Nova Resource:Deployment-prep/Overview

Reviewed on May 2014

The mw:Beta Cluster project reproduces a good part of the WMF production cluster so we can use it as a test and staging area for code changes and new extensions. It runs on the wmflabs.org virtual cluster which provides us with a way to easily create virtual machines that are automatically configured by puppet using our puppetmaster. The MediaWiki and MediaWiki extensions are constantly up to date to the tip of their master branches.

Architecture

DNS entries are regrouped under the `beta.wmflabs.org` DNS subdomain and point to 5 public IP (as of July 2014). DNS and public IP allocations are entirely managed through Special:NovaAddress.

Static assets are served by varnish on the deployment-cache-bits01 instance which serves the bits.beta.wmflabs.org domain.

Wikis follow the <language>.<project>.beta.wmflabs.org hostname schema. The beta English Wikipedia is thus served via http://en.wikipedia.beta.wmflabs.org/.

The mobile version uses <language>.m.<project>.beta.wmflabs.org like in production. The mobile version of enwiki is thus at http://en.m.wikipedia.beta.wmflabs.org/.

The text traffic (*.beta.wmflabs.org) is served by the deployment-cache-text02 instance which load balance requests to two backend Apaches. Apaches are sharing their configuration and MediaWiki install over a shared NFS file system mounted at /data/project/ on the deployment-bastion.

The mobile traffic (*.m.*.beta.wmflabs.org) is served by the deployment-cache-mobile03 instance. It uses as backend the same Apaches as for the text traffic.

MediaWiki and its extensions are running out of their master branches and are updated automatically from time to time. See How code is updated for more details.

Data can be cached in the memcached instances (deployment-memc04 and deployment-memc05) which run on dedicated instances or are fetched from the MariaDB database hosted on deployment-db1.

Files required by the Apaches (configuration, MediaWiki code, MediaWiki configuration) are available via the shared NFS directory. This mean that any modification made to the files are instantly deployed upon saving the file, that makes things easy to alter but also very easy to break.

Files and thumbnails are served by a Varnish upload cache (deployment-cache-upload02). It uses a specific backend (deployment-upload) which serves the files and generates the thumbnails. As of May 2014 that server is unpuppetized and it runs an nginx server to invoke the thumb handler. This will be migrated to Swift eventually.

Components

This section describe the components for each layer starting from page caches till the database.

Varnish (text cache)

The text cache is served by Varnish on the deployment-cache-text02 instance.

The load balancing is done by having Apaches listed as Varnish backends in the puppet cache configuration.

Varnish (bits cache)

bits.beta.wmflabs.org is served by deployment-cache-bits01. It relays all uncached queries to the backend Apache virtualhost bits.beta.wmflabs.org.

MediaWiki is asked to send traffic to it by configuring $wgResourceBasePath, $wgExtensionAssetsPath and $wgStyleSheetPath in operations/mediawiki-config.git:wmf-config/CommonSettings.php.

Apaches

The Apaches serve MediaWiki requests. The document root is /usr/local/apache/common which is a shared NFS directory.

The configuration is available in operations/apache-config.git, in the branch betacluster. It is kept up to date by puppet, you will have to reload the apaches manually though (as of May 2014) (see also bug 36422 - easily reload all apaches).

Logs

Various server logs are written to the remote syslog server deployment-fluorine.eqiad.wmflabs in /srv/mw-log. This matches the production cluster setup. Logs describes some of the production logs, and many have equivalents on the beta cluster.

Apache access logs are written to /var/log/apache2/other_vhosts_access.log on each host.

Configure

See also how code is updated

The work machine is deployment-bastion.eqiad.wmflabs, basically an equivalent to fenari in production. (To access machines on our internal wmflabs network, see Help:Access.)

File structure

Project files are hosted on a NFS server, mounted at /data/project like other WMF labs projects.

/data/project/apache/common => operations/mediawiki-config.git
/data/project/apache/common/php-master => mediawiki/core.git
/data/project/apache/common/php-master/extensions => mediawiki/extensions.git
/data/project/apache2 (Apache configuration)
/data/project/logs (udp2log)
/data/project/upload7 (uploaded files + thumbs)

Thus to view beta cluster logs such as exception.log and fatal.log, ssh deployment-bastion.eqiad.wmflabs and view /data/project/logs.

The git repositories are maintained automatically by Jenkins jobs, see how code is updated.

MediaWiki config

The MediaWiki configuration files are located on /data/project/apache/common. It is a clone of the operations/mediawiki-config.git repository and is maintained automatically by Jenkins (see how code is updated). Any local hacks/changes will get reverted when Jenkins updates the code, so if you do any hack, make sure to send it in Gerrit or your change will be lost.

You will mostly want to add your configuration changes in wmf-config/InitialiseSettings-labs.php or wmf-config/CommonSettings-labs.php and it will need to be reviewed via Gerrit. Once merged the change will be applied automatically.

There is a single local hack which is wmf-config/PrivateSettings.php. It holds secret keys and passwords. git happily ignores the file and it must not be made public.

Database connection

To connect to a database use the mwscript wrapper and MediaWiki sql.php script:

mwscript sql.php --wiki=enwiki

New extension

Warning: new extensions should only be deployed on the 'beta' cluster after they have been scheduled for production deployment. 'beta' is not meant to be a testing/alpha area, you will want your own project and instance if you want to try out an extension.

Extensions live in /data/project/apache/common-local/php-master/extensions, which is a clone of mediawiki/extensions.git. Which mean that to get an extension deployed on the beta cluster it must be registered as a submodule of the mediawiki/extensions.git repository. It will then be automatically applied.

You will need to do all the steps needed for a production deployment:

1. add a configuration snippet in operations/mediawiki-config.git:wmf-confg/CommonSettings-labs.php
2. add the entry point for the message cache to load the i18n files. It would be added to wmf-config/extension-list or extension-list-labs (if it is labs only)

Once the change is merged, the Jenkins jobs will take care of refreshing the l10n cache and invalidating the serialized configuration by touching InitiaseSettings.php.

You might have to apply schema changes, for now with update.php --wiki=SOMEWIKIDB. The foreachwiki script might work too. A Jenkins job also does this hourly (see how code is updated).

New wiki

Duplicate content with Nova_Resource:Deployment-prep/Add_a_wiki.

need to be expanded

Basically, we do not create any new wiki yet. A plan is to be able to setup wiki dedicated to staging a new extension. We most likely will probably create them using the addWiki.php script which should work (mwscript addWiki.php <options>). Still need to find out a naming scheme, maybe ext-<name>-wiki and bind that to <name>.extension.beta.wmflabs.org.

The mwscript wrapper is installed on deployment-dbdump so one should use addWiki.php.

# Creating a new beta cluster wiki. Must be logged in as mwdeploy
# Private 
sudo -u mwdeploy -- mwscript addWiki.php --wiki=aawiki en wikimedia zerowiki zero.wikimedia.beta.wmflabs.org
sudo -u mwdeploy -- mwscript addWiki.php --wiki=aawiki ru wikipedia ruwiki ru.wikipedia.beta.wmflabs.org

# SQL manipulation. sudo is not needed.
mwscript sql.php --wiki=aawiki
> drop database <dbname>    # be careful here, make sure you re-create it afterwards

Edit the wmf-config git repo, +2. Deploying to production is a good practice, but not required.

• add-labs.dblist
• wikiversions-labs.json
• wmf-config/InitializeSettings-labs.php - $wgServer

Edit the apache-config git repo, betacluster branch and restart apache (ssh to deployment-apache01 & 02, service apache2 restart)

• Added <virtualhost> to wikimedia.conf (make sure it is no longer a sym-link)
• Added wikimedia.conf to sites.conf

# Flushed varnish cache on deployment-cache-text02
sudo varnishadm "ban req.url ~ /" && sudo varnishadm -n frontend "ban req.url ~ /"

# Add a user account to the new wiki from deployment-bastion
# Do not type your actual password, it will be stored in public history
# use something temporary and change it after logging in
mwscript maintenance/createAndPromote.php --wiki=zerowiki --sysop --username myusername --password temp