Heterogeneous deployment

It has been suggested that this page should be merged with How to deploy code

See also: Heterogeneous deployment/old for information about older setup

This document describes the current deployment infrastructure, which allows for different wikis to be running different MediaWiki versions as the same time.

Where things live

WMF configuration lives in /srv/mediawiki-staging/wmf-config on the deployment host 'tin', and tools such as scap sync this directory to /usr/local/apache/common-local on cluster machines. (The same common directory also has subdirectories for each live branch of the MediaWiki codebase.) Most files are maintained in the git repository operations/mediawiki-config.

mediawiki-staging/

wikiversions.json: a JSON object mapping wikiname -> version. Versions must have a "php-" prefix, like "php-1.24wmf2". This is here for admins to edit as needed. Note that addwiki.php updates this automatically (rebuilding the cdb too).
wikiversions.cdb: a cdb database of the wikiversions.json file. Never edit this directly.

mediawiki-staging/multiversion

Note: these files are all checked out from operations/mediawiki-config/multiversion

MWMultiVersion.php: Class for determining the database name and version of a wiki. Supports loading from three entry points:
- Web access of an uploaded file through apache - setSiteInfoForUploadWiki()
- Web access of a other wiki content through apache - setSiteInfoForWiki()
- From within a maintenance script - setSiteInfoForMaintenance()
MWScript.php: Wrapper for running the right version of a maintenance script for a wiki.
MWVersion.php: Contains two functions. Both locate the path of a MW file for a given wiki, set $IP, and set PHP's working directory to $IP. One functions is for web hits through apache, the other handles maintenance scripts.
activeMWVersions: This returns a space separated list of items based on the common/wikiversions.json file. If --home is set then it uses the file on /home instead of /usr. If the --withdb param is set, then each item in the list is the form "version=wiki", otherwise they are of the form "version". The versions do not have a "php-" prefix. This is used for setting bash arrays to iterate over different MW versions.
checkoutMediaWiki: git clone a new installation from the wmf/<version> branch. It creates some files and symlinks to common/ that the install will need in order to be used.
refreshWikiversionsCDB: rebuilds the wikiversions.cdb database from the common/wikiversions.json file.
deleteMediaWiki: delete symlinks to a wmf/<version> branch that is no longer needed for a live wiki or the varnish cache.
updateWikiversions: switch all wikis in a .dblist to a given MediaWiki version in the common/wikiversions.json file.

mediawiki-staging/wmf-config

This is were most of the site configuration lies. Main files include things like:

CommonSettings.php: Mostly handles shared settings or settings with few exceptions. Also handles the task of getting the InitialiseSettings.php options into globals that can be used for MW, and the caching thereof. Site, language, and DB name globals are extracted from the MWMultiVersion object. A MWMultiVersion object should always be initialized by the time this file is loaded. It also sets the i18n message directory locations for extensions. Other random things are performed here.
InitialiseSettings.php: Mostly contains MW configuration settings for individual wikis as well as wiki groups (like all "wikibooks" for example).
extension-list: The list of extensions available for use on the site (though some may not be enabled at the moment). These extensions must be installed on cluster. This list is used to automatically generate the ExtensionsMessages-*.php files needed for localization.
CommonSettings-labs.php, InitialiseSettings-labs.php, and other *-labs files contain overrides for the Beta Cluster (deployment) cluster.

mediawiki-staging/php-*

This is where the various MW directories lie.

/usr/local/bin

See Wikimedia binaries for a more exhaustive list.

RefreshWikiversionsCDB: wrapper script to call the RefreshWikiversionsCDB in multiversion/.
scap: script to sync all hosts with the state of the deployment server (tin.eqiad.wmnet)
sync-wikiversions: script to rebuild wikiversions.cdb (by running RefreshWikiversionsCDB), sync wikiversions.json and wikiversions.cdb, and log the action.

How to do stuff

General remarks on deploying code

Basic common sense

Be careful. Breaking the site is surprisingly easy!
If you're deploying code written by someone else, ask them to be around during deployment so they can troubleshoot if necessary.
Make sure you know about anything hairy, such as additional prerequisites (e.g. schema changes) or potential complications when rolling back.

Deployment Requirements

Cluster account request through Phabricator (set Security: Access Request!); request deployment access (requires manager and/or sr dev approval).
- If you can ssh into tin, and ssh into a random mw box (e.g. mw1001) from there, you already have this.
Deployment branch merge access requested from an existing user in the group.
Some shiny new code with which to terrorize end-users.

Don't leave town

Even if your deploy appears to be working, it's important to be reachable in the hours immediately following your deploy. Ideally, stay online and in IRC channels like #wikimedia-tech for a couple of hours.

If you must go offline, let people know how to reach you (and keep your mobile phone or other communications device on your person). You can use /away messages on IRC, or perhaps send a short email to the ops list.

If you are on Wikimedia staff, now might be a great time to check if your contact info is up to date. If you aren't on staff, ask a staffer to add your contact info to that page, under "Important volunteers".

Basic code deployment steps

Really, you should check out the How to deploy code page. It is the canonical place for documentation on how to deploy code, obviously.

Change wiki configuration

See Configuration files for a list of configuration files and their formats

In your own repo via gerrit

If you need someone else to merge and push your change, follow this path.

for the person making the change
- in your own checkout of the operations/mediawiki-config project, make the changes you need
- push the change to gerrit for review
for the person merging and pushing the change live
- on tin, cd to /srv/mediawiki-staging/
- run git status and verify there are no pending changes. Untracked files are ok.
- in gerrit: rebase, review, and merge the change
  - The rebase is to make sure the merge commit below isn't a false positive. Old patches will create FF-only merge commits.
- on tin, run git fetch origin && git diff HEAD origin. Review the diff and make sure it's only the changes that you expect
  - if more than the changes you're making show up, STOP and find out why and make sure they're ok to merge.
- if it looks good, run git merge origin/master
  - if this creates a merge commit, that means someone has committed a local change without pushing; this isn't the end of the world but it should be fixed, so either fix it yourself or ask for help
- run sync-file wmf-config/filename 'Summary of change' for each changed file separately, sync-dir wmf-config 'Summary of change' if a lot of files in one directory changed, and/or sync-dir dblists/ 'Summary of change' if multiple dblist files changed

Change on tin then commit

If you are going to make the change, review, and push all by yourself, follow this path.

Edit the file(s) you need to edit.
cd to /srv/mediawiki-staging/
Run git diff wmf-config , which will display a diff of recent changes to the wmf-config directory. Verify that these changes are sane and that there are no traces of someone else messing with the same files
If everything looks good, run sync-file wmf-config/filename 'Summary of change' for each changed file separately, sync-dir wmf-config 'Summary of change' if a lot of files in one directory changed, and/or sync-dir dblists/ 'Summary of change' if multiple dblist files changed
Run git commit wmf-config/filename and add a decent commit summary. Then do git push origin HEAD:refs/for/master
Do not forget the git push step above, things will get messed up if you do (and nobody can review your changes in Gerrit)

Run a maintenance script on a wiki

To run a maintenance script, log into terbium, go into the directory for the version of MediaWiki that you want to run the maintenance script from (for example php-1.22wmf2):

cd /usr/local/apache/common/php-1.22wmf2

Use the mwscript command-line utility:

mwscript scriptname.php dbname

The above command will run the script scriptname.php located in /maintenance since no directory was specified. You can also use a path relative to the MediaWiki installation, for example:

mwscript extensions/MyExtension/maintenance/scriptname.php dbname

Internally, this executes /home/wikipedia/common/multiversion/MWScript.php. It will run the version of the maintenance script in /home/wikipedia/common. If you want to run the version under /usr/local/apache/common-local/, or you're on a box without /home mounted, you need to change to /usr/local/apache/common-local/multiversion and run ./MWScript.php instead.

Example:

$ mwscript eval.php enwiki
> print $wgDBname;
enwiki
> print $wgVersion;
1.18wmf1
> exit
$

If you need to pass parameters to your maintenance script, you can add them to the end. You may also want to explicitly declare the database parameter to avoid confusion:

mwscript scriptname.php --wiki=enwiki --days=1

In this case, the 'wiki' parameter is handled by mwscript and the 'days' parameter is handled by scriptname.php. The "wiki" must be in wikiversions.cdb so the maintenance script machinery can figure out what MediaWiki version it is running.

Run a maintenance script on a group of wikis

If you want to run a maintenance script on all wikis in a .dblist file, you can issue the following command from anywhere:

mwscriptwikiset scriptname.php listname.dblist

This will run the script scriptname.php located in /maintenance since no directory was specified. As with MWScript.php and mwscript, you can also use a script path relative to the MediaWiki installation, for example:

mwscriptwikiset extensions/MyExtension/maintenance/scriptname.php listname.dblist

The .dblist file must be located in /usr/local/apache/common-local.

As with mwscript, mwscriptwikiset will run the "home" version of the maintenance script found in /home.

Switch a wiki to a different MW version

First make sure that the new MW version exists. If not, see the section of this page concerned with checking out new versions.
First check if another wiki is already using the version. You can use the command mwversonsinuse --withdb. Note that this will only return *one* wiki using the version, not all of them.
If no other wiki is already using the MW version:
- Locate the testwiki entry in wikiversions.json (on /a/) and change it to the new MW version. Make sure it has 'php-' in it.
- Do cd /php-<MW VERSION>/extensions/TrustedXFF and then run php /srv/mediawiki-staging/multiversion/MWScript.php extensions/TrustedXFF/generate.php <WIKI NAME>
- Run scap. This should update ExtensionMessages-*.php for the version.
- Run dsh -F30 -cM -g mediawiki-installation -o -oSetupTimeout=10 'sudo -u mwdeploy /usr/bin/scap-recompile' to compile binaries for the version, like texvc.
- Check that testwiki is operational.
Edit the wikiversions.json file by finding the wiki's DB name and changing the corresponding version value. Make sure it has 'php-' in it.
Run sync-wikiversions [DESCRIPTIVE MESSAGE] to rebuild the cdb from the json file and sync them.

Install a new extension on a wiki

If you're deploying an extension or feature that can be switched off, it's usually best to leave it switched off while you deploy and carefully switch it on after that using a simple configuration change (this is called a dark launch). Even if you do this, you should build any configuration infrastructure (e.g. $wmg variable, adding entry in InitialiseSettings with default false) at this time so all you'll have to do later is flip a switch.

For specific preparations, see the sections below as well as How to do a schema change and How to do a configuration change. Best to perform schema changes before making config changes.

Add a configuration switch for an extension

In /srv/mediawiki-staging/wmf-config/CommonSettings.php, add:

if ( $wmgEnableMyExtension ) {
	require_once( "$IP/extensions/MyExtension/MyExtension.php" );
	// Set config vars if needed

	// If you want to export config vars through InitialiseSettings.php, you need to set $wmgMyExtensionThingy there and do
	#$wgMyExtensionThingy = $wmgMyExtensionThingy;
}

In /srv/mediawiki-staging/wmf-config/InitialiseSettings.php, add something like:

'wmgEnableMyExtension' => array(
	'default' => false,
	'eswikibooks' => true,
	// etc.
),
// If needed, set $wmgMyExtensionWhatever vars here too

If your extension requires a large-ish amount of configuration, consider putting it in a separate file instead. Currently, AbuseFilter, LiquidThreads and FlaggedRevs do this.

For more documentation on these files and their formats, see Configuration files.

Add new extensions to extension-list

When adding a new extension, you need to add it to the extension list, or its i18n messages won't get picked up. For more information about this setup, see this.

cd to /srv/mediawiki-staging/wmf-config
Edit extension-list and add the path to the extension setup file on a line by itself
Where X.XX is the current MediaWiki version, run mwscript maintenance/mergeMessageFileList.php --wiki=aawiki --list-file=extension-list --output=ExtensionMessages-X.XX.php and watch the output for errors. Your extension should appear in the output the same way the others do
- If you get an error, you probably got the path wrong, or the extension setup file is wonky
- If you get an error, you can also try running /home/catrope/sync-common , then running mergeMessageFileList.php again like above

Deploy the extension

Do what needs to be done to deploy and register changes in these config files. You will need to run scap afterwards (so the config and message cache is updated).

Add to make-wmf-branch

To have your extension added to newly created WMF branches, please add the newly added extension to make-wmf-branch/config.json in the release tools repository.

Make a wiki use a special version of an extension

This scenario occurs when we have MyExtension/ and MyExtensionBeta/ in one of the MW deployment directories, say php-X.XX, and we want some wikis using one extension version while other wikis use the other.

Edit wikiversions.json by finding the row with [target wikidb] [version] and changing it to [target wikidb] [version] [target wikidb]. This defines the wiki as having a separate i18n cache.
If this was already in wikiversions.json:
- Edit CommonSettings.php to include the desired extension version if $wgDBname equals the target wiki and sync it
...otherwise, perform the following steps:
- Run refreshWikiversionsCDB to rebuild wikiversions.cdb.
- cd to /srv/mediawiki-staging/wmf-config
- Edit CommonSettings.php to include the desired extension version if $wgDBname equals the target wiki
- Run mwscript mergeMessageFileList.php --wiki=TARGETWIKI --list-file=extension-list --output=ExtensionMessages-MWVERSIONFORTHATWIKI-TARGETWIKI.php and watch the output for errors. MWVERSIONFORTHATWIKI should not include "php-", it should be like "X.XX".
  - If you get an error, you probably got the path wrong, or the extension setup file is wonky.
  - If you get an error, you can also try running /home/catrope/sync-common , then running mergeMessageFileList.php again like above.
- Check that a new ExtensionMessages-MWVERSIONFORTHATWIKI-TARGETWIKI.php file exists in wmf-config/ and makes sense. If it does, then svn commit the file.
- Run scap.

Switch all wikis of one version to a different MW version

First make sure that the new MW version exists. If not, see the section of this page concerned with checking out new versions.
If no other wiki is already using the MW version then you probably should stop what you're doing.
Run /srv/mediawiki-staging/multiversion/updateWikiversions all php-NEWVERSION to switch all the wikis to using a new version. This can be for upgrades or for downgrades. This will rebuild the wikiversions.json and wikiversions.cdb files.
Run scap.

Rebuild the wikiversions.cdb file

Run refreshWikiversionsCDB to rebuild the local cdb from the json file. More commonly you will want to run sync-wikiversions to compile and sync wikiversions.cdb to all hosts in the cluster.

Create a new MediaWiki version

See the Train deploys subpage for full details.