Nova Resource:Deployment-prep/How code is updated

From Wikitech

The cluster runs the master branch of MediaWiki core and extensions.

The extensions are deployed using the Gerrit project mediawiki/extensions.git in which extensions are manually registered as submodules. The Jenkins job beta-code-update-eqiad runs on the deployment-bastion to update MediaWiki core and all the submodules under /extensions. The job then refreshes the message cache and start again every ten minutes.

An overview of all the jobs for this project is available under the Jenkins 'Beta' view as well as on the Continuous Integration dashboard. As of May 2014 the jobs are:

beta-code-update-eqiad (jenkins)
Update MediaWiki core and extensions to their master and trigger the beta-scap-eqiad Jenkins job.
beta-mediawiki-config-update-eqiad (jenkins)
Update the working copy of operations/mediawiki-config.git which hold the MediaWiki settings for both production and labs and trigger the beta-scap-eqiad Jenkins job.
beta-scap-eqiad (jenkins)
Run scap to update localization cache and copy MediaWiki core, extensions and settings to the rest of the beta cluster.
beta-parsoid-update-eqiad (jenkins)
Job to keep in sync parsoid. Whenever a change is made to the mediawiki/services/parsoid or mediawiki/services/parsoid/deploy repository, the code is updated on deployment-parsoid04.eqiad.wmflabs instance and parsoid is restarted via an upstart script.
beta-update-databases-eqiad (jenkins)
A job that run through a hardcoded list of database and run the update.php script against each of them on a hourly basis.
beta-recompile-math-texvc-eqiad (jenkins)
Recompile texvc from the Math extension whenever a commit is made there.

The job status can be seen on the Wikimedia Continuous integration portal or by looking at the Beta Jenkins view

Puppet and Salt

We are using a self-hosted puppermaster and a local salt master in the project.

Converting a host to use local puppetmaster and salt master

This has been automatically set up with hiera and ENC (See Hiera::deployment-prep). Just run puppet a few times and wait :)

Updating puppet manifests to match production

To update puppet configuration you will need to log into deployment-puppetmaster and update /var/lib/git/operations/puppet manually before forcing the puppet run. We've been doing this:

  1. ssh deployment-puppetmaster.eqiad.wmflabs
  2. sudo -s
  3. cd /var/lib/git/operations/puppet
  4. git fetch
  5. git rebase origin/production

Cherry-picking a patch from gerrit

The major benefit of using the local puppet master in the beta project is the ability to quickly add and test new puppet configuration before getting someone with +2 in operations/puppet.git to merge. This is often needed because an ugly temporary fix is needed for beta while a nicer log term solution is worked out for production.

We use cherry-picks rebased on top of the current operations/puppet.git production branch because git is smart about these sorts of changes and will recognize on subsequent git rebase runs patches that have been merged upstream.

  1. Create a patch against operations/puppet.git
  2. git review
  3. Go to review in gerrit and find the "cherry-pick" git command that gerrit generates for the patch. It will look something like: git fetch refs/changes/88/125888/1 && git cherry-pick FETCH_HEAD
  4. ssh deployment-puppetmaster.eqiad.wmflabs
  5. sudo -s
  6. cd /var/lib/git/operations/puppet
  7. git fetch
  8. git rebase origin/production
  9. Paste cherry-pick command from gerrit: git fetch refs/changes/... && git cherry-pick FETCH_HEAD
    • Occasionally there will be a conflict resulting from the cherry-pick; figure out how to resolve it if that happens or git cherry-pick --abort to roll back the attempt
    • You may find that you need to go back to your local working copy and rebase your patch on top of another cherry-pick that is already pending review in gerrit to cleanly resolve the conflict.
  10. Ssh to a host that the patch should effect and force a puppet run with sudo puppet agent --test --verbose
  11. Log the change (i.e. go to #wikimedia-releng connect and use "!log").

Updating/removing a cherry-pick

When you find a bug in your proposed patch (it happens to the best of us), you'll want to remove the old cherry-picked patch and possibly add an amended one.

  1. ssh deployment-puppetmaster.deployment-prep.eqiad.wmflabs
  2. sudo -s
  3. cd /var/lib/git/operations/puppet
  4. git rebase --interactive
  5. Find the line for the cherry-pick you want to remove/update and delete it from the git-rebase-todo file that opens in your editor
  6. Save the file/close your editor (:q in vim; you do use vim right?)
  7. Cross you fingers that the rebase completes with no errors
    • Clean up git with git rebase --abort if there are conflicts you can't figure out how to resolve.

If you wanted to update the cherry-pick, go back to the Cherry-picking a patch from gerrit instructions now that the old cherry-pick is removed.


How do I get my code on the beta cluster?

To get your code deployed, you thus just have to have it merged in the master branch. It will eventually land on beta after a few minutes. Have a look at the Jenkins beta-code-update-eqiad job.

How does MediaWiki configuration get updated?

We use a Jenkins job that reacts on merges happening in operations/mediawiki-config.git. That triggers a Jenkins job beta-mediawiki-config-update-eqiad which connects to the bastion instance and checkout the merged revision. The job invalidates the cached configuration by touching InitialiseSettings.php.

How does the database get upgraded?

If your change is supported by MediaWiki update.php, it will be applied via the hourly Jenkins job beta-update-databases-eqiad. The job spawn a child job for each of the databases. They will simply run upgrade.php.

How to add a new extension on beta

At first, beta is a staging area, it is not meant to be a development playground. The extension you want to add to beta must have been planned for deployment on production. If you want a dev sandbox, you should create an instance in a different labs project.

Steps to deploy a new extension on beta cluster:

  • Make sure you have tests registered in Jenkins / Zuul. You don't want to deploy a php linting error
  • Verify the extension is registered as a submodule of mediawiki/extensions.git
  • Register the extension in operations/mediawiki-config.git
    • Add entry point in /wmf-config/extension-list-labs (used to generate l10n messages)
    • Add a require_once() in CommonSettings-labs.php
  • Eventually create a global $wmgUse<your extension name>
  • Create settings using either CommonSettings-labs.php or InitialiseSettings-labs.php

Whenever your change to operations/mediawiki-config.git is merged, it will be deployed by Jenkins job beta-mediawiki-config-update-eqiad as described above.

Remember: you don't want the extension to be enabled in production if it is not ready / added to wmf branches yet.

My code introduces a feature that is not yet ready for production, should I wait to merge in master?

You definitely do not want your experiment to land on production which would potentially cause a lot of havoc for our millions of readers. A best practice we have been following for more than a decade is to safe guard your new feature using a MediaWiki global setting. Having a setting set to false by default, would make sure the feature is not going to be enabled on production. You could then explicitly enable it on beta by editing the wmf-config/InitialiseSettings-labs.php file of operations/mediawiki-config.git.

Example highlighting how to safeguard a new feature:

$wgEnableFeatureOne = false;

function something() 
  global $wgEnableFeatureOne;
 // some code
 if( $wgEnableFeatureOne) {
  // Your new code
 // some more code

Then to enable this feature on labs:

// wmf-config/InitialiseSettings-labs.php under wmfLabsSettings()
  'wgEnableFeatureOne' =>array( 'default' => true ),

Get it merged. Jenkins deploys the new configuration, the feature is enabled \O/