Heterogeneous deployment/Train deploys

From Wikitech

Tuesday: New branch creation and deploy

Create the new branch in gerrit

The new branch can be created in gerrit from anywhere. It is often faster to do this step on a host in the cluster to minimize the time needed to clone from gerrit.

Warning Warning: For now, make-wmf-branch requires a newer version of php than is on the deploy host (tin), so running this on your laptop is required. See this comment in phab

  • git clone ssh://gerrit.wikimedia.org:29418/mediawiki/tools/release.git
  • cd release/make-wmf-branch
  • ./make-wmf-branch -n VERSION -o master
    • (e.g. php make-wmf-branch -n 1.27.0-wmf.9 -o master)

Remove symlinks to expired branches

Any branch checkout on the deployment server that has not been used for more than 5 weeks can be safely removed to reduce disk usage across the cluster.

  • /srv/mediawiki-staging/multiversion/deleteMediaWiki STALE_VERSION
  • git rm -r docroot/bits/static-STALE_VERSION
  • git rm -r w/static-STALE_VERSION
  • DOLOGMSGNOLOG=1 git commit -m "Remove STALE_VERSION symlinks"

Clone new branch

  • On tin, to create a new /srv/mediawiki-staging/php-VERSION directory run:
    • /srv/mediawiki-staging/multiversion/checkoutMediaWiki VERSION php-VERSION
    • (e.g. /srv/mediawiki-staging/multiversion/checkoutMediaWiki 1.27.0-wmf.9 php-1.27.0-wmf.9)

Apply security patches

  • Check /srv/patches/core/*.patch for any patches for /srv/mediawiki-staging/php-VERSION
  • Check /srv/patches/extensions/EXTENSION-NAME/*.patch for any patches for /srv/mediawiki-staging/php-VERSION/extensions/EXTENSION
  • Can use git apply --check PATCH_FILE to check if patches apply cleanly (e.g., cd /srv/mediawiki-staging/1.27.0-wmf.9; git apply --check /srv/patches/core/TXXXXXX.patch
  • Some patches may require git apply --check --3way to apply cleanly
  • If any patch doesn't apply cleanly, contact Chad (ostriches) or Chris (csteipp) in IRC.
  • There is never a patch that shouldn't apply in /srv/patches if that patch has merged it should be removed.

Create symlink patch for common/docroot

  • Create symlinks patch for changes made in common/docroot and common/w by checkoutMediaWiki
    • cd /srv/mediawiki-staging
    • git add docroot/bits/static-VERSION w/static-VERSION
    • DOLOGMSGNOLOG=1 git commit -m "Add VERSION symlinks"

Create patches to update wikiversions.json

  • Create group0 to VERSION patch
    • /srv/mediawiki-staging/multiversion/updateWikiversions group0 php-VERSION
      • (e.g. /srv/mediawiki-staging/multiversion/updateWikiversions group0 php-1.27.0-wmf.9)
    • git add wikiversions.json
    • DOLOGMSGNOLOG=1 git commit -m "Group0 to VERSION"
      • (e.g. DOLOGMSGNOLOG=1 git commit -m "Group0 to 1.27.0-wmf.9")

Send staged patches to gerrit for review

  • cd /srv/mediawiki-staging
  • git push origin HEAD:refs/for/master/VERSION
    • (e.g. git push origin HEAD:refs/for/master/1.27.0-wmf.9)

Discard changes to working directory and index on tin

  • cd /srv/mediawiki-staging
  • DOLOGMSGNOLOG=1 git reset --hard origin/master

Restore symlinks on deployment server

  • Review and submit symlinks patches in gerrit
    • This includes the patches for any expired branches that were removed and the new branch that was added.
  • Wait for gerrit/zuul/jenkins to merge the patch(es)
  • Pull patch(es) to deployment server
    • cd /srv/mediawiki-staging
    • git fetch
  • Check diff to ensure it is what you expect
    • git diff HEAD..origin/master
  • Apply changes
    • git rebase origin/master

Remove clones of expired branches

For each branch that you removed symlinks for, remove the clone from /srv/mediawiki-staging:

  • rm -rf /srv/mediawiki-staging/php-STALE_VERSION

Sync to cluster and verify on testwiki

  • Edit /srv/mediawiki-staging/wikiversions.json and set testwiki to php-VERSION
    • Do not commit and push to gerrit, only make this change locally on the deployment seerver
  • Run scap to (re)build localization caches and sync changes across the cluster
    • scap "testwiki to php-VERSION and rebuild l10n cache"
  • Verify version change and l10n cache on testwiki
  • Revert local changes: git checkout -- wikiversions.json

Remove left over files from expired branches

Because we don't rsync the localization CDB files during scap, some files will be left behind after scap runs for expired branches that were removed. Clean them up across the cluster using dsh

  • SSH_AUTH_SOCK=/run/keyholder/proxy.sock dsh -F 20 -M -g mediawiki-installation -r ssh -o -oUser=mwdeploy -- rm -rf /srv/mediawiki/php-STALE_VERSION

Wait for deploy window

All of the changes above can be done at any time prior to the actual deployment window. Depending on how practiced you are and where you choose to run commands (full clones of mediawiki-core from outside the cluster can take a while) the steps up to this point will typically take 60 to 120 minutes.

Switch group0 wikis to VERSION

  • Review and submit group0 to VERSION patch in gerrit
  • Wait for gerrit/zuul/jenkins to merge the patch(es)
  • Pull patch(es) to deployment server
    • cd /srv/mediawiki-staging
    • git fetch
  • Check diff to ensure it is what you expect
    • git diff HEAD..origin/master
  • Apply changes
    • git rebase origin/master
  • Sync the change across the cluster
    • sync-wikiversions "group0 to VERSION"
      • (e.g. sync-wikiversions "group0 to 1.24wmf4")
  • Verify that mediawikiwiki switched to the new version
  • Monitor irc and logstash and/or fatalmonitor for problems

Purge localization cache for now unused versions

  • scap-purge-l10n-cache --version VERSION-2
    • (e.g. scap-purge-l10n-cache --version 1.24wmf2)

Update deploy notes

Setup the tools you need to do this:

For the new branch and any branches that may have changed in the last week:

  • Check out the branch locally
    • git checkout wmf/VERSION
    • (e.g. git checkout wmf/1.24wmf4)
  • Update the submodules
    • git submodule update --init --recursive
  • Run the change log generation and upload script
    • php path/to/make-deploy-notes/uploadChangelog.php wmf/VERSION
      • (e.g. php path/to/make-deploy-notes/uploadChangelog.php wmf/1.24wmf4)
  • Repeat for additional branches as needed

Update roadmap

Wednesday: group0 to group1 deploy

  • Clone the MediaWiki release tools to your home directory on tin
  • Use the deploy-promote script inside ~/release/bin/ to update wikiversions.json
    • You will be prompted to verify the update, then you will be asked to approve relevant patches in gerrit
    • ~/release/bin/deploy-promote
      • Promote group1 from VERSION-1 to VERSION? Enter to continue, ctl-c to cancel:
    • Once you have +2'd the patches in gerrit, and they have merged into the config repo, hit enter at the 2nd prompt
      • Now merge the patch and press enter to continue with git pull && sync-wikiversions
    • After the script run is complete, group1 wikis should be running VERSION

Thursday: group{0,1} to all deploy

  • Thursday deploy is very similar to the Wednesday deploy, the only difference in terms of procedure is the target group
  • Clone the MediaWiki release tools to your home directory on tin
  • Use the deploy-promote script inside ~/release/bin/ to update wikiversions.json
    • You will be prompted to verify the update, then you will be asked to approve relevant patches in gerrit
    • ~/release/bin/deploy-promote all
      • Promote all from VERSION-1 to VERSION? Enter to continue, ctl-c to cancel:
    • Once you have +2'd the patches in gerrit, and they have merged into the config repo, hit enter at the 2nd prompt
      • Now merge the patch and press enter to continue with git pull && sync-wikiversions
    • After the script run is complete, all wikis should be running VERSION
Retrieved from "https://labtestwikitech.wikimedia.org/w/index.php?title=Heterogeneous_deployment/Train_deploys&oldid=9208"