LocalisationUpdate

Purpose

This MediaWiki extension fetches updates to MediaWiki core and extension localization files and makes them available to an installation without updating its own source. This allows us to pull updated translations being committed to development trunk even though our live install is running a deployment branch.

Source

• http://www.mediawiki.org/wiki/Extension:LocalisationUpdate

Big picture overview

• Translators tear it up on translatewiki.net
• translatewiki.net staff commit translations to trunk
• LocalisationUpdate cron job runs every night at 02:00 UTC on tin.eqiad.wmnet
  • Pulls translations from local git clones and writes updated messages to a cache directory
  • Rebuilds the l10n cache(s) including updates found above
• Syncs the l10n cache(s) to apaches

What the cron job on tin does

• Runs as the l10nupdate user. This user has a passphraseless SSH key
• Updates the git clones in /var/lib/l10nupdate/mediawiki/core and /var/lib/l10nupdate/mediawiki/extensions
• Runs /srv/mediawiki-staging/php-$VERSION/extensions/LocalisationUpdate/update.php for each MediaWiki version
  • For core and for each extension, this script reads the i18n file(s) from the trunk checkout and compares them with the current translations on the cluster
  • If a message has a new/updated translation, and the English version of the message has not changed, updates that message
  • Writes message updates for each language to /srv/mediawiki-staging/php-$VERSION/cache/l10n/l10nupdate-$LANGCODE.json
• Runs /usr/local/bin/sync-l10nupdate which syncs /srv/mediawiki-staging/php-$VERSION/cache/l10n on tin to /srv/mediawiki/php-$VERSION/cache/l10n on the Apaches.
• Runs /srv/mediawiki-staging/php-$VERSION/maintenance/wmf/clearMessageBlobs.php, which clears message blobs cached by ResourceLoader
• Logs to /var/log/l10nupdatelog/l10nupdate.log in append mode. Log file is rotated by logrotate.
• logmsgbot !log's success or failure in #wikimedia-tech

Running LU manually

Run l10nupdate on tin.eqiad.wmnet. Syncing the message updates requires deploy access (i.e. ssh access to the Apaches).

Files

• /usr/local/bin/l10nupdate-1
  • Script that does all the work. The cron job runs this as the l10nupdate user
• /usr/local/bin/l10nupdate
  • Wrapper that runs l10nupdate-1 as the l10nupdate user using sudo
• /var/lib/l10nupdate/mediawiki
  • MediaWiki checkouts of trunk core and extensions
• /srv/mediawiki-staging/php-$VERSION/cache/l10n
  • Storage for serialized updates
• /srv/mediawiki/php-$VERSION/cache/l10n
  • Synced home of serialized updates per web server
• /usr/local/bin/sync-l10nupdate-1
  • Script that syncs the updates to the Apaches
• /usr/local/bin/sync-l10nupdate
  • Wrapper that runs sync-l10nupdate-1 as the l10nupdate user using sudo

Where this are in puppet

• misc::l10nupdate (in manifests/misc-servers.pp)
  • Installs cron job
  • Installs l10nupdate, l10nupdate-1, sync-l10nupdate and sync-l10nupdate-1 scripts
  • Creates logging directory and SVN checkout directory
  • Installs logrotate config file
• files/misc/l10nupdate
  • Scripts live here
• files/logrotate/l10nupdate
  • logrotate config file

Config

MediaWiki configuration variables:

• $wmgUseLocalisationUpdate (to enable/disable extension; lives in InitialiseSettings.php)
• $wgLocalisationUpdateDirectory = "/var/lib/l10nupdate/caches/cache-$wmgVersionNumber"; (set in CommonSettings.php)