Wikimedia binaries
The following executable binaries are maintained in version control (operations/puppet:/files/misc/scripts
) and copied/synced to servers (such as tin) by puppet into /usr/local/bin
.
bin
apache-graceful
Note: this script is NOT in puppet.
- Performs the action only on the one server from which the command is ran
apache-graceful-all
Note: this script is in /usr/local/bin/ on tin and in puppet.
- This script operates on all servers in the following dsh groups: apaches
- Runs
apache-graceful
locally on all apaches.
fatalmonitor
- Runs a
watch
command on the apache logs with various exclusion and aggregation filters to monitor current php notices, warnings and errors on the cluster.
sync-apache
- This script operates on all servers in the following dsh groups: apaches, image_scalers, snapshot, searchidx
- Note: The conf files are outside of
/a/common
, and as such are not synced by thesync-common
-like scripts.
- syncs
/home/wikipedia/conf/httpd
to/apache/conf
sync-common
Note: this script is from mediawiki/tools/scap
.
- Performs the action only on the one server from which the command is ran
- syncs
/srv/mediawiki-staging
from a deployment rsync server->/srv/mediawiki
on the local server
sync-dir
Note: this script is from mediawiki/tools/scap
.
- This is an all-script!
- for a single directory
- checks PHP syntax
- syncs
/srv/mediawiki-staging/(some dir)
->/srv/mediawiki
sync-file
Note: this script is from mediawiki/tools/scap
.
- This is an all-script!
- for a single file
- checks PHP syntax
- syncs
/srv/mediawiki-staging/(some file)
->/srv/mediawiki
sync-wikiversions
Note: this script is from mediawiki/tools/scap
.
- This is an all-script!
- syncs
/srv/mediawiki-staging/wikiversions.{json,cdb}
->/srv/mediawiki
scap
Note: this script is from mediawiki/tools/scap
. Generated documentation is published here
- This is an all-script!
- "sync-common-all-php"
- lints PHP files in
./wmf-config
,./multiversion
- Sync deploy directory (/srv/mediawiki-staging) on localhost with staging area (/srv/mediawiki)
- for all mediawiki versions currently deployed (usually 2) it:
- rebuilds localization caches of core and extensions
- creates json files from localization caches for better rsync performance
- runs sync-common on rsync slave servers to update
/srv/mediawiki
- runs sync-common on "all" servers to update
/srv/mediawiki
from nearest rsync slave - rebuilds localization caches on "all" servers from json files that were synced by sync-common
- compiles wikiversions.json to wikiversions.cdb on localhost
- runs sync-wikiversions
mwgrep
Search for CSS and JavaScript code fragments in MediaWiki namespace pages on all Wikimedia projects (powered by Elasticsearch).
It's located in operations/puppet/files/misc/scripts/mwgrep
All-script
An all-script is a script that operates on mediawiki-installation boxes via ssh. These scripts performs the action on all servers.
The specific servers include:
- app servers
- image scalers
- misc other hosts running apache:
Examples
sync-apache
example
- On tin:
- (optionally) run:
sync-apache-simulated
This just has an "-n" added to the rsync commands. (--dry-run |
show what would have been transferred) - If that looks fine, actually sync:
sync-apache
- Connect to a single random "srv" server, confirm the expected change has arrived here, and restart Apache to check your change did not break things.
- (how to restart that apache?)
- If that also looks fine, restart all Apaches:
apache-graceful-all
After this you might want to purge a URL from Squid cache. Also see Squids#Purge a given external (non WMF) url.
In most cases this is enough. However if the change involves more than a simple change to redirects, depooling/ repooling may be necessary. Make sure you find out whether this is needed ahead of time (while at it, start the depooling page and link to it, we need documentation!).