Memcached

Wikimedia infrastructure
	Data centres and PoPs Networking HTTP Caching Wiki HHVM; Nutcracker; Memcached; Redis; Parser cache; Media Logs

The Memcache setup changed in 2015 to be multi-datacenter aware. See also:

mw:Manual:WANObjectCache.php
nutcracker (memcached proxy on all application servers)

Disabling broken servers

When a memcache server is offline, it should be entirely removed from the configuration (if permanent) or commented out (if temporary.) Unlike the old slot format config file, the new format doesn't require you to keep an identical number of members deployed in the memcached pool. Please note that one shouldn't add and remove a lot of members from this list in a short period of time, as the consistent hashing may result in a server being unpooled, then repooled to cause the keys to remap directly back to the same server, resulting in undesired performance. So if a server is down, fix it ASAP. If it cannot be rebooted and fixed ASAP, then it should be removed. If it is removed, it may be prudent to leave it unpooled for 24 hours? (Aaron advises we have some items that can cache in mc for 24 hours.)

The memcached configuration php files are maintained in gerrit, under the operations/mediawiki-config branch.

mediawiki-config/wmf-config/mc-eqiad.php
mediawiki-config/wmf-config/mc-pmtpa.php

Memcached installation is handed via the puppet memcached role; and are currently deployed to dedicated memcache systems (mcX and mcXXXX).

You can test the servers to see which memcached are functioning and which are not with the following on fenari:

cd /home/w/common/wmf-deployment/maintenance; mwscript mctest.php

Note that all servers that are up will return incr: 100 get: 100 and servers that are down will return incr: 0 get: 0.

Editing mc.php

The memcached configuration php files are maintained in gerrit, under the operations/mediawiki-config branch.

mediawiki-config/wmf-config/mc-eqiad.php
mediawiki-config/wmf-config/mc-pmtpa.php

If you want to add a server from the spare list to the active list, please test it first. You can run on fenari

 cd /home/w/common/wmf-deployment/maintenance; mwscript mctest.php enwiki ip-address-here:11000

(it runs on a nonstandard port). As above, servers that are up will return incr: 100 get: 100 and servers that are down will return incr: 0 get: 0.

Once you finish updating the file, you must git commit and then git review. You should be logged in as your own username, as your user keys will be checked. Then you need to git fetch and merge on fenari in the /home/wikimedia/common/wmf-deployment directory and sync out changes.

Deleting a memcache key

mwscript mcc.php --wiki=$wiki
> delete $wiki:$key
> exit

If the deletion is successful, no output will be shown.