Debugging in production
Using testwiki
If you have ever had to debug live code on the production cluster, you know that it can be very difficult to reason about the effects of caching and load-balancing. testwiki is specially configured to help control for these factors. Unlike other wikis, testwiki is served by a single Apache server, mw1017, so you know exactly where your code is executing. This is very useful if the problem you are trying to debug is reproducible on https://test.wikipedia.org.
Accessing mw1017 with a magic header
Using a special HTTP header X-Wikimedia-Debug: 1 any request can bypass Varnish cache and be routed to mw1017, regardless of wiki or hostname.
Other caching layers (such as by MediaWiki, Memcached, ResourceLoader, Parser cache, etc.) may still be in effect.
You can use cURL's -H command line option to craft requests with this header.
$ curl -H 'X-Wikimedia-Debug: 1' --dump-header - https://it.wikiquote.org/wiki/Pagina_principale
HTTP/1.1 200 OK
...
Age: 0
X-Cache: cp1055 miss (0), cp4008 miss (0), cp4016 frontend miss (0)
...
<!DOCTYPE html>
...
mw.config.set({"wgBackendResponseTime":313,"wgHostname":"mw1017"});
..
The WikimediaDebug Chrome extension and WikimediaDebug Firefox extension (by Ori Livneh) enables this header on all requests to Wikimedia domains for 5 minutes. It can be toggled on or off at any time via its Chrome toolbar icon.
Debugging a maintenance script on mw1017
ssh into mw1017
source /usr/local/lib/mw-deployment-vars.sh
sudo -u "$MEDIAWIKI_WEB_USER" hhvm -m debug "$MEDIAWIKI_DEPLOYMENT_DIR/multiversion/MWScript.php" someScript.php --wiki=testwiki --scriptSpecificParameters "goHere"
Pushing code to mw1017
Developers can put code updates on mw1017 shortly before deploying to the entire production cluster, see Pre-deployment testing in production in How to deploy code.
Showing profiling data for MediaWiki requests
If the debug header is sent, adding ?forceprofile=1 to the URL will result in profiling data being show in an HTML comment at the bottom of the page (e.g. using "view source").
The easiest ways to so for pages which require a cookie are probably:
- in Chromium, open developer tools, right click on the request for the URL, select "copy as curl command" and add the debug header to the CLI;
- in Firefox, open developer tools, select the request, select the "Headers" tab, click "Edit and resend", add the header, send.