Obsolete:Upload.wikimedia.org

Uploaded image and media files for Wikimedia's wikis appear on the separate subdomain upload.wikimedia.org.

This provides separate physical hosting for efficiency, a separate domain name to aid in browser connection splitting, and a separate domain name to aid in JavaScript security rules.

DNS

upload.wikimedia.org is handled by the PowerDNS geobackend, which CNAMEs it to upload.pmtpa.wikimedia.org, upload.knams.wikimedia.org or upload.yaseo.wikimedia.org, depending on the location of the querying resolver.

Caching layer

Uploads have a separate set of squid proxy caches from the text squids; this avoids contention between the two data sets, which have different characteristics for object size, update rate, etc.

Backend: storage

server	path	purpose
ms1	/mnt/upload5	uploaded images and texvc-rendered images
ms4	/mnt/thumbs	thumbnails

Some not-too-horrible stanza snipped from upload-settings.php:

'apaches' => array(
	'pmtpa' => array(
		'ms1.wikimedia.org',
                '=ms4_thumbs'=> 'ms4.wikimedia.org',
	),
),

Thumbnails are served by ms4.

Backend: scaling

Wikimedia's wikis are configured to defer most image scaling/rasterization operations ($wgGenerateThumbnailOnParse off), just putting "optimistic" URLs <img> tags and letting a 404 handler on the thumbnail server deal with making sure the images are in place.

This does a couple nice things:

• Less interaction between primary servers and NFS
• Encapsulation of thumbnailing/scaling problems on the "image scaling cluster" so the entire grid doesn't go down due to runaway 'convert' processes etc.

The thumbnail server (ms4) farms out the scaling to the "image scaling cluster" machines, a sub-cluster of apaches running MediaWiki just for thumb.php.

These backend scalers produce the rendered images and save them back to NFS, and they also get served back out to the requesting user-agent none the wiser.

Note that thumb.php can be accessed directly from the web as well on the regular "text" Apaches. Ideally we shouldn't do that? :)

Private wikis

Our private wikis do not serve their files to the public through this interface; they're served through img_auth.php on the local domain, which enforced authentication.

Compatibility links

Most of our wikis have a rewrite rule to redirect requests from /upload and /math on the primary wiki domain to the appropriate subdirectory on upload.wikimedia.org. This provides compatbility for old direct image links from the days before the separate image hosting.

Special:Filepath can also send redirects here.