APT repository

From Wikitech

Old information as of 2012. See Reprepro

Wikimedia maintains its own APT repository, under http://apt.wikimedia.org/wikimedia which lives on brewster. This repository contains Debian/Ubuntu packages modified for use by Wikimedia, and Wikimedia specific native packages, that don't exist elsewhere.

Previously, we were using one big package pool for all distribution versions we had installed, i.e., packages in hardy-wikimedia and lucid-wikimedia were the same. However, this gives all kinds of version incompatibilities, so we have now split the repository per distribution, and we are now managing the repository using a nice tool called reprepro.

It's important that we always specify the right distribution (hardy-wikimedia or lucid-wikimedia) for which the package is built in the package's Changelog (debian/changelog). The package should also be put under the right component, see below for details.

Using

To use this repository, the following lines need to be present in /etc/apt/sources.list:

## Wikimedia APT repository
deb http://apt.wikimedia.org/wikimedia hardy-wikimedia main universe
deb-src http://apt.wikimedia.org/wikimedia hardy-wikimedia main universe

Use the right distribution (e.g. lucid-wikimedia) instead of hardy-wikimedia, depending on which Ubuntu version was installed.

Additionally, to make sure that the system prefers packages from this repository, and not packages from the origin distributions even if they have a higher version number, this APT source is "pinned" with a higher priority in /etc/apt/preferences:

Package: *
Pin: release o=Wikimedia
Pin-Priority: 1001

The Wikimedia repository is signed with the Wikimedia Archive Automatic Signing Key <root@wikimedia.org>. This public key must be installed in APT's GPG keyring for it to verify packages without nagging about it. This can be accomplished by installing the wikimedia-keyring package.

Alternative options:

  • get the key from /Key, then add with apt-key add
  • pull the key from a labs host: ssh <host> apt-key export root@wikimedia.org, then add with apt-key add.

An automated installation of a Wikimedia server should take care of all of the above.

Updating the repository

To add packages to the Wikimedia repository, so they can be deployed on the servers, take the following steps.

Commit the debian/ directory of the package to Subversion, under:

svn.wikimedia.org/svnroot/mediawiki/trunk/debs/package basename/debian/

Make sure that the package is built on the relevant architectures (these days, only amd64). If you need to build a special package for each distribution version (as you likely will need to for at least binary packages), you should add the distribution name to the package version number. For example:

Package

puppet_2.6.1-0ubuntu1

becomes:

puppet_2.6.1-0ubuntu1~hardy1

and

puppet_2.6.1-0ubuntu1~lucid1

Be aware that a suffix starting with ~ sorts as a lower version than the original package without that suffix!

Upload all the package files, including the .changes file, to directory /srv/wikimedia/incoming/ on brewster. You can use the convenient tool dupload for this, on hosts on which it has been setup (e.g. ragweed):

$ dupload path-to-.changes-file

Then, on brewster, run:

# reprepro -C component-name include wikimedia-distribution-name path-to-.changes-file

For example:

# reprepro -C main include hardy-wikimedia php5-apc_3.1.3p1-1wm1_amd64.changes

If any previous version of the package was present in the repository, it will be removed completely by reprepro.

For component, use the following as a guide:

main
for Wikimedia native packages, as well as Debian/Ubuntu packages that have had source-modifications
universe
for existing Debian/Ubuntu packages that just have been recompiled/backported for the given distribution.

For distribution, use the distribution that the package has been compiled for, and under. Usually, any given compiled package should be for one distribution only, e.g. hardy-wikimedia OR lucid-wikimedia. This should match the field in the package's Changelog. Only in special circumstances a given package can be used under multiple distribution versions, .e.g. if the package contains only scripts (no binaries). Only do this if you've tested it and are sure this is working. In this case, include the package in the oldest distribution version, and then copy the package references using reprepro copy.

Under some circumstances (when you don't have a .changes file) you can use different reprepro commands, such as reprepro includedsc and reprepro includedeb. See reprepro for additional details.

The package should now be available for deployment/installation on the target hosts, after they have done a apt-get update, of course.

Setup of a Wikimedia repository for a new Ubuntu release

Edit /srv/wikimedia/conf/distributions, adding a block for the new distribution that matches the previous ones. For example:

Origin: Wikimedia
Label: Wikimedia
Suite: lucid-wikimedia
Codename: lucid-wikimedia
Version: 10.04
Architectures: source amd64
Components: main universe
UDebComponents: main
Description: Wikimedia specific packages for Ubuntu Lucid
SignWith: default
DebOverride: deb-override

New packages can now be imported for this distribution as normally.

See also

  • pbuilder for building packages for different distribution versions and architectures
  • reprepro for additional information on how to use this tool