Cowbuilder

This is a generic guide for cowbuilder, to use it on wmflabs see Cowbuilder on labs

Apply role::package::builder on a labs VM will allow you to avoid all the following as it is already being done by puppet. Documentation is kept up to date at: https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/package_builder/

Intro

cowbuilder is a Debian packaging tool that takes advantages of Copy On Write (COW) to speed up the building process. You first bootstrap Debian distributions you are interested in and configure them as it may fit your use case. When building a package, cowbuilder hardlink the files in a new hierarchy and would write to the disk only on actual change to the files. That makes the startup lightning fast.

This page is a step by step guide to setup a cowbuilder environement. It assumes your machine is running Debian/Jessie machine and you have root access. We will cover the initial image creations for stock Debian/Jessie and the various custom distribution Wikimedia is using. We will then look at how they work out when using git-buildpackage.

cowbuilder images

Installation

apt-get install build-essential git-buildpackage cowbuilder

Done.

Create images

TODO Configure pbuilder:

/root/.pbuilderrc
Is APTCACHE really needed ?

Stock Debian Jessie

cowbuilder --create --distribution=jessie --debootstrapopts --variant=buildd

It takes the Debian mirror from your sources.list and install it in a chroot under /var/cache/pbuilder/base-jessie.cow. That will be known as the distribution jessie. More options are available via man cowbuilder and the underlying command being used man pbuilder.

Debian Jessie Wikimedia

Wikimedia uses its own apt repositories which comes with additional packages and backports. You will want to create another jessie flavor image. It would be similar but points to wikimedia and ensure the packages receive an higher priority. For a start, just copy:

cp -a /var/cache/pbuilder/base-jessie.cow /var/cache/pbuilder/base-jessie-wikimedia.cow

We will further tweak that image by login into it and having our modifications saved back to disk:

cowbuilder --login --basepath /var/cache/pbuilder/base-jessie-wikimedia.cow --save-after-login

You now have a prompt in this image, add in the Wikimedia repository pinned to receive priority:

echo deb http://apt.wikimedia.org/wikimedia jessie-wikimedia main backports thirdparty \
  > /etc/apt/sources.list.d/wikimedia.list
echo -e "Package: *\nPin: release o=Wikimedia\nPin-Priority: 1001" \
  > /etc/apt/preferences.d/wikimedia

Since our packages are signed with GPG, you will want to grab the gpg keys we use and verify them. To do so:

apt-get install wget
wget http://apt.wikimedia.org/autoinstall/keyring/wikimedia-archive-keyring.gpg
apt-key add wikimedia-archive-keyring.gpg
apt-get update

You will want to clear out some of the modifications you just made:

dpkg --purge wget
apt-get autoremove --purge
rm wikimedia-archive-keyring.gpg

And upgrade the image:

apt-get upgrade

exit / Ctrl+d and cowbuilder will save the result of your modification to the base-jessie-wikimedia.cow image.

Ubuntu images

TODO

   apt-get install ubuntu-archive-keyring
   # We need universe!
   cowbuilder --create \
       --basepath /packaging/base-trusty.cow \
       --mirror http://nova.clouds.archive.ubuntu.com/ubuntu/ \
       --distribution trusty \
       --components 'main universe' \
       --debootstrapopts --variant=buildd \
       --debootstrapopts --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg \

  echo 'deb http://apt.wikimedia.org/wikimedia precise-wikimedia main universe thirdparty' > /etc/apt/sources.list.d/wikimedia.list
  echo -e "Package: *\nPin: release o=Wikimedia\nPin-Priority: 1001" > /etc/apt/preferences.d/wikimedia

Probably need to get trusty updates as well.

Result

An example result of the above steps:

$ ls -1d /var/cache/pbuilder/*cow
base-jessie.cow
base-jessie-wikimedia.cow
base-precise.cow
base-precise-wikimedia.cow
base-trusty.cow
base-trusty-wikimedia.cow
$

This let you build a package against either the stock or the Wikimedia flavors for all distributions in use.

Maintaining patch series

Placeholder for gbp pq.