Using git-buildpackage with source that already has a debian/ directory

(extracted from #Experiments below.)

Initially cloning the git repository from Upstream

Clone the repository from upstream if it doesn't already exist in Gerrit, e.g. from git.debian.org:

$ cd ~/git/debs/
$ git clone git://git.debian.org/git/pkg-puppet/repo.git

Set git username/email

$ git config --global user.name "Your Name"
$ git config --global user.email you@example.com

Set up the local branches upstream and pristine-tar from the remote branches (but don't switch to them):

$ git branch upstream origin/upstream
$ git branch pristine-tar origin/pristine-tar

Build a source package:

$ git-buildpackage --git-export-dir=/var/tmp/build-area/packagename -S

Build the binary packages with Pbuilder:

# pbuilder --basetgz /var/cache/pbuilder/lucid.tgz /var/tmp/build-area/packagename/packagename_version.dsc

Note: Applying role::package::builder on a labs VM will pre-populate a pbuilder/cowbuilder environment for you. Documentation on it is kept up to date at: https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/package_builder/

Making changes and testing snapshot builds

Make changes to the package files
Make commits or amended commits (uncommitted changes are not exported to the build directory)
Build snapshot builds for testing with:

$ git-dch --snapshot
$ git-buildpackage --git-export-dir=/var/tmp/build-area/packagename -S
# pbuilder --build --basetgz /var/cache/pbuilder/lucid.tgz /var/tmp/build-area/packagename/packagename_version.dsc

Doing a release

When a package is ready to be released, use the following steps:

$ git-dch -R -N version

Commit the debian/changelog file (or amend a previous commit)
Build the final source package with --git-tag:

$ git-buildpackage --git-tag --git-export-dir=/var/tmp/build-area/varnish -S

Build the binaries with Pbuilder as usual

Pushing changes into Gerrit

Push changes for each of the three branches (if applicable): $ git push gerrit master:refs/for/master $ git push gerrit upstream:refs/for/upstream $ git push gerrit pristine-tar:refs/for/pristine-tar

Pushing the repository into Gerrit

You should push any repository with local changes into Gerrit.

Create a new repository, operations/debs/packagename using the following Gerrit command:

$ ssh username@gerrit.wikimedia.org -p 29418 'gerrit create-project -d "Package packagename" -n operations/debs/packagename -o ldap/ops -p operations/debs'

Create a new remote called gerrit:

$ git remote add gerrit ssh://username@gerrit.wikimedia.org:29418/operations/debs/packagename

Push the three local branches (master, upstream and pristine-tar) into the fresh remote repository:

$ git push gerrit --all

How to build a Python deb package using git-buildpackage

First, it might be good to familiarize yourself with git-buildpackage. This manual is actually very helpful: http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.html

Particularly, read the bit about different ways to import packages. The package that you are importing may vary in its own packaging. Do they release tarballs? Are you importing from an upstream git repository? Etc. etc.

Building a python package from a release tarball

I just built python-jsonschema. I'll use it as an example case.

Prep your git repository

Download the release tarball:

 wget https://pypi.python.org/packages/source/j/jsonschema/jsonschema-1.1.0.tar.gz

If you don't yet have a repository in gerrit, you'll need to create one:

 ssh username@gerrit.wikimedia.org -p 29418 'gerrit create-project -d "Package python-jsonschema" -n operations/debs/python-jsonschema -o ldap/ops -p operations/debs'
 git remote add gerrit ssh://username@gerrit.wikimedia.org:29418/operations/debs/python-jsonschema

The gerrit repository we just created is empty. Let's start a new one locally.

 mkdir -p python-jsonschema
 cd python-jsonschema
 git init .

We are importing from a tarball, and not a git repository and git-import-orig wants an empty 'upstream' branch to exist on our repository. See: http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.import.html#AEN212

 git checkout --orphan upstream
 git commit --allow-empty -m 'Initial upstream branch.'
 # switch back to master
 git checkout -f master

NOTE: master will be the name of the git-buildpackage specific 'debian-branch'. IF you want to use a different branch for this name, you will have to tell git-buildpackage via a config file, or via the CLI options. git-buildpackage suggests using master, so we'll stick with it.

Import the tarball

Since we're building a python module, be sure to name the package with the 'python-' prefix. We're also using --pristine-tar as git-buildpackage recommends.

 git-import-orig  --pristine-tar ../jsonschema-1.1.0.tar.gz
 What will be the source package name? [jsonschema] python-jsonschema
 What is the upstream version? [1.1.0]
 gbp:info: Importing '../jsonschema-1.1.0.tar.gz' to branch 'upstream'...
 gbp:info: Source package is python-jsonschema
 gbp:info: Upstream version is 1.1.0
 gbp:info: Merging to 'master'
 Merge made by the 'recursive' strategy.
 gbp:info: Successfully imported version 1.1.0 of ../jsonschema-1.1.0.tar.gz

Use stdeb to create the debian/ directory

Faidon says not to use stdeb, but it creates a decent debian/ directory that gives a good starting point. You can create your debian/ directory however you like. See python-jsonschema/debian for a decent version of a python package debian/ directory that will use dh_python2.

For our example, we'll use stdeb and then edit the debian/ files and use dh_python2 to build the package.

See: https://pypi.python.org/pypi/stdeb

 python setup.py --command-packages=stdeb.command debianize

Since we don't like stdeb's output, follow these instructions to modify debian/ files to work with dh_python2:

http://wiki.debian.org/Python/TransitionToDHPython2

In summary:

Changes in small were not relevant when I tried this valhallasw (talk) 20:04, 9 May 2015 (UTC)[reply]

changelog:
- Change maintainer to you.
- Change changelog message to 'Initial release.'
compat:
- change to 8
control:
- Change maintainer to you.
- debhelper (>= 8)
- remove python-support from Build-Depends
- Add python (>= 2.6.6-3~) to Build-Depends
- Put relevant dependencies (from setup.py / requirements.txt) in Depends
- Change Standards-Version to 3.9.3
- Remove XB-Python-Version
- Modify description to remove any special formatting (markdown, etc.)
python-<package-name>.preinst
- remove this file completely.
rules
- remove comments about stdeb
- remove unexports
- remove export DH_OPTIONS
- add --with python2 to dh command: dh $@ --with python2
source/format
- change to 3.0 (quilt)

Add the debian/ directory to git:

 git add debian
 git commit -m 'Initial deb packaging'

Build the package

Now try to build the package, yay!

 mkdir -p ../build-area/python-jsonschema
 git-buildpackage --git-export-dir=../build-area/python-jsonschema

Check to see that the .deb looks like it should:

 dpkg -c ../build-area/python-jsonschema/*.deb

If that looks good, push everything to gerrit, minus your most recent commit (that one needs review).

Push to gerrit for review

Directly push the upstream and pristine-tar branches, those don't need review:

 git push gerrit upstream
 git push gerrit pristine-tar

Directly push everything in master except for the commit in which you added the debian/ directory:

 git push gerrit HEAD~1:refs/heads/master

Finally git-review (or whatever) to push the last unpushed commit for review:

 git review

Add the reviewed package to apt.wikmedia.org

Once your review has been approved and merge, copy the files in build-area/python-jsonschema to carbon, and add the .changes file with reprepro. We need to add the --ignore=wrongdistribution flag, since we didn't build the package with a special wikimedia distribution name.

 reprepro --ignore=wrongdistribution -C main include precise-wikimedia jsonschema_1.1.0-1_amd64.changes

Building a python package from an existing git repository

Based on [1]

$ git clone https://github.com/sripathikrishnan/redis-rdb-tools
$ cd redis-rdb-tools
$ vim .git/config
 # hack out the branch "master" upstream tracking

# upstream doesn't have tags, hack these in
$ git reset --hard d2792069291484af1d9d64abd849aaa44ddf6211
$ git tag upstream/0.1.6

then follow the instructions above from 'Use stdeb to create the debian/ directory'.

Ottomata TODO: understand and write this section.

FAQ

You might have dpkg-source complains about local change. You can get them ignore by editing debian/source/options and add the file to ignore list:

$ cat debian/source/options
extend-diff-ignore = '^\.gitreview$'

Reference: http://www.debian.org/doc/manuals/maint-guide/dother.en.html#sourceopt

Experiments

Mark is experimenting with building puppet using git-buildpackage, in labs.

On i-00000080:

The debian puppet package git repository was cloned into ~/git/debs/puppet:

$ cd ~/git/debs/
$ git clone git://git.debian.org/git/pkg-puppet/puppet.git

Package build was attempted with git-buildpackage --git-ignore-new --git-pristine-tar
Signing the package failed, as the Debian maintainer's secret key is not available.
Package build was attempted without signing:

$ git-buildpackage --git-ignore-new --git-pristine-tar -us -uc

This worked. We ended up with built packages in ~/git/debs, and a dirty git working dir.
The git working directory was made clean again, and another build was attempted using

$ git-buildpackage --git-ignore-new --git-pristine-tar -us -uc --git-export-dir=/var/tmp/build-area/puppet

This worked as well. Packages were being built, and the git working dir remained clean. Packages ended up in the build dir.
The debian git dir only had version 2.7.6, but we want 2.7.7rc2. So I tried importing the dev release with

$ git-import-orig /tmp/puppet-2.7.7rc2.tar.gz

This failed, as there was no local upstream branch in the cloned git repository yet. Ok, so let's create it from origin/upstream:

$ git branch upstream origin/upstream

Now the import worked. But commits were made, with the default git name/mail address. Do set it:

$ git config --global user.name "Your Name"
$ git config --global user.email you@example.com

Try another build.

$ git-buildpackage --git-ignore-new --git-pristine-tar -us -uc --git-export-dir=/var/tmp/build-area/puppet

Oops. This used version 2.7.6. But, the git-buildpackage manual says changing the changelog itself all the time isn't a good idea. Let's try git-dch:

$ git-dch --snapshot
Changelog has been prepared for snapshot #1 at 48473567008ad82cdb0d0a5654dffe7c868fad91

$ git diff
diff --git a/debian/changelog b/debian/changelog
index 796b61e..f6da810 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,11 @@
+puppet (2.7.6-1ubuntu1~1.gbp484735) UNRELEASED; urgency=low
+
+  ** SNAPSHOT build @48473567008ad82cdb0d0a5654dffe7c868fad91 **
+
+  * Imported Upstream version 2.7.7rc2
+
+ -- Mark Bergsma <mark@i-00000080.pmtpa.wmflabs>  Wed, 23 Nov 2011 14:06:45 +0000
+

Cool, but not quite what I need. There was no commit, so let's simply revert that, and try again after reading the git-dch manual:

$ git checkout debian/changelog
$ git-dch --snapshot -N 2.7.6+2.7.7rc2 --git-author

$ git diff
diff --git a/debian/changelog b/debian/changelog
index 796b61e..06a9cac 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,11 @@
+puppet (2.7.6+2.7.7rc2~1.gbp484735) UNRELEASED; urgency=low
+
+  ** SNAPSHOT build @48473567008ad82cdb0d0a5654dffe7c868fad91 **
+
+  * Imported Upstream version 2.7.7rc2
+
+ -- Mark Bergsma <mark@wikimedia.org>  Wed, 23 Nov 2011 14:23:22 +0000
+

Now let's try a new build:

$ git-buildpackage --git-ignore-new --git-pristine-tar -us -uc --git-export-dir=/var/tmp/build-area/puppet

Whoops. It built 2.7.6 instead. Probably because it exported HEAD into the build area. Let's see what it does within the git repository.

$ git-buildpackage --git-ignore-new --git-pristine-tar -us -uc

So that worked. Hmm, inconsistent. :( I guess we could choose not to care about version numbers for snapshots, and only care about them for releases.

experiment build mathoid debian package on vagrant and publish to ppa =

The result https://launchpad.net/~physikerwelt/+archive/ubuntu/mathoid/+packages

 sudo apt-get install git-buildpackage
 sudo apt-get install libdistro-info-perl

without the libdistro-info-perl the changes between the last release could not be calculated

cd

it does not work in a folder that is mounted to the windows host machine (i.e. /vagrant/srv/mathoid can not be used)

 mkdir container
 ls
 cd container/
 git clone https://github.com/physikerwelt/mathoid-server
 cd mathoid-server/
 npm install
 git-dch -R -N 0.2.8
 git commit -a -m "Update debian release notes"
 sudo apt-get install dh-make
 sudo apt-get install gnupg2
 gpg2 --gen-key
 apt-get install haveged

this command was executed in a second terminal window because it was hard to generate entropy

 gpg2 --send-keys --keyserver=keyserver.ubuntu.com 7271B791
 vi test.msg

you get an encrypted email that contains a link that you have to click... you need to find some way to read the contents... probably there is also a plugin for the gmail webui... but I could not find it

 gpg -d test.msg
 rm test.msg
 git-buildpackage --git-tag -S -k7271B791

normally the k argument should not be required

 sudo apt-get install dput
 dput ppa:physikerwelt/mathoid ../mathoid_0.2.8_source.changes
 git push origin debian/0.2.8

External links