Jenkins

Jenkins is a software coming from the Java community, to assist in building a continuous integration system. It offers a highly modular system which has allowed other community to add their own plugins (such as GIT, PHP...). Wikimedia Foundation is using Jenkins to run MediaWiki tests and generates nightly build of its Android applications.

Upgrading

Upstream does a release every week and a stable one from time to time, also named Long Term Support (LTS). We will certainly want to at least upgrade to LTS whenever it is released and possibly to a snapshot in between depending on bug fixes.

Updating

Debian packages can be installed on Ubuntu and are made available at:

http://pkg.jenkins-ci.org/debian/ http://pkg.jenkins-ci.org/debian-stable/

Per ops convention, the packages need to be copied to apt.wikimedia.org. Reprepro is configured to do so semi-automatically, so on carbon one only needs to do

 cd /srv/wikimedia
 reprepro checkupdate precise-wikimedia

verify the output and then

reprepro update precise-wikimedia

Warning:

It seems that recent Jenkins repository layout changes have broken the mechanism above by triggering a reprepro bug/missing feature (double redirects). You can always fall back by fetching the LTS deb manually from http://pkg.jenkins-ci.org/debian-stable/ and then run:

 reprepro -C thirdparty includedeb precise-wikimedia jenkins_1.480.3_all.deb
 reprepro export precise-wikimedia

Deploying

Jenkins is deployed on gallium host, reserved for continuous integration purposes. It can safely be restarted at anytime as long as the developer community is aware about it ( #mediawiki / #wikimedia-dev ). You will most probably want to have one continuous integration person floating around while doing the upgrade since plugin dependencies might broke once Jenkins is upgraded.

To upgrade the package, just apt-get update && apt-get upgrade jenkins. (or tell puppet to ensure "latest")

How to

Restart Jenkins

Warning:

Jenkins init script is bugged bug 51817, the init script cant find the pid

See mw:Continuous integration/Jenkins#Restart.

Logs

Main application logs:

tail -f /var/log/jenkins/jenkins.log

Disable plugin

Jenkins plugins are placed in /var/lib/jenkins/plugin/ and ends with a .hpi extension. To disable a plugin, renames it to .hpi.disable and restart Jenkins.

Java thread dump

Whenever Jenkins appears to be stuck or facing high CPU usage, you will want to look at the Java threads: https://integration.wikimedia.org/ci/threadDump

~~You can also send signal 3 to the java process, Jenkins will write a thread dump in /var/log/jenkins/jenkins.log~~

~~kill -3 "pid of jenkins"~~

kill -3 apparently kills Jenkins :(

Another way:

jstack -F <pid of jenkins>

And there is a more verbose dump written to /var/log/jenkins/jenkins.log

Links

bug 31877 bug about upgrade done in April 2012
bug 41973 bug tracking the next LTS upgrade.

Worklog

Issues about upgrade done on April 5th 2012 :