Nova Resource:Integration/Setup
Roles
integration-slave-{type}-XXXX
- Create instance:
m1.large
- Security group: Default
(Puppet master is auto-configured via Hiera:Integration)
- Wait 10 minutes (during which the instance is created, puppetmaster configured and certificate signed).
- Connect to the instance over SSH. If your
/home
is not yet mounted from NFS, reboot the instance (race condition with new instances). - Reconfigure instance from wikitech: Enable
role::ci::slave::labs
. - Via SSH, force a puppet run (provisions the instance, takes about an hour).
- Apply patches: #Ubuntu 14 Trusty or #Ubuntu 12 Precise
- Reboot the instance (Before adding to Jenkins). This cleans state, launches deamons, and fixes Shinken monitoring (phabricator:T91351).
- Create "New Node" in Jenkins management
- Name: (short hostname of instance)
- Type: Dumb
- Executors: 4
- Filesystem root:
/mnt/jenkins-workspace
- Labels:
contintLabsSlave
UbuntuTrusty phpflavor-hhvm
orUbuntuPrecise phpflavor-zend
- Usage:
EXCLUSIVE
(Only build jobs with label restrictions matching this node) - Launch method: SSH
- Host: (internal IP of instance)
- Credentials: jenkins-deploy (key from role::ci::slave::labs::common)
- Availability:
Always
(Keep this slave on-line as much as possible)
integration-dev
- Create instance:
m1.medium
- Security group: Default
- Wait 10 minutes
- Reconfigure instance from wikitech: Enable
role::ci::slave::labs
. - Via SSH, force a puppet run (applies role).
Patches
Ubuntu 14 Trusty
- See phabricator:P466 for a shell script that automates this.
- Assert Node.js v0.10.25 is installed.
- Ensure npm is upgraded to v2.14.12.
- Ensure grunt-cli v0.1.13 is installed.
Ubuntu 12 Precise
- See phabricator:P502 for a shell script that automates this.
- Assert Node.js v0.8.2 is installed.
- Ensure npm is upgraded to v2.14.12.
- Ensure grunt-cli v0.1.13 is installed.
Utilities
npm upgrade
Always use the original npm that came with Ubuntu to perform the upgrade. This installs the new version in /usr/local/bin
, preserving the original.
- For each instance, gracefully depool it (!log it), wait for any running builds on it to complete, and run the following:
- If Ubuntu Precise:
# Note: Must sudo without preserving user environment, using -i $ sudo -i # Precise default npm has outdated cert set null. Note this only applies to this command. # Jenkins builds always use the new/local cert which works fine. root$ /etc/alternatives/npm install -g npm@2.7.6 --ca=null # Purge both caches root$ rm -rf /mnt/home/jenkins-deploy/.npm /root/.npm
- If Ubuntu Trusty:
# Note: Must sudo without preserving user environment, using -i $ sudo -i root$ /usr/share/npm/bin/npm-cli.js install -g npm@2.7.6 # Purge both caches root$ rm -rf /mnt/home/jenkins-deploy/.npm /root/.npm
- Re-pool instance.
puppet
Use sudo /usr/local/sbin/puppet-run
. Avoid using sudo puppet agent -t
, since that is not what cron uses and leads to inconsistencies with e.g. umask and other factors affecting default values used at runtime.
dsh-ci-slaves
When debugging or performing post-puppet setup patches documented on this page, dsh-ci-slaves
can come in handy to quickly perform checks or commands. A few examples:
## Check node/npm version
$ dsh-ci-slaves 'node --version && npm --version'
integration-slave1204.eqiad.wmflabs: v0.8.2
integration-slave1204.eqiad.wmflabs: 2.4.1
integration-slave1402.eqiad.wmflabs: v0.10.25
integration-slave1402.eqiad.wmflabs: 2.4.1
..
## Get IP of instances
# Useful when registering them in Jenkins for which you need the IP
$ dsh-ci-slaves 'ping $HOSTNAME -c1 | grep PING'
integration-slave1203.eqiad.wmflabs: PING integration-slave1203.eqiad.wmflabs (10.68.17.195) 56(84) bytes of data.
integration-slave1401.eqiad.wmflabs: PING integration-slave1401.eqiad.wmflabs (10.68.17.208) 56(84) bytes of data.
..
screenshot
# at integration-slave1010 over ssh
$ import -display :94 -window root "image $(date).png"
# local shell
$ scp integration-slave1010.eqiad.wmflabs:image*.png . && open
Debug MediaWiki frontend
Take slave of choosing offline in Jenkins web interface (e.g. press "Mark offline" after selecting a node)
sudo -iu jenkins-deploy
$ cd /mnt/jenkins-workspace/workspace/mediawiki-core-qunit/
$ export WORKSPACE=$PWD; export BUILD_TAG=debug; export EXECUTOR_NUMBER=999
$ cd src/
$ git remote update && git checkout origin/master && git reset --hard HEAD
$ /srv/deployment/integration/slave-scripts/bin/mw-install-mysql.sh
$ /srv/deployment/integration/slave-scripts/bin/mw-apply-settings.sh
$ . /srv/deployment/integration/slave-scripts/bin/mw-set-env-localhost.sh
$ echo -e "<?php\n\$wgServer = '${MW_SERVER}';\n\$wgScriptPath = '${MW_SCRIPT_PATH}';\n\$wgScript=\$wgStylePath=\$wgLogo=false;\n\$wgEnableJavaScriptTest = true;\n" >> "$MW_INSTALL_PATH/LocalSettings.php"
$ . /srv/deployment/integration/slave-scripts/bin/npm-setup.sh
$ rm -rf node_modules && npm install
$ ln -s "$MW_INSTALL_PATH" /srv/localhost-worker/$BUILD_TAG
Now the install is ready for interacting with.
For example:
$ curl --include "$MW_SERVER/$MW_SCRIPT_PATH/index.php?title=Special:BlankPage"
$ curl --include "$MW_SERVER/$MW_SCRIPT_PATH/load.php?debug=true&modules=startup&only=scripts"
Or
$ cd $MW_INSTALL_PATH
$ grunt qunit
Remember: Clean up MySQL database afterwards by running (needs the same environment, e.g. WORKSPACE
, EXECUTOR_NUMBER
):
/srv/deployment/integration/slave-scripts/bin/mw-teardown-mysql.sh