Dumps/How to deploy

Deployment of dumps code is done from a salt-master, at this writing palladium. For deployment of other parts of the dumps infratructure, see the puppet modules 'dataset', 'snapshot', 'download', and their roles.

Scripts for doing the deployment live in palladium:/root/dumpsdeploy/scripts, see the dumps git repository, branch ariel. A local anonymous git clone of the dumps repo is in /root/dumpsdeploy/dumps with the ariel branch checked out.

To ue the scripts, do the following from a salt master with the above directories and code in place, with the latest changes to the repo and the checkout you want to push from.

setup

1. be root (note that the script uses salt to push files)
2. cd /root/dumpsdeploy/scripts
3. python hack-deploy.py dumps prep

   this creates a copy of the relevant files in /root/dumpsdeploy/staging in a subdirectory with today's date

4. make your desired changes if any

testing

1. python hack-deploy.py dumps deploy snapshot100x.eqiad.wmnet (or other snapshot host for testing)

   this will take effect *immediately* for the next dump run on that host

2. on the speciied snapshot host, force a run of a small wiki, choosing one from near the bottom of this page: [1]:
   • attach to screen session on the snapshot host and open a new window
   • su - datasets; cd /srv/dumps
   • bash ./worker --configfile confs/wikidump.conf --basedir /srv/dumps --wiki name-of-wiki-here
   • watch it run, check the output files in /mnt/data/xmldatadumps/public/nameofwikihere/todays-date-here/ against the prev date for sanity (size, # of titles, etc)
3. if it looks bad, remove the contents and revert the production push from the snapshot host:
   • on palladium, cd /root/dumpsdeploy
   • python dumps deploy snapshot100x.eqiad.wmnet name-of-previously-deployed-dir-here
   • cleanup the local staging area dir if you aren't going to fix and repdeploy:
   • rm -rf /root/dumpsdeploy/staging/todays-date-here (do an ls to see the format)
   • on the snapshot host, rm -rf /mnt/data/xmldatadumps/public/name-of-wiki-here/todays-date-here/

full deployment after testing

1. if the test results look good, make it so on all snaps:
   • on palladium, python dumps deploy 'snapshot*'
2. you are done.

misc ccrapola

TODO: dump test instance in labs, to be used before live testing on a snapshot host