Help:Self-hosted puppetmaster

Wikimedia Labs documentationThis is a documentation page for Wikimedia Labs. If you can improve it, please do that. You can also request help on #wikimedia-labs ^connect. For all documentation pages, please see the List of all documentation pages for labs. If you are seeking a help with a specific project, please contact the project owners.

This page documents how you change puppet manifests locally on a labs instance without having to submit, review and deploy it from a branch to the central puppetmaster first. Most labs instances are managed by the central labs-puppetmaster which uses puppet classes directly out of the central git repository. This will continue to be the case except that you can now override classes locally, so that you can test changes before pushing a change to gerrit for review.

Set up

The role::puppet::self class makes this possible. By default, including this class in an instance configuration causes that instance to host its own puppet repo, and to apply that same repo to the instance. That means you can write, test and debug all on one instance without needing to push anything to gerrit until you're ready.

You can also use role::puppet::self to manage a group of labs instances from a single self hosted puppetmaster. role::puppet::self has a 'puppetmaster' global variable. When undefined, the behavior is the same as described above. When set to an instance name, the current instance will use value of 'puppetmaster' global variable as its puppetmaster.

NOTE: most likely, you will only need to follow the instructions for setting up a single instance self hosted puppetmaster. A multi-instance setup is only necessary for users that need to configure a cluster of labs instances using a self hosted puppetmaster.

Set up a single instance self hosted puppetmaster

Set up an instance to your liking. "trusty" is a good choice, as most new development is happening on trusty systems and production has been fully migrated to puppet 3 masters.
Configure this instance as you would normally in wikitech, and include the classes that you want to hack on.
Force a puppet run ($ sudo puppet agent -tv) to make sure all is well. (You might want to watch /var/log/syslog to be sure that the initial puppet run has finished.)
Now, go back to the instance configuration page and enable the role::puppet::self class.
Force another puppet run, and behold!
- If you see "Could not start Service[puppetmaster]: Execution of '/etc/init.d/puppetmaster start' returned 1" on console, you will need to manually start the puppetmaster with sudo service puppetmaster start.

You now have a local puppet repo in /var/lib/git/operations/puppet. Changes to that repo will be applied to this instance any time that puppet runs (which is about every 30 minutes). Proper symlinks have been setup in /etc/puppet so that you can edit there without having to remember the repository's URL.

In case the changes include the creation of additional classes, these should be added to the instance's configuration as they would normally be; the labs central puppetmaster will ignore them but it will take effect on the self-hosted one.

Set up a multi-instance self hosted puppetmaster

Just don't do it. You'll make Yuvi and the other labs team people sad. Just request your own project and setup a per-project puppetmaster instead.

Set up project-wide puppetmaster

Setup a puppet master inside your project following the steps above
Visit Special:NovaProject and filter down to your project, then click, "View Hiera Config" under actions (or visit https://wikitech.wikimedia.org/wiki/Hiera:<your project name>)
Edit this config and add:

   classes:
     - role::puppet::self
   role::puppet::self::master: <your puppetmaster domain>

After a $ sudo puppet agent -t run on each project instance, each instance will look to the puppetmaster set in your project's hiera config.

If you want it to automatically sign new clients, add 'puppetmaster::autosigner' to your puppetmaster instance via 'Edit configuration'.

Your puppetmaster will automatically pull in changes to the puppet repository (while keeping your local commits, but not your uncommited changes). This is necessary so that Labs administrators can apply changes to the underlying system. To disable this, you can either set the role::puppet::self::autoupdate_master variable in the wikitech configuration page for your puppetmaster to false, or set role::puppet::self::autoupdate_master: false in your Hiera config. Note that in this case you will be responsible for updating the Puppet repository on a regular and long-term basis!

Caveats

CAUTION: There is currently no easy way to rollback from role::puppet::self to the central puppetmaster. It's not hard to manually force puppet to run against the central puppetmaster, but there is no programmatic way of removing packages, git clones etc.

NOTE: puppetmaster::self is now deprecated in favor of role::puppet::self. puppetmaster::self is retained for backwards compatibility, and is now just a wrapper for role::puppet::self with server => localhost.

TODO: Git repo is owned by root, so committing and pushing is messy. How to organize so that patches get the proper ID? (a solution is proposed below)

Enabling debug information for the puppetmaster itself is easiest done by setting DAEMON_OPTS="--debug --logdest=/tmp/puppet.log" in /etc/default/puppetmaster and restarting puppetmaster (service puppetmaster restart). Do not forget to disable it when no longer needed.

Working on server-side functions (for example Hiera backends, Puppet reports, etc.) can be confusing: While client-side facts and providers are "live" at the moment when you edit their files in /var/lib/git/operations/puppet, and for server-side types you only need to restart puppetmaster to clear its code cache, for server-side internals it takes a Puppet run to copy them to /var/lib/puppet/lib where puppetmaster sources them after another restart. As a Puppet run can take a long time, you can also edit the files directly in /var/lib/puppet/lib and restart puppetmaster afterwards, or use your favourite programmable editor to do all of that automatically.

Set up a remote machine for pushing to labs

Typical development is done on remote (non-labs) machines, this applies to puppet too so a common use case is to develop locally and then push to the labs puppet master for testing. Due to the way git works you can't push directly to a non-bare repository (i.e. /var/lib/git/operations/puppet). A simple but hackish solution is to push to an intermediate bare repo and then update the non-bare repo.

The git hook added in https://gerrit.wikimedia.org/r/#/c/159720/ achieves this. So assuming you have a puppet::self master configured, you can setup a local bare repo e.g. in your labs home:

 cd ~
 [ -d puppet.git/.git ] || git clone --bare --no-hardlinks /var/lib/git/operations/puppet/ puppet.git
 install -m755 /var/lib/git/operations/puppet/modules/puppet/files/self-master-post-receive puppet.git/hooks/post-receive

and on the development host add the relevant remote:

 git remote add labs_project_puppetmaster ssh://PUPPET_MASTER_HOSTNAME/~/puppet.git

then to test a puppet change in your labs puppet master from your local host you can push to that remote in the "production" branch, e.g.

 git push -f labs_project_puppetmaster HEAD:production

NOTE: if the puppet master is shared among more than one person, pushing will overwrite each other's changes on the puppet master!

NOTE: this works only if the post-receive hook runs, which isn't the case when pushing no changes e.g. when sharing the same bare git repo in /home among multiple machines. Running the hook boils down to ssh machine -- puppet.git/hooks/post-receive or pushing to a throw-away branch also does the trick (git push -f MACHINE HEAD:_bogus_branch ; git push -f MACHINE :_bogus_branch)

FAQ

How do I update?

The repo is set with labs-puppet as user, we provide a simple ssh wrapper that will let you authenticate at that user. To run an update, you will need to be allowed sudo access by a projectadmin:

$ cd /var/lib/git/operations/puppet
$ sudo GIT_SSH=/var/lib/git/ssh git pull --rebase
[sudo] password for username: <enter your labs password>
First, rewinding head to replay your work on top of it...
Fast-forwarded production to 5c540c3f298098ededae20cb14f54a0abad26d33.

You should also update the repo that contains dummy passwords, and then restart the puppetmaster:

$ cd /var/lib/git/labs/private
$ sudo GIT_SSH=/var/lib/git/ssh git pull --rebase
[sudo] password for username: <enter your labs password>
First, rewinding head to replay your work on top of it...
Fast-forwarded production to 5c540c3f298098ededae20cb14f54a0abad26d33.

And, for good measure, restart the puppetmaster so it catches up with the changes:

$ sudo service puppetmaster restart

Can't find module?

It might happen that /etc/puppet/modules is an empty directory. It needs to be symlinked to the git working tree:

rm /etc/puppet/modules; ln -sf /var/lib/git/operations/puppet/modules /etc/puppet/modules

Looking at the layout of the puppet repo – shall I write modules or manifests?

Modules are the future! Manifests that stand outside of modules are legacy code which will (someday) be refactored into modules. If you're altering existing puppet code and don't have grand refactoring ambitions, you should just edit whatever is there now (manifest or module). If you're building a new feature, you should dive in and make a module.

How do I get my puppet classes to display in wikitech?

If you have written a new class you want to be able to select it in the configuration interface in wikitech. Click "Manage Puppet groups" in the sidebar. On top of the puppet group list you see your project. Click "Add group" and give the new group a descriptive name. Click submit. To add your class, click "Add class" under your group and give the new class the exact name it has in your manifest, for example role::mediawiki-install::labs. Click submit and you are done. Verify this by returning to "Manage Instances" and "configure" in your instance. Again, on top you see your project specific groups where now your class should be selectable with a checkbox.

If you don't see the "Add class" link then you'll have to add a group to your project. Click "Add group", name the group, and then go back to "Manage Puppet groups" and you should have an "Add class" link.

How do I add input fields in wikitech and hand over global variables to my puppet manifests?

To add an input field, go to Special:NovaPuppetGroup in wikitech, click "add variable" (in the upper part where your project's classes are). Give it the same name it has in your manifest. For readability you might want to give it a prefix hinting to your project.

An example: Let's say the global variable in your manifest is called "$::install_repo", you call it "install_repo" in wikitech (no $:: sign here). Puppet inserts the value you enter in wikitech when you have this in your manifests/templates: <%= install_repo %>. With a prefix (wikidata in this example) the same would look like this: In wikitech you would call the variable "wikidata_install_repo". In your manifests you would define the variable with $install_repo = $::wikidata_install_repo. Like this you can keep <%= install_repo %> in your manifests but have more understandable names in wikitech.

When do I check the puppetmaster self checkbox in wikitech?

Using that checkbox downloads the files to /var/lib/git/operations/puppet. This allows you to modify them locally on your instance and test your modifications. The instance will not get newly merged changes automatically. You can pull them manually (see How do I update?).

If you have developed something that got merged into the repository, you can create instance and choose your puppet classes in wikitech without ticking role::puppet::self. Then all your manifests are kept up to date by puppet, but you are not able to customize the manifests on your instance.

Where do I put my documentation?

In wikitech, next to the classes you can choose, there are little question marks linking to doc.wikimedia.org. That's our doxygen. The docs on that page are pulled straight from class-header comments in the puppet files, so please add inline documentation in your files and doxygen will fill the respective pages for you. Caution: You must not break the syntax.

An example:

# Write down some instructions
# more here
# do not leave an empty line before declaring the class you are explaining!
class blah::blubb {

How do I submit my changes to gerrit?

As mentioned above, right now, you don't do this directly from your Labs instance. Go to your local computer and set up git/gerrit there.

Clone the puppet repo locally: git clone https://gerrit.wikimedia.org/r/p/operations/puppet
Copy the files you modified to the right place in the working copy and submit them from there.

If you want your Labs instance to act as yet another remote Git repository, you have to configure Git to use sudo as well. To do that, in your local computer's Puppet repository:

git remote add my-labs-instance my-labs-instance.eqiad.wmflabs:/var/lib/git/operations/puppet
git config remote.my-labs-instance.receivepack "sudo git-receive-pack"
git config remote.my-labs-instance.uploadpack "sudo git-upload-pack"

After that, you can push and pull branches from and to your Labs instance and after finishing testing for example reset authorship information with git commit --amend --reset-author and then push to Gerrit.