Puppet Testing

Puppet CI

Lints
- [done] Puppet-typos
- [done] Puppet-validate
- [ongoing] Misc language-specific stuff, e.g. pep8
- Puppet-lint, someday?
  - Easy to implement, but can probably never actually block.
Unit Tests
- Spec Ruby version https://gerrit.wikimedia.org/r/#/c/72721/
  - Insecure w/volunteer code, must be jailed somehow
  - Should we run on all modules, or just the ones touched by patch?
Compile Tests
- [ON HOLD] --noop tests within modules
  - Mostly done
  - Insecure
- Run compiles for expressly-written test manifests
  - Probably safe to run on gallium if we use a fake node with ready-made facts
- Enumerate node definitions, run compile test on each (?)
  - Do this later or never
  - Need way to inject/mock up facts
Integration Tests
- Build VMs from short list of test node definitions
- Can test successfull puppet runs, but not really test that the servers work
  - This might be easy once we figure out vagrant/labs integration for Compile tests.
- Build VMs from every existing test node definition (?)
  - This will be insanely expensive, probably can't happen per patch.

Andrew idea: create an empty VM, apply a bunch of roles and make sure everything works fine after a single puppet run.

Isolation requirements: - VM must not have access network - VM must have a predefined time to live

To run the puppet compiler on a gerrit change set go to the Build with Parameters page of the puppet compilers Jenkins job, fill in GERRIT_CHANGE_NUMBER and add fully qualified domains to LIST_OF_NODES. After its done look on http://puppet-compiler.wmflabs.org/ for the build number.