Salt

Salt is a remote execution framework built on a pub/sub model (0mq), used for instance for Trebuchet. Hosts executing remote commands are called minion hosts; they receive their orders from and send data back to the master host.

For information on upgrading Salt in production/labs, see Salt/Upgrades.

Feature overview

Grains

Salt grains are like puppet facts. Grains are a set of information specific to a minion. Many grains are automatically generated on hosts, but grains can also be set via a minion's configuration file, via puppet (using the salt::grain definition) or using salt itself via the grains.setval module call.

Grains can be used for targeting, or can be accessed from within custom modules for system information.

To list all grains on a host, do:

 sudo salt-call grains.items

Pillars

Pillars are information set by the salt master for minions. They are similar to providing variables via external node classifiers (ENC) in puppet. On our salt masters, the pillar location is /srv/pillars. Pillars are defined in state files [1] under directories and these state files are applied to minions via /srv/pillars/top.sls. We're currently managing pillar information via puppet, so you should not modify this information manually.

Pillars can be used for targeting, but are very useful as global configuration for runners or custom modules.

See modules/deployment/templates/pillars/deploy.sls.erb for an example of a pillar state file, and [2] for official documentation.

Targeting

Salt's major feature is remote execution, and to properly use remote execution being able to easily target minions is key. Salt has a number of methods for targeting minions; feel free to try any of these examples from the production salt master:

Glob matching:
- salt 'mw*' test.ping
Regex matching:
- salt -E '(mw|srv).*' test.ping
Grain matching:
- salt -G 'deployment_target:parsoid' test.ping
Pillar matching:
- (no good pillar examples right now)
Compound matching:
- salt -C '*eqiad* and G@deployment_target:mediawiki'

Salt master locations

Production: sockpuppet and palladium as multimasters (keys must be accepted/rejected/deleted on both)

Labs: virt0

Useful examples

adding and removing system keys

List keys:

 salt-key -L

Add key:

 salt-key -a <hostname>

Delete key

 salt-key -d <hostname>

get kernel versions

salt '*' grains.item kernelrelease

run command

salt '*.wmflabs' cmd.run 'echo "ran echo on `hostname`"'

Restart service

salt 'mw*.eqiad.wmnet' service.restart 'apache2'

List/ping all nodes with a puppet role

salt -G 'rolename:role::zuul::production' test.ping

List/ping with wildcards on multiple roles

salt -G 'rolename:role::ci::*' test.ping

Run command on all nodes in a puppet role

salt -G 'rolename:role::bugzilla' cmd.run 'uname -a'

-G for grain, "rolename" is our custom grain we add via puppet. "role::bugzilla" is the value of rolename as it appears in puppet.

Which other grains are there / List all the grains a node has

salt 'zirconium.wikimedia.org' grains.ls

zirconium.wikimedia.org:
    - biosreleasedate
    - biosversion
    - cluster
    - cpu_flags
    - cpu_model
    - cpuarch
    - defaultencoding
    - defaultlanguage
    - deployment_target    (custom, added by WMF puppet code)
    - domain
    - fqdn
    - fqdn_ip4
    - fqdn_ip6
    - gpus
    - host
    - id
    - ip_interfaces
    - ipv4
    - ipv6
    - kernel
    - kernelrelease
    - localhost
    - lsb_distrib_codename
    - lsb_distrib_description
    - lsb_distrib_id
    - lsb_distrib_release
    - manufacturer
    - master
    - mem_total
    - nodename
    - num_cpus
    - num_gpus
    - os
    - os_family
    - osarch
    - oscodename
    - osfinger
    - osfullname
    - osrelease
    - path
    - productname
    - ps
    - pythonpath
    - pythonversion
    - realm
    - rolename      (custom, added by WMF puppet code)
    - saltpath
    - saltversion
    - saltversioninfo
    - serialnumber
    - server_id
    - shell
    - site
    - virtual

Get all other grains and their values for a node

salt 'zirconium.wikimedia.org' grains.items

This is the same as above but also lists the actual grain values.

Get all grains and their values for an entire puppet role

salt -G 'rolename:role::otrs::webserver' grains.items

The same on a role by combining the 2 examples above.

Where are docs on other salt commands besides just "cmd.run"

salt 'palladium.eqiad.wmnet' sys.doc  | less