SSH access

From Wikitech

SSH configuration

Ideally one restricts and does NOT forward a key into shared systems. This helps to prevent SSH agent hijacking. However, setting up your configuration to avoid this may be confusing, so here is sample configuration for your ~/.ssh/config file.

Please note that we do not generally allow agent forwarding. Ensure you don't have any alias to automatically add -A to your ssh commands.

Labs

Host bastion.wmflabs.org
    ProxyCommand none
    ControlMaster auto

Host *.wmflabs.org *.wmflabs
    User your_username_here
    IdentityFile ~/.ssh/your_labs_ssh_key
    ProxyCommand ssh -a -W %h:%p bastion.wmflabs.org

If you're in ops, you can replace bastion in the example above with bastion-restricted.

Production

Host bast1001.wikimedia.org
    ProxyCommand none
    ControlMaster auto

Host *.wikimedia.org *.wmnet
    User your_username_here
    ProxyCommand ssh -a -W %h:%p bast1001.wikimedia.org
    IdentityFile ~/.ssh/your_production_ssh_key

In the example above you may replace bast1001.wikimedia.org with a bastion that's physically closer to you:

  • bast1001.wikimedia.org (at eqiad; in Ashburn, Virginia, United States)
  • bast2001.wikimedia.org (at codfw; in Carrolton, Texas, United States)
  • hooft.esams.wikimedia.org (at esams; in Amsterdam, The Netherlands)
  • bast4001.wikimedia.org (at ulsfo; in San Francisco, United States)
Map of bastion hosts

Production SSH keys

Make a custom key just for production servers, and make sure your ssh key is listed in modules/admin/data/data.yaml in the puppet repo.

Production ops

Ops go through the restricted bastion iron.wikimedia.org (eqiad) instead.

## Production & External Zones

Host iron.wikimedia.org bast1001.wikimedia.org bast2001.wikimedia.org hooft.esams.wikimedia.org bast4001.wikimedia.org
    ProxyCommand none
    ControlMaster auto

Host *.wikimedia.org
    User your_username_here
    IdentityFile ~/.ssh/your_production_ssh_key
    ProxyCommand ssh -a -W %h:%p iron.wikimedia.org

## Internal Zones

Host *.wmnet
    User your_username_here
    IdentityFile ~/.ssh/your_production_ssh_key

Host *.eqiad.wmnet
    ProxyCommand ssh -a -W %h:%p iron.wikimedia.org

Host *.codfw.wmnet
    ProxyCommand ssh -a -W %h:%p bast2001.wikimedia.org

Host *.esams.wmnet
    ProxyCommand ssh -a -W %h:%p hooft.esams.wikimedia.org

Host *.ulsfo.wmnet
    ProxyCommand ssh -a -W %h:%p bast4001.wikimedia.org

## Networking Equipment

Host *-eqiad.mgmt.wikimedia.org
    ProxyCommand ssh -a -W %h:%p iron.wikimedia.org

Host *-codfw.mgmt.wikimedia.org
    ProxyCommand ssh -a -W %h:%p bast2001.wikimedia.org

Host *-esams.mgmt.wikimedia.org *-knams.wikimedia.org
    ProxyCommand ssh -a -W %h:%p hooft.esams.wikimedia.org

Host *-ulsfo.mgmt.wikimedia.org
    ProxyCommand ssh -a -W %h:%p bast4001.wikimedia.org

Fundraising infrastructure

Host tellurium.wikimedia.org
    User your_username_here
    IdentityFile ~/.ssh/your_frack_ssh_key
    ProxyCommand none

Host *.frack.* payments100* indium pay-lvs* silicon boron samarium db1025 db1008 barium thulium lutetium backup4001
    User your_username_here
    IdentityFile ~/.ssh/your_frack_ssh_key
    ProxyCommand ssh -a -W %h:%p tellurium.wikimedia.org

Managing multiple SSH agents

This describes a method for maintaining a separate ssh-agent to hold your ssh key for connecting to Labs.

The problem

You use an ssh-agent to connect to your personal or company systems. You want to connect to Labs using an agent and you created a separate ssh key to connect to Labs, but you don't want to forward your personal key to Labs systems. If you just add both keys to your existing agent, they both get forwarded to Labs. It's a pain to constantly remove your personal key from your agent each time you want to connect to Labs. Additionally, you might be connected to both your personal system and labs simultaneously, so just removing the key is insufficent; you must run a separate ssh-agent. You don't want to run one agent per connection because then you have to type your passphrase on every connection (and you have a nice long secure passphrase on your key).

This page describes a method for getting your shell to maintain two agents, your primary agent and your labs agent. When you connect to labs you connect to the existing labs agent (or create one if it doesn't exist) and the rest of the time you use your default agent.

OS X solution

The default terminal application can be modified in how it runs to have every tab run its own ssh-agent.

  • Open Terminal
  • Open Terminal preferences
  • Select the terminal style and settings on the left for the one in use (usually already defaulted to this choice)
  • Select the 'Shell' Tab.
  • Check the box to run the command on startup, and populate the command eval `ssh-agent` in the field.
  • Also ensure 'Run in Shell' is checked.
  • New tabs you open will now use this setting.

Linux solutions

Using multiple agents via systemd

This requires the use of a Linux distribution using systemd as the init system (all current releases do that, e.g. Debian jessie or Ubuntu 15.10).

You can start multiple ssh-agents through systemd user units. The following unit would e.g. connect to labs, copy it to /usr/lib/systemd/user/ssh-labs.service (and similar to wherever you else want to connect):

[Unit]
Description=SSH authentication agent for labs
Before=default.target

[Service]
Type=forking
Environment=SSH_AUTH_SOCK=%t/ssh-labs.socket
ExecStart=/usr/bin/ssh-agent -a $SSH_AUTH_SOCK

[Install]
WantedBy=default.target

Then run the following command as your regular user (and similar for the other agent(s)):

systemctl --user enable ssh-labs

This will create the agent socket ssb-labs.socket inside the $XDG_RUNTIME_DIR directory (which is automatically created).

Start the agent as follows to check if the systemd user unit works properly. There is no need to do this afterwards, later on the unit will be started during your first login.

systemctl --user start ssh-labs.service

Finally whenever you want to connect to either labs or elsewhere, point SSH_AUTH_SOCK to the respective socket.

The simplest solution

There is an easy answer to this problem, though it's not very flexible. Run two terminals on your workstation. Load a fresh agent in one of them. Always use one to connect to labs and the other to connect other places.

A more complex solution

The items listed here are entirely untested by current staff, and left over from the past.

This solution has the advantage of being able to connect to Labs or other hosts indiscriminately from any terminal running on your workstation (or in screen) etc. It protects you against accidentally attempting to authenticate against labs with the wrong key.

Setup

This solution assumes you are running bash as your local shell. It can probably be adapted for other shells with minimal effort. It involves creating a socket connected to your ssh-agent at a predictable location and using a bash function to change your environment to use the labs agent when connecting to labs.

This solution is also geared towards running screen. It's a little more complicated than necessary because when disconnecting then reconnecting to a screen session, the SSH_AUTH_SOCK has usually changed. We override that with a predictable location so that as the agent moves around the old screen sessions still have access to the current agent.

We start by creating a socket that can talk to our regular agent at a predictable location every time we start a new shell. In .bashrc:

 if [ -f ~/.persistent_agent ]; then source ~/.persistent_agent; fi
 persistent_agent /tmp/$USER-ssh-agent/valid-agent

Next we set up a function specifically for connecting to labs

 # ssh into labs with an isolated agent
 function labs() {
   oldagent=$SSH_AUTH_SOCK
   SSH_AUTH_SOCK=
   persistent_agent /tmp/$USER-ssh-agent/labs-agent
   # add the key if necessary
   if ! ssh-add -l | grep -q labs-key-rsa; then
       ssh-add ~/.ssh/labs-key-rsa
   fi  
   ssh -A -D 8080 bastion.wmflabs.org
   SSH_AUTH_SOCK=$oldagent
 }

And one to copy content into labs (scp into labs)

 # scp into labs with an isolated agent
 function labscp() {
   oldagent=$SSH_AUTH_SOCK
   SSH_AUTH_SOCK=
   persistent_agent /tmp/$USER-ssh-agent/labs-agent
   # add the key if necessary
   if ! ssh-add -l | grep -q labs-key-rsa; then
       ssh-add ~/.ssh/labs-key-rsa
   fi  
   scp "$@"
   SSH_AUTH_SOCK=$oldagent
 }

Last, we make sure we clean up our old agents if we completely disconnect from the system otherwise we'll wind up with the agent running even when we're not connected to labs. This is a little tricky because we don't want to kill the agent when we close the first connection we made to labs but only when we're actually done working. As a proxy for 'done working', I use 'I log out of the last shell i have open on this system'. This is not a great solution because if the connection dies or I just quit Terminal or something like that instead of specifically logging out, .bash_logout doesn't get run. Add to .bash_logout:

 # if this is the last copy of my shell exiting the host and there are any agents running, kill them.
 if [ $(w | grep $USER | wc -l) -eq 1 ]; then
   pkill ssh-agent
 fi

Just for good measure, let's throw a line in my user crontab that will kill any agents running if I'm not logged in:

 # if I'm not logged in, kill any of my running ssh-agents.
 * * * * * if ! /usr/bin/w | /bin/grep ben ; then /usr/bin/pkill ssh-agent; fi > /dev/null 2>&1

Finally, here is the code for the persistent_agent function

 ## preconditions and effects:
 ## $validagent already exists and works, in which case we do nothing
 ## SSH_AUTH_SOCK contains a valid running agent, in which case we update $validagent to use that socket
 ## SSH_AUTH_SOCK is empty, in which case we start a new agent and point $validagent at that.
 ## SSH_AUTH_SOCK exists but doesn't actually connect to an agent and there's no existing validagent; we'll start a new one.
 ## end result:
 ## validagent always points to a running agent, either local or your existing forwarded agent
 function persistent_agent() {
   validagent=$1
   validagentdir=$(dirname ${validagent})
   # if it's not a directory or it doesn't exist, make it.
   if [ ! -d ${validagentdir} ]
   then
       # just in case it's a file
       rm -f ${validagentdir}
       mkdir -p ${validagentdir}
       chmod 700 ${validagentdir}
   fi  
   # only proceed if it's owned by me
   if [ -O ${validagentdir} ]
   then
       # update the timestamp on the directory to make sure tmpreaper doesn't delete it
       touch ${validagentdir}
       # if the validagent arleady works, we're done
       orig_sock=$SSH_AUTH_SOCK
       SSH_AUTH_SOCK=${validagent}
       if ssh-add -l > /dev/null 2>&1; then
           return
       fi  
       SSH_AUTH_SOCK=$orig_sock
       # ok, the validagent doesn't arleady work, let's move on towards setting it up.
       # if SSH_AUTH_SOCK is a valid agent, we'll use it.
       if ssh-add -l > /dev/null 2>&1; then
           ln -svf $SSH_AUTH_SOCK $validagent
           SSH_AUTH_SOCK=$validagent
           return
       fi  
       # note - inverting the order of the previous two tests changes behavior from 'first valid agent gets $validagent' to 'most recent valid agent gets $validagent'.
       # ok, at this point SSH_AUTH_SOCK doesn't point to a valid agent (it might be empty or have bad contents)
       # let's just start up a new agent and use that.
       echo "triggering new agent"
       eval $(ssh-agent)
       ln -svf $SSH_AUTH_SOCK $validagent
       SSH_AUTH_SOCK=$validagent
       return
   fi  
   # at this point, I failed to own my $validagentdir.  Someone's trying to do something nasty?  Who knows.
   # I've failed to create a validagent.  Announce that and bail.
   echo "Failed to create a valid agent - bad ownership of ${validagentdir}"
   return
 }
Use

Note that I already have my regular key loaded:

 ben@green:~$ ssh-add -l
 2048 25:9e:91:d5:2f:be:73:e8:ff:37:63:ae:83:5b:33:e1 /Users/ben/.ssh/id_rsa (RSA)

The first time (in a given day) you connect to labs, you are prompted to enter the passphrase for your key, and when you get to bastion, it can only see your labs key:

 ben@green:~$ labs
 triggering new agent
 Agent pid 32638
 `/tmp/ben-ssh-agent/labs-agent' -> `/tmp/ssh-YfZWc32637/agent.32637'
 Enter passphrase for /home/ben/.ssh/labs-key: 
 Identity added: /home/ben/.ssh/labs-key (/home/ben/.ssh/labs-key)
 [motd exerpted]
 ben@bastion:~$ ssh-add -l
 2048 60:a2:b5:a5:fe:47:07:d6:d5:78:50:50:ba:50:14:46 /home/ben/.ssh/labs-key (RSA)

When connecting the subsequent shells (until the end of the day when you log out of your workstation and all your agents are killed), you are connected without being prompted for your passphrase.

 ben@green:~$ labs
 [motd exerpted]
 ben@bastion:~$

Copying files means just using labscp instead of scp:

 ben@green:~$ labscp foo bastion.wmflabs.org:/tmp/
 foo                                    100%   43KB  43.0KB/s   00:00

But when you log out of bastion (in any connection), your normal key is once again available for connecting to personal or other hosts:

 ben@bastion:~$ logout
 Connection to bastion.wmflabs.org closed.
 ben@green:~$ ssh-add -l
 2048 25:9e:91:d5:2f:be:73:e8:ff:37:63:ae:83:5b:33:e1 /Users/ben/.ssh/id_rsa (RSA)

See also

Here are some of the devs' own configs: greg