Help:Access

Wikimedia Labs documentationThis is a documentation page for Wikimedia Labs. If you can improve it, please do that. You can also request help on #wikimedia-labs ^connect. For all documentation pages, please see the List of all documentation pages for labs. If you are seeking a help with a specific project, please contact the project owners.

This page explains how to gain access to this wiki and thereafter access Wikimedia Labs instances (see Terminology),for example to ssh into one to run terminal commands.

The same key and password also lets you access our git repositories located at https://gerrit.wikimedia.org/.

Prerequisites

set up an SSH key if you do not already have one.
Create an account on this wiki (Wikitech) and log in to this wiki. Your wiki username will be your ssh username.
After logging in, upload your public ssh key to Labs.
Also log in to gerrit using the same username and password. Click your name in the upper-right corner, choose Settings > SSH Public Keys, and upload your public key there as well.

Basic instructions for what to do next: Git workflow, getting started on Labs.

Accessing public and private instances

Most labs instances are not visible on the internet – a hostname like i-0000050f.eqiad.wmflabs means nothing. To access them, you must proxy through another machine that is visible to the Internet and knows about WMF's labs network; we name these "bastion" instances.

In order to access a labs instance you must be a member of the bastion project. Verify whether you are a member at 'bastion' project page by clicking "Members [Expand]".

If not, either:

please fill a Shell access request form
make an appearance on #wikimedia-labs ^connect IRC channel and ask someone to add you.

Which bastion do I use?

Almost everyone should proxy through bastion.wmflabs.org. The Wikimedia operations team must use bastion-restricted.wmflabs.org.

Accessing instances with ProxyCommand ssh option (recommended)

See SSH access for the main article about this topic.

You can tell ssh to access labs machines by proxying through a bastion instance. Using this configuration, it's possible to directly connect to your instances ( nb: you need ssh version >= 5.4 in order to use the -W option ).

Add this to the bottom of your ~/.ssh/config on your local system:

Host *.eqiad.wmflabs
    ProxyCommand ssh -a -W %h:%p <your-labs-shell-name>@bastion.wmflabs.org

Host *.wmflabs
    User <your-labs-shell-name>

Host * !gerrit.wikimedia.org
     Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
     KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256

The ssh proxy command, -a, is a security safeguard that makes sure you are never going to forward your ssh keys to Wikimedia Labs. If you are using a non-default ssh key, the next two lines should come right after User your-labs-shell-name and refer to the location of the ssh key you want to use for Wikimedia Labs.

The chacha20 and GCM ciphers are only supported in OpenSSH 6.4 and later. If you run Mac OS X with the stock SSH version you can either update to a more recent version from brew or modify the configuration to use:

Ciphers aes256-ctr,aes192-ctr,aes128-ctr
KexAlgorithms diffie-hellman-group-exchange-sha256
IdentityFile <key-filename>
IdentitiesOnly yes

Now you should be able to run the following directly from your local computer:

 ssh your-instance.eqiad.wmflabs

the ec2-style instance IDs (for example i-0000057a) exist largely for internal use. This style of instance name will not work with the proxy-command; instead, you should use the normal human-readable instance name.

Accessing instances with a graphical file manager

You can directly connect to your Labs instance (through the bastion) via ssh with a graphical file manager.

Windows: PuTTY and WinSCP.
Linux (Gnome/KDE), the setup described above (Using ProxyCommand ssh option) is a prerequisite. Afterwards, you can connect like so:
Gnome: In Nautilus, File -> Connect to server -> Type SSH, enter the instance's host name your-instance.eqiad.wmflabs and connect. In Nautilus version 3.6 and newer, use "Connect to server" and enter ssh://your-instance.eqiad.wmflabs as the address.
Mac: Using sshfs you can mount a folder on labs to an (empty, existing) local folder for browsing with your favourite file browser. sshfs can be installed e.g. from the osxfuse project or from macports (port install sshfs). To write to folders where you have only group permissions, such as projects, you can specify defer_permissions:

 sshfs -o defer_permissions your-instance.eqiad.wmflabs someLocalFolder

Accessing instances using agent forwarding

If configuring ProxyCommand doesn't work, you can use an SSH agent, and forward that agent when connecting to the bastion instance so that from it you can ssh to particular labs instances. For example:

On your local system:

eval `ssh-agent`
ssh-add ~/.ssh/your_key_file_for_labs  # (for instance: ~/.ssh/id_rsa )
ssh -A your-labs-shell-name@bastion.wmflabs.org

Then on bastion:

ssh your-instance.eqiad.wmflabs

The above would let you ssh into your instance; note that FQDN isn't completely necessary; you can also just use ssh your-instance from within bastion.

If you already use ssh or an ssh agent to connect to other hosts, you should consider using a separate agent for your labs key to avoid exposing your other keys to the labs hosts. Also, depending on your operating system, noticed this on Mac OS X, you may want to purge keys that are automatically loaded using ssh-add -d after creating the agent. This article on ssh-agent may be helpful in that regard.

See Help:Putty for using PuTTY on Windows.

Accessing services using port forwarding

If your instance doesn't have a public IP address, but you need to access a service on it, you can use port forwarding. For instance, here's how you would access a web service:

On your local system
requires port 80 and 8080 to be part of the security group rules for the instance

ssh username@bastion.wmflabs.org -L 8080:your-instance:80

# Using agent forwarding to connect to bastion
ssh username@bastion.wmflabs.org -L 8080:your-instance:80 -A

Accessing web services with a labs proxy

The easiest way to expose a web server to the public is via a labs proxy. This will provide public access to selected ports (generally just port 80) on an instance.

Accessing web services using a SOCKS proxy

If you need web access to an instance but do not wish it to be open to the public, a SOCKS proxy will allow you to set up access that's limited to your ssh bastion login.

Setting up the proxy

A SOCKS proxy makes it possible to connect directly to a labs instance from your browser's location field, so browsing to http://your-instance.eqiad.wmflabs:8080 will work.

On your local system:
ssh username@bastion.wmflabs.org -D 8080
Configure your computer to use localhost:8080 as a SOCKS proxy. You can configure this in a few different ways (whichever fits you best):
- On a Mac, open System Preferences > Network > Advanced > Proxies. Tick "Socks Proxy" and enter localhost and 8080. After saving and applying this you can browse to http://your-instance.eqiad.wmflabs:8080 in the browser.
- If you use foxy proxy (a Firefox, IE or Chrome add-on), you can use the pre-defined URL pattern defined here. After adding the URL patterns, you'll also need to edit the defined proxy to specify localhost as the host and 8080 as the port.

Troubleshooting the proxy

Test to see if the proxy is working:

load http://whatismyipaddress.com/. If you see your own IP address, your browser is not sending traffic through the proxy. If you see 208.80.153.207, the browse is correctly sending traffic through the proxy.
If you see a Server Not Found error when you browse to http://your-instance.eqiad.wmflabs you may need to tell your browser to use remote DNS instead of local.
- To do this on Firefox, point your browser at about:config and search for network.proxy.socks_remote_dns. If it is set to false double click on the row to set it to true.

Troubleshooting

In general, adding ssh option -v, -vv, or -vvv may help identify possible issues.

#when using Agent Forwarding
ssh -Av username@bastion.wmflabs.org

#when using ProxyCommand
ssh -v your-instance.eqiad.wmflabs

Into Bastion

Permission denied (publickey)

Make sure you have uploaded the correct ssh key to your preferences
Use lowercase letters for your username
Your ssh user name is your instance shell account name name (see User Profile > Basic Information in your WikiTech account's Preferences page). It is not necessarily the same as your account's username

Connection closed by remote host

Make sure you have uploaded the correct ssh key to your preferences
If you have access to other ssh servers, can you connect to them? If not, then there may be an issue with your ssh client.
If you use Windows, is Pageant set up with correct keys and running?
Contact a Wikimedia Labs admin via #wikimedia-labs ^connect IRC and let them know about your issue.
They can look into the following :

make sure your username is in the bastion project - on the Special:NovaInstance page you should see “bastion” as one of the options under the project filter.
clear any cache settings or deny host settings that may also prevent you from ssh’ing into bastion.

Blocking connection on OS X with no error message

If you are running OS X and your ssh connection blocks without any error message (while pinging the server works), try

unset SSH_AUTH_SOCK

and then ssh again. This will unset the socket to ssh-agent.

Into your-instance

Permission denied (publickey)

Make sure the instance build has actually completed.
Look in the console output for “Finished puppet run”, BEGIN SSH HOST KEY FINGERPRINTS and BEGIN SSH HOST KEY KEYS output.
If using Using agent forwarding, make sure you ssh’d into Bastion with the -A option.

Banners

If you log into any instance apart from bastion, ssh will display:

If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances

This message cannot be suppressed, and you need to manually filter it if you want to process the output.

Rights

Anonymous users

You'll need to have an account created for you. If you currently have SVN access, then you have an account, but need to have it linked to Labs (how-to for admins). As of November 2011‎, we are still working out the account activation process, but hope to have it done soon.

Logged-in users

After creating an account, you can:

Add and manage your public SSH keys

Once you add a key, you'll be able to log into the instance of any project you are a member of. Have a current project member, or an admin user add you to a project.

To create instances within a project, you'll need to have a projectadmin on the project, or an admin add you to the projectadmin role for that project.

You can make queries for nova resources; currently only Nova instances have semantic properties enabled.

After logging in, you can also access Gerrit; if you wish to do git checkouts of the puppet repositories, you'll need to log into Gerrit, and add your SSH key there as well. Note: it would be nice if Gerrit could use LDAP for its SSH keystore, instead of its database; I've opened a bug for this, if you'd like to help, please add that feature to Gerrit!

Project Members

If you are a member of a project (to check, in its Nova_Resource:projectName page expand the list of Members) then you can:

ssh to project instances

Admins

Project Admins

If you are a Project Admin, you can:

Cloud Admins

In addition to all actions that projectadmins can do, cloudadmins can:

Create or manage DNS domains.

Wiki Admin

If you are a wiki admin, you can:

Create/delete Nova projects and manage project and project role membership.

Puppet

Labs instance run from the operations/puppet repository and branch 'production'.

To trigger a puppet update, run: sudo puppet agent -tv (used to be sudo puppetd -tv)

The cron job that updates the master puppet repo in the labs cluster runs every minute. So your change got merged in, you will have to wait a bit.

Access FAQ

Q: How do I get access to Labs?: A: Just sign up, then upload your SSH key.

Q: I just want git access; I don't care about wikitech. How do I get Git access?: A: You should care about wikitech! However, that doesn't matter. Wikitech access and Git access are currently given using the same mechanism, therefore it is a unified login.

Q: I was added to a group that gives me access to something in git, but it isn't working, what's wrong?: A: Once you have been added to the group, you need to log out of gerrit, then back in. Gerrit pulls its groups from LDAP, but caches them. Logging out, then back in re-synchronizes your groups, and thus clears the cache.

Q: How do I add a new tool to Labs or migrate a tool from the toolserver to labs?: A: Such as the first question above, sign up, upload your SSH key, but you also need to ask for a Nova Administrator to grant you Nova accesss before you are able to create a new tool. To migrate a tool, or for general insight, read these instructions from User:Magnus Manske.