For getting assistance from operations, see Operations requests.

The Ops Clinic Duty triage duty was established to ensure that tickets (and thus requests and projects) are triaged and processed in a timely fashion, providing feedback and regular updates to operations supported projects/responsibilities.

This is a duty that is fulfilled by a member of the Wikimedia Operations Team.

Schedule & Assignments

Monday to Monday
Generally volunteered/assigned during Operations Meetings, sometimes via mailing list.
During Clinic Duty the operations team member on duty should remain available in IRC & email.

Folks will follow up with the person on Ops Clinic Duty about existing tasks, as well as how to create new ones.

This duty is fairly intensive, and will interrupt a person's normal workflow on the week they are on duty..
This duty shouldn't normally require any adjustment to one's normal working schedule; if you work business hours in CET, then you wouldn't shift your hours on clinic duty for another time zone.

This should result in regularly having ops clinic duty coverage in most overlapping working timezones.

Hand-off / Takeover

Ideally all phabricator tasks are replied/commented upon in the process of reviewing and triaging, so no actual handoff of duties is required between weeks

Update the topic in IRC channel #wikimedia-operations, section 'On Ops duty:' with the person's name for that week.
Update the list in the Duty desk rotation - who is next? section below.

This is currently the public facing method of determining who is on duty.

Responsibilities

If Clinic Duty is a relaxing week for you, you are doing it wrong.
All incoming Clinic Duty tasks in phabricator can be viewed on the Ops Clinic Duty Dashboard

The idea is folks tend to have their own dashboard, which is fine when they are NOT on clinic duty. When you take clinic duty, you can install this dashboard to your homescreen during that time, and swap back to your own when finished.
Please try to refrain from editing the ops clinic dash to reflect non-clinic duties. There is a panel for 'tasks assigned to myself' at the bottom, since most of the ops clinic duty is triaging and knocking down tasks, but tend not to involve long-running personal tasks. However, even on clinic duty you need to see your tasks, so its at the bottom.

Review all incoming tasks to the #ops-access-requests, #blocked-on-operations, #operations, #patch-for-review (when its also #operations), and #wmf-nda-requests (when its also #operations) projects.

These are all included on the | Ops Clinic Duty Dashboard
Escalate, update, and follow up as needed for any incoming tasks to ensure they are worked upon.

Assign a priority to tasks that come in.
Ask for more data from requester if needed in order to confirm the request, such as date it must be completed by, additional details, etc.
Assign to the proper person.
Communicate ETA to requester, based on the workload of the person working on it.
If the request is relatively quick, just do it yourself

Maintain the 'maint-announce' queue and calendar:

This is the ONLY RT queue left for Ops Clinic Duty coverage.
Modify ticket Subject to prepend dates of effect in big-endian order (ex: 2014-11-06 to 2014-11-09: Equinix chiller maintenance)
Merge follow-up tickets as needed so that there is one per maintenance event
There is a gcal shared with all WMF named 'Ops maintenance & contracts'. All maint-announce queue tickets should be entered into this calendar.

Include the circuit IDs and RT#s in the entry. (See entries on 2014-10-07 for examples.)
Update the ticket in RT from 'new' to 'open' and comment that it has been added to the ops tracking gcal.

Review older tickets and determine if they are still active.

Follow up with ticket owners and requestors as needed on old tickets to resolve, re-assign, or escalate as needed.

Be a person of first contact, including on IRC (timezone/availability permitting).
Triage any mailing list requests for operations lists.
Try to improve the manual below.

Tips

RT can sort the ticket history with "newest first" which helps esp. with long histories and frequently-updated tickets, check "Show oldest history first" under https://rt.wikimedia.org/Prefs/Other.html

RT sometimes won't display the full ticket body, if you see an empty body click "show" on top right to show the message verbatim.

There is a clinic duty dashboard for Phabricator

Manual

This is a manual for the current "op on duty" in charge of triaging the Phabricator #Operations project.

How to handle IRC requests

If reasonable, politely ask requestor to turn into an RT ticket in the "ops-requests" queue by either simply mailing to RT or by using the web interface. If requestor lacks a real RT user (real = not just auto-created via mail without decent permissions) and is staff, offer to create one. If user relies on email replies to a ticket ensure to use "Reply" in RT instead of just commenting.

Common, small "#Operations" tickets

Mail aliases

note: ops handles only role/group mail aliases, individual mail aliases are handled by OIT as outlined here [1]

In the past these were handled manually on mchenry, but now they are puppetized. So please don't: ~~Just ssh there and edit files in /etc/exim4/aliases/~~anymore.

Instead, go to the puppet master (palladium) to the private puppet repo (/root/private/) and git commit.

There are 3 types of domains:

a) domains that have their own alias file (wikimedia.org, wikipedia.org and a few others), you will find these files in ./modules/privateexim/files, just edit them there, git commit, and presto!!!, as with any other change in the private repo.

b) domains that just link to wikimediafoundation.org. These are just symlinks and puppet generates them. If you need to add a new one or change links, go to ./manifests/mail.pp. You will find it in class exim::aliases::private and should be self-explanatory.

c) domains that link to another domain. currently just wikivoyage.de to .org, same as in b) but a separate definition in the puppet class.

It is nice to add the corresponding Phab ticket number in a comment near changed aliases. Experience shows that it can be quite handy to be able to quickly answer questions like when exactly something has been changed and who requested it. There is one file or symlink per domain name. 95% of the time the requests are just regarding the "wikimedia.org" file. In other cases make sure you check for possible symlinks and realize which domains you are actually changing when editing a specific file.

Mailman mailing lists

Public mailing lists should typically be requested through Bugzilla in the "Mailing lists" component of the Wikimedia product. Google mailing lists are managed by OIT. However, there are sometimes Mailman lists that are requested by staff through RT for internal use. Follow the normal procedure to create a Mailman mailing list and if the requested list is a private list, ensure you complete all of step 7. For all Mailman related tickets, please link the RT ticket to the master ticket by entering ticket number 3173 in the Referred to by : part of the links page.

LDAP group changes

For example, new hires should be added to "wmf" or newly created Labs users who were Subversion users in the past.

Check if they are a member of the group from the labs LDAP server: ldaplist -l group grpname | grep username
Add them if they are not there: modify-ldap-group --addmembers=username grpname
To remove someone from an ldap group you can modify-ldap-group --deletemembers=username grpname

For further instructions see Help:Access and LDAP.

Access requests

Access and reasoning for requesting it are documented on Requesting shell access. Please read and understand entirely before processing any access requests, as this very brief summary documentation may not cover all required points in the linked page.

If a request asks for things like new shell accounts, access to additional servers, log files, personal data, admin roles in systems like Mailman, Bugzilla, data center access, opening a firewall rule etc, then it is an access request and should be moved into the | Ops-Access-Requests Project. Once the initial request is made, a number of follow up steps must be confirmed:

Clinic duty person must make a linked subtask in Ops-Access-Review, no access tasks should be approved until this blocking ticket has been both created and then reviewed after the 3 day period for any objections.

This will be made to occur automatically once task | T87467 is completed.

User's direct supervisor has approved of access request via comment on phabricator task.
Approval from project lead where user's access will be granted via comment on phabricator task.

2. Confirmation that the user has read, comprehend, and signed the Acknowledgement of Wikimedia Server Access Responsibilities document.

If non-sudo, a 3 day waiting period for security review must pass AFTER the task is moved into the | Ops-Access-Requests Project.
If sudo, restrict it down as much as possible and put on the agenda for the following weeks operations meeting for team review.
Approvals must be on the phabricator ticket.
SSH public key has to be submitted via gerrit patchset by user, or by some confirmed (non-email) method (suggestion: wiki user page).
Please update the Task in phabricator, as the requestor will get update.
Please raise any security concerns on ticket via comments.

Analytics Groups

There are multiple potential groups. They have been detailed on Analytics/Data_access#Access_Groups.
- The clinic duty person can often link to this page for the person requesting access, and require the requestor to define which of the groups are required.

Creating new shell users

Please see instructions in the puppet admin module's README.

IRC channel access

/query chanserv
help access
access #channel list
access #channel add *!*@wikimedia/cloak

14:07 -ChanServ(ChanServ@services.)- Flags +Aiortv were set on ...

Removing access

Please check this section for accuracy.

Disabling an ssh key only

The most common case is that someone's laptop is gone and their ssh key must be disabled.

In admins.pp in puppet, change their ssh key from

ensure => present

to

ensure => absent

Because puppet is not always running on all hosts, you can remove the user's authorized keys file from any salt master by

salt '*' cmd.run 'rm -f /home/<username>/.ssh/authorized_keys'

And because salt doesn't run on hardy, you can check and clean up those hosts manually if puppet is not running on one or more of them.

If the user had root access, their key will also be in root-authorized-keys in the private repo for puppet and you'll need to make the corresponding change there as well.

Removing the account

Make the puppet changes for all keys as described above.

Then in the user's account class in admins.pp in puppet change

$enabled = true

to

$enabled = false

which doesn't actually do anything more than the previous step; the user account and home directory will be untouched though ssh keys will be gone.

The previous caveats about checking hosts without puppet or salt apply.

Note: there is an optional parameter that is sometimes passed to unixaccount, which would cause the user account to be removed, i.e. enabled => $enabled. Do we want to start using this regularly or get rid of it everywhere we have it? Current use is inconsistent.

Powercycling / reboots

RT duty paging for reboots is usually due to hardware failure, or immediate concerns of exploits. Anything outside those issues would be handled by normal operations workflow, and would not necessarily fall to the RT triage duty person.

Powercycling requires a passing familiarity with the different out of band management options we use (based on vendor). Hardware type can be determined by looking up the hardware in question in Racktables; then you can determine the instructions from Platform-specific_documentation.

Duty desk rotation - who is next?

Currently we assign this to folks within the team who volunteer for the duty. Please keep this list with oldest date to bottom (easier to maintain and get most relevant info faster.)

2015-11-30: Filippo
2015-11-23: Ariel
2015-11-16: Andrew Bogott
2015-11-09: Robh
...
2015-09-07: Jcrespo (jynus)
2015-08-31: Rob (robh)
...
2015-05-11: Ariel
2015-05-04: Marc-André (Coren)
2015-04-27: Brandon
2015-04-20: Rob (robh)
2015-04-13: Gage (jgage)
2015-04-06: Andrew Bogott
2015-03-30: Filippo
2015-03-23: Rob
2015-03-16: Yuvi
2015-03-02: Marc-André (Coren)
2015-02-02: Alexandros Kosiaris
2015-02-02: Andrew Bogott
2015-01-26: Rob
2015-01-19: Rob
2015-01-12: Jeff Green
2015-01-05: Chase
2014-10-13: Faidon
2014-10-06: Andrew Bogott
2014-09-29: Marc-Andre
2014-09-22: Daniel
2014-09-15: Filippo
2014-09-08: Ariel
2014-09-01: Marc-Andre
2014-08-25: Giuseppe
2014-08-18: Andrew Otto
2014-08-11: ?
2014-08-04: Alex
2014-07-28: Andrew Bogott
2014-07-21: Chris
2014-07-14: Ariel
2014-07-07: Gage
2014-06-30: Rob
2014-06-23: Otto
2014-06-16: Sean
2014-06-09: Giuseppe (harassing Faidon for help)
2014-05-19: Filippo (harassing Alex for help)
2014-05-12: Gage
2014-05-05: Jeff Green
2014-04-28: Andrew Bogott
2014-04-21: Marc-Andre
2014-04-14: Ariel
2014-04-07: Andrew O
2014-03-31: Jeff Green
2014-03-24: Rob
2014-03-17: Alex
2014-03-10: bblack
2014-03-03: ottomata
2014-02-24: apergos
2014-02-17: Jeff Green
2014-02-10: apergos
2014-02-03:
2014-01-27:
2014-01-20: ottomata
2014-01-13: Jeff_Green
2014-01-06: LeslieCarr
2013-12-30: akosiaris
2013-12-23: andrewbogott
2013-12-16: mutante
2013-12-09: apergos
2013-12-02: akosiaris
2013-11-25: RobH
2013-11-18: ottomata <-- Three weeks for real? The horror.
2013-11-11: ottomata
2013-11-04: Andrew Otto
2013-10-28: Chris
2013-10-20: Daniel
2013-10-14: Andrew Bogott
2013-10-07: Ariel Glenn
2013-09-30: Leslie Carr
2013-09-23: Andrew Otto
2013-09-16: Chris
2013-09-09: Rob
2013-09-02: Chris
2013-08-26: Marc-Andre
2013-08-19: Asher Feldman
2013-08-12 (Wikimania): Andrew Otto
2013-08-05 (Wikimania): Rob
2013-07-29: Faidon Liambotis
2013-07-22: Ryan Lane
2013-07-15: Ariel Glenn
2013-04-22: Daniel Zahn
2013-04-15: Ariel
2013-04-08: Rob
2013-04-01: Andrew B
2013-03-07: Jeff
2013-02-11: Peter
2013-01-08: Andrew Otto
2012-12-31: Faidon
2012-12-24: Leslie
2012-12-17: Ariel
2012-10-29: Faidon
2012-10-26: Asher
2012-10-15: Peter