phabricator.wikimedia.org runs on iridium in eqiad.

The Phabricator install relies on db1043 (eqiad, master). Other DB hosts (backup slaves) are db1048 (eqiad) and db2012 (codfw). Databases access is routed through dbproxy1003, a.k.a. m3-master.

General use information can be found here. This page will outline operations specific phabricator items. This includes administration as well as operations project workflows.

Operations Projects Workflows

The operations specific projects on phabricator include:

Project	Description
operations	General Operations Project
domains	Domain support/changing/issues
hardware requests	Server Allocation Requests
procurement	Vendor & Procurement Tasks. Direct ordering of SSL certificates.
network	Network Requests
Ops Access Requests	Access requests to any Operations systems
ops-codfw	Onsite queue for codfw
ops-eqiad	Onsite queue for eqiad
ops-esams	Onsite queue for esams
ops-ulsfo	Onsite queue for ulsfo
DBA	Database administration requests

,

Hardware Request Stage

User requests hardware via Operations_requests#Hardware_Requests
Rob reviews the hardware-requests project for all tasks that are assigned to him, or unassigned.

Tasks assigned to others are not reviewed as often, as they are awaiting input from the assignee. If they are left neglected by the assignee long term, they will likely be rejected, or have the hardware-requests project removed from the task.

If the system specification meets an on-site spare, system allocation may proceed.

This allocation step is typically processed by Rob and approved by Mark. (It involves a general overview of the roadmap and system procurement planning.)

If the system specifications require an order of hardware, the following occurs:

A RT procurement queue ticket is created for each set of vendor quotes.

Example: A caching system at this time could be Dell or HP, we create two RT tickets. One for each vendor to provide quotes for the system specification in question.

Quotes are generated and reviewed by Rob, Mark, and the requestors for the hardware.
Quotes are approved for purchase by Mark/Damon/Lila (escalation dependent on overall cost) and are typically placed by Rob (for US ordering) or Mark (for EU ordering).

The hardware-requests task will have the system details noted (hostname/asset tag) and the task will be linked to the system setup task.

These are kept separate for easy future search history on hardware allocations; thus its nice to leave a task with the hardware-request in said project.

Hardware/Server Setup / Deployment Stage Workflow

A new phabricator task is created in the operations project.

This task is the primary tracking task for the setup and deployment of the server.

Task should include the following (base template):
System Deployment Steps:

   [] - mgmt dns entries created/updated (both asset tag & hostname) [link sub-task for on-site work here, sub-task should include the ops-datacenter project]
   [] - system bios and mgmt setup and tested [link sub-task for on-site work here, sub-task should include the ops-datacenter project]
   [] - network switch setup (port description & vlan) [link sub-task for network configuration here, sub-task should include the network project]
   [] - production dns entries created/updated (just hostname, no asset tag entry) [link sub-task for on-site work here, sub-task should include the ops-datacenter project]
   [] - install-server module updated (dhcp and netboot/partitioning) [done via this task when on-site subtasks complete]
   [] - install OS (note jessie or trusty) [done via this task when network sub-task(s) complete]
   [] - accept/sign puppet/salt keys [done via this task post os-installation]
   [] - service implementation [done via this task post puppet/salt acceptance]

The main task is basically for all the software setup, and the sub-tasks are for the specific on-site or networking tasks.

Many times, the network task isn't created, as the person doing the software work can also do the network configuration.

Misc. Production Virtual Machine Requests Workflow

User requests hardware via Operations_requests#Virtual_Machine_Requests_.28Production.29
Alex reviews the vm-requests project for all tasks that are assigned to him, or unassigned.

Tasks assigned to others are not reviewed as often, as they are awaiting input from the assignee. If they are left neglected by the assignee long term, they will likely be rejected, or have the vm-requests project removed from the task.

If the system specifications meet all requirements for approval/allocation of a production virtual machine, Alex will process and grant the request.

Administrative Commands

All Phabricator documentation refers to scripts in the phabricator bin directory. On our setup, that is: /srv/phab/phabricator/bin/

Remove a repo

First you need the repo's callsign. This is an all-uppercase identifier with 'r' prefixed that is used in urls and such in Phabricator for the repo. For example, Puppet's is OPUP. First SSH to iridium. Then:

cd /srv/phab/phabricator
sudo ./bin/remove rFOO

Removing Two Factor Authentication

Please note that removal of 2FA is a serious request, and all too easily socially engineered. All requests of this nature should be treated with the same degree of security and confirmation as ssh key changes. The user guidelines require one month between the paste of the user committed identity hash on the wiki user page and the reset request.
Once confirmed, the actual command is quite simple, run on the phabricator host:

  sudo /srv/phab/phabricator/bin/auth strip --user <username> --all-types

You will be prompted with a yes or no to remove the multi-authentication types on the user.

External link

https://phabricator.wikimedia.org