Server Lifecycle

This page describes the lifecycle of Wikimedia servers, starting from the moment we acquire them and until the time we unrack them. A server has various states that it goes through, with several steps that need to happen in each state. The goal is to standardize our processes for 99% of the servers we deploy or decomission and ensure that some necessary steps are taken for consistency, manageability & security reasons.

Server states

Requested

New hardware is requested for use via the instructions on Operations_requests#Hardware_Requests.
Hardware Allocation Tech will review request, and detail on ticket if we already have a system that meets these requirements, or if one must be ordered.

If further details are needed, the task will be assigned back to requester for additional information.

If hardware is already available and request is approved by operations management, system will be allocated, skipping the rest of this process to the Existing System Allocation step.
If hardware must be ordered, the buyer will gather quotes from our approved vendors & perform initial reviews on quote(s).

At this time, quotes are still in RT

Technical review is done by operations team members familiar with hardware in question (see below), attaching their confirmation or corrections.

If there are corrections, ticket goes back to buyer and requester as needed until issues are clarified otherwise escalate to Systems Architect(s).

System Architect(s) perform final technical review to ensure the technical correctness, cost effectiveness, & architecture/roadmap alignment; attaching confirmation or corrections.

If there are corrections, ticket goes back to buyer and requester as needed until issues are clarified otherwise escalate to Operations Management.

Operations Management reviews ticket and attaches approval(s) or questions as needed & assigns ticket back to buyer.
Buyer may create an on-site hardware confirmation task, this ticket will confirm all parts, cables, and assorted items are available for incoming system.

Ticket is assigned to on-site tech, who must confirm or request the required hardware accessories needed to support the system.
Order may proceed even if all hardware is not on site, depending on missing hardware and lead times.

Existing System Allocation

Only existing systems (not new) use this step if they are requested.

If a system must be ordered, please skip this section and proceed to Ordered section.

System name is changed if required.
All puppet files are ensured to be absent of old server declarations.

System should have already been cleared out of monitoring.
If system listed in puppet decommission files to clear said monitoring, it should be removed from those files before it is allocated again.

All old system keys are confirmed to be removed from puppet master.
All ssh authorized
If all the above are good, the Hardware Allocation Tech will update your RT ticket and the Server Spares page to reflect the allocation.
Skip following steps until the Installation section.

Ordered

Only new systems (not existing/reclaimed systems)
Buyer purchases hardware, attaching ordering details to ticket.
Once order ships, buyer places inbound shipment ticket with datacenter vendor. Point of Contact details here.
Buyer assigns RT procurement ticket to the on-site technician to receive in hardware.

Post Order

An installation/deployment task should be created (if it doesn't already exist) for the overall deployment of the system/OS/service & place in the #operations project.

You can include the following steps on this ticket for ease of reference (taken from the entirely of the lifecycle document):

 System Deployment Steps:
  [] - mgmt dns entries created/updated (both asset tag & hostname) [link sub-task for on-site work here, sub-task should include the ops-datacenter project]
  [] - system bios and mgmt setup and tested [link sub-task for on-site work here, sub-task should include the ops-datacenter project]
  [] - network switch setup (port description & vlan) [link sub-task for network configuration here, sub-task should include the network project]
  [] - production dns entries created/updated (just hostname, no asset tag entry) [link sub-task for on-site work here, sub-task should include the ops-datacenter project]
  [] - install-server module updated (dhcp and netboot/partitioning) [done via this task when on-site subtasks complete]
  [] - install OS (note jessie or trusty) [done via this task when network sub-task(s) complete]
  [] - accept/sign puppet/salt keys [done via this task post os-installation]
  [] - service implementation [done via this task post puppet/salt acceptance]

Receiving Systems On-Site

Before the new hardware arrives on site, a shipment ticket must be placed to the datacenter to allow it to be received.

If the shipment has a long enough lead time, the buyer should enter a ticket with the datacenter site. Note sometimes the shipment lead times won't allow this & a shipment notification will instead be sent when shipment arrives. In that event, the on-site technician should enter the receipt ticket with the datacenter vendor.

New hardware arrives on site & datacenter vendor notifies us of shipment receipt.
Packing slip for delivery should list an RT # & the RT ticket should have been assigned to the on-site technician for receipt at this time.
Open boxes, compare box contents to packing slip. Note on slip if correct or incorrect, scan packing slip and attach to ticket.
Compare packing slip to order receipt in the RT ticket, note results on ticket.
If any part of the order is incorrect, reply on RT ticket with what is wrong, and assign the ticket to the buyer on the ticket.
If the entire order was correct, please note on the procurement ticket. Unless the ticket states otherwise, it can be resolved by the receiving on-site technician at that time.
Assign asset tag to system, enter system into Racktables immediately, even if not in rack location.

Some systems will have a hostname assigned to them at time of order (usually for clustered systems), if it has been assigned, it will be on the procurement ticket. If it has not been assigned, name the systems in racktables with the Asset Tag under server name, and leave visible label blank.
Entry into racktables should always include the following: asset tag, OEM Serial Number (or service tag), hardware type (dropdown), procurement rt#, purchase date (on order), hardware warranty expiration, and location of system (tag).

The location tag lets us see whats in a location but NOT racked, so please make sure to check which site for each system.
Hardware warranties should be listed on the order ticket, most servers are three years after ship date.
Network equipment has one year coverage, which we renew each year as needed for various hardware.

Racked

A Phabricator task should exist with racking location and other details; made during the post-order steps above.
Hostname may be assigned (or system may refer to asset tag name until it is allocated for specific role)

Please see Server naming conventions for details on how hostnames are determined.
If hostname was not previously assigned, a label with name must be affixed to front and back of server.

If system has a front LCD, please see instructions on how to set the name on it via Platform-specific documentation.

DNS is updated for the mgmt network connections.

DNS for mgmt should include both the assettag.mgmt.site.wmnet as well as hostname.mgmt.site.wmnet.
DNS for production network will be set only for hostname, since a system will have a hostname before going on the production network, systems may not have this set if their usage isn't yet determined.

Racktables entry updated to reflect rack location.
System Bios & out of band mgmt setttings are configured at this time..

See the Platform-specific documentation for setup instructions for each system type.
Serial Redirection and mgmt must be tested at this time

On-site Tech should fully test the mgmt interface to ensure it responds to ssh, they are able to login, reboot the system, and watch a successful BIOS POST over serial console.

Switch port(s) are assigned and labeled.

Label with hostname, if not available label with asset tag.
VLAN assignment is completed at this time only if system role is known.

After systems have been racked, if they are not immediately allocated to a service (IE: they are spare), a ticket should be created in core-ops and assgined to the HW Allocation Tech with the asset tags, so they can add to spares list.

Installation

Hostname must be assigned at this point.

Please see Server naming conventions for details on how hostnames are determined.
If hostname was not previously assigned, a label with name must be affixed to front and back of server.

If system has a front LCD, please see instructions on how to set the name on it via Platform-specific documentation.

DNS setup for production network.

$assettag.mgmt.$loc.wmnet should have been setup when the system was racked.
set up $name.mgmt.$loc.wmnet to the same IP as $assettag.mgmt.$loc.wmnet.

VLAN: Network port is set to proper vlan (and labeled with hostname if not yet labeled.)

Folks who can handle vlan assignments: Chris J, Faidon L, Mark B, Rob H.

Any ops folks who want this ability should speak to our network admins.

DHCP: Add server to appropriate file in Puppet, based on serial console port and speed:

modules/install-server/files/dhcpd/linux-host-entries.ttyS0-9600 = com port 1, speed of 9600
modules/install-server/files/dhcpd/linux-host-entries.ttyS0-115200 = com port 1, speed of 115200
modules/install-server/files/dhcpd/linux-host-entries.ttyS1-115200 = com port 2, speed of 115200 (most hosts)
You can pull this information from the management of most systems, as described in their specific pages under Platform-specific documentation.

Decide on partition mapping & add server to modules/install-server/files/autoinstall/netboot.cfg

Detailed implementation details for our Partman install exist here.
The majority of systems should use automatic partitioning, which is set by inclusion on the proper line in netboot.cfg.
Any hardware raid would need to be setup manually via rebooting and entering raid bios.

Right now there is a mix of hardware and software raid availability.

File located @ puppet modules/install-server.

partman recipe used located in modules/install-server
Please note if you are uncertain on what to pick, you should lean towards LVM.

Many reasons for this, including ease of expansion in event of filling the disk.

Proceed with Installation

Reboot system and boot from network / PXE boot.
acquires hostname in DNS
acquires DHCP/autoinstall entries
gets installed

Post-Install: Get puppet running

Nothing replaces fully understanding how our deployment of Puppet operates, as detailed on the service info page.
Warning: if you are rebuilding a pre-existing server (rather than a brand new name), on the puppet master (palladium), run puppet cert destroy $server_fqdn to clear out the old certificate before beginning this process. If you already began, also run (on the server you're building, not the puppet master) find /var/lib/puppet/ssl -type f -exec rm {} \; to clean out the client.
Login to restricted bastion (iron, used for boxes in the labs-private vlan) or the puppet master (palladium, used for everything else).

from iron or palladium, sudo /usr/local/sbin/install-console $server_fqdn to log into $server

on $server, run puppet agent --test

It should whine that it can't get its cert automatically.

on the puppet master (definitely palladium, this time), run puppet cert -l to list all pending certificate signings.
on the puppet master, run puppet cert -s $server_fqdn for the specific server you wish to sign keys for.
Now again on $server, run puppet agent --enable to administratively enable puppet, and then puppet agent --test. It should now succeed.
Now that puppet has run, sign the new salt key on the puppetmaster: salt-key -a $server_fqdn
After your first couple of successful puppet runs, you should reboot just to make sure it comes up clean.

Your host should now appear in puppet stored configs and therefore in ganglia and icinga.

Important: check that salt-minion is properly running on the host. If your host does not include the base or standard classes from puppet, include the specific class for the salt minion (at this writing role::salt::minion).

Look to see if the process 'salt-minion' is running.
Check /var/log/salt/minion for messages about keys; if you see continuing messages "The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate", you need to accept the key on the salt master host, see Salt for how to do this.

In Service

Where all servers aspire to be.

Reinstallation

A Phabricator ticket should be created detailing the reinstallation in progress.
System services must be confirmed to be offline. Checking everything needed for this step and documenting it on this specific page is not feasible at this time. Please ensure you understand the full service details and what software configuration files must be modified. This document will only list the generic steps required for the majority of servers.
The following instructions assume the system is online and responsive. If the system is offline, simply skip the steps requiring you do run something on the host.
If server is part of a service pool, ensure it is set to false or removed completely from pybal/LVS.

Instructions on how to do so are listed on the LVS page.

If server is part of a service group, there will be associated files for removal or update. The service in question needs to be understood by tech performing the decommission (to the point they know when they can take things offline.) If assistance is needed, please seek out another operations team member to assist.

Example: db class machines are in associated db-X.php, memcached in mc.php.

Remove server entry from DSH node groups.

These files are maintained in operations/puppet:/files/dsh/group.

Put the downtime into icinga if said downtime will generate pages.
Disable puppet on the host (puppet agent --disable)

Admin log whenever you disable or enable puppet on a host!

Manually revoke keys from puppet.

Instructions on how to do so are on the Puppet service details page.
$ puppet cert clean <fqdn>

Power down system.
Manually revoke keys from salt.

Instructions on how to do so are on the Salt service details page.
$ salt-key -d <fqdn>

After the above is done, system installation can proceed from the Installation section above.

Reclaim or Decommission

These steps can be done one at a time and/or with long breaks in between.

A Phabricator ticket should be created detailing if system is being decommissioned (and removed from datacenter) or reclaimed (wipe all services and set system as spare for reallocation).

Place the operations project on the task.

System services must be confirmed to be offline. Checking everything needed for this step and documenting it on this specific page is not feasible at this time(but we are working to add them all). Please ensure you understand the full service details and what software configuration files must be modified. This document will only list the generic steps required for the majority of servers.
The following instructions assume the system is online and responsive. If the system is offline, simply skip the steps requiring you do run something on the host.
If server is part of a service pool, ensure it is set to false or removed completely from pybal/LVS.

Instructions on how to do so are listed on the LVS page.

If server is part of a service group, there will be associated files for removal or update. The service in question needs to be understood by tech performing the decommission (to the point they know when they can take things offline.) If assistance is needed, please seek out another operations team member to assist.

Example: db class machines are in associated db-X.php, memcached in mc.php.

Remove server entry from DSH node groups.

These files are maintained in operations/puppet:/modules/dsh/files/group.

Remove from puppet stored configuration files.

remove from site.pp (puppet:///manifests/site.pp)
remove from netboot.cfg (puppet:///modules/install-server/files/autoinstall/netboot.cfg)
remove from DHCPD lease file (puppet:///modules/install-server/files/dhcpd/linux-host-entries.ttyS... filename changes based on serial console settings)

End steps

These steps, once started, must be completed without interruption.

Disable puppet on the host (puppet agent --disable)

Admin log whenever you disable or enable puppet on a host!

Remove from Icinga monitoring:

Manually run puppetstoredconfigclean.rb <fqdn> on the puppet master.
Run puppet on the icinga host (at this writing, neon) and be sure it completes successfully, including refresh of Icinga. If it fails, try /usr/sbin/icinga -v /etc/icinga/icinga.cfg to see what's wrong with the configuration.
If you can't get puppet to run happily on icinga, get help. If help is not available, stop, renable puppet on the host, and start these steps again once you can get help with icinga.
You should either ensure monitoring is removed, or at minimum disable notifications for that host. Don't generate paging alerts for systems you are decommissioning.

Manually revoke keys from puppet.

Instructions on how to do so are on the Puppet service details page.
$ puppet cert clean <fqdn>

Power down system.
Manually revoke keys from salt.

Instructions on how to do so are on the Salt service details page.
$ salt-key -d <fqdn>

End steps

The following can be done one at a time and/or with long breaks in bewtween.

Remove DNS entries for the production network.

Don't remove the mgmt DNS entries at this time!

Reclaims never have mgmt entries removed, and decom servers should keep them until they are wiped and unracked.

Disable the production switch port.

If system is being reclaimed for spare, do not change port label.
If sysetm is being fully decommissioned, blank the port label in switch software.
If system is being reclaimed in Tampa and shipped elsewhere, blank the port label in switch software.

Update associated Phabricator ticket, detailing steps taken and resolution.

If system is decommissioned by on-site tech, they can resolve the ticket.
If system is reclaimed into spares, ticket should be assigned to the HW Allocation Tech so he can update spares lists for allocation.

End steps

Decommission Specific

A Phabricator ticket for the decommission of the system should be placed in the operations project and the appropriate datacenter-specific ops-* project.

All further decommission steps are handled by the on-site technician.

Wipe all disks on system with minimum of 3 passes.

We presently boot off USB version of DBaN.

Reset all system bios, mgmt bios, & raid bios settings to factory defaults.
Unrack system & update Racktables, moving system from rack location to decommissioned rack.

Unless another system will be placed in the space vacated immediately, please remove all power & network cables from rack.

Once server is un-racked, remove its mgmt DNS entries.

Position Assignments

The cycle above references specific position/assignments, without referring to name. To keep the document generic, we'll keep the cycle with positions listed, and just list those folks here.

Buyer / HW Allocation Tech: Rob H (US), Mark B (EU)
On-site Tech EQIAD: Chris J
On-site Tech CODFW: Papaul T
On-site Tech ULSFO: Rob H
Director Technical Operations : Mark B
Operations Technical Review: Mark B, Faidon L