Lightning deployments

Rationale

Until the deploy and failure cost is reduced, it doesn't make sense to be able to move away from totally unscheduled windows. However some deployments are now easy enough that they can be completed within a couple minutes, so reservering in advance an exclusive multi-hour deployment window is gratuitous.

The idea is to set aside a block of time every day for quick deploys (ones that do not take scap and are not very likely to be high-impact), so that engineers using the window can be made available to each other to verify releases.

Process

Your deployment must be small (nothing that would require running a full scap) and simple.

Before the Lightning Deploy:

1. Warn Greg (greg-g) of your intention of using the LD as soon as possible; there are a limited number of slots (2, to be specific).
2. Add the planned changes to the Deployments schedule for the date you intend to use it.
   • The order of deploys will be the order they are listed on the Deployments page.

During the Lightning Deploy

1. Be present on #wikimedia-tech ^connect and #wikimedia-operations ^connect
2. Sync up with others. (particularly greg-g)
3. Deploy (in the order that is on the Deployments page)
   • Optional: exclaim LIGHTNING DEPLOY!!!11 to the channel.
4. Don't leave town. Stick around to verify your deployment and to help others.
5. Check relevant logs
   • # of fatals in the last hour (ganglia graph)

Be mindful of the law of programmer hubris, which states that the more straightforward and trivial a change-set appears, the more profound and devilish the breakage it causes. Make sure you're not just being over-eager by taking the time to describe to other participants in a sentence or two the changes you intend to deploy. If other developers express concern, avoid the temptation to reassure them that everything will be fine. Take it as a sign that you may be underestimating the complexity of your deployment in your eagerness.

Karma

• Participating in lightning deploy means making an implicit commitment to help other participants fix things if they break.
• If you use the lightning deploy window, you must be around for its entirety.
• If you cause things to break and require the assistance of others, you'll make up for it by making yourself available to help others during this window even on occasions when you do not have things to deploy.
• Because responsibility for the consequences of lightning deploy is shared by all participants (not just those responsible for syncing the broken code), you should take the time to describe your intentions to the channel. If it makes people nervous, it's a good sign you should defer to a scheduled deployment window.

Humor

As a pre-requisite to participation in lightning deployments, you must become a devotee of a thunder god of your choosing. Prior to syncing any change to the cluster, you will (publicly or privately, at your discretion) supplicate your deity for a smooth passage to production.