Incident documentation
This page tracks postmortems of past site outages. How to create a new post-mortem:
- Deal with the issue (See also Incident response)
- Write up what happened
- Use the template
- Send to the ops@ list
- Add to wikitech (this wiki)
- Fix the longer term issues
Quarterly(ish) reviews of post-mortems
Incident reports
- 20110524-PlannedMaintenance
- 20110623-Performance
- 20110926-DBandPower
- 20110926-PowerCable
- 20110926-s5switchover
- 20110926-s7switchover
- 20111127-Reddit
- 20120120-Varnish
- 20120206-Squid
- 20120430-MWCoreVersions
- 20120607-LastModifiedExtension
- 20120620-srv268Load
- 20120806-Fibercut
- 20120916-GeoDataTemplate
- 20121211-TranslationNotifications
- 20130315-AFTv5
- 20130418-Jenkins
- 20130419-Parsoid
- 20130508-memcached
- 20130607-CI
- 20130617-AccountCreation
- 20130617-Email
- 20130628-Site
- 20130725-Wikivoyage
- 20131030-Wikidata-ZeroRatedMobileAccess
- 20131104-Parsoid
- 20131118-CirrusSearch
- 20131205-Swift
- 20140113-Poolcounter
- 20140121-BitsApplicationServers
- 20140203-LVS
- 20140206-Math
- 20140211-Parsoid
- 20140228-Cirrus
- 20140313-API-Parsoid
- 20140313-Deploy
- 20140318-EventLogging
- 20140328-DB-Queries
- 20140403-Deploy
- 20140503-Thumbnails
- 20140509-EventLogging
- 20140517-bits
- 20140526-m1
- 20140529-appservers
- 20140607-Elasticsearch
- 20140608-Kafka
- 20140612-Math
- 20140613-Videoscalers
- 20140616-LuceneSearch
- 20140618-Wikitech
- 20140619-parsercache
- 20140619-parsercache/IRC log
- 20140622-es1006
- 20140622-imagescaler
- 20140625-CirrusSearch
- 20140714-Etherpad
- 20140714-Lists
- 20140715 CirrusSearch
- 20140716-DNS
- 20140730-Icinga
- 20140806-Upload
- 20140820-SwiftMemcache
- 20140905-BitsVarnishMobileApp
- 20140910-BetaCluster
- 20140910-swift-syslog
- 20140919-s1
- 20141027-CirrusSearch
- 20141106-Labstore
- 20141113-EventLogging
- 20141114-EventLogging
- 20141118-EventLogging
- 20141118-m2
- 20141122-m2
- 20141125-EventLogging
- 20141126-oauth
- 20141126-ocg
- 20141130-Eqiad-Rack-C4
- 20150103-Parsoid
- 20150106-Zuul
- 20150107-EventLogging
- 20150205-EventLogging
- 20150205-SiteOutage
- 20150206-EventLogging
- 20150217-LabsOutage
- 20150224-LabsOutage
- 20150227-LabsOutage
- 20150320-EventLogging
- 20150324-Eqiad-Rack-C4
- 20150331-LabsNFS-Filesystem-Switch
- 20150331-LabsNFS-Overload
- 20150401-LabsNFS-Overload
- 20150403-API
- 20150406-EventLogging
- 20150406-Flow
- 20150408-s2-SiteStatsInit
- 20150409-EventLogging
- 20150413-LabsNFS
- 20150422-LabsOutage
- 20150423-Commons
- 20150424-Parsoid
- 20150428-EventLogging
- 20150429-Parsoid
- 20150503-JobQueueRedis
- 20150506-EventLogging
- 20150515-LabsOutage
- 20150518-LabsOutage
- 20150519-LabsOutage
- 20150519-Mailman
- 20150519-RESTBase
- 20150527-Cookie
- 20150527-GridEngine
- 20150529-Tagging
- 20150602-gridengine-dns-failure
- 20150605-redis
- 20150607-LabsNFS-Outage
- 20150613-Esams-Text-80
- 20150615-Elasticsearch
- 20150616-Elasticsearch
- 20150617-LabsNFSOutage
- 20150617-LabsNFSOutage/Updates
- 20150617-LabsNFSOutage/Updates/details
- 20150622-EventLogging
- 20150625-LabsNFSOutage
- 20150702-RESTBase
- 20150709-poolcounter
- 20150729-HttpsRedirects
- 20150803-Kafka
- 20150806-poolcounter
- 20150810-Kafka
- 20150812-LabsOutage
- 20150814-MediaWiki
- 20150817-ToolLabs-WebgridOutage
- 20150820-OCSP
- 20150820-PageCategorization (Privacy)
- 20150825-Redis
- 20150830-labs-nfs
- 20150901-Elasticsearch
- 20150908-ores
- 20150914-extension1
- 20150925-LabsOutage
- 20150930-enwiki-readonly
- 20151003-Parsoid
- 20151005-MediaWiki
- 20151014-MediaWiki
- 20151015-EventLogging
- 20151022-EventLogging
- 20151023-graphite-grafana
- 20151024-LabsNFS-Lag
- 20151026-MediaWiki
- 20151105-LabsOutage
- 20151118-LVS-PyBal
- 20151123-LabsOutage
- 20151126-graphite-grafana
- 20151127-EventLogging
- 20151127-PartialNetworkOutage
- 20151203-SiliconOutage
- 20151214-Math-and-Mathoid
- 20151215-swift-syslog
- 20151216-Labs-NFS
- 20151216-ores
- 20151217-cxserver
- 20151218-payments wiki
- 20151229-SiliconOutage
- 20160112-20160111-toollabs-SGE
- Editintro
- Preload
- QR20140319
- QR201407
- QR201407/group1
- QR201407/group1/notes
- QR201407/group2
- QR201407/group2/notes
- Report Template
- Table of outages