Jump to content
Server admin log/Archive 22
July 31
23:40 logmsgbot: rmoen synchronized wmf-config/InitialiseSettings.php 'Remove config variables for VE gender survey'
23:39 logmsgbot: rmoen synchronized wmf-config/CommonSettings.php 'Remove config variables for VE gender survey'
23:32 logmsgbot: rmoen Finished syncing Wikimedia installation... : update visualeditor to master
23:25 logmsgbot: rmoen Started syncing Wikimedia installation... : update visualeditor to master
23:18 logmsgbot: bsitu synchronized php-1.22wmf12/extensions/Echo 'Update Echo to master'
22:14 logmsgbot: csteipp synchronized php-1.22wmf12/extensions/CentralAuth 'eventlogging patch'
18:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:33 cmjohnson1: amaranth currently unreachable according to nosy....going to try rebooting
18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.22wmf12
15:42 logmsgbot: hashar synchronized wmf-config '71774 set some paths to use 'bits' and 'test
02:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 31 02:46:48 UTC 2013
02:31 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Wed Jul 31 02:29:48 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Wed Jul 31 02:15:56 UTC 2013
01:09 springle: restarting db45 slave threads
01:08 springle: restarting db43 slave threads
July 30
23:36 logmsgbot: catrope synchronized php-1.22wmf12/extensions/VisualEditor 'Update VE to master'
23:35 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor 'Update VE to master'
23:15 logmsgbot: bsitu synchronized php-1.22wmf12/extensions/Echo/modules/alert/ext.echo.alert.js 'Refresh js file'
23:14 logmsgbot: bsitu synchronized php-1.22wmf11/extensions/Echo/modules/alert/ext.echo.alert.js 'Refresh js file'
23:11 logmsgbot: bsitu synchronized php-1.22wmf12/extensions/Echo/modules/alert/ext.echo.alert.js 'Refresh js file'
23:10 logmsgbot: bsitu synchronized php-1.22wmf11/extensions/Echo/modules/alert/ext.echo.alert.js 'Refresh js file'
23:07 awight: listeners updates from 246f50c to 243aeba
22:20 awight: payments updated from 74b8b03 to fd276e0
21:35 awight_: payments updated from 855aa0f to 74b8b03
21:29 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:28 awight_: listeners updated from 90ddcc3 to 246f50c
21:18 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
20:35 mutante: gracefull'ing apaches, RT-5539 & bug 51593
20:19 paravoid: switching wikidata back to squid; reports of "bad token" messages
19:24 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo, Thanks, PageTriage to master
19:19 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo, Thanks, PageTriage to master
19:17 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Revert Enable Secure Login everywhere'
19:16 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings-labs.php 'Revert Enable Secure Login everywhere'
18:41 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo, Thanks, PageTriage to master
18:20 mutante: deploying fix for Bug 51593 - Correct the mime type for WOFF to application/font-woff
18:05 awight: fr-civicrm updated from 6e4e641 to 661a1af
18:04 awight: fr-civicrm updated from 820c54a to 6e4e641
17:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Translate on collabwiki'
17:37 Reedy: Created Translate tables on collabwiki
17:15 logmsgbot: reedy synchronized wmf-config/ 'Disable UploadBlacklist'
17:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:09 logmsgbot: reedy synchronized wmf-config/
15:22 Reedy: Purging old SecurePoll private data from all wikis
12:21 mark: Put all non-European wikidata traffic on cp1052 and cp1065
11:21 mark: Pooled cp1052 as text varnish box in PyBal group 'text-varnish' with weight 1, to serve a tiny fraction of wikidata traffic
08:56 Nikerabbit: Translation memory migrated from vanadium to zinc and repopulated
08:12 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'Translation memory to zinc from vanadium'
02:58 sbernardin: racked and plugged back pappas
02:54 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 30 02:52:51 UTC 2013
02:34 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Tue Jul 30 02:33:27 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Tue Jul 30 02:17:54 UTC 2013
July 29
23:23 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php 'Enabling wgNoticeUseLanguageConversion for automatic zh conversion in CentralNotice'
23:21 logmsgbot: mwalker synchronized php-1.22wmf12/extensions/CentralNotice 'Updating CentralNotice to master on wmf12'
23:18 logmsgbot: mwalker synchronized php-1.22wmf11/extensions/CentralNotice 'Updating CentralNotice to master on wmf11'
23:03 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'enable captcha on ptwiki'
22:35 andrewbogott: moved private puppet manifests into modules
22:31 mutante: creating operations/debs/etherpad-lite gerrit project
22:21 logmsgbot: maxsem synchronized php-1.22wmf11/extensions/MobileFrontend/javascripts/modules/editor/EditorApi.js
22:19 logmsgbot: maxsem synchronized php-1.22wmf12/extensions/MobileFrontend/javascripts/modules/editor/EditorApi.js
21:18 mutante: removing unused mysql-server from hooper, the only service uses db9
21:16 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Hack for Cirrus in labs'
21:13 mutante: upgrading packages on hooper (etherpad/racktables)
20:34 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php 'More Cirrus goodies'
20:17 logmsgbot: reedy synchronized php-1.22wmf11/extensions/DataTypes
20:13 logmsgbot: reedy synchronized wmf-config/
20:10 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Syncing Cirrus config'
20:09 logmsgbot: demon synchronized wmf-config/CirrusSearch-labs.php 'Syncing Cirrus config'
20:08 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php 'Syncing Cirrus config'
20:03 logmsgbot: reedy synchronized php-1.22wmf12/extensions/CirrusSearch
19:49 logmsgbot: reedy synchronized php-1.22wmf12/extensions/DataTypes
19:41 springle: delaying slave db45 during OSC bug 49199
19:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikibooks to 1.22wmf12
19:06 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Actually disable VE for anons on dewiki, hopefully'
19:01 Jeff_Green: thulium kernel update, reboot, tweak CPU power settings
19:01 springle: delaying slave db43 during OSC bug 49199
19:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikisource to 1.22wmf12
19:00 andrewbogott: set root password to ! on all labs instances.
18:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews, wikivoyage and wikiversity
18:55 ^demon: restarted gitblit service
18:54 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, wikimedia and special to 1.22wmf12
18:42 sbernardin: server pappas being taken offline to ship to EQIAD
18:41 logmsgbot: reedy synchronized php-1.22wmf12/extensions/Wikibase
18:18 mutante: DNS update - added voyagewiki.org and .com
18:12 springle: restarting db56 slave threads
16:44 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable VE for anons on es/fr/he/it/pl/ru/svwiki; set dewiki back to opt-in mode'
16:35 logmsgbot: catrope synchronized php-1.22wmf12/extensions/VisualEditor 'Update VE to master'
16:34 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor 'Update VE to master'
14:50 logmsgbot: maxsem synchronized wmf-config/squid.php 'Amsterdam varnish IPv6'
14:25 andrewbogott: restarted apache on virt0, fixing (at least temporarily) bug 52217
13:37 Jeff_Green: removed spammy deprecated cron from gadolinium
13:24 ottomata: stopping nginx on maerlant.esams.wikimedia.org (this machine should probably be fully decommissioned)
04:31 springle: delaying slave db56 during OSC bug 49199
02:34 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 29 02:33:00 UTC 2013
02:20 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Mon Jul 29 02:19:08 UTC 2013
02:19 springle: enabling slave db45
02:11 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Mon Jul 29 02:10:33 UTC 2013
01:23 springle: delaying slave db45
July 28
22:37 MaxSem: mw1157 was running out of /tmp space, nuked all *.v older than 1 day. VIPS bug?
02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 28 02:07:37 UTC 2013
02:04 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sun Jul 28 02:02:52 UTC 2013
02:03 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Sun Jul 28 02:02:01 UTC 2013
July 27
23:43 Krinkle: Graceful reload of Zuul (in attempt to purge the ghost entries caused by bug 52161 )
02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 27 02:49:20 UTC 2013
02:32 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sat Jul 27 02:31:17 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Sat Jul 27 02:16:34 UTC 2013
July 26
23:12 mutante: activated softwarewikipedia.com/.org
23:08 mutante: activated http://wikimediacommons.pt/
23:02 mutante: DNS update - removing zones that link to other links. use wikipedia.org not wikipedia.com
20:05 mutante: synced apache, gracefull'ed, activate visualwikipedia.com and .net
18:22 mutante: installing package upgrades on iron
18:15 binasher: db1042 unresponsive on serial console, power cycling
18:13 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1042 from s4'
17:07 mark: Rebooting cp3011
15:34 logmsgbot: reedy synchronized php-1.22wmf12/extensions/Wikibase
15:32 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
02:36 ori-l: ran the git-clone update component of l10nupdate (l10nupdate-1, lines 17-45) as 'l10nupdate' user to try and reproduce "git pull of extensions failed" error, but it completed successfully for me.
02:04 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
01:11 logmsgbot: reedy synchronized php-1.22wmf11/languages/messages/MessagesTl.php
00:48 logmsgbot: bsitu synchronized php-1.22wmf11/extensions/Echo/modules/special/ext.echo.special.js 'update cache for resourceloader'
00:47 logmsgbot: bsitu synchronized php-1.22wmf11/extensions/Echo/modules/overlay/ext.echo.overlay.js 'update cache for resourceloader'
00:17 logmsgbot: maxsem synchronized php-1.22wmf11/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/ 76043 /'
00:16 logmsgbot: maxsem synchronized php-1.22wmf12/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/ 76043 /'
00:11 logmsgbot: bsitu Finished syncing Wikimedia installation... : IE ajax browser cache fix for 1.22wmf11 and scap to update message cache for 1.22wmf12
00:00 logmsgbot: bsitu Started syncing Wikimedia installation... : IE ajax browser cache fix for 1.22wmf11 and scap to update message cache for 1.22wmf12
July 25
20:50 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/GettingStarted 'E3 deployment: bugfixes for GuidedTour and GettingStarted (2/2)'
20:49 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/GuidedTour 'E3 deployment: bugfixes for GuidedTour and GettingStarted (1/2)'
20:47 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/GettingStarted 'E3 deployment: bugfixes for GuidedTour and GettingStarted (2/2)'
20:47 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/GuidedTour 'E3 deployment: bugfixes for GuidedTour and GettingStarted (1/2)'
20:35 logmsgbot: olivneh synchronized php-1.22wmf11/extensions/GettingStarted 'E3 deployment: bugfixes for GuidedTour and GettingStarted (2/2)'
20:35 logmsgbot: olivneh synchronized php-1.22wmf11/extensions/GuidedTour 'E3 deployment: bugfixes for GuidedTour and GettingStarted (1/2)'
20:29 MaxSem: Ran sync-common on mw1041 - looked outta sync
18:51 paravoid: authdns-update: change ns1's IP to new service IP; glues were updated
18:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:40 logmsgbot: reedy synchronized wmf-config/
18:32 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.22wmf11
18:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki, testwikidatawiki, loginwiki and mediawikiwiki to 1.22wmf12
18:24 logmsgbot: reedy Finished syncing Wikimedia installation... :
18:18 Jeff_Green: authdns-update to change IPs for db1008
18:06 logmsgbot: reedy Started syncing Wikimedia installation... :
17:54 logmsgbot: reedy synchronized php-1.22wmf12/extensions/Wikibase
17:44 logmsgbot: reedy synchronized php-1.22wmf11/cache/l10n/
17:30 logmsgbot: reedy Started syncing Wikimedia installation... :
17:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki back to wmf11
17:12 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
17:11 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Thu Jul 25 17:10:36 UTC 2013
16:42 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Thu Jul 25 16:40:47 UTC 2013
16:30 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Thu Jul 25 16:29:28 UTC 2013
16:14 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to php-1.22wmf12 and rebuild l10n cache
15:55 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to php-1.22wmf12 and rebuild l10n cache
15:30 logmsgbot: reedy synchronized docroot and w
15:11 logmsgbot: reedy synchronized php-1.22wmf12
12:49 hashar: gallium cleared out /etc/update-motd.d/05* which adds the roles in motd. They got out of sync. Rerunning puppet
08:25 paravoid: running swift->ceph thumb sync
07:58 paravoid: upgrading ceph to 0.67-rc2
02:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 25 02:47:27 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Thu Jul 25 02:29:13 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Thu Jul 25 02:15:32 UTC 2013
00:15 Ryan_Lane: upgrading wikitech-static to 1.22wmf11
July 24
23:55 logmsgbot: krinkle synchronized php-1.22wmf11/extensions/VisualEditor 'Idc7b094f8eb2788c48'
23:50 logmsgbot: krinkle synchronized php-1.22wmf10/extensions/VisualEditor 'Idc7b094f8eb2788c48'
23:15 Ryan_Lane: upgrading wikitech to 1.22wmf11
23:14 logmsgbot: maxsem synchronized php-1.22wmf11/extensions/MobileFrontend/
23:12 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/
22:46 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/
22:45 logmsgbot: maxsem synchronized php-1.22wmf11/extensions/MobileFrontend/
21:59 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:44 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
21:30 binasher: dropped all databases on s3 that were migrated to s7 [rt5506]
21:29 mwalker: updating fundraising civicrm from 0834a16 to 820c54a
20:21 paravoid: authdns-update: change ns0's IP to new service IP; glues were updated
19:04 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor 'Fix copyright notice bug'
19:03 logmsgbot: catrope synchronized php-1.22wmf10/extensions/VisualEditor 'Fix copyright notice bug'
18:04 logmsgbot: catrope synchronized php-1.22wmf11/resources/mediawiki/mediawiki.notification.js 'Fix mw.notify positioning bug'
18:04 logmsgbot: catrope synchronized php-1.22wmf10/resources/mediawiki/mediawiki.notification.js 'Fix mw.notify positioning bug'
17:42 logmsgbot: catrope synchronized php-1.22wmf11/extensions/FlaggedRevs 'Fix FlaggedRevs fatal'
17:42 logmsgbot: catrope synchronized php-1.22wmf10/extensions/FlaggedRevs 'Fix FlaggedRevs fatal'
17:32 logmsgbot: catrope synchronized php-1.22wmf11/extensions/FlaggedRevs 'Fix excessive FlaggedRevs notices'
17:31 logmsgbot: catrope synchronized php-1.22wmf10/extensions/FlaggedRevs 'Fix excessive FlaggedRevs notices'
17:21 RobH: torrus deadlocked, fixing
17:10 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable VisualEditor on 8 more wikis'
17:09 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Show either visualeditor-enable or visualeditor-betatempdisable, not both'
16:48 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor/modules/ve-mw/dm/annotations/ve.dm.MWNowikiAnnotation.js 'touch'
16:47 logmsgbot: catrope synchronized php-1.22wmf10/extensions/VisualEditor/modules/ve-mw/dm/annotations/ve.dm.MWNowikiAnnotation.js 'touch'
16:47 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor/modules/ve-mw/dm/annotations/ve.dm.MWExternalLinkAnnotation.js 'touch'
16:47 logmsgbot: catrope synchronized php-1.22wmf10/extensions/VisualEditor/modules/ve-mw/dm/annotations/ve.dm.MWExternalLinkAnnotation.js 'touch'
16:42 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor 'Fix nowiki bug'
16:41 logmsgbot: catrope synchronized php-1.22wmf10/extensions/VisualEditor 'Fix nowiki bug'
16:32 RoanKattouw: Fixed /etc/dsh/group/bits ... again :(
16:28 logmsgbot: catrope Finished syncing Wikimedia installation... : Deploy code ahead of VE roll-out to another 8 wikis
16:25 logmsgbot: catrope synchronized php-1.22wmf10/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.ViewPageTarget.js 'touch'
16:22 logmsgbot: catrope synchronized php-1.22wmf10/resources/startup.js 'touch'
16:04 logmsgbot: catrope Started syncing Wikimedia installation... : Deploy code ahead of VE roll-out to another 8 wikis
15:32 logmsgbot: reedy synchronized wmf-config/
12:18 logmsgbot: mark synchronized wmf-config/squid.php
11:00 MaxSem: Aborted TTM migration, will file a bug report
10:25 MaxSem: Attempting to migrate TTM to zinc
02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 24 02:26:44 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Wed Jul 24 02:15:04 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Wed Jul 24 02:08:46 UTC 2013
00:12 ^demon: rebooting formey
00:07 ^demon: dist-upgrading formey lol
July 23
23:41 MaxSem: Rebuilding Solr index to catch up new schema
23:35 andrewbogott: done messing with sockpuppet, stafford, puppet-merge, etc for now.
21:28 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:56 mwalker: updating civicrm on aluminium from e4062d0 to 180ecd2
20:50 mwalker: kludging payments data -- adding 7 hours to contribution timestamps between IDs 4380557 and 4605567
20:36 mwalker: paused all audit and QC jenkins jobs on the payments cluster in preparation for fixing timestamps
19:55 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
19:47 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
19:31 Jeff_Green: dist-upgrade & reboot for payments*
18:40 logmsgbot: reedy synchronized php-1.22wmf10/extensions/Wikibase
18:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:07 logmsgbot: reedy synchronized wmf-config/
18:03 logmsgbot: reedy synchronized database lists files:
18:01 logmsgbot: reedy synchronized wmf-config/
17:59 Reedy: Populated sites table on all wikivoyage wikis
17:41 logmsgbot: reedy synchronized wmf-config/
17:32 logmsgbot: reedy synchronized wmf-config/
17:28 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
17:27 logmsgbot: reedy synchronized php-1.22wmf10/extensions/Wikibase
17:14 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php '26162fb '
16:41 Jeff_Green: authdns-update to propagate fundraisingdb-* cname changes
15:59 Jeff_Green: stop icinga notifications for db1025 mysql
15:58 Jeff_Green: stop icinga notifications for db1008
15:22 Jeff_Green: dist-upgrade &reboot db1025
15:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:14 logmsgbot: reedy synchronized database lists files:
15:14 Jeff_Green: dist-upgrade & reboot payments2-4
14:58 Jeff_Green: dist-upgrade & reboot payments2
14:48 ottomata: stopping puppet on gadolinium, looking into filters causing socat process packet loss
14:23 hashar: Jenkins: lanthanum has been added as a Jenkins slave node though NO job should be running there.e
10:48 logmsgbot: mark synchronized wmf-config/squid.php 'Add ssl1005/ssl1006'
05:45 paravoid: pooling ssl1005/1006 for https/ipv6
05:35 paravoid: authdns-update for cr1-ulsfo <-> cr2-eqiad, dns-auth-lb -> nsN(-new), ssl100x fixed IPv6
02:22 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 23 02:21:20 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Tue Jul 23 02:12:36 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Tue Jul 23 02:08:03 UTC 2013
01:02 Ryan_Lane: enabled hyperthreading on ssl1005
00:40 Ryan_Lane: depooling ssl1005/6 due to ipv6 issues
00:40 Ryan_Lane: pooled new https node ssl1006.wikimedia.org (with hyperthreading enabled)
00:40 Ryan_Lane: pooled new https node ssl1005.wikimedia.org
July 22
22:32 logmsgbot: reedy synchronized php-1.22wmf11/extensions/UploadWizard
22:05 cmjohnson1: restarting ns2
22:01 cmjohnson1: dns update
21:48 LeslieCarr: restarting powerdns on ns1
21:05 logmsgbot: asher synchronized wmf-config/StartProfiler.php 'only randomly profile web reqs in eqiad'
20:40 sbernardin: hot swapping disk 13 on db78 storage array
20:21 paravoid: deploying bmc-config fix across all pmtpa/eqiad mgmt
19:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everyhting else no wikipedia to 1.22wmf11
19:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote, wikinews and wikibooks to 1.22wmf11
19:30 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, wikivoyage and wiktionary to 1.22wmf11
19:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, wikimedia, private, fishbowl and closed to 1.22wmf11
19:26 paravoid: restarting & repooling cp1047/cp1059
19:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Move 3 wikis with UW to 1.22wmf11
16:04 hashar: Jenkins back up
15:53 hashar: jenkins accidentally killed Jenkins :(
15:18 hashar: gallium / jenkins : blacklisting a bunch of user agent 75105
14:42 apergos: (btw docs would be nice, is that really the right way to kick it?)
14:42 apergos: shot and restarted gitblit: on antinomy, cd /var/lib/gitblit, java -jar gitblit.jar & (see bug 51769 )
12:47 paravoid: depooling cp1047/cp1059 for further investigation
12:29 paravoid: restarting cp1046/cp1060's varnish-frontend
11:22 hashar: Jenkins: deleting -merge jobs ([[bugzilla:51395 |bug 51395 ]] sequentially to avoid deadlocking jenkins [[bugzilla:51802 |bug 51802 ]]
11:20 hashar: jenkins restarted
11:02 hashar: restarting Jenkins, some threads are deadlocked ( see [[bugzilla:51802 |bug 51802 ]] )
09:27 paravoid: adding new ns0/ns1 service ip static routes to dobson/linne on cr1-sdtpa/cr2-pmtpa
07:19 paravoid: restarting pybal and manually ipvsadm removing dns_auth services from lvs1/lvs5/lvs1002/lvs1005
02:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 22 02:17:55 UTC 2013
02:11 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Mon Jul 22 02:10:16 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Mon Jul 22 02:06:01 UTC 2013
July 21
20:20 hashar: crashed Jenkins while debugging it :(
19:29 hashar: Jenkins deleting remaining -merge jobs (not triggered anymore since that is handled by Zuul now bug 51395 )
19:28 hashar: logging bot is dead, no idea for how long though.
July 20
14:05 Nemo_bis: antimony was overloaded (100 % CPU, around 15 load) again, still serving internal errors
08:46 netsplit kills morebots
07:01 paravoid: apt: including dclass 2.2.2-1
02:31 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 20 02:30:42 UTC 2013
02:20 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Sat Jul 20 02:18:54 UTC 2013
02:11 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Sat Jul 20 02:10:26 UTC 2013
July 19
20:32 logmsgbot: hashar synchronized wikiversions-labs.dat
20:31 logmsgbot: hashar synchronized all-labs.dblist
20:21 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php 'Slightly increase Parsoid dequeue rate'
19:44 sbernardin: shutting down spence for decommissioning
19:38 logmsgbot: gwicke synchronized php-1.22wmf11/extensions/Parsoid/php/ParsoidCacheUpdateJob.php 'restore Parsoid to previously deployed newer code'
19:38 logmsgbot: gwicke synchronized php-1.22wmf11/extensions/Parsoid/php/Parsoid.php 'restore Parsoid to previously deployed newer code'
19:37 logmsgbot: gwicke synchronized php-1.22wmf10/extensions/Parsoid/php/ParsoidCacheUpdateJob.php 'restore Parsoid to previously deployed newer code'
19:36 logmsgbot: gwicke synchronized php-1.22wmf10/extensions/Parsoid/php/Parsoid.php 'restore Parsoid to previously deployed newer code'
18:56 gwicke: updated Parsoid to d5fe6c9
14:51 mark: Upgraded mobile caches to varnish 3.0.3~rc1-wm14
14:40 hashar: continuous integration : slightly adjusted main page https://integration.wikimedia.org/ (see integration/docroot.git )
10:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable ShortUrl on testwiki'
10:12 logmsgbot: reedy synchronized php-1.22wmf11/includes/Setup.php
09:47 logmsgbot: reedy synchronized wmf-config/
07:56 paravoid: stopping radosgw across the ceph cluster, post-upgrade issues
02:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 19 02:49:50 UTC 2013
02:35 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Fri Jul 19 02:33:48 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Fri Jul 19 02:15:29 UTC 2013
July 18
23:55 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor and CentralAuth to master
23:53 Tim: deploying new php5-fss, to fix segfault bug
23:41 springle: restarting db45 slave threads
23:39 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor and CentralAuth to master
23:18 springle: completed wikidatawiki wb_terms.term_weight OSC
22:48 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Second scap attempt for E3 deployment of GettingStarted and GuidedTour
22:31 logmsgbot: mflaschen Started syncing Wikimedia installation... : Second scap attempt for E3 deployment of GettingStarted and GuidedTour
22:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove bigdelete from sysyop'
22:30 Ryan_Lane: dist-upgrading all virt nodes
21:57 superm401: Scap failed due to Wikibase issue
21:54 Aaron|home: Finally running "mwscriptwikiset maintenance/purgeDeletedFiles.php all.dblist --starttime 20130529000000 "
20:27 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
19:59 springle: starting wikidatawiki wb_terms.term_weight OSC
19:53 springle: delaying slave db45 for wikidatawiki wb_terms OSC duration
19:15 notpeter: adding elasticsearch v. 0.90.2 package to brewster
18:21 LeslieCarr: adding new links into cr1/cr2-eqiad asw-c-eqiad bundle
18:14 Reedy: Added term weight to wb_terms on testwikidatawiki
18:13 Reedy: Created wb_property_info on testwikidatawiki
18:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwikidatawiki to 1.22wmf11
18:07 Reedy: Ran extensions/UploadWizard/maintenance/migrateCampaigns.php against testwiki
18:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, wikidatawiki and loginwiki to 1.22wmf11
18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikipedias to 1.22wmf11
17:58 cmjohnson1: authdns update
17:58 logmsgbot: reedy synchronized wmf-config/
16:49 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf11 and build l10n cache
16:35 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf11 and build l10n cache
16:28 logmsgbot: reedy synchronized docroot and w
16:23 logmsgbot: reedy synchronized php-1.22wmf11/ 'Initial code sync'
15:55 logmsgbot: csteipp synchronized php-1.22wmf9/extensions/CentralAuth 'Fix SUL issue - wmf9'
15:53 logmsgbot: csteipp synchronized php-1.22wmf10/extensions/CentralAuth 'Fix SUL issue'
14:33 ^demon: zuul back up too
14:32 ^demon: gerrit service back online
14:04 cmjohnson1: shutting down manganese
14:04 ^demon: stopped gerrit & zuul services on manganese and gallium respectively
14:03 cmjohnson1: removing/swapping sdb2 on manganese
12:28 mark: Inserted varnish 3.0.3plus~rc1-wm14 packages into the precise-wikimedia APT repository
02:44 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 18 02:43:17 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Thu Jul 18 02:28:28 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Thu Jul 18 02:15:04 UTC 2013
July 17
22:17 awight: chongo
22:02 mwalker: updating payments-listeners from b2ac16f to 90ddcc3
20:43 logmsgbot: csteipp synchronized wmf-config/InitialiseSettings.php
20:26 logmsgbot: csteipp synchronized php-1.22wmf9/extensions/CentralAuth 'hooks for SUL2 - wmf9'
20:23 logmsgbot: csteipp synchronized php-1.22wmf10/extensions/CentralAuth 'hooks for SUL2'
20:19 Ryan_Lane: deploying OpenStackManager commit 46f3be1 to virt0
19:54 Ryan_Lane: clarification (those two hashes for OpenStackManager)
19:54 Ryan_Lane: deploying 5e1289b and 40ca623 to virt0
18:54 ^d: killed *another* pywikibot/compat clone. Please stop cloning this right now, you're gerrit cry.
18:43 awight: payments cluster updated from from 2a9169765b94e0 to 855aa0f8c0bbc5
18:03 ^d: killed a clone of pywikibot/compat. Until I figure out why it's soooo big, please don't kill gerrit trying to clone it
17:55 ^d: force running jgit gc on pywikibot/compat
17:41 ^d: restarting gerrit
16:58 logmsgbot: reedy synchronized wmf-config/
16:52 logmsgbot: reedy synchronized wmf-config/
16:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Revert live hack of setting testwiki to zh'
16:24 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'debugging over'
16:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'testwiki wgLanguageCode to zh'
16:10 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'debug'
15:52 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'debug'
14:32 akosiaris: adding bigtop-tomcat package to apt for analytics
14:13 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'debug'
11:32 paravoid: updating gdnsd in apt to 1.9.0-1~precise1
08:41 akosiaris: added oozie and hue packages to apt.wikimedia.org as per analytics team request
02:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 17 02:47:27 UTC 2013
02:32 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Wed Jul 17 02:31:39 UTC 2013
02:18 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Wed Jul 17 02:17:00 UTC 2013
00:09 logmsgbot: krinkle synchronized php-1.22wmf9/extensions/VisualEditor 'I12f52719ecafd7488bb00419'
00:07 logmsgbot: krinkle synchronized php-1.22wmf10/extensions/VisualEditor 'I12f52719ecafd7488bb00419'
July 16
23:38 logmsgbot: krinkle Finished syncing Wikimedia installation... : 'Deploying Ifeea7885d67af6e53bf5c19ccaa231f86c7ead90'
23:31 logmsgbot: krinkle Started syncing Wikimedia installation... : 'Deploying Ifeea7885d67af6e53bf5c19ccaa231f86c7ead90'
23:15 logmsgbot: bsitu synchronized php-1.22wmf9/extensions/Echo 'Update Echo to master'
23:14 logmsgbot: bsitu synchronized php-1.22wmf10/extensions/Echo 'Update Echo to master'
21:00 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
20:58 springle: finished rechecking tag_summary enwiki re bug 51254 comment 11
20:49 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
18:04 springle: rechecking tag_summary enwiki re bug 51254 comment 11
17:54 Ryan_Lane: deployed LdapAuthentication fdb01877 to virt0 and purged memcache to remove bad memcache entries for users
17:10 ^demon: gitblit restarted to grab a bugfix
16:59 mwalker: updated payments-listeners from 8a38a49 to b2ac16f
16:33 cmjohnson1: that's sdb2 on manganese
16:31 cmjohnson1: swapping sdb1 on manganese (software raid1) rt5442
15:34 logmsgbot: reedy synchronized php-1.22wmf10/extensions/Wikibase
14:54 hashar: zuul / jenkins -merge jobs are no more triggered by Zuul since it detects whether a patch apply cleanly on the tip of the branch. That saves up a bunch of overhead [[bugzilla:51395 |bug 51395 ]] and 73947
12:58 mark: Sending all European mobile traffic to esams
12:21 mark: Sending dutch mobile traffic to esams
12:06 mark: Created georecord m.wikimedia.org
10:30 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php ' now uses ['bits'] 71777 '
10:29 logmsgbot: hashar synchronized wmf-config/CommonSettings.php ' now uses ['bits'] 71777 '
10:22 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '$wgCopyrightIcon now vary using $wmfHostnames[bits]'
10:21 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '$wgCopyrightIcon now vary using $wmfHostnames[bits]'
09:55 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS: Bug 51019 '
09:53 logmsgbot: nikerabbit synchronized php-1.22wmf9/extensions/UniversalLanguageSelector/ 'ULS to master'
09:49 logmsgbot: nikerabbit synchronized php-1.22wmf10/extensions/UniversalLanguageSelector/ 'ULS to master'
09:42 ori-l: Resumed reporting of MediaWiki fatals & exceptions to Ganglia
09:28 hashar: upgrading packages on gallium
09:13 apergos: restarting gerrit
02:45 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 16 02:43:55 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Tue Jul 16 02:28:30 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Tue Jul 16 02:15:23 UTC 2013
01:01 logmsgbot: catrope Finished syncing Wikimedia installation... : Update VisualEditor to master again
00:53 logmsgbot: catrope Started syncing Wikimedia installation... : Update VisualEditor to master again
July 15
23:40 cmjohnson1: reinstalling carbon
23:06 logmsgbot: csteipp synchronized php-1.22wmf10/extensions/CentralAuth 'Fix sul2 regression'
23:03 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php 'Slightly increase Parsoid dequeue rate'
22:31 logmsgbot: kaldari synchronized php-1.22wmf10/extensions/WikiLove/WikiLove.hooks.php 'Fixing WikiLove regression on wmf10'
21:51 ori-l: All EventLogging services start/running; data looks good.
20:47 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable VisualEditor for all users (anons too) on enwiki'
20:45 Krinkle: Graceful reload of Zuul to fast-forward deployment to Ia53d412b029205
20:41 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master
20:30 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master
20:26 ori-l: Disabled MediaWiki errors Ganglia module on vanadium
20:03 ori-l: Rebooting vanadium to complete kernel upgrade to 3.2.0-49.
19:49 ori-l: Shutting down EventLogging services on vanadium ahead of Iba8cc5d7b deployment.
18:59 hashar: labstore3 system CPU rocketed from ~10% to ~60%
18:43 cmjohnson1: powercycling mw1163
18:42 cmjohnson1: depooling mw1163 to troubleshoot DIMM error
18:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: everything non wikipedia to 1.22wmf10
18:23 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wiktionary to 1.22wmf10
18:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks, wikivoyage, wikiversity to 1.22wmf10
18:17 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia, closed, special to 1.22wmf10
17:55 cmjohnson1: powering down mw1173
17:36 cmjohnson1: depooling mw1173 to troubleshoot DIMM Failure
17:10 RobH: Fixing blog setup of themes, theme may reset to defaults for next few minutes as i tinker with it.
15:28 MaxSem: Recreating Solr index
12:41 apergos: rebooting kaulen to pick up some upgrades, per rt 5460
02:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 15 02:42:20 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Mon Jul 15 02:27:57 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Mon Jul 15 02:14:48 UTC 2013
July 14
18:44 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php 'Throttle titles per template update job further from 10 to 6'
17:50 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php 'Re-enable Parsoid updates after throttling template update rate'
16:55 logmsgbot: reedy synchronized php-1.22wmf10/extensions/Gadgets/
16:54 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Gadgets/
14:05 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Resync for boxes that came back up'
14:02 mark: Repooled stolen apaches back in the appserver cluster
14:00 logmsgbot: catrope synchronized php-1.22wmf9/resources/startup.js 'touch'
13:51 mark: Depooled stolen apaches from the api cluster
13:21 mark: Upping PyBal weight to 15 for stolen appservers
13:20 RoanKattouw: Hand-syncing CommonSettings.php with dsh to set $wgParsoidSkipRatio = 0
13:18 mark: Stealing 10 app servers, for the API pool
13:06 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Set $wgParsoidSkipRatio to 1 to let the API cluster breathe'
12:55 RoanKattouw: Restarted Apache on mw1118, had to stop by hand using killall -9 (/etc/init.d/apache stop didn't work)
12:39 RoanKattouw: Restarted pybal on lvs1003
12:38 RoanKattouw: Restarted pybal on lvs1006 (set api depool threshold to .4 from .6)
06:39 apergos: powercycled mc1005, unreachable via console, from ganglia guess it was in swap death
02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 14 02:07:10 UTC 2013
02:03 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Sun Jul 14 02:02:31 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Sun Jul 14 02:01:43 UTC 2013
July 13
02:30 mutante: kernel upgrade, rebooting zirconium
02:30 mutante: copied etherpad-lite from lucid to precise
02:24 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 13 02:22:51 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Sat Jul 13 02:13:55 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Sat Jul 13 02:07:36 UTC 2013
00:17 mutante: swift install on iron (swift cleaner & utils only) had a depends issue python-swift (= 1.5.0-3) but 1.7.4-0ubuntu2+wmf1 is installed, upgraded anyway with -f
00:07 mutante: installing package upgrades on iron
July 12
23:40 gwicke: updated Parsoid to f6d3742
23:38 springle: restarted db32 slave
23:32 springle: finished tag_summary rebuild
22:40 logmsgbot: aaron cleared profiling data
22:34 mutante: completely stopped and then started squid front and back on cp1001 (had socket already in use errors)
22:26 springle: rebuilding tag_summary on enwiki
22:25 mutante: restarting squid on cp1001
22:07 mutante: restarting job runners
21:07 springle: stopped db32 mysql slave threads
19:58 logmsgbot: gwicke synchronized php-1.22wmf9/extensions/Parsoid/php/ParsoidCacheUpdateJob.php 'updating Parsoid'
19:57 logmsgbot: gwicke synchronized php-1.22wmf9/extensions/Parsoid/php/Parsoid.php 'updating Parsoid'
18:45 RobH: authdns-update all ulsfo now 4XXX
18:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove wgOldChangeTagsIndex on all remaining wikis'
18:00 logmsgbot: gwicke synchronized php-1.22wmf9/extensions/Parsoid/php/ParsoidCacheUpdateJob.php 'updating Parsoid master'
17:59 logmsgbot: gwicke synchronized php-1.22wmf9/extensions/Parsoid/php/Parsoid.php 'updating Parsoid master'
16:27 Krinkle: Graceful reload of Zuul to forward deployment to I4de1670e11d58e
16:10 logmsgbot: reedy synchronized docroot and w
15:51 logmsgbot: reedy synchronized docroot and w
15:35 logmsgbot: reedy synchronized docroot and w
15:33 logmsgbot: reedy synchronized docroot and w
15:31 logmsgbot: reedy synchronized docroot and w
15:26 logmsgbot: reedy synchronized docroot and w
15:20 Krinkle: Resetting commons:User:CommonsDelinker account e-mailaddress (bug 51016 )
10:43 mark: Sending European wikidata and wikivoyage traffic to esams
05:13 mutante: restarting ircecho on neon
03:03 mwalker: and again... payments from 1c360e2 to 2a91697
02:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 12 02:47:23 UTC 2013
02:41 mwalker: attempting again on the payments cluster! from 1c360e2 to 7a5a3c3
02:35 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Fri Jul 12 02:34:20 UTC 2013
02:21 mwalker: aaand nope; didn't work, reverting payments back to 1c360e2
02:18 mwalker: trying again -- updating payments from 1c360e2 to 8a801e8
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Fri Jul 12 02:16:00 UTC 2013
01:40 mwalker: aaaahhhhh reverting payments to 1c360e2 because banners are sending the wrong gateway (and I was ignoring it before!)
01:33 mwalker: updating payments (including the formchooser) from 1c360e2 to 1a2130b
00:49 springle: paused updating change_tags indexes on remaining wikis in bug 40867 comment 6 (adds done, drops todo)
00:11 logmsgbot: maxsem synchronized php-1.22wmf9/extensions/GeoData/solrupdate.php
00:09 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/GeoData/solrupdate.php
July 11
23:46 logmsgbot: catrope Finished syncing Wikimedia installation... : Update VE to master
23:38 logmsgbot: catrope Started syncing Wikimedia installation... : Update VE to master
22:53 springle: updating change_tags indexes on remaining wikis in bug 40867 comment 6
22:42 logmsgbot: tstarling synchronized wmf-config
22:41 logmsgbot: tstarling restarted twemproxy on all servers
22:40 logmsgbot: tstarling synchronized wmf-config/mc.php
22:37 logmsgbot: tstarling synchronized wmf-config/twemproxy.yaml
22:37 logmsgbot: tstarling synchronized wmf-config/twemproxy-pmtpa.yaml
22:33 logmsgbot: springle synchronized wmf-config/InitialiseSettings.php 'simplewiki change_tags indexes updated, bug 40867 '
21:49 logmsgbot: spage synchronized wmf-config/InitialiseSettings.php 'Config changes for VE survey'
21:38 RobH: move internal ip, not wiki. bleh
21:38 RobH: authdns update for zinc to move internal wiki
21:38 Ryan_Lane: deployed squid change to switch test to mw1017
21:30 Ryan_Lane: depooling mw1017 to make it the new test.wm.o
21:23 logmsgbot: spage Finished syncing Wikimedia installation... : E3 deploy latest GettingStarted and VE gender survey config
21:22 RobH: authdns-update for tmc1/2 and mgmt
21:12 logmsgbot: spage Started syncing Wikimedia installation... : E3 deploy latest GettingStarted and VE gender survey config
20:21 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Wikibase
20:21 logmsgbot: reedy synchronized php-1.22wmf10/extensions/Wikibase
19:11 LeslieCarr: clearing out stuck icinga processes , except higher than normal icinga spam and do not worry
18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: loginwiki testwiki testwikidatawiki mediawikiwki to 1.22wmf10
18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: all 'pedias to 1.22wmf9
17:22 cmjohnson1: commenting out mc1-16 in node groups
17:17 sbernardin: bringing down ms-be5 for hardware replacement
16:56 gwicke: updated Parsoid to 6d17a80
15:13 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf10 and rebuild l10n cache
15:00 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf10 and rebuild l10n cache
14:49 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.22wmf10 and rebuild l10n cache
14:46 logmsgbot: reedy synchronized docroot and w
14:45 logmsgbot: reedy synchronized php-1.22wmf10 'initial sync of 1.22wmf10'
10:21 Krinkle: Graceful reload of Zuul (forward to I3fc6baba8c6d65)
07:39 hashar: gallium : fix permissions for android-common nightly build (need to be jenkins owned, not jenkins-slave) [[bugzilla:51137 |bug 51137 ]]
02:52 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 11 02:51:16 UTC 2013
02:38 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Thu Jul 11 02:37:40 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Thu Jul 11 02:17:47 UTC 2013
00:26 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor 'Update VE to pick up cssText fix'
00:26 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Update VE to pick up cssText fix'
July 10
23:39 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor 'Update VE to master'
23:38 RoanKattouw: [00:23] gwicke !updated Parsoid to f571475
23:38 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Update VE to master'
23:30 logmsgbot: csteipp synchronized php-1.22wmf9/extensions/CentralAuth 'Updating CentralAuth on wmf9 for SUL'
22:27 mutante: reprepro copying etherpad from wikimedia-lucid to wikimedia-precise
21:36 mutante: removing old racktables Apache site and other remnants from hooper
21:03 mutante: package upgrades on hooper (etherpad)
18:29 logmsgbot: nikerabbit synchronized php-1.22wmf8/extensions/UniversalLanguageSelector/ 'ULS perf fix'
18:25 logmsgbot: nikerabbit synchronized php-1.22wmf9/extensions/UniversalLanguageSelector/ 'ULS perf fix'
18:10 gwicke: updated Parsoid to 8ea8ab0
17:58 mutante: reverting removing spence, NTP server issue
17:47 mutante: DNS update - killing spence
17:21 Jeff_Green: dist-upgrade and reboot aluminium, db78
17:21 Jeff_Green: grosley switched to frack puppetmaster
17:21 Jeff_Green: db78 switched to frack puppetmaster
16:35 mutante: upgrading ruby and other packages on sockpuppet and stafford
14:40 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/ 72957 '
14:36 apergos: restarted gerrit, its queue was backed up
14:24 hashar: gerrit being super slow, it has 32 items in its queue ( lot of git-upload-pack p/mediawiki/core.git ). Ganglia for manganese does not show anything obvious.
14:22 logmsgbot: maxsem synchronized php-1.22wmf8/extensions/GeoData/solrupdate.php
14:20 logmsgbot: maxsem synchronized php-1.22wmf9/extensions/GeoData/solrupdate.php
11:34 mark: Upgraded and restarted eqiad mobile caches (front/back)
09:48 mark: Upgrading Varnish on bits servers
09:17 mark: Inserted varnish 3.0.3plus-rc1-wm12 packages into the precise-wikimedia APT repository
03:03 logmsgbot: tstarling Finished syncing Wikimedia installation... : ProofreadPage update for bug 51085
02:59 logmsgbot: tstarling Started syncing Wikimedia installation... : ProofreadPage update for bug 51085
02:53 logmsgbot: tstarling synchronized php-1.22wmf9/extensions/ProofreadPage/ProofreadPage.body.php
02:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 10 02:42:32 UTC 2013
02:39 logmsgbot: tstarling synchronized php-1.22wmf8/maintenance/showJobs.php
02:38 logmsgbot: tstarling synchronized php-1.22wmf8/includes/job/JobQueueRedis.php
02:37 logmsgbot: tstarling synchronized php-1.22wmf8/includes/job/JobQueue.php
02:37 logmsgbot: tstarling synchronized php-1.22wmf8/includes/MappedIterator.php
02:30 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Wed Jul 10 02:29:28 UTC 2013
02:22 Tim: on tin: attempted to do a git pull in wmf8 but it updated a whole lot of extensions for some reason. Checking out the old versions manually.
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Wed Jul 10 02:15:40 UTC 2013
01:16 gwicke: finished Parsoid config update with latest dependencies
01:14 gwicke: starting Parsoid config update with latest dependencies
July 9
23:17 logmsgbot: mwalker synchronized php-1.22wmf9/extensions/ContributionReporting 'Updating ContributionReporting for 2013 fundraiser'
22:16 mwalker: updating payments.wm.o from eee8689 to 1c360e2
22:04 logmsgbot: maxsem synchronized php-1.22wmf9/extensions/MobileFrontend
22:03 logmsgbot: maxsem synchronized php-1.22wmf8/extensions/MobileFrontend
21:42 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:27 Jeff_Green: db78 switched to mariadb
21:21 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
21:13 Robh: authdns-update for bastion5001.mgmt
20:13 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'activating Disambiguator extension on all WMF wikis'
18:26 LeslieCarr: authdns-update on ns0
18:21 logmsgbot: mlitn synchronized wmf-config/ 'Enable ArticleFeedbackv5 auto-archive'
18:02 logmsgbot: mlitn synchronized wmf-config/ 'Update ArticleFeedbackv5 & Echo config'
17:58 logmsgbot: mlitn synchronized php-1.22wmf9/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
17:57 logmsgbot: mlitn synchronized php-1.22wmf8/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
16:50 logmsgbot: anomie synchronized php-1.22wmf9/extensions/AbuseFilter 'Update AbuseFilter to master for global filter bug fixes'
16:39 cmjohnson1: going to tinkering with dysprosium...will see some icinga alerts
15:10 ottomata: re-starting puppet on analytics1006,1008 ,1009
15:01 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
14:40 ottomata: stopping puppet on analytics1006, 1008 and 1009 to manually apply udp2log puppetization
13:20 paravoid: upgraded ceph to 0.66
13:04 logmsgbot: hashar synchronized php-1.22wmf9/extensions/Wikibase/lib/resources/wikibase.ui.SiteLinksEditTool.js
13:04 hashar: touched and synced extensions/Wikibase/lib/resources/wikibase.ui.SiteLinksEditTool.js on php-1.22wmf8 and php-1.22wmf9
13:04 logmsgbot: hashar synchronized php-1.22wmf8/extensions/Wikibase/lib/resources/wikibase.ui.SiteLinksEditTool.js
11:48 paravoid: apt: updating to ceph 0.66
09:34 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 51019 '
09:33 logmsgbot: nikerabbit synchronized php-1.22wmf9/extensions/UniversalLanguageSelector/ 'ULS to master'
09:30 logmsgbot: nikerabbit synchronized php-1.22wmf8/extensions/UniversalLanguageSelector/ 'ULS to master'
08:56 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS phase 5'
08:53 logmsgbot: nikerabbit synchronized php-1.22wmf8/extensions/UniversalLanguageSelector/ 'ULS to master'
08:44 logmsgbot: nikerabbit synchronized php-1.22wmf9/extensions/UniversalLanguageSelector/ 'ULS to master'
04:17 logmsgbot: tstarling synchronized php-1.22wmf8/includes/api/ApiPurge.php
04:16 logmsgbot: tstarling synchronized php-1.22wmf8/includes/WikiPage.php
02:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 9 02:38:31 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Tue Jul 9 02:26:54 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Tue Jul 9 02:14:12 UTC 2013
01:57 logmsgbot: tstarling synchronized php-1.22wmf9/includes/api/ApiPurge.php
01:57 logmsgbot: tstarling synchronized php-1.22wmf9/includes/WikiPage.php
00:46 gwicke: updated Parsoid to d0e3603
July 8
23:56 mutante: graceful'ing Apaches
23:53 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VE and AbuseFilter
23:43 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VE and AbuseFilter
22:48 mutante: restarting pdns on ns0
22:47 mutante: DNS update - point download.mw away from empty dir on kaulen
21:36 Robh: updating blog
19:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
19:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non 'pedia to 1.22wmf9
19:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wiktionary to 1.22wmf9
18:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikiquote to 1.22wmf9
18:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks, wikimania and wikivoyage to 1.22wmf9
18:36 logmsgbot: reedy synchronized php-1.22wmf8/extensions/TimedMediaHandler/
18:35 logmsgbot: reedy synchronized php-1.22wmf9/extensions/TimedMediaHandler/
17:31 mark: Gave row A and row C upload caches equal share of traffic again
17:16 logmsgbot: csteipp synchronized php-1.22wmf9/includes/SkinTemplate.php 'Deploy missing function for SUL2'
16:44 mutante: installing package upgrades on kaulen (bugzilla)
15:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
14:38 paravoid: apt: installing gdnsd 1.8.3-1~precise1
12:44 hashar: jenkins: reduced number of executors on master from 2 to 0.
10:28 paravoid: (re)installing colby, rubidium, eeden
09:09 hashar: srv281 and srv1173 have ssh timeout when synchronizing configuration.
09:09 logmsgbot: hashar synchronized images/wikimedia-button.png 'added to git as of 71775 '
08:36 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'touch config for $wgLoadScript beta/prod variance 70322 '
08:35 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '$wgLoadScript beta/prod variance 70322 '
08:33 hashar: Testing out $wgLoadScript beta/prod variance 70322 , pulled change and running sync-common on srv193
04:44 Tim: on enwiki: discarded refreshLinks2 entries for templates Yesno, If_pagename, Category_handler/numbered, Category_handler, by setting their root job timestamp to a fake new timestamp. See bug 50785
02:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 8 02:15:21 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Mon Jul 8 02:09:13 UTC 2013
July 7
23:29 Tim: on fenari: removed /a/common directory so that mwscript will work
23:28 Tim: restarted morebots
02:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 7 02:14:42 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Sun Jul 7 02:09:01 UTC 2013
02:06 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Sun Jul 7 02:05:09 UTC 2013
July 6
02:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 6 02:38:04 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Sat Jul 6 02:27:28 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Sat Jul 6 02:14:24 UTC 2013
July 5
21:55 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor/ 'Update VE to master in wmf8 (did wmf9 earlier)'
21:25 mutante: graceful Apaches, activate "ontology" alias for wikidata
21:10 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor/ 'Update VE to master in wmf9'
20:56 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/ 72144 '
18:34 sbernardin: bringing down colby to remove extra drives
13:54 hashar: [[bugzilla:50228 |bug 50228 ]]
13:48 hashar: Jenkins: reduced # of executors on [master] from 4 to 2. Will bring it to 0 next monday (see [[bugzilla:50228 |bug 50228 ]] )
05:13 logmsgbot: tstarling Finished syncing Wikimedia installation... :
05:09 logmsgbot: tstarling Started syncing Wikimedia installation... :
03:01 logmsgbot: tstarling Finished syncing Wikimedia installation... :
02:58 logmsgbot: tstarling Started syncing Wikimedia installation... :
02:52 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 5 02:51:29 UTC 2013
02:45 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Fri Jul 5 02:44:41 UTC 2013
02:40 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Fri Jul 5 02:38:54 UTC 2013
02:34 Tim: testing l10nupdate
02:04 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 5 02:03:10 UTC 2013
02:04 logmsgbot: LocalisationUpdate failed (1.22wmf9) at Fri Jul 5 02:03:09 UTC 2013
02:04 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Fri Jul 5 02:03:09 UTC 2013
July 4
22:10 LeslieCarr: icinga restarted, was down due to bad check
16:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
13:58 pp-pdf1: restarted all services
13:57 pp-pdf1: - update mwlib to 0.15.10
12:01 akosiaris: uploaded buck version 0+git20130612-0+wmf1 at apt.wikimedia.org
09:37 logmsgbot: tstarling Finished syncing Wikimedia installation... :
09:34 logmsgbot: tstarling Started syncing Wikimedia installation... :
07:07 logmsgbot: tstarling synchronized README
02:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 4 02:34:13 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Thu Jul 4 02:27:40 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Thu Jul 4 02:14:32 UTC 2013
01:47 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor 'Fix <a href bug'
01:47 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Fix <a href bug'
01:15 logmsgbot: catrope synchronized php-1.22wmf9/skins/vector/collapsibleTabs.js 'touch'
01:15 logmsgbot: catrope synchronized php-1.22wmf9/skins/vector/vector.js 'touch'
01:15 logmsgbot: catrope synchronized php-1.22wmf9/resources/startup.js 'touch'
01:14 logmsgbot: catrope synchronized php-1.22wmf9/resources/Resources.php 'collapsibleTabs fix'
01:14 logmsgbot: catrope synchronized php-1.22wmf8/skins/vector/collapsibleTabs.js 'touch'
01:14 logmsgbot: catrope synchronized php-1.22wmf8/skins/vector/vector.js 'touch'
01:13 logmsgbot: catrope synchronized php-1.22wmf8/resources/startup.js 'touch'
01:13 logmsgbot: catrope synchronized php-1.22wmf8/resources/Resources.php 'collapsibleTabs fix'
00:52 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor 'Latest fixes'
00:51 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Latest fixes'
00:46 RoanKattouw: Add cp1056, cp1057, cp1069 and cp1070 to /etc/dsh/group/bits , they were missing and so the purge-varnish script didn't actually work
July 3
23:30 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor 'Update VE again'
23:30 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Update VE again'
23:25 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master
23:18 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master
22:12 ^d: restarting gerrit for firefox workaround
20:42 mark: Trippled PyBal weights of row A upload caches to reduce traffic on row C
20:38 ottomata: restarting gmetad on nickel to reload an rrd file
20:37 hashar: reloading Zuul to lower gerrit debug messages 71916
19:47 logmsgbot: mwalker synchronized php-1.22wmf9/extensions/CentralNotice 'Updating CentralNotice to master on wmf9 (mobile stability and emergency messaging)'
19:46 logmsgbot: mwalker synchronized php-1.22wmf8/extensions/CentralNotice 'Updating CentralNotice to master on wmf8 (mobile stability and emergency messaging)'
19:45 binasher: migrated ishmael to neon (eqiad)
19:45 binasher: pdns hung on ns2, restarted
19:41 gwicke: updated Parsoid to a53e198
18:07 logmsgbot: yurik synchronized php-1.22wmf9/extensions/ZeroRatedMobileAccess/
17:59 logmsgbot: yurik synchronized php-1.22wmf8/extensions/ZeroRatedMobileAccess/
17:32 gwicke: updated Parsoid to 732c3e6
17:28 logmsgbot: yurik synchronized php-1.22wmf8/extensions/ZeroRatedMobileAccess/
17:25 logmsgbot: yurik synchronized php-1.22wmf9/extensions/ZeroRatedMobileAccess/
17:18 Ryan_Lane: deploying change 60783 to virt0
17:10 RobH: authdns-update rt 4547
16:56 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Wikibase
15:39 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/ 71375 /'
15:30 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/ 70597 /'
15:26 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/ 71813 /'
14:13 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Wikibase changes'
14:12 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Wikibase changes'
14:06 logmsgbot: demon synchronized php-1.22wmf9/extensions/Wikibase/ 'Updating wikibase to 5a65117 '
13:40 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Wikibase: I2e9d03b2'
13:37 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/ 70557 /'
02:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 3 02:38:05 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Wed Jul 3 02:28:35 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Wed Jul 3 02:15:00 UTC 2013
00:02 ^d: manganese back up
00:00 ^d: rebooting manganese (gerrit)
July 2
22:47 logmsgbot: bsitu synchronized php-1.22wmf8/extensions/Echo 'update Echo to master'
22:46 logmsgbot: bsitu synchronized php-1.22wmf9/extensions/Echo 'update Echo to master'
20:57 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
20:46 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
19:41 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo, GettingStarted, PageTriage, Thanks to master
19:32 mwalker: updated production civi from 6e78674 to 5bcbab5
19:32 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo, GettingStarted, PageTriage, Thanks to master
18:08 bblack: upgraded vhtcpd to 0.0.8 on brewster
17:48 sbernardin: replacing disks slot5 & slot6 on db45
17:38 sbernardin: replacing disk on db44
17:29 logmsgbot: csteipp rebuilt wikiversions.cdb and synchronized wikiversions files: bumping loginwiki to wmf9
16:30 cmjohnson1: disabling puppet freshness check
16:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
16:16 sbernardin: rebooting mw80 for bios update
16:11 cmjohnson1: depooling mw80
16:10 bblack: updated vhtcpd package on brewster to 0.0.7-1
16:09 Jeff_Green: authdns update, switching mx records for donate.wikimedia.org to mchenry/sodium
16:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:15 paravoid: apt: include jmxtrans 242-1
14:59 pp-pdf1: restarted all services
14:59 pp-pdf1: - update mwlib to 0.15.9
10:18 logmsgbot: maxsem synchronized wmf-config/mobile.php
10:10 logmsgbot: maxsem synchronized wmf-config/mobile.php
10:03 logmsgbot: maxsem synchronized wmf-config/mobile.php
10:01 logmsgbot: maxsem synchronized wmf-config/mobile.php
09:27 hashar: jenkins upgraded.
09:15 apergos: reinstalling dataset1001
09:01 hashar: gallium : upgraded Jenkins from 1.509.1 to 1.509.2 bug 50522
08:46 hashar: gerrit : added Tntnet user to the 'openzim' group. Grants him push/submit rights on the openzim git repository. Requested by Kelson.
08:33 logmsgbot: nikerabbit synchronized php-1.22wmf9/extensions/UniversalLanguageSelector/ 'ULS to master'
08:17 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'ULS on enwiki part 2'
08:16 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS on enwiki part 1'
08:11 logmsgbot: nikerabbit synchronized php-1.22wmf8/extensions/UniversalLanguageSelector/ 'ULS to master'
05:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 2 05:25:52 UTC 2013
05:22 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Tue Jul 2 05:21:48 UTC 2013
05:15 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Tue Jul 2 05:14:00 UTC 2013
05:06 Tim: testing LU with a real shell (not /bin/false)
02:24 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 2 02:23:13 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Tue Jul 2 02:15:57 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Tue Jul 2 02:09:03 UTC 2013
01:22 gwicke: updated Parsoid to 65bd7ae9
00:56 logmsgbot: ori testing dologmsg. mutante ftw.
00:06 mutante: restarted logmsgbot
July 1
23:51 ori-l: logsmsgbot's cidr-filtering was broken by https://gerrit.wikimedia.org/r/#/c/ 69624 /, manifested now because of neon restart earlier
22:49 mutante: restarted snmptt and snmptrapd on neon
22:43 mutante: deleting files from snmptt spool dir on neon
22:18 LeslieCarr: restarting neon for new kernel
21:37 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable VisualEditor for all logged-in users on enwiki'
21:36 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Plumbing for VE settings'
21:34 gwicke: updated Parsoid to 613287b
21:24 logmsgbot: catrope Finished syncing Wikimedia installation... : Scap for i18n changes in VE
21:14 logmsgbot: catrope Started syncing Wikimedia installation... : Scap for i18n changes in VE
21:04 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor/ 'Update VE to master in wmf8 (did wmf9 earlier)'
21:02 logmsgbot: catrope synchronized php-1.22wmf8/extensions/TimedMediaHandler/ 'Deploy TMH API cherry-pick to wmf8 (already in wmf9)'
20:55 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor 'Updating VE to master in wmf9; for real this time'
20:47 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor 'Updating VE to master in wmf9'
20:36 Krinkle: Zuul was clogging up with ~ 350 queued up events. Added 1 extra executor on Jenkins master and slave.
20:11 gwicke: updated Parsoid to 55d011d
18:02 logmsgbot: csteipp synchronized wmf-config/CommonSettings.php 'enablign silent redirects for test wikis'
18:00 logmsgbot: csteipp synchronized wmf-config/InitialiseSettings.php 'pushing config flag for centralauth silent login'
17:22 logmsgbot: csteipp synchronized php-1.22wmf9/extensions/CentralAuth 'Updating wmf9 to latest CentralAuth'
16:00 cmjohnson1: reseating disk 3 db44
14:58 ^demon: gerrit: brought offline, removed /var/lib/gerrit2/review_site/cache/*, brought back up
13:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
12:09 mark: Depooled old eqiad mobile cache servers
11:25 mark: Depooled old eqiad upload varnish servers
10:54 apergos: rebooting labstore3, /srv reporting errors, couldn't xfs_check it
10:08 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/Campaigns 'I58a424c70: Restore extensions/Campaigns submodule; was accidentally removed by change I2cafcb595.'
03:53 Tim: restarted morebots
03:33 TimStarling: testing new scap
June 29
02:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 29 02:11:56 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Sat Jun 29 02:06:08 UTC 2013
02:04 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Sat Jun 29 02:03:41 UTC 2013
00:00 jvandavier: jvandavier is the best
June 28
23:59 K4-713: updated payments cluster to eee8689ab646
20:55 gwicke: updated Parsoid to 454f87a
19:20 mutante: attempting clean stop and restart of snmptt service on neon
19:08 mutante: deleting /var/spool/snmptt/ files on neon running out of inodes
17:57 ^demon: restarting gerrit, bringing back missing documentation lost during upgrade
17:33 mutante: upgrading binutils and libldap on bast1001
17:20 mutante: installing some pkg upgrades on fenari
16:56 paravoid: cause of brief outage: operator error caused restart of varnish on all upload caches eqiad, cascaded into a swift overload, cascaded into imagescaler outage & page
16:39 LeslieCarr: restarting snmptrapd and smptt on neon
16:28 hashar: created operations/debs/jenkins-debian-glue and force pushed upstream sources from https://github.com/mika/jenkins-debian-glue
15:46 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.22wmf9
14:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 28 14:05:42 UTC 2013
13:58 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Fri Jun 28 13:56:55 UTC 2013
13:52 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Fri Jun 28 13:51:49 UTC 2013
13:44 logmsgbot: reedy Finished syncing Wikimedia installation... : Scaptastic
13:42 logmsgbot: reedy Started syncing Wikimedia installation... : Scaptastic
13:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything back from wmf7 to wmf8
13:39 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Wikibase
13:28 logmsgbot: reedy Finished syncing Wikimedia installation... : Ensure all l10n caches are built and properly built
13:22 logmsgbot: reedy Started syncing Wikimedia installation... : Ensure all l10n caches are built and properly built
13:21 logmsgbot: reedy synchronized wmf-config/
13:16 hashar: jenkins: reducing # of executor on master from 4 to 2
12:24 hashar: jenkins: migrating mwext-Wikibase-client-tests and mwext-Wikibase-repo-tests from master to slaves
09:57 hashar: gallium : updating some files under /srv/org to belong to jenkins-slave:jenkins-slave instead of jenkins.
09:24 hashar: jenkins: made the .erb linting jobs to be voting 70994
06:28 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to wmf8
05:07 Krinkle: Note, anons were eventually getting HTTP 500 too (even for enwiki Main Page) as action=view threw as well squid cache got purged somehow and apaches unable populate it back. And then LVS depooled stuff for apache serving 500 on action=view for main page.
05:07 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 're-enabling GeoData'
05:05 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
05:00 Tim: reverting to 1.22wmf7
04:58 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'disable GeoData'
04:55 RoanKattouw: sync-dir php-1.22wmf8/cache/l10n
04:55 Krinkle: catrope Removing that extension from extension-list
04:54 Krinkle: Cause from GeoData "coordinates" magic word. Deployed through broken i18n stuff from WikibaseDataModel deployment earlier that caused fatal during the scap just now
04:54 Krinkle: About 20 minutes ago the site went down for logged in users (and anon users hitting an uncached page) throwing HTTP 500
04:54 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.22wmf8.php
04:53 logmsgbot: tstarling synchronized wmf-config/ExtensionMessages-1.22wmf9.php
04:51 logmsgbot: catrope Finished syncing Wikimedia installation... : Trying to fix the site
04:47 logmsgbot: catrope Started syncing Wikimedia installation... : Trying to fix the site
04:47 RoanKattouw: Running scap in an attempt to fix things
04:46 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.22wmf8.php
04:31 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master
04:22 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master
02:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 28 02:18:38 UTC 2013
02:11 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Fri Jun 28 02:10:26 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Fri Jun 28 02:07:05 UTC 2013
01:53 mwalker: updating payments from bf6cbd6 to 9d7748c
00:57 gwicke: updated Parsoid to 68a086a
00:00 mutante: graceful Apaches, activate wiikipedia and wekipedia
June 27
22:31 gwicke: updated Parsoid to 8b38dcc
20:52 hashar: jenkins: migrating mediawiki-core-phpunit-parser from master to slaves
20:44 logmsgbot: spage synchronized php-1.22wmf8/extensions/Campaigns 'updating Campaigns in 1.22wmf8'
20:43 hashar: deployment-prep jenkins: migrating mediawiki-core-phpunit-misc from master to slaves
20:42 logmsgbot: spage synchronized php-1.22wmf8/extensions/GuidedTour 'updating GuidedTour in 1.22wmf8'
20:36 logmsgbot: spage synchronized php-1.22wmf9/extensions/GuidedTour 'updating GuidedTour in 1.22wmf9'
20:33 hashar: jenkins: migrating mediawiki-core-phpunit-databaseless form master to slaves
20:21 hashar: jenkins: migrating mediawiki-core-phpunit-api from master to slaves
18:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add Extension:NewUserMessage to de.wikiversity'
18:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:35 Reedy: Created Echo tables on enwikivoyage
18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, mediawikiwiki and testwikidatawiki to 1.22wmf9
18:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Rest of wikipedias to 1.22wmf8
18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.22wmf8
17:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki back to 1.22wmf9
17:38 logmsgbot: reedy synchronized php-1.22wmf8/extensions/Wikibase 'Updating to master of 1.22wmf6 branch'
17:37 logmsgbot: reedy synchronized php-1.22wmf7/extensions/Wikibase 'Updating to master of 1.22wmf6 branch'
17:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki back to 1.22wmf8
17:21 hashar: zuul: removing --force-message from layout 70849 and reloading zuul. Caused [[bugzilla:50300 |bug 50300 ]]
17:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 27 17:15:49 UTC 2013
17:13 hashar: Zuul/Jenkins merging bug is [[bugzilla:50300 |bug 50300 ]]
17:12 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Thu Jun 27 17:11:43 UTC 2013
17:11 hashar: Jenkins/Zuul is no more merging changes due to --force-message options no more being available in Gerrit :/
17:02 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 27 17:01:20 UTC 2013
17:01 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Thu Jun 27 17:00:37 UTC 2013
16:45 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf9 rebuild l10n cache
16:30 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf9 rebuild l10n cache
16:28 mark: Corrected IPv6 problem, repooled the new servers, depooled the old bits servers
16:19 logmsgbot: reedy synchronized php-1.22wmf9/extensions/WikibaseDataModel
16:15 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Wikibase/
16:15 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Diff
16:14 logmsgbot: reedy synchronized php-1.22wmf9/extensions/DataValues
16:12 logmsgbot: reedy synchronized php-1.22wmf9/
16:03 mark: Depooled row C bits caches, repooled old bits caches to investigate IPv6 problem
15:56 logmsgbot: reedy synchronized docroot and w
15:47 mark: Depooled old eqiad bits caches
15:41 mark: Pooled new eqiad bits caches
14:50 logmsgbot: mark synchronized wmf-config/squid.php
14:50 logmsgbot: mark synchronized wmf-config/squid.php
14:11 ^demon: gerrit: running puppet and restarting service
09:39 mark: Pooled new eqiad upload caches with 1% load
03:09 Tim: on fenari: moved away the outdated source copy that was at /home/wikipedia/common . Will update noc.
02:56 Tim: added springle to wmf and ops LDAP groups
02:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 27 02:19:05 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 27 02:13:40 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Thu Jun 27 02:07:37 UTC 2013
01:35 gwicke: Roan cleared Parsoid caches
01:33 gwicke: updated Parsoid to 091ebece
00:24 gwicke: updated Parsoid to eccca39
00:17 logmsgbot: catrope synchronized php-1.22wmf8/resources/startup.js 'touch'
00:17 logmsgbot: catrope synchronized php-1.22wmf7/resources/startup.js 'touch'
00:10 logmsgbot: catrope Finished syncing Wikimedia installation... : Scap for VE update, contained i18n changes
00:03 logmsgbot: catrope Started syncing Wikimedia installation... : Scap for VE update, contained i18n changes
June 26
23:48 logmsgbot: yurik synchronized php-1.22wmf7/extensions/ZeroRatedMobileAccess/
23:47 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Update VE to master'
23:47 logmsgbot: catrope synchronized php-1.22wmf7/extensions/VisualEditor 'Update VE to master'
23:46 logmsgbot: yurik synchronized php-1.22wmf8/extensions/ZeroRatedMobileAccess/
23:22 gwicke: updated Parsoid to 8c83524
22:22 LeslieCarr: killed slow query on db1025
21:15 Jeff_Green: changing fundraising db config to point to service aliases
18:37 Jeff_Green: commented out cron-msp queue flush in stat1:/etc/cron.d/sendmail to stop pointless cronspam, host is running exim
18:13 logmsgbot: yurik synchronized php-1.22wmf7/extensions/ZeroRatedMobileAccess
18:10 logmsgbot: yurik synchronized php-1.22wmf8/extensions/ZeroRatedMobileAccess
16:38 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Update VisualEditor to master'
16:38 logmsgbot: catrope synchronized php-1.22wmf7/extensions/VisualEditor 'Update VisualEditor to master'
16:08 gwicke: updated Parsoid to e79f70a
14:26 logmsgbot: reedy synchronized php-1.22wmf8/extensions/VisualEditor/modules/ve/ve.js
14:24 logmsgbot: reedy synchronized php-1.22wmf7/extensions/VisualEditor/modules/ve/ve.js
14:01 logmsgbot: reedy synchronized php-1.22wmf7/resources
14:00 logmsgbot: reedy synchronized php-1.22wmf8/resources
13:59 logmsgbot: reedy synchronized wmf-config 'touch 1'
12:08 hashar: Jenkins let jobs roam on gallium slave node and keep master only for tied jobs.
10:39 hashar: jenkins moving unit test jobs for mw extensions to use the slave tmpfs and let them roam on slaves
08:42 MaxSem: recreating Solr index
08:40 hashar: jenkins installed Simple Theme Plugin
08:35 hashar: jenkins reducing number of executors on (master) from 10 to 4. Most jobs are now running on the 'gallium' slave node.
08:35 hashar: jenkins: restarted it twice to finally have it up and running.
08:15 hashar: restarting Jenkins to apply plugins updates.
07:13 Tim: deploying squid config change to block API DoS attack
06:41 paravoid: upgrading ceph/ms-fe.eqiad.wmnet to 0.65, ignore warnings/alerts for the next few hours
03:59 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Updating VisualEditor to master'
03:58 logmsgbot: catrope synchronized php-1.22wmf7/extensions/VisualEditor 'Updating VisualEditor to master'
02:57 RoanKattouw: Updated Parsoid config for bug 50173
02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 26 02:27:09 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Wed Jun 26 02:13:32 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Wed Jun 26 02:07:35 UTC 2013
01:41 logmsgbot: catrope synchronized php-1.22wmf8/skins/vector/collapsibleTabs.js 'I6e277a9503a1e6003bc7bf9f9468ed7b35552e60'
June 25
23:26 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Updating VisualEditor to master'
23:25 logmsgbot: catrope synchronized php-1.22wmf7/extensions/VisualEditor 'Updating VisualEditor to master'
21:47 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving pool4 and 5 search traffic back to eqiad for realz'
21:34 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving pool2 and 3 search traffic back to eqiad for realz'
21:28 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:25 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving en and prefix search traffic back to eqiad for realz'
21:19 LeslieCarr: restarted pybal on lvs1003
21:16 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
19:34 Ryan_Lane: test
19:32 Ryan_Lane: test
19:26 logmsgbot: reedy synchronized .
19:26 Ryan_Lane: upgrading adminbot to 1.7.4 on wikitech-static
19:04 logmsgbot: reedy synchronized docroot and w
18:34 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving en and prefix search traffic back to pmtpa'
18:27 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving en and prefix search traffic back to eqiad'
18:14 logmsgbot: reedy synchronized php-1.22wmf8/extensions/UploadWizard
17:19 logmsgbot: aaron cleared profiling data
16:57 logmsgbot: reedy synchronized wmf-config/
15:25 hashar: jenkins: raising # of executors on 'gallium' slave node from 4 to 8
15:03 gwicke: updated Parsoid to 5ef05e6c9
14:54 paravoid: deluser icinga; rm -rf /home/icinga on all hosts but neon
14:45 paravoid: restarting nrpe on all hosts
14:25 logmsgbot: reedy synchronized php-1.22wmf8/extensions/ProofreadPage/
12:37 mark: Pooled the new eqiad mobile frontend caches with lowest weight in PyBal
12:27 hashar: jenkins: all mediawiki extension jobs got updated.
12:12 logmsgbot: mark synchronized wmf-config/squid.php
11:45 mark: Depooled old parsoid cache servers - parsoidcache migration complete
11:17 hashar: jenkins: migrating extensions jobs to use gallium slave ( lint , phpcs-HEAD ) or be bound on master (test extensions)
10:56 mark: Pooled cp1045 and cp1058 Varnish frontends in the parsoid cache pool
08:13 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS phase3'
02:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 25 02:26:25 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Tue Jun 25 02:13:22 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Tue Jun 25 02:07:28 UTC 2013
01:51 logmsgbot: krinkle synchronized php-1.22wmf7/extensions/VisualEditor/ 'If96bed '
01:49 logmsgbot: krinkle synchronized php-1.22wmf8/extensions/VisualEditor/ 'If96bed '
01:23 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings-labs.php 'Iac6a3f2f4d: wmgMFLogEvents => true on beta labs'
June 24
23:49 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'I82fe6063a: Disable AFTv4 (Bug 43892)'
23:06 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/GettingStarted 'I329f59929: Don't log when schemaAction is falsy. (Bug: 50065)'
23:06 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/GettingStarted 'I329f59929: Don't log when schemaAction is falsy. (Bug: 50065)'
22:44 Krinkle: Graceful reload of Zuul to deploy Ie51c7ce9
20:54 mutante: DNS update - switching RT over to magnesium and 4.x
19:53 logmsgbot: reedy synchronized php-1.22wmf8/extensions/ProofreadPage/
19:47 gwicke: updated Parsoid to 6ab78c8dd8a
19:46 mutante: moving backups from db9 to tridge
19:39 mutante: temp shutting down webserver on streber (RT scheduled downtime)
18:22 gwicke: updated Parsoid to b3a872fa7
18:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiooks to 1.22wmf8
18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote, wikiversity, wikinews, wiktionary to 1.22wmf8
18:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage and wiktionary to 1.22wmf8
18:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, wikimedia and closed to 1.22wmf8
17:09 apergos: er, "shot" many dead snmptt processes
17:09 apergos: while kicking icinga into working again on neon, also cleared out a pile of inodes in /var/spool/snmptt again, show many many dead processes too
16:55 logmsgbot: krinkle synchronized wmf-config/CommonSettings.php 'I94d74fd8abeba465f'
16:55 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'I94d74fd8abeba465f'
16:48 logmsgbot: krinkle synchronized php-1.22wmf7/extensions/VisualEditor/ 'I73522f63d9a49a4b7'
16:47 logmsgbot: krinkle synchronized php-1.22wmf8/extensions/VisualEditor/ 'I73522f63d9a49a4b7'
15:54 logmsgbot: reedy synchronized wmf-config/
15:50 ^demon: restarting gerrit
15:49 Reedy: Created WikiLove tables on enwikivoyage
14:51 hashar: jenkins: migrating pyflakes and pep8 jobs to slaves having the label 'hasContintPackages'
14:09 hashar: jenkins raising 'gallium' slave # of executors from 1 to 4. I have migrated some jobs to run on that slave.
02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 24 02:24:40 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Mon Jun 24 02:12:57 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Mon Jun 24 02:07:08 UTC 2013
June 23
07:06 logmsgbot: krinkle Finished syncing Wikimedia installation... : Update VisualEditor I7f875f48ce2d
06:39 logmsgbot: krinkle Started syncing Wikimedia installation... : Update VisualEditor I7f875f48ce2d
06:32 RoanKattouw: Made ExtensionMessages-1.22wmf8.php group-writeable
06:28 Krinkle: Aborted scap due to php fatal errors with 1.22wmf8 (Permission error on ExtensionMessages-1.22wmf8.php; Missing files: Narayam.i18n.php, WebFonts.i18n.php and WebFonts.alias.php)
06:27 logmsgbot: krinkle Started syncing Wikimedia installation... : Update VisualEditor I7f875f48ce2d
02:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 23 02:16:43 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Sun Jun 23 02:09:09 UTC 2013
02:06 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Sun Jun 23 02:05:29 UTC 2013
June 22
15:46 Reedy: tin is down, dead to SSH
02:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 22 02:19:26 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Sat Jun 22 02:13:00 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Sat Jun 22 02:07:29 UTC 2013
June 21
22:47 binasher: created a wb_entity_per_page view in labsdb wikidatawiki_p
21:24 mutante: deleting /home/daniel on fenari (confirmed not needed)
20:31 mutante: installing upgrades on bast1001
20:27 mutante: cleaning up /home/daniel/ (the other Daniel) ssh keys on fenari and letting puppet recreate them
19:31 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js 'Gerrit change I3db3cf47b / bug 49967'
19:30 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js 'Gerrit change I3db3cf47b / bug 49967'
18:53 gwicke: updated Parsoid to 21e1e7d
18:09 mutante: installing package upgrades on kaulen (bugzilla)
18:03 gwicke: updated Parsoid to 37cd852f
14:38 hashar: jenkins: updating all phpcs-HEAD jobs to rely on git-changed-in-head script 69863
10:38 hashar: upgrading packages on gallium
06:09 gwicke: updated Parsoid to 90515ab5
02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 21 02:24:03 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Fri Jun 21 02:13:50 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Fri Jun 21 02:07:39 UTC 2013
00:05 gwicke: updated Parsoid to 393a3263
June 20
23:30 gwicke: updated Parsoid to 6dadde3
22:58 gwicke: updated Parsoid to 6240c19
22:06 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/GuidedTour 'Updating GuidedTour to 950ee2c70417be517f4cdce3a1b590f6fc28d388'
22:02 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/GuidedTour 'Updating GuidedTour to 950ee2c70417be517f4cdce3a1b590f6fc28d388'
21:01 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'I24bbe0d4e: Update EventLogging config to utilize API'
20:52 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/GettingStarted 'GettingStarted to bf05656766'
20:52 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/GuidedTour 'GuidedTour to a6fdf3c910'
20:51 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/GettingStarted 'GettingStarted to bf05656766'
20:51 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/GuidedTour 'GuidedTour to a6fdf3c910'
20:28 Krinkle: Gracefully reloading Zuul to deploy Ie7513f356cd8
20:24 logmsgbot: reedy synchronized php-1.22wmf8/extensions/ProofreadPage/
20:24 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/EventLogging 'EventLogging to 2351b4ccbb'
20:21 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/EventLogging 'EventLogging to 2351b4ccbb'
20:20 Krinkle: Gracefully reloading Zuul to deploy I8c0ac58d9498979b
20:11 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/CoreEvents 'CoreEvents to d291b64248'
20:11 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/CoreEvents 'CoreEvents to d291b64248'
20:07 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings-labs.php 'I83fcaa5ec: enable NavigationTiming on beta cluster'
19:49 logmsgbot: maxsem synchronized php-1.22wmf7/extensions/MobileFrontend
19:47 logmsgbot: maxsem synchronized php-1.22wmf8/extensions/MobileFrontend
19:23 logmsgbot: reedy synchronized wmf-config/
18:06 gwicke: updated Parsoid to b206b54
18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rest of wikipedias to 1.22wmf7
18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki, test2wiki and testwikidatawiki to 1.22wmf8
17:58 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 20 17:57:01 UTC 2013
17:55 logmsgbot: reedy synchronized wmf-config/
17:52 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Thu Jun 20 17:51:50 UTC 2013
17:41 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 20 17:40:46 UTC 2013
17:41 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 20 17:39:59 UTC 2013
17:38 logmsgbot: reedy synchronized php-1.22wmf8/extensions/ 'Sync Campaigns and CentralAuth'
17:34 logmsgbot: reedy Finished syncing Wikimedia installation... : rebuild localisation cache and testwiki to 1.22wmf8
17:10 logmsgbot: reedy Started syncing Wikimedia installation... : rebuild localisation cache and testwiki to 1.22wmf8
16:56 logmsgbot: reedy synchronized docroot and w
16:54 logmsgbot: reedy synchronized php-1.22wmf8/ 'initial sync'
16:09 Jeff_Green: nuked neon puppet.log b/c /var/log was 99% full
15:55 gwicke: updated Parsoid to bf8d3df
14:17 hashar: jenkins updating all mediawiki extensions unit testing jobs ( mwext-.*-testextensions-master'
13:28 cmjohnson1: dns update
02:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 20 02:22:16 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 20 02:12:59 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 20 02:07:13 UTC 2013
00:13 gwicke: updated Parsoid to 45944a0
June 19
21:06 RobH: rebooting tridge due to updates
20:57 Ryan_Lane: updating OpenStackManager on virt0 to 3e92211313ed8b73ee44fb395230bafb14dc9bdc
20:34 ottomata: analytics1019 and analytics1020 are fully puppetized. rebalancing them back into the hadoop cluster
18:42 logmsgbot: nikerabbit synchronized php-1.22wmf6/extensions/UniversalLanguageSelector/ 'ULS to master again'
18:35 gwicke: updated Parsoid to ddc47d8
18:35 logmsgbot: nikerabbit synchronized php-1.22wmf7/extensions/UniversalLanguageSelector/ 'ULS to master'
18:32 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'pull db64'
18:29 mutante: sync-apache and graceful for votewiki rewrites (change 67203)
18:15 logmsgbot: nikerabbit synchronized php-1.22wmf6/extensions/UniversalLanguageSelector/ 'ULS to master'
15:02 cmjohnson1: authdns update
14:58 cmjohnson1: powering down and relocating searchidx1001
14:55 Jeff_Green: update & reboot boron
14:36 cmjohnson1: powering down mw1171 to replace hard drive and reinstalling
13:52 cmjohnson1: powering down carbon to replace bad disk
10:01 logmsgbot: hashar synchronized wmf-config '69466 beta: $wmg for AbuseFilterCentralDB'
10:00 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '69466 beta: $wmg for AbuseFilterCentralDB'
02:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 19 02:18:52 UTC 2013
02:16 logmsgbot: catrope synchronized php-1.22wmf7/resources/startup.js 'touch'
02:16 logmsgbot: catrope synchronized php-1.22wmf6/resources/startup.js 'touch'
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Wed Jun 19 02:14:07 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Wed Jun 19 02:07:42 UTC 2013
02:05 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master
01:50 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master
00:57 gwicke: updated Parsoid to 0445108
June 18
22:52 logmsgbot: olivneh synchronized wmf-config/CommonSettings-labs.php 'Iaade2591b: Configure $wgEventLoggingBaseUri & $wgEventLoggingFile on beta cluster'
22:31 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings-labs.php 'Ibeb07a4b9: Re-enable EventLogging on beta cluster'
22:30 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'I7fc1a5ad4: Ensure forward-compatibility by setting '
22:16 mutante: scheduled 24 hour downtime for eqiad search-pools to stop paging
22:04 mutante: rebooting iron to finish upgrade
22:04 logmsgbot: maxsem synchronized php-1.22wmf6/extensions/MobileFrontend/
21:55 logmsgbot: maxsem synchronized php-1.22wmf6/extensions/MobileFrontend/
21:50 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:39 mutante: apache-graceful-all for redirects
21:37 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
20:41 LeslieCarr: added temp logging filter to cr2-eqiad analytics4 filter
20:34 mutante: upgrading iron to precise
19:44 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Turn Disambiguator on on en.wiki'
19:17 logmsgbot: kaldari Finished syncing Wikimedia installation... :
18:58 logmsgbot: kaldari Started syncing Wikimedia installation... :
18:50 mutante: rebooting iron
18:44 mutante: installing kernel and package upgrades on iron
18:33 mwalker: updating payments cluster (listeners) from aa0039104e71417845ec14a5ad666c2de464cfab to 8ca174296d570b6fda365cab37b2222d81c47c3d for adyen USD capture bug
18:12 greg-g: it's better than bad, it's good!
18:09 RobH: log log log log
18:09 mutante: LVS monitoring pages caused by neon recovering
18:08 RobH: ignore eqiad based search errors for now, as tampa is hosting all search traffic while hardware is repaired
18:07 RobH: search is based in tampa
17:40 cmjohnson1: dns update
17:28 cmjohnson1: wtp1008 is going down for main board replacement
17:09 mutante: reloading squid frontends in squid_esams_text and eqiad.text
17:07 mutante: reloading squid frontends in pmtpa.text
14:01 cmjohnson1: powering down search1001-1024 to relocate to row c5
08:26 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS phase 2'
08:16 logmsgbot: nikerabbit synchronized php-1.22wmf6/extensions/UniversalLanguageSelector/ 'ULS to master'
08:14 logmsgbot: nikerabbit synchronized php-1.22wmf7/extensions/UniversalLanguageSelector/ 'ULS to master'
07:19 ori-l: Force-confirmed e-mail for 160 accounts. full list in fluorine:/home/olivneh/users-forced-confirm-email-18-Jun-2013.list
07:12 ori-l: Marking e-mail addresses of accounts created in last 24h with no password as "confirmed" on all wikis on which VE is enabled.
04:13 ori-l: Manually marked the e-mail address of 54 enwiki users with e-mail address configured but no password (due to bug 49727) as confirmed.
03:41 logmsgbot: mflaschen synchronized php-1.22wmf7/extensions/VisualEditor/VisualEditor.hooks.php 'Hotfix to fix account creation'
03:15 logmsgbot: mflaschen synchronized php-1.22wmf6/extensions/VisualEditor/VisualEditor.hooks.php 'Emergency hotfix to fix account creation'
03:10 logmsgbot: mflaschen synchronized php-1.22wmf6/extensions/VisualEditor/VisualEditor.php 'Emergency hotfix to fix account creation'
03:10 superm401: Started hotfix for account creation on wmf6.
02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 18 02:25:42 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Tue Jun 18 02:13:24 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Tue Jun 18 02:07:31 UTC 2013
01:04 Krinkle: Gracefully reloading Zuul to deploy I5695a3
00:51 mutante: reloading squid config on pmtpa.txt
00:48 mutante: reloading squid config on esams text squids
00:40 mwalker: updating the payments cluster (listeners) from 5f67ceddd9d495cffd22f6b303443aba58041d47 to aa0039104e71417845ec14a5ad666c2de464cfab for additional smashpig content
00:11 mutante: touch /etc/exim4/defer_domains on streber
June 17
23:52 mutante: restarting exim on streber
23:52 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master
23:40 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master
23:36 ^demon: restarting gerrit
23:31 RoanKattouw: Running scap to update VisualEditor
23:11 mutante: install package upgrades on streber
23:06 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Change I95a8785c7 / Bug 49312: Add 'Programs' NS to MetaWiki'
23:05 Krinkle: Gracefully reloading Zuul to deploy I773e84
22:22 gwicke: updated Parsoid to b2a7b101e
22:20 mutante: built package and upgrading to php-luasandbox 1.7-1
19:38 Jeff_Green: dist-upgrade and reboot indium
19:29 gwicke: updated Parsoid to d1ae92720c4e
18:50 hashar: upgrading packages on gallium
18:41 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'bookkeeping changes for pmtpa dbs'
18:16 logmsgbot: aaron synchronized php-1.22wmf7/includes/db/DatabaseMysql.php 'ac676a2d83af283701dee111a28a52ea1730d657'
18:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: everything non 'pedia to 1.22wmf7
17:15 notpeter: rebooting es7 for lvm snapshot cleanup (again)
17:05 apergos: neon should be back in business
16:41 apergos: neon out of inodes again, clearing out /var/spool/snmpttp, will return to service shortly
13:11 apergos: need to find out why sfo-intranet.corp.wikimedia.org can't be resolved in order to have opendj back and restore mail service
12:40 apergos: restarting opendj on sanger
10:50 hashar: manually rebased mediawiki-config on fenari 7d10d3f --> 1d2093e
02:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 17 02:12:33 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Mon Jun 17 02:06:18 UTC 2013
02:04 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Mon Jun 17 02:03:45 UTC 2013
June 16
23:36 paravoid: restarting pybal on lvs1003/lvs1006 for ms-fe4.eqiad.wmnet ip change
02:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 16 02:13:03 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Sun Jun 16 02:06:29 UTC 2013
02:05 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Sun Jun 16 02:03:54 UTC 2013
June 15
08:38 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Mobile host for Commons'
02:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 15 02:19:50 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Sat Jun 15 02:13:57 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Sat Jun 15 02:08:17 UTC 2013
02:06 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
00:58 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
00:05 mutante: reloading squid config on cp1001-cp1020 (text eqiad) for RT-5296
June 14
23:25 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
22:27 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
22:26 logmsgbot: reedy synchronized php-1.22wmf7/extensions/SecurePoll/
22:06 logmsgbot: reedy synchronized php-1.22wmf7/extensions/SecurePoll/
21:57 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
21:56 logmsgbot: py synchronized wmf-config/db-eqiad.php 'repooling es1007 and es1010'
21:56 logmsgbot: reedy synchronized php-1.22wmf7/extensions/SecurePoll/
21:41 notpeter: reboot es1010 to deal with messed up lvm snapshot
21:36 logmsgbot: mholmquist Finished syncing Wikimedia installation... : Fixing UploadWizard in Chrome
21:33 notpeter: reboot es1007 to deal with messed up lvm snapshot
21:31 logmsgbot: py synchronized wmf-config/db-eqiad.php 'temp depooling es1007 and es1010 for reboots'
21:28 logmsgbot: mholmquist Started syncing Wikimedia installation... : Fixing UploadWizard in Chrome
21:03 notpeter: rebooting es10 to deal with fucked up snapshot volume
19:33 mutante: deploying small changes to bugzilla login screen
19:30 notpeter: installin updates on es7 while I'm at it
19:28 notpeter: rebooting es7 to deal with messed up vlm snapshot
18:46 gwicke: updated Parsoid config and code back to latest after finding and fixing Varnish config bug
18:31 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Re-enable VisualEditor on the rest of the wikis'
18:01 logmsgbot: aaron synchronized php-1.22wmf7/maintenance/purgeDeletedFiles.php '9e2ffededde6e7752a8bd64d2ae791af768213c0'
17:32 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Re-enable VE on mw.org'
15:44 cmjohnson1: powering down wtp1008 to swap cpu's
15:26 cmjohnson1: carbon going offline to run h/w test at Dell support request.
15:15 gwicke: rolled back Parsoid dependency update to investigate issues
15:14 cmjohnson1: working mw1041 updating bios
14:54 logmsgbot: ariel synchronized wmf-config/InitialiseSettings.php 'emergency disabling of Visual Editor, see bug 49577'
04:46 gwicke: updated Parsoid to 64921430b1
04:33 gwicke: updating Parsoid dependencies from config repository
02:20 cmjohnson1: dns update
02:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 14 02:17:44 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Fri Jun 14 02:11:02 UTC 2013
02:06 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Fri Jun 14 02:05:57 UTC 2013
01:25 paravoid: starting swift->ceph scripts on a screen on ms-fe1002
00:29 logmsgbot: catrope Finished syncing Wikimedia installation... : Fixup for VE deployment
00:20 logmsgbot: catrope Started syncing Wikimedia installation... : Fixup for VE deployment
00:10 RoanKattouw: Scapping to fix up VE deployment
June 13
23:59 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'I3728a4cd9'
23:42 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Enabling Campaigns & CoreEvents (4/4)'
23:41 logmsgbot: olivneh synchronized wmf-config/extension-list 'Enabling Campaigns & CoreEvents (3/4)'
23:41 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling Campaigns & CoreEvents (1/4)'
23:40 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings-labs.php 'Enabling Campaigns & CoreEvents (1/4)'
23:37 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/CoreEvents 'Adding CoreEvents extension to wmf7'
23:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 13 23:35:36 UTC 2013
23:31 logmsgbot: olivneh synchronized php-1.22wmf6/extensions/Campaigns 'Adding Campaigns extension to wmf6'
23:31 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 13 23:30:06 UTC 2013
23:30 logmsgbot: olivneh synchronized php-1.22wmf6/extensions/CoreEvents 'Adding CoreEvents extension to wmf6'
23:21 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 13 23:20:43 UTC 2013
23:15 logmsgbot: mholmquist synchronized php-1.22wmf6/extensions/UploadWizard/resources/mw.LanguageUpWiz.js 'Opera UploadWizard fix'
23:14 logmsgbot: mholmquist synchronized php-1.22wmf6/extensions/UploadWizard/includes/specials/SpecialUploadWizard.php 'Opera UploadWizard fix'
23:08 logmsgbot: anomie synchronized php-1.22wmf7/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'Update LocalisationUpdate to master in wmf7'
23:07 logmsgbot: anomie synchronized php-1.22wmf7/extensions/LocalisationUpdate/LocalisationUpdate.php 'Update LocalisationUpdate to master in wmf7'
22:57 logmsgbot: catrope synchronized php-1.22wmf6/resources/startup.js 'touch'
22:52 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Enable EventLogging for VE on enwiki'
22:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 13 22:49:02 UTC 2013
22:48 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable EventLogging for VE on enwiki'
22:43 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 13 22:42:42 UTC 2013
22:40 logmsgbot: catrope Finished syncing Wikimedia installation... : VisualEditor update to master, and adding Campaigns in wmf7
22:39 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 13 22:38:14 UTC 2013
22:34 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 13 22:33:09 UTC 2013
22:31 logmsgbot: catrope Started syncing Wikimedia installation... : VisualEditor update to master, and adding Campaigns in wmf7
22:18 RoanKattouw: Fixing perms on php-1.22wmf7, not group-writable
22:10 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
22:08 LeslieCarr: restarted squid on cp1004 due to "Jun 13 21:40:34 cp1004 squid[1961]: commBind: Cannot bind socket FD 23 to *:0: (98) Address already in use"
22:04 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
21:55 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
21:15 logmsgbot: spage synchronized php-1.22wmf6/extensions/GuidedTour 'latest GuidedTour'
21:13 logmsgbot: spage synchronized php-1.22wmf6/extensions/EventLogging 'latest EventLogging'
20:51 LeslieCarr: mw1173 has bad memory, rt 5294
20:48 logmsgbot: spage synchronized php-1.22wmf7/extensions/GuidedTour 'latest GuidedTour'
20:48 LeslieCarr: repuppetizing mw1020 after a reinstall
20:41 Jeff_Green: authdns update
20:41 logmsgbot: spage synchronized php-1.22wmf7/extensions/EventLogging 'latest EventLogging'
20:19 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/UploadWizard/resources/mw.LanguageUpWiz.js 'Ia09ace5dc: Fix UploadWizard's config variables (2/2)'
20:18 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/UploadWizard/includes/specials/SpecialUploadWizard.php 'Ia09ace5dc: Fix UploadWizard's config variables'
20:16 logmsgbot: kaldari synchronized php-1.22wmf6/extensions/Thanks/modules/ext.thanks.thank.js 'fix IE Thanks bug'
20:16 logmsgbot: reedy synchronized php-1.22wmf5/cache/
20:07 gwicke: updated Parsoid to a33980cf
19:44 binasher: rebooted mc1009 as a firedrill, chaos monkey ahoy!
19:36 cmjohnson1: dns update
19:06 logmsgbot: kaldari synchronized php-1.22wmf7/extensions/Thanks/modules/ext.thanks.thank.js 'fix IE Thanks bug'
18:59 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving last of search traffic to pmtpa'
18:50 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving more search traffic to pmtpa'
18:33 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving search traffic to pmtpa for en, de, fr, and ja'
18:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki, testwikidatawiki and mediawikiwiki to 1.22wmf7
18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikipedia to 1.22wmf6
16:29 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf7 and build l10n cache
16:12 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf7 and build l10n cache
16:04 logmsgbot: reedy synchronized w
16:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:46 logmsgbot: reedy synchronized docroot
15:45 logmsgbot: reedy synchronized php-1.22wmf7 'Initial file sync'
07:33 apergos: the last of the c2100s in the swift pool has been replaced (and new rings pushed), finally. now we "only" have the H310 controllers to replace...
02:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 13 02:28:27 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Thu Jun 13 02:14:58 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 13 02:08:12 UTC 2013
00:29 logmsgbot: asher synchronized wmf-config/mc-eqiad.php 'disable persistent conns to twmprxy'
00:24 paravoid: ceph readding osd.81, osd.125, osd.137, osd.141 after disk replacement (#5202, #5228, #5248, #5263)
00:22 logmsgbot: asher synchronized wmf-config/mc-eqiad.php 'trying twemproxy on all eqiad hosts'
00:14 logmsgbot: asher synchronized wmf-config/mc-eqiad.php 'large server_failure_limit, small retry_timeout for twemproxy test hosts'
June 12
23:57 logmsgbot: asher synchronized wmf-config 'revert twemproxy'
23:48 logmsgbot: asher restarted twemproxy on all servers
23:46 logmsgbot: asher synchronized wmf-config/twemproxy-pmtpa.yaml 'pull 10.0.12.15'
23:42 logmsgbot: aaron cleared profiling data
23:35 logmsgbot: asher synchronized wmf-config 'use twemproxy for memcached access from all hosts'
23:33 logmsgbot: tstarling cleared profiling data
23:10 paravoid: apt: updating libmemcached/php5-memcached (#5268)
23:00 logmsgbot: asher synchronized wmf-config/mc-eqiad.php 'set (server_failure_limit, retry_timeout) = 0 for twemproxy test hosts'
22:26 logmsgbot: aaron synchronized php-1.22wmf6/includes/objectcache/MemcachedPeclBagOStuff.php '7f091934ceae0280b44292e5ce426b24c5a579a2'
22:24 logmsgbot: aaron synchronized php-1.22wmf5/includes/objectcache/MemcachedPeclBagOStuff.php 'f6287b2e2faba20aea7379dae84ee7c5dd84983f'
21:35 mutante: deploying new favicon on wikitech
21:22 LeslieCarr: restarting caesium
20:56 mutante: PXE booting analytics1020
18:51 logmsgbot: asher synchronized wmf-config/mc-eqiad.php 'use twemproxy on mw113* (10 api apaches)'
17:29 cmjohnson1: powering down wtp1008 for h/w problem/testing
17:23 logmsgbot: yurik synchronized php-1.22wmf5/extensions/ZeroRatedMobileAccess/
17:21 yurik: mw1020 complained during depl about fingerprint 81:2e:56:16:83:fd:b1:65:08:7b:ad:f8:66:98:19:c9.
17:20 ori-l: mw1020: Warning: Remote Host Identification Has Changed
17:18 logmsgbot: yurik synchronized php-1.22wmf6/extensions/ZeroRatedMobileAccess/
17:06 cmjohnson1: removing/replacing disk slot 9 ms-be1007
16:36 cmjohnson1: removing/replacing failed disks in slot5 and slot9 from ms-be1012
16:00 cmjohnson1: powering down carbon for h/w testing/reports
15:19 cmjohnson1: mw1041 powering down for h/w issue
15:10 gwicke: updated Parsoid to 058da0822
02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 12 02:27:19 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Wed Jun 12 02:13:49 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Wed Jun 12 02:07:39 UTC 2013
00:21 Ryan_Lane: upgraded mediawiki on wikitech-static
00:11 Tim: installing the experimental libmemcached packages on srv193
June 11
22:58 cmjohnson1: wtp1008 rebooting to check bios setting
21:26 cmjohnson1: mw1171 powering down via racadm
21:21 logmsgbot: maxsem synchronized php-1.22wmf5/extensions/MobileFrontend/
21:19 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:11 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
20:33 mutante: deploying bugzilla frontpage css/html changes (bug 22170)
20:28 cmjohnson1: powering off asw-c8-eqiad and removing from rack
20:18 LeslieCarr: halting asw-c8-eqiad
20:17 LeslieCarr: disabling asw-c8-eqiad
19:46 gwicke: updated Parsoid to ab8d15596f
19:35 logmsgbot: kaldari Finished syncing Wikimedia installation... :
19:23 logmsgbot: kaldari Started syncing Wikimedia installation... :
18:56 Aaron|home: Cleared rows for queued transcodes on commonswiki
18:41 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Enable Echo interaction schema and cohort study'
18:40 cmjohnson1: depooling and powering down wtp1001-1002 and wtp1005-1tp1008
18:40 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Echo interaction schema and cohort study'
18:15 Aaron|home: Parsoid jobs pipeline fixed with 866e66104fa6d285911da1888b9f9dcece2f53a4, pipeline should be working now
17:21 logmsgbot: asher restarted twemproxy on all servers
17:20 logmsgbot: asher synchronized wmf-config
16:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
16:03 logmsgbot: reedy synchronized docroot/bits/favicon/testwikidata.ico
15:56 logmsgbot: asher restarted twemproxy on all servers
15:22 hashar: cleaned up permissions on operations/debs/buck , that is now a regular ops repo (ops are owner + force push )
15:13 logmsgbot: reedy synchronized php-1.22wmf6/extensions/DataValues/
15:13 logmsgbot: reedy synchronized php-1.22wmf6/extensions/Wikibase/
15:12 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Wikibase/
15:09 mark: Inserted varnish 3.0.3plus~rc1-wm11 packages into the precise-wikimedia APT repository
14:41 hashar: granted myself force push rights on operations/debs/buck to push in git://github.com/facebook/buck.git@master to 'upstream' branch
08:16 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Enable ULS'
08:14 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'Disable narayam and WebFonts'
08:10 Nikerabbit: Starting ULS deployment phase 1
07:21 Jamesofur: updating Board Election translations on cluster
02:31 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 11 02:30:24 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Tue Jun 11 02:16:57 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Tue Jun 11 02:09:34 UTC 2013
01:31 paravoid: restarting all jobrunners per Aaron's request
01:07 logmsgbot: aaron synchronized php-1.22wmf6/includes/db/LoadMonitor.php '2e62b818a01d714b8464689eec2394a993209a70'
00:56 logmsgbot: catrope synchronized php-1.22wmf5/extensions/Wikibase
00:56 logmsgbot: catrope synchronized php-1.22wmf6/extensions/Parsoid
00:55 logmsgbot: catrope synchronized php-1.22wmf5/extensions/Parsoid
00:46 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Added parsoid jobs to $wgJobTypesExcludedFromDefaultQueue'
00:39 RoanKattouw: mw1020: @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
00:35 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Point Parsoid updates to LVS IP'
00:20 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Re-enabled TTM server message jobs.'
June 10
23:33 paravoid: disabling asw-c-eqiad ge-4/0/7, enabling ge-4/0/6, moving description ms-fe1004 from one to the other and fixing VLANs
22:50 logmsgbot: reedy synchronized php-1.22wmf6/extensions/Wikibase/
22:48 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Wikibase/
22:10 mutante: restarting lucene on search prefix hosts
21:55 gwicke: updated Parsoid to 6e40256821
21:08 mutante: dpkg --configure -a on analytics1020 to fix interrrupted dpkg
21:02 mutante: puppet re-signing analytics1020
20:48 mutante: restarting lsearchd on all pool4 search nodes
19:39 cmjohnson1: dns update
19:36 RoanKattouw: Restarted Apache on gallium
19:34 logmsgbot: olivneh synchronized php-1.22wmf6/includes/content 'Reverting change Ibfb2cbefe49398'
19:34 mutante: reinstalling analytics1020
19:26 mutante: powercycling analytics1020
19:24 mutante: deleting and recreating /var/spool/snmptt on neon
19:08 cmjohnson1: powering down ms-fe1004 to relocate to different row
18:58 RoanKattouw: Restarted Varnish on cerium and titanium
18:47 mutante: deleting files from /var/spool/snmptt to fix neon again
18:35 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikibooks to 1.22wmf6
18:29 paravoid: depooling mw1171, broken hardware #5231
18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikisource to 1.22wmf6
18:26 Reedy: mw1171 is running with readonly file system
18:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage and wiktionary to 1.22wmf6
18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia, special, private and fishbowl to 1.22wmf6
17:48 mutante: rebooting analytics1020
17:46 mutante: killed and restared lucene on search indexers after imports, starting incremental updater
17:38 mutante: restarting search indexers to let them know about new wikis, running import-db scripts for elwikivoyage, vecwiktionary, testwikidatawiki
17:22 logmsgbot: catrope synchronized php-1.22wmf5/extensions/VisualEditor/ApiVisualEditor.php 'Undo live hack'
17:15 logmsgbot: catrope synchronized php-1.22wmf5/extensions/VisualEditor/ApiVisualEditor.php 'Live hack for debugging'
17:14 logmsgbot: catrope synchronized php-1.22wmf5/extensions/VisualEditor/ApiVisualEditor.php 'Live hack for debugging'
16:29 ^demon: restarting apache on gerrit box
08:49 apergos: restarted gerrit again, poor thing
08:26 logmsgbot: nikerabbit synchronized php-1.22wmf5/extensions/UniversalLanguageSelector/ 'ULS to master'
08:10 logmsgbot: nikerabbit synchronized php-1.22wmf6/extensions/UniversalLanguageSelector/ 'ULS to master'
07:34 Jamesofur: updated Board Election translations on cluster
02:36 gwicke: updated Parsoid to 956117df0e
02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 10 02:06:49 UTC 2013
02:03 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Mon Jun 10 02:02:14 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Mon Jun 10 02:01:29 UTC 2013
00:12 Krinkle: Gerrit has once again closed the Zuul socket, ssh event stream down
June 9
22:33 Nemo_bis: gitblit fixed as of 1h ago: ^demon> Yeah, I saw the e-mails.
18:03 gwicke: deploy Parsoid 836ae1e, hopefully fixing the hanging worker issue
16:11 gwicke: restarting Parsoid
05:20 ^demon: restarted gerrit, again
03:29 Krinkle: git.wikimedia.org is 503 Service Temporarily Unavailable
02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 9 02:07:43 UTC 2013
02:03 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Sun Jun 9 02:02:15 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Sun Jun 9 02:01:31 UTC 2013
01:59 Krinkle: Zuul is halted on a broken Gerrit event stream again.
June 8
23:22 RoanKattouw: Restarted Parsoid on a few machines where it was stuck
23:02 Jamesofur: synced Board Election translations on cluster
16:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Actually enable EducationProgram on the swedish wikipedia'
10:40 Nemo_bis: git.wikimedia.org 503: Service Temporarily Unavailable
07:54 Jamesofur: synced Board Election translations on cluster
02:25 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/
02:25 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
02:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 8 02:19:43 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Sat Jun 8 02:08:16 UTC 2013
02:06 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Sat Jun 8 02:05:56 UTC 2013
01:37 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
01:37 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/
01:00 RoanKattouw: Restarting Parsoid on all backends
00:54 RoanKattouw: Restarted Apache on gallium AGAIN. Patch for the underlying issue is https://gerrit.wikimedia.org/r/#/c/67551/
June 7
23:09 RoanKattouw: Restarted Apache on gallium agai
22:47 gwicke: update Parsoid to 9ccf7a3365
22:46 RoanKattouw: and on cerium and titanium too
22:46 RoanKattouw: Rerunning puppet on nickel for parsoidcache change
22:40 LeslieCarr: restart snmptrapd and smptt on neon
22:24 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php
22:03 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php
21:58 logmsgbot: reedy synchronized php-1.22wmf6/extensions/EducationProgram/
21:57 logmsgbot: reedy synchronized php-1.22wmf5/extensions/EducationProgram/
21:49 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Enable REL1_21 for extdist'
21:45 cmjohnson1: auth-dns update
21:21 mutante: installing package upgrades on neon now that it has inodes again
21:08 cmjohnson1: auth-dns update removing tampa parsoid dns entries
21:06 mutante: neon ran out of inodes, culprit /var/spool/snmptt has >300k files, deleting -mtime +2 in batches
20:49 RoanKattouw: Actually restarting pybal on lvs4 now
20:48 RoanKattouw: Running puppet on lvs4
20:47 RoanKattouw: Restarting pybal on lvs4
20:32 RoanKattouw: Restarting pybal on lvs3
19:50 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Tweaked redis queue timeout and enabled gzip'
19:43 RoanKattouw: Restarting Parsoid manually because git-deploy is not cooperating
19:29 mutante: installing package upgrades on neon (Icinga, nagios-plugins)
19:28 mutante: installing package upgrades on bast1001
19:24 mutante: installing package upgrades on fenari
19:20 RoanKattouw: Restarted Apache on gallium (APC cache pollution)
19:13 cmjohnson1: shutting down wtp1001 and wtp1002 to relocate to c8-eqiad
19:06 RoanKattouw: Depooled wtp1008 for investigation, Chris saw some sort of error on it
19:04 RoanKattouw: Depooled wtp1001, wtp1002 so Chris can move them
18:52 cmjohnson1: restarting ns1
18:47 cmjohnson1: auth-dns update (removing/renaming db1013)
18:28 mutante: DNS update - removing singer
17:52 cmjohnson1: shutting down db1013 to move to frack and rename
17:45 logmsgbot: aaron synchronized php-1.22wmf5/maintenance/eraseArchivedFile.php '2aa62f6afbd48a3362b78788b81aed847e6e805f'
16:29 logmsgbot: reedy synchronized wmf-config/
16:22 logmsgbot: reedy synchronized wmf-config/
16:04 logmsgbot: reedy synchronized wmf-config/
15:59 Reedy: Created EducationProgram tables on hewiki
15:58 Reedy: Created EducationProgram tables on sewiki
15:56 Reedy: Creating wikilove tables on viwiki
14:17 hashar: Zuul was no more triggering Jenkins jobs bug 49294 . Fixed by Chad who restarted Gerrit.
13:55 ^demon: restarting gerrit, queue is overloaded and it's killing the bridge to jenkins
13:44 ^demon: restarting zuul per https://bugzilla.wikimedia.org/show_bug.cgi?id=49294#c4
12:46 logmsgbot: reedy synchronized php-1.22wmf6/extensions/Parsoid/
12:45 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Parsoid/
02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 7 02:27:05 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Fri Jun 7 02:16:10 UTC 2013
02:06 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Fri Jun 7 02:05:54 UTC 2013
00:48 logmsgbot: catrope synchronized php-1.22wmf6/extensions/VisualEditor 'Updating VisualEditor to master'
June 6
23:39 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php
23:31 gwicke: updating Parsoid to 020c7b2
23:24 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Plumbing for $wmgVisualEditorExperimental'
23:23 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Set $wmgVisualEditorExperimental = true on mediawiki.org'
23:20 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Parsoid/
23:17 logmsgbot: reedy synchronized php-1.22wmf6/extensions/Parsoid/
23:15 logmsgbot: reedy synchronized php-1.22wmf6/extensions/Parsoid/
23:08 logmsgbot: tstarling cleared profiling data
23:04 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Parsoid/php/Parsoid.php 'fix hook call'
22:56 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'touch'
22:47 logmsgbot: catrope synchronized visualeditor.dblist 'Enabling VisualEditor on the remaining 270 Wikipedias'
22:44 logmsgbot: gwicke Finished syncing Wikimedia installation... :
22:36 logmsgbot: gwicke Started syncing Wikimedia installation... :
22:28 gwicke: deploying https://gerrit.wikimedia.org/r/#/c/67338/
22:22 logmsgbot: csteipp synchronized wmf-config/InitialiseSettings.php 'enable SUL2 on testwikidatawiki'
22:14 RoanKattouw: WTF, puppet ran just fine on srv283
22:13 RoanKattouw: Running puppet on srv283 to figure out what broke
22:11 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deploying PostEdit change to core, GettingStarted and GuidedTour for E3 deployment
22:01 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploying PostEdit change to core, GettingStarted and GuidedTour for E3 deployment
21:43 Krinkle: Zuul gave up on connecting to Gerrit. Restarting Zuul made it try again. Restarting Gerrit again made it connect and it seems back up.
21:40 logmsgbot: catrope synchronized php-1.22wmf5/extensions/Parsoid 'Adding Parsoid to wmf5'
21:26 Krinkle: Restarting zuul on gallium
21:11 kaldari: completed updateSpecialPages.php on testwiki
21:10 kaldari: Running updateSpecialPages.php on testwiki to test new Disambigutor special pages
21:00 logmsgbot: yurik synchronized php-1.22wmf6/extensions/ZeroRatedMobileAccess
20:58 RoanKattouw: Restarting zuul on gallium
20:57 logmsgbot: yurik synchronized php-1.22wmf5/extensions/ZeroRatedMobileAccess
20:45 logmsgbot: reedy synchronized database lists files:
20:36 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Send all Parsoid requests to eqiad'
20:34 logmsgbot: reedy synchronized php-1.22wmf6/extensions/
19:42 logmsgbot: reedy synchronized php-1.22wmf6/extensions/WikimediaMaintenance/
19:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:35 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
19:34 logmsgbot: reedy synchronized php-1.22wmf5/extensions/WikimediaMaintenance/
19:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:04 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Set wgSecurePollScript'
19:01 logmsgbot: reedy synchronized w/auth-api.php
18:53 logmsgbot: reedy synchronized database lists files:
18:53 logmsgbot: reedy synchronized wmf-config/
18:44 logmsgbot: reedy synchronized database lists files:
18:39 logmsgbot: reedy synchronized wikidata.dblist
18:39 LeslieCarr: removed some extraneous packages from neon which had filled its root partition
18:29 Reedy: Created translate tables on testwikidatawiki
18:28 logmsgbot: reedy synchronized database lists files:
18:27 logmsgbot: reedy synchronized database lists files:
18:26 logmsgbot: reedy synchronized wmf-config/
18:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
18:20 logmsgbot: reedy synchronized database lists files:
18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf6
17:25 logmsgbot: reedy Finished syncing Wikimedia installation... : Take 2
17:19 logmsgbot: reedy Started syncing Wikimedia installation... : Take 2
17:09 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf6 and rebuild l10n cache
16:58 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf6 and rebuild l10n cache
16:55 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.22wmf6 and rebuild l10n cache
16:50 logmsgbot: reedy synchronized w
16:49 logmsgbot: reedy synchronized docroot
16:47 logmsgbot: reedy synchronized w
16:46 logmsgbot: reedy synchronized docroot
16:45 logmsgbot: reedy synchronized php-1.22wmf6/
15:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
13:44 ^demon: zuul seems to not be yelling about invalid json anymore
13:41 ^demon: rolling gerrit back to 2.6-rc0-144-gb1dadd2 (version before any updates). Too many rough edges we didn't catch in testing, zuul is not happy
12:53 ^demon: gerrit rollback to 2.7-rc1-420-g5d5c5c3, some of the new work wasn't quite ready for prime time
12:30 ^demon: gerrit's back, running 2.7-rc1-424-gef469ac
12:26 ^demon: bringing gerrit down for a few minutes to pick up some changes
07:28 apergos: wtp1008 rebooted itself a few minutes ago (hence the icinga whine), no idea why
07:22 apergos: and the rest of the ms-fes
07:18 apergos: restarted swift on most backends as well
07:15 apergos: swift init restart all on ms-fe1 and 3
05:16 Tim: fixed high enwiki master query load due to GuidedTour
05:15 logmsgbot: tstarling synchronized php-1.22wmf5/extensions/GuidedTour
05:13 logmsgbot: tstarling synchronized php-1.22wmf5/extensions/GuidedTour/GuidedTourHooks.php
05:13 logmsgbot: tstarling synchronized php-1.22wmf5/extensions/GuidedTour/modules/tours/test.js
04:19 logmsgbot: tstarling synchronized php-1.22wmf5/includes/Revision.php 'remove debug hack'
03:55 logmsgbot: tstarling synchronized php-1.22wmf5/includes/Revision.php 'temp debug hack'
03:48 logmsgbot: tstarling synchronized php-1.22wmf5/includes/cache/MessageCache.php 'revert temp hack'
03:46 logmsgbot: tstarling synchronized php-1.22wmf5/includes/cache/MessageCache.php 'temp debug log for high master query load'
02:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 6 02:15:33 UTC 2013
02:06 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Thu Jun 6 02:06:01 UTC 2013
02:06 Krinkle: Graceful reload of Zuul to deploy I4e7978
00:56 mwalker: updating listeners from a9fdfd328a8d06d4efb5f4037c281cf4341e23ba to e5f30211ae89ba52a1234715a6d59a7a2d421169; updating payments from cf2d1639a9a8b2854d5302db2aba86199b4e4101 to bf6cbd63cb3cc6148f8abf35767c7dba4ce92b0e
00:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:10 Tim: deployed apache configuration for vote.wikimedia.org
00:09 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Enabled message cache debug log'
June 5
23:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:57 logmsgbot: reedy synchronized wmf-config/
23:56 logmsgbot: reedy synchronized database lists files:
23:56 logmsgbot: reedy synchronized docroot
23:44 logmsgbot: spage Finished syncing Wikimedia installation... :
23:40 Tim: added vote.wikimedia.org to DNS
23:38 logmsgbot: spage Started syncing Wikimedia installation... :
23:29 logmsgbot: mlitn synchronized php-1.22wmf5/extensions/ArticleFeedback/ 'Update ArticleFeedbackv5 to master'
23:13 mutante: enabling wikidata "entity" URLs like http://www.wikidata.org/entity/Q1
22:52 logmsgbot: maxsem synchronized wmf-config
22:21 Ryan_Lane: upgrading salt in labs and production
22:09 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
22:00 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
21:39 logmsgbot: dzahn synchronized docroot/testwikidata
21:23 RoanKattouw: Killed old process on wtp1004 that was hogging the port
21:22 mutante: shutting down mw1171
21:06 ^demon: killed gitweb, long live gitblit
20:27 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/
20:20 mutante: attempting parsoid restart on wtp1004
20:17 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/resources/SecurePoll.js
20:05 mutante: enabling test.wikidata virtual host
19:56 logmsgbot: dzahn synchronized docroot/wikidata_test/
19:55 mutante: sync_dir docroot/wikidata_test/ on tin
18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything over to 1.22wmf5
16:50 paravoid: upgrading Ceph to cuttlefish, ignore warnings for the next half/one hour
15:32 andrewbogott: upgraded adminbot on wikitech-static to 1.7.3. Should be more likely to survive netsplits.
10:24 paravoid: starting icinga
03:58 apergos: restarted morebots
June 4
02:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 4 02:36:22 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Tue Jun 4 02:16:08 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Tue Jun 4 02:09:55 UTC 2013
June 3
23:08 mutante: DNS update - point contacts away from broken singer
22:58 logmsgbot: kaldari synchronized php-1.22wmf5/extensions/TimedMediaHandler/resources/mw.MediaWikiPlayerSupport.js 'syncing mw.MediaWikiPlayerSupport.js for TimedMediaHandler bug'
22:26 binasher: replaced redis-server in precise-wikimedia with 2.6.13
21:18 binasher: upgrading php5-redis on all appservers to version bult from github master 2d0f29bdaf29b071aea29b8fb9ee4158c2b69d72
20:28 LeslieCarr: dist-upgrading vanadium
20:28 LeslieCarr: allowing salt out of analytics subnet
19:37 logmsgbot: aaron synchronized php-1.22wmf5/includes/diff/DifferenceEngine.php '22a544be01809226c1069cff90cdc174df81b7e5'
19:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf5
19:16 logmsgbot: aaron synchronized php-1.22wmf5/includes/WebRequest.php '7715ffc9e4216497edc696fa686103dcc9698e9d'
19:15 logmsgbot: aaron synchronized php-1.22wmf4/includes/WebRequest.php '7f31f0029bdc4526b41ad09968e17e10838087a6'
17:56 binasher: dropped old rowiki db from s3, left over from the migration to s7 4+ years ago
17:01 cmjohnson1: ms-be1011 replacing disk at slot5
02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 3 02:07:36 UTC 2013
02:03 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Mon Jun 3 02:02:19 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Mon Jun 3 02:01:35 UTC 2013
June 2
23:01 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll
23:00 logmsgbot: reedy synchronized php-1.22wmf4/extensions/SecurePoll
02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 2 02:07:28 UTC 2013
02:03 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sun Jun 2 02:02:14 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Sun Jun 2 02:01:30 UTC 2013
June 1
15:56 paravoid: staggered reboot of ms-fe1001-4, standard maintenance
02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 1 02:07:32 UTC 2013
02:03 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sat Jun 1 02:02:25 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Sat Jun 1 02:01:38 UTC 2013
May 31
21:34 hashar: Zuul / Jenkins had some delay most probably related to a lot of patchsets being sent at the same time. Zuul catching up right now.
21:15 binasher: completed echo event_page_id schema migrations
20:50 binasher: restarted gerrit
19:53 AaronSchulz: Running a terbium script to copy missing files to swift as well as ceph
19:24 logmsgbot: aaron synchronized php-1.22wmf5/maintenance/copyFileBackend.php '82352634594d0a67d3b9f6597c3bb2c0911f8028'
19:23 logmsgbot: aaron synchronized php-1.22wmf4/maintenance/copyFileBackend.php '3d0e3f8e4d09d4a2462f583c25dd284e6ee3e466'
14:42 logmsgbot: reedy synchronized php-1.22wmf4/extensions/SecurePoll/
14:41 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/
14:34 Reedy: Created bv2013_edit tables on all wikis
14:30 logmsgbot: reedy synchronized php-1.22wmf4/extensions/SecurePoll/
14:29 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/
12:04 apergos: uh... it would help if I looked on tin, the actual deploy host so nm. anyways texvc now working
12:01 apergos: ran scap-recompile to get texvc built for 1.22wmf5 and btw this had to be done from fenari, there's no dsh on terbium
06:57 apergos: ceph osds 50 and 132 on ms-fe1002 are logging 'slow requests', don't know how to restart specific osds in bobtail though
06:43 Aaron|home: Running a script on terbium to list out files that were updated (overwritten) but are not synced on all wikis (it found none on commons/enwiki)
05:26 Tim: on msfe1001-1004: disabled apache_status ganglia module
04:18 Tim: on nickel: restarting gmetad to see if that fixes ceph cluster reporting
04:02 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Disabled ceph backends and switched to just swift.'
02:32 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 31 02:31:40 UTC 2013
02:11 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Fri May 31 02:11:03 UTC 2013
02:06 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Fri May 31 02:06:06 UTC 2013
00:11 mwalker: Updating payments deploy from dce20cbeb501a9945d97b2e64c59dbb4dd0a9620 to cf2d1639a9a8b2854d5302db2aba86199b4e4101 for RTBT attempted fix
00:07 logmsgbot: kaldari Finished syncing Wikimedia installation... :
May 30
23:58 LeslieCarr: fixing packages/security upgrades on db45
23:56 logmsgbot: kaldari Started syncing Wikimedia installation... :
23:55 LeslieCarr: mw1171 appears to have a bad hard drive
22:08 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deployment for E3 and E2
21:55 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deployment for E3 and E2
21:23 LeslieCarr: powercycling mw16
20:35 cmjohnson1: ms-be106 replacing disk slot 6
20:11 cmjohnson1: es1001 replacing bad disk (slot 11)
20:06 logmsgbot: kaldari Finished syncing Wikimedia installation... :
19:55 LeslieCarr: enabling routing-engine on asw-c1-eqiad
19:54 LeslieCarr: asw-c1-eqiad successfully replaced
19:48 logmsgbot: kaldari Started syncing Wikimedia installation... :
19:21 LeslieCarr: powering on new asw-c-eqiad switch -- possible disruption to row c connectivity (low risk)
19:20 LeslieCarr: replacing asw-c1-eqiad with new switch
18:15 bblack: updated vhtcpd deb to 0.0.6-1 on brewster, puppet will upgrade the varnishes
16:19 andrewbogott_afk: updated OpenStackManager on virt0 to git head.
14:59 paravoid: restarting ms-fe4 proxy
14:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable SecurePoll on testwiki'
13:21 RoanKattouw: (it's not broken, never mind, I'm just incapable of reading things)
13:21 RoanKattouw: Synced afe912f7a335ae037b8d8b0ba9616dc55848a2f0 for VisualEditor updates (logmsgbot seems to be broken)
13:20 hashar: gallium: killed stalled Jenkins threads that were producing high I/O.
13:16 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'afe912f7a335ae037b8d8b0ba9616dc55848a2f0'
13:15 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'afe912f7a335ae037b8d8b0ba9616dc55848a2f0'
13:15 logmsgbot: catrope synchronized visualeditor.dblist 'afe912f7a335ae037b8d8b0ba9616dc55848a2f0'
13:11 hashar: gallium: blackholed some web crawler robot.
12:01 Krinkle: Gracefully reloading Zuul to deploy 'Ica1137438a1b9137f01276b4dfe49a82227de518'
10:19 logmsgbot: mark synchronized wmf-config/squid.php 'Add cp1037-1040 as text Varnish'
02:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 30 02:37:00 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Thu May 30 02:14:49 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Thu May 30 02:08:18 UTC 2013
May 29
23:36 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'enwikispecies -> specieswiki'
23:26 AaronSchulz: Tried running some metawiki TTMServerMessageUpdateJob jobs again, still very slow for some jobs (though at least not hours)
23:20 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Enabling new 'VForm' layout for account creation & login interfaces on 31 wikis. (2/2)'
23:19 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling new 'VForm' layout for account creation & login interfaces on 31 wikis. (1/2)'
23:19 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings-labs.php 'Labs: consolidate 'wgUseVFormCreateAccount' & 'wgUseVFormUserLogin' into 'wmgUseVForm
19:53 logmsgbot: yurik synchronized php-1.22wmf5/extensions/ZeroRatedMobileAccess/
19:07 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
18:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.22wmf5
18:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
18:57 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Wikibase
18:56 logmsgbot: reedy synchronized php-1.22wmf4/extensions/Wikibase
18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikinews to 1.22wmf5
18:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wiktionary to 1.22wmf5
18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikiversity to 1.22wmf5
18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, special and wikimedia to 1.22wmf5 (not wikidata)
18:10 logmsgbot: reedy synchronized database lists files: private and wikivoyage to 1.22wmf5
17:47 cmjohnson1: mw1085 powering down to reseat DIMM
17:41 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
17:18 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
16:18 cmjohnson1: replacing failed disk on dataset1001 slot9
15:02 bblack: deployed vhtcpd puppet stuff via sockpuppet...
14:06 cmjohnson1: ms-be4..fixing raid cfg
02:34 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 29 02:33:19 UTC 2013
02:11 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Wed May 29 02:10:48 UTC 2013
02:06 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Wed May 29 02:05:47 UTC 2013
May 28
21:28 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:17 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
18:54 logmsgbot: mlitn synchronized php-1.22wmf4/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
18:53 sbernardin: taking down ms-be1 for server replacement
18:23 logmsgbot: mlitn Finished syncing Wikimedia installation... : Update ArticleFeedbackv5 & Echo to master
18:14 logmsgbot: mlitn Started syncing Wikimedia installation... : Update ArticleFeedbackv5 & Echo to master
17:28 sbernardin: taking down ms-be4 for server replacement
16:46 cmjohnson1: ms-be4 going down for h/w replacement
14:37 RoanKattouw: Gracefulled Apache on gallium, looks like there might have been APC corruption
14:23 notpeter: reimaging db1033 for testing
07:02 apergos: search23 apparently rebooted itself, nothing of use in the logs nor in atop
02:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 28 02:17:19 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Tue May 28 02:11:34 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Tue May 28 02:01:26 UTC 2013
May 27
20:30 logmsgbot: reedy synchronized php-1.22wmf5/extensions/UniversalLanguageSelector 'Revert back to 1.22wmf4 version of ULS'
20:19 apergos: restarted morebots
18:41 reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf5
18:39 reedy Finished syncing Wikimedia installation... : Take 3
18:24 reedy Started syncing Wikimedia installation... : Take 3
18:16 reedy Started syncing Wikimedia installation... : Take 2
18:15 reedy synchronized php-1.22wmf5/cache/
17:30 reedy synchronized wmf-config/squid.php 'Repush'
17:15 reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf5 and build localisation cache
17:13 reedy synchronized php-1.22wmf5/extensions/Wikibase/
17:12 reedy synchronized php-1.22wmf5/extensions/DataValues/
17:11 reedy synchronized php-1.22wmf5/extensions/Diff/
17:04 reedy synchronized docroot
17:04 reedy synchronized w
17:03 reedy synchronized php-1.22wmf5 'Initial file sync out'
02.15 LocalisationUpdate ResourceLoader cache refresh completed at Mon May 27 02:15:05 UTC 2013
02.03 LocalisationUpdate completed (1.22wmf4) at Mon May 27 02:03:50 UTC 2013
May 26
18.41 reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf5
18.39 reedy Finished syncing Wikimedia installation... : Take 3
18.24 reedy Started syncing Wikimedia installation... : Take 3
18.16 reedy Started syncing Wikimedia installation... : Take 2
18.15 reedy synchronized php-1.22wmf5/cache/
17.30 reedy synchronized wmf-config/squid.php 'Repush'
17.15 reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf5 and build localisation cache
17.13 reedy synchronized php-1.22wmf5/extensions/Wikibase/
17.12 reedy synchronized php-1.22wmf5/extensions/DataValues/
17.11 reedy synchronized php-1.22wmf5/extensions/Diff/
17.04 reedy synchronized docroot
17.04 reedy synchronized w
17.03 reedy synchronized php-1.22wmf5 'Initial file sync out'
15:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
14:32 RobH: authdns-update for netmon1001
13:21 logmsgbot: mark synchronized wmf-config/squid.php 'Update cache list'
13:09 logmsgbot: mark synchronized wmf-config/squid.php 'Add amssq47-62'
12:44 Jeff_Green: authdns-update
12:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
12:31 logmsgbot: reedy synchronized wikidata.dblist
12:22 mutante: DNS update - add test.wikidata cname
12:17 ottomata: taking oxygen down for precise upgrade
09:43 LeslieCarr: upgrading some pmtpa mw* machines - may be a few alerts
09:18 logmsgbot: catrope synchronized wmf-config/throttle.php 'Raise account creation throttle for Amsterdam hackathon'
08:49 logmsgbot: tstarling cleared profiling data
06:11 Aaron|home: De-duplicated base refreshLinks2 jobs on enwiki, didn't do much though
02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 26 02:27:21 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sun May 26 02:06:36 UTC 2013
May 25
14:51 Jeff_Green: authdns-update
13:36 mutante: restarting pdns on ns0
13:11 LeslieCarr: rm -rf /home/wikipedia/common/php-1.21wmf12 on fenari
12:46 logmsgbot: reedy synchronized php-1.22wmf4/extensions/CentralAuth
12:43 Jeff_Green: authdns-update
12:07 logmsgbot: reedy synchronized php-1.22wmf4/extensions/CentralAuth
12:04 bblack: cp1029.eqiad : manually testing s/varnishhtcpd/vhtcpd/
10:08 Krinkle: Gracefully reloading Zuul to deploy I8c95150a688973
02:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 25 02:17:03 UTC 2013
02:05 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sat May 25 02:04:49 UTC 2013
May 24
23:38 logmsgbot: aaron synchronized php-1.22wmf4/maintenance '3d47dae94d7c416a1e2d4acc506c5e6a212592af'
23:37 logmsgbot: aaron synchronized php-1.22wmf4/includes/profiler/Profiler.php '3d47dae94d7c416a1e2d4acc506c5e6a212592af'
23:03 logmsgbot: reedy synchronized wmf-config/trusted-xff.cdb
20:39 logmsgbot: aaron synchronized php-1.22wmf4/includes/job/jobs/RefreshLinksJob.php '88ffcabd2b0039d18b805bd1e5d8d350c26588e1'
14:05 mutante: dzahn graceful'ed apaches and api servers , add redirect for be-x-old.wikisource
13:56 RobH: tmh1 back online, tmh2 rebooting for kernel updates
13:43 RobH: tmh1 rebooting for kernel updates
13:01 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'testwiki captcha'
12:47 logmsgbot: reedy synchronized wmf-config/
10:42 Jeff_Green: authdns-update again...
10:28 hashar: jenkins: updating all phplint jobs to get rid of the old ant building script
10:22 Jeff_Green: authdns-update to add fundraisingdb service aliases
10:21 mutante: installing upgrades on fenari
10:07 RobH: tridge upgrading, will require reboot
08:29 K4-713: updated payments cluster to dce20cbeb501a
02:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 24 02:28:20 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Fri May 24 02:07:17 UTC 2013
May 23
21:23 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable echo email bundling on enwiki'
20:31 logmsgbot: spage synchronized php-1.22wmf4/extensions/GettingStarted 'E3 latest GettingStarted'
20:29 logmsgbot: spage synchronized php-1.22wmf4/extensions/GuidedTour 'E3 latest GuidedTour'
19:17 logmsgbot: kaldari synchronized php-1.22wmf4/extensions/Echo/modules 'syncing Echo modules for css updates'
18:54 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to Master
18:48 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to Master
18:31 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Add new eventlogging schema: EchoPrefUpdate'
12:19 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 23 02:25:13 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Thu May 23 02:06:32 UTC 2013
May 22
23:40 logmsgbot: aaron synchronized php-1.22wmf4/maintenance/runJobs.php '2188c14239b74d202b2109933612259193f6fa41'
22:44 logmsgbot: krinkle synchronized php-1.22wmf4/extensions/VisualEditor/modules/ve/ve.EventEmitter.js 'touch, attempt to fix VisualEditor cache snafu on dewiki'
22:33 paravoid: removing users sara & mwang from ldap/ops
19:12 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
19:05 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
18:51 apergos: restarted gerrit
18:42 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Swithced remaining wikis to 1.22wmf4
18:26 logmsgbot: kaldari synchronized php-1.22wmf4/includes/Preferences.php 'syncing preferences.php for bug 48693'
18:02 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
18:00 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
17:46 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
17:41 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
07:50 apergos: restarted swift proxy on ms-fe1
02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 22 02:35:44 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 22 02:14:48 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Wed May 22 02:08:27 UTC 2013
00:21 Ryan_Lane: deployed change 64875 to virt0
May 21
21:11 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Regular weekly MobileFrontend deployment
21:04 logmsgbot: awjrichards Started syncing Wikimedia installation... : Regular weekly MobileFrontend deployment
19:27 logmsgbot: kaldari Finished syncing Wikimedia installation... :
19:22 logmsgbot: kaldari Started syncing Wikimedia installation... :
18:31 mwalker: (previous log referring to fundraising jenkins!) now upgrading fundraising jenkins for auth plugins
18:31 mwalker: restarted jenkins for rekeying of jobs dir; then updated non auth plugs
18:17 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Disable AFTv5 talk page links'
18:16 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Disable AFTv5 talk page links'
17:49 cmjohnson1: powering down db26 for last time
17:45 Jeff_Green: rebooting several fundraising hosts for kernel upgrades
17:14 cmjohnson1: replacing disk10 (L11) on ms-be1009
14:38 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'Disable $wgCentralAuthLoginWiki for testwiki'
13:59 logmsgbot: anomie synchronized wmf-config/CommonSettings.php 'Set $wgCentralAuthLoginWiki for test & test2 (SUL2 testing)'
13:59 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'Set $wgCentralAuthLoginWiki for test & test2 (SUL2 testing)'
13:14 logmsgbot: anomie synchronized php-1.22wmf4/extensions/CentralAuth 'Update CentralAuth to master, for SUL2 testing on testwiki and test2wiki'
13:13 logmsgbot: anomie synchronized php-1.22wmf3/extensions/CentralAuth 'Update CentralAuth to master, for SUL2 testing on testwiki and test2wiki'
09:41 hashar: made Jeroen De Dauw a maintainer of https://packagist.org/packages/mediawiki/data-values
08:10 hashar: zuul: running git gc on all Zuul git repositories: find /var/lib/zuul/git -maxdepth 4 -type d -name .git -print -exec git --git-dir={} gc \;
08:02 hashar: Running git gc on Zuul git repositories (gallium: /var/lib/zuul/git/operations/puppet )
07:18 LeslieCarr: allowed labs access to the new labsdb hosts
02:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 21 02:26:30 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue May 21 02:09:52 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Tue May 21 02:06:41 UTC 2013
01:31 cmjohnson1: cycling titanium to check bios settings
May 20
23:59 mutante: dist-upgrading celsus
23:36 logmsgbot: mflaschen synchronized php-1.22wmf4/extensions/GettingStarted/ '-m Sync GettingStarted for E3 lightning deploy'
22:26 mutante: rebooting Tampa parsoid servers, one by one
22:20 mutante: dist-upgrading, rebooting kuo
22:11 mutante: dist-upgrading, rebooting arsenic
22:04 mutante: dist-upgrading, rebooting strontium
21:48 mutante: dist-upgrading, rebooting palladium
21:31 mutante: dist-upgrading, rebooting niobium
20:58 logmsgbot: krinkle synchronized php-1.22wmf4/extensions/VisualEditor/modules/ve/ve.EventEmitter.js 'touch, fixup cache snafu'
20:51 Ryan_Lane: temporarily disabling nova-api ratelimiting
20:46 akosiaris: added cloudera debian packages hadoop, zookeeper, bigtop-jsvc, bitop-utils to apt.wikimedia.org
20:07 Ryan_Lane: rebooting all pmtpa virt nodes
20:07 Ryan_Lane: shutdown all labs instances
19:12 mutante: dist-upgrading cp1(sq51)
19:09 logmsgbot: csteipp synchronized php-1.22wmf4/includes/ 'Security fix'
19:08 logmsgbot: csteipp synchronized php-1.22wmf3/includes 'Security fix'
19:04 logmsgbot: reedy synchronized w/
19:01 logmsgbot: reedy synchronized wmf-config/wgConf.php
19:00 logmsgbot: reedy synchronized wmf-config/wgConfVHosts.php
18:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable translate on commonswiki'
18:46 Reedy: Created translate tables on commonswiki
18:44 logmsgbot: krinkle synchronized php-1.22wmf4/resources/mediawiki/mediawiki.js 'touched'
18:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:30 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache
18:21 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache
18:15 logmsgbot: reedy synchronized php-1.22wmf4/extensions/Wikibase/
18:14 logmsgbot: reedy synchronized php-1.22wmf4/extensions/ProofreadPage/
18:08 logmsgbot: reedy synchronized wmf-config/ 'Enable TemplateData on all wikis'
18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf4
17:49 MaxSem: Reindexing solr
17:48 mutante: upgrading/rebooting mobile varnish cache cp1041-cp1044
17:35 mutante: kernel upgrade cp1043
17:00 mutante: resurrecting icinga-wm and nsca
16:59 mutante: ping logmsgbot
16:52 mutante: rebooting neon for upgrade
16:51 cmjohnson1: shutting down lanthanum to add ssd rt5074
15:49 ottomata: rebooting gadolinium (source of webrequest multicast stream) to upgrade kernel
14:55 ottomata: rebooting emery to upgrade to 3.2.0-43-generic kernel
14:53 ottomata: upgraded all analytics* machines to linux 3.2.0-43-generic kernel
02:30 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 20 02:29:25 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Mon May 20 02:13:42 UTC 2013
02:05 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Mon May 20 02:05:04 UTC 2013
May 19
02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 19 02:06:55 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sun May 19 02:01:54 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sun May 19 02:01:19 UTC 2013
May 18
02:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 18 02:29:09 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sat May 18 02:11:28 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sat May 18 02:06:30 UTC 2013
01:23 mutante: singer boot issue - affects service contacts.wm (but no others)
00:13 mutante: dist-upgrading yvon
00:09 mutante: dist-upgrading nitrogen
May 17
23:48 mutante: dist-upgrading singer
23:46 mutante: dist-upgrading nickel
23:45 RobH: all tampa based apaches have had kernel upgrades
23:40 mutante: dist-upgrading praseodymium
23:33 RobH: upgrading all srv*.pmtpa.wmnet via dist-upgrade in salt.
23:28 mutante: dist-upgrading gurvin
23:13 RobH: rebooting all pmtpa mw servers
23:13 mutante: dist-upgrading hydrogen, manutius
23:12 RobH: still doing pmtpa mw upgrades, ignore all icinga alarms for now
23:05 mutante: dist-upgrading chromium
23:02 RobH: using salt to dist-upgrade all the tampa apaches... nothing could go wrong....right?
22:58 akosiaris: upgrading and rebooting cp1001-1020 (3 at a time)
22:55 mutante: dist-upgrading capella ( IPv6 tunnel relay)
22:34 mutante: shutting down zinc
22:23 mutante: !!log doesn't log
22:21 Reedy: [23:18:29] <binasher> !!log running hotbackup of db71 to pre-labsdb for s1
22:14 mutante: rebooting spence for upgrades
22:10 logmsgbot: reedy synchronized w
22:08 logmsgbot: reedy synchronized docroot
22:08 mutante: rm -rf php-1.21wmf12 on tin per reedy
22:03 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
21:58 apergos: mw80 memtest failure dimm1, all other image scalers in pmtpa updated
21:53 mutante: dist-upgrading zhen (mobile vumi)
21:49 mutante: DNS update - kill storage1 and 2
21:34 akosiaris: about to reboot fenari
21:13 hashar: gallium Manually blackholed some web engine crawler (via ip route)
19:55 mutante: Wikimedia IRC server working again
19:46 Ryan_Lane: rebooted ssl1/2 ssl3001 ssl1001/1002
19:42 hashar: Jenkins restarted successfully.
19:40 Ryan_Lane: depooling ssl3001 ssl1/2 ssl1001/2
19:40 Ryan_Lane: repooling ssl1003/4 ssl3/4
19:31 Ryan_Lane: repooling ssl3002/3
19:30 hashar: restarted Jenkins
19:29 Ryan_Lane: rebooted ssl3/4 ssl3002/3 ssl1003/1004
19:29 Ryan_Lane: depooled ssl3/4 ssl3002/3 ssl1003/1004
19:19 RobH: blog back up, whew.
19:16 RobH: holmium (blog server) rebooting for kernel upgrade
19:12 cmjohnson1: shutting titanium down to add disks (take 2)
19:04 apergos: eqiad imagescalers done
19:04 mutante: rebooting gallium
19:03 mutante: wikimedia irc is back
19:00 hashar: gallium: stopping jenkins and zuul
18:56 RobH: cerium rebooting for kernel upgrade
18:56 Ryan_Lane: dist-upgrading all ssl boxes
18:55 mutante: dist-upgrading ekrem (IRC)
18:55 cmjohnson1: shutting down titanium for add'l disk
18:50 RobH: i hate you morebots.
18:50 RobH: caesium rebooting
18:47 mutante: dist-upgrading calcium
18:42 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php 'Removing france as a redirect country for fundraising (7649276ddb3f20678de265c37fe75e1d5e642956)'
18:31 hashar: Jenkins: created a Jenkins slave on gallium using jenkins-slave as user and /srv/ssd/jenkins-slave as working directory.
18:17 RobH: carbon rebooting for kernel update, tftp in eqiad will be offline for a few
17:50 apergos: updating and rebooting image scalers one by one
17:46 cmjohnson1: shutting down cerium to add add'l disk
17:39 apergos: rebooted snapshots 1001-4 (new kernel)
17:35 mutante: bugzilla is back
17:31 ^demon: rebooting formey (svn) in 2 minutes...why are you still using SVN?
17:30 mutante: rebooting kaulen
17:24 mutante: rebooting kaulen (Bugzilla) in 5 minutes, please save your bugs and expect 5 minutes downtime
16:49 ^demon: rebooting antimony
15:00 paravoid: setting Varnish send_timeout to 600 on upload, mobile, bits (eqiad), upload, bits (esams) for both frontend & backend
13:31 hashar: Zuul: applying project templates 63674
12:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
12:29 logmsgbot: reedy synchronized php-1.22wmf4/maintenance/checkUsernames.php
05:06 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
02:34 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 17 02:33:21 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Fri May 17 02:12:44 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Fri May 17 02:07:14 UTC 2013
00:56 K4-713: Woohoo! Logs!
00:51 logmsgbot: reedy synchronized php-1.22wmf3/maintenance/checkUsernames.php
May 16
23:48 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/AccountAudit/ 'Updating AccountAudit to master. Adding aa_method'
23:45 logmsgbot: pgehres synchronized php-1.22wmf4/extensions/AccountAudit/ 'Updating AccountAudit to master. Adding aa_method'
23:43 logmsgbot: reedy synchronized database lists files:
23:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:39 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
22:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:29 logmsgbot: mwalker synchronized php-1.22wmf4/extensions/CentralNotice/ 'Updating CentralNotice to master'
22:28 logmsgbot: reedy synchronized wmf-config/
22:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:02 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
22:00 logmsgbot: mflaschen Finished syncing Wikimedia installation... : E3 scap for deployment of GettingStarted and GuidedTour
21:53 logmsgbot: mflaschen Started syncing Wikimedia installation... : E3 scap for deployment of GettingStarted and GuidedTour
21:50 superm401: Scap failed due to wrong ssh configuration
21:49 Ryan_Lane: rebooting virt1005 and virt1007
21:49 logmsgbot: mflaschen Finished syncing Wikimedia installation... : E3 deployment
21:48 logmsgbot: mflaschen Started syncing Wikimedia installation... : E3 deployment
21:47 Ryan_Lane: rebooting virt1000
21:40 superm401: Started scap for E3 deployment of GettingStarted and GuidedTour
21:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
21:29 binasher: running accountaudit_login aa_method schema migrations on all wikis
21:20 Ryan_Lane: rebooting virt0
20:56 Ryan_Lane: running dist-upgrade on virt cluster
20:51 logmsgbot: reedy synchronized wmf-config/
20:40 logmsgbot: reedy synchronized php-1.22wmf4/extensions/Wikibase/
20:35 paravoid: restarting a ceph-osd osd.3 for testing
20:03 logmsgbot: reedy synchronized wmf-config/ 'Updating interwiki cache'
19:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
19:58 logmsgbot: reedy synchronized database lists files:
19:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:51 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
19:49 logmsgbot: reedy synchronized database lists files:
19:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:01 logmsgbot: reedy synchronized wmf-config/
18:56 logmsgbot: reedy synchronized wmf-config/
18:55 logmsgbot: reedy synchronized wmf-config/session.php
18:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'EP on dewikiversity'
18:41 mwalker|0_0: updating payments from fdf308c8fa92ca75adbb3154b8b31a9f7237e885 to head
18:34 gwicke: updated Parsoid to b0229051eb08c8
18:29 mutante: magnesium (racktables): upgrading kernel and rebooting
18:21 mutante: zirconium (planet): upgrading kernel and rebooting
18:17 paravoid: restarting a ceph-osd osd.0 for testing
18:02 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuilding l10n cache for wikidata deploy
17:51 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuilding l10n cache for wikidata deploy
17:27 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
17:13 logmsgbot: reedy synchronized php-1.22wmf4/extensions/
16:53 ottomata: rebooting stat1 for kernel upgrade
16:24 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pooling db1043'
16:18 gwicke: Parsoid update to current master: done
16:16 gwicke: Parsoid update to current master: start
15:57 Krinkle: Graceful reload of Zuul deploying Ic9194c079fa7d46
15:28 Krinkle: Graceful reload of Zuul deploying I7565eff83a7a128e
13:50 Krinkle: Graceful reload of Zuul deploying I0ea73ca1439c8aa6
07:54 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/GeoData/ 'I9d5afff9c328 '
07:47 logmsgbot: maxsem synchronized php-1.22wmf4/extensions/GeoData/ 'I9d5afff9c328 '
07:30 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
06:42 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
02:31 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 16 02:30:28 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Thu May 16 02:11:55 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Thu May 16 02:06:34 UTC 2013
00:37 mutante: puppetstoredconfigclean.rb ocg3.pmtpa.wmnet
00:14 mutante: puppetstoredconfigclean.rb ms2.pmtpa.wmnet
00:06 logmsgbot: reedy synchronized database lists files:
00:05 logmsgbot: reedy synchronized database lists files:
May 15
23:44 mutante: pushing wikimedia-task-appserver 2.9-1
23:32 logmsgbot: reedy synchronized database lists files:
23:30 logmsgbot: reedy synchronized database lists files:
22:30 logmsgbot: krinkle synchronized php-1.22wmf4/resources/mediawiki/mediawiki.js 'touched'
22:01 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
21:54 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
20:57 Ryan_Lane: OpenStackManager is reenabled on virt0. login to wikitech is working again
20:56 Ryan_Lane: openstack in pmtpa now at the folsom release
20:54 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/
20:51 logmsgbot: maxsem synchronized php-1.22wmf4/extensions/MobileFrontend/
20:15 Ryan_Lane: force running puppet on all virt nodes to bring them back up
20:14 Ryan_Lane: upgrading schema for keystone, then glance, then nova
20:13 Ryan_Lane: backing up keystone, nova, and glance databases
20:12 logmsgbot: aaron cleared profiling data
20:12 logmsgbot: aaron cleared profiling data
20:10 Ryan_Lane: upgrading glance, keystone and nova-scheduler on virt0
20:09 Ryan_Lane: upgrading nova services on compute nodes
20:07 logmsgbot: aaron synchronized php-1.22wmf4/includes/specials/SpecialActiveusers.php 'df79a82 '
20:05 Ryan_Lane: disabling OpenStackManager
20:05 Ryan_Lane: stopping all openstack services. it won't be possible to log into wikitech during this process
19:44 paravoid: running refreshFileHeaders.php on terbium
19:25 logmsgbot: aaron cleared profiling data
19:19 Ryan_Lane: adding temporary ubuntu-cloud archive apt changes in a non-puppetized way on pmtpa virt nodes
19:18 Ryan_Lane: disabling puppet on pmtpa virt cluster
19:09 logmsgbot: reedy synchronized wmf-config/ 'touch!'
19:09 logmsgbot: reedy synchronized php-1.22wmf4/resources/ 'touch!'
18:42 Ryan_Lane: cherry-picked change 62100 on virt0 until permanent fix for jquery.chosen minimum width is added
18:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commons back to 1.22wmf4
18:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commons back to 1.22wmf3 due to broken UploadWizard
18:35 ^demon: restarting gerrit
18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: aaaand wikisource to 1.22wmf4
18:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikiquote to 1.22wmf4
18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikibooks to 1.22wmf4
18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage and wiktionary to 1.22wmf4
18:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: fishbowl, special, wikimedia and private to 1.22wmf4
18:01 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/
17:58 RobH: authdns update, killin dataset1 entries
17:56 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
17:50 Ryan_Lane: deploying new project filter on virt0
17:44 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
16:57 RobH: authdns update, cleaning out old decom entries
15:35 Jeff_Green: rebooting boron
11:16 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
11:15 logmsgbot: reedy synchronized wmf-config/squid-labs.php
11:14 logmsgbot: reedy synchronized wmf-config/squid.php
11:13 logmsgbot: reedy synchronized wmf-config/squid-labs.php 'wmf-config/squid.php'
11:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 15 02:34:37 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Wed May 15 02:14:58 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 15 02:08:15 UTC 2013
00:49 mutante: graceful apache on stat1001
00:31 logmsgbot: maxsem synchronized php-1.22wmf4/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/63806/'
00:29 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/63806/'
May 14
23:57 mwalker: lies -- went from d895730067e3325fd630d09196ee81eeba24b5ee back to head on civi because it was working -- the tester was just broken
23:42 mwalker: rolling back civi from deploy head to d895730067e3325fd630d09196ee81eeba24b5ee because thank you isn't working
23:38 logmsgbot: aaron synchronized wmf-config/StartProfiler.php
23:23 logmsgbot: maxsem synchronized php-1.22wmf3/resources/startup.js
22:38 RobH: authdns-update for a bunch of decom updates
22:32 mutante: DNS update - removing owa servers
22:23 logmsgbot: tstarling synchronized wmf-config/db-pmtpa.php
22:17 Ryan_Lane: finished upgrading wikitech
22:10 Reedy: Running sync-common on srv193
22:04 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
22:03 Ryan_Lane: upgrading mediawiki to 1.22wmf4 on wikitech
22:03 mwalker: upgrading fundraising civi (d895730067e3325fd630d09196ee81eeba24b5ee --> head)
21:53 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
21:30 RoanKattouw: Repooled wtp1004, added wtp1005-wtp1024 to the pool
21:01 sbernardin: changing HD in ms-be5
20:52 binasher: s4 now all mariadb, new master info = MASTER_LOG_FILE='db1059-bin.000012', MASTER_LOG_POS=473448023
20:45 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/63596/'
20:44 sbernardin: changing drives on ms-be10
20:39 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/Gadgets/ 'https://gerrit.wikimedia.org/r/#/c/60954/'
20:33 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
20:32 mutante: apply umask 0002 for wikidev users fix on tin and terbium, class generic::wikidev-umask
20:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'setting s4 to writeable on new master'
20:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'setting s4 to read-only for a sec'
20:22 logmsgbot: maxsem synchronized docroot/bits/mobile/W.png
20:20 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
19:53 RoanKattouw: Running salt-call state.highstate on wtp1005-24
19:51 logmsgbot: aaron synchronized php-1.22wmf3/includes/filerepo/file/LocalFile.php 'd58dc596a25607deaecf0aa1c3be877f834da533 '
19:47 RoanKattouw: Accepted salt keys for wtp1005-1024 on sockpuppet
19:22 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to master
19:12 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to master
18:52 mutante: gracefulling apaches with dsh
18:46 Ryan_Lane: running an apache-graceful-all
18:44 ottomata: deployed puppet change to make nginx send webrequest logs back to main multicast firehose. NOTE: These logs are not sent to the udp2log instance on emery.
18:35 sbernardin: replacing bad drive in ms-be11
18:34 logmsgbot: kaldari synchronized php-1.22wmf4/extensions/Echo 'syncing Echo for 1.22wmf4'
18:31 logmsgbot: reedy synchronized wmf-config/
18:30 logmsgbot: aaron synchronized php-1.22wmf3/includes/filerepo/file/LocalFile.php '0cf6b27aa43ad2b65725081ee4d36ccb0f8a34c6'
18:28 logmsgbot: reedy synchronized wmf-config/
18:27 mutante: powercycling frozen mw1173
18:25 logmsgbot: kaldari synchronized php-1.22wmf4/extensions/Echo 'syncing Echo for 1.22wmf4'
18:12 logmsgbot: reedy synchronized wmf-config/
17:56 logmsgbot: reedy synchronized wmf-config/
17:53 logmsgbot: reedy synchronized wmf-config/
17:28 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Enable upload of XML files on Incubator (for MediaWiki dumps)'
17:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Namespace configuration for ur.wikipedia'
17:15 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
17:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
16:18 logmsgbot: reedy synchronized wmf-config/
15:14 Reedy: Created EducationProgram tables on dewikiversity
14:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'thwiki collation'
13:53 cmjohnson1: running dns update
13:53 hashar: Zuul: applying a patch to prevent it from fetching a change multiple time.
10:51 apergos: re-enabling puppet on ssl1001 for a bit of real traffic
10:42 apergos: repooled ssl1
09:58 apergos: puppet temp disabled on all ssl terminators except ssl1 in pmtpa
09:37 apergos: depooled ssl1 in pmtpa for testing
08:51 logmsgbot: tstarling synchronized php-1.22wmf4/LocalSettings.php 'remove testwiki special case'
08:51 logmsgbot: tstarling synchronized php-1.22wmf3/LocalSettings.php 'remove testwiki special case'
08:34 logmsgbot: tstarling Finished syncing Wikimedia installation... :
08:26 logmsgbot: tstarling Started syncing Wikimedia installation... :
08:11 logmsgbot: tstarling Started syncing Wikimedia installation... :
07:44 logmsgbot: tstarling Started syncing Wikimedia installation... :
07:34 logmsgbot: tstarling Started syncing Wikimedia installation... :
07:16 logmsgbot: tstarling synchronized w/MWVersion.php
07:15 logmsgbot: tstarling synchronized refresh-dblist
07:14 logmsgbot: tstarling synchronized multiversion
07:12 logmsgbot: tstarling synchronized docroot/noc/db.php
06:40 logmsgbot: tstarling synchronized README
04:34 Tim: moved scap to tin. Copied source from NFS to tin. Fixed up broken git submodule config using sed.
04:03 logmsgbot: tstarling synchronized cgi-bin
03:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 14 03:09:49 UTC 2013
02:31 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Tue May 14 02:31:15 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue May 14 02:12:10 UTC 2013
02:11 logmsgbot: LocalisationUpdate failed: mwversionsinuse returned empty list
01:13 logmsgbot: tstarling synchronized README
01:00 logmsgbot: tstarling synchronized README
00:20 akosiaris: added myself to root email alias at mchenry
00:17 akosiaris: installing package-upgrades on zirconium
May 13
23:19 logmsgbot: catrope synchronized php-1.22wmf4/extensions/VisualEditor/ 'VE bug fix'
22:46 RobH: authdns-update for wikipedia.is
21:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
21:11 hashar: jenkins: updating all phpcs-HEAD jobs to make them ignore i18n.php files bug 48420 / 63571
21:02 logmsgbot: reedy synchronized wmf-config/
20:34 logmsgbot: aaron cleared profiling data
20:33 cmjohnson1: morebots back
14:40 gallium: deleted an ip route blacklist
02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 13 02:35:49 UTC 2013
02:06 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Mon May 13 02:06:03 UTC 2013
May 12
14:59 Krinkle: Graceful reload of Zuul to deploy Ic22d7c6cb811fd7e3
14:31 jeremyb: imported missing !log entries from channel log
14:23 bblack: restarted adminbot on wikitech-static
14:22 bblack: fixed typo in rt.wikimedia.org vhost hostname, was breaking RT http stuff
14:19 bblack: morebots is broken, someone should fix that when they read this
09:32 logmsgbot: olivneh synchronized wmf-config/throttle.php '(Bug 48301) Add throttle exception for Haifa University workshop'
03:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 12 03:03:03 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sun May 12 02:13:17 UTC 2013
01:28 logmsgbot: aaron synchronized php-1.22wmf3/maintenance/copyFileBackend.php '3e103e93f01c24d7e454aaa5db07418405d273c3'
01:22 logmsgbot: aaron synchronized php-1.22wmf3/includes/filebackend/FileBackendMultiWrite.php '308ebddc237e76a9602b20585386fc26876f0bee'
May 11
13:24 hashar: restarted Jenkins. Got hit by a memory bug again.
03:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 11 03:48:00 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sat May 11 02:16:23 UTC 2013
May 10
23:20 mutante: DNS update - remove various old Tampa hosts
22:49 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/CentralAuth/ 'Small little update to maintenance script'
21:46 mutante: shutdown -h blondel .. bye
21:39 RobH: switch port disabled for blondel & bellin
21:27 RobH: decom of bellen/blondel : setting network ports to disabled
21:27 AaronSchulz: Running listing based copy-sync scripts in terbium for all originals+transcodes on all wikis for swift->ceph to fix any outdated/missing files
21:05 mutante: fixing some more varnishncsa monitoring
20:58 logmsgbot: aaron synchronized php-1.22wmf3/includes/filebackend/FileBackendStore.php '2717d876ba2507df26ac2c5ab746ad7895af1bbf'
20:31 logmsgbot: aaron synchronized php-1.22wmf3/includes/filebackend
20:06 logmsgbot: reedy synchronized docroot/bits/WikipediaMobileFirefoxOS
19:57 logmsgbot: aaron synchronized php-1.22wmf3/maintenance
19:56 logmsgbot: aaron synchronized php-1.22wmf3/includes '6c01414775bb9c5269b391bee7ffd1611f9ff74f & d1940caa1f4a11e7c7d8d2ff5b178da7de8e2a92'
18:50 mutante: killing and restarting nagios-nrpe-server on a bunch of cp10xx boxes to fix monitoring of varnishncsa
18:23 mutante: powercycling srv291
18:17 RobH: authdns-update to remove blog.wikimediafoundation as it wasnt setup and pointed to decom server
17:42 RobH: authdnsupdate - removing racktables2 entry
17:19 hashar: jenkins: cleared all occurrences of an old plugin in build histories. bug 48025 . Jenkins should be fine now. Will monitor tonight.
13:20 mark: Migrated toolserver routing to cr1-esams and cr2-esams via csw2-esams access
12:28 mark: Removed knsq23-knsq30 from the text squid config
12:25 hashar: jenkins: cleaning up all occurrences of an old plugin in build histories. The script run in a tmux on gallium (see https://bugzilla.wikimedia.org/show_bug.cgi?id=48025#c19 )
12:09 mark: Depooled knsq23-knsq30 text squid frontends
11:52 mark: Moved amslvs1 and 2 from csw1-esams to csw2-esams
11:52 mark: Started PyBal on amslvs1
11:51 mark: Stopped PyBal on amslvs1
11:49 mark: Reenabled PyBal on amslvs2
11:44 hashar: restarting jenkins
11:44 mark: Disabled Puppet and stopped PyBal on amslvs2
11:27 mark: Moved ssl3001-3004 from csw1-esams to csw2-esams
10:09 hashar: jenkins: upgrading plugins and restarting.
09:43 hashar: restarted jenkins twice to get it to finally serve something :(
09:27 hashar: killed -9 jenkins :(
09:25 hashar: restarting jenkins
09:18 hashar: gallium: dumping java heap of Jenkins using jmap -dump:format=b,file=/root/jenkins-bug48025.jmap -F 24508
08:34 hashar: Jenkins hit by the evil bug again bug 48025 . Taking as much trace as possible before restarting it
04:46 paravoid: swift: pushing new rings, set weight 0 for ms-be5 sdb1, ms-be11 sdh1 (broken disks); balance 999.99, needs further rebalance
03:38 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 10 03:37:41 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Fri May 10 02:15:49 UTC 2013
00:15 logmsgbot: pgehres synchronized wmf-config 'Actually enabling wgCentralAuthAutoMigrate'
May 9
23:34 logmsgbot: mwalker synchronized wmf-config/InitialiseSettings.php 'Enabling siteNotice div on all alpha and beta mobile sites'
23:27 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/CentralAuth/CentralAuthUser.php 'Moving one more CentralAuth log line'
23:22 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/ 'Updating CentralNotice to master for some bugfixes'
23:07 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/CentralAuth/ 'Updates to CentralAuth'
21:55 hashar: Jenkins / Zuul is back up (for now).
21:45 hashar: restarted Jenkins to enable access logs in /var/log/jenkins/access.log 63068
21:42 hashar: forced Zuul to stop. All events have been dropped :(
21:31 hashar: restarting Zuul to make it query Jenkins directly without SSL + apache proxy 63069 .
21:29 Jeff_Green: pdns-update to cut back to aluminium, restarted pdns on nescio b/c it failed
21:28 ^demon: gerrit repositories will now recursively merge
21:17 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/WikimediaMessages 'Updating WikimediaMessages to master'
20:48 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/GuidedTour 'Actually updating GuidedTour to master'
20:46 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/EventLogging 'Updating GuidedTour to master'
20:46 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/EventLogging 'Updating EventLogging to master'
20:46 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/ConfirmEdit 'Updating ConfirmEdit to master'
20:21 hashar: restarting Jenkins.
20:20 hashar: Jenkins died :/
19:17 mutante: fixing file permissions in /a/e3/E3Analysis repo on stat1001
19:09 mutante: restarted Apache and then puppetmaster on stafford, that revived it
18:56 mutante: attempting to restart broken puppetmaster on stafford
18:27 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php 'Enabling CentralAuth debug log'
18:00 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php 'Enabling wgCentralAuthAutoMigrate on all wikis that use CentralAuth'
15:56 cmjohnson1: powering down barium /relocate server to c1
15:27 logmsgbot: anomie synchronized wmf-config/CommonSettings.php 'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653'
15:27 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653'
15:26 logmsgbot: anomie synchronized wmf-config/InitialiseSettings-labs.php 'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653'
11:25 hashar: upgrading Zuul to latest master (makes report times friendlier)
08:54 hashar: Jenkins apparently working fine. I have upgraded a few plugins, that seems to have fixed the Gerrit jobs.
08:22 hashar: restarting Jenkins again
08:21 hashar: upgrading Jenkins plugins 'promoted build' and 'M2 release'. That should fix the Gerrit jobs disappearing (ping qchris)
07:39 hashar: restarting jenkins. All its http thread are locked :/
07:25 hashar: Jenkins HTTP interface died :/
03:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 9 03:39:55 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Thu May 9 02:15:54 UTC 2013
May 8
22:27 RobH: authdns update to migrate racktables from hooper to magnesium. both servers are already using db1001 as master, which is why hooper is slow to load racktables data
22:24 logmsgbot: mholmquist synchronized php-1.22wmf3/extensions/UploadWizard/UploadWizardHooks.php 'Fixing Special:Preferences for wikis with UploadWizard'
21:45 logmsgbot: mholmquist Finished syncing Wikimedia installation... : Fix UploadWizard concurrency and minor message updates
21:37 logmsgbot: mholmquist Started syncing Wikimedia installation... : Fix UploadWizard concurrency and minor message updates
21:07 Krinkle: Running small maintenance script foreachwiki to gather data about odd wgRC2UDPPrefix values (bug 28276)
20:21 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/ 'CentralNotice security bugfix 48255'
20:15 Jeff_Green: authdns-update to swap in grosley for dead aluminium
19:41 RobH: magnesium is future racktables host, disregard errors for now
19:41 cmjohnson1: accidently removed power from magnesium
19:00 Reedy: Ensuring securepoll database tables exist on all wikis
18:35 logmsgbot: reedy synchronized php-1.22wmf3/extensions/CentralAuth
18:31 Jeff_Green: updating and rebooting grosley
18:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:17 logmsgbot: reedy synchronized wmf-config/
18:16 logmsgbot: reedy synchronized database lists files:
18:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable EducationProgram on mkwiki'
18:07 Reedy: Created EducationProgram tables on mkwiki
18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.22wmf3
18:00 RobH: authdnsupdate for racktables2
17:46 apergos: )
17:46 apergos: about to graceful apaches *and api servers* (which somehow are not in the actual apache graceful all script
17:42 Jeff_Green: kernel updates and reboot on pappas
17:29 logmsgbot: yurik Finished syncing Wikimedia installation... : Zero extension
17:18 RobH: restarting all mw jobrunners
17:13 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Disabled TTMServerMessageUpdateJob jobs'
17:12 logmsgbot: aaron synchronized php-1.22wmf3/includes/job/JobQueueGroup.php 'revert hack'
17:11 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/JobQueueGroup.php 'revert hack'
17:08 logmsgbot: yurik Started syncing Wikimedia installation... : Zero extension
15:01 logmsgbot: mark synchronized wmf-config/mc-eqiad.php
14:22 paravoid: ceph osd out 39, primary for stuck unclean pg 3.5c6
14:21 paravoid: ceph osd out 128, primary for stuck unclean pg 3.19e1
13:59 logmsgbot: mark synchronized wmf-config/mc-eqiad.php
13:59 mark: Removing mc1009 from the eqiad memcached pool
06:14 apergos: restarted adminbot (morebot) on wikitech-static
May 7
23:42 logmsgbot: catrope synchronized php-1.22wmf3/resources/startup.js 'touch'
23:42 logmsgbot: catrope synchronized php-1.22wmf3/extensions/VisualEditor/modules/ve/ve.js 'touch'
23:39 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
23:35 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/ 'Appling update to fix CentralNotice message editing bug'
23:30 mutante: running addWiki.php for iegcomwiki
23:26 logmsgbot: dzahn synchronized wmf-config/InitialiseSettings.php
23:23 mutante: depool mw1041 in pybal
23:22 mutante: removed mw1041 from dsh groups mediawiki-installation and apaches
23:20 mutante: mw1041 - RT-4517, RT-5065, RT-5105 ...and broken again :p
23:17 logmsgbot: mflaschen synchronized php-1.22wmf3/extensions/GettingStarted/ 'E3 Lightning deploy for GettingStarted'
23:17 LeslieCarr: upgrading srv* machines
22:48 Tim: on marmontel: disabled logrotate
22:36 logmsgbot: aaron synchronized wmf-config/mc-pmtpa.php 'removing mc15 for hw maint'
22:34 mutante: upgrading PHP_CodeSniffer on gallium
22:31 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/62744/'
22:10 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/62650/'
22:04 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
22:04 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/
21:47 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
21:09 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/JobQueueGroup.php
20:53 logmsgbot: aaron synchronized php-1.22wmf3/includes/job/JobQueueGroup.php 'Live hack to avoid translationNotificationJob jobs.'
20:44 logmsgbot: reedy synchronized database lists files:
20:30 andrewbogott: copied passwords from stafford to sockpuppet, then reset --hard on stafford to get things back in sync.
20:17 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
20:11 logmsgbot: reedy synchronized docroot/login/
20:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:02 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
19:57 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: loginwiki
19:56 logmsgbot: reedy synchronized wmf-config/
19:55 logmsgbot: reedy synchronized database lists files: loginwiki
19:37 mutante: DNS update - switch old secure.wm
19:30 mutante: apache-graceful-all, move old secure.wm redirects
18:54 cmjohnson1: authdns update
18:42 logmsgbot: dzahn synchronized docroot
18:42 mutante: sync-docroot for login.wm
18:38 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1042,1059 to s4'
18:03 logmsgbot: awjrichards synchronized php-1.22wmf2/extensions/MobileFrontend/stylesheets/common/ui.css 'Cherry-pick e37e9d9ae6b8a581569c39ede38fc21827983128'
17:59 logmsgbot: awjrichards synchronized php-1.22wmf3/extensions/MobileFrontend/stylesheets/common/ui.css 'Cherry-pick e37e9d9ae6b8a581569c39ede38fc21827983128'
17:43 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/FundraiserLandingPage/ 'Updating FundraiserLandingPage for WMF fundraising in FR'
17:38 kaldari: killed copyExistingEmailPreference.php on enwiki
17:29 Reedy: Killed php-1.21wmf1 through 1.21wmf5 on fenari and mediawiki-installation hosts
17:26 logmsgbot: mwalker Finished syncing Wikimedia installation... : Update CentralNotice and core for HTMLForms, iframe preview, and devices!
17:19 mutante: killed hanging pdns related processes on ns1 and restarted pdns
17:15 kaldari: begin running extensions/Echo/maintenance/copyExistingEmailPreference.php on enwiki db
17:02 mutante: DNS update - add login.wikimedia and login.m.wikimedia
16:57 logmsgbot: mwalker Started syncing Wikimedia installation... : Update CentralNotice and core for HTMLForms, iframe preview, and devices!
16:37 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
16:35 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
16:31 RobH: professor being worked on, network is offline =(
16:29 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
16:19 gwicke: updated Parsoid to current master
16:03 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
15:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:46 logmsgbot: reedy synchronized docroot
14:47 cmjohnson1: authdns update
14:33 cmjohnson1: authdns update
13:09 mark: Power cycled line card 2 in csw1-esams (which has the SSL servers) to fix esams SSL reachability issues
09:15 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'New ULS config part 2'
09:14 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'New ULS config part 1'
09:09 logmsgbot: nikerabbit synchronized php-1.22wmf3/extensions/UniversalLanguageSelector/ 'ULS to master'
08:21 paravoid: ceph: all radosgw & mon + osd.24 upgraded to 0.56.6
07:59 paravoid: partially upgrading ceph to 0.56.6
07:27 Tim: on holmium: reverted firewall, too hard
07:22 Tim: on holmium: added some ad hoc iptables rules intended to keep the rest of the site mostly safe from it overnight
07:00 Tim: on holmium: chown'd to root all script files writable by www-data and all directories which are configured to allow PHP execution
06:28 Tim: on marmontel: disabled apache logrotate, stopped apache, disabled apache init script
04:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 7 04:19:39 UTC 2013
02:25 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue May 7 02:25:11 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue May 7 02:16:06 UTC 2013
01:29 logmsgbot: bsitu synchronized php-1.22wmf3/includes/User.php 'Two fixes to User::getNewMessageRevisionId() from kaldari'
01:29 logmsgbot: reedy synchronized php-1.22wmf3/extensions/WikimediaMaintenance
00:38 logmsgbot: reedy synchronized php-1.22wmf3/includes/User.php 'Live hack to stop fatals'
00:07 logmsgbot: bsitu synchronized php-1.22wmf3/extensions/Echo 'Update Echo to master'
00:07 logmsgbot: bsitu synchronized php-1.22wmf3/docs/hooks.txt 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
00:06 logmsgbot: bsitu synchronized php-1.22wmf3/RELEASE-NOTES-1.22 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
00:06 logmsgbot: bsitu synchronized php-1.22wmf3/includes/WikiPage.php 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
00:03 logmsgbot: bsitu synchronized php-1.22wmf3/includes/User.php 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
00:03 logmsgbot: bsitu synchronized php-1.22wmf3/includes/Skin.php 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
00:02 logmsgbot: bsitu synchronized php-1.22wmf3/includes/OutputPage.php 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
May 6
23:25 mutante: restarting prelabsdbdb on db1054
22:48 mutante: apache-graceful-all, add iegcom virtual host
21:00 mutante: blog is back up
21:00 mutante: holmium: bring up interface lo, comment "search" line in resolv.conf, restart varnish, ...
20:46 paravoid: swift replication scripts running on ms-fe1002 under screen -- kill if they have any effect on production
20:15 mutante: DNS update - sync with ns2
20:04 mutante: restarting parsoid on wtp1001
20:03 mutante: restarting pdns on ns0
20:01 mutante: killing parsoid server.js on wtp1001
19:59 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'f257ddac39194f8bfcfd145104999fdba83d33f5'
19:59 mutante: DNS update - switch all blog cnames over to holmium
19:30 logmsgbot: reedy synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
19:25 LeslieCarr: moved gateway from csw1-esams to cr1-esams
19:18 Krinkle: Reloading Zuul to deploy I259585e78ee
19:18 LeslieCarr: rebooting ns2
19:16 logmsgbot: reedy synchronized wmf-config/
19:13 binasher: rebuilt db1016 from hotbackup of db1001; pmtpa side of m1 needs to be rebuilt as well
18:42 RobH: ns2/nescio appears offline, investigating
18:37 RobH: ns2 appears offline
18:37 RobH: authdns-update crashed on ns2, rerunning
18:33 RobH: holmium
18:33 RobH: authdns-update for new server homium
18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf3
17:18 LeslieCarr: marmontel's interface is now in the sandbox vlan
17:16 LeslieCarr: turned off marmontel's network port
16:12 logmsgbot: reedy synchronized docroot
13:55 hashar: gallium: installed manually libcidr0-dev libanon0-dev . Puppet change is 62434
09:51 paravoid: upgrading Ganglia Web to 3.5.7
08:25 hashar: Zuul upgrading to upstream version 7191ee8 + 1 wmf hack.
06:45 Nemo_bis: Translate search on vanadium keeps being broken https://toolserver.org/~nemobis/tmp/SearchTranslations.log (now RT-5085)
06:15 paravoid: pushing new swift rings: ms-be1 weight 50->0, ms-be2 33->75
02:19 paravoid: restarting gerrit
02:18 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
May 5
10:56 Nemo_bis: "Search service is temporarily unavailable" since unknown time https://meta.wikimedia.org/wiki/Special:SearchTranslations?query=amen&uselang=en
03:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 5 03:36:17 UTC 2013
02:26 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sun May 5 02:25:44 UTC 2013
02:18 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sun May 5 02:17:22 UTC 2013
May 4
22:42 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
21:44 Reedy: Manually force ran TorBlock/loadExitNodes.php
21:43 logmsgbot: reedy synchronized php-1.22wmf3/extensions/TorBlock/
16:03 logmsgbot: reedy synchronized wmf-config/
16:02 logmsgbot: reedy synchronized docroot/bits/WikipediaMobileFirefoxOS/
04:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 4 04:36:09 UTC 2013
02:21 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sat May 4 02:20:30 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sat May 4 02:12:01 UTC 2013
01:17 Krinkle: binasher restarted gerrit
May 3
23:49 Krinkle: Gerrit is completely unresponsive (all http and ssh time out, serving HTTP 500). Filed details at https://bugzilla.wikimedia.org/show_bug.cgi?id=48061
22:19 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'adding new pmtpa slave'
21:10 cmjohnson1: db10 is decommissioned...powering down for the final time
20:04 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/MobileFrontend.hooks.php
19:54 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/includes/MobileFrontend.hooks.php
07:52 MaxSem: OMG logmsgbot is a terminator!
06:16 logmsgbot: tstarling synchronized wmf-config/missing.php
04:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 3 04:03:12 UTC 2013
02:26 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Fri May 3 02:26:02 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Fri May 3 02:15:45 UTC 2013
01:25 hashar: Zuul processed all its events, seems to be fine now. *sleep*
00:33 logmsgbot: dzahn synchronized docroot
00:32 mutante: sync-docroot for iegcom wiki
00:25 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php '6512451451ac6617bedbfce7be17045bed794d70'
00:16 mutante: DNS update - add iegcom
May 2
23:20 mutante: restarting lucene on search1021, search1022, search1017, search1018 (with some waiting in between)
23:18 hashar: yeah after hours and hours of fighting, Jenkins is finally working again.
23:11 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php
22:59 hashar: restarting jenkins… it got a few threads blocked and the main process is at 100% usage for now reason
22:59 mutante: repooling search1016
22:40 mutante: gallium, run puppet, graceful Apache to deploy split log files
22:34 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1020 for upgrade'
22:32 mutante: depooling search1016, restarting lucene, etc.. (Search#Adding_new_wikis)
22:29 mutante: repooling search1015
22:23 hashar: Zuul restarted. The bug about slowness is bug 48025
22:15 hashar: restarting Zuul
22:13 hashar: Deploying a workaround on Zuul to make it stop querying the Jenkins API when it just want to check whether a job exist. 62095
22:04 mutante: depooling search1015 in pybal
21:43 logmsgbot: mflaschen synchronized php-1.22wmf3/extensions/GuidedTour/ 'Sync GuidedTour to 1.22wmf3 for E3 deployment'
21:42 logmsgbot: mflaschen synchronized php-1.22wmf2/extensions/GuidedTour/ 'Sync GuidedTour to 1.22wmf2 for E3 deployment'
21:38 mutante: importing wikimania2014wiki into search indexers
21:35 mutante: restarting search indexers on searchidx2, searchidx1001 to make sure the indexer knows about new wiki
21:28 logmsgbot: mflaschen synchronized php-1.22wmf3/skins/common/shared.css 'Sync font-size change for edit section links'
21:15 logmsgbot: spage synchronized php-1.22wmf2/extensions/ConfirmEdit 'update 1.22wmf2 to wmf3 version of ConfirmEdit'
21:14 LeslieCarr: adding GEANT via fiberring to avoid-paths
20:21 logmsgbot: reedy synchronized php-1.22wmf3/includes/
20:15 logmsgbot: reedy synchronized docroot/bits/WikipediaMobileFirefoxOS/
19:51 Jeff_Green: authdns-update to move db1025 to frack.eqiad.wmnet
19:47 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
19:40 ottomata: rebooting oxygen
19:33 ottomata: re-enabling puppet on cp1031, it was administratively disabled. running puppet there.
19:24 RobH: professor was already powered down (why?) starting it back up now
19:23 ottomata: all webrequest udp2log loggers (squid and varnish) now send to gadolinium for socat unicast -> multicast relay
19:22 RobH: rebooting professor
19:16 Jeff_Green: moving db1025 into frack-fundraising1-c-eqiad
19:14 logmsgbot: kaldari synchronized php-1.22wmf2/extensions/Echo 'sync Echo ext for en.wiki'
19:14 ottomata: deployed squid frontend.conf.php changes to remove locke and send logs directly to gadolinium for multicast relay
19:11 logmsgbot: kaldari synchronized php-1.22wmf3/extensions/Echo 'sync Echo ext'
19:09 logmsgbot: kaldari synchronized php-1.22wmf3/extensions/Echo 'sync Echo ext'
18:52 ottomata: removed varnishcsa-locke instance from varnish hosts: (dsh -c -g varnishncsa-all 'test -f /etc/init.d/varnishncsa-locke && service varnishncsa-locke stop && update-rc.d -f varnishncsa-locke remove && rm -v /etc/init.d/varnishncsa-locke')
18:47 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
18:41 ottomata: varnishncsa now sends traffic to gadolinium instead of oxygen for multicast relay
18:30 Krinkle: Installed Monitoring plugin from Jenkins control panel <https://wiki.jenkins-ci.org/display/JENKINS/Monitoring >
17:59 Krinkle: Jenkins restart complete. No visible improvement. Jenkins is still idling most of the time while Zuul is still halted by an unknown factor on spawning jobs.
17:56 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
17:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:48 logmsgbot: reedy synchronized wmf-config/
17:37 Krinkle: Jenkins keeps clogging up. Starting an emergency restart.
17:00 Krinkle: Jenkins is nearing 100% CPU on gallium, what is Jenkins doing?
16:59 Krinkle: Zuul is somehow having trouble kicking off Jenkins jobs (less than 1 event processed per minute). Jenkins shows that 10/10 executors are idle. Investigating...
16:52 logmsgbot: nikerabbit synchronized php-1.22wmf3/extensions/UniversalLanguageSelector/ 'ULS to master'
15:13 hashar: Jenkins made Jenkins to instantly restart (was bug 47120 ) ) by deleting the downstream-buildview plugin.
15:11 hashar: restarted Zuul
14:12 hashar: Upgraded Jenkins from 1.480.3 to LTS 1.509.1. Restarted it.
14:10 hashar: shutting down Zuul
14:09 hashar: upgrading Jenkins (unscheduled maintenance).
13:01 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/61941/'
12:59 mark: Migrated amslvs3 and amslvs4 PyBal BGP peerings from csw2-esams to cr2-knams
09:55 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/'
09:53 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/'
09:50 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/'
09:40 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/GeoData/GeoData.body.php 'https://gerrit.wikimedia.org/r/#/c/61962/'
06:04 apergos: powercycle snapshot1001, upgrade to precise
05:46 apergos: powercycle snapshot1004, upgrade to precise
05:28 apergos: snapshot1003 powercucle, upgrading to precise
05:28 Tim: on professor: manually started carbon-cache.py
05:15 Tim: professor is down, no response on serial console, rebooting
05:11 logmsgbot: tstarling synchronized wmf-config/jobqueue-pmtpa.php
05:08 logmsgbot: tstarling synchronized wmf-config/jobqueue-eqiad.php
04:47 Tim: on rdb1001-1002, set requirepass and masterauth in soft state
04:44 Tim: on mc1-16 and mc1001-1016, setting requirepass and masterauth to the new password in soft state
04:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 2 04:39:48 UTC 2013
04:04 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
03:59 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
03:57 logmsgbot: tstarling synchronized wmf-config/jobqueue-eqiad.php
03:47 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/GettingStarted
03:40 logmsgbot: tstarling synchronized php-1.22wmf3/extensions/GettingStarted
02:46 logmsgbot: kaldari Finished syncing Wikimedia installation... :
02:32 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Thu May 2 02:32:10 UTC 2013
02:22 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Thu May 2 02:21:30 UTC 2013
02:21 Tim: on wikibugs-l: disabled bounce processing and re-enabled mail delivery to wikibugs-irc (was disabled due to excessive bounces)
02:21 logmsgbot: kaldari Started syncing Wikimedia installation... :
01:20 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js 'Poking bits to try and get the new banner controller deployed for CentralNotice'
01:17 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js 'Poking bits to try and get the new banner controller deployed for CentralNotice'
00:16 mutante: @stafford:~# puppetstoredconfigclean.rb db10.pmtpa.wmnet
May 1
23:28 logmsgbot: tstarling synchronized php-1.22wmf3/extensions/GettingStarted
23:23 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/GettingStarted
23:18 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/GettingStarted
22:42 mutante: powercycling frozen mw1041
22:14 logmsgbot: mwalker Finished syncing Wikimedia installation... : Updating CentralNotice from a8797938bc for historical allocations and some refactoring
22:06 logmsgbot: dzahn synchronized ./multiversion/MWMultiVersion.php
22:02 logmsgbot: dzahn synchronized docroot
21:44 logmsgbot: mwalker Started syncing Wikimedia installation... : Updating CentralNotice from a8797938bc for historical allocations and some refactoring
21:32 logmsgbot: dzahn gracefulled all apaches
21:25 ottomata: syncing new redirects.conf to apachces to redirect wikipediazero.org
21:14 Jeff_Green: stopping mysql on db78 for cloning
20:50 ottomata: adding new wikipediazero.org zone alias, running authdns-update on dobson
19:45 logmsgbot: reedy synchronized php-1.22wmf3/cache/l10n/
19:15 mutante: DNS update - remove project2, anthony and yongle :p
19:05 logmsgbot: reedy Started syncing Wikimedia installation... : rebuild l10n cache for wikidata
18:36 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 1 18:36:02 UTC 2013
18:32 mutante: DNS update - add wikimania2014 entries
18:31 RobH: disabled owa1-3 switchports, starting remote wipe
18:21 RobH: owa1/2/3 offlinging and wiping
18:20 logmsgbot: reedy synchronized php-1.22wmf3/extensions/ 'Update extensions for wikibase deploy'
18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wiktionary to 1.22wmf3
18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource, wikivoyage and wikiversity to 1.22wmf3
18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikiquote to 1.22wmf3
18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special and wikimedia to 1.22wmf3
18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, private and fishbowl to 1.22wmf3
17:58 logmsgbot: kaldari synchronized php-1.22wmf3/extensions/Echo/Hooks.php 'syncing Hooks in Echo'
17:33 logmsgbot: kaldari synchronized php-1.22wmf2/extensions/Echo/Hooks.php 'syncing Hooks in Echo'
17:20 LeslieCarr: puppet working on bits caches again after gerrit change 61793
17:01 RobH: torrus compiling, manutilus apache offline
16:53 RobH: torrus deadlocked, fixin
15:54 mark: Converted link between cr2-knams:xe-1/0/0 and csw2-esams:xe-0/1/1 to LACP aggregated link cr2-knams:ae1 / csw2-esams:ae2
14:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
14:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
10:30 logmsgbot: reedy synchronized php-1.22wmf2/extensions/ParserFunctions/ 'Update to master to fix common fatals'
10:08 mark: Removed old Toolserver IP range 91.198.174.192/27 on primary and backup router interfaces
03:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 1 03:49:55 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 1 02:27:30 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed May 1 02:16:40 UTC 2013
02:09 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Turned autoResync back on again for multiwrite backend'
00:46 logmsgbot: tstarling synchronized wmf-config/PrivateSettings.php
April 30
23:52 Krinkle: Gracefully restarted Zuul on gallium
23:52 Krinkle: Ryan_Lane restarted the gerrit service
23:44 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Syncing CommonSettings for Echo'
23:34 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Syncing InitialiseSettings for Echo'
23:28 logmsgbot: kaldari synchronized /php-1.22wmf3/extensions/Echo 'Updating Echo for some minor bugfixes'
23:18 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/ 'Revert MobileFrontend'
23:15 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/ 'Revert MobileFrontend'
23:12 logmsgbot: tstarling synchronized wmf-config/missing.php 'rewrite (I17989fc4)'
23:01 Krinkle: Graceful restart of Zuul on gallium
22:56 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php 'Enabling Extension:AccountAudit on all wikis and wmgEchoHelpPage'
22:09 Jeff_Green: rolling back db1025 change b/c "mysqld: unknown option '--federated'"
22:06 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/ 'grrr'
22:00 logmsgbot: maxsem synchronized wmf-config
21:58 Jeff_Green: restarting mysqld on db1025 to enable federated engine for testing
21:56 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/ 'Weekly mobile deployment'
21:40 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/ 'Weekly mobile deployment'
20:04 logmsgbot: kaldari synchronized echowikis.dblist 'syncing echowikis.dblist for cron jobs'
19:59 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning on Echo for English Wikipedia'
19:49 logmsgbot: kaldari Finished syncing Wikimedia installation... :
19:19 logmsgbot: kaldari Started syncing Wikimedia installation... :
19:03 cmjohnson1: removing db10 certs from puppet
17:48 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Syncing CommonSettings for Echo deployment'
17:45 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Syncing InitialiseSettings for Echo deployment'
13:57 cmjohnson1: jenkins seems to not be working -- restarting
11:27 Nikerabbit: refresh-translatable-pages.php finished for mediawikiwiki and metawiki
04:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 30 04:06:11 UTC 2013
02:34 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue Apr 30 02:33:50 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue Apr 30 02:15:03 UTC 2013
02:04 paravoid: cleaning up neon's /var/log (100% full), restarting icinga...
00:59 logmsgbot: catrope synchronized php-1.22wmf3/extensions/VisualEditor/ 'Update VE'
00:44 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php 'Touching InitialiseSettings'
00:32 logmsgbot: pgehres Finished syncing Wikimedia installation... : Deploying Extension:AccountAudit and an Echo thing
00:00 logmsgbot: pgehres Started syncing Wikimedia installation... : Deploying Extension:AccountAudit and an Echo thing
April 29
23:20 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'updated s5 pmtpa topology'
23:09 binasher: s5 pmtpa master changed to db73 - MASTER_LOG_FILE='db73-bin.000090', MASTER_LOG_POS=978211000
23:08 AaronSchulz: finished creating AccountAudit tables
23:06 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'Re-enable WikiLove, bug fix is deployed'
23:04 logmsgbot: catrope Finished syncing Wikimedia installation... :
22:40 logmsgbot: catrope Started syncing Wikimedia installation... :
22:28 anomie: (Including backport of gerrit change 61494 in the scap)
22:27 RoanKattouw: Running scap
22:23 RoanKattouw: Running scap again
22:19 logmsgbot: catrope Started syncing Wikimedia installation... :
22:13 hashar: gallium: restarted apache2 to kill off http connections established by mod_proxy to the jenkins backend.
22:03 hashar: jenkins web server is no more responding on port 8080. Might have to restart it.
22:00 hashar: gallium / jenkins : blackholed a web drawer using ip route add blackhole <IP>/32
21:54 hashar: Jenkins is in trouble
21:52 logmsgbot: reedy synchronized php-1.22wmf3/includes/DefaultSettings.php 'Fix wgVersion'
21:31 notpeter: upgrading db45 to precise
21:24 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enabling VisualEditor on viwiki and test2wiki'
20:31 AaronSchulz: Doing another original/math/timeline sync/copy run
20:28 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Enabled multiwrites for ceph/swift'
19:19 logmsgbot: kaldari synchronized php-1.22wmf2/extensions/Echo/maintenance/setEmailOptionTemp.php
19:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf3
19:06 logmsgbot: reedy Finished syncing Wikimedia installation... : Scap again to see if that fixes ROOTPAGENAME and ROOTPAGENAMEE. Otherwise I'm reverting it
18:46 logmsgbot: reedy Started syncing Wikimedia installation... : Scap again to see if that fixes ROOTPAGENAME and ROOTPAGENAMEE. Otherwise I'm reverting it
18:37 logmsgbot: reedy synchronized php-1.22wmf2/.git/modules/extensions/WikiLove
18:21 logmsgbot: reedy synchronized php-1.22wmf3/extensions/
18:06 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf3 and rebuild l10n cache
18:03 notpeter: deploying new frontend squid conf
17:51 LeslieCarr: restarting icinga-wm
17:35 cmjohnson1: removing asw-c1 for replacement
17:30 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf3 and rebuild l10n cache
17:09 mutante: fix NRPE checks on ms-be2, running as "weird user id" bug
17:05 mutante: removing reinstalled ms-be2 from ssh_known_hosts with ssh-keygen and then making it world-readable again manually so deployers don't get errors :p
17:05 logmsgbot: csteipp synchronized php-1.22wmf2/includes/
17:05 logmsgbot: reedy synchronized live-1.5
17:02 mutante: attempt to fix NRPE checks on ms-be9, restart as correct user
16:49 logmsgbot: reedy synchronized docroot
16:44 logmsgbot: reedy synchronized php-1.22wmf3/ 'initial sync of 1.22wmf3 files'
03:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 29 03:25:35 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Mon Apr 29 02:12:33 UTC 2013
April 28
13:14 hashar: restarting Zuul, it has a finished build stuck in the `postmerge` pipeline.
02:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 28 02:48:51 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sun Apr 28 02:12:10 UTC 2013
April 27
03:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 27 03:06:47 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sat Apr 27 02:13:53 UTC 2013
00:08 mutante: make SSL on Bugzilla also use SSLCACertificatePath and c_rehash certs on kaulen
April 26
23:58 mutante: upgrading mysql packages on marmontel (blog)
23:53 mutante: make SSL on the Blog use SSLCACertificatePath and c_rehash certs on marmontel
20:11 hashar: killed a node process on gallium. Probably a daemon that has gone standalone when jenkins died
19:19 hashar: jenkins died again sorry :( But this time I most probably found out the reason why it is slowed down: a web spider attempt to crawl all our builds!
18:04 mwalker: Upgraded fundraising civicrm install from a3db312836b6ac004694f831071b1c7d00560fbf to a65fbcec0dbb5bb79a97aca220a998e56d08b067
15:56 hashar: jenkins: enabled ExtraSettings.php loading 61022 . That enables a bunch of MediaWiki settings for all mediawiki related jobs, namely displaying errors, moving the temp dir to tmpfs, showing exceptions details and enabling development warnings.
13:49 mark: Rebooting hooft
13:48 mark: dist upgrade of hooft to precise
13:48 mark: Moved udpmcast from hooft to nescio
13:16 ^demon: gerrit replication freaked out, restarting
06:38 hashar: Restarted Jenkins. It was stuck in some while loop doing tons of stat() on unexisting file :( Would be up in roughly 40 minutes :(
03:46 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 26 03:45:44 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Fri Apr 26 02:12:38 UTC 2013
00:57 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'temporarily disabling WikiLove per bug 47457'
00:53 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor/ 'Update VisualEditor to master'
00:33 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Clean up VisualEditor config'
00:33 Tim: on fenari: fixed permissions on /etc/ssh/ssh_known_hosts
00:33 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Remove wmgVisualEditorHide'
April 25
23:09 logmsgbot: mwalker synchronized docroot/bits/favicon/ 'Pushing new favicons for incubator, wiktionary, wiktionary.en, wikibooks, wikinews, and wikispecies'
22:32 logmsgbot: mflaschen Finished syncing Wikimedia installation... :
22:22 ^demon: new ssl cert (gerrit.wikimedia.org) live on gerrit.
20:55 notpeter: reimaging db55
20:45 logmsgbot: kaldari Finished syncing Wikimedia installation... :
20:17 logmsgbot: kaldari Started syncing Wikimedia installation... :
20:07 awight: payments cluster updated from 7f70ccd580d to fdf308c8f
19:12 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'updating for new schema number'
18:52 logmsgbot: reedy synchronized docroot
18:19 LeslieCarr: cleaning up old home directories on fenari
17:00 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Removing oldskin notice'
16:50 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable VisualEditor on 14 more Wikipedias'
16:38 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor/ 'Update VisualEditor to master'
16:20 hashar: Jenkins is busy reparsing all the build history for some reason. That slow down the continuous integration box by an order of magnitude :/ Should take roughly 40 minutes (no idea when it started)
16:09 hashar: Zuul git operations are slow again :/
14:57 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php 'rm debug hack'
14:56 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Turn mwsearch debug log back off, its not phps fault'
14:53 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Turn mwsearch debug log on'
14:52 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php 'Debugging'
11:56 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'I2f0990fa087a'
11:56 logmsgbot: krinkle synchronized docroot
03:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 25 03:39:44 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Thu Apr 25 02:14:20 UTC 2013
00:45 mutante: upgrading packages on manutius
00:30 mutante: DNS update - removing internproxy, removing tesla (minus mgmt)
April 24
23:04 logmsgbot: dzahn gracefulled all apaches
22:39 logmsgbot: dzahn gracefulled all apaches
22:39 LeslieCarr: upgrading mw's
22:38 LeslieCarr: replaced smokeping targets for sdtpa c3 and sdtpa d3
22:26 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'New location for icons - Ib5554c54729'
22:26 mutante: sync-apache, add coffee
22:24 logmsgbot: krinkle synchronized docroot/bits/apple-touch/ 'New location for icons - Ib5554c54729'
21:47 mutante: DNS update - remove survey, remove wikitech-old, remove argon
21:47 logmsgbot: demon synchronized php-1.22wmf2/extensions/Wikibase 'b4b5fb4046175ca29834a4669c253c97ac5ccd07'
21:38 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
21:34 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Turn mwsearch log back on'
21:33 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php
21:26 mutante: DNS update - readding voip
21:17 cmjohnson1: removing sq33 from dsh node groups and pybal cfg
20:32 RobH: racktables on hooper upgraded to recent stable
20:26 RobH: racktables flap again, working on it
20:16 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch
20:04 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch 'Deploying I48480cb6'
20:03 logmsgbot: demon synchronized php-1.22wmf2/includes/api/ApiQuerySearch.php 'Deploying Ib7abe8d5'
19:57 logmsgbot: aaron synchronized php-1.22wmf2/includes/filerepo/file/LocalFile.php 'deployed 0e42bdadd9349fe5b6119ff65322e6c52d813a6e'
19:17 mutante: deploying Bugzilla wikimedia extension changes for improved wikilinking
19:16 RobH: racktables going offline a moment for upgrade
18:45 logmsgbot: reedy synchronized wmf-config/ 'wikidata stuffs'
18:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non enwiki to 1.22wmf2
18:09 mutante: DNS update - removing unused remnants
18:09 sbernardin: replacing bad disk on db59
17:31 sbernardin: taking down ms-be2 for hw replacement
15:09 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/skins/MinervaTemplate.php 'bug 47597'
15:05 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/skins/MinervaTemplate.php 'bug 47597'
09:05 hashar: restarted Zuul. Upgraded to e403579 (latest master)
08:58 hashar: upgrading Zuul
06:04 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
03:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 24 03:42:12 UTC 2013
02:32 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 24 02:31:43 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed Apr 24 02:18:59 UTC 2013
00:03 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Disable mobile sitenotice everywhere but testwiki'
00:03 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disable mobile sitenotice everywhere but testwiki'
April 23
23:38 mutante: cleaning up all unused Apache sites on singer, gzip, move to /root for backup
23:33 mutante: disabling old techblog apache site config on singer
23:32 logmsgbot: maxsem synchronized wmf-config/mobile.php
23:15 logmsgbot: mwalker synchronized docroot/bits/favicon/ 'Pushing wikidata, wikimania, wikiquote, and wikiversity favicons from Isarra'
23:12 logmsgbot: dzahn gracefulled all apaches
23:07 logmsgbot: maxsem synchronized wmf-config 'Mobile banners again, hopefully fixed'
23:05 mutante: sync-apache, add wikepdia.org and .com typo domains
22:28 logmsgbot: reedy synchronized php-1.22wmf1/extensions/Score
22:21 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score
22:12 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score/Score.body.php 'debugging'
22:05 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score/Score.body.php 'debugging'
22:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'rv debugging'
22:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Score'
21:59 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score/Score.body.php 'debugging'
21:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable score debug log'
21:40 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/60512/'
21:23 logmsgbot: maxsem synchronized wmf-config
21:17 logmsgbot: reedy synchronized php-1.22wmf1/extensions/Score
21:15 logmsgbot: maxsem synchronized wmf-config 'Mobile banners everywhere - https://gerrit.wikimedia.org/r/#/c/60362/'
21:14 logmsgbot: aaron synchronized php-1.22wmf2/includes/db/Database.php 'temp logging'
21:13 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/ 'Weekly mobile deployment'
21:12 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/ZeroRatedMobileAccess/ 'Weekly mobile deployment'
21:06 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/ZeroRatedMobileAccess/ 'Weekly mobile deployment'
21:05 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/ 'Weekly mobile deployment'
20:56 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Wikibase
19:36 logmsgbot: anomie synchronized php-1.22wmf2/extensions/Cite 'Update to master for bug 47291'
18:44 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Re-enable AFTv5 on enwiki'
17:49 logmsgbot: mlitn synchronized php-1.22wmf2/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
17:49 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
16:34 logmsgbot: reedy synchronized php-1.22wmf2/extensions/TorBlock
16:34 logmsgbot: reedy synchronized php-1.22wmf1/extensions/TorBlock
12:34 paravoid: reprepro python-statsd 2.0.1-1~wmf1 & voluptuous 0.6.1-1~wmf1 for precise-wikimedia
09:57 pp-pdf1: restarted all services
09:56 pp-pdf1: - alow numprocs to be configured via initenv.sh
09:16 pp-pdf1: restarted all services
09:12 pp-pdf1: restarted all services
09:10 pp-pdf1: restarted all services
08:46 logmsgbot: maxsem synchronized docroot/ 'https://gerrit.wikimedia.org/r/#/c/60375/'
08:41 logmsgbot: maxsem synchronized wmf-config 'Beta changes, no-op for production'
02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 23 02:25:25 UTC 2013
02:11 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 23 02:10:32 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue Apr 23 02:06:54 UTC 2013
00:02 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'bookkeeping for pmtpa slaves (this time for real)'
00:01 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'bookkeeping for pmtpa slaves'
April 22
23:51 notpeter: reimaging db44
22:57 binasher: mysql query logs / ishmael migrated from db9 to db1001
21:41 LeslieCarr: upgrading even more mw servers in tampa
20:12 Reedy: Running namespaceDupes on kowikiversity
19:57 Reedy: Running namespaceDupes on zhwikibooks
19:55 Reedy: Running namespaceDupes on uzwiki
19:55 logmsgbot: reedy synchronized wmf-config/
19:50 logmsgbot: reedy synchronized images/
19:50 logmsgbot: reedy synchronized docroot
19:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Score extension on the cluster'
18:49 logmsgbot: catrope synchronized echowikis.dblist
18:44 LeslieCarr: sq33 is going to be decomissioned… long live…. nothing else in tampa
18:39 mutante: changing #mediawiki to #wikimedia-dev in wikibugs startup script (just in case, should not even be needed by puppetized ircecho)
18:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:26 LeslieCarr: still doing package upgrades on mw* machines
18:20 LeslieCarr: rebooting nonresponsive sq33
18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf2
17:37 LeslieCarr: dist-upgrading some mw machines, should be low/no impact
16:51 mutante: restarting wikibugs on mchenry
16:43 logmsgbot: mark synchronized wmf-config/InitialiseSettings.php
16:37 mutante: added abaso to wmf-deployment group on gerrit (new mediawiki deployer)
16:11 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'New caching everywhere'
16:02 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Wikibase
15:57 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php '$wgMFVaryResources on enwiki, https://gerrit.wikimedia.org/r/#/c/60286/'
14:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 22 02:14:58 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 22 02:08:56 UTC 2013
02:06 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Mon Apr 22 02:05:30 UTC 2013
00:13 Reedy: Running namespaceDupes.php --fix via foreachwiki in screen session on terbium
April 21
02:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 21 02:37:21 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sun Apr 21 02:16:56 UTC 2013
02:11 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sun Apr 21 02:10:46 UTC 2013
April 20
16:54 ottomata: resetarting apache2 on stafford in attempt to fix puppetmaster
02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 20 02:35:55 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sat Apr 20 02:14:31 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sat Apr 20 02:09:07 UTC 2013
00:09 logmsgbot: aaron synchronized php-1.22wmf1/includes/job/JobQueueRedis.php 'deployed c9e702e7f73f41f7bcd2deb045085689d5cb4229'
00:03 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/JobQueueRedis.php 'deployed d2e4fdfed03148fdd1a564925ce351bec6f9b935'
April 19
23:07 Coren: brief reboot of labstore3 -> test to see that everything starts up as expected.
20:39 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Setup redis job queue debug log'
18:36 Jeff_Green: increased php memory limit for fundraising civicrm instance
17:55 LeslieCarr: restarting etherpad on hooper
17:15 binasher: running enwiki.abuse_filter_log migration - drop afl_user index, added user_timestamp
16:10 LeslieCarr: flushed celsus varnish cache
06:31 Tim: deploying apache conf change for www.wikidata.org redirect (I7bb872fd)
03:33 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 19 03:32:40 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Fri Apr 19 02:27:17 UTC 2013
02:26 logmsgbot: aaron cleared profiling data
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Fri Apr 19 02:16:42 UTC 2013
01:54 logmsgbot: aaron cleared profiling data
01:54 logmsgbot: aaron synchronized wmf-config/StartProfiler.php 'Removed cli profiling for now.'
00:29 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor 'Update VisualEditor to master'
00:29 logmsgbot: catrope synchronized php-1.22wmf1/extensions/VisualEditor 'Update VisualEditor to master'
April 18
23:40 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor 'Update VisualEditor to master'
23:40 logmsgbot: catrope synchronized php-1.22wmf1/extensions/VisualEditor 'Update VisualEditor to master'
23:38 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Deploy 5c5455c81bc8dee0ae2a0410655da639e2f84e87 ($wgVisualEditorParsoidProblemReportURL)'
23:25 logmsgbot: bsitu synchronized php-1.22wmf2/extensions/Echo/processEchoEmailBatch.php 'More echo db fix'
23:25 logmsgbot: bsitu synchronized php-1.22wmf2/extensions/Echo/includes/DbEchoBackend.php 'More echo db fix'
22:51 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Enable mobile sitenotice on testwiki'
22:51 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enable mobile sitenotice on testwiki'
22:48 notpeter: reimaging db35
22:46 logmsgbot: mwalker synchronized docroot/bits/favicon/wikisource.ico 'New and shiney ws.o favicon by Isarra'
22:45 logmsgbot: mwalker synchronized docroot/bits/favicon/mediawiki.ico 'New and shiney mw.o favicon by Isarra'
22:35 logmsgbot: reedy synchronized php-1.22wmf2/extensions/EducationProgram
22:35 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'adjusting weights on pmtpa slaves'
22:26 logmsgbot: reedy synchronized php-1.22wmf1/extensions/EducationProgram
22:15 logmsgbot: reedy synchronized php-1.22wmf1/extensions/EducationProgram
22:09 notpeter: correction: that was db33 out, not db31
22:03 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'db31 and db35 out, db72 and db73 in'
21:56 logmsgbot: bsitu synchronized php-1.22wmf2/extensions/Echo/includes/EchoDbFactory.php 'Db cache fix'
21:41 logmsgbot: reedy synchronized php-1.22wmf2/extensions/EducationProgram
21:39 logmsgbot: reedy synchronized php-1.22wmf1/extensions/EducationProgram
21:38 mutante: merging changes on bugzilla to now support RT in "See also" field
20:45 logmsgbot: mflaschen synchronized php-1.22wmf2/extensions/GettingStarted/ 'Sync GettingStarted 1.22wmf2 for E3 deploy'
20:43 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GettingStarted/ 'Sync GettingStarted for E3 deploy'
20:36 logmsgbot: reedy synchronized wmf-config/
20:15 logmsgbot: kaldari Finished syncing Wikimedia installation... :
19:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgMaxImageArea and wgMaxAnimatedGifArea to 50MP'
19:52 logmsgbot: kaldari Started syncing Wikimedia installation... :
19:14 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Echo email config + db cluster update to testwiki'
19:13 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Echo email config + db cluster update to testwiki'
18:57 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/jobs 'deployed 1e3eaed6fbe7fd91c15399cc17eac8602b34c9cd'
18:28 logmsgbot: reedy Finished syncing Wikimedia installation... : rebuild l10n cache for wikidata deploy
17:57 logmsgbot: reedy Started syncing Wikimedia installation... : rebuild l10n cache for wikidata deploy
17:50 RobH: authdns0update for git.w.o
17:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.22wmf2
17:06 logmsgbot: reedy synchronized php-1.22wmf2/extensions/
16:58 logmsgbot: reedy synchronized php-1.22wmf1/extensions/Wikibase '5d496b4ea2c8a68b9e1280a282530f7ce5232b68'
16:47 logmsgbot: ariel synchronized wmf-config/throttle.php
16:10 logmsgbot: mlitn synchronized php-1.22wmf2/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
16:10 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
15:39 cmjohnson1: authdns update
15:10 mark: Reversed replication of volume fr_archive, now nas1001-a:fr_archive -> nas1-a:fr_archive
14:26 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/59818/'
14:24 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/59818/'
14:14 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
14:06 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/59818/'
14:04 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/59818/'
13:24 logmsgbot: maxsem synchronized wmf-config/mobile.php
13:03 mark: Rebooting cp1041
08:48 logmsgbot: maxsem synchronized wmf-config/mobile.php
07:57 logmsgbot: olivneh synchronized php-1.22wmf2/extensions/ConfirmEdit 'Updating extension/ConfirmEdit (Bug 46132 / Change I8ee3fd136)'
07:57 logmsgbot: olivneh synchronized php-1.22wmf1/extensions/ConfirmEdit 'Updating extension/ConfirmEdit (Bug 46132 / Change I8ee3fd136)'
07:43 hashar: jenkins raised max # of executors from 6 to 8
07:07 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/59767/'
05:59 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:21 logmsgbot: tstarling synchronized php-1.22wmf1/includes/specials/SpecialSearch.php
05:15 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php
05:12 logmsgbot: tstarling synchronized php-1.22wmf1/extensions/MWSearch/MWSearch_body.php
05:09 logmsgbot: tstarling synchronized php-1.22wmf1/includes/specials/SpecialSearch.php
03:57 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/MWSearch
03:55 logmsgbot: tstarling synchronized php-1.22wmf2/includes/HttpFunctions.php
03:48 logmsgbot: tstarling synchronized php-1.22wmf1/extensions/MWSearch
03:46 logmsgbot: tstarling synchronized php-1.22wmf1/includes/HttpFunctions.php
03:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 18 03:26:48 UTC 2013
02:24 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 18 02:23:54 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Thu Apr 18 02:14:26 UTC 2013
01:03 mutante: re-enabling Tomasz' RT account
00:42 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'setting db1004 to full weight'
00:37 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1004 at a warmup weight'
00:09 logmsgbot: demon synchronized docroot
April 17
23:39 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1004'
23:21 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/ContributionReporting 'Finished update of CN and CR to latest'
23:20 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/CentralNotice/
23:14 hashar: If jenkins/zuul goes wild overnight: connect to gallium, killoff puppet in crontab then /etc/init.d/zuul stop and drop me an email. Will take of it whenever I wake up.
23:12 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/ContributionReporting/
23:12 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralNotice 'Beginning update of CentralNotice and ContributionTracking to latest'
22:59 hashar: restarting Zuul
22:20 hashar: Jenkins repointing all jobs to use the new Zuul repo ( /srv/ssd/zuul/git ) 59744
22:06 hashar: jenkins restarted. Restarting Zuul.
21:20 hashar: stopped Zuul. Will wait a bit till jenkins start up
21:19 hashar: restarted Jenkins
21:19 hashar: restarted Zuul
21:16 hashar: finished migrating zuul git repositories ( rsync -av /var/lib/zuul/git /srv/ssd/zuul )
21:09 hashar: Zuul ETA: it is copying MobileFrontend extension right now so half way done
21:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionaries to 1.22wmf2
21:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiquote to 1.22wmf2
21:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikiversity to 1.22wmf2
21:00 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki back to 1.22wmf1
20:59 hashar: stopping Jenkins that somehow restarted
20:59 logmsgbot: reedy synchronized php-1.22wmf2/includes/db/DatabaseMysql.php 'moar'
20:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:55 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews.dblist and special wikis to 1.22wmf2
20:51 logmsgbot: reedy synchronized php-1.22wmf2/includes/db/DatabaseMysql.php 'debugging'
20:49 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage, wikimania and wikimedia to 1.22wmf2
20:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Some wikis to create some noise
20:42 logmsgbot: reedy synchronized php-1.22wmf2/includes/db/DatabaseMysql.php 'debugging'
20:30 hashar: Switching zuul dir to /srv/ssd/zuul ( 58898 and 58899 )
20:18 hashar: stopping Zuul gracefully
20:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Revert that due to large number of warnings...
20:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: fishbowl, private and special to 1.22wmf2
19:57 hashar: gallium slowness is partly due to the ssd install. The git repositories are on the slow raid where as the workspaces are on the ssd. That means `git clone` can't do hardlink anymore :(
19:43 binasher: enwiki, dewiki, wikidata are fully on mariadb 5.5.30
19:33 hashar: jenkins: reducing number of executors from 10 to 6.
19:32 hashar: jenkins gallium has a huge queue of jobs. investigating.
19:31 hashar: jenkins / gallium has a huge queue of jobs. Investigating.
19:19 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 's1/s5 writeable from pmtpa'
19:17 logmsgbot: asher synchronized wmf-config/db-eqiad.php 's1/5 writeable'
19:13 logmsgbot: asher synchronized wmf-config/db-eqiad.php 's1/5 read-only'
19:12 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'preparing for master swaps'
19:03 logmsgbot: aaron synchronized php-1.22wmf2/maintenance/doMaintenance.php 'deployed 1437b57bd475e5ecb06800f9f64b14ccffd0dedf'
19:00 logmsgbot: aaron synchronized php-1.22wmf2/maintenance/doMaintenance.php 'deployed 1306792ad24be5b8d17355c58da655eb88c2fcc6'
18:27 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1050'
18:18 LeslieCarr: installing new linecard in cr2-eqiad , .1% chance of issues
18:16 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1050'
17:50 logmsgbot: aaron synchronized php-1.22wmf2/includes/User.php 'deployed 6a66776fd7afdf883ad6f82270453fc76833508b'
16:20 ^demon: running puppet on manganese (will restart gerrit)
13:05 apergos: restarted icinga on neon (the usual: Could not create external command file '/var/lib/nagios/rw/nagios.cmd' as named pipe)
13:03 hashar: jenkins: refreshing job mediawiki-core-phpcs-HEAD (points to a wrong git repo path)
09:50 mark: Inserted libvpx 1.1.0-1+wmf1 packages into the precise-wikimedia APT repository
06:57 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 17 06:57:02 UTC 2013
06:51 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed Apr 17 06:51:28 UTC 2013
06:48 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 17 06:47:37 UTC 2013
03:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 17 03:13:56 UTC 2013
02:20 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed Apr 17 02:19:57 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 17 02:10:22 UTC 2013
00:38 mutante: blog: enabled new eventlogging by ori, removed old logging class, converted plugins/WMBlog from svn to git and sync with gerrit
00:31 Tim: on hume: changed UID for l10nupdate to be 10002, for consistency with puppet admins.pp
April 16
23:58 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'raising db1058 weight'
23:57 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1005, adding db1058 to s5'
23:55 logmsgbot: aaron cleared profiling data
23:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:48 logmsgbot: reedy synchronized wmf-config/abusefilter.php
23:46 logmsgbot: aaron synchronized wmf-config/StartProfiler.php 'Enabled 1:1 profiling for cli scripts and put "cli" in the profile ID'
23:22 logmsgbot: awjrichards synchronized php-1.22wmf2/extensions/MobileFrontend/
23:22 logmsgbot: awjrichards synchronized php-1.22wmf1/extensions/MobileFrontend/
23:00 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/ContributionReporting/ 'Contribution reporting now pulls 2013 data'
22:59 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/ContributionReporting/
22:51 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 16 22:51:08 UTC 2013
22:37 Tim: running l10nupdate manually on fenari to test change 58911
22:03 logmsgbot: anomie synchronized php-1.22wmf2/extensions/WikimediaMaintenance 'Update WikimediaMaintenance to master for l10nupdate fix'
22:02 logmsgbot: anomie synchronized php-1.22wmf1/extensions/WikimediaMaintenance 'Update WikimediaMaintenance to master for l10nupdate fix'
21:49 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/59449'
21:46 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/ZeroRatedMobileAccess/ 'https://gerrit.wikimedia.org/r/#/c/59525/'
21:43 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:13 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
19:54 logmsgbot: pgehres synchronized php-1.22wmf2/extensions/CentralAuth/maintenance/migrateAccount.php 'Updating CentralAuth/migrateAccount.php'
19:53 logmsgbot: pgehres synchronized php-1.22wmf1/extensions/CentralAuth/maintenance/migrateAccount.php 'Updating CentralAuth/migrateAccount.php'
19:38 logmsgbot: pgehres synchronized php-1.22wmf2/extensions/CentralAuth/maintenance/migrateAccount.php 'Updating CentralAuth/migrateAccount.php'
19:33 logmsgbot: pgehres synchronized php-1.22wmf1/extensions/CentralAuth/maintenance/migrateAccount.php 'Updating CentralAuth/migrateAccount.php'
19:18 binasher: upgrading db1005 to precise + mariadb
19:07 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1005 from s5 for upgrade'
19:02 binasher: streaming s5 hotbackup from db45 to db1058
18:49 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1058 from s1'
18:34 logmsgbot: aaron synchronized php-1.22wmf2/includes/cache/HTMLCacheUpdate.php 'deployed 344ddf85b6d7b79c40a1b6b9ec7e0a1e05c0705b'
18:26 logmsgbot: mlitn synchronized php-1.22wmf2/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
18:26 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
18:17 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Turning on footer contact link for en.wiki'
18:13 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable email bundling for test2wiki and mediawikiwiki'
18:13 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Enable email bundling for test2wiki and mediawikiwiki'
16:42 logmsgbot: aaron synchronized php-1.22wmf2/includes/WikiPage.php 'deployed 38d60cdafe036625a6ec8fa3a9e8a94ef450c85c'
16:41 logmsgbot: aaron synchronized php-1.22wmf2/includes/db/Database.php 'deployed fa819f8e3579855f6d95552655de98f92ec1789c'
16:40 logmsgbot: aaron synchronized php-1.22wmf2/includes/cache/BacklinkCache.php 'deployed 1a94124385df4c452c68fadaa74bc2714a204761'
16:37 logmsgbot: aaron synchronized php-1.22wmf2/includes/OutputPage.php 'deployed 997ab92458946d10fed96ff4ba41c74eeaee0d20'
05:04 Tim: moving most cron jobs from hume to terbium
03:23 legoktm: test
02:35 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue Apr 16 02:34:59 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 16 02:16:58 UTC 2013
April 15
23:57 logmsgbot: csteipp synchronized php-1.22wmf2/includes/ 'User gerrit versions of security fixes'
23:18 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/CentralNotice
23:13 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralNotice/ 'CentralNotice banner= fix'
22:31 logmsgbot: dzahn gracefulled all apaches
22:16 andrewbogott: Moved labs-morebots to tools project.
21:43 LeslieCarr: doing a few more random host upgrades of misc hosts
21:32 LeslieCarr: upgrading mw1070-mw1079
21:24 LeslieCarr: upgrading mw1060-mw1069
21:22 RobH: linode wikitech is now shut down \o/
21:13 logmsgbot: dzahn gracefulled all apaches
20:32 mutante: restarting pdns on ns1
20:30 mutante: DNS update - add wikpedia.org
19:37 logmsgbot: ariel gracefulled all apaches
18:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf2
15:54 logmsgbot: reedy Finished syncing Wikimedia installation... : Build localisation cache for 1.22wmf2
15:19 logmsgbot: reedy Started syncing Wikimedia installation... : Build localisation cache for 1.22wmf2
14:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.22wmf2, scap to come
14:56 logmsgbot: reedy synchronized docroot
14:55 logmsgbot: reedy synchronized w
14:54 logmsgbot: reedy synchronized wmf-config/
14:49 logmsgbot: reedy synchronized php-1.22wmf2/ 'Initial sync of 1.22wmf2'
12:24 mark: Depooled amssq47-amssq62 frontend squids
12:04 mark: Depooled knsq16-knsq22 frontend squids
09:20 hashar: jenkins died :(
08:22 hashar: updating Jenkins jobs to fetch ZUUL_COMMIT instead of ZUUL_BRANCH. That would solve a race condition between jobs ( bug 46723 / 58865 ).
05:50 logmsgbot: olivneh synchronized wmf-config/throttle.php 'Adding throttling exception for itwiki GLAM event which starts in four hours (Bug 47221, change Id9ae600ac)'
02:04 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 15 02:03:51 UTC 2013
April 14
02:04 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sun Apr 14 02:04:17 UTC 2013
April 13
23:54 Coren: rebooting labstore3. autofs is being silly
15:11 Coren: Fixed boot problem on labstore3; back to operating.
04:54 Coren: rebooted labstore3 again; kernel and lvm2 upgrade
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sat Apr 13 02:12:27 UTC 2013
01:10 Coren: rebooted labstore3 again -> we hates lockd. We hates it!
00:41 Coren: rebooting labstore3 -> silly lockd module
April 12
23:31 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Add 'Contact Wikipedia' footer link on test/test2 (Iab1f5f527)'
23:31 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Add 'Contact Wikipedia' footer link on test/test2 (Iab1f5f527)'
23:14 Coren: rebooted labstore3 -> hung MegaCLI processes
22:41 notpeter: temp depooling search1015
20:53 mutante: install package upgrades on sanger
19:57 Ryan_Lane: finished opendj upgrade on virt0
19:23 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Disable mwsearch log'
18:54 mutante: restarting parsoid on wtp1004
18:25 logmsgbot: aaron synchronized wmf-config/jobqueue-pmtpa.php 'Set pmtpa queue config to match eqiad'
18:17 logmsgbot: demon synchronized php-1.22wmf1/extensions/MWSearch/MWSearch_body.php 'Moar logging'
18:16 logmsgbot: demon synchronized php-1.21wmf12/extensions/MWSearch/MWSearch_body.php 'Moar logging'
18:00 Ryan_Lane: upgrade opendj on virt0
16:00 LeslieCarr: adding another link to asw-c-eqiad's uplink
15:05 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Debug group for mwsearch'
15:02 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Debug group for mwsearch'
15:01 hashar: Changing Jenkins workspace directories from ${ITEM_ROOTDIR}/workspace to /srv/ssd/jenkins/workspace/${ITEM_FULLNAME} . My change has not been taken in account early on :(
14:54 hashar: jenkins is back up. Starting Zuul
14:44 cmjohnson1: analytics1007 has never worked..but working on it
14:21 hashar: restarting jenkins
14:03 hashar: gallium: created primary partition on /dev/sdb . formatting using: mkfs.xfs /dev/sdb1
13:36 hashar: gallium can't find the SSD, shut downing, will attempt another slot
12:59 hashar: gracefully shut-downing Jenkins and Zuul for scheduled maintenance. Will shutdown server gallium just after.
09:39 hashar: jenkins: updated mediawiki-core-whitespaces job to use ZUUL_COMMIT as a refspec specifier (for bug 46723 )
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Fri Apr 12 02:15:30 UTC 2013
00:08 mutante: DNS update - point old bugzilla3 entry over to actual bugzilla server
April 11
23:35 LeslieCarr: upgrading mw1052 to mw1060
23:30 logmsgbot: csteipp synchronized php-1.22wmf1/extensions/RSS/
23:30 LeslieCarr: upgrading mw1051
23:24 logmsgbot: csteipp synchronized php-1.22wmf1/includes/
23:22 mutante: shutting down xenon
23:20 LeslieCarr: upgrading mw1050
23:11 LeslieCarr: upgrading 1040 to 1049
23:04 mutante: install package upgrades on neon
23:00 LeslieCarr: upgrading mw1030 - 1039
22:51 K4-713: synchronized payments to 7f70ccd580d64
22:47 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralAuth 'Peter's log line change to CentralAuth'
22:41 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php 'Enable translate on CentralNotice'
22:39 logmsgbot: mwalker synchronized docroot/bits/favicon/wikipedia.ico 'Bi resolution wiki favicon'
22:38 LeslieCarr: upgrading mw1120 to mw1129
22:37 mutante: removing spence from dsh group mediawiki-installation
22:33 logmsgbot: mwalker synchronized php-1.21wmf12/extensions/CentralNotice
22:26 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralNotice 'Updating CentralNotice to the latest and greatest (from 68f0262be838320ecc9c87eb4485c28b0699bc1f)'
22:22 LeslieCarr: upgrading mw1110 to mw1119
21:41 logmsgbot: kaldari synchronized php-1.22wmf1/extensions/Echo 'syncing Echo for bugfix'
21:35 ottomata: emery back up and upgraded to precise (thanks mutante!)
21:32 ^demon: fenari: php-1.21wmf12 and php-1.22wmf1 both rebased to remove silly merge commits.
21:14 mutante: emery - revoking old puppet cert, resiging, initial puppet run
21:04 logmsgbot: olivneh synchronized php-1.22wmf1/extensions/ArticleFeedback/ArticleFeedback.hooks.php 'Remove jquery.clickTracking from AFT dependencies'
21:01 logmsgbot: olivneh synchronized php-1.22wmf1/extensions/GettingStarted/SpecialGettingStarted.php
20:53 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GettingStarted/ 'Update GettingStarted extension for E3 deployment'
20:34 logmsgbot: mflaschen synchronized wmf-config 'Config change for GettingStarted'
20:32 logmsgbot: kaldari Finished syncing Wikimedia installation... :
20:11 logmsgbot: kaldari Started syncing Wikimedia installation... :
19:58 mutante: emery - upgrading from lucid to precise
19:46 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'syncing InitialiseSettings to turn off VectorSectionEditLinks on testwiki'
19:41 Jeff_Green: running authdns-update to propagate frdata cname
19:31 AaronSchulz: Refreshed the job queue aggregator hash table
19:31 mutante: running puppet on brewster, testing lucid image on emery, rebooting again
19:17 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Turn Echo back on for testwiki'
19:16 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Add checkDelay to Echo email bundle jobs'
19:07 AaronSchulz: All jobs finished copying into redis
19:07 sbernardin: taking down ms-be9 for hw replacement
19:02 LeslieCarr: uninstalling nagios from spence
18:51 mutante: PXE booting emery into installer
18:48 ottomata: emery going down for upgrade to precise
18:21 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Turn Echo off on testwiki'
17:35 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Moved remaining jobs to redis'
17:24 sbernardin: taking down db11 for decommissioning
17:19 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 11 17:19:14 UTC 2013
17:15 cmjohnson1: europium power disconnected during routine data center work
13:14 logmsgbot: reedy synchronized wmf-config/ 'Disable ClickTracking :D'
13:08 mark: Inserted python-extras packages into the precise-wikimedia APT repository (backport by hashar)
10:10 hashar: Jenkins is back up
09:23 hashar: restarting Jenkins to get the git verbose mode :(
08:44 hashar: jenkins: stop git cloning mw/core over and over by reusing existing clone 58502 . That should help with I/O starvation on gallium.
08:40 Nikerabbit: translation memory database rebuild finished
08:17 Nikerabbit: emptying Translate solr instance and starting rebuilds
08:15 logmsgbot: nikerabbit synchronized php-1.22wmf1/extensions/Translate/ 'to master'
07:11 logmsgbot: hashar synchronized docroot/noc/conf/highlight.php 'Revert highlight.php recent modifications bug 47112 58664 '
07:02 hashar: Put a live hack on fenari for bug 47112
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 11 02:16:35 UTC 2013
01:10 Ryan_Lane: disabling puppet on labstore3/4 - making sure gluster won't die without them and we're going to rebuild them
April 10
23:31 LeslieCarr: deactivated XO ipv6 transit
22:25 Ryan_Lane: added live hack to OpenStackManager on wikitech, ensuring that DynamicSidebar support is only enabled for logged-in users
22:24 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'moved refreshLinks to redis.'
22:20 ^demon|sick: restarting gerrit, was hung
22:15 Ryan_Lane: enable DynamicSidebar support in OpenStackManager
22:14 Ryan_Lane: temporarily disabling DynamicSidebar support in OpenStackManager
21:59 notpeter: gracefulling apache on stafford
21:36 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Moved htmlCacheUpdate jobs to redis'
21:19 ottomata: added python-statsd to apt repo for hashar
21:18 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php
21:07 ottomata: added python-voluptuous to apt repo (for hashar)
20:46 ottomata: added python-jsonschema to wikimedia apt repo: ori-l :)
20:00 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Moved async upload jobs to redis'
19:13 notpeter: running puppetd --enable on all nodes via salt
19:09 mwalker: eqiad frack is now using clustered memcache and has site local configuration for DB, ActiveMQ, MemCacheD shtuff
18:24 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Enabled use of redis for null test jobs'
18:06 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Switched all remaning wikis to 1.22wmf1
17:45 RobH: authdns-update to rename db1012 to stat1002
17:29 logmsgbot: aaron synchronized php-1.22wmf1/maintenance/copyJobQueue.php 'deployed 54f74111901ddebdc69f26847e3904140b3723d5'
17:24 LeslieCarr: restart unresponsive cp1037
17:14 RobH: authdns-update
15:48 mark: Ran dist-upgrade (for varnish upgrade) on cp1021-1036
15:40 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Nostalgia ext'
15:40 logmsgbot: demon synchronized wmf-config/extension-list 'Nostalgia ext'
15:38 logmsgbot: demon synchronized php-1.22wmf1/extensions/Nostalgia
14:33 logmsgbot: maxsem synchronized wmf-config/mobile.php
14:19 logmsgbot: maxsem synchronized wmf-config 'Enable $wgMFVaryResources on test'
14:12 mark: Inserted varnish 3.0.3plus-rc1-wm10 packages into the APT repository
12:23 apergos: removed outright puppet.log on neon, was > 7gb and no room left on device
07:53 logmsgbot: krinkle synchronized php-1.22wmf1/resources/startup.js 'Ia54dd738b3ce0995fa'
05:57 Tim: on stafford: swapdeath repeat narrowly averted via killall ruby && /etc/init.d/apache2 stop
05:34 Tim: on brewster: proxy down due to full root partition, again. Will fix it properly this time.
05:28 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Add 'Zero' & 'Zero talk' namespaces to metawiki (If6b3ce5e4)'
04:53 Tim: stafford went into swapdeath. I killed all ruby processes a couple of times, and then eventually stopped apache2 for a while, while it recovered
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Apr 10 02:28:06 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 10 02:17:19 UTC 2013
00:21 mutante: removing xenon and db1012 from icinga configs, running puppetstoredconfigclean.rb on them, restarting icinga
00:07 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'https://gerrit.wikimedia.org/r/#/c/58436/'
00:06 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'https://gerrit.wikimedia.org/r/#/c/58436/'
April 9
23:29 logmsgbot: tstarling synchronized php-1.22wmf1/extensions/Nostalgia
23:21 notpeter: running upgrades on db56 and rebooting
23:19 notpeter: running upgrades on db64 and rebooting
22:55 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend/
22:53 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/
22:36 logmsgbot: reedy synchronized wmf-config/
22:34 LeslieCarr: clearing a few varnish caches - fyi if db load is seen to increase
22:17 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:45 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
21:45 hashar: Zuul is slowly dequeuing a large number of jobs. There is currently rougly 30 minutes lag between change submission and actually processing of the job. No idea why but investigating.
19:56 Ryan_Lane: installed Nuke on wikitech
18:11 K4-713: updated payments anti-fraud rules
13:07 hashar: jenkins plugin upgrade is tracked in bug 47040 . Luckily git plugin has been fixed just in time ;-D Just need to migrate the config history.
12:37 hashar: jenkins plugins seems to have been upgraded overnight :(
10:26 mark: Deployed experimental varnish build (3.0.3plus~rc1-wm10) on dysprosium and cp1022
08:54 hashar: Jenkins tests for mediawiki/core are broken (bug 47031 ) due to a faulty change that has been merged in master branch. Root cause is the gating job not testing the proper change bug 46723 :/
08:43 Nikerabbit: Got rid of local changes and undeployed commit related to noc
08:41 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Narayam and WebFonts'
05:42 Tim: on mw1149: enabled apache access log
02:31 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr 9 02:30:59 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 9 02:18:53 UTC 2013
00:26 ^demon: jenkins is back. java gc took awhile.
00:19 ^demon: jenkins is hung, looking into it.
00:16 awight: payments cluster updated from 22943dc28 to d3954768e4dd
April 8
23:16 logmsgbot: catrope synchronized php-1.22wmf1/extensions/WikiEditor 'Deploying 17a90bc74e87341199a50ebe5348576e05be028a'
23:15 logmsgbot: catrope synchronized php-1.21wmf12/extensions/WikiEditor 'Deploying 17a90bc74e87341199a50ebe5348576e05be028a'
22:55 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Changing from emery to vanadium'
22:13 logmsgbot: demon synchronized php-1.21wmf12/includes/AutoLoader.php 'Backported Poolcounter convenience class'
22:13 logmsgbot: demon synchronized php-1.21wmf12/includes/PoolCounter.php 'Backported Poolcounter convenience class'
21:59 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-pmtpa.php 'Enable poolcounter for search'
21:59 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-eqiad.php 'Enable poolcounter for search'
21:42 logmsgbot: demon synchronized php-1.21wmf12/extensions/MWSearch/MWSearch_body.php 'Deploying Poolcounter updates'
21:17 notpeter: temp stopping slave on db55
20:53 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Bumped file journal ttl to a year'
20:18 AaronSchulz: Moved ceph sync scripts to terbium and started a second pass
20:15 logmsgbot: aaron synchronized php-1.22wmf1/extensions/FlaggedRevs 'deployed c61688baef682b7d97068fefdcc399256786a387'
19:08 ^demon: gerrit back
19:07 ^demon: gerrit restarting, config change to fix idle timeouts.
19:00 ottomata: removing some unused filters from emery's udp2log instance
18:34 Jeff_Green: deployed 1.22 on payments
18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf1
17:09 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1058 to s1 at full weight'
17:05 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1058 to s1 at a low warmup rate'
11:57 ^demon: gerrit back, deployed 2.6-rc0-154-gfcdb34b which contains a temporary fix for the stream-events timeout issues. See bug 46917 for info.
11:53 ^demon: bringing gerrit down for urgent update
10:31 hashar: gerrit: created operations/debs/libvpx for j^ . Initialized pristine-tar and upstream branches using empty commits and force push.
07:48 hashar: restarting Zuul for demo purposes :-)
04:00 Tim: on brewster: root partition was full, removed squid access.log and store.log and started squid
03:55 Tim: on all apaches: upgrading libpoppler
02:25 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 8 02:25:07 UTC 2013
02:18 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 8 02:18:18 UTC 2013
April 7
22:46 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
22:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:31 logmsgbot: reedy synchronized wmf-config/ 'Kill readerfeedback config, it aint coming back'
22:26 logmsgbot: reedy synchronized wmf-config/
22:20 logmsgbot: reedy synchronized php-1.22wmf1/extensions/CentralAuth
11:35 ^demon: restarting gerrit, stream-events seized up again.
07:26 hashar: gallium: restarted Zuul just to be sure. Gerrit on manganese is stuck again and need to be restarted bug 46917
02:17 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sun Apr 7 02:17:09 UTC 2013
02:11 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sun Apr 7 02:11:21 UTC 2013
April 6
12:27 ^demon: also restarted zuul
12:19 ^demon: restarting gerrit again, stream-events halted like friday.
02:23 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sat Apr 6 02:22:48 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sat Apr 6 02:11:56 UTC 2013
01:55 binasher: iwlinks iwl_prefix_from_title migration completed for all projects
00:52 K4-713: adjusted anti-fraud rules on payments cluster
April 5
23:11 binasher: running iwlinks index migration on s4 (commons)
21:27 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1021 to full weight, pulling db1023'
21:13 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1021 at a low weight'
20:05 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1021 for upgrade'
19:53 binasher: streaming backup of db1056 to db1050 (converting s1 snapshot slave to file_per_table)
18:58 logmsgbot: asher synchronized wmf-config/db-eqiad.php 's1: moving watchlist to db1049, pulling db1043, adding db1056'
18:50 Jeff_Green: increased jvm heap size on silicon (payments activemq)
18:42 notpeter: restarting gerrit
18:22 logmsgbot: reedy synchronized php-1.21wmf12 'Killing active users in 1.21wmf12'
09:39 hashar: Ariel kindly restarted gerrit on manganese which unbroken the stream-events process in Zuul. It again receives events.
09:33 hashar: restarted Zuul
09:23 Nemo_bis: varnish esams purging still broken, e.g. https://upload.wikimedia.org/wikipedia/commons/f/fe/Wikipedia-logo-v2-km.png
08:58 hashar: for Jenkins / Gerrit issue see bug 46917
08:48 hashar: Jenkins no more receiving ANY events from Gerrit. Effect: no jobs are triggered.
02:23 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Fri Apr 5 02:23:11 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Fri Apr 5 02:12:34 UTC 2013
00:49 logmsgbot: csteipp synchronized php-1.21wmf12/includes/
00:46 logmsgbot: csteipp synchronized php-1.22wmf1/includes/
00:30 Tim: on fenari: testing a new version of mw-update-l10n which doesn't use stdout of mergeMessagesFileList.php
00:20 logmsgbot: tstarling synchronized wmf-config/ExtensionMessages-1.22wmf1.php
00:19 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to master
00:07 Tim: all 1.22wmf1 wikis briefly showed error messages due to display_errors being enabled when mergeMessageFileList.php generated ExtensionMessages-1.22wmf1.php. Fixed temporarily.
00:04 logmsgbot: tstarling synchronized wmf-config/ExtensionMessages-1.22wmf1.php
00:00 logmsgbot: mflaschen synchronized wmf-config/extension-list 'Remove E3Experiments and LastModified'
April 4
23:51 logmsgbot: demon synchronized wmf-config/extension-list 'Remove e3 since it was deleted'
23:49 logmsgbot: mflaschen synchronized wmf-config/CommonSettings.php
23:46 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043'
23:46 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to master
23:45 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043'
23:44 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043'
23:25 binasher: shutting down mysql on db1043, preparing to reboot
23:24 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043'
23:17 mutante: wikinews.com now redirects to wikinews.org
23:11 mutante: DNS update - adding wikinews.com
23:08 logmsgbot: dzahn gracefulled all apaches
22:52 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Re-enable Echo on testwiki'
22:49 mutante: running updateSpecialPages on cswikinews per request
22:14 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Disable Echo on testwiki'
22:10 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Bump Echo EventLogging schema for new notification types.'
22:02 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GuidedTour/ 'GuidedTour 1.22wmf1 deployment for E3'
22:02 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GettingStarted/ 'GettingStarted 1.22wmf1 deployment for E3'
22:01 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GuidedTour/ 'GuidedTour 1.21wmf12 deployment for E3'
22:00 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GettingStarted/ 'GettingStarted deployment for E3'
20:21 LeslieCarr: stopped cp3001 varnish
20:18 LeslieCarr: restarting varnish on cp3001
20:15 LeslieCarr: restarted udpmcast on hooft
20:13 mutante: puppetstoredconfigclean.rb xenon.eqiad.wmnet - Killing xenon ...
19:34 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/57525/'
19:29 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/57525/'
19:20 binasher: enabled 1:100 full query logging on db1042/1043, disabled puppet [planning to revert in 4-8 hours]
19:09 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/57528'
19:01 binasher: resuming iwlinks migrations on s3
17:31 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
17:31 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
17:27 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Updating ArticleFeedbackv5 frwiki config'
16:16 paravoid: upgrading image & video scalers for libav
15:55 ottomata: now generating webstatscollector dumps on gadolinium. snapshot1 syncs dumps from there over to dataset2 for dumps.wikimedia.org
15:38 K4-713: adjusted anti-fraud rules on the payments cluster
13:26 ^demon: running jgit gc on all repos
05:43 Ryan_Lane: clearing squid logs and restarting squid on brewster
04:16 Ryan_Lane: opendj upgrade on virt1000 complete
03:56 Ryan_Lane: upgrading opendj on virt1000
02:21 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Thu Apr 4 02:21:01 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 4 02:13:31 UTC 2013
April 3
23:19 mwalker: Updating CentralNotice schema on testwiki & metawiki with patch-centralnotice-2_3.sql
22:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
21:45 binasher: started iwlinks index migrations on all wikis [gerrit 43389]
21:32 cmjohnson1: Running DNS update
20:34 ottomata: udpated webstatscollector package in apt repo to 0.1-3
18:54 mutante: DNS update - also adding softwarewikipedia.org
18:49 RoanKattouw: Removed caesium and xenon from /home/wikipedia/common/docroot/noc/pybal/eqiad/parsoid
18:43 mutante: restarting pdns on ns1
18:42 mutante: DNS update - adding softwarewikipedia.com for RT-4355
18:41 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikibooks to 1.22wmf1
18:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary to 1.22wmf1
18:35 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikisource to 1.22wmf1
18:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikivoyage to 1.22wmf1
18:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, wikimania and wikimedia wikis to 1.22wmf1
18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: private, closed and fishbowl to 1.22wmf1
17:14 logmsgbot: aaron synchronized php-1.22wmf1/includes/objectcache/SqlBagOStuff.php 'deployed b61053ca4b554f6bd18fb6408967839cdb5ccde2'
17:12 logmsgbot: aaron synchronized php-1.21wmf12/includes/objectcache/SqlBagOStuff.php 'deployed 61587acc64cb62400ff7978271c54e8bd8b1f02d'
16:54 logmsgbot: reedy synchronized php-1.21wmf12/includes/Collation.php
16:34 cmjohnson1: authdns update
14:22 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php 'setting weight on db1028 to 400'
13:15 mark: Stopped and started MySQL on pc1003 after finding mysql deadlocked
13:10 logmsgbot: reedy synchronized wmf-config/
12:20 mark: Rebooting cp3003 for RAID reconfiguration
10:43 mark: Deactivated AS6908 peering on cr2-knams
06:09 ^demon: gerrit: finished doing jgit gc on mediawiki/core. Repo size 3G -> 323M. Fresh clone time to <5m to localhost, <1.5m to other hosts inside wmf. I rock. Backup's in /home/demon/core.git in case something goes wrong. Bed time.
03:05 LeslieCarr: the final upload varnish is on 2gig instead of 1gig ! win!
02:32 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 3 02:31:44 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Apr 3 02:19:30 UTC 2013
01:05 LeslieCarr: applying a big change to varnish interface groups - risk higher than normal
00:05 binasher: removed sda3/varnish.persist on cp1021, restarted varnish
00:00 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/PostEdit
April 2
23:46 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php '(Ia2665c4fe) Enable PostEdit on bn, br, ca, cs, et, ka and zh wikis'
23:46 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php '(Ibc2633f1c) : 10000 => 5000.'
23:25 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqiad'
23:11 logmsgbot: anomie synchronized php-1.22wmf1/includes/api/ApiQueryRecentChanges.php 'Fix for API list=recentchanges rccontinue'
23:10 logmsgbot: anomie synchronized php-1.21wmf12/includes/api/ApiQueryRecentChanges.php 'Fix for API list=recentchanges rccontinue'
23:01 logmsgbot: maxsem synchronized php-1.21wmf12/resources/Resources.php 'https://gerrit.wikimedia.org/r/#/c/57212/'
22:49 logmsgbot: maxsem synchronized php-1.21wmf12/resources/Resources.php 'https://gerrit.wikimedia.org/r/57208'
22:39 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/56696/'
22:34 awight: payments cluster updated from 12bc36c238 to 0ab980b8b
22:24 logmsgbot: maxsem synchronized php-1.21wmf12/includes/resourceloader/ResourceLoaderStartUpModule.php
22:20 LeslieCarr: authdns-update for cp1039 and cp1040 move
22:14 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend 'touch'
22:11 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend
22:08 logmsgbot: maxsem Finished syncing Wikimedia installation... :
21:42 logmsgbot: maxsem Started syncing Wikimedia installation... :
21:38 logmsgbot: reedy synchronized wmf-config/
21:25 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
20:58 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1028'
20:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:41 logmsgbot: maxsem synchronized php-1.21wmf12/includes/OutputPage.php 'https://gerrit.wikimedia.org/r/#/c/49071/'
20:34 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/57078/'
20:11 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'moving s1 watchlist to db1052, db1043 to full weight'
20:09 Ryan_Lane: added marc to the ops ldap group
20:01 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1001'
19:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
19:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:56 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php 'adding db1028 back'
18:54 notpeter: actually starting innopack from db65 to db1054 for pre-labsdb db (sanitarium)
18:48 notpeter: starting innopack from db31 to db1054 for pre-labsdb db (sanitarium)
18:39 binasher: completed image img_media_mime migration on all projects
18:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
17:42 LeslieCarr: restarted varnishncsa-locke on cp1023
17:41 LeslieCarr: restarted varnishncsa-multicast_relay on cp1028
17:34 cmjohnson1: correction db1028
17:34 cmjohnson1: removing disk 0 from db1048 to replace
17:28 Jeff_Green: disable fundraising banner log rotation on locke
17:21 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php 'adding db1001 back to production removing db1028'
17:04 cmjohnson1: setting weight to 100 on db1001
15:42 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Message for ancient skin users'
15:41 logmsgbot: anomie synchronized php-1.21wmf12/cache/l10n/l10n_cache-en.cdb
15:34 logmsgbot: anomie synchronized php-1.21wmf12/extensions/WikimediaMessages/WikimediaTemporaryMessages.i18n.php
15:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 2 15:12:45 UTC 2013
15:09 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr 2 15:08:37 UTC 2013
14:54 logmsgbot: demon synchronized php-1.21wmf12/extensions/WikimediaMessages/WikimediaTemporaryMessages.i18n.php '8th time is the charm'
14:51 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'I hate l10nupdate'
14:50 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Notice for users of disabled skins'
14:24 paravoid: deploying squid config, diverting all of upload to swift
04:09 binasher: increased 'wikiadmin'@'10.64.%' max connections to 200 on s1
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 2 02:17:33 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr 2 02:10:24 UTC 2013
01:15 Ryan_Lane: removing labstore1 and labstore2 entries from projectstorage.wmnet rr dns entry in preparation for shrinking volumes
00:44 logmsgbot: reedy synchronized wmf-config/
00:43 binasher: now running the image img_media_mime migration on commons (the big one)
00:41 mutante: restarting lucene on all pool4 servers (one by one)
00:36 mutante: creating search index for transitionteamwiki
00:30 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
00:14 logmsgbot: reedy synchronized php-1.22wmf1/cache/interwiki.cdb 'Updating 1.22wmf1 interwiki cache'
00:10 logmsgbot: reedy synchronized php-1.21wmf12/cache/interwiki.cdb 'Updating 1.21wmf12 interwiki cache'
00:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
April 1
23:55 LeslieCarr: search1024 is our guinea pig of using sfp-t's and is connected to asw2-a5-eqiad ge 0/0/18
23:48 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
23:40 mutante: creating swift containers for private wiki
23:39 logmsgbot: dzahn synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
23:37 mutante: dumpInterwiki.php to update interwiki cache
23:36 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic to pmtpa temporarily'
23:33 mutante: adding transitionteamwiki to private.dblist
23:25 mutante: running addWiki.php for transitionteamwiki - Access denied for user ...
23:21 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
23:14 cmjohnson1: swapping disk1 (slot1) db1001
23:06 mutante: creating upload dir on upload7 for transitionteamwiki
22:57 binasher: beginning image.img_media_mime scehma migrations, starting with s1
22:52 logmsgbot: aaron synchronized php-1.22wmf1/includes/job 'deployed ec8f42daf19f386d9af2eece6dfc5f8fa6fc42f3'
22:26 logmsgbot: aaron synchronized php-1.22wmf1/maintenance 'deployed c7832c8956d79892be93452e5f4d6b52df1d3bf0'
22:26 logmsgbot: aaron synchronized php-1.22wmf1/includes/job 'deployed c7832c8956d79892be93452e5f4d6b52df1d3bf0'
22:26 mutante: recreating transitionteam wiki docroot from skel-1.5 and syncing docroot
22:26 logmsgbot: root synchronized docroot
21:09 LeslieCarr: rebooting lvs1001
21:04 logmsgbot: demon synchronized php-1.22wmf1/cache/l10n 'Manually syncing 1.22wmf1 l10ncache'
20:56 LeslieCarr: dist-upgrading lvs1001
20:48 LeslieCarr: failing over lvs from lvs1001 to lvs1004
20:44 LeslieCarr: CORRECTION restarting sshd on lvs1001
20:44 LeslieCarr: restarting lvs1001
20:21 logmsgbot: demon synchronized php-1.21wmf12/cache/l10n 'Manually syncing 1.21wmf12 l10ncache'
20:20 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 1 20:20:29 UTC 2013
20:00 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 1 19:59:44 UTC 2013
19:54 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 1 19:54:21 UTC 2013
19:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki
19:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki and testwiki to 1.22wmf1
19:11 logmsgbot: reedy synchronized php-1.22wmf1/extensions/ 'Wikidata extensions to same as 1.21wmf12'
19:04 logmsgbot: reedy synchronized php-1.22wmf1/includes/
18:56 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf1 and rebuild l10n cache
18:49 LeslieCarr: testing logbot
18:20 Reedy: Ran scap-recompile to build texvc
18:20 jeremyb_: re!logged from earlier today while morebots was gone (all times UTC)
18:20 jeremyb_: 17:56:29 <logmsgbot> !log reedy synchronized w
18:20 jeremyb_: 17:55:45 <logmsgbot> !log reedy synchronized docroot
18:20 jeremyb_: 17:55:04 <logmsgbot> !log reedy synchronized php-1.22wmf1 'Initial sync of php-1.22wmf1'
18:19 jeremyb_: 17:27:13 <logmsgbot> !log reedy synchronized wmf-config/ExtensionMessages-1.22wmf1.php
18:19 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf1 and rebuild l10n cache
18:19 jeremyb_: 17:14:31 <Jeff_Green> !log moved aside williams:/opt/otrs-home/.spamassassin/auto-whitelist and restarted spamd to test whether the bloated AWL is resulting in poor filtering
18:19 jeremyb_: 14:39:19 <andrewbogott> !log restarted glusterfs-server on labstore1 and labstore2, because of read errors in public/datasets/public in bots project
18:19 jeremyb_: 13:55:28 <ottomata> !log added webstatscollector-0.1-2 to apt repo
18:18 jeremyb_: 11:54:15 <paravoid> !log repooling ms-fe1; staggered depool/restart/pool of ms-fe2-4
18:17 Ryan_Lane: restarted morebots. wikitech-static had 100% / partition. Have also fixed that and its cause.
18:16 jeremyb_: 07:45:24 <paravoid> !log depooling ms-fe1 for staging
18:16 jeremyb_: 03:58:05 <logmsgbot> !log asher synchronized wmf-config/db-eqiad.php 'returning db1051'
18:16 jeremyb_: 03:44:43 <logmsgbot> !log asher synchronized wmf-config/db-eqiad.php 'pulling db1051'
18:16 jeremyb_: 03:22:54 <binasher> !log reinstated max_user_connections = 80 for wikiadmin on s1
18:15 cmjohnson1: restarted morebots
03:23 binasher: reinstated max_user_connections = 80 for wikiadmin on s1
02:12 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 1 02:12:24 UTC 2013
March 31
02:10 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sun Mar 31 02:10:18 UTC 2013
March 30
02:14 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sat Mar 30 02:13:49 UTC 2013
00:01 LeslieCarr: rebuilding mailman archives …. very slowly
March 29
23:51 LeslieCarr: stopping mailman to remove sensitive message
22:38 LeslieCarr: upgrading packages on aluminium
22:25 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
21:02 hashar: jenkins: raised the number of execution for deployment-bastion from 2 to 4. Will let us execute more jobs in parallel
20:14 mutante: scheduling a 2-year downtime for db11 :p
19:33 logmsgbot: dzahn gracefulled all apaches
19:30 logmsgbot: dzahn synchronized ./docroot/transitionteam
19:29 mutante: creating transitionteam docroot
19:25 logmsgbot: aaron synchronized php-1.21wmf12/includes 'deployed fa5c0a6a82c9a1cb63f3cedd87e6c4aba59c994d'
18:49 sbernardin: hot-swapping disk on db29
18:04 ^demon: gerrit back up, now running 2.6-rc0-76-g52fb5ae
18:00 mutante: DNS update - adding transitionteam wiki
17:47 sbernardin: taking down db29 to replace bad hw
17:45 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Re-enabled async upload for commons'
17:23 ^demon: gerrit offline for an update, brb
09:25 logmsgbot: olivneh synchronized wmf-config/throttle.php 'Updating throttle rules for guwiki workshop (Bug 46686)'
04:16 logmsgbot: aaron synchronized php-1.21wmf12/maintenance/runJobs.php 'deployed 651884d4bdc76c29dd717dfdbbd698632223f3b5'
04:15 logmsgbot: aaron synchronized php-1.21wmf12/includes/job/JobQueueGroup.php 'deployed 651884d4bdc76c29dd717dfdbbd698632223f3b5 '
02:17 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Fri Mar 29 02:17:06 UTC 2013
01:24 ori-l: Stopping EventLogging daemons to allow Puppet to change 'eventlogging' user's home dir
00:11 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/MoodBar 'Updating MoodBar to remove ClickTracking integration'
00:03 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GuidedTour/GuidedTour.php 'Small bug fix to GuidedTour; removing unneeded dependency. https://gerrit.wikimedia.org/r/#/c/56546/1/GuidedTour.php'
00:00 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GuidedTour/GuidedTour.php 'Small bug fix to GuidedTour; removing unneeded dependency. https://gerrit.wikimedia.org/r/#/c/56546/1/GuidedTour.php'
March 28
23:24 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Remove E3Experiments and LastModified (2/2)'
23:24 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Remove E3Experiments and LastModified (1/2)'
22:35 logmsgbot: khorn synchronized php-1.21wmf12/includes/cache/MessageCache.php 'Reverting changes in language fallback behavior'
22:34 logmsgbot: khorn synchronized php-1.21wmf12/languages/Language.php 'Reverting changes in language fallback behavior'
22:20 logmsgbot: spage Finished syncing Wikimedia installation... : E3: Updated GettingStarted and GuidedTour extensions
22:12 LeslieCarr: removed some log files from ms1004 - it was full
22:10 RobH: bugzilla admin stuff: noc@w.o is now a disabled account cuz im tired of folks who arent on the alias attempting to sign up with it
22:02 logmsgbot: spage Started syncing Wikimedia installation... : E3: Updated GettingStarted and GuidedTour extensions
21:46 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming/modules/ext.navigationTiming.js 'Fix mobileMode property name (was: 'mobile')'
21:36 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming
20:11 Jeff_Green: add hostname samarium.wikimedia.org and pdns-update
19:51 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1051'
19:44 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1051'
19:44 LeslieCarr: killed gmetad on neon
19:36 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/ArticleFeedback/populateAFRevisions.php 'Fix fatal error triggered by populateAFRevisions.php cronjob when AFT is not loaded.'
19:33 mutante: starting gmetad on neon again
19:24 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo 'syncing Echo on wmf12'
19:17 mutante: manually deleting db11 references from icinga configs and restarting it
19:12 mutante: pkill -u snmptt on neon, restarting snmptt deamon
19:10 binasher: replaced collector on professor with locally built version from I2bc117d5620b9e545a8e5a6cf3b654ee835b75d7 (testing fix for issues only under very high load)
19:10 mutante: stopping gmetad and ganglia-parser on neon
18:53 mutante: adding gjg (greg-g) to wmf ldap group
18:23 mutante: restarting lucene on pool4 search nodes to pickup new wikivoyage indices
18:12 ottomata: adding reportcard and analytics .wikimedia.org CNAMEs to stat1001
18:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:56 mutante: creating search indices for new wikivoyages
17:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
11:49 hashar: search19 and search20 issue with enwiki.prefix is in RT #4845
11:42 hashar: search19 and search20 have enwiki.prefix marked as FAILED. (see: curl --silent http://search19.pmtpa.wmnet:8123/status |grep FAILED and curl --silent http://search20.pmtpa.wmnet:8123/status |grep FAILED ).
10:39 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/Translate/ 'Translate to master'
04:19 paravoid: updating jsduck in apt and upgrading it on gallium
03:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:19 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Thu Mar 28 02:19:24 UTC 2013
00:47 K4-713: synchronized payments cluster to 12bc36c2385d8e56792c
00:27 mutante: once again disabling notifications for db11
00:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
March 27
23:59 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
23:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:40 Ryan_Lane: finished updating wikitech-static
23:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:38 Ryan_Lane: updating wikitech-static to wmf/1.21wmf12
23:24 Ryan_Lane: finished upgrading wikitech
23:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:23 Ryan_Lane: upgrading wikitech to wmf/1.21wmf12
23:17 logmsgbot: reedy synchronized php-1.21wmf12/cache/interwiki.cdb 'Updating 1.21wmf12 interwiki cache'
23:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:50 logmsgbot: maxsem synchronized php-1.21wmf11/extensions/MobileFrontend 'touch'
22:47 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend 'touch'
22:24 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
22:22 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
22:05 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment, 2nd attempt
21:48 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment, 2nd attempt
21:12 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'changing s1 slave weights for testing'
20:10 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild localisation cache to fix he related ParserFunctions problems
19:56 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild localisation cache to fix he related ParserFunctions problems
19:43 logmsgbot: reedy synchronized php-1.21wmf12/extensions/ParserFunctions/
19:38 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for WikiData deployment
19:05 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for WikiData deployment
18:51 cmjohnson1: pushing mw1209-1220 into service
18:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:39 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf12
18:23 mutante: installing package upgrades on neon (Icinga), incl. ganglia-monitor,gmeta
18:09 mutante: bunch of mw* hosts: fix puppet runs
18:08 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo 'syncing Echo for wmf12'
18:05 mutante: mw1130 - kill and fix puppet runs
17:41 Jeff_Green: authdns-update, pdns on ns1 croaked, restarted
15:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: be_x_oldwiki to 1.21wmf12
14:09 hashar: jenkins has been restarted by mistake :-( Job building will be unavailable for up to half an hour.
13:58 hashar: Jenkins: disabled the old Gerrit Trigger Plugin bug 46415
09:56 Nikerabbit: Testing Translate bug fixes on test.wikipedia.org
05:20 Tim: on fluorine: running mw-log-cleanup once to test Id2196e7b
05:02 Ryan_Lane: disabling Bayes-collector cron on sanger
04:55 Ryan_Lane: purging BayesLearning files older than 70 days on mchenry
02:45 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Wed Mar 27 02:44:56 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Mar 27 02:19:11 UTC 2013
00:55 LeslieCarr: ms-be3 was crashed with stack traces for swift-container
00:54 LeslieCarr: rebooting ms-be3
00:38 awight: payments cluster updated from 4862fb8eef to f295d851c
00:34 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'db1052 to full weight'
00:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1001, adding db1052 at a warmup weight'
00:11 awight: payments cluster updated from fa2e2 to a06a03f05 wheee!
March 26
23:54 awight: payments cluster rolled back from e758102 to fa2e240b6e9
23:53 awight: payments cluster updated from fa2e240b6e9 to e758102
23:53 Tim: on all apaches: replacing /usr/local/apache/common-local/live-1.5 with a symlink to w
23:44 logmsgbot: tstarling synchronized w
23:35 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/56019/'
23:33 awight: payment cluster rollback, from 4862fb8 to fa2e240b
23:31 mutante: restarting nrpe on professor
23:29 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming/modules/ext.navigationTiming.js 'Workaround for buggy NavTiming implementation in IE9 (bug 46474)'
23:20 awight: payments cluster updated from fa2e240b to 4862fb8
23:10 mutante: db11 - disabling all notifications
23:07 mutante: mw13 - killing puppet and re-running
23:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:00 mutante: removing mysql packages from db1036 (broken dpkg, status "pi")
22:55 mutante: re-installing ganglia-monitor on mw1135, fix dpkg
22:54 logmsgbot: reedy synchronized docroot
22:52 mutante: starting udpprofile (carbon-cache.py) on professor
22:51 logmsgbot: reedy synchronized wmf-config/
22:49 mutante: search21 - remove sun-j2sdk , replaced by oracle-jdsdk
22:47 mutante: search21 - broken DPKG: iF sun-j2sdk1.6
22:45 mutante: mw2 - broken DPKG: pi librsvg2-2 , pi librsvg2-bin
22:39 LeslieCarr: did a service apache2 start on mw108*
22:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:10 notpeter: icinga mostly trustworthy again :)
22:08 andrewbogott: restarted nova-api on virt0 because it appeared to have crashed.
21:31 andrewbogott: restarted memcached on virt0 in a futile attempt to address https://bugzilla.wikimedia.org/show_bug.cgi?id=46583
21:30 mutante: killed stuck snmptt procs on neon to fix the others.. apparently :p
21:28 mutante: stopping gmetad on neon
20:52 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'syncing CommonSettings.php for Echo event tracking'
20:41 mutante: more degraded RAIDs: db1001->RT-4832, db1028->4834
20:35 mutante: broken disks/degraded RAID on db servers: db11->RT-4828, db29->RT-4829, db45->RT-4831
20:34 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/ContributionReporting 'syncing ContributionReporting extension'
20:07 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'raising db1051 to full weight'
19:49 logmsgbot: khorn synchronized php-1.21wmf12/extensions/CentralNotice 'Fixing CentralNotice bug in which it was possible to initialize multiple times'
19:46 logmsgbot: khorn synchronized php-1.21wmf11/extensions/CentralNotice 'Fixing CentralNotice bug in which it was possible to initialize multiple times'
19:40 logmsgbot: kaldari synchronized php-1.21wmf12/resources/startup.js 'syncing php-1.21wmf12/resources/startup.js'
19:33 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase/client
19:27 cmjohnson1: mw1209 re-imaging
19:20 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo/modules/icons 're-syncing Echo icons dir'
19:19 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Include AFTv5 on frwiki'
19:12 logmsgbot: mlitn synchronized wmf-config 'Enable AFTv5 on frwiki'
19:05 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Collection/
19:04 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
19:04 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
19:02 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo/modules/icons 'syncing Echo icons dir'
18:58 mutante: starting gmetad on neon - we failed to resolve data source name ms-fe1001.eqiad.wmnet
18:56 mutante: restarting ircecho on fenari
18:48 mutante: stopping gmetad on neon
18:44 mutante: killing all snmp related processes on neon and restarting
18:43 logmsgbot: reedy synchronized php-1.21wmf12/extensions/MobileFrontend/includes/specials/SpecialMobileWatchlist.php
18:41 logmsgbot: kaldari synchronized php-1.21wmf11/extensions/Echo/modules/icons 'syncing Echo icons dir'
18:39 mutante: restarting snmtrapd on neon
18:25 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Turning Thanks extension on on MediaWiki wiki'
18:13 logmsgbot: krinkle synchronized php-1.21wmf12/resources/mediawiki/mediawiki.js 'I80af730daa815 fixing bug 46575'
18:12 Krinkle: Change I34a488247 (mw.loader debug) has been reverted in wmf/1.21wmf12 and pulled on fenari
17:51 cmjohnson1: removing disk from slot 0 ms-be1012
17:14 Krinkle: Change I34a488247 is now merged in wmf/1.21wmf12 and pulled on fenari for test.wikipedia.org, do not sync.
17:04 ottomata: added (crappy, but better than nothing) .deb for webstatscollector to apt.wikimedia.org
15:24 cmjohnson1: swapping disk 6 ms-be1012
14:34 logmsgbot: hashar synchronized multiversion '47564 wikiversions.cdb now vary by realm'
14:10 cmjohnson1: running sync-common on mw1209
10:23 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/UniversalLanguageSelector/UniversalLanguageSelector.php 'Testing fix for wikidata'
09:28 pp-pdf1: updated mwlib to 0.15.4
08:47 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/Translate
08:46 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/UniversalLanguageSelector
07:27 mark: Restarted swift-proxy on ms-fe*
04:16 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
02:46 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Tue Mar 26 02:45:52 UTC 2013
02:21 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Mar 26 02:21:08 UTC 2013
01:41 mutante: analytics1026 - delete 16G udp2log.log to free disk
01:38 mutante: analytics1003 - gzipping jmxtrans logs to free some disk
01:36 mutante: analytics1003 - gzipping kafka.log to free some disk
00:39 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1051 at a low warmup weight'
March 25
23:36 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'done'
23:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'changing s1 weights, returning db1050'
23:27 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Quick profiling'
22:10 hashar: reloading Zuul
21:44 logmsgbot: khorn synchronized wmf-config/InitialiseSettings.php 'Re-enabling CentralNotice'
21:39 mutante: killing and restarting nagios-nrpe-server on '*'
21:32 logmsgbot: khorn synchronized php-1.21wmf12/extensions/CentralNotice/ 'CentralNotice jQuery fix'
21:28 logmsgbot: khorn synchronized php-1.21wmf11/extensions/CentralNotice/ 'CentralNotice jQuery fix'
21:12 mutante: killing and restarting nagios-nrpe-server on gadolinium
21:07 mutante: --last-lines 20 | RT ..would be nice sometimes:)
20:13 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
20:07 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Use ffmpeg for ogg thumbs as is done for webm'
19:55 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild l10n cache for wikibase
19:33 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild l10n cache for wikibase
18:57 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
18:46 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
18:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf12
18:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable CN everywhere'
18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rb
18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf12
17:25 logmsgbot: reedy synchronized wmf-config/
March 24
18:13 Ryan_Lane: restarting glusterd processes on labstore*
02:31 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sun Mar 24 02:31:22 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sun Mar 24 02:08:00 UTC 2013
March 23
02:37 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sat Mar 23 02:36:53 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sat Mar 23 02:13:44 UTC 2013
00:17 RoanKattouw: Deployed master of Parsoid onto wtp1004; the code is broken but the box is depooled for Gabriel's benchmarking work
00:15 RoanKattouw: That's me messing with wtp1004
March 22
23:33 ottomata: restarting hadoop setting yarn.nodemanager.resource.memory-mb to 16G
23:06 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
23:04 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
23:04 mutante: sync-wikiversions
22:29 logmsgbot: aaron synchronized php-1.21wmf12/extensions/Translate 'deployed 0aab5984415a94bd214088767ff1de33ab57c4e0'
22:12 mutante: killall nrpe via salt, then restart nagios-nrpe-server , runs as wrong user
22:09 mutante: kill and restart nrpe on oxygen
22:07 mutante: installing package upgrades on oxygen
21:40 mutante: killing nagios-nrpe on gallium and testing if puppet restores it properly
21:03 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Set all non-Wikipedias back to 1.21wmf12 again.
20:51 logmsgbot: aaron synchronized php-1.21wmf12/includes/Title.php 'deployed 0f4914bb4ff2d13cb124cad94dc5ded663123a18'
20:46 logmsgbot: aaron synchronized php-1.21wmf12/includes/WikiPage.php 'deployed e372b635ae6d6a589f6d87f14fe4c9452d7a1b4d'
20:38 logmsgbot: aaron synchronized php-1.21wmf12/includes/Title.php 'Reverted follow-up fixes too'
19:32 ottomata: removed unicast udp2logging to gadolinium; gadolinium will consume from multicast stream
18:44 K4-713: synchronized payments cluster to fa2e240b6e92440b
18:28 K4-713: synchronized payments cluster to df3c8dbc937c971
17:33 logmsgbot: aaron synchronized php-1.21wmf12/includes/WikiPage.php 'deployed 1862a57aef08432e8b1194a54dc179649f5cae63'
09:08 MaxSem: Recreating Solr index
02:52 Tim: on searchidx1001: started incremental indexer, apparently it died on March 21 at 02:08 when it ran at the same time as a cron job import
02:50 binasher: power cycling db1050, unresponsive on serial console
02:49 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1050, crashed'
02:47 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043'
02:42 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Fri Mar 22 02:42:32 UTC 2013
02:37 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043, oops'
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Fri Mar 22 02:30:31 UTC 2013
01:28 ori-l: Finished correcting E3Experiments extension's entry point from 'Experiments.php' (which does not match the submodule name) to 'E3Experiments.php'. Updated extension-list, CommonSettings.php, and the extension itself in both wmf11 and 12.
01:24 logmsgbot: olivneh synchronized wmf-config/extension-list 'Remove temporary workaround for E3 entry-point migration (2/2)'
01:23 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Remove temporary workaround for E3 entry-point migration (1/2)'
01:18 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/E3Experiments 'Updating E3Experiment extensions's entry point'
01:18 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/E3Experiments 'Updating E3Experiment extensions's entry point'
01:11 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Temporarily accepting either E3Experiments.php or Experiments.php as extension entry point to facilitate clean-up'
00:55 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'NavigationTiming: log mobile mode'
00:16 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.21wmf12.php 'Remove PHP errors'
00:15 logmsgbot: aaron synchronized php-1.21wmf12/includes/Title.php 'deployed cef327e945e2593fe0291880a7d1976cc5a2f248'
00:09 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Thanks 'syncing Thanks ext'
March 21
23:24 logmsgbot: robh Finished syncing Wikimedia installation... :
23:17 logmsgbot: robh Started syncing Wikimedia installation... :
23:04 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo, PageTriage to master, Add Thanks
22:44 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo, PageTriage to master, Add Thanks
21:50 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Syncing change to CommonSettings for Thanks ext'
21:49 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Syncing change to InitialiseSettings for Thanks ext'
21:46 mutante: running puppet and reloading zuul on gallium. enables hashar's new shiny https://integration.wikimedia.org/zuul/
21:38 RobH: authdns-update for terbium
21:05 RoanKattouw: Added fenari back to /etc/dsh/groups/mediawiki-installation per https://rt.wikimedia.org/Ticket/Display.html?id=4794
21:02 logmsgbot: cmjohnson Finished syncing Wikimedia installation... :
20:58 binasher: pulling mw1070, may have network issues
20:57 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/PageTriage/modules/ext.pageTriage.views.list 'syncing PageTriage js on wmf12'
20:57 hashar: fenari disappeared from dsh group mediawiki-installation :-(
20:52 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
20:51 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
20:48 awight: payments cluster updated from 18fd11a4 to df3c8dbc9
20:43 logmsgbot: cmjohnson Started syncing Wikimedia installation... :
20:37 LeslieCarr: restarting gmond on mw1153
20:19 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
20:19 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming 'Enable NavigationTiming on mobile alpha and beta'
20:19 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
20:19 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'Enable NavigationTiming on mobile alpha and beta'
19:53 logmsgbot: mflaschen synchronized php-1.21wmf11/extensions/GuidedTour/ 'E3 deploy'
19:52 logmsgbot: mflaschen synchronized php-1.21wmf11/extensions/GettingStarted/ 'E3 deploy'
19:52 logmsgbot: mflaschen synchronized php-1.21wmf11/extensions/E3Experiments/ 'E3 deploy'
19:49 logmsgbot: mflaschen synchronized wmf-config/CommonSettings.php 'E3 deployment'
17:39 logmsgbot: csteipp rebuilt wikiversions.cdb and synchronized wikiversions files: Update test2wiki to 1.21wmf12
17:32 logmsgbot: csteipp synchronized php-1.21wmf12/includes/Title.php 'update for loadPageData()'
17:24 notpeter: running puppet on all lucid nodes via salt
17:23 mutante-away: purging wikispecies.net from squid
17:08 notpeter: install timeout package on all lucid nodes to get puppet back on track
12:49 mark: Rebooting palladium with hyperthreading disabled
11:27 mark: Rebooting niobium with hyperthreading disabled
09:29 hashar: created git repo operations/debs/python-statsd
09:15 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Add 'wikidata.org' to $wgCrossSiteAJAXdomains to allow CORS'
09:14 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Add 'wikidata.org' to $wgCrossSiteAJAXdomains to allow CORS'
06:34 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
06:34 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
06:32 Tim: reverting all wikis to 1.21wmf11 due to move page being totally broken (bug 46397)
06:32 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
06:32 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
01:10 logmsgbot: catrope synchronized php-1.21wmf12/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'
01:10 logmsgbot: catrope synchronized php-1.21wmf12/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'
01:10 logmsgbot: catrope synchronized php-1.21wmf11/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'
01:10 logmsgbot: catrope synchronized php-1.21wmf11/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'
March 20
23:57 RobH: yay peter!
23:56 notpeter: freeing up roughly 500G of ram across our cluster by calling puppet via cron :)
23:52 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning es100[69]'
23:52 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning es100[69]'
23:51 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s2 servers'
23:51 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s2 servers'
23:36 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling es100[69]'
23:36 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling es100[69]'
23:36 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s1 servers'
23:35 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s1 servers'
23:31 RobH: authdns-update for lanthanum
22:45 RobH: ignore mw1209+ they arent deployed
22:04 LeslieCarr: restarting ircecho on fenari to ensure logmsgbot is on operations and tech irc channels
22:03 ottomata: stopping puppet on gadolinium to test for packet loss weirdness
22:01 pgehres: pgehres synchronized php-1.21wmf12/includes/GlobalFunctions.php 'Fix wfWaitForSlaves() so the parameter actually works correctly'
22:01 logmsgbot: pgehres synchronized php-1.21wmf11/includes/GlobalFunctions.php 'Fix wfWaitForSlaves() so the parameter actually works correctly'
21:35 LeslieCarr: restarted ircecho on fenari
21:01 Krinkle: asher synchronized wmf-config/db-eqiad.php 'pulling db1049 from s1'
20:53 pgehres: pgehres synchronized php-1.21wmf11/extensions/CentralAuth 'Updating CentralAuth to master'
20:53 pgehres: pgehres synchronized php-1.21wmf12/extensions/CentralAuth 'Updating CentralAuth to master'
20:42 Reedy: reedy synchronized wmf-config/InitialiseSettings.php
19:59 mutante: DNS update - remove zuul CNAME
19:53 Reedy: wikibooks to 1.21wmf12
19:51 Reedy: sync'd wmf-config/InitialiseSettings.php
19:50 Reedy: wikiversity and wikiquote to 1.21wmf12
19:49 Reedy: wikivoyage and wiktionary to 1.21wmf12
19:47 Reedy: wikinews and wikisource to 1.21wmf12
19:47 Reedy: Synchroni[sz]ed php-1.21wmf12/extensions for Wikidata deploy
19:46 Reedy: special and wikimedia dblists to 1.21wmf12
19:46 Reedy: wikimania, private and fishbowl dblists to 1.21wmf12
19:45 Reedy: closed.dblist to 1.21wmf12
19:38 RobH: sync-common on mw23
19:36 mutante: restarting hung apache on mw1202
19:28 LeslieCarr: sync-common on mw1085
19:24 mutante: .tar.gz'ing /var/mwdocs on formey (SVN) and then manually delete it
19:18 LeslieCarr: upgrading mw118* and then restarting apache2 on these boxes
19:11 LeslieCarr: installing package upgrades on mw1170-1179
19:10 mutante: installing package upgrades on formey
18:50 mutante: apache-graceful-all, remove boards.wm redirect
16:03 ottomata: added python-voluptuous package to apt repo
15:12 mark: Fixed purging for all sites but eqiad (which wasn't broken)
07:56 hashar: gallium: reloaded apache2 to make sure recent changes are applied.
02:44 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Mar 20 02:44:03 UTC 2013
02:32 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Wed Mar 20 02:32:16 UTC 2013
March 19
23:52 K4-713: synchronized payments cluster to fe4fb96bea1378e
23:33 mutante: DNS update - add wikiartpedia.org
23:30 logmsgbot: dzahn gracefulled all apaches
23:08 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Re-enable AFTv5 on dewiki/testwiki'
22:38 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
22:36 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
22:33 ottomata: adding udp-filter_0.3.22 for precise to apt repo (lucid version already was there)
22:17 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
22:15 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
21:52 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
21:52 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
21:37 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:25 mutante: installing package upgrades on kaulen
21:24 mutante: installing package upgrades on sockpuppet
21:13 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
20:36 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5/SpecialArticleFeedbackv5.php 'Touched AFT Special page file'
20:35 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/SpecialArticleFeedbackv5.php 'Touched AFT Special page file'
19:42 ottomata1: added gadolinium as a udp2log host on frontend caches
19:14 logmsgbot: mlitn synchronized wmf-config/CommonSettings.php 're-include AFTv5'
19:08 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
19:07 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
19:03 paravoid: adding ruby-multi-json (universe), ruby-parallel, ruby-dimensions, ruby-execjs, ruby-jsduck (main) to apt
18:46 mutante: dist-upgrade on gallium
18:03 logmsgbot: robh synchronized docroot
18:01 logmsgbot: robh synchronized docroot
17:21 mark: Ended bits.esams performance testing
16:29 mark: Starting load testing with esams bits servers
16:20 mark: Rebooted cp3019 with hyperthreading and DAPC disabled
16:16 logmsgbot: krinkle synchronized extract2.php
14:29 mark: Rebooting arsenic with hyperthreading disabled
12:54 Tim: on professor: restarted collector
08:46 hashar: gallium: killed stalled puppet agents. Locked on apt-get and update-java-alt
08:41 hashar: gallium: restarted puppet and zuul
08:38 hashar: gallium: stopped zuul and upgrading
08:37 hashar: gallium: stopped puppet to deploy Zuul whenever the queue is empty
08:36 hashar: zuul : giving results processing priority over enqueued jobs. Cherry picked 263fba9 from upstream. That should resolve Zuul being slow to report back to Gerrit bug 46176
03:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
03:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
03:12 Tim: started incremental updater on searchidx1001 and searchidx2, apparently has not been running since March 13
02:50 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Mar 19 02:50:19 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Tue Mar 19 02:29:02 UTC 2013
01:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:22 Ryan_Lane: adding dns entries for labstore systems in eqiad
00:08 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning servers'
March 18
23:34 binasher: i <3 running "dpkg -P mysql-server-5.1 mysqlfb-client-5.1 mysqlfb-common mysqlfb-server-5.1 mysqlfb-server-core-5.1 libmysqlfbclient16 libmysqlclient16"
23:11 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db10[10-11],[26-28] at low weights'
22:45 binasher: converted rogue s3 tables (moodbar_feedback, trackbacks, transcache) to innodb
22:28 ottomata: added python-flask-login-0.1.2-1 to apt
22:23 logmsgbot: robh gracefulled all apaches
22:20 LeslieCarr: intradatacenter link is flapping , switching links, this may cause some higher latency
22:14 logmsgbot: robh gracefulled all apaches
22:11 logmsgbot: robh gracefulled all apaches
21:58 mutante: purging wikimediafoundation.org from squid
21:57 logmsgbot: dzahn gracefulled all apaches
21:56 logmsgbot: robh gracefulled all apaches
21:36 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling a db from s3-7 for upgrade'
21:30 paravoid: authdns-update: removing *.ts.wikimedia.org records
21:29 logmsgbot: dzahn gracefulled all apaches
21:25 binasher: ran "ddsh -g apaches -cM '/etc/init.d/apache2 start'"
21:24 RobH: authdns-update rt4683
21:22 RobH: gracefuling all apaches resulted in rendering cluster overload, they didnt restart apache, odd
21:21 logmsgbot: robh synchronized docroot
21:18 LeslieCarr: service apache2 start on the mw-eqiad group via dsh
21:14 binasher: started apache on all imagescalers
21:00 logmsgbot: robh gracefulled all apaches
21:00 RobH: troubleshooting erros in apache restart script
20:59 logmsgbot: robh gracefulled all apaches
20:36 RobH: authdns-update rt4694
20:20 RobH: authdns-update
20:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
20:00 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for 1.21wmf12
19:44 ^demon: jetty freaked out again, forcing a gerrit restart
19:32 binasher: upgraded all coredb mariadb replicas to 5.5.30
19:25 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for 1.21wmf12
19:08 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043 db1009'
19:04 logmsgbot: reedy synchronized docroot
19:03 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'db1050 for special s1'
18:56 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043, db1009 for upgrade'
18:40 logmsgbot: reedy synchronized php-1.21wmf12 'Initial sync of php-1.21wmf12'
18:16 awight: payments cluster updated from c19cc66 to fe4fb96b
18:06 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
18:03 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php 'Removed old testing cruft'
16:19 cmjohnson1: removing/replacing disk 4 ms-be1004
16:01 hashar: Hard restarted Zuul which was dead locked again :-(
13:23 paravoid: depooled, upgraded, restarted and pooled again all ms-fe[1-4]
13:02 paravoid: depooling ms-fe1 for new swift version (patched 1.7.4)
11:27 apergos: restarted both the varnishncsas on niobium, they were giant again
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Mon Mar 18 02:30:03 UTC 2013
March 17
02:31 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sun Mar 17 02:31:16 UTC 2013
March 16
14:58 logmsgbot: reedy synchronized php-1.21wmf11/extensions/EducationProgram
14:48 logmsgbot: reedy synchronized php-1.21wmf11/extensions/EducationProgram
14:40 logmsgbot: reedy synchronized php-1.21wmf11/extensions/EducationProgram
13:07 hashar: Mar 16, 2013 1:06:47 PM Jenkins is fully up and running
12:55 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/EducationProgram/includes/pages/EducationPage.php 'Fix fatal error in EducationProgram (Bug 46188)'
12:33 hashar: restarting Jenkins on a quiet day to have it using umask 0002 (aka g+w). That needs a full restart since the umask is set by /usr/bin/daemon See 53990 .
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sat Mar 16 02:29:19 UTC 2013
March 15
23:24 logmsgbot: laner synchronized wmf-config/CommonSettings.php 'Disabling aft5'
22:45 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning es1007,10 at lower weights'
22:35 binasher: rebooted es1007,1010 for kernel upgrades
22:33 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'temporarily pulling es1007, es1010'
22:23 Ryan_Lane: hid domain dropdown for account creation and login on wikitech
22:23 Ryan_Lane: changed wikitech configuration to require email address on account creation
22:23 Ryan_Lane: deployed change 54093 to wikitech
22:03 hashar: Zuul : opened a bug about this week slowness, breakage: https://bugzilla.wikimedia.org/show_bug.cgi?id=46176
21:51 logmsgbot: aaron synchronized php-1.21wmf11/includes 'deployed 08295adcdc4a4130954f52fb1eb5377638cd97bc'
21:36 binasher: db59 -> mariadb 5.5.30
21:30 hashar: restarted Zuul with the safe version ff79197
21:28 hashar: downgrading Zuul to safe version ff79197 (aka stop doing `git remote update` on ref-update events)
21:25 binasher: added new build of mariadb 5.5.30 to precise-wikimedia on brewster
21:20 hashar: hard restarted Zuul. It was locked down again :/
21:18 mutante: installing package upgrades on gallium
21:14 hashar: restarting Zuul, it is locked somehow. Definitely need to update it :(
21:09 RobH: professor services seem back up
20:51 RobH: professor shutting down for fan swap, short downtime (hopefully)
20:47 hashar: integration.mediawiki.org is not redirecting to the new integration.wikimedia.org (apache conf is disabled). Patch in 54068
20:33 mutante: puppet runs on neon broken due to mysql puppet issue - mysql-client-5.5 can not create alias mysql-client: object already exists at /var/lib/git/operations/puppet/manifests/mysql.pp:513
20:31 mutante: restarting pdns on ns0
20:30 mutante: DNS update - removing ipv4/ipv6/ipv6and4/results.labs.wm
19:37 mutante: maerlant - re-adding 2620:0:862:1::80:2 to eth0, starting nginx
19:22 sbernardin: taking down raskin for decommissioning
19:13 mutante: maerlant - dpkg: error processing ganglia-monitor | backing up /var/www/ to tridge /data/other then deleting contents | removing sites-enabled/ipv6and4 | nginx can't bind to [2620:0:862:1::80:2]:443
18:50 mutante: installing more package upgrades on maerlant (wikimedia-lvs-realserver, libc6, nginx...)
18:46 mutante: installing package upgrades on formey
18:44 mutante: fixing broken dpkg/APT on maerlant - dpkg --configure -a , re-run puppet and let it install package upgrades
18:34 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/
18:19 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/
18:19 notpeter: ran puppetd --enable on all mw hosts
18:15 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/
18:07 logmsgbot: mlitn Finished syncing Wikimedia installation... : Update ArticleFeedbackv5 to master
17:52 logmsgbot: mlitn Started syncing Wikimedia installation... : Update ArticleFeedbackv5 to master
17:36 hashar: jenkins: updated jenkins configuration to have its URL default to wikimedia.org domain instead of the legacy mediawiki.org
17:22 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/ 'Update ArticleFeedbackv5 to master'
17:21 hashar: gallium : reloading Zuul so it reports the status url with the wikimedia.org domain
17:19 hashar: integration website going down while we deploy a new layout.
17:03 hashar: gallium : restarted puppet
17:03 hashar: gallium : update /srv/ to latest version of integration/docroot.git ( aaa4b0c ) and moved all content from /srv/org/mediawiki/integration to the new /srv/org/wikimedia/integration
17:00 hashar: stopped puppet on gallium to fix some permissions before puppet changes are merged in and applied.
05:09 Tim: restarting squid on cp1010 for squid redirector test
05:06 Tim: testing new squid redirector
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Fri Mar 15 02:30:02 UTC 2013
02:16 logmsgbot: reedy synchronized docroot
02:13 logmsgbot: dzahn gracefulled all apaches
01:53 logmsgbot: reedy synchronized wmf-config/
01:52 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
01:52 logmsgbot: reedy gracefulled all apaches
01:44 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/GettingStarted/SpecialGettingStarted.php 'Fix deployed bug in check for editability'
01:44 logmsgbot: reedy synchronized wmf-config/
01:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
01:35 logmsgbot: reedy synchronized wmf-config/
01:35 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
01:34 logmsgbot: reedy synchronized docroot
01:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
01:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
01:21 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/GettingStarted/CategoryRoulette.php 'Fix to use single-parameter version of SRANDMEMBER'
01:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
01:13 logmsgbot: dzahn gracefulled all apaches
01:12 logmsgbot: reedy synchronized docroot
01:10 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'Missed a file in earlier sync'
00:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'chapcomwiki readonly'
00:47 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming/modules/ext.navigationTiming.js 'Additional fields for NavTiming'
00:43 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deployment for Mobile and E3
00:17 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deployment for Mobile and E3
March 14
23:48 mutante: DNS update - add affcom in prep. to replace chapcom (bug 39482)
23:47 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Updating MF copyright logo path for enwiki'
23:47 logmsgbot: awjrichards synchronized php-1.21wmf11/extensions/MobileFrontend/ 'Updating MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-03-14'
23:44 mutante: installing package upgrades on sockpuppet
23:30 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Updating MF copyright logo for testwiki'
22:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:43 mutante: pa.us.wikimedia.org now redirects to pa-us.wikimedia.org (Wikimedia Pennsylvania) (no sub.sub domains)
22:41 logmsgbot: dzahn gracefulled all apaches
22:34 mutante: disabling notifications for puppet freshness on celsus
22:14 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling NavigationTiming at 1:10,000 sampling factor across cluster.'
22:02 logmsgbot: olivneh synchronized wmf-config 'Enable NavigationTiming on test2'
21:57 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'Update of extension to use EventLogging'
21:51 hashar: Gerrit took roughly 5 minutes to merge the change https://gerrit.wikimedia.org/r/#/c/53412/ ( mediawiki/core ) which means Zuul does nothing during that time waiting for the merge to happen.
21:48 hashar: apparently Gerrit takes ages to submit a change that in turn cause a hard lock on Zuul which is waiting for the merge to happen.
21:39 hashar: Zuul is hanging out not reporting changes :/
21:01 mutante: DNS update - add pa-us wiki to replace pa.us
20:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:50 cmjohnson1: authdns update (labsdb1002-3 vlan)
20:45 Ryan_Lane: deleting the jenkins user from ldap
20:36 mutante: DNS update - add zuul as CNAME for gallium
20:34 Ryan_Lane: installed WikiLove on wikitech
20:30 ori-l: Added article edit event to EventLogging, which upped events/sec from ~3.5 to ~30. Expect increased load on vanadium and db1047; bits caches unaffected.
20:20 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/EventLogging 'Log ['HTTP_HOST'] as webHost'
20:05 logmsgbot: olivneh synchronized php-1.21wmf11/includes/WikiPage.php 'Adds CategoryAfterPageAdded / CategoryAfterPageRemoved hooks'
19:32 logmsgbot: aaron synchronized php-1.21wmf11/includes/job/jobs 'deployed 6f76ede163cb114724bce7a1c1b8938e1e30606f '
18:49 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Enable wgEnableAsyncUploads only on testwikis for now'
18:44 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/ 'Update ArticleFeedbackv5 to master'
18:41 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1009 to full weight'
18:32 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1009 at a low weight for warmup'
18:12 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Enabled $wgEnableAsyncUploads on all wikis'
18:08 binasher: cleaned up myisam cruft (unused t���ables) on s2
17:49 logmsgbot: aaron synchronized wmf-config 'Enabled redis job aggregator fully'
17:48 cmjohnson1: labsdb1001 powering down
17:42 logmsgbot: aaron synchronized wmf-config 'Set redis job aggregator on for testwiki.'
17:27 logmsgbot: reedy synchronized php-1.21wmf11/extensions/Math/
17:24 binasher: shutting down mysql on db1009, upgrading to mariadb
16:59 cmjohnson1: powering down labsdb1002 and labsdb1003
16:59 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1009 for mariadb migration'
16:57 cmjohnson1: addin mw1041 back to dsh groups
16:26 logmsgbot: mlitn synchronized wmf-config 'Prepare AFTv5 config for deployment new features'
13:15 mark: Set OSPF cost to 10 on csw1-esams:ve7 to facilitate ip multipath
13:15 mark: Fixed OSPF on cr2-knams:xe-0/0/0 - csw1-esams:e8/2 (earlier)
12:17 hashar: gallium setting up /srv/ as a copy of integration/docroot.git
09:03 apergos: restarted varnishncsa.vanadium on niobium, it was using 9gb memory or so
02:25 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Thu Mar 14 02:25:19 UTC 2013
02:21 hashar: jenkins: made Zuul to block changes on operations/debs/ircecho whenever pep8/pyflakes fails.
01:35 mutante: running puppet on celsus to investigate puppet freshness reports from icinga - it started parsoid, but also celsus is a Wikimedia DECOMMISSIONED server (base::decommissioned).
01:10 logmsgbot: reedy synchronized php-1.21wmf11/cache/l10n/
01:00 logmsgbot: reedy synchronized php-1.21wmf11/extensions/Scribunto
00:53 logmsgbot: reedy synchronized wmf-config/
March 13
23:44 mutante: installing package upgrades on zirconium
23:41 logmsgbot: demon Finished syncing Wikimedia installation... :
23:38 Ryan_Lane: deploying change 53464 to OpenStackManager on wikitech
23:23 RobH: no one sync apaches, we wre doing crazy magic on them
23:17 logmsgbot: demon Started syncing Wikimedia installation... :
23:05 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqiad. upgrades over.'
23:00 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqiad. upgrades over.'
22:53 logmsgbot: reedy synchronized wmf-config/
22:44 logmsgbot: demon synchronized php-1.21wmf11/extensions/Scribunto/ 'Updating Scribunto to master'
22:30 logmsgbot: robh gracefulled all apaches
21:46 awight: payments cluster updated to 61d3f1f
21:43 pp-pdf1: upgraded mwlib to 0.15.3
21:38 Tim: on professor: restarted collector
21:22 RobH: repooling all ssl servers (except broken ssl3004) in all sites. new unified certificate is now in use.
21:15 RobH: depooling ssl servers 3 and 4 in each dc site
21:14 RobH: repooling the first two ssl servers per site
21:08 logmsgbot: py synchronized wmf-config/lucene-production.php 'temp moving all earch traffic to pmtpa for upgrades in eqiad'
21:05 RobH: setting first 2 ssl servers per dc site to false in pybal
20:21 Ryan_Lane: repooling ssl1001
20:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable collection'
20:16 Ryan_Lane: depooling ssl1001
20:08 LeslieCarr: cp1022 unresponsive, power cycling
19:44 LeslieCarr: switched preferred path between esams and eqiad
19:41 LeslieCarr: deploying a possibly mobile front end traffic effecting ipv6 apache request firewall filter
19:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable Collection extension'
19:08 LeslieCarr: site was popedotted !
18:47 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqaid'
18:31 logmsgbot: py synchronized wmf-config/lucene-production.php 'temp moving all search traffic to pmtpa for upgrades in eqiad'
18:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable scribunto EVERYWHERE'
18:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf11
17:23 pp-pdf1: downgrade to 0.14.3
16:39 pp-pdf1: upgraded mwlib to 0.15.1.
16:32 apergos: cleaned up borked puppet agent on ms-be12 and re-enabled/restarted it
12:30 logmsgbot: anomie synchronized php-1.21wmf11/extensions/Scribunto/common/Hooks.php 'Fix for bug 46031'
12:30 logmsgbot: anomie synchronized php-1.21wmf11/extensions/Scribunto/common/Common.php 'Fix for bug 46031'
12:07 Danny_B: reenabled mail delivery to wikibugs 30 mins ago (disabled automatically for excessive bounces)
11:42 apergos: restarted morebots
01:10 Tim: running updateCollation.php on all uca-* wikis in 6 screens on hume
00:45 logmsgbot: dzahn gracefulled all apaches
00:39 Ryan_Lane: depooling ssl1 ssl2 ssl1001 ssl1002 ssl3001 ssl3002
00:35 Ryan_Lane: pooling ssl3 ssl4 ssl1003 ssl1004 ssl3003
00:31 Ryan_Lane: depooling ssl3 ssl4 ssl1003 ssl1004 ssl3003
00:30 Ryan_Lane: pooling ssl1 ssl2 ssl1001 ssl1002 ssl3001 ssl3002
00:16 Ryan_Lane: depooling ssl1 ssl2 ssl1001 ssl1002 ssl3001 ssl3002
00:12 Tim: manually creating missing /etc/apache2/wmf symlinks on servers that lack them
00:07 logmsgbot: dzahn gracefulled all apaches
March 12
23:39 Tim: updating PHP to php5-3.5+wmf1 for new ICU
23:39 notpeter: upgrading pmtpa search nodes to lucene-search-2 2.1.7wm1
23:34 logmsgbot: tstarling synchronized php-1.21wmf11/maintenance/updateCollation.php
23:34 logmsgbot: tstarling synchronized php-1.21wmf10/maintenance/updateCollation.php
22:33 Ryan_Lane: pooling ssl3003 ssl3 ssl4 ssl1003 ssl1004
22:29 Ryan_Lane: depooling ssl3003 ssl3 ssl4 ssl1003 ssl1004
22:28 Ryan_Lane: pooling ssl3001 ssl3002 ssl1 ssl2 ssl1001 and ssl1002
22:23 mutante: pushing new version 2.7 of wikimedia-task-appserver to wikipedia-precise repo (to fix apache-sanity-check)
22:21 Ryan_Lane: depooling ssl3001 ssl3002 ssl1 ssl2 ssl1001 and ssl1002
21:23 Ryan_Lane: seems that the unified certificate is missing mediawiki.org, depooling the ssl servers that are serving it
21:22 Ryan_Lane: installing the new unified ssl certificate on all ssl servers
19:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:54 RobH: authdns-update for europium to gadolinium ip
18:47 RobH: europium shuttin down
18:43 mark: Imported ganglia 3.5.0 packages into the precise-wikimedia APT repository
18:19 RobH: europium reclaimed per rt4689
18:02 RobH: en-wp.com/org now redirect to en.wikipedia per rt 4669
18:00 RobH: ran apache restart as myself, not root, had a bunch of fails. restarted them as root, hopefully nothing noticable to users
17:58 logmsgbot: robh gracefulled all apaches
17:53 Jeff_Green: live-hack trivial OTRS patch to support filtering on X-Spam-Score for RT #4713, will check in once project is copied to git
17:42 logmsgbot: anomie synchronized php-1.21wmf11/extensions/Scribunto 'Update Scribunto to master'
17:29 logmsgbot: reedy synchronized php-1.21wmf10/includes/Collation.php
17:28 logmsgbot: reedy synchronized php-1.21wmf11/includes/Collation.php
15:42 paravoid: apache graceful to fix bz 46018 / RT 4695
15:39 logmsgbot: root gracefulled all apaches
15:32 hashar: Restarted Zuul process. Was stuck somehow waiting for a job that has been canceled.
13:30 hashar: Jenkins restarted!
12:55 hashar: gallium : upgrading Jenkins (unscheduled)
08:32 hashar: jenkins: live hacked the ant build script for MediaWiki jobs. Now using the local replicated git directory instead of a handmade replication.
05:03 logmsgbot: tstarling synchronized php-1.21wmf10/extensions/Math
05:02 logmsgbot: tstarling synchronized php-1.21wmf11/extensions/Math
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Tue Mar 12 02:52:43 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Tue Mar 12 02:29:54 UTC 2013
02:16 Ryan_Lane: deploying change 53248 and 52951 to OpenStackManager on wikitech
02:10 Ryan_Lane: deployed change 53304 to OATHAuth on wikitech
00:38 logmsgbot: kaldari synchronized php-1.21wmf11/extensions/UploadWizard/ 'deploying some bugfixes for UploadWizard'
00:15 awight: payments cluster updated to 207a112b5b36a9e125999acc6f4098d9058ef594
March 11
23:48 AaronSchulz: syncFileBackend.php Running for ceph for all wikis now
23:48 AaronSchulz: copyFileBackend.php finished for ceph for all originals and transcodes (plus math/captcha)
23:38 logmsgbot: aaron synchronized php-1.21wmf11/includes/clientpool/RedisConnectionPool.php 'deployed 5af8702a4b4208ac42fab2fb36da69d13f3e3b92'
23:37 logmsgbot: aaron synchronized php-1.21wmf10/includes/clientpool/RedisConnectionPool.php 'deployed b9fdedc57295a4dd1da684859871aeb734bf70fe'
23:10 Ryan_Lane: adminbot upgraded to 1.6.3
23:09 Ryan_Lane: upgrading adminbot on wikitech-static
22:31 LeslieCarr: upgrading ircecho on neon
22:30 Ryan_Lane: upgrading ircecho on manganese
21:20 mutante: gracefulling eqiad Apaches - wikiquote.net now redirects to wikiquote.org
21:19 logmsgbot: dzahn gracefulled all apaches
21:07 Ryan_Lane: deployed change 53267 to OpenStackManager on wikitech
20:59 logmsgbot: reedy synchronized php-1.21wmf10/cache/interwiki.cdb 'Updating 1.21wmf10 interwiki cache'
20:59 logmsgbot: reedy synchronized php-1.21wmf11/cache/interwiki.cdb 'Updating 1.21wmf11 interwiki cache'
20:30 ^demon: Upgraded gerrit to 2.5.2-1636-g353e384, all's well
20:18 Ryan_Lane: deploying change 52950 to OpenStackManager on wikitech
19:22 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf11
19:05 mutante: patching Bugzilla for security - RT-4553
17:45 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
17:34 mwalker: Shields up! Payments deploy from a6fe052a73f1421028b44305908ec6984dbc50fa to head for Japanese localization updates.
17:29 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
17:19 RobH: search23 shutting down for mb sap per rt4323, has been disabled in pybal
17:18 RobH: search23 out of search pool1, other servers in pool online
17:17 RobH: setting search23 to false for mb swap
17:10 sbernardin: taking down search23 for MB replacement: https://rt.wikimedia.org/Ticket/Display.html?id=3423
16:48 logmsgbot_: aaron synchronized php-1.21wmf11/includes/limit.sh 'deployed e77397fc1de324dfc6d7b515e9440bd564311dd9'
15:57 ottomata: deployed new webstatscollector filter on locke. This replaces spaces in urls.
11:39 apergos: ook that was wrong started apache2 on stafford (puppet uses passenger), now things look ok
10:51 apergos: that would be 10.5. can't add.
10:50 apergos: shot old puppet agents hanging around on stafford, also old apaches from dead puppet master, restarted puppet master, it died around 6.5 hours ago
02:52 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Mon Mar 11 02:52:06 UTC 2013
02:43 logmsgbot_: krinkle synchronized wmf-config/CommonSettings.php 'I7c8174bcf6ca411014a712a6416d3a74e9fecb3d'
02:41 logmsgbot_: krinkle synchronized wmf-config/InitialiseSettings.php 'I7c8174bcf6ca411014a712a6416d3a74e9fecb3d'
02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Mon Mar 11 02:29:27 UTC 2013
March 10
22:19 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild message cache
21:59 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild message cache
21:10 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/CentralAuth/
21:00 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
March 9
02:54 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Sat Mar 9 02:54:23 UTC 2013
02:37 mutante: deploying blog WMBlog.php changes (gerrit 28835, gerrit 29823)
02:33 Reedy: Created cn_notice_log on enwiki. Not sure it's actually needed, but there's dberrors appearing here and htere
02:31 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Sat Mar 9 02:31:06 UTC 2013
02:01 mutante: deploying WMBlog.php changes for EventLogging (gerrit 52945)
01:26 logmsgbot_: olivneh synchronized wmf-config/InitialiseSettings.php '(Change If7ab3d016 / bug 45830) Separate slow parse log destination for private wikis'
01:13 Ryan_Lane: cherry-picking in change 52950 on wikitech
01:12 Ryan_Lane: scratch that
01:11 Ryan_Lane: deploying OATHAuth master to wikitech
00:32 mutante: deleting squid logs on brewster to prevent out-of-disk, and setting logrotate to compress
00:25 Ryan_Lane: upping OpenStackManager to master on wikitech
00:22 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'removing a second node per shard for streaming dumps. stopping slave'
00:00 Ryan_Lane: upping OpenStackManager to master on wikitech
March 8
22:28 logmsgbot_: reedy synchronized wmf-config/
22:28 logmsgbot_: py synchronized wmf-config/lucene-production.php 'moving pool4 and pool5 search traffic back to eqiad'
22:21 logmsgbot_: reedy synchronized php-1.21wmf10/includes/Collation.php
21:47 awight: payments cluster update to a6fe052a73
20:29 Ryan_Lane: enabling quorum feature for all gluster volumes
19:55 RobH: authdns-update, why? because i can.
19:34 Ryan_Lane: added Babel extension to wikitech
19:20 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'removing a second node per shard for streaming dumps. stopping slave'
15:35 ottomata: adding metrics.wikimedia.org domain, running authdns-update on dobson
06:21 Tim: updated luasandbox to version 1.4 (with pauseUsageTimer)
04:01 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/Translate
02:53 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Fri Mar 8 02:52:54 UTC 2013
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Fri Mar 8 02:28:34 UTC 2013
01:27 mutante: installing package upgrades on ekrem
01:12 logmsgbot_: catrope synchronized wmf-config/InitialiseSettings.php '20388a06 - Normalize VE config on testwiki'
00:57 mutante: added Set(@ReferrerWhitelist, qw(rt.wikimedia.org:443 rt.wikimedia.org:80));
00:55 Ryan_Lane: deploying change 52592 to wikitech
00:55 mutante: killall lighttpd and restart lighttpd and apache on streber
00:19 logmsgbot_: kaldari Finished syncing Wikimedia installation... :
00:06 mutante: installing openjdk/icedtea jre and other package upgrades on gallium (jenkins)
00:05 LeslieCarr: upgrading cp1001
March 7
23:54 logmsgbot_: kaldari Started syncing Wikimedia installation... :
23:47 LeslieCarr: doing tmh upgrades
23:47 mutante: bravely rebooting streber - expect short RT and Observium outage
23:25 LeslieCarr: upgrading solr boxes
23:14 mutante: installing kernel on streber
23:11 ^demon: gerrit: disabling hooks-its plugin for the time being. It's working (yay) but it's overly spammy until we fix it next week.
23:04 LeslieCarr: dist-upgrading pmtpa memcache
23:04 mutante: dist-upgrading zhen
22:38 mutante: rebooting yvon
22:34 mutante: installing package upgrades on yvon despite (defunct apt-get procs)
22:31 mutante: salt pkg.upgrade on yvon
22:28 Ryan_Lane: rebooting marmontel
22:25 mutante: salt pkg.upgrade on spence
22:21 mutante: salt pkg.upgrade on zirconium
22:18 LeslieCarr: upgrading all analytics machiens
22:06 mutante: reboot formey after dist-upgrade via salt
20:05 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'fa0a56a89 - parsoidcache VIP in pmtpa'
19:37 logmsgbot_: bsitu synchronized wmf-config/CommonSettings.php 'Add Eventlogging configuration to Echo'
19:17 logmsgbot_: pgehres synchronized php-1.21wmf11/extensions/CentralNotice/ 'Updating CEntralNotice to master, resolving bug 45846'
18:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
18:39 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
17:36 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
16:07 cmjohnson1: reseating disk4 on ms-be1004
15:28 mark: Reinstalled cp3003
15:00 mark: Restarted gmetad on nickel
02:53 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Thu Mar 7 02:53:50 UTC 2013
02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Thu Mar 7 02:29:38 UTC 2013
01:53 mutante: installing package upgrades on sodium
00:54 RobH: kicked icinga
00:49 LeslieCarr: restarted pybal on lvs3
00:40 RobH: puppet locked on ms1004, killall puppet, refired
00:39 RobH: puppet halted run on cp1003, left in locked state, killed all puppet instances and refired, cleared.
00:33 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/RSS
00:15 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
March 6
23:19 LeslieCarr: reinstalling celsus as an internal host
23:07 logmsgbot_: reedy synchronized php-1.21wmf10/resources/ 'touch'
23:06 logmsgbot_: reedy synchronized php-1.21wmf11/resources/ 'touch'
23:05 logmsgbot_: reedy synchronized wmf-config 'touch'
22:54 logmsgbot_: reedy synchronized wmf-config/
22:53 logmsgbot_: reedy synchronized wmf-config/
22:40 mutante: starting haproxy on brewster
22:39 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild message cache for Wikibase deploy
22:39 notpeter: reslaving db36 and db38
22:29 RobH: authdns-update
22:20 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/Wikibase
22:20 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild message cache for Wikibase deploy
22:11 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
22:10 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
22:09 logmsgbot_: reedy synchronized wikidataclient.dblist
22:07 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
22:02 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
21:55 logmsgbot_: reedy cleared profiling data
21:55 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Wikibase Client on all wikipedias'
21:41 awight: updated payments cluster to 39a265489
21:16 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/
20:54 logmsgbot_: reedy synchronized wmf-config/
20:38 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/DataValues/
20:29 Reedy: Creating and populating site_identifiers table on all Wikipedias
20:28 Reedy: Creating and populating sites table on all Wikipedias
20:27 logmsgbot_: olivneh synchronized wmf-config/CommonSettings.php 'Adding redis host config. var for GettingStarted'
20:12 logmsgbot_: reedy synchronized wmf-config/
20:07 logmsgbot_: reedy synchronized wmf-config/
20:04 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/Wikibase
20:04 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/DataValues
20:03 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/Diff
19:57 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/Wikibase
19:56 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/DataValues
19:43 notpeter: restarting all jobrunners
19:34 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
19:29 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
19:28 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource to 1.21wmf11
19:25 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: aawiki back to 1.21wmf10 for Aaron
19:21 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
19:20 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia and wikiquote to 1.21wmf11
19:18 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikinews to 1.21wmf11
19:15 Reedy: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage, wikiversity and wiktionary to 1.21wmf11
19:10 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Special wikis to 1.21wmf11
19:07 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Closed, private and fishbowl to 1.21wmf11
19:03 mutante: gracefulling apaches to remove 1.17 aliases (gerrit 50609)
19:03 logmsgbot_: dzahn gracefulled all apaches
18:58 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
18:47 mutante: regenerating mailman list archives for wmfreqs
18:44 mutante: renaming mailman list 'allhands' to 'wmfreqs' (RT-4640)
18:36 logmsgbot_: aaron synchronized php-1.21wmf11/maintenance/nextJobDB.php 'deployed cbc98b09fb0a13b387d82bafab825608e2bb0b3b'
18:35 logmsgbot_: aaron synchronized php-1.21wmf11/includes/job/JobQueueGroup.php 'deployed cbc98b09fb0a13b387d82bafab825608e2bb0b3b'
18:35 logmsgbot_: aaron synchronized php-1.21wmf11/includes/DefaultSettings.php 'deployed cbc98b09fb0a13b387d82bafab825608e2bb0b3b'
18:10 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'clear config cache'
18:09 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'clear config cache'
18:01 logmsgbot_: hashar synchronized wmf-config 'deploying 52340 adapt Lucene configuration file to support beta'
17:59 hashar: deploying 52340 adapt Lucene configuration file to support beta
07:08 jeremyb_: 02:52:46 <+logmsgbot> !log LocalisationUpdate completed (1.21wmf11) at Wed Mar 6 02:52:46 UTC 2013
07:08 jeremyb_: 02:36:35 <+logmsgbot> !log reedy synchronized wmf-config/InitialiseSettings.php
07:08 jeremyb_: 02:28:21 <+logmsgbot> !log LocalisationUpdate completed (1.21wmf10) at Wed Mar 6 02:28:21 UTC 2013
07:08 jeremyb_: 01:33:50 <+logmsgbot> !log csteipp synchronized wmf-config/CommonSettings.php 'Revert https patch'
07:07 jeremyb_: 01:08:27 <+logmsgbot> !log reedy synchronized wmf-config/CommonSettings.php 'Revert wgSecureLoginDefaultHTTPS'
07:07 Ryan_Lane: restarted morebots
01:10 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Set wgSecureLoginDefaultHTTPS to false for testing'
00:37 binasher: added new deb of twemproxy 0.2.3 built with --enable-debug=log to precise-wikimedia
00:21 logmsgbot_: csteipp synchronized wmf-config/CommonSettings.php 'Enabling HTTPS login'
00:14 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings-labs.php
00:09 hashar: snapshot1002: rsync: change_dir#3 "/apache/common-local" failed: No such file or directory (2)
00:09 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings-labs.php
March 5
23:57 logmsgbot_: py synchronized wmf-config/db-eqiad.php 'increasing db1009 to full weight'
23:54 LeslieCarr: reinstalling constable as an internal host
23:49 RoanKattouw: Removed decommissioned srv226-234 from /etc/dsh/group/mediawiki-installation
23:47 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php '6fd144c44f26 - Use new service IP for Parsoid in eqiad'
23:46 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'repooling previously crashed mariadb pmtpa slaves'
23:36 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'ombudsmenwiki logo'
23:26 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'commenting out one slave per shard that's mysqldumping'
23:18 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
23:15 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
23:13 notpeter: stopping slave and starting mysql dumps on dbs 71, 57, 66, 65, 55, 50, and 68
23:00 logmsgbot_: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
22:40 logmsgbot_: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
22:36 logmsgbot_: py synchronized wmf-config/db-eqiad.php 'repooling db1009 at 25% weight'
22:18 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
22:17 logmsgbot_: reedy synchronized wmf-config/
22:14 LeslieCarr: restarting pybal on lvs3
22:14 LeslieCarr: restarting pybal on lvs4
22:13 LeslieCarr: restarting pybal
22:07 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
22:05 logmsgbot_: reedy synchronized wmf-config/
21:57 LeslieCarr: restarting pybal on lvs1003
21:55 LeslieCarr: restarting pybal on lvs1006
21:44 Coren: tools rejiggered the webproxy config to be smarter about paths not leading to specific tools
21:43 LeslieCarr: adding parsoidcache service to lvs1006
21:42 mutante: gracefulling eqiad Apaches - add ombudsmen.wm (bug 45744)
21:41 logmsgbot_: dzahn gracefulled all apaches
21:35 mutante: restarting pdns on ns2
21:34 mutante: DNS update - add ombudsmen (bug 45744)
20:49 logmsgbot_: reedy synchronized docroot
20:35 mutante: installing package upgrades on hooper
20:26 logmsgbot_: reedy synchronized wmf-config/
20:22 mutante: installing package upgrades on hume
20:00 mutante: wikipedia.ee now redirects to et.wikipedia, wikimedia.ee now redirects to et.wikimedia
19:56 mutante: gracefulling eqiad Apaches, push wiki(m|p)edia.ee redirects
19:56 logmsgbot_: dzahn gracefulled all apaches
19:18 K4-713: updated payments cluster to 9b063ee2e399a3b
18:42 logmsgbot_: mlitn synchronized wmf-config/InitialiseSettings.php 'Disable AFTv4 (entirely) and AFTv5 (leaving only opt-in) on enwiki'
18:04 cmjohnson1: restarting dhcpd3 service brewster
16:55 cmjohnson1: mw1085 powering off to troubleshoot DIMM error
05:35 ori-l: EventLogging db on db1047: ALTER DATABASE log CHARACTER SET utf8 COLLATE utf8_unicode_ci; (was: latin1)
02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Tue Mar 5 02:48:58 UTC 2013
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Tue Mar 5 02:28:16 UTC 2013
02:03 K4-713: updated payments cluster to 5361c795f4bf31980f
00:46 mutante: fixing exim/mail permissions on neon to make Icinga paging work. missing +x on /var/log/exim4 dir
March 4
23:12 mutante: starting dhcp3 on brewster
23:11 LeslieCarr: p.s. bonding on foundry sucks
23:10 LeslieCarr: bringing down virt2 network to bond it
22:18 mutante: gracefulling eqiad Apaches to push indiawikipedia redirect
22:17 logmsgbot_: dzahn gracefulled all apaches
22:03 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'shuffling around pmtpa enwiki slaves'
21:50 LeslieCarr: messing around with the network of virt2 - expect possible pmtpa labs network issues
21:50 ottomata: staring puppet back up on oxygen, packet loss is better after disabling a filter
21:30 cmjohnson1: authdns update
21:17 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf11
21:16 logmsgbot_: reedy synchronized php-1.21wmf11
21:15 logmsgbot_: reedy synchronized php-1.21wmf10
21:14 RobH: shutting down professor for a fan swap, should be a short window
21:11 cmjohnson1: authdns update
21:05 ottomata: stopping puppet on oxygen to see if we can disable filters and reduce packet loss
20:55 Ryan_Lane: restarted icinga
20:53 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild Message Cache for 1.21wmf11
20:52 RobH: had to restart morebots, but it wasnt really dead, opps
20:51 RobH: had to restart icinga
20:51 RobH: Why arent you taking all the !log
20:50 RobH: i hate you morebots
20:28 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild Message Cache for 1.21wmf11
19:55 logmsgbot_: reedy Started syncing Wikimedia installation... : Build 1.21wmf11 message cache
19:41 logmsgbot_: reedy synchronized php-1.21wmf11/
19:34 logmsgbot_: reedy synchronized wmf-config/
19:34 logmsgbot_: reedy synchronized live-1.5/
19:33 logmsgbot_: reedy synchronized docroot
19:30 hashar: on gallium: stopping puppet, cleaning defunct and restarting puppet
19:23 logmsgbot_: reedy synchronized wmf-config/
19:20 mutante: installing package upgrades on sockpuppet
19:13 sbernardin: taking down amaranth for hw replacement: https://rt.wikimedia.org/Ticket/Display.html?id=4497
18:44 cmjohnson1: authdns update
18:17 K4-713: updated payments cluster to 059969c3d29f685c
17:44 cmjohnson1: authdns update
16:56 logmsgbot_: py synchronized wmf-config/db-eqiad.php 'depooling db1009 while investigating error'
16:24 sbernardin: rebooting db70 to reset idrac: https://rt.wikimedia.org/Ticket/Display.html?id=4628
16:14 sbernardin: db27 being taken down for decom per: https://rt.wikimedia.org/Ticket/Display.html?id=4623
10:12 apergos: cleared some space on brewster by rmeoving old squid access/storage logs (2.2gb worth in 3 files!), its root partition was full, restarted squid
10:12 apergos: reinstalled wikimedia-app-taskserver on fenari (how did it get removed?), needed for various cron jobs among other things
02:04 logmsgbot_: LocalisationUpdate failed (1.21wmf10) at Mon Mar 4 02:04:10 UTC 2013
March 3
02:25 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sun Mar 3 02:25:44 UTC 2013
March 2
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sat Mar 2 02:27:35 UTC 2013
01:17 mutante: powercycling emery
00:35 mutante: rebooting emery for kernel upgrade
00:29 mutante: installing package upgrades on emery
March 1
23:59 cmjohnson1: authdns update
23:56 K4-713: updated payments cluster to c8442435a9e2f5c
23:29 notpeter: dropping limesurvey from db9, backup in db9:/a/backup_dumps
22:53 RoanKattouw: Make that apache2-utils, that's the actual name of the package
22:51 RoanKattouw: Temporarily installing apachebench on wtp1004 per Gabriel's request
22:49 RoanKattouw: Depooled wtp1004 so Gabriel can use it for benchmarking
22:46 mwalker: reverting payments to dced5328fd6ae4b395622abc41d2afa0ccad21c8 from 4a3ca6cc36b9d78dbb015c1bf93bbafb14eb0799 because of bug report that some people are not able to make donations due to 'required field city' being missing and then being unable to provide it
22:34 cmjohnson1: authdns update
22:15 cmjohnson1: auth-dns update
22:12 mutante: DNS update - remove status cname
21:56 cmjohnson1: taking wtp1001 down to replace w/new box
21:32 mwalker: pushing 4a3ca6cc36b9d78dbb015c1bf93bbafb14eb0799 to the payments cluster (again, because the instant reviewer, me, was broken)
21:23 mwalker: attempted deploy of payments cluster to new version, but then had to roll back to dced5328fd6ae4b395622abc41d2afa0ccad21c8
20:02 mutante: blocked en.wikivoyage from bugmenot
20:01 logmsgbot: dzahn synchronized ./docroot/wikivoyage.org/block_8708f.html
19:55 logmsgbot: dzahn synchronized ./docroot/wikivoyage.org/block_ab2c3.html
19:55 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'parser cache max age 30d'
19:37 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043 to s1 watchlist service'
19:31 mutante: DNS update - add CNAME for status.wp/status.wm
19:28 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/51291/'
19:13 mutante: install package upgrades on marmontel
19:06 mutante: install package upgrades on streber
19:04 mutante: create missing backup dir on streber to enable RT_Shredder plugin
19:04 Ryan_Lane: maxsem synchronized wmf-config/InitialiseSettings.php 'make upload endpoint for Commons local https://gerrit.wikimedia.org/r/#/c/51612/'
19:03 Ryan_Lane: LocalisationUpdate completed (1.21wmf10) at Fri Mar 1 02:31:13 UTC 2013
19:03 Ryan_Lane: csteipp synchronized php-1.21wmf10/includes
19:03 mutante: db9 - zipping old sql dumps of drupal and civirm to prevent full disk
19:03 Ryan_Lane: dzahn synchronized ./docroot/wikivoyage.org/mywotf83102384bfcd1e52152.html
19:03 Ryan_Lane: restarted glusterd service on labstore1-4
19:03 Ryan_Lane: pgehres Finished syncing Wikimedia installation... : Updating i18n for DonationInterface-langonly
19:02 Ryan_Lane: pgehres Started syncing Wikimedia installation... : Updating i18n for DonationInterface-langonly
19:02 Ryan_Lane: binasher: shutting down mysql on db1043, converting to mariadb
19:02 LeslieCarr: powering back on spence
19:02 Ryan_Lane: asher synchronized wmf-config/db-eqiad.php 'pulling db1043 and shifting watchlist q's to db1050'
19:02 Ryan_Lane: pgehres synchronized php-1.21wmf10/extensions/DonationInterface/ 'Updating DonatonInterface-langonly'
19:02 Ryan_Lane: LeslieCarr: powering off spence
19:01 Ryan_Lane: asher synchronized wmf-config/lucene.php 'sending search-pool4 traffic to pmtpa, where the index/host distribution has been rebalanced'
19:01 Ryan_Lane: notpeter: upgrading mariadb boxes to 5.5.29
19:01 Ryan_Lane: tstarling synchronized wmf-config/InitialiseSettings.php
19:01 Ryan_Lane: Jeff_Green: stopped gmetad on neon for testing
19:01 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/GettingStarted 'Update to split test (3/3)'
19:00 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/E3Experiments 'Update to split test (2/3)'
19:00 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/GuidedTour 'Update to split test (1/3)'
19:00 Ryan_Lane: mutante: create missing dump directory on streber, enables "RT-shredder" plugin which comes with RT since 3.8
19:00 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/EventLogging 'Updates to JavaScript API'
19:00 Ryan_Lane: changing A record for wikitech.wm.o and changing labsconsole.wm.o into a cname to wikitech
18:59 Ryan_Lane: disabled editing on wikitech.wikimedia.org
18:59 Ryan_Lane: binasher: resharding search-pool4 in pmtpa - restarted lsearchd on all local nodes and indexers on searchidx2
18:59 mwalker: ... therefore the payments cluster is now running dced5328fd6ae4b395622abc41d2afa0ccad21c8
18:59 mutante: restart nagios-nrpe,gmond,apache,salt-minion on bastion hosts
18:59 Ryan_Lane: binasher: rebuilding db35 from a hotbackup of db55 (s5); testing a new build of mariadb 5.5.29
18:59 Ryan_Lane: reedy synchronized php-1.21wmf10/cache/interwiki.cdb 'Updating 1.21wmf10 interwiki cache'
18:59 Ryan_Lane: reedy synchronized php-1.21wmf10/cache/interwiki.cdb 'Updating 1.21wmf10 interwiki cache'
18:58 Ryan_Lane: reedy synchronized php-1.21wmf10/extensions/WikimediaMaintenance
18:58 Ryan_Lane: apergos: restarted icinga (not sure why it stopped, log file simply said 'caught TERM signal', in any case it complained that Error: Could not create external command file '/var/lib/nagios/rw/nagios.cmd' as named pipe )
18:58 mwalker: enabled sofort forms for CH and DE on the payments cluster
18:39 mutante: installing package upgrades on bast1001
18:37 mutante: installing package upgrades on fenari
17:40 ^demon: rescanned all trackingids for gerrit on manganese
17:31 ^demon: upgrading gerrit to 2.5.2-1506-g278aa9a on manganese/formey
17:07 Ryan_Lane: test
February 28
12:58 apergos: restarted icinga (not sure why it stopped, log file simply said 'caught TERM signal', in any case it complained that Error: Could not create external command file '/var/lib/nagios/rw/nagios.cmd' as named pipe )
06:28 apergos: stabbed searchpool4 and restarted lucene search etc gah hate hate hate
03:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
03:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
03:12 logmsgbot: reedy synchronized docroot
03:10 logmsgbot: reedy synchronized docroot
03:09 logmsgbot: reedy synchronized docroot
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Thu Feb 28 02:30:14 UTC 2013
01:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:58 RobH: dsh restarted eqiad apaches in progress
00:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:57 logmsgbot: robh gracefulled all apaches
00:54 logmsgbot: reedy synchronized wmf-config/
00:32 RobH: authdns-update to support wikinews.de
00:27 logmsgbot: reedy synchronized wmf-config/
00:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Lowering throttle to 5/day for new users, and'
February 27
23:26 Reedy: Running sync-common on mw1111. Reporting failed opening wmf-config/mc.php
22:57 mutante: rebooting zirconium (planet)
22:54 mutante: installing package upgrades on zirconium
22:52 logmsgbot: reedy synchronized wmf-config/
22:46 mutante: rebooting bast1001
22:36 mutante: installing package upgrades on bast1001
22:16 logmsgbot: reedy synchronized wmf-config/
21:44 RobH: rdb1001 & rdb1002 allocated, network setup, and ready for handoff, per rt4608
21:36 RobH: authdns-update for rdb1001-1002
21:36 RobH: authdns-update for sdb1001-1002
21:31 Tim: running resetUserTokens.php on all wikis, in screen on hume, to fix user_token field corruption (bug 41586 )
21:26 logmsgbot: tstarling synchronized php-1.21wmf10/includes/User.php 'user_token fixes'
20:05 K4-713: updated payments cluster to fcbf9211c9836b
19:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf10
18:55 LeslieCarr: rebooting arsenic
18:42 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for EducationProgram
18:25 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for EducationProgram
17:57 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/
17:51 K4-713: updated payments cluster to eab489d0889a
17:13 mutante: rebooting fenari for kernel upgrade
08:33 apergos: restarted lucene search on search1015
08:04 logmsgbot: reedy synchronized live-1.5/static-1.21wmf4/
08:04 logmsgbot: reedy synchronized docroot/bits/static-1.21wmf4/
07:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
07:05 Krinkle: done adding 1.21wmf3, 1.21wmf4, 1.21wmf5 (bug 44570 )
07:04 logmsgbot: krinkle Finished syncing Wikimedia installation... :
06:29 apergos: snapshot1002 down for reinstallation
06:27 logmsgbot: krinkle Started syncing Wikimedia installation... :
06:15 Krinkle: adding 1.21wmf3, 1.21wmf4, 1.21wmf5 (bug 44570 )
06:15 Krinkle: added 1.21wmf1, 1.21wmf2 (bug 44570 )
05:35 apergos: restarted lucene search on search1016
04:57 logmsgbot: krinkle Finished syncing Wikimedia installation... :
04:37 logmsgbot: krinkle Started syncing Wikimedia installation... :
02:57 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 27 02:57:22 UTC 2013
02:31 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Wed Feb 27 02:31:06 UTC 2013
01:35 Krinkle: Bringing back checkouts of MediaWiki 1.21wmf5 - 1.21wmf1 on fenari for bug 44570
01:35 Jeff_Green: dist-upgrade and reboot grosley
00:25 logmsgbot: tstarling cleared profiling data
February 26
23:29 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/51053 https://gerrit.wikimedia.org/r/51057'
23:27 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/51053 https://gerrit.wikimedia.org/r/51057'
23:04 K4-713: synchronized payments cluster to 596b854787132c730
22:35 mutante: installing package upgrades on fenari
22:31 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
22:31 LeslieCarr: nagios is dead! spence.wikimedia.org if you want to access it directly
22:16 mutante: gracefulling eqiad Apaches to push redirect for wikivoyage.eu
22:15 logmsgbot: dzahn gracefulled all apaches
22:15 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
20:58 RobH: stat1 puppet runs fail due to dan account error, i am working on it, but i also want to run out and get something to eat, so leaving in broken state for 10 minutes until i return to work on it
20:21 mutante: DNS update - add wikiovyage.eu
19:34 Jeff_Green: package updates & reboots of payments hosts
19:13 paravoid: freezing all exim4 bounces to restore normal email delivery
18:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
17:54 paravoid: restarting opendj on sanger
17:45 AaronSchulz: Copying math files into ceph
15:14 mark: suspended i-000001d7 and i-0000039e for sending out malicious emails
15:06 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/GeoData/solrupdate.php
15:05 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/GeoData/solrupdate.php
12:26 apergos: temp blacklist live hack on sodium for exim4 to prevent some spam, see /etc/exim4/blacklist for that
11:20 logmsgbot: mlitn synchronized php-1.21wmf10/extensions/ArticleFeedback/api/ApiArticleFeedback.php 'Update ArticleFeedback to master'
06:53 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/
06:43 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
06:40 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
06:28 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
06:26 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
06:21 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
03:10 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
03:06 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
03:04 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
03:03 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 26 02:52:01 UTC 2013
02:46 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
02:44 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
02:41 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
02:37 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
02:29 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Tue Feb 26 02:28:06 UTC 2013
02:11 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
01:59 Ryan_Lane: test
01:59 Ryan_Lane: test
01:58 Ryan_Lane: test
01:40 K4-713: updated payments cluster to ce28b210374d8
00:20 mutante: mc1001,ms1004,mw1029,mw1059,mw1070,mw1157 - fixed puppet freshness
00:03 mutante: amssq33,amssq44 - fix more puppet runs
February 25
23:54 mutante: knsq26,knsq28 - deleting puppet lock file, fix puppet runs
23:45 logmsgbot: reedy synchronized wmf-config/
23:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable SearchExtraNS on commonswiki'
23:34 mutante: re-enabling disabled puppet on srv253,srv235
22:16 Ryan_Lane: restarting opendj/pdns on virt0
22:02 Reedy: Ran namespaceDupes.php on huwikt
21:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:28 logmsgbot: reedy synchronized docroot
20:27 logmsgbot: reedy synchronized wmf-config/
19:50 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuilding message cache to update WikimediaMessages
19:40 mutante: powercycling frozen niobium
19:32 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuilding message cache to update WikimediaMessages
19:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf10
18:54 mutante: wikivoyager.de and wikivoyager.org transferred and both redirect to wikivoyage.org
18:50 mutante: deleting foundation-news-l (deprecated, replaced by WikimediaAnnounce-l)
18:49 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMessages
18:31 AaronSchulz: Copying all captchas into ceph
18:11 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Wikibase
06:38 apergos: restarted lucene search on search1016
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 25 02:52:32 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Mon Feb 25 02:28:55 UTC 2013
01:15 mark: Powercycled strontium
01:07 paravoid: powercycling niobium (both unresponsive from the load)
01:06 paravoid: powercycling arsenic
February 24
06:28 apergos: restarted search1016 lucene
02:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 24 02:54:30 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sun Feb 24 02:29:29 UTC 2013
February 23
21:45 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb
21:44 logmsgbot: reedy synchronized php-1.21wmf10/cache/interwiki.cdb
21:43 Reedy: mw110 is asking for a password
06:13 apergos: restarted lucene search on search1016
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 23 02:52:10 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sat Feb 23 02:27:17 UTC 2013
00:43 RobH: lvs1003 had a ton of hung puppet processes (no output on them when tracing, so dunno whats up). killed them and kicked manual run
February 22
23:59 RobH: argon is shut down now
23:55 RobH: argon is going offline, killing our limesurvey install
23:29 paravoid: deploying squid config for upload's new transcoded container
23:26 paravoid: deploying squid
22:55 Ryan_Lane: changing dns entry for wikitech-static
22:41 RobH: mw110 throwing odd puppet errors, will come back to it later
22:41 RobH: sq41 reinstalled, throws endless puppet errors during redeployment
22:07 MaxSem: reindexing complete
21:39 RobH: harmon and copper online with OS installed for datadumps misc host support
21:39 MaxSem: Nuking GeoData Solr index and reindexing
21:38 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/GeoData/solrupdate.php 'https://gerrit.wikimedia.org/r/#/c/50388/'
21:35 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/GeoData/solrupdate.php 'https://gerrit.wikimedia.org/r/#/c/50388/'
21:05 RobH: installing harmon and copper into dump generation misc use
20:17 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/GuidedTour/modules/tours/gettingstarted.js 'Resume split test (3/3)'
20:16 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/GettingStarted/resources/ext.gettingstarted.js 'Resume split test (2/3)'
20:16 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/E3Experiments/experiments/openTask.js 'Resume split test (1/3)'
20:14 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Math 'Updating Math extension to fix db caching behavior (Change-Id: I9b690e55001859c97fd40330272791d49ec6de75)'
20:13 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/Math 'Updating Math extension to fix db caching behavior (Change-Id: I9b690e55001859c97fd40330272791d49ec6de75)'
20:09 RobH: authdns-update
19:58 RobH: sq41 boot order was fubar, fixed, reinstalling (as reboot into installer wiped mbr)
19:56 RobH: wtp1002-1004 os installed
19:34 Jeff_Green: start mysqldump of otrs database on db49
19:18 RobH: kicked ntpd manually on mw1045
19:09 RobH: sq41 reinstalling
18:57 RobH: mw112-125 back into pybal api pool
18:56 RobH: mw86-99 & mw111 back into pybal pool
18:49 logmsgbot: robh Finished syncing Wikimedia installation... :
18:37 logmsgbot: robh Started syncing Wikimedia installation... :
17:53 sbernardin: taking down sq41 to replace controller card.... https://rt.wikimedia.org/Ticket/Display.html?id=4550
16:24 mutante: tartupeedia.ee now redirects to et.wp portal page (bug 44893 )
16:12 mutante: gracefulling eqiad Apaches via dsh to push tartupeedia.ee redirect (bug 44893 )
16:11 logmsgbot: dzahn gracefulled all apaches
16:07 mutante: mw110 - sits at partioning screen in installer
15:47 hashar: gallium removed live hack in /srv/org/wikimedia/doc/puppetsource/manifests/role/labsmediawiki.pp
14:52 ^demon: restarted gerrit on manganese, jetty flapping again
14:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
13:14 mark: Halted ms-be3001-3003 for controller replacement
13:11 mark: Halted ms-be3004 for controller replacement
13:07 mark: Halted cp3009 for controller replacement
12:20 mark: Halted cp3010 for controller replacement
11:23 mutante: DNS update - add indiawikipedia.com
07:33 apergos: restarted lucene search on search1016
06:49 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:33 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:33 logmsgbot: tstarling synchronized docroot/bits/favicon/wikipedia.ico
05:20 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:18 logmsgbot: tstarling synchronized docroot/bits
02:53 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 22 02:53:05 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Fri Feb 22 02:29:09 UTC 2013
01:57 binasher: migrating wikivoyage and wikidata external blob stores to innodb
01:08 logmsgbot: aaron synchronized wmf-config/filebackend.php 'deployed f699c5a9ad2f780dc18863592dcae4a51ba6d841'
February 21
23:21 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/CodeEditor 'Syncing patch that disables background linting'
23:18 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, and GuidedTour
23:02 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, and GuidedTour
22:35 logmsgbot: awjrichards synchronized php-1.21wmf9/extensions/MobileFrontend/
22:33 logmsgbot: awjrichards synchronized php-1.21wmf10/extensions/MobileFrontend
22:12 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Add photo upload schema for event logging'
21:56 notpeter: running do-release-upgrade on dbs 52, 39, 51, and 35
21:49 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/EventLogging
21:48 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing 1 slave for s2-5 from db-secondary for upgrades and maria'
21:48 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
21:47 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/CodeEditor
21:37 notpeter: restarting mysql on db1047 to pick up conf changes
21:31 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/SyntaxHighlight_GeSHi 'Disable highlighting of very large files.'
21:23 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/SyntaxHighlight_GeSHi 'Disable highlighting of very large files.'
21:05 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php Enabling CodeEditor for Schema namespace on test2 and metawiki.
20:02 binasher: some wikis have myisam page_props tables. converting all to innodb via osc
18:39 Ryan_Lane: enabled Extension:OpenID as a provider on labsconsole
11:04 hashar: Ariel saved Gerrit by restarting it!
10:51 hashar: huge CPU spike on manganese (gerrit host)
10:06 hashar: Jenkins: reseting git branch specifier :-/
09:32 hashar: Jenkins: removing git branch specifier from all mediawiki-core jobs.
05:50 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'shell cgroup'
02:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 21 02:54:17 UTC 2013
02:30 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'new extension1 shard (as an externalload config) - initially for AFTv5'
02:30 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'new extension1 shard (as an externalload config) - initially for AFTv5'
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Thu Feb 21 02:28:20 UTC 2013
00:53 RobH: issues with new mw servers 86+, took back out of pybal until i can troubleshoot
00:49 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance/
00:48 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMaintenance/
00:34 binasher: moved wikidatawiki tables on s3 to wikidatawiki_old to keep around for a time just in case; then "drop database wikidatawiki"
00:20 logmsgbot: asher synchronized wmf-config 'enabling wikidatawiki on shard s5'
00:18 logmsgbot: asher synchronized s5.dblist 'adding wikidatawiki'
00:17 logmsgbot: asher synchronized s3.dblist 'removing wikidatawiki'
February 20
22:48 RobH: authdns-update
22:35 binasher: importing wikidatawiki with schema changes into the s5 master, with sql_log_bin disabled
22:24 binasher: importing wikidatawiki with schema changes into the inactive pmtpa side of the s5 shard, with replication of the import just in pmtpa. ignore repl warnings for db{35,44,45,55}
21:48 RobH: authdns-update for wtp1002-1004
21:45 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'Take pmtpa Solr servers out of rotation, increase load on master: https://gerrit.wikimedia.org/r/#/c/50056/'
21:41 RobH: ignore db66 errors, asher is working on it
21:25 binasher: running wikidata wb_items / wb_items_per_site schema migrations in parallel on the db66 slave (creating primary keys, cannot be done via online-schema-change)
21:24 RobH: kicked knsq23 to clear puppet freshness alarm
21:24 RobH: maerlant still offline (200 days?). dropped esams ticket for onsite troubleshooting, as mgmt is offline.
21:20 logmsgbot: asher synchronized wmf-config/InitialiseSettings.php 'setting wgReadOnly for wikidatawiki durring maintenance'
21:18 RobH: forced puppet run on lardner, was tired of it showing error for no runs
21:13 RobH: mw112-125 now in service as api apaches in tampa
20:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:38 RobH: mw86-mw111 tested ok, pushing into apache service in pmtpa
20:32 RobH: mw86-mw111 being added to nodelists, ignore errors on them for next few minutes
19:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks to 1.21wmf10
19:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, wikisource and wikiquote to 1.21wmf10
19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage, wikinews and wiktionary to 1.21wmf10
19:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, special, wikimedia, private and fishbowl to 1.21wmf10
05:34 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 're-enable Scribunto profiler'
04:32 Tim: updating luasandbox to version 1.2, with profiler bug fixes
02:53 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Wed Feb 20 02:53:32 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 20 02:29:11 UTC 2013
01:28 RobH: mw1161-1208 have wrong power setting, changing all to system.power.hotspare.enable 0
01:00 notpeter: do-release-upgrade-ing on db58
00:57 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing db58 from db-secondary for upgrades and maria, replacing db43'
February 19
23:47 notpeter: do-release-upgrade-ing on db43
23:38 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing db43 from db-secondary for upgrades and maria'
23:21 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/49946/'
23:04 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
23:03 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/
22:49 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
22:41 logmsgbot: preilly synchronized php-1.21wmf9/extensions/ArticleFeedback 'fix MAX call with GREATEST'
22:29 RobH: sq41 offline, rt 4550
22:20 RobH: sq41 puppet freshness bad, wont accept ssh connections, serial redirection results in no output. will need to reboot it
22:17 RobH: sq41 not allowing ssh, drac resetting
22:15 sbernardin: taking down ms-be12 for hardware replacement per: https://rt.wikimedia.org/Ticket/Display.html?id=4546
22:15 RobH: sq79/80 puppet in locked state, manually restarted puppet, removed lock file, ran manually
22:11 logmsgbot: preilly synchronized php-1.21wmf9/extensions/ArticleFeedback 'fix mysql 5.5 / ansi sql incompatibility'
21:54 RobH: mw48 was set to puppet agent disabled, re-enabled and fired puppet run
20:04 logmsgbot: mflaschen synchronized . 'Syncing GuidedTour submodule omitted when new branch cut was first cloned.'
19:59 Ryan_Lane: disabling project storage volumes for all labs projects with no instances
15:02 hashar: restarting jenkins to enable some startup setting
14:58 hashar: manganese CPU spike ongoing. Started roughly at 14:40 UTC
13:29 mutante: wikimedia.jp.net now redirects to wikimedia.org | wikimediacommons.jp.net now redirects to commons.wm
13:27 mutante: DNS update - adding wikimedia.jp.net
13:25 logmsgbot: dzahn gracefulled all apaches
13:20 mutante: gracefulling eqiad Apaches via dsh to push redirect change for .jp.net domains
12:39 apergos: powercycling ms-be7 (unresponsive to ping or from console)
11:29 mutante: installing package upgrads on neon/icinga
02:49 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Tue Feb 19 02:49:32 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 19 02:28:25 UTC 2013
01:12 logmsgbot: anomie synchronized php-1.21wmf10/includes/api/ApiQuerySiteinfo.php 'Backport fix for ugly API bug'
00:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
00:30 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
February 18
23:51 logmsgbot: tstarling synchronized php-1.21wmf9/extensions/Scribunto
23:51 logmsgbot: tstarling synchronized php-1.21wmf10/extensions/Scribunto
23:46 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
23:25 Tim: on snapshot1002 -- fixed incorrect owner on some MW files, killed broken apt-get and upgraded wikimedia-task-appserver
23:24 Tim: fixed rsyncd.conf on scap proxies to allow connections from new apaches
23:14 logmsgbot: tstarling Finished syncing Wikimedia installation... :
23:11 logmsgbot: tstarling Started syncing Wikimedia installation... :
22:45 logmsgbot: tstarling Finished syncing Wikimedia installation... :
22:42 logmsgbot: tstarling Started syncing Wikimedia installation... :
22:37 logmsgbot: tstarling synchronized php-1.21wmf9/extensions/Scribunto
22:37 logmsgbot: tstarling synchronized php-1.21wmf10/extensions/Scribunto
22:20 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.21wmf10
20:37 logmsgbot: reedy synchronized php-1.21wmf10/cache/l10n/ 'Scap is a liar'
20:31 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild l10n cache for wikidata related extensions
20:15 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild l10n cache for wikidata related extensions
19:59 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Wikibase
19:59 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Diff
19:58 logmsgbot: reedy synchronized php-1.21wmf10/extensions/DataValues
19:58 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMessages
19:44 logmsgbot: reedy synchronized php-1.21wmf10/cache/l10n/
19:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf10
19:03 logmsgbot: reedy Finished syncing Wikimedia installation... : Build 1.21wmf10 message cache, and push it this time
18:45 logmsgbot: reedy Started syncing Wikimedia installation... : Build 1.21wmf10 message cache, and push it this time
17:54 logmsgbot: reedy Finished syncing Wikimedia installation... : Build 1.21wmf10 message cache
17:53 logmsgbot: reedy Started syncing Wikimedia installation... : Build 1.21wmf10 message cache
17:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf10
17:28 logmsgbot: reedy synchronized php-1.21wmf10 'Initial sync out'
17:03 logmsgbot: reedy synchronized wmf-config/
17:02 logmsgbot: reedy synchronized docroot
17:01 logmsgbot: reedy synchronized live-1.5/
12:33 mutante: wikiartpedia.(biz|co|info|me|mobi|net) activated - redirect to wikipedia.org (RT-4240)
12:30 mutante: adding wikiartpedia domains to DNS / authdns-update
12:28 logmsgbot: dzahn gracefulled all apaches
12:24 mutante: gracefulling eqiad Apaches via dsh for redirect.conf change
12:22 mutante: mw1045 - eth0 NO-CARRIER - check cable (RT-4545)
12:13 mutante: mw1041 - down - CPU Machine Chk error on mgmt - removing from dsh groups (RT-4517)
05:45 logmsgbot: tstarling synchronized wmf-config/db-eqiad.php
05:16 logmsgbot: tstarling synchronized wmf-config/db-eqiad.php
05:10 Tim: on db1043: running alter table to convert the relevant field to signed, since I figure this is less disruptive than sql_mode = 'NO_UNSIGNED_SUBTRACTION' and less likely to break again than skipping the query
04:58 Tim: replication broken on db1043, possibly others, due to update of article_feedback_revisions with negative value for afr_total
03:40 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 18 02:29:47 UTC 2013
February 17
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 17 02:29:31 UTC 2013
February 16
13:38 apergos: restarted lucene-search on search1015 about 20 minutes ago
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 16 02:28:26 UTC 2013
01:47 logmsgbot: reedy synchronized php-1.21wmf9/
01:11 logmsgbot: reedy synchronized wmf-config/filebackend.php
February 15
23:56 RobH: authdns-update to correct mw110 reverse entry
23:28 Ryan_Lane: *modified them in LDAP
23:28 Ryan_Lane: modified project objects to indicate whether or not they are using gluster volumes for project and/or home storage
23:04 RobH: disregard mw1161 flaps, im working on it
20:15 Ryan_Lane: restarting opendj and pdns on virt0 and virt1000
20:15 Ryan_Lane: removed opendj from nfs1/2
19:07 RobH: no one sign mw86-mw125 puppet certs, im still working on them
18:23 RobH: mw87-mw125 installing
17:33 binasher: reimaging db1029-1031
16:55 mutante: removing mw1085 from dsh groups - broken hardware (RT-4453)
16:00 hashar: Enabled a linter for operations/apache-config . Any apache typo in operations/apache-config will publicly write your name on the typo wall of shame. See 49251 for implementation details and https://integration.mediawiki.org/ci/job/operations-apache-config-lint/ for some build examples.
14:31 mutante: wikimediafoundation.info activated to redirect .org
14:28 mutante: gracefulling Apaches in eqiad using dsh
14:25 logmsgbot: dzahn gracefulled all apaches
14:15 ^demon: [gerrit] updated replication plugin to 20c6fc7e (1.1-SNAPSHOT) which is back in line with upstream
08:02 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Echo 'Syncing Iab077a77aa75a6ff0cf03acecca38403595dda93 to fix exception 640c60c4'
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 15 02:30:05 UTC 2013
01:19 paravoid: deploying squid changes for s/knams/esams/ for knsq* squids (see git log for rationale)
01:09 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Enabling Schema NS on test2wiki'
February 14
22:49 RobH: mw1194 online and now in service as api apache
22:48 RobH: mw1200 added to api pool, online and in service
22:44 RobH: mw1189-1199 (sans 1194) now online as eqiad api apaches
22:37 logmsgbot: mflaschen Finished syncing Wikimedia installation... :
22:29 logmsgbot: mflaschen Started syncing Wikimedia installation... :
22:21 sbernardin: taking storage1 offline for decommissioning per: https://rt.wikimedia.org/Ticket/Display.html?id=4529
22:18 RobH: pushing mw1161-1188 into apache service (sans 1161/1164) from false to true in pybal, otherwise all online
22:16 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Updating for test2wiki'
22:14 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Math
22:03 Ryan_Lane: replacing gluster upstarts on all labstore nodes with a regular init script
22:01 logmsgbot: reedy synchronized docroot
22:00 logmsgbot: olivneh synchronized wmf-config
21:49 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling PostEdit on {be|mediawiki|outreach}wikis'
21:45 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
21:39 sbernardin: taking down srv266 and srv268 for decommissioning ... https://rt.wikimedia.org/Ticket/Display.html?id=4534
21:31 ottomata1: restarted solr jetty on solr2
21:30 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update ArticleFeedbackv5, Echo, PageTriage to master
21:25 ottomata: restarted solr jetty on solr3 and solr1003 for MaxSem
21:03 logmsgbot: bsitu Started syncing Wikimedia installation... : Update ArticleFeedbackv5, Echo, PageTriage to master
20:24 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'updating CommonSettings.php for Echo'
20:15 apergos: rebooting mw1010 after some issues with OOM
19:22 RobH: new servers are NOT in pybal yet, and thus not getting traffic
19:22 RobH: just added mw1161-1200 into node lists for eventual service, some may throw sync script errors as I track them down
19:15 Ryan_Lane: rebooting labstore3
19:08 Ryan_Lane: rebooting labstore4
19:01 Ryan_Lane: rebooting labstore2
18:52 Ryan_Lane: rebooting labstore1
15:22 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Updating wikibase sort order'
15:22 logmsgbot: demon synchronized php-1.21wmf9/extensions/Wikibase/client/includes/SortUtils.php 'Updating wikibase sort order'
14:52 mutante: re-enabled cp1044 in pybal - mobile varnish cache servers all on precise now
14:39 hashar: Jenkins: adding in several extension to Zuul configuration 49041
14:30 hashar: jenkins : updated all jobs using latest version jjb-config
13:35 mark: Fixed fixing
13:27 mutante: disabling cp1044 in pybal, reinstalling with precise
13:22 mutante: re-enabling cp1043 in pybal
13:11 mark: Upgraded varnish to latest version with updated range patch on cp1021-1028
12:16 mark: Inserted varnish 3.0.3plus~rc1-wm7 into the precise-wikimedia APT repository
12:14 mutante: stopping varnish backend on cp1043 and starting reinstall
12:10 mutante: disabling cp1043 in pybal
11:31 mutante: ms1004 - full disk, deleted large HTCPpurger.log file, fixed puppet runs
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 14 02:29:05 UTC 2013
01:17 andrewbogott: updated OpenStackManager to git head on virt0. Previous state is preserved (temporarily) in the local 'premerge' branch.
01:13 paravoid: removing mysql & apache from emery(!)
00:03 RobH: db29 os installed, handed off to peter
February 13
23:50 RobH: db29 reinstalling
23:48 paravoid: reinstalling sq48
23:22 RobH: db27 and db29 will be up and down for reinstallations
23:17 sbernardin: rebooting db29 per https://rt.wikimedia.org/Ticket/Display.html?id=4528
23:17 sbernardin: rebooting db27 per https://rt.wikimedia.org/Ticket/Display.html?id=4528
23:11 RobH: db1013 online and puppet updated, needs to be pushed into some db cluster service
22:45 RobH: mw27 locked up, powercycling
22:36 notpeter: powercycling all eqiad jobrunners
22:13 RobH: adding mw1201-1208 to eqiad apache pool
21:44 Reedy: Running sync-common on mw1203
21:43 logmsgbot: reedy synchronized php-1.21wmf9 'resync to fix file permission errors'
21:32 Reedy: Running sync-common on mw1204
21:30 Reedy: Running sync-common on mw1202
21:27 RobH: db27 rebooting for reinstallation
21:27 RobH: db29 offline due to hardware issue, related ticket 4526
21:19 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
21:11 logmsgbot: csteipp synchronized wmf-config
21:09 logmsgbot: reedy cleared profiling data
20:25 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Uncommenting enwiki from localClientDatabases'
19:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Wikibase Client on enwiki'
19:47 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase/
19:30 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved remaning wikis to 1.21wmf9
19:21 logmsgbot: aaron synchronized php-1.21wmf9/maintenance/runJobs.php 'deployed b03384f0ebd95e7f79638fb14ccd55da9c186d97'
19:12 logmsgbot: py gracefulled all apaches
18:59 cmjohnson1: mw1041 is borked and will be powered down and removed from network for h/w checks
18:27 logmsgbot: aaron synchronized php-1.21wmf9/extensions/TranslationNotifications 'deployed d7bf60b57f472e5f75101f58505f61842848f961'
16:26 Nemo_bis: Job queue weirdness and TranslationNotifications duplicates again noticed on Meta (bug 42715 )
13:33 logmsgbot: reedy synchronized php-1.21wmf9/resources/mediawiki
09:17 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48804/'
09:10 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48804/'
08:52 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48797/'
08:25 hashar: jenkins: changing encryption key and regenerating secrets. See bug 44592
05:50 logmsgbot: tstarling synchronized multiversion/MWScript.php
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Feb 13 02:52:39 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 13 02:28:06 UTC 2013
00:42 notpeter: restarting varnishncsa on cp1043
00:03 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend 'Touchhhhh'
February 12
23:54 logmsgbot: awjrichards synchronized php-1.21wmf9/extensions/MobileFrontend/javascripts/specials/donateimage.js 'touch file'
22:58 logmsgbot: maxsem Finished syncing Wikimedia installation... :
22:46 notpeter: disabling innodb-flush-log-at-trx-commit=2 on db1047 w/asher's blessing. will schedule downtime to apply
22:38 logmsgbot: maxsem Started syncing Wikimedia installation... :
22:04 notpeter: creating metrics user on all s1-7 masters
20:25 logmsgbot: reedy synchronized php-1.21wmf9/resources/mediawiki
20:22 binasher: flushed mobile varnish
20:21 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend
20:19 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend
20:06 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:05 binasher: flushed mobile varnish cache �:(
19:59 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend
19:57 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend
19:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable WikibaseClient on enwiki'
19:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable WikibaseClient on enwiki'
18:49 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
18:41 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
18:36 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
18:25 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
18:20 cmjohnson1: nas1001-a swapping mainboard
18:10 logmsgbot: aaron synchronized php-1.21wmf9/extensions/UploadWizard 'deployed 4548f9817d3d8025f4c19e279e9fb1f05b178087'
18:09 logmsgbot: catrope synchronized php-1.21wmf9/extensions/Wikibase 'Un-rollback Wikibase'
18:06 RoanKattouw: Reverted Wikibase update earlier, now reapplying
17:53 paravoid: powercycling niobium, SSH & console unresponsive
17:50 mark: Blocking modules=mobile.startup requests with 503 on bits varnish (earlier)
17:50 mark: Forced mw1149-1150 to healthy in bits.eqiad varnish
17:30 logmsgbot: catrope synchronized php-1.21wmf9/extensions/Wikibase 'Rolling back Wikibase'
16:18 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 12 16:18:31 UTC 2013
16:13 Jeff_Green: restarted apache on mw1152
16:01 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
16:00 Reedy: mw1161 and mw1165 are both missing mediawiki deployment files
15:58 logmsgbot: reedy synchronized php-1.21wmf9/includes/site/
15:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 12 15:54:11 UTC 2013
14:25 mutante: ran puppet on professor
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 12 02:52:42 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 12 02:28:53 UTC 2013
01:01 ^demon: stopped puppet and gerrit on manganese and formey
00:28 logmsgbot: aaron synchronized php-1.21wmf9/includes/db/Database.php 'deployed d8705542627f006a7ec9f81a9fb488fcc9a367bd'
February 11
23:19 RobH: unpooled mw1161-1188 from pybal config
22:03 logmsgbot: demon synchronized php-1.21wmf9/extensions/VisualEditor/modules/ve/init/mw/ve.init.mw.Platform.js 'Touching file'
21:08 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Wikibase client throwing fatals, disabling on enwiki pending fix'
21:04 cmjohnson1: auth-dns update
21:03 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'wikidata client on enwiki'
21:03 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'wikidata client on enwiki'
20:30 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: huwiki and itwiki to 1.21wmf9
20:27 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: hewiki and itwiki to 1.21wmf9
20:22 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf9
20:11 logmsgbot: demon synchronized php-1.21wmf9/includes/site 'Updating sites code to pull in I8964a03c, I42221832'
18:39 RobH: disregard, pushing one to test, mw1161
18:36 RobH: mw1161-mw1188 being pushed into pybal pool as general apaches
16:58 logmsgbot: demon synchronized php-1.21wmf9/extensions/Wikibase/lib/includes/modules/SitesModule.php 'Touching file'
16:50 logmsgbot: demon synchronized php-1.21wmf9/extensions/TorBlock 'TorBlock to master, now with less annoying logs'
16:41 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Enable logging group for torblock'
16:40 logmsgbot: demon synchronized php-1.21wmf9/extensions/TorBlock 'Updating TorBlock to master'
16:27 cmjohnson1: commenting out srv226-234 on pybal/pmtpa
14:59 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'bug 44251 hardcode $wgDBuser = "wikiuser" 47074 '
13:55 hashar: jenkins: regenerating all phpcs-HEAD jobs so they stop failing when no files need to be processed bug 44567 48450
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Mon Feb 11 02:52:39 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 11 02:27:44 UTC 2013
February 10
22:16 Tim: fixed syslog.eqiad.wmnet DNS entry, restarting rsyslogd on the eqiad apaches (RT 4500)
19:20 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/48015/'
02:53 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sun Feb 10 02:53:56 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 10 02:28:46 UTC 2013
February 9
12:52 Reedy: mw1161 is missing /usr/local/apache/common-local
12:51 logmsgbot: reedy synchronized php-1.21wmf8/resources/
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sat Feb 9 02:52:45 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 9 02:28:28 UTC 2013
February 8
23:48 logmsgbot: anomie synchronized php-1.21wmf8/extensions/Scribunto/engines/LuaCommon/LuaCommon.php
23:48 logmsgbot: anomie synchronized php-1.21wmf9/extensions/Scribunto/engines/LuaCommon/LuaCommon.php
23:09 RoanKattouw: and on cerium too
23:08 RoanKattouw: Running puppet on titanium
20:20 RobH: disregard errors for mw1161-1164
20:13 RobH: mw1161-1164 offline for reallocation from job to general apache
20:10 cmjohnson1: decommissioning srvs226-234
19:43 RobH: mw1165-1188 os installed for apache use, initial puppet runs in progress
19:33 RobH: celsus back online
19:23 RobH: celsus rebooting to fix power mgmt cpu spike issue
19:02 RobH: mw1161-1164 in service as eqiad jobrunners
16:32 ottomata: updated vumi packages in apt to vumi_0.5.0~a+143-0~ppa3 and vumi-wikipedia_0.1~a+14-0~ppa3
16:09 mark: Upgraded and rebooted cr2-eqiad (both REs)
15:11 mark: Set cr2-eqiad VRRP prio to 50 to transition to backup role
15:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
15:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: srwikibooks back to 1.21wmf9 to test variant fixes
14:39 hashar: gallium installed php-luasandbox manually. Gerrit change is 48127 .
05:47 logmsgbot: tstarling synchronized php-1.21wmf9/includes/context/RequestContext.php
02:54 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Fri Feb 8 02:54:49 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 8 02:30:42 UTC 2013
02:02 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
01:48 LeslieCarr: cr2-knams is back up on the new code
01:36 LeslieCarr: cr2-knams reloading after upgrade
01:34 LeslieCarr: deactivating european bgp sessions - may cause some connectivity issues as connections reset
01:24 LeslieCarr: disabling notifications on icinga and nagios
01:03 logmsgbot: reedy synchronized php-1.21wmf9/languages/classes/
00:25 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/GettingStarted
February 7
23:31 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling GuidedTour on additional wikis'
23:25 LeslieCarr: manually cleared arp and turned managmeent interface off and on of asw-c-eqiad to solve reachability issue
23:02 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deploy GuidedTour
22:47 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy GuidedTour
22:39 RobH: mw82 pooled into tampa apaches
22:04 ottomata: pulled power and rebooted amaranth toolserver for multichill. Its back online. multichill will notify whomever needs to do more work.
21:58 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/RelatedArticles/ 'Fixes 44761'
21:58 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/RelatedArticles/ 'Fixes 44761'
21:50 RobH: mw81-mw85 in pmtpa apache service
21:39 cmjohnson1: performing a hard reboot on amaranth
21:34 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/E3Experiments
21:33 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/E3Experiments
21:12 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/PostEdit
21:11 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/PostEdit
21:10 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/EventLogging
21:09 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
20:29 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Being paranoid, node lists were in flux during last sync
20:23 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: All ku/sr wikis back to 1.21wmf8 - lang converter breakage
20:20 Nemo_bis: !Wikipedia and !Wikimedia projects are/were down, issue being investigated
20:14 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: kuwiktionary back to 1.21wmf8
20:07 RobH: removing srv219-224 from rendering config in pmtpa
20:06 mark: Rebooting cr1-sdtpa
19:56 RobH: adding mw75-mw80 into tampa rendering pool
18:42 mark: Rebooted re0.cr1-eqiad
18:05 mark: Rebooting re1.cr1-eqiad
16:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:44 mark: Rebooting cr1-esams
15:40 logmsgbot: reedy synchronized php-1.21wmf8/cache/interwiki.cdb 'Updating 1.21wmf8 interwiki cache'
15:39 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb 'Updating 1.21wmf9 interwiki cache'
14:54 mutante: granting RT group 'toolserver admins' the 'modify ticket' permission in queue esams
14:38 mutante: planet now updating hourly instead of daily
14:13 mutante: planet speaks SSL now for all languages. (it does not force http->https but it can if there is a demand)
12:00 Nemo_bis: amaranth dead/unreachable from Europe, ping dies at xe-5-3-1.cr2-eqiad.wikimedia.org (173.241.131.218)
10:08 hashar: jenkins : regenerating jobs to make phpcs to show the sniff codes being used. 47847
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Thu Feb 7 02:52:26 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 7 02:27:52 UTC 2013
02:23 logmsgbot: spage synchronized php-1.21wmf9/extensions/GuidedTour/modules/externals/mediawiki.libs.guiders/mediawiki.libs.guiders.submodule 'sync GuidedTour submodule omitted from 1.21wmf9'
01:57 RoanKattouw: Force-running puppet on wtp1 to figure out why it's broken
01:53 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'I9fd98309'
00:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:20 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance
00:00 logmsgbot: reedy synchronized langlist
February 6
23:59 RobH: authdns-update to add min.wikipedia.org via addwiki automation
23:56 logmsgbot: reedy synchronized php-1.21wmf8/cache/interwiki.cdb 'Updating 1.21wmf8 interwiki cache'
23:55 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb 'Updating 1.21wmf9 interwiki cache'
23:50 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:43 logmsgbot: reedy synchronized langlist
23:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:28 RobH: srv190-srv223 & srv225 should be offline, they are being decommissioned, can ignore nagios errors
23:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:13 RobH: specifically (i can spell, maybe)
23:13 RobH: removed from pybal specifially
23:12 cmjohnson1: removing srv192-218 and srv225 per rt 4438
23:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:05 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance/
21:36 cmjohnson1: srv193 is back online
21:21 cmjohnson1: srv193 power down to relocate to different rack
19:47 RobH: authdns-update
19:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikibooks to 1.21wmf9
19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiquote to 1.21wmf9
19:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews, wikivoyage and wikiversity to 1.21wmf9
19:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special and wikimedia wikis to 1.21mwf9
19:11 cmjohnson1: powerdowing mw1 to reseat DIMM B1
19:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, private and fishbowl wikis to 1.21wmf9
18:43 ^demon: running puppet on manganese, restarting gerrit
18:41 RobH: authdns-update
17:48 LeslieCarr: restarted morebots
02:51 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 6 02:51:36 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Feb 6 02:28:05 UTC 2013
02:05 logmsgbot: reedy synchronized wmf-config/
00:42 RobH: mailing lists returned to normal
00:41 RobH: done with data removal on mailing list server, rebuilding mbox
00:37 RobH: stopping mailman to scrub an entry out of archives, will restart shortly
February 5
23:29 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend/javascripts/modules/mf-photo.js 'https://gerrit.wikimedia.org/r/#/c/47670/'
23:26 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/javascripts/modules/mf-photo.js 'https://gerrit.wikimedia.org/r/#/c/47670/'
22:44 logmsgbot: maxsem Finished syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-02-05
22:28 logmsgbot: maxsem Started syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-02-05
22:08 RobH: analytics1007 coming down for troubleshooting, disregard nagios errors
22:01 RobH: analytics1001 reinstalled, not puppet signed, back online
22:00 RobH: analytics1001 respects my authority.
21:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 44688 - Please disable MoodBar extension on English Wikipedia'
21:28 logmsgbot: robh synchronized wmf-config/InitialiseSettings.php
21:12 RobH: analytics1001 going down for troubleshooting/reinstallation
20:21 logmsgbot: kaldari Finished syncing Wikimedia installation... :
20:12 RobH: blog update sucessful, caching plugin update nonsuccessful, rolling it back to older plugin
20:07 logmsgbot: kaldari Started syncing Wikimedia installation... :
20:04 RobH: updating the blog software, here goes nothing
19:17 hashar: gallium : Zuul updated!
19:13 hashar: gallium : updating Zuul by cherry-picking a set of changes that let us fetch tags. ff79197..1854e32
17:05 cmjohnson1: authdns update
16:37 LeslieCarr: copied all equinix ashburn IPv4 peering sessions to their new /22 on cr2-eqiad
16:37 Jeff_Green: dist-upgrades for payments systems
16:36 Jeff_Green: restarted morebots :-P
05:37 Ryan_Lane: restarting opendj on all hosts
05:36 Ryan_Lane: remove virt1 replication agreement from all ldap servers
02:51 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 5 02:51:10 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 5 02:29:02 UTC 2013
01:07 logmsgbot: reedy synchronized php-1.21wmf9/extensions/TimedMediaHandler
00:51 logmsgbot: reedy synchronized php-1.21wmf8/extensions/ZeroRatedMobileAccess
00:39 logmsgbot: reedy synchronized php-1.21wmf8/extensions/ZeroRatedMobileAccess
00:29 logmsgbot: reedy synchronized php-1.21wmf9/extensions/ZeroRatedMobileAccess
February 4
23:49 logmsgbot: reedy synchronized multiversion/
23:46 logmsgbot: reedy synchronized multiversion/
23:45 logmsgbot: reedy synchronized wmf-config/
23:17 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
21:02 hashar: gallium moved a git extension : mv /var/lib/git/rt-authen-externalauth.git /var/lib/git/operations/debs/
19:44 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
19:39 ottomata: manually gracefully reloaded eqiad apaches. See RT 4449
19:37 logmsgbot: root gracefulled all apaches
19:20 logmsgbot: reedy synchronized wmf-config/
19:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, mediawikiwiki and wikidatawiki to 1.21wmf9
18:35 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
17:56 logmsgbot: reedy Finished syncing Wikimedia installation... : Build messages for 1.21wmf9
17:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:24 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
17:14 logmsgbot: reedy synchronized php-1.21wmf9/extensions/GuidedTour/
17:09 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
17:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
17:06 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
16:45 logmsgbot: reedy synchronized php-1.21wmf9/ 'Initial sync'
16:22 logmsgbot: reedy synchronized wmf-config/
16:06 Ryan_Lane: force starting any glusterfs volume that has a downed process
15:12 Ryan_Lane: rebooting labstore1
15:11 Ryan_Lane: upgrading labstore1 to precise
14:33 apergos: powercycled ms-be3
14:32 Ryan_Lane: make that rebooting labstore3
14:32 Ryan_Lane: rebooting labstore2
14:07 Ryan_Lane: upgrading labstore3 to precise
11:14 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'cbd432f..82ed797 47393 Vector micro design on mlwiki, 29792 wikipedia translation on pawiki'
07:37 Ryan_Lane: upgrading labstore2 to precise
02:25 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Mon Feb 4 02:25:58 UTC 2013
00:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
February 3
20:58 hashar: jenkins : generating mwext-**-phpcs-HEAD jobs
17:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:42 logmsgbot: reedy synchronized wmf-config/
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sun Feb 3 02:28:29 UTC 2013
02:03 logmsgbot: reedy synchronized wmf-config/
02:02 logmsgbot: reedy synchronized images/sul/foundation.png
01:14 logmsgbot: reedy synchronized wmf-config/
00:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
February 2
23:59 logmsgbot: reedy synchronized wmf-config/
23:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:34 logmsgbot: reedy synchronized wmf-config/
23:30 logmsgbot: reedy synchronized php-1.21wmf8/extensions/SearchExtraNS
23:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
16:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
16:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
09:23 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/PageImages/initImageData.php 'https://gerrit.wikimedia.org/r/#/c/44556/'
08:45 MaxSem: Running foreachwikiindblist wikipedia.dblist extensions/PageImages/initImageData.php --earlier-than=20121229000000
03:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sat Feb 2 02:28:18 UTC 2013
February 1
23:02 K4-713: synchronized payments cluster to b421871b43822
20:00 RobH: authdns-update
19:43 RobH: authdns-update
14:45 cmjohnson1: authdns update "adding mw1161-1200 to eqiad mgmt and production zone files
12:04 cmjohnson1: authdns update adding db1051-60 to zone files
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Fri Feb 1 02:27:09 UTC 2013
01:50 K4-713: updated payments cluster to 45f8a6c6009a4bd7
January 31
23:14 andrewbogott: rebooting virt0 in a fit of optimism and/or desperation
22:44 logmsgbot: spage Finished syncing Wikimedia installation... : take 2: Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
22:37 logmsgbot: spage Started syncing Wikimedia installation... : take 2: Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
22:26 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
22:20 ottomata: deployed tab delimiter log format change on squids, varnishes and nginxes
21:40 ottomata: reverted tab separartor change to squid
21:37 ottomata: deloyed tab separator log format changes to squids, merged the corresponding puppet changes to varnish and nginx
20:47 paravoid: deploying patched ganglia and removing .htaccess. Ganglia is public again!
14:59 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase
14:58 notpeter: taking down db1047 for upgrade to precise
13:21 hashar: gallium: installed mercurial manually (puppet change is 46931 PS2)
13:19 hashar: gallium manually removed bzr: apt-get remove bzr python-keyring python-httplib2 python-launchpadlib python-zope.interface python-oauth python-bzrlib bzr python-simplejson python-configobj python-lazr.uri python-lazr.restfulclient python-wadllib
13:19 hashar: gallium: manually installed "bzr"
12:12 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase
10:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Remove strategyappswiki
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Thu Jan 31 02:29:47 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Thu Jan 31 02:28:18 UTC 2013
January 30
22:40 binasher: deployed new text/upload squid confs with eqiad in proxySubnets
22:38 logmsgbot: reedy synchronized wmf-config/
22:23 logmsgbot: tstarling cleared profiling data
22:12 logmsgbot: aaron synchronized php-1.21wmf8/includes/filerepo/file/LocalFile.php 'deployed f884911f5562fc41980664c424e7037bde6ba110'
22:11 logmsgbot: aaron synchronized php-1.21wmf8/includes/filerepo/file/OldLocalFile.php 'deployed f884911f5562fc41980664c424e7037bde6ba110'
21:59 LeslieCarr: killing varnishhtcpd on cp3009 for realzies this time
21:58 RobH: mw23 offline, reinstalled with precise and signed into puppet, puppet throwing errors on initial run.
21:52 binasher: started udplog with correct mw config on fluorine
21:49 LeslieCarr: restarting varnishhtcpd on cp3010 - it appears to be locked up , netstat -l shows a huge Recv-Q for port 4827 and appears to be the previously discovered issue
21:43 binasher: restarting udp2log on fluorine
20:59 logmsgbot: reedy cleared profiling data
20:50 RobH: authdns-update
20:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:20 RobH: mw23 was stuck in installer, restarting and setting up.
20:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:45 logmsgbot: reedy synchronized wmf-config/ 'Enabling wikidata client on itwiki and hewiki'
19:42 ottomata: added updated udplog_1.8-5 .debs to apt.wikimedia.org. No real changes, these now include the packet-loss program in the package.
19:22 logmsgbot: reedy synchronized wmf-config/
19:12 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase/
19:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf8
18:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:48 Ryan_Lane: downgrading memcached to ubuntu version on virt0
17:18 Reedy: Created wb_changes_dispatch on wikidatawiki
16:55 Reedy: Created sites/site_identifier tables on hewiki and itwiki
16:34 cmjohnson1: mw1072 going to down to check h/w state currently failed rt4381
15:38 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 44395 ) Allow bureaucrats to remove the translateadmin group on wikidatawiki 46011 '
15:37 logmsgbot: hashar synchronized wmf-config/throttle.php '(bug 44506 ) raise throttle for an Israel editthon 46747 '
12:54 logmsgbot: maxsem synchronized php-1.21wmf7/extensions/GeoData
12:52 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/GeoData
05:55 Tim: rebooted kaulen, was not responding on network or serial console
03:11 mutante: wikimania.asia redirect now works, it did not before because of a bug in apache-graceful-all not actually restarting Apaches in eqiad
03:07 mutante: reloading Apaches in eqiad using manual dsh (because apache-graceful-all sanity check has a bug)
02:55 logmsgbot: catrope gracefulled all apaches
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Wed Jan 30 02:52:03 UTC 2013
02:39 logmsgbot: dzahn gracefulled all apaches
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Jan 30 02:28:22 UTC 2013
00:44 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
00:43 mutante: sync InitialiseSettings.php for bug 44473
00:19 binasher: restarted pdns-recursor on dobson
00:13 logmsgbot: reedy synchronized wmf-config/
January 29
23:54 logmsgbot: reedy synchronized wmf-config/
23:21 logmsgbot: pgehres synchronized php-1.21wmf8/extensions/CentralNotice/ 'Updating CentralNotice to master, resolving namespace issues'
23:17 logmsgbot: pgehres synchronized php-1.21wmf7/extensions/CentralNotice/ 'Updating CentralNotice to master, resolving namespace issues'
23:00 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Enable Special:Nearby everywhere'
22:51 logmsgbot: maxsem synchronized php-1.21wmf7/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/46630'
22:48 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/46630'
22:47 Tim: rotated SAL wikitech page - test logging
Archives
Toggle limited content width