23:57 Krinkle: Restore localised messages (nl) for CVNBot12
June 5
17:00 Krinkle: Unscheduled reboot of cvn-app5. Bots are having trouble writing to local disk.
April 24
19:03 Krinkle: Restarted cvn-app4. Somehow it became unreachable for over an hour.
18:11 Krinkle: Promoted Rxy from member to projectadmin
April 23
17:13 andrewbogott: rebooted cvn-app5 and cvn-app4 (not at the same time) because I suspect them of running amok
15:42 andrewbogott: migrating cvn-app4 to labvirt1001 in an attempt to redistribute load a bit
April 22
12:47 andrewbogott: migrating cvn-app4 and cvn-app5 to labvirt1005 and labvirt1006, respectively. Seems to be working now.
03:00 Krinkle: Archived cvn-apache5 access logs to /data/project/cvn-common/backup/cvn-apache5-accesslogs.tar.gz
02:54 Krinkle: https://cvn.wmflabs.org now points to cvn-apache8 (UbuntuTrusty; Apache 2.4). The old cvn-apache5 (UbuntuPrecise; Apache 2.2) will be deleted shortly after archiving web logs to NFS.
April 21
21:26 Krinkle: Apply role::labs::lvm::srv via Hiera
21:01 andrewbogott: cold-migrating cvn-app4 to labvirt1005
September 20
14:43 andrewbogott: moved cvn-apache5 to virt1007
June 12
01:51 Krinkle: Disabled role::labs::lvm::mnt, and enabled role::labs::lvm::srv instead (separate changes)
01:12 Krinkle: Enable role role::labs::lvm::mnt on all cvn instances
May 4
18:30 Krinkle: CVNBot1 (feeds #cvn-wp-en; migrated from rcbot) is now hosted in Wikimedia Labs, runs on cvn-app4.
03:50 Krinkle: CVNBot13 (feeds #cvn-wp-da; migrated from KyluBot) is now hosted in Wikimedia Labs, runs on cvn-app5.
03:50 Krinkle: CVNBot12 (feeds #cvn-wp-nl; migrated from KrinkleBot7) is now hosted in Wikimedia Labs, runs on cvn-app4.
03:50 Krinkle: Installed pidentd package on cnv-app servers.
March 28
00:00 Krinkle: Migrated web services from pmtpa to eqiad. cvn-apache*.pmtpa.wmflabs can be deleted.
March 27
19:11 Krinkle: Migrated all applications from pmtpa to eqiad. cvn-app*.pmtpa can be shut down.
17:36 Krinkle: Installing packages on new eqiad instances (cvn-app4, cvn-app5, cvn-apache5)
March 18
05:34 Krinkle: Finished migration of all bots from the old and small app server (cvn-app1) to the large cvn-app2. Instance to be decomissioned.
05:26 Krinkle: Created cvn-app3 as the first cvn server in wmflabs-eqiad. Don't use for now (not yet set up with proper packages), but will replace cvn-app1 and cvn-app2 from pmtpa soon.
February 18
20:45 Krinkle: Setting up cvn-app2 as app server (same packages as on cvn-app1)
20:42 Krinkle: Deleted cvn-apache3, re-creating as cvn-apache4 with 'web' security group
17:52 Krinkle: Setting up cvn-apache3 as web server
February 13
23:34 Krinkle: Apache instances keeps going brain dead (time out, no significant spike in ganglia, just gradually going OOM). Rebooted again..
October 21
23:09 Krinkle: The https://cvn.wmflabs.org entry point now supports HTTPS (thanks YuviPanda!)
05:16 Krinkle: Moving CVNBot14 from cvn@willow.toolserver.org to cvn-app1.wmflabs
June 28
17:29 Krinkle: Rebooted cvn-app1
June 10
23:23 Krinkle: Added seahorseruler to cvn access group.
April 24
14:03 matanya: rebooted by krinkle due to out of memory
13:25 matanya: migrating Cluekie from Toolserver to labs
April 10
21:40 Krinkle: CVNBot-nl and CVNBot-es were down because it was unable to open i18n files. Files restored from git, bots automatically started by stillalive.
April 4
23:12 Krinkle: Set up symlinks for cvn-config. Bot in #cvn-wp-nl now uses the Dutch msgs file, #cvn-wp-es uses Spanish msgs file.
22:11 Krinkle: cvn-app1 went OOM and kernel failed because of it (/home and /data unavailable). stillalive has auto-started the bots and we're back up.
March 29
00:19 Krinkle: Installing python-twisted-words and python-mysqldb from apt on cvn-app1 (for cvnclerkbot.py)
March 28
23:40 Krinkle: Installing mysql-server and mysql-client from apt on cvn-app1 (for cvnclerkbot.py)
23:06 Krinkle: Installing subversion from apt on cvn-app1 (for pywikipedia)
March 25
14:04 Krinkle: gluster had another hickup, /data became unaccessible ("Transport endpoint is not connected"). Bots down for almost 24 hours. Rebooting the instance fixed it.
16:12 Krinkle: Depooling cvn-apache1 from application
16:11 Krinkle: Installing php5-cli and mono-complete on cvn-app1
08:05 Krinkle: cvn-apache2 is getting too small for all the cvn apps (wasn't supposed to run them in the first place, just apache). Created instance cvn-app1 with more cpu/memory. Moving over application pool later today.
