Labs Eqiad Migration

Tools to use during migration (well, one tool at least) are documented here: Labs_Eqiad_Migration_Tools

Steps

Set up 2nd openstack infra in labs Done
1. get eqiad labs servers properly organized and named Done
2. Make sure to bond ports for the network node, or use 10Gb Done
3. Prepare new puppet classes for Havana Done
Ensure OpenStackManager works properly with two zones Done
Try the upstream OpenStack modules (Nope!)
Investigate neutron (Nope!)
Set up eqiad, using havana nova-network (Andrew)
1. Set up keystone Done
2. Set up glance, migrate images from pmtpa Done
3. Set up nova services Done
  1. All services running and communicating Done
4. Update eqiad labs network config Done
  1. Move to a larger fixed IP range (currently limited to 10.68.16.0/24) Done
  2. Rearrange interfaces on labnet1001 so that we're using 10g everywhere Done
    1. Reassign eth4.1102 to eth1.1102 Done
    2. Switch eth0 to 10g Done
  3. Figure out why labnet1001 has virbr0/192.168.122 (removed) Done
  4. Instances cannot access virt0/virt1000/other public wmf boxes Done
  5. DNS Done
5. Set up /public/keys shared volume in eqiad nfs Done
6. Create new eqiad bastion cluster
7. Set up wikitech mediawiki mirror on virt1000
8. Rewrite the instance wikistatus code to work with havana Done
Test Done
Migrate shared storage
Make region-specific puppet changes so instances can move from tampa to eqiad Done
Update OpenStackManager on wikitech to be aware of both datacenters Done
1. Test keystone/redis with havana Done
2. Deploy keystone/redis on virt1001 Done
3. Update virt1001 keystone with tampa services, test Done
4. Point wikitech DNS to virt1001 -- or -- deploy multi-region changes on virt0 Done (the latter)
Set up new service group schema to support nfs on eqiad Done
Move some test instances from a project (begin week of 2014-01-27) (mostly Andrew): Done
1. Creating test instances in pmtpa
2. Creating instances with the same name in eqiad
3. Shut down the instances in eqiad and pmtpa
4. Transfer the images from pmtpa to eqiad
5. Start the instances in eqiad and ensure a puppet run makes the system work properly
6. Delete the instances in pmtpa
Get deployment-prep and tools to start moving (begin week of 2014-02-03)
Start moving projects, scheduling each one

Timeline

2012-02-21: Internal test migrations. Requires use of wikitech and virt1000.
2012-02-24: Tools, Deployment-prep migration begins (carefully). Wikitech exposes access to both datacenters.
2012-03-03: Public migration period begins. New instance creation and IP allocation disabled in pmtpa.
2012-03-18: Deadline for self-migration. After this, labs staff starts to shutdown and transfer remaining instances, where they will remain in SHUTOFF state until a specific request is made.

Relevant RT tickets

Relevant Bugzilla tickets

Add eqiad as a production region

All open Labs related bugs in Bugzilla

Meeting notes and other pages

project migration example (gluster)

Eqiad already knows about all the projects in pmtpa. So nothing needs to be done regarding project creation, security groups, or service groups.
Migrate shared storage (home) by rsyncing from labstore2:

   root@labstore2:/tmp# mkdir -p /tmp/testlabs/home
   root@labstore2:/tmp# mount -t glusterfs localhost:/testlabs-home /tmp/testlabs/home
   root@labstore2:/tmp# rsync -S -av /tmp/testlabs/home labstore1001.eqiad.wmnet:/srv/project/testlabs/
   root@labstore2:/tmp# umount /tmp/testlabs/home
   root@labstore2:/tmp# rmdir /tmp/testlabs/home

Migrate shared storage (project)

   root@labstore2:/tmp# mkdir -p /tmp/testlabs/project
   root@labstore2:/tmp# mount -t glusterfs localhost:/testlabs-project /tmp/testlabs/project
   root@labstore2:/tmp# rsync -S -av /tmp/testlabs/project labstore1001.eqiad.wmnet:/srv/project/testlabs/
   root@labstore2:/tmp# umount /tmp/testlabs/project
   root@labstore2:/tmp# rmdir /tmp/testlabs/project

Alternately, with tar (from NFS):

   root@labstore1001:~# ssh labstore4.pmtpa.wmnet tar cf - -C /srv deployment-prep | tar xf - -C /srv/project