Labs Eqiad Migration

From Wikitech
Jump to navigation Jump to search

Tools to use during migration (well, one tool at least) are documented here: Labs_Eqiad_Migration_Tools


  1. Set up 2nd openstack infra in labs Yes check.svg Done
    1. get eqiad labs servers properly organized and named Yes check.svg Done
    2. Make sure to bond ports for the network node, or use 10Gb Yes check.svg Done
    3. Prepare new puppet classes for Havana Yes check.svg Done
  2. Ensure OpenStackManager works properly with two zones Yes check.svg Done
  3. Try the upstream OpenStack modules (Nope!)
  4. Investigate neutron (Nope!)
  5. Set up eqiad, using havana nova-network (Andrew)
    1. Set up keystone Yes check.svg Done
    2. Set up glance, migrate images from pmtpa Yes check.svg Done
    3. Set up nova services Yes check.svg Done
      1. All services running and communicating Yes check.svg Done
    4. Update eqiad labs network config Yes check.svg Done
      1. Move to a larger fixed IP range (currently limited to Yes check.svg Done
      2. Rearrange interfaces on labnet1001 so that we're using 10g everywhere Yes check.svg Done
        1. Reassign eth4.1102 to eth1.1102 Yes check.svg Done
        2. Switch eth0 to 10g Yes check.svg Done
      3. Figure out why labnet1001 has virbr0/192.168.122 (removed) Yes check.svg Done
      4. Instances cannot access virt0/virt1000/other public wmf boxes Yes check.svg Done
      5. DNS Yes check.svg Done
    5. Set up /public/keys shared volume in eqiad nfs Yes check.svg Done
    6. Create new eqiad bastion cluster
    7. Set up wikitech mediawiki mirror on virt1000
    8. Rewrite the instance wikistatus code to work with havana Yes check.svg Done
  6. Test Yes check.svg Done
  7. Migrate shared storage
  8. Make region-specific puppet changes so instances can move from tampa to eqiad Yes check.svg Done
  9. Update OpenStackManager on wikitech to be aware of both datacenters Yes check.svg Done
    1. Test keystone/redis with havana Yes check.svg Done
    2. Deploy keystone/redis on virt1001 Yes check.svg Done
    3. Update virt1001 keystone with tampa services, test Yes check.svg Done
    4. Point wikitech DNS to virt1001 -- or -- deploy multi-region changes on virt0 Yes check.svg Done (the latter)
  10. Set up new service group schema to support nfs on eqiad Yes check.svg Done
  11. Move some test instances from a project (begin week of 2014-01-27) (mostly Andrew): Yes check.svg Done
    1. Creating test instances in pmtpa
    2. Creating instances with the same name in eqiad
    3. Shut down the instances in eqiad and pmtpa
    4. Transfer the images from pmtpa to eqiad
    5. Start the instances in eqiad and ensure a puppet run makes the system work properly
    6. Delete the instances in pmtpa
  12. Get deployment-prep and tools to start moving (begin week of 2014-02-03)
  13. Start moving projects, scheduling each one


  • 2012-02-21: Internal test migrations. Requires use of wikitech and virt1000.
  • 2012-02-24: Tools, Deployment-prep migration begins (carefully). Wikitech exposes access to both datacenters.
  • 2012-03-03: Public migration period begins. New instance creation and IP allocation disabled in pmtpa.
  • 2012-03-18: Deadline for self-migration. After this, labs staff starts to shutdown and transfer remaining instances, where they will remain in SHUTOFF state until a specific request is made.

Relevant RT tickets

Relevant Bugzilla tickets

All open Labs related bugs in Bugzilla

Meeting notes and other pages

project migration example (gluster)

  1. Eqiad already knows about all the projects in pmtpa. So nothing needs to be done regarding project creation, security groups, or service groups.
  2. Migrate shared storage (home) by rsyncing from labstore2:
   root@labstore2:/tmp# mkdir -p /tmp/testlabs/home
   root@labstore2:/tmp# mount -t glusterfs localhost:/testlabs-home /tmp/testlabs/home
   root@labstore2:/tmp# rsync -S -av /tmp/testlabs/home labstore1001.eqiad.wmnet:/srv/project/testlabs/
   root@labstore2:/tmp# umount /tmp/testlabs/home
   root@labstore2:/tmp# rmdir /tmp/testlabs/home
  1. Migrate shared storage (project)
   root@labstore2:/tmp# mkdir -p /tmp/testlabs/project
   root@labstore2:/tmp# mount -t glusterfs localhost:/testlabs-project /tmp/testlabs/project
   root@labstore2:/tmp# rsync -S -av /tmp/testlabs/project labstore1001.eqiad.wmnet:/srv/project/testlabs/
   root@labstore2:/tmp# umount /tmp/testlabs/project
   root@labstore2:/tmp# rmdir /tmp/testlabs/project
  1. Alternately, with tar (from NFS):
   root@labstore1001:~# ssh labstore4.pmtpa.wmnet tar cf - -C /srv deployment-prep | tar xf - -C /srv/project