Link aggregation

Wikimedia infrastructure
Data centres and PoPs Networking Network design Peering Link aggregation LVS HTTP Caching Wiki Media Logs

There are several ways to aggregate multiple links into one, bigger, more redundant link.

Server link aggregation

Before we did (sporadically) aggregate links using manual configuration on Linux servers, usually statically, without LACP. Now, we have a Puppet definition for it: interface_aggregate in generic-definitions.pp. This sets up an LACP aggregated link.

Use the following switch side configuration:

Juniper

[edit interfaces]
ge-4/0/22 {
    apply-groups-except access-port;
    description arsenic;
    ether-options {
        802.3ad {
            lacp {
                force-up;
            }
            ae3;
        }
    }
}
...
ge-4/0/30 {
    apply-groups-except access-port;
    description arsenic:eth1;
    ether-options {
        802.3ad ae3;
    }
}
ge-4/0/31 {
    apply-groups-except access-port;
    description arsenic:eth2;
    ether-options {
        802.3ad ae3;
    }
}
ge-4/0/32 {
    apply-groups-except access-port;
    description arsenic:eth3;
    ether-options {
        802.3ad ae3;
    }
}
...
ae3 {
    description arsenic:bond0;
    mtu 9192;
    aggregated-ether-options {
        lacp {
            periodic fast;
        }
    }
    unit 0 {
        family ethernet-switching {
            port-mode access;
            vlan {
                members public1-a-eqiad;
            }
        }
    }
}

• Remove the individual (ge-x/x/x or whatever) interfaces from any interface-range lists that put them into vlans, since the aggregated port is added directly. Look at

  interfaces interface-range

• Check and increment if you added a new aggregated device:

  chassis aggregated-devices ethernet device-count

The apply-group access-port needs to deactivated with apply-groups-except access-port on the individual interfaces used for the aggregated link, as individual MTU and 802.1q configuration is now allowed.

Note that we define the interface connected to eth0 as force-up (always up). When Linux isn't running, for example during PXE boot or during an installation, LACP is not active, and then the switch accepts traffic from eth0 (only) as if there is no aggregated link.

Foundry

The following configuration sets up an LACP (dynamic) aggregate with fast PDU rate:

lag "nas1-a" dynamic
 ports e 15/5 to 15/6 
 primary-port 15/5
 lacp-timeout short
 deploy
 port-name "nas1-a:e0a" ethernet 15/5                             
 port-name "nas1-a:e0b" ethernet 15/6
!

Unfortunately, Foundry doesn't seem to support a way to force one interface to up in the absence of LACP packets. Therefore the LAG will need to be undeployed during PXE boots. :-(

Linux/Puppet

In Puppet, the interface_aggregate definition can be used, like this:

interface_aggregate { "bond0":
	orig_interface => "eth0",
	members => [ "eth0", "eth1", "eth2", "eth3" ],
	lacp_rate => "fast"
}

orig_interface contains the original interface from which IP configuration should be taken. This interface is converted to the bonding interface (bond0 here). lacp_rate, default fast defines the LACP PDU transmission rate. Our Juniper and Foundry switches support fast, but some may only support "slow".

This sets up file /etc/network/interfaces. Check /sys/class/net/bond0/bonding/ for debugging.