Understanding NIC Bonding with Linux

Network card bonding is an effective way to increase the available bandwidth, if it is done carefully. Without a switch that supports 802.3ad, you must have the right hardware to make it work. In this article we will explain how bonding works so you can deploy the right mode for your situation.

Most administrators assume that bonding multiple network cards together instantly results in double the bandwidth and high-availability in case a link goes down. Unfortunately, this is not true. Let’s start with the most common example, where you have a server with high network load, and wish to allow more than 1Gb/s.

Bonding With 802.3Ad

You connect two interfaces to your switch, enable bonding, and discover half your packets are getting lost. If Linux is configured for 802.3ad link aggregation, the switch must also be told about this. In the Cisco world, this is called an EtherChannel. Once the switch knows those two ports are actually supposed to use 802.3ad, it will load balance the traffic destined for your attached server.

This works great if a large number of network connections from a diverse set of clients are connecting. If, however, the majority of the throughput is coming from a single server, you won’t get better than the 1Gb/s port speed. Switches are load balancing based on the source MAC address by default, so if only one connection takes place, it always gets sent down the same link. Many switches support changing of the load balancing algorithm, so if you fall into the single server-to-server category, make sure you allow it to round-robin the Ethernet frames.

Alternatively, you don’t need to burn the expensive switch ports at all. Both servers can be connected together via crossover cables to the bonded interfaces. In this configuration, you want to use balance-rr mode on both sides, which we will explain momentarily.

Generic Bonding

There are multiple modes you can set in Linux, and the most common “generic” one is bonding-alb. This mode works effectively in most situations, without needing to configure a switch or trick anything else. It does, however, require that your network interface support changing the MAC address on the fly. This mode works well “generically” because it is constantly swapping MAC addresses to trick the other end (be it a switch or another connected host) into sending traffic across both links. This can wreak havoc on a Cisco network with port security enabled, but in general it’s a quick and dirty way to get it working.

Channel Bonding Modes

Channel Bonding modes can be broken into three categories: generic, those that require switch support, and failover-only.

The failover-only mode is active-backup: One port is active until the link fails, then the other takes over the MAC and becomes active.

Modes that require switch support are:

• balance-rr: Frames are transmitted in a round-robin fashion without hashing, to truly load balance.
• 802.3ad: This mode is the official standard for link aggregation, and includes many configurable options for how to balance the traffic.
• balance-xor: Traffic is hashed and balanced according to the receiver on the other end. This mode is also available as part of 802.3ad.

Note that modes requiring switch support can be run back-to-back with crossover cables between two server as well. This is especially useful, for example, when using DRBD to replicate two partitions.

Generic modes include:

• broadcast: This mode is not really link aggregation – it simply broadcasts all traffic out both interfaces, which can be useful when sending data to partitioned broadcast domains for high availability (see below). If using broadcast mode on a single network, switch support is recommended.
• balance-tlb: Outgoing traffic is load balanced, but incoming only uses a single interface. The driver will change the MAC address on the NIC when sending, but incoming always remains the same.
• balance-alb: Both sending and receiving frames are load balanced using the change MAC address trick.

High Availability

How often have you seen a network die catastrophically? So bad that the link died? Chances are: never. More often you will see packet loss and very strange behavior. The failover part of NIC bonding is quite attractive to many administrators, but it rarely ever works. When the switch that both ports is connected to reboots for a firmware upgrade, you are down.

The easy fix is to connect each port to two distinct switches, right? If you are using a bonding mode that doesn’t require switch support this will work fine. If, however, you are using a mode that requires switch support, this is not possible on most devices. Switches that support stacking, and are managed from a single point, often support EtherChannel across multiple switches. Ideally, you would would connect one port to each, and never reboot the whole stack of switches simultaneously.

Decisions

Bonding is simple once you understand the limitations of each mode. if you’re working in an environment where switches support 802.3ad and you have no special needs, use that mode. Conversely, if you have no switch support and just want to increase throughput and enable failover, use balance-alb. Finally, if you just need a data replication link between two servers, balance-rr is the way to go.