Tips and Tuning for Ethernet Bonding With Linux

Getting the Linux bonding driver to work is just the start: Getting the most out of it requires a little digging.

 By Carla Schroder
Page 1 of 2
Print Article

Last week we whipped up a quick Ethernet bonding setup that aggregated the bandwidth of two Ethernet interfaces without needing a special switch to do it, so it works with plain old cheap dumb switches, which is a very nice thing.

There is a dark side to this wonderful feature of the Linux bonding driver–it only works with network interfaces that allow the MAC address to be changed when the interface is open. The balance-alb mode depends on swift ARP trickery to fool the kernel into thinking the two physical interfaces are one by rewriting the MAC address on the fly. So the driver for the interface must support this, and many of them don't. Here are some of the more popular drivers that don't support balance-alb:

  • The 3c59x driver which supports the EtherLink Vortex, Boomerang, Tornado and Cyclone series of chipsets
  • The tulip driver, which supports some ADMtek, D-Link, Netgear, SMC, and Znyx interfaces
  • The via-rhine driver, which supports onboard VIA chipsets

Some interfaces that do support balance-albare:

  • Intel Pro 100 and Pro 1000, using the e100 and e1000 drivers
  • Broadcom NetXtreme and NetLink (tg3), NetXtreme II (bnx2), and 4401 (b44)
  • Nvidia CK8S Ethernet Controller, which is the onboard controller for motherboards that use the nForce and GA-K8NS chipsets (forcedeth)

There is no way to compile a definitive list of supported NICs, so I am going to show you a slick trick for quickly finding out which drivers support balance-alb. You need a kernel source tree, which you can have on your system, or just find one online. To find one online simply search for "kernel lxr." lxr is the Linux cross-referencing utility, and happily many sites use it to maintain online kernel source trees, such as lxr.linux.no/source. (This also has all the kernel documentation, which is a gold mine of great information.) Find the "freetext search" box and enter set_mac_address. You'll get a bunch of hits like this:

97% drivers/net/qla3xxx.c
97% drivers/net/defxx.c
96% drivers/net/wireless/atmel.c

set_mac_address is the function that allows the MAC address to be changed while the interface is open. If this doesn't exist the balance-alb mode won't work. If it does exist, most likely it will.

To search your own local kernel source tree, use grep to search your drivers/net directory. You only want the C files, because these contain the source code in plain text:

$ grep -r 'set_mac_address' /home/carla/kernel/linux-*.c

This doesn't tell you brand or model names, but you'll find these quickly enough with a web search.

Other Bonding Options

Up to this point we've only talked about enabling the bonding-alb option. But that's not all the bonding driver can do. The mode option gives you seven choices, and you don't have to worry about interface compatibility. However you do need to consider what your switches support. The balance-rr, balance-xor and broadcast modes need switch ports grouped together. This goes by all sorts of different names, so look for "trunk grouping", "etherchannel", "port aggregation", or some such. 802.3adrequires 802.3ad support in the switch.

balance-rr or 0

This is the default, transmitting packets in order from the first available slave through the last, for load balancing and fault tolerance.

active-backup or 1

Only one of the bond slaves is used, and the others sit idle until the active slave fails.

balance-xor or 2

This has two possible behaviors, which you control with the xmit_hash_policy option. The default (layer2) is to generate a simple transmit hash based on the MAC addresses to decide which slave will carry a particular traffic stream. The nice thing about this is all traffic destined for a network peer will use the same interface, rather than splitting it up and possibly creating a bunch of out-of-order packets.

The other option, layer3+4, mimics some Cisco and other big name network devices. This stripes traffic across several slaves, which can be very fast, but it can also create out-of-order delivery. Under most circumstances on well-behaved networks this should not be a problem.

broadcast or 3

This transmits everything on all slave interfaces, so it's useful mainly when each interface is connected to a separate switch going to separate networks.

802.3ad or 4

This is the official standards-based link aggregation technology for LAN switch ports, and for high-end server NICs. You still need a switch that supports 802.3ad, but you can use any Ethernet interface with a driver that supports Ethtool. Most Linux Ethernet drivers support Ethool, but a very few don't. Just run the ethtool command to find out:

# ethtool eth0
Settings for eth0:
No data available

Oops, no support here. You should see some output like this:

Settings for eth1:
Supported ports: ' TP MII '
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full

balance-tlb or 5

Adaptive transmit load balancing- outgoing traffic gets the benefit of bonded interfaces without needing support in the switch, but not incoming. The interface driver must support Ethtool. This also provides redundancy and failover.

balance-alb or 6

We know all about this–it's similar to balance-tlb, except both send and receive traffic are bonded. This requires Ethtool support, and support for changing MAC addresses on active interfaces in the interface's driver.

active-backup and broadcast don't have that many practical applications. balance-rr and 802.3ad are probably the most useful on networks that use managed switches, for both performance and availability.

This article was originally published on Sep 4, 2007
Get the Latest Scoop with Networking Update Newsletter