Tips and Tuning for Ethernet Bonding With Linux
Last week we whipped up a quick Ethernet bonding setup that aggregated the bandwidth of two Ethernet interfaces without needing a special switch to do it, so it works with plain old cheap dumb switches, which is a very nice thing.
There is a dark side to this wonderful feature of the Linux bonding driverit only works with network interfaces that allow the MAC address to be changed when the interface is open. The balance-alb mode depends on swift ARP trickery to fool the kernel into thinking the two physical interfaces are one by rewriting the MAC address on the fly. So the driver for the interface must support this, and many of them don't. Here are some of the more popular drivers that don't support balance-alb:
- The 3c59x driver which supports the EtherLink Vortex, Boomerang, Tornado and Cyclone series of chipsets
- The tulip driver, which supports some ADMtek, D-Link, Netgear, SMC, and Znyx interfaces
- The via-rhine driver, which supports onboard VIA chipsets
Some interfaces that do support balance-alb are:
- Intel Pro 100 and Pro 1000, using the e100 and e1000 drivers
- Broadcom NetXtreme and NetLink (tg3), NetXtreme II (bnx2), and 4401 (b44)
- Nvidia CK8S Ethernet Controller, which is the onboard controller for motherboards that use the nForce and GA-K8NS chipsets (forcedeth)
There is no way to compile a definitive list of supported NICs, so I am going to show you a slick trick for quickly finding out which drivers support balance-alb. You need a kernel source tree, which you can have on your system, or just find one online. To find one online simply search for "kernel lxr." lxr is the Linux cross-referencing utility, and happily many sites use it to maintain online kernel source trees, such as lxr.linux.no/source. (This also has all the kernel documentation, which is a gold mine of great information.) Find the "freetext search" box and enter set_mac_address. You'll get a bunch of hits like this:
set_mac_address is the function that allows the MAC address to be changed while the interface is open. If this doesn't exist the balance-alb mode won't work. If it does exist, most likely it will.
To search your own local kernel source tree, use grep to search your drivers/net directory. You only want the C files, because these contain the source code in plain text:
$ grep -r 'set_mac_address' /home/carla/kernel/linux-18.104.22.168/drivers/net/*.c
This doesn't tell you brand or model names, but you'll find these quickly enough with a web search.
Other Bonding Options
Up to this point we've only talked about enabling the bonding-alb option. But that's not all the bonding driver can do. The mode option gives you seven choices, and you don't have to worry about interface compatibility. However you do need to consider what your switches support. The balance-rr, balance-xor and broadcast modes need switch ports grouped together. This goes by all sorts of different names, so look for "trunk grouping", "etherchannel", "port aggregation", or some such. 802.3ad requires 802.3ad support in the switch.
balance-rr or 0
This is the default, transmitting packets in order from the first available slave through the last, for load balancing and fault tolerance.
active-backup or 1
Only one of the bond slaves is used, and the others sit idle until the active slave fails.
balance-xor or 2
This has two possible behaviors, which you control with the xmit_hash_policy option. The default (layer2) is to generate a simple transmit hash based on the MAC addresses to decide which slave will carry a particular traffic stream. The nice thing about this is all traffic destined for a network peer will use the same interface, rather than splitting it up and possibly creating a bunch of out-of-order packets.
The other option, layer3+4, mimics some Cisco and other big name network devices. This stripes traffic across several slaves, which can be very fast, but it can also create out-of-order delivery. Under most circumstances on well-behaved networks this should not be a problem.
broadcast or 3This transmits everything on all slave interfaces, so it's useful mainly when each interface is connected to a separate switch going to separate networks.
802.3ad or 4This is the official standards-based link aggregation technology for LAN switch ports, and for high-end server NICs. You still need a switch that supports 802.3ad, but you can use any Ethernet interface with a driver that supports Ethtool. Most Linux Ethernet drivers support Ethool, but a very few don't. Just run the ethtool command to find out:
# ethtool eth0
Settings for eth0:
No data available
Oops, no support here. You should see some output like this:
Settings for eth1:
Supported ports: ' TP MII '
Supported link modes: 10baseT/Half 10baseT/Full
balance-tlb or 5Adaptive transmit load balancing- outgoing traffic gets the benefit of bonded interfaces without needing support in the switch, but not incoming. The interface driver must support Ethtool. This also provides redundancy and failover.
balance-alb or 6
We know all about thisit's similar to balance-tlb, except both send and receive traffic are bonded. This requires Ethtool support, and support for changing MAC addresses on active interfaces in the interface's driver.
active-backup and broadcast don't have that many practical applications. balance-rr and 802.3ad are probably the most useful on networks that use managed switches, for both performance and availability.
In part 1 we learned how to configure Debian. Fedora is just the same, only different. First create a configuration file for your bond interface, like /etc/sysconfig/network-scripts/ifcfg-bond0. Remember that these files have a strict naming convention–they must say ifcfg- followed by the interface's kernel name. Configure it as you would any interface:
Then create files for each of your slaves:
Then add your alias and module options to /etc/modules.conf:
alias bond0 bonding
options bond0 mode=balance-rr miimon=100
Run modprobe bonding, restart networking, and you should see your nice new bonded interface come up with happy messages like
Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
And ifconfig will show all of your interfaces, both the bond and the slaves.
You can test everything on the command line before committing your configurations to files. (See part 1.) You can unload kernel modules without rebooting using rmmod:
# rmmod bonding
Remember lsmod for listing loaded kernel modules, and cat /proc/net/bonding/bond0 to view your interface statistics.
What if things don't work and you get weirdo error messages that don't help? Just fire up dmesg and see if it brings enlightenment. You should also check /var/log/messages, and you can even follow along in realtime by running tail -f /var/log/messages. But most likely dmesg will tell you more.
Well now, that was quite a whirlwind tour. There are many more useful things to know, such as controlling your bonded interfaces with sysfs instead of ifenslave, additional bonding driver parameters, creating multiple bonded interfaces on a single machine, configuring switches, routing, and lots of additional information on the bits we already covered. You'll find all of this in the Documentation/networking/bonding.txt file in the kernel sources, which is a complete, well-written document.
- Thank you to Akkana Peck and the bonding driver developers for their excellent assistance