Harden Your Sun Server Against Network Flickers
Having a redundant layer two network à la STP doesn't keep your servers up and running when the switch they're connected to becomes unavailable. As long as there are bugs, you will always have to patch and reboot switches and routers; there's no getting around that. But you don't have to render your file and application server unreachable just because its switch has to reboot.
First, let's clear up some confusion.
We aren't talking about load-balancing, link aggregation, or trunking. Also, trunking in this sense is referring to load-sharing, not 802.1q VLAN trunks. You can achieve other sorts of redundancy using VRRP (virtual router redundancy protocol) (define) and similar technologies, but we're going to talk about multi-NIC failover today. While Sun's IPMP can use two IP addresses, we aren't really interested in that feature for high availability purposes. The goal is to use a single public IP address, and have the OS decide which NIC it should use based on availability.
IP Multipathing (IPMP) from Sun allows you to connect the same server to multiple switches on the same subnet. The implication is that you will have an 802.1q trunk between the switches, e.g. VLAN 2 will be the same everywhere. It is also important to have multiple standby routers on the same subnet, but that is beyond the scope of this article. You clearly can't assign the same IP address to two network cards, but the net effect of IPMP can be thought of in this manner.
Active-active configurations allow for load-balancing and high throughput, but active-standby configurations allow for more resilience. In active-standby mode, the server won't use the secondary link unless an active link fails. In the event of failure, all active connections are moved to the working link, and everything just keeps ticking. You don't have to configure your services to listen on multiple interfaces, because the same IP is used in a failure situation. Conversely, if you wanted to load-balance with an active-active configuration, you'd have to configure two active IP addresses, and make sure that all your services listened on both addresses.
Active-standby mode works differently in Solaris 9 than it does in Solaris 10. In both versions the ifconfig group command configures IPMP, and you can accomplish the same things. The difference is in how it works under the covers. In both 9 and 10, the in.mpathd daemon will monitor the interfaces to verify the link state. If the physical media goes down, failover doesn't necessarily happen in Solaris 9. Instead, it relies on ping responses from other devices on the subnet to determine if the subnet is unreachable. Solaris 10's daemon is a bit more advanced, as it appears to monitor more than just the link state of the network card, and does not rely on ping responses. The rate of pings is configurable in /etc/default/mpathd , but the default of 10 seconds seems to be a reasonable trade-off of 10-second failover vs. increased ping traffic.
It's surprisingly easy to configure a server that is connected as described above. First, you must undo Sun's strange default, which assigns every interface on a machine the same MAC address, with the following command:
Be sure to quote the argument or escape the question mark so your shell doesn't try to eat it.
Next, you have the option of configuring everything live, or setting up the configuration files and rebooting. We're going to choose the configuration file method with reboot, so we're certain the server will survive a future reboot. The first step is to configure the existing interface in /etc/hostname. <interface>:
10.1.1.1 netmask + broadcast + group test up
addif 10.1.1.2 deprecated -failover netmask + broadcast + up
Replace the first IP address with the real IP for the server, and "test" with the group name you wish to use. The second IP address is the address that will be used for testing (the pings), so it must be a unique and unused address on the same subnet. The "deprecated" option is important; it keeps applications from binding to that IP address and using it.
Next, the standby interface must be configured. In the other interface configuration file, /etc/hostname.<interface2>, place:
10.1.1.3 netmask + broadcast + deprecated group test -failover standby up
This sets up the test interface (for pings) only. If you attempt to configure a second IP address similar to the primary NIC's configuration, you'll be accidentally configuring an active-active setup.
Reboot and you're configured with standby capabilities! The output of ifconfig -a will reveal the details, and indicate which interfaces are in standby mode. Now start a ping to the server (10.1.1.1 in this example) and unplug the primary network cable. You will notice in.mpathd logging that it's failing over, and the pings will continue working. When the link returns, another syslog entry will say that it's back, and the standby interface goes idle again. You'll actually want to use hostnames in the interface configuration files, and add those entries to /etc/hosts , so that in the event of IP renumbering the change only has to happen in one place.
Failover in this manner is attainable with Linux and FreeBSD too. Linux channel bonding, coupled with an appropriate Cisco EtherChannel configuration on the switch side will offer both increased throughput and failover capabilities. In FreeBSD, if you have some serious netgraph knowledge, a similar setup can be accomplished.
Sun's implementation of IPMP works very well, and is really simple to configure. More involved setups, including active-active configuration examples, can be found on Sun's website. You'll have to search for "ip multipathing" on their website, because their URLs seem to change daily. Now go forth and bring network failover capabilities to your important servers.