Storage Networking 101: Understanding Fibre Channel Domains
Anyone can plug in a SAN. Understanding Fibre Channel domains, address assignment and VSANs will help you make that SAN resilient to change.
First, we must understand how a SAN fabric exists without loops. Everything you see here will look suspiciously familiar to Spanning Tree. A few terms are different, of course, but the same concept applies.
The Domain_ID is dynamically assigned to a FC switch when it comes online. The Principal Switch (PS) election begins, which is very similar to a root bridge election in Spanning Tree, followed by the Domain_ID Distribution process.
Briefly, this is the election process for determining the PS:
- Clear Domain_ID list
- On each inter-switch link (E-Ports), transmit the Build Fabric (BF) frame; do not send one on a port that you've received a BF on, to prevent loops
- Wait for the Fabric Stability Timeout, to ensure the BF frames have been flooded throughout the entire fabric
- Transmit an EFP (Exchange Fabric Parameters) frame, and send SW_ACC (Switch Accept) to each transmitter of these frames
- Examine the EFP frame, looking for PS_Priority, PS_Name (the Node WWN of the switch), and the Domain_ID list
- Concatenate the PS_Priority and PS_Name to select the winner; lowest number wins
- Repeat until everyone attached agrees on the PS
After completion of the PS election, a switch must begin the Domain_ID Distribution process. Even if the Domain_ID is manually configured, the distribution process still occurs, because the PS needs to compile a list of Domain_IDs. The Domain_ID election process isn't really important, because most people configure the domain manually. Just know that a change of a Domain_ID results in everyone sending an EFP frame with the updated information.
Configuring the Domain_ID is important, because merging fabrics can be disruptive if conflicting Domain_IDs are present. When you have a single switch, and want to extend the fabric by connecting the two together, everything goes fine unless they're both Domain_ID 1, as some vendors set by default. Every new switch that's brought online needs to be configured with a unique Domain_ID before connecting it to the fabric.
Conflicting Domain_IDs frequently happen when using VSANs. A VSAN is the same as a VLAN, but for FC networks. You can configure a VSAN-capable switch (usually a Cisco) to segment ports into separate fabrics. One node connected to switch port 1 may be in fabric 322, while the node right next to it lives in fabric 4; two completely separate fabrics. Each fabric may have a domain 31, for example. For the most part, excluding some fanciness implemented by a few vendors, there is no inter-fabric routing, so nodes in different fabrics won't be able to talk to each other. This is wonderful, but often times it's necessary to merge two fabrics together.
Merging two fabrics is normally accomplished by connecting multiple switches together. If a "core" switch already had a link to two switches, and suddenly decides to merge the fabrics by placing them in the same VSAN, those switches better have unique Domain_IDs. If not, traffic will suddenly be spotty, since the FCIDs include the Domain_ID. Furthermore, each PS in a domain runs its own name server containing information about N_Ports, and when receiving a frame, a switch will not know which way to send it if it has conflicting information.
Just like VLANs, a VSAN can be used to implement arbitrary boundaries, in ways that make administration much more tolerable, compared to manually moving wiring. The Cisco VSAN technology is gaining widespread adoption since ANSI blessed its implementation, calling it "Virtual Fabrics." The neat thing about a VSAN is that it's more capable than the Ethernet's VLANs.
The Virtual Fabric model takes virtualization to the next level. It is possible to configure a zone server, so that all fabric-attached nodes know how to reach it. FC services run on a switch, unlike the IP world where services like DHCP and DNS normally run on a host. In a VSAN environment, the switch actually runs each service multiple times, once in each fabric.
Speaking of fabric services, there are a few well-known FC addresses associated with SAN services. The brief list is:
- 0xFF FF F5: Multicast server
- 0xFF FF F6: Clock Sync server
- 0xFF FF F7: KDC (key distribution)
- 0xFF FF F8: Alias server (for multicast, or hunt groups)
- 0xFF FF F9: QoS information
- 0xFF FF FA: Management server
- 0xFF FF FB: Time server
- 0xFF FF FC: Directory server
- 0xFF FF FD: Fabric Controller
- 0xFF FF FE: Fabric Login server
FC addresses (the FCID) aren't actually necessary for SCSI over FC operation. Unicast FC frames are sent to and from the WWN of the node, so the FC address is really only needed in two cases: during link initialization, or when sending IP over FC. When sending IP over FC, and IP address needs to be turned into the FCID. Very similar to the Ethernet world, ARP is used in FC land. Either "ARP over FC" or FARP, which are two distinct protocols, is possible, depending on what the devices support. And you wondered why FC has so many interoperability issues?
Join us next time when we talk about Fabric Zoning and switch configuration.