As we dive deeper into SAN technology, it’s Fibre Channel’s turn to be examined. Fibre Channel, or FC, is the underpinning of all SAN technologies these days, as it won the protocol war roughly 25 years ago.
FC wouldn’t be much use without something on top of it, namely SCSI. FC is the low-level transport that ships data, but hosts are normally communicating via SCSI as far as they’re concerned. The hubs, switches and HBAs in a SAN all speak FC, while the applications that use SAN storage continue to use familiar protocols, like SCSI.
The idea behind FC was to create a high throughput, low latency, reliable and scalable protocol. Ethernet wouldn’t quite cut it for highly-available storage needs. FC can currently operate of speeds up to 10Gb/s (10GFC) for uplinks, and 4Gb for standard host connections. FC also provides small connectors. As silly as it sounds, SCSI cables become unruly after time, and small strands of fiber are certainly easier to manage. The equipment required to connect to a FC SAN (multiple HBAs for each host, fiber, and switches) is extremely expensive, and was the main reason SAN technologies took so long to become widely adopted.
Topologies
In reality, two different protocols, or topologies, make up the FC protocol. FC supports all topologies, but the behavior of the protocol changes depending on the topology. The following three types of topologies are supported:
- PTP (point to point): normally used for DAS configurations.
- FC-AL (FC Arbitrated Loop): Fabric Loop ports, or FL ports on a switch, and NL_Ports (node loop) on an HBA, support loop operations.
- FC-SW (FC Switched): the mode when operating on a switched SAN.
Useful Terms
- » HBA
- » Fibre Channel
- » SCSI
- » DAS
FC-AL operation has its share of problems, but sometimes a device doesn’t support FC-SW operations, and there’s no choice. A hub has no choice but to operate in FC-AL mode, and therefore attached hosts must as well. When a device joins an FC-AL, or when there’s any type of error or reset, the loop must reinitialize. All communication is temporarily halted during this process, so it can cause problems for some applications. FC-AL is limited to 127 nodes due to the addressing mechanism, in theory, but in reality closer to 20. FC-AL is mostly relegated to niche uses now, including but not limited to internal disk array communications and internal storage for high-end servers.
FC switches can be connected any way you please, since the FC protocol avoids the possibility of a loop by nature. Ethernet isn’t so lucky. The addressing scheme used does impose a limit of 239 switches though. FC switches use FSPF, a link-state protocol like OSPF in the IP world, to ensure loop-free and efficient connectivity.
FC networks are generally designed in one of two ways: either one big star, or one big star with edge switches hanging off it. These are commonly known as “core-only” and “core-edge” configurations. Normally a SAN will contain two of these networks, and each host’s HBA or storage device’s controller will attach to each. Keeping these networks separate isn’t as necessary as it is with FC-AL topologies, but even with FC-SW setups it still provides complete isolation and assurance that a problem in one fabric won’t impact the other. An FSPF recalculation, for example, could cause a brief interruption in service.
Ports
As previously mentioned, there are different port types in a SAN, and it can get confusing. Let’s try to clear up some of that terminology:
- N_Port: Node Port; the node connection point; end points for FC traffic
- F_Port: Fabric Port; a switch-connected port, that is a “middle point” connection for two N_Ports
- NL_Port: Node Loop Port; connects to others via their NL_Ports, or to a switched fabric via a single FL_Port; or NL_port to F_Port to F_Port to N_Port (through a switch)
- FL_Port: Fabric Loop Port; a shared point of entry into a fabric for AL devices; example NL_Port to FL_Port to F_Port to N_Port
- E_Port: Expansion Port; used to connect multiple switches together via ISL (inter-switch links)
- G_Port: Generic Port; can switch between F_Port and E_Port operation depending on how it’s connected
- TE_Port: Trunked Expansion Port; link aggregation of multiple E_Ports for higher throughput
You’ll generally only see F_Ports and FL_Ports when looking at a single SAN switch, and knowing the difference helps. FL means that you’re talking FC-AL, and there’s a device attached that is either a hub, something that can’t do anything but FC-AL, or something strange. Ports will automatically configure themselves as an FL_Port if the attached device is Loop-only, otherwise it will be an F_Port. It’s also worth noting that some brands of FC switches don’t allow you to have an E_Port unless you pay a higher licensing fee. It’s something to think about if you ever plan to connect multiple switches together.
FC Layers
FC has its own layers, so in fact, calling it “like Ethernet” isn’t quite accurate, even if it helps for understanding. They are:
- FC-0: The interface to the physical media; cables, etc
- FC-1: Transmission protocol or data-link layer, encodes and decodes signals
- FC-2: Network Layer; the core of FC
- FC-3: Common services, like hunt groups
- FC-4: Everything! Protocol mapping for SCSI, iSCSI, FCP, IP, and others
The bulk of FC is really in FC-2. FC-PH refers to FC-0 through FC-2, which are strangely dubbed “the physical layers.”
FC also supports its own naming and addressing mechanism, which sheds light on the previously mentioned limitations in FC-AL and FC-SW topologies. Next time, we’ll discuss the header format for FC-2 as well as FC address assignment and name resolution.
In a Nutshell
- FC is the transport mechanism, and SCSI or even IP ride atop FC
- FC-AL is a loop, where all connected devices see each other, and a re-initialization takes out the entire SAN
- Port types reveal what is actually happening, and knowing what they stand for can aid in topology visualization when looking at a switch’s configuration