Storage Networking 101: Understanding the Fibre Channel Protocol

Understanding the guts of the Fibre Channel (FC) protocol itself, including the naming format and addressing scheme, allows one to better understand what’s happening on a SAN. Quickly glancing at a problem and knowing what’s wrong requires thorough knowledge of all the protocols involved. While it’s possible to operate a SAN with only point-and-click GUIs and limited knowledge, it certainly isn’t recommended. So let’s learn about the FC protocol.

To reiterate: Fibre Channel is not a replacement for SCSI; SCSI generally rides on top of Fibre Channel. Now that we have that out of the way, let us get to work. FC generally refers to the FC-PHY layers: FC0-FC2, which were briefly discussed in our last installment. The term FCP, Fibre Channel Protocol, refers to the interface protocol for SCSI, or the FC-4 mapping. We’re talking about the inner-workings of FC here, not FCP.

FC data units are called Frames. FC is mostly a layer 2 protocol, even though it has its own layers. The maximum size for a FC frame is 2148 bytes, and the header FC frame itself is a bit strange, at least when compared to Ethernet with IP and TCP. FC uses one frame format for many purposes, and at many layers. The function of the frame determines the format, which is strange and wonderful, compared to our notions in the IP world.

FC frames begin with a start-of-frame (SOF) marker followed by frame the header, which will be described in a moment. The data, or FC content, comes next, followed by an EOF. The reason for the encapsulation is so that FC can be carried over other protocols, such as TCP if desired.

Figure 1. The FC Encapsulation Header

The FC frame itself, the general format that is, varies in size quite a bit. In Figure 1 you can see the SOF and EOF markers we mentioned before. The strange part about FC headers is that they are word-oriented, and an FC word is 4 bytes. Up to 537 words are allowed, which gives us our 2148-byte capacity.

The components of the header, with all the optional items listed, are:

• SOF (1 word): The start of a frame.
• Frame Header (24 bytes): The header that specifies what protocol is being used, as well as the source and destination address. Varies depending on the protocol in question.
• Optional ESP Header (8 bytes): Provides encryption; includes the SPI and ESP sequence number.
• Optional Network Header (16 bytes): So that you can connect an FC-SAN to non-FC networks.
• Optional Association Header (32 bytes): Not used by FCP, but can be used to identify processes within a node.
• Optional Device Header (up to 64 bytes): Not used by FCP, and is application specific.
• Payload: The data, up to 2048 bytes.
• Optional Fill Bytes (variable): Used to ensure the variable-length payload ends on a word boundary.
• Optional ESP Trailer (variable): Contains check values for ESP.
• CRC (4 bytes): A CRC of the header and FC data fields.
• End of Frame (4 bytes): Ends the frame, and says whether or not it’s the last in a sequence.

Figure 2. The FC Frame Header

The FC frame format includes FC-specific information, including the source and destination, among others. Hopefully it clear now why FC is so flexible, which also explains why there’s so darn many FC-blah protocols available to give you a headache.

The actual FC Header, depicted in Figure 2 includes the following fields:

• Routing Control (1 byte): The routing portion says if this is a data frame or a link-control frame (either an ACK or a Link_Response), and the information portion indicates the type of data.
• Destination ID (3 bytes): The FC address of the destination.
• Class Specific Control/Priority (1 byte): Essentially, Quality of Service.
• Source ID (3 bytes): The FC address of the originating node.
• Type (1 byte): Indicates the next protocol (what’s in the Payload), unless R_CTL indicates a control frame.
• Frame Control (3 bytes): Various crazy FC options, such as sequencing information and what to do in case of a problem.
• Sequence ID (1 byte): A sequence number, just like IP.
• Data Field Control (1 byte): Indicates the presence of optional headers, and the size.
• Sequence Count (2 bytes): The number of frames that have been transmitted in a sequence.
• Originator Exchange ID (2 bytes): Assigned by the initiator, used to group related sequences.
• Responder Exchange ID (2 bytes): Same as the OX_ID, but assigned by a target node.
• Parameter (4 bytes): Mostly used as a “relative offset” in sequences, much like IP’s offset.

Yes, it is confusing, and there’s a lot of new terminology, compared to the IP world. We’ll continue to refer back to these headers in future Storage Networking 101 articles, so hopefully the fields and their purposes will become second nature after some real-world examples.

The next important concept to grasp is the way FC assigns names. Notice that the D_ID and S_ID fields in the FC Frame Header only allow for 24 bits. Each HBA is assigned a WWN, and each port on it is assigned a Port WWN, or PWWN. These WWNs are 64-bits in length, which are larger than the 24 bits in FC. The ANSI T11 Address Identifier Format says that the FCID is made up of three parts, which are the Domain_ID, the Area_ID, and the Port_ID.

FC networks are broken up into hierarchies, dynamically. The Domain_ID is assigned to each switch when a fabric comes online using a Domain_ID distribution process. Normally the Domain_ID is administratively configured. The Domain_ID, along with the Area_ID, a second hierarchical level, are combined with a Port_ID (assigned by the switch) to identify each FC node in a fabric. So the WWN doesn’t really mean anything as far as SAN routing goes.

Domain_IDs are distributed by a Principal Switch, which ensures that everyone has the correct information. In short, an FCID will be completely random the first time a node connects, which is generally fine, unless an administrator manually configures it. Some Domain_IDs are reserved for multicast and other purposes, but the details are a bit outside our scope here. Refer to the ANSI T11 FC-SW-3 specification for more details.

Next we’ll explore a bit into the hierarchical world of FC fabrics, including VSANs, a fairly new concept in the SAN world.

• A 2148 byte FC frame is composed of an encapsulating header (Figure 1) and the FC frame (Figure 2), which lends FC easily to being transported over other protocols.
• FC addresses are FCIDs, which get assigned by a switch, based on its internal representation of its ports. Each node is identified by an 8-bit Port_ID.
• Domains are the top level of hierarchical structure in a SAN fabric, and Areas are the second. Areas are for groups of ports on a switch, and they cannot span multiple switches.