Storage Networking 101: Is IP Storage Viable?
Proponents of iSCSI are quick to point out some attractive facts. These consist of, but are not limited to:
- iSCSI is cheaper because it uses your existing IP infrastructure
- iSCSI is simpler to deploy
- iSCSI is "faster" because you can use 10GbE
These things are true, to varying degrees, depending on the environment and performance needs. Unfortunately, iSCSI cannot meet high-demand enterprise needs.
Let us first address the performance concerns. The truth is that iSCSI is certainly capable of some impressive throughput. Most any protocol, standard, or device can perform really well in a few niche applications of the technology, and iSCSI does excel in a few areas.
It is no mystery that iSCSI-only devices are targeted to the SMB market. Nobody is claiming you're supposed to put huge transactional databases on iSCSI storage, but the proliferation of iSCSI hype has led to this cost-saving conclusion for many IT managers. Be extremely careful to avoid this trap.
For many reasons, Fibre Channel (FC) does offer better performance. We often relegate the performance discussion to factors that do not accurately isolate the things being compared. Frequently we see arguments about iSCSI performance turn in to a discussion about how horrible SATA is, simply because many SATA array vendors offer iSCSI support. Yes, SATA will crumble under a load with lots of random IO, but it has nothing to do with the underlying access protocols.
FC is designed for large block IO and, as such, is heavily optimized to pass storage data around. Ethernet is not, but using jumbo frames (9K instead of 1.5K data units) can alleviate this concern a bit. FC HBA cards are, however, more efficient than Ethernet cards. Without getting into too much detail, FC essentially requires less CPU overhead because Ethernet usage requires an interrupt for each frame. Ethernet is really designed for small and frequent packet handling, not large data streams. iSCSI also rides on top of TCP, so traffic needs to be passed through many more layers in the OS before actually being sent out on the wire, further increasing latency. TCP checksum offload engines exist, so we exclude the added TCP overhead for practical purposes. In short, FC provides far less latency, and far better throughput. Again, deciding on FC-based SAN versus iSCSI, depends on the specific performance needs that you are addressing.
Network utilization is another important concern. A primary selling point of iSCSI is that you can utilize your existing network infrastructure; in fact, you can use the same NIC that other IP traffic passes on. This is fine, for casual uses, but a high-traffic (in the TCP/IP sense) servers that also need decent storage access times will clearly suffer with iSCSI deployed over a single network interface. In fact, iSCSI users often find that, because they don't really need good performance, NAS-based solutions (i.e. mounting an NFS or CIFS share) would have worked just as well.
Options exist help alleviate network congestion at the host. Obviously we can deploy a second network card, and luckily most servers these days ship with two to four Gb network interfaces. When we start talking about the need for 10GbE, we're likely going to be running into other performance issues as was outlined above. Assuming the access characteristics are ideally suited for iSCSI, the 10GbE option does exist.
How will a network handle 10GbE, though? Likely we'd need to upgrade some infrastructure to make this happen. Network congestion in IP storage networks often demand the separation of duties—a completely separate IP network for storage. People might be inclined to directly connect an Ethernet cable to their storage device if 10GbE would put too much strain on the network and there was only one server that needed the performance boost. Unfortunately, that devolves your situation back to DAS (direct-attached) days, and truth be told, a DAS setup could provide much better performance anyway.
The FC world is used to high availability configurations. Each node connecting to a SAN may use two HBA ports, and "see" each storage LUN twice, but over multiple paths. When configured correctly, these LUNs are accessed from a virtual LUN, and the driver transparently fails over between the actual LUNs in the event of an outage. This is how SAN storage is done, and everyone loves it.
In an FC SAN, we can upgrade or replace storage device controllers and switches with zero downtime. In the iSCSI world, we cannot. Each host is directly connected to a single switch, regardless of whether or not they share the same NIC with TCP/IP traffic. If that switch disappears, there is no failover capability. Some vendors have likely implemented proprietary multipathing solutions for iSCSI, but it is surely only going to work with only one operating system and their own storage device.
The good news in all of this drudgery is that you don't really have to choose one technology over the other.
I urge everyone to consider iSCSI as an intermediary between high-performance FC-based SAN and file shares (NAS). Most FC storage array vendors now offer iSCSI support directly on the array itself, enabling businesses to use a hybrid deployment methodology. Your SAN-attached disk array can now be IP-attached, and serve two distinct demands, based on performance and reliability needs.
Don't dismiss iSCSI because of the naysayers, and don't adopt it without careful consideration.