Storage Distance by Protocol, Part II – Physical Layer

In the last article, we looked at the big picture of what is involved in creating a SAN distance extension. In this article, we’re going to take a slightly closer look at the physical requirements and with luck we’ll be able to clear up some general confusion and misconceptions along the way.

There is a lot of information about these different elements available via a quick search on your favorite search engine. What I find, though, is that there is usually very little context that accompanies the descriptions or, at best, the authors assume that you may have more of an understanding about some of these technologies than you do. In this case, if I’m going to err it will likely be on the side of making it too accessible and in Plain English, which is something I can live with.

As usual, this is a mid-level view. There are many deep dives that will go into each subject in fine-toothed detail available on the web, but we’re going to stay focused on what you need to know for extending SANs across distances.

Again, this is a rather long post, but hopefully it will be useful as a reference point for you.

Optics

Let’s start off by talking about optics. It seems that for the most part, the exposure to optics is generally limited to looking up a chart to determine what optics are supported, what kind of distance is supported, and then finding the appropriate SKU to add to the shopping cart.

When we start looking at distances, though, we start to realize that there is much more than meets the eye (get it? Optics, eye… never mind).

The transceivers that are used in SANs are a big deal. That’s one of the reasons why they can cost so much. They are, effectively, the power plants that send a signal over the wire. Obviously, the more powerful the optical transceiver, the farther you can go, right? Simple! And away we go!

Not so fast there, pard’ner.

SFP
For speeds up to 4 Gbps, optics for Fibre Channel use Small Form-factor Pluggables (SFPs). These come in three flavors: Shortwave (SW), Medium-wave (MW) and Longwave (LW). While it’s not a direct, 1:1 comparison, for our purposes here it’s useful to think of “shortwave = short distance,” “medium-wave = medium distance,” and “longwave = long distance,” at least as far as determining what kind of optics will get you to a certain distance.

(Of course, what is “short,” “medium” and “long range” distances is all relative.)

SFP+
At speeds at 8Gbps and above, you’ll need to use SFP+ optics. In addition to the SW and LW above, you have Short Range (SR), Long Range (LR) and Extended Range (ER) optics. Again, we’re not being comprehensive in order to keep things brief and succinct. It’s more important that you get the general idea here, rather than feel like you have to memorize every single optics type.

[NB: At the time of this writing, these characterizations have been confirmed to be correct. However, you should always check out the MDS optics data sheet to find additional information and the most up-to-date specs.]

The important thing to remember here is that the distance is mandated by the power of the optics, not the speed of the card. In other words, if you have a shortwave SFP+ optic it, too, will be limited to around 860 m and a Long Wave optic will still be limited to the physical 10km distance, regardless of how fast your HBA is running.

By the way, you may be tempted to say to yourself, “Hey! I’ve got 10G Fibre Channel ER optics and I’ve got 10G Ethernet ER optics. I can just mix and match, right? Both Long Wave and Long Range optics have the same distance, so it doesn’t matter which one I use, right?”

You might think that, but you’d be wrong. Fibre Channel and Ethernet have different clocking speeds and the optical transceivers are designed specifically to carry one type of traffic or the other. (In fact, the colors I’ve included in these images above have nothing to do with the optics or cables themselves; I just chose them because I typically tend to try to use green as a Fibre Channel convention and blue as an Ethernet convention.It’s my own personal quirk.)

xWDM
From a distance extension perspective, this is where things get interesting.

“WDM” stands for “Wavelength Division Multiplexing, and there are two types: Coarse and Dense. That means that “xWDM” is really a ‘catch-all’ phrase for CDWM and DWDM types of optical transceivers.

(Actually, there is another type, “Enhanced Wavelength Division Multiplexing,” or EWDM, but I don’t discuss that here.)

CWDM is a simpler technology that uses a lower power consumption (typically about 20% less) with smaller space requirements (typically about 30% less). Generally these optics run at a typical capacity of up to 4 Gbps, so they’re not capable of running high-speed links such as FCoE, nor do they have the power to run very far (when compared to, say, DWDM) but they are cheaper and do provide cost savings on start-up and expansion.

DWDM optics are generally used in the long-haul requirements of undersea and transcontinental routes, and can operate at speeds up to 10Gbps. While it increases the maximum distance, it does require more space, more power, and more accurate lasers.

Between the two of them, DWDM is the most common optic for use in high-capacity, long-distance applications. If yo0u want more information, it’s useful to look at the dedicated page on optics for the MDS product line of Fibre Channel solutions.

Cabling

Generally, there are three different types of cables that we’re going to be concerned about here (there are more, but let’s try to keep things simple for the time being): TwinAx (copper), Single Mode Fibre (SMF) and MultiMode Fibre (MMF).

Before we get ramped up, however, let’s settle something about “dark fibre.”

Dark Fibre
The thing about Dark Fibre that’s interesting to me is that I often hear people talk about Dark Fibre as the “magic link” that can do anything

“Oh, I’ll just run it over dark fibre.”

“We’ll just connect it over dark fibre and we’ll be all set.”

Thing is, dark fibre isn’t a magic bullet, and it isn’t the cure-all for distance extensions. In other words: “You keep using that word. I don’t think it means what you think it means!” I have this mental image of people thinking that Dark Fibre is like Darth Vader using the Force to transmit data from one Death Star to the other…

In reality, dark fibre is merely the miles and miles of cabling that was laid out by telecommunication companies that isn’t being used (dark fibre, as opposed to ‘lit’ fibre). Telecoms lease or sell the lines to companies who wish to communicate across great distances.

Nevertheless, there is nothing magic about these cables, and they are subject to the same physical laws that affect all cables. In other words, yes, you still need to know about the physical layer capabilities of handling FC-based traffic even over dark fibre.

Copper Cabling
Since we’re talking about SAN extension and distance, we’re not going to be spending much time on copper cabling. Copper has a higher resistance than fibre optics, and as a result the amount of distance you can use copper for is more limited than spun glass, when you look at long, unsegmented units.

There are many different forms of copper cabling, but for handling high-speed traffic (say, over typical 1Gbps speeds) you’re looking at either Category 6a, Category 7, or TwinAx 10Gb cabling (again, this is not a comprehensive list, just a starting point).

Copper cabling is cheap, which is why it is used so often. At 10Gb speeds, especially for Fibre Channel over Ethernet (FCoE), copper cabling is found in the TwinAx or 10GBASE-T form. The good news is that not only are they cheap to purchase, but they’re also cheep to operate: A TwinAx copper cable can run up to 10m (which is about 9 rack-lengths) and Cat 6a can run FCoE up to 30m (100m for non-FCoE traffic). We’ll cover why there are different distances for FCoE in the next article.

Copper is a great medium for running Fibre Channel or FCoE inside a Data Center, but given that we’re focusing on SAN extension, we’ll leave the conversation simply at the point where we’ll state that copper cabling is appropriate for intra-Data Center SAN connectivity. That is, even though it’s inexpensive, it’s not the best candidate for long distances.

Optical Fibre Cabling: Multimode Fibre (MMF) and Single Mode Fibre (SMF)
A five-minute search on your favorite web engine will give you more information about fibre optic cabling than you’ll know what to do with. In truth, it’s difficult to discuss optical cabling without getting too deep into the physics, but I’m going to try to hover around the critical points to remember.

Multimode Fibre (MMF) has a larger diameter core which can allow multiple paths (modes) of light to be sent at the same time. Single Mode Fibre (SMF), as you may have guessed, has a smaller diameter core which allows only one mode of light to be sent. Both types suffer from signal degradation over distance; the MMF more so because of the varying angles of light being shot through the cabling affects the amount that light is distorted (or dispersed) over time.

The key thing to remember about MMF is that it has a shorter effective distance than single-mode, usually around 10 miles is the limit. This means that for most general-purpose applications such as campus or building deployments, MMF is a good, popular choice. MMF also has an advantage of using somewhat cheaper equipment and being easier to interconnect. It also means that it is used for shorter, slower network connections. While cables can be made in any color, most of the orange cabling you may have seen in your data center is MMF.

You’ve heard the phrase, “focus like a laser beam?” Well, SMF does exactly that. It shrinks down the size of the core so that only one ray of light can be shown through the cable, which greatly increases the distance. In theory, this is limitless, but in practical applications cable lengths are 100 km or less (specialized cables like submarine cables are a notable exception, but fall outside the scope of what we’re talking about here). SMF cabling often has a yellow color.

Note that you still need to be able to have the appropriate transceiver to have enough power to send the signal those distances!

When you look at Cisco’s guide to MDS transceivers you will see this relationship between optics and cabling specifications. In Fibre Channel, Shortwave optics are used with MMF, and Longwave optics are used for SMF. You will also notice that any distance over 1000m requires SMF.

SONET and SDH

If you’re looking for truly long distances – the kind that are global in nature – you can move beyond the xWDM components into the realm of SONET and SDH. Going into excruciating detail isn’t the point here, so if you don’t think you’re going to be doing cross-national or intercontinental distances, feel free to skip this section. Conversely, if you are interested there is also a brief overview of SONET technology available.

Generally speaking, SONET (which is used in North America) or SDH (used in the rest of the world) is a managed optical technology that supports much longer distances than CWDM or DWDM.

Why use SONET/SDH? Well, there is a significant installed infrastructure that provides a few key benefits:

• Services are available in many areas
• Allow longer distances than xWDM at a lower cost
• Support longer distances with lower latencies than Fibre Channel over IP (FCIP) (more about FCIP in a later article)
• Offer robust network management and troubleshooting capabilities
• Offer numerous and flexible topologies (e.g., point-to-point, switched, ring, mesh, and more)

If SONET/SDH has so many benefits, why isn’t it deployed even more than it already is? One reason is scalability. Unless you have a Optical Carrier 192 (OC-192) cable – and these aren’t widely available – you’ll not have enough bandwidth to run more than one 2-Gb/s Fibre Channel connection. OC-3, which is the most widely available SONET and SDH service level, provide less than 1-Gb/s of bandwidth.

Fortunately, SONET/SDH can be used in conjunction with other distance physical technologies, such as dark fibre, to allow multiple SONET or SDH channels to be carried over a single DWDM link, for instance.

While a list and description of SONET/SDH topologies and best practices (and dark fibre implications) falls far, far outside the scope of this article, Cisco does have several solutions and technologies that can help address some of the needs of advanced SAN extension environments, such as the ONS 15454 Multiservice Transport Platform (MSTP).

Putting It Together

Optics + Cabling = Physical Distance

Obviously, we’ve only scratched the surface here (and not very deeply at that, it’s barely a flesh wound!). If you want to find out more information about the different optical fibre types, take a glance at this TechwiseTV Networking 101 Video.

Given what we’ve talked about so far, the following chart might be helpful to see how the optics and cabling match up for various distances, useful when you start looking at planning all the way up to geoclusters.

Since we are only looking at the physical components of the links, we’re obviously missing out on a few important bits: speed of the link and how we’re going to fill it up with Fibre Channel data

The next article is going to look into that aspect in greater detail.

[NB: Special thanks to Cisco’s Mark Allen for the use of some of his presentation visuals in this post.]