INTRODUCING BROCADE MULTIPROTOCOL ROUTING SERVICES

New software capabilities support cost-effective SAN island connectivity and SAN extension.

SAN SOLUTIONS

To help organizations maximize the value of their Storage Area Networks (SANs) Brocade continues to develop innovative solutions that extend SAN benets throughout the enterprise. As part of this effort, Brocade has developed a unique set of multiprotocol routing services that increase the functionality, scalability, and versatility of todays SANs. These new services include: Brocade FC-FC Routing Service for SAN connectivity Brocade FCIP Tunneling Service for SAN extension over distance Brocade iSCSI Gateway Service for sharing Fibre Channel resources with iSCSI devices Delivered on the Brocade SilkWorm Multiprotocol Router, these services provide new options for connecting SAN islands and extending SAN benefits over multiple networks, to larger SAN sizes, and across longer distances. A key aspect of this approach is the unprecedented capability to configure SAN protocols on a portby-port basis within the Multiprotocol Router.

Multiprotocol Routing Services Overview

The Brocade multiprotocol routing services include three types of services: the FC-FC Routing Service for SAN island connectivity, the FCIP Tunneling Service for distance extension, and the iSCSI Gateway Service for sharing Fibre Channel resources with iSCSI devices. This document describes FC-FC routing in detail and provides an overview of the FCIP and iSCSI capabilities. To better understand these services, it helps to know the general terminology (a glossary of terms also appears at the end of the paper). The primary purpose of the services is to provide device connectivity between two or more fabrics without merging those fabrics. The Multiprotocol Router enables the creation of Logical Storage Area Networks (LSANs) that connect multiple fabrics without merging them, thereby providing strategic advantages for change management, network management, scalability, reliability, availability, and serviceability. An LSAN is similar to a Virtual Private Network (VPN): it connects different networks but allows only specic devices on those networks to communicate rather than enabling unrestricted communication. However, it is most useful to think of an LSAN in terms of zoning: an LSAN is a zone that spans multiple SAN fabrics.

The basic requirement for this type of LSAN solution is to create large, flat Fibre Channel SANs that can continue to grow in a scalable, cost-effective manner. For example, organizations with many Fibre Channel SAN islands might not want to merge them, because they would have to contend with domain ID conicts, zoning inconsistencies, fabric-wide parameter mismatches, and other challenges. It might simply be too much administrative work and risk to production services to justify the benet of enhanced connectivity. By using FC-to-FC routing, however, organizations could interconnect devices without having to solve any of those problems, or face any of those risks. Each SAN island would remain its own Fibre Channel fabric, known as an edge fabric in this context. It is important to note that edge fabrics retain their own separate fabric services: nameservers, zoning databases, routing tables, domain ID spaces, and so on.This means, for example, that one fabric could have a domain ID 1 switch, and another fabric could also have a domain ID 1 switch.Without the Multiprotocol Router, these fabrics could not merge until such conflicts were resolved, which could be a time-consuming and risky process in a production environment. With the Multiprotocol Router, these conicts are irrelevant. Moreover, FC-to-FC routing provides additional strategic advantages (none of which would be true with merged fabrics): The scalability of one edge fabric does not affect another. Fabric recongurations do not propagate across edge fabrics. Faults in fabric services are contained. The resulting routed network would consist of multiple individual fabrics that nevertheless form one storage network connectivity model. This kind of network is a level above the traditional definition of a SAN, in which a SAN equals a fabric and a dual-redundant SAN equals a pair of fabrics. This higher-level network is therefore called a Meta SAN. Figure 1 illustrates a Meta SAN comprised of two Fibre Channel routers and n edge fabrics.
Meta SAN
Fabric 1 Fabric n

Figure 1.

A Meta SAN that includes multiple edge fabrics

Edge Fabrics

Standard E_Port connections from installbase from Brocade switches

EX_Port connections from FC Routers

Multiprotocol Routers

SAN SOLUTIONS
2

In the Brocade LSAN model, Multiprotocol Routers connect to the edge fabrics and export devices between them by using EX_Ports.These ports look just like normal Brocade E_Ports to the edge fabrics but limit what each edge fabric sees by using Fibre Channel Network Address Translation (FC-NAT).This is similar to the hide-behind NAT used by most IP rewalls. FC-to-FC routing can use FC-NAT to hide an entire n-domain remote edge fabric behind one translation phantom domain. An EX_Port can be thought of as being an E_Port lite since it appears like a standard E_Port to the edge fabric. An E_Port-to-EX_Port connection is called an Inter-Fabric Link (IFL). Unlike an E_Port, an EX_Port terminates at the router and does not propagate fabric services or routing topology information to other edge fabrics. Moreover, EX_Ports do not switch between themselves except when crossing between different edge fabrics. Figure 2 shows a pair of devices exported between two fabrics.
Figure 2.

A pair of devices exported between two edge fabrics

Fabric 1

Exporting Devices
A host exported from Fabric 2 now also appears in the name server on Fabric 1 A storage array port exported from Fabric 1 appears in Fabric 2
Fabric 2

Each EX_Port has a user-assigned Fabric Identier (FID) that species the fabric to which it is attached. Any number of EX_Ports can have the same FID if they attach to the same edge fabric. In Figure 2, all EX_Ports on all routers connected to Fabric 1 would have FID=1. In a Meta SAN, a fabric reconguration in one edge fabric is not propagated to the others. Nor is zoning database and fabric topology data propagated, so scalability is not limited by the sum of all the fabrics zoning, routing, or convergence timing limits. Even nameserver entries are exported between fabrics only for devices that have been explicitly added to a relevant LSAN for sharing. For example, if Fabric 1 has a scalability limit of 1024 nameserver entries and currently has 768 devices, and so does Fabric 2, an administrator could not merge the fabrics. However, the fabrics could be connected by Multiprotocol Routers and some devices could be shared between fabrics without reaching the Simple Name Server (SNS) scalability limit. When a set of devices on different edge fabrics is allowed to communicate through the Multiprotocol Router, the resulting connectivity group is known as an LSAN, as shown in Figure 3. Many different LSANs can exist in the Meta SAN, and many different LSANs can exist between a given pair of edge fabrics. Likewise, devices can be members of multiple LSANs, and LSANs can overlap with traditional zoning mechanisms on local fabrics.
3

LSAN
Fabric 1 Fabric 2

Figure 3.

LSAN

An LSAN allows connectivity that spans two or more fabrics

Multiprotocol Router Installation

Installing the Multiprotocol Router is a relatively simple process that mirrors the installation steps of any other Brocade SilkWorm switch, such as conguring an IP address for management.The appropriate ports must be congured as EX_Ports and set to the appropriate FIDs using the same tools for traditional switch administrative tasks:WEB TOOLS, Fabric Manager, the Fabric OS command line interface, and so on. Conguring each port in this manner is simple enough that relatively junior SAN administrators can easily install and congure an a Multiprotocol Router. After the installation, day-to-day administration is performed through zoning on each edge fabric.This practice enables existing tools from Brocade and many third-party vendors to work as usual, thereby eliminating the need to extensively retrain SAN administrators. If a specially named zone (known as an LSAN zone) is created on each of two fabrics that the Multiprotocol Router can accessand the devices in that zone are onlinethe Multiprotocol Router would automatically create FC-NAT entries between those fabrics.This is all that must be done to create an LSAN, as shown in Figure 4.
LSAN Zones
Fabric 1 Fabric 2

Figure 4.

Creation of LSAN zones

An LSAN is created by building LSAN zones, which are indistinguishable from regular zones to edge fabrics

SAN SOLUTIONS
4

LSANs spanning multiple fabrics to share devices

The Multiprotocol Router obtains the zoning database from each edge fabric and parses the database for zones with LSAN_ in the name. (The prex is case-insensitive, so LSAN_ and lsan_ would be equivalent.) The Multiprotocol Router then compares the WWNs from LSAN zones in each fabric with entries from the other fabrics. Matching entries dene which devices can communicate through the Multiprotocol Routers across fabrics.As a result, these devices are considered to be in the same LSAN. The user-dened portion of the LSAN zone name from one fabric does not have to match the user-dened portion of the LSAN zone name from another fabric for devices to reside in the same LSAN. It is important to note that these are real zones in the edge fabrics, and the devices that exist in these zones are subject to normal zoning enforcement by the switches in each edge fabric. If the administrator of Fabric 1 does not zone a host with a storage array from Fabric 2, it doesnt matter if the Fabric 2 administrator did so.The devices will be able to communicate only when the zoning policy on both fabrics allows it.
LSAN Administration Models

There are two primary administration models for LSANs: the model in which one administrator owns the routers and all edge fabrics involved, and the model in which different administrators own each component. An administrator who wants to allow connectivity between two fabrics and who has administrative access to both can accomplish this task by using traditional zoning tools or by using a single operation through Fabric Manager. Because Fabric Manager can access both fabrics, it can simultaneously create both LSAN zones.When instructed to create an LSAN, Fabric Manager determines which fabrics the devices reside in and creates the appropriate LSAN zones in each fabric.This is known as the Super Admin model. In contrast, if different administrators have access to each fabric, they can create LSAN zones containing the devices they want to export.This is the Multi Admin model. If a large number of SAN islands will be combined into a very large Meta SAN, administrators could network Multiprotocol Routers over a centralized backbone fabric, as shown in Figure 5.
Figure 5.

Networking Multiprotocol Routers over a centralized backbone fabric

Backbone Fabric Meta SAN

Fabrics 1 through 8

Fabrics 9 through 16

NR_Ports are virtual N_Port devices attached to the router Routers connect to the backbone with standard E_Ports Router domains talk across the Backbone fabric with FCRP The Backbone Fabric uses standard E_Ports on standard switches additional backbone fabrics

additional edge fabrics

A host in Fabric 1 could be exported into Fabric 16 across the backbone fabric, even though those fabrics are not attached to the same Multiprotocol Router.To accomplish this, Brocade uses a standards-track Fibre Channel Router Protocol (FCRP) that operates on backbone-attached E_Ports.The Multiprotocol Router does not use a special E_Port type (such as an EX_Port) for a router-to-backbone connection. Instead, it uses a standard Brocade E_Port. Each Multiprotocol Router on the backbone fabric can see that other routers have entered that fabric, and can send FCRP messages from its domain address to the other routers domain addresses. FCRP also operates between domains projected by EX_Ports into an edge fabric. In addition to these routing tasks, FCRP handles coordination between domains, exchanging LSAN zone information as well as device and fabric state information. Each Multiprotocol Router also projects special virtual N_Ports (known as NR_Ports) onto the backbone fabric. NR_Ports serve as sources and destinations for data frames sent across the backbone.They are similar to router ports in IP networks and can be thought of as the back side of an EX_Port.They are discovered via FCRP and do not exist in the nameserver of the backbone fabric. Data frames sent between NR_Ports use an encapsulated global header that contains items such as the source and destination fabric IDs, so that a receiving router knows the true origin and destination of a frame.This process is transparent to switches in the backbone fabric.
Deployment Examples

To help illustrate the Brocade multiprotocol routing services, this section contains deployment examples of possible solutions. However, this section does not represent a comprehensive list of all possible applications of the Multiprotocol Router.
Basic Resilient Meta SAN

The dening characteristic of a resilient SAN is that there is no single point of failure within the core-to-edge connectivity model. Each Inter-Switch Link (ISL) has one or more alternate paths, and each core switch is likewise duplicated. It is possible to lose an edge switch, but critical nodes are always connected to at least two edge switches. In a resilient Meta SAN, each edge fabric that exports devices must have at least two Multiprotocol Routers providing paths to every other relevant edge fabric. Nodes would generally be connected to A/B fabrics within this model, as shown in Figure 6.
Resilient Meta SAN
Figure 6.

A resilient Meta SAN with redundant paths between devices

Fabric 1 (A)

Fabric 2 (B)

Fabric 3 (A)

Fabric 4 (B)

SAN SOLUTIONS
6

Basic Dual-Redundant Meta SAN

A fully redundant SAN duplicates the entire connectivity model: a dual-redundant SAN fabric has two completely separate fabrics. Similarly, a dual-redundant Meta SAN has two completely separate (A/B) Meta SANs (see Figure 7). Hosts and storage arrays are dual-attached, with at least one connection to each Meta SAN.This provides the greatest possible fault isolation, such as preventing a misbehaving Host Bus Adapter (HBA) on one Meta SAN from interfering with nodes on the other.
Figure 7.

A dual-redundant Meta SAN with enhanced fault isolation

Dual-Redundant Meta SAN

Meta SAN A

Meta SAN B

Fabric 1A

Fabric 2A

Fabric 1B

Fabric 2B

Tape Consolidation Meta SAN

One key FC-to-FC routing application is centralizing backup and restore resources across SAN islands. In this example, 30 isolated fabrics can share a central tape pool consisting of two large backup and restore fabrics, as shown in Figure 8.
Tape Consolidation Meta SAN

Figure 8.

A centralized tape pool shared across SAN islands

Fabric 1 Fabric 30

Distributed Host and Storage Edge Fabrics

Centralized Backup and Restore Edge Fabrics

Fabric 31

Fabric 32

SAN Island Consolidation Meta SAN

Island Consolidation Meta SAN

Fabric ID = 1 PID format = 1 Domain IDs = 1, 2, 3, 4 Fabric ID = 4 PID format = 0 Domain IDs = 1, 2 Fabric ID = 15 PID format = 0 Domain IDs = 1, 6, 8, 9 Fabric ID = 19 PID format = 1 Domain IDs = 1, 2

Figure 9.

SAN island consolidation in a Meta SAN

Each fabric can have its own set of domain IDs, its own zoning, its own PID format, its own TOVs, etc.

This example shows many small fabrics which could be merged from a scalability standpoint, but not without making many changes to resolve conflicts.

Distance Extension Meta SAN (FC-to-FC Routing and FCIP)

In another example, an organization might have separate A/B fabric pairs for disastertolerant production environments in distinct geographical locations, and a separate fabric for pre-production.This implementation does not provide optimal sharing of resources since there is no connectivity between fabrics unless hosts and storage arrays have ve network connections each.This implementation also requires the use and management of expensive external FCIP gateways (see Figure 10).
Disaster-Tolerant Without FC-to-FC routing
Production A Disaster-Tolerant A

Figure 10.

SAN distance extension over FCIP without FC-to-FC routing

Production B IP WAN

Pre-Production Disaster-Tolerant B Hosts and storage involved in Disaster-Tolerant and pre-production operaions in any way need five HBAs Dedicated Pre-Production Nodes

Disaster-Tolerant Nodes

Unreliable WAN links can create instability for all Disaster-Tolerant fabrics

SAN SOLUTIONS
8

In many environments, even if SAN islands could be merged without crossing fabric scalability limits, the process of doing so would be too difficult and risky to be considered. In contrast, FC-to-FC routing enables the connection of SAN islands without needing to resolve domain conicts, rework zoning congurations, or resolve fabric-wide parameter conflicts such as Timeout Value (TOV) and Port ID (PID) formats (see Figure 9).

In contrast, the implementation of a dual-redundant Meta SAN enables fault isolation between each fabric, retaining the fully redundant model but allowing connectivity as needed. Hosts and storage can communicate across all fabrics but need only two HBAs for fully redundant operation.There are still ve fabrics, but the LSAN switches allow cross-fabric communication (see Figure 11).
Figure 11.

SAN distance extension with FC-to-FC routing

Disaster-Tolerant With FC-to-FC routing

Production A

Backbone A Disaster-Tolerant A Production B

IP WAN

Pre-Production Disaster-Tolerant B Backbone B Disaster-Tolerant Nodes

Integrated FCIP capabilities simplify management and eliminate cost and potential failures caused by external WAN tunneling equipment. Because the Multiprotocol Routers are running FC-to-FC routing, the WAN forms a completely redundant and isolated pair of backbone fabrics.This capability increases fault isolation in the WAN: IP instability is no longer translated into edge fabric recongurations. A similar design would also work with external gateways for SONET, ATM, or other third-party solutions.
Service Provider Meta SAN

In the example of a service provider model, the provider might have a central set of resources that are projected as needed into fabrics owned and managed by its customers. No customer could access any other customers fabric without permission. Nor could any customer access resources on the service providers fabric without permission. As a result, the appropriate group could autonomously manage each fabric without faults in one section impacting any other section (see Figure 12).
Figure 12.

A Meta SAN in a service provider environment

Service Provider Meta SAN

Fabric 31

The IP WAN is really one or more tunnelled backbone fabrics. It could also be a native Fibre Channel network, or use other gateways such as SONET.

IP WAN

Customer A

Customer B

Customer C

Customer D

iSCSI Gateway Service Overview

The Brocade iSCSI Gateway Service enables organizations to integrate low-cost Ethernet-connected servers into their Brocade Fibre Channel SANs via the iSCSI protocol.This practice facilitates storage consolidation and improves management of applications such as centralized backup in cases where hosts do not need the performance or reliability of Fibre Channel but could benet from the management simplicity and white space optimization of a SAN infrastructure.This type of integration reduces the cost of connecting a low-tier server to centrally managed storage within a SAN. Organizations that use this service would not typically attach iSCSI hosts directly to gateway ports. Direct attachment of an iSCSI TCP Ofoad Engine (TOE) adapter to an iSCSI gateway is less cost-effective than using a Fibre Channel HBA. Instead, there would be an external fan-out using existing Ethernet infrastructure. When an iSCSI host accesses the service, it is projected onto the backbone fabric of the router and can then communicate with any storage device in that fabric.
FCIP Tunneling Service Overview

The Brocade FCIP Tunneling Service enables organizations to extend their Fibre Channel SANs over longer distances that would be impractical with native Fibre Channel links, or situations where dark fiber links would be impractical but in which IP WANs already exist. This service offers two important advantages.The rst advantage is full integration with the switch. It is easier and more cost-effective to deploy and manage an FCIP link integrated into the switch as opposed to one that requires an external gateway. In addition to easier management, tighter integration means lower cost and a smaller rack footprint. The second advantage is integration with FC-to-FC routing. It is possible to have a Multiprotocol Router in which a port is both an E_Port into the backbone fabric and an FCIP-to-FCIP port.This prevents faults on the WAN link from turning into Meta SAN-wide events.This is a key advantage, because IP networks in general and WANs in particular are less reliable than Fibre Channel networks. A appingWAN link might disturb the backbone fabric, but these disturbances are isolated from all edge fabrics so no host/storage conversations would be affected other than those that actually crossed the unreliable WAN. This service will initially be most useful for campus networks and WANs that have full-bandwidth links. In addition, Brocade plans to continue partnering with CNT for enterprise-class FCIP-to-FCIP distance extension solutions, and with other leading vendors such as Alcatel and Nortel for other WAN protocols.
For More Information

For more information about the Brocade SilkWorm Multiprotocol Router and Brocade Multiprotocol SAN Routing Services, visit www.brocade.com.

10

SAN SOLUTIONS

Glossary of Terms

Backbone Fabric: A capability that enables scalable Meta SANs by allowing the networking of multiple Multiprotocol Routers that connect to the backbone fabric via E_Port interfaces. Devices attached to Multiprotocol Routers via F_Port or FL_Port, or imported via the iSCSI Gateway Service, are also considered part of the backbone. E_Port: A standard Fibre Channel mechanism that enables switches to network with each other. Edge Fabric: A Fibre Channel fabric connected to a Multiprotocol Router via one or more EX_Ports.This is where hosts and storage are typically attached in a Meta SAN. EX_Port: The type of E_Port used to connect a Multiprotocol Router to an edge fabric. An EX_Port follows standard E_Port protocols and supports FC-NAT but does not allow fabric merging across EX_Ports. Exported Device: A device that has been mapped between fabrics. A host or storage port in one edge fabric can be exported to any other fabric through LSAN zoning. Fabric: A collection of Fibre Channel switches and devices, such as hosts and storage. Fabric ID (FID): Unique identier of a fabric in a Meta SAN. FCIP Tunneling Service: A service that enables SANs to span longer distances than could be supported with native Fibre Channel links. FCIP is a TCP/IP-based tunneling protocol that allows the transparent interconnection of geographically distributed SAN islands through an IP-based network. Fibre Channel: The primary protocol for building SANs. Unlike IP and Ethernet, Fibre Channel is designed to support the needs of storage devices of all types. Fibre Channel Network Address Translation (FC-NAT): A capability that allows devices in different fabrics to communicate when those fabrics have addressing conicts. This is similar to the hide-behind NAT used in rewalls. Fibre Channel Router Protocol (FCRP): A Brocade-authored standards-track protocol that enables LSAN switches to perform routing between different edge fabrics, optionally across a backbone fabric. FC-FC Routing Service: A service that extends hierarchical networking capabilities to Fibre Channel fabrics. It enables devices located on separate fabrics to communicate without merging the fabrics. It also enables the creation of LSANs. Inter-Fabric Link (IFL): A connection between a router and an edge fabric. Architecturally, these can be of type EX_Port-to-E_Port or EX_Port-to-EX_Port. The former method is supported in the first release. iSCSI Gateway Service: A service that maps the SCSI protocol to the IP transport. This service projects iSCSI hosts onto the backbone fabric of a gateway switch. Logical Storage Area Network (LSAN): A logical network that spans multiple fabrics. The path between devices in an LSAN can be local to an edge fabric or cross one or more Multiprotocol Routers and up to one intermediate backbone fabric. LSANs are administered through LSAN zones in each edge fabric.
11

Meta SAN: The collection of all devices, switches, edge and backbone fabrics, LSANs, and Multiprotocol Routers that make up a physically connected but logically partitioned storage network. In a data network, this would simply be called the network. However, an additional term is required to specify the difference between a single-fabric network (SAN), a multifabric network without cross-fabric connectivity (for example, a dual-redundant fabric SAN), and a multifabric network with connectivity (Meta SAN). Multiprotocol Router: A device that enables the Brocade multiprotocol routing services. Multiprotocol routing services: An optionally licensed software bundle available on certain Brocade platforms, such as the Multiprotocol Router, that includes the FC-FC Routing Service, the iSCSI Gateway Service, and the FCIP Tunneling Service. NR_Port: A port used as a source and destination address for frames traversing a backbone fabric. A normal E_Port (not an EX_Port) is used to connect a Multiprotocol Router to a backbone. An NR_Port appears to the rest of the backbone as a standard N_Port connected to the Multiprotocol Router domain.

12

SAN SOLUTIONS

LSAN Zone: The mechanism by which LSANs are administered. A Multiprotocol Router attached to two fabrics will listen for the creation of matching LSAN zones on both fabrics. If this occurs, it will create phantom domains and FC-NAT entries as appropriate, and insert entries for them into the nameservers on the fabrics. LSAN zones are compatible with standard zoning mechanisms.

Corporate Headquarters San Jose, CA USA T: (408) 333-8000 info@brocade.com European Headquarters Geneva, Switzerland T: +41 22 799 56 40 europe-info@brocade.com

Asia Pacic Headquarters Tokyo, Japan T: +81 3 5402 5300 apac-info@brocade.com Latin America Headquarters Miami, FL USA T: (305) 716-4165 latinam-sales@brocade.com

2004 Brocade Communications Systems, Inc. All Rights Reserved. 02/04 GA-WP-642-02 Brocade, the Brocade B weave logo, Secure Fabric OS, and SilkWorm are registered trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries. FICON is a registered trademark of IBM Corporation in the U.S. and other countries. All other brands, products, or service names are or may be trademarks or service marks of, and are used to identify, products or services of their respective owners. Notice:This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use.This informational document describes features that may not be currently available. Contact a Brocade sales ofce for information on feature and product availability. Export of technical data contained in this document may require an export license from the United States government.