Академический Документы
Профессиональный Документы
Культура Документы
Welcome to Introduction to Fibre Channel. The AUDIO portion of this course is supplemental to the material and is not a replacement for the student notes accompanying this course. EMC recommends downloading the Student Resource Guide from the Supporting Materials tab, and reading the notes in their entirety.
Copyright 2005 EMC Corporation. All rights reserved. These materials may not be copied without EMC's written consent. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Celerra, CLARalert, CLARiiON, Connectrix, Dantz, Documentum, EMC, EMC2, HighRoad, Legato, Navisphere, PowerPath, ResourcePak, SnapView/IP, SRDF, Symmetrix, TimeFinder, VisualSAN, where information lives are registered trademarks. Access Logix, AutoAdvice, Automated Resource Manager, AutoSwap, AVALONidm, C-Clip, Celerra Replicator, Centera, CentraStar, CLARevent, CopyCross, CopyPoint, DatabaseXtender, Direct Matrix, Direct Matrix Architecture, EDM, E-Lab, EMC Automated Networked Storage, EMC ControlCenter, EMC Developers Program, EMC OnCourse, EMC Proven, EMC Snap, Enginuity, FarPoint, FLARE, GeoSpan, InfoMover, MirrorView, NetWin, OnAlert, OpenScale, Powerlink, PowerVolume, RepliCare, SafeLine, SAN Architect, SAN Copy, SAN Manager, SDMS, SnapSure, SnapView, StorageScope, SupportMate, SymmAPI, SymmEnabler, Symmetrix DMX, Universal Data Tone, VisualSRM are trademarks of EMC Corporation. All other trademarks used herein are the property of their respective owners.
Course Objectives
Upon completion of this course, you will be able to: Understand the Fibre Channel Architecture Identify the different Fibre Channel layers Learn Fibre Channel Topologies Know SAN concepts
The objectives for this course are shown here. Please take a moment to read them.
The objectives for this lesson are shown here. Please take a moment to read them.
Fibre Channel
Fibre Channel is a serial data transfer interface intended for connecting high-speed storage devices to computers The high-speed is obtained through of the process following:
Networking and I/O protocols (such as SCSI commands) are mapped to Fibre Channel constructs And encapsulated and transported within Fibre Channel frames With this is possible high-speed transfer of multiple protocols over the same physical interface
Fibre Channel is a serial data transfer interface that operates over copper wire and/or optical fiber at data rates up to 200 MB/s (megabytes per second). Networking and I/O protocols (such as SCSI commands) are mapped to Fibre Channel constructs, and then encapsulated and transported within Fibre Channel frames. This process allows high-speed transfer of multiple protocols over the same physical interface. Fibre Channel systems are assembled from familiar types of components: adapters, hubs, switches and storage devices. Host bus adapters are installed in computers and servers in the same manner as a SCSI host bus adapter or a network interface card (NIC). Hubs link individual elements together to form a shared bandwidth loop. Fibre Channel switches provide full bandwidth connections for highly scalable systems without a practical limit to the number of connections supported (16 million addresses are possible). The word fiber indicates the physical media. The word fibre indicates the Fibre Channel protocol and standards.
Channels
The channel connections, such as parallel bus and tag, ESCON, and SCSI provide fixed connections between host systems and their peripheral devices Some characteristics of channel technologies are:
High performance Low protocol overhead Static configuration Short distance (although ESCON is somewhat of an exception) Connectivity within a single system
Traditionally, host computer operating systems have communicated with storage devices over channel connections, such as parallel bus and tag, ESCON, and SCSI. These channel technologies provide fixed connections between host systems and their peripheral devices. The static connections are defined to the operating system in advance. Tight integration between the transmission protocol and the physical interface minimizes the overhead required to establish communication and transport large amounts of data to the statically defined devices. Some characteristics of channel technologies are: High performance Low protocol overhead Static configuration Short distance (although ESCON is somewhat of an exception) Connectivity within a single system
Network
The network technologies offer more flexibility and distance capabilities than channel technologies Some characteristics of network technologies are:
Low performance High protocol overhead Dynamic configuration Long distance Connectivity among different systems
Network technologies are more flexible than channel technologies, and provide greater distance capabilities. Most networks provide connectivity between client or host systems, and carry a variety of data between the devices. A simple example is a network of desktop PCs within a company. This type of setup can provide each PC with connectivity to file and print services, server-based applications, and corporate intranets. The networks these PCs are connected to provide shared bandwidth and the ability to communicate with many different systems. This flexibility results in greater protocol overhead and reduced performance. Some characteristics of network technologies are: Low performance High protocol overhead Dynamic configuration Long distance Connectivity among different systems
Channels (benefits)
Networks (benefits)
Fibre Channel
Fibre Channel captures some of the benefits of both channels and networks. A Fibre Channel Fabric is a switched network, providing a set of generic, low-level services onto which host channel architectures and network architectures can be mapped. Fibre Channel standards define a layered communications architecture similar to other networking environments. Each level of the Fibre Channel protocol stack provides a specific set of functions.
SAN
Network-Attached Storage
NAS
CAS
Fibre Channel
Type of transport Type of data Key requirement Typical applications (FCP, FICON) IP (iSCSI, FCIP, iFCP) Block Deterministic performance OLTP, data warehousing, ERP
Fibre Channel integrates the SAN (Storage Area Networks) and NAS (Network-Attached Storage) solutions.
Networks of host and storage devices (called Storage Area Networks, or SANs) are often connected over Fibre Channel Fabrics. A common method of managing the variety of devices on a SAN is SNMP (Simple Network Management Protocol), popular because it is widely supported and can be run out of band (which is advantageous because it does not rely on the Fibre Channel network). An open industry consortium called FibreAlliance is defining an SNMP MIB (Management Information Base) to facilitate SAN management. The MIB is a group of parameters (variables) whose values define, and describe the status of, a network and its components. The Fibre Channel Management Integration (FCMGMT-INT) MIB provides a heterogeneous method of managing multiple devices across a SAN.
The objectives for this lesson are shown here. Please take a moment to read them.
Levels
The Fibre Channel standards defines a Level protocol stack
Fibre Channel Level Function Defines physical media including connectors, cables, transmitter and receiver technology. Supports a variety of data rates. Defines transmission protocol utilizing the 8B/10B code/decode method, which improves the transmission characteristics and enhances error recovery. Defines the rules by which nodes communicate including data framing, frame sequencing, flow control, and class of service. Defines a set of services to support advanced functions. Defines the Fibre Channel Link Encapsulation (FC-LE) Upper level Protocol - SCSI-3, IP, ESCON/FIPS, etc..
The Fibre Channel standards define a layered protocol: The FC-0 is the lowest level and defines the physical link in the system, including the fiber, connectors, optical and electrical components. This level covers a variety of media and the associated drivers and receivers capable of operating at a wide range of speeds The FC-1 defines the transmission protocol including serial encoding and decoding, special characters and error control. Information is encoded 8 bits at a time into a 10 bit transmission character and transmitted over the cable The FC-2 level serves as the transport mechanism of Fibre Channel. The transported data is transparent to FC-2 and visible to FC-3 and above The FC-3 Defines a set services to support advanced functions The FC-4 level of Fibre Channel is designed to hand off to another protocol such as SCSI. Fundamentally, the commands at FC-4 for SCSI allow SCSI initiators and targets to communicate over Fibre Channel ULP (Upper level Protocol) is not actually part of Fibre Channel
FC-0
The FC-0 level of FCPH describes the Fibre Channel link
Physical Interface Optical and Electrical Interfaces Cables, Connectors, etc
ULP ULP FC-4 FC-4 FC-3 FC-3 FC-2 FC-2 FC-1 FC-1 Fibre Channel Level 0 (FC0) Fibre Channel Level 0 (FC0)
Each fiber is attached to a transmitter of a Port at one end and a receiver of another Port at the other end
2005 EMC Corporation. All rights reserved.
The FC-0 level describes the Fibre Channel link. It covers a variety of media and the associated drivers and receivers capable of operating at a wide range of speeds. It is designed for maximum flexibility and allows the use of a large number of technologies to meet the widest range of system requirements. Each fiber is attached to a transmitter of a Port at one end and a receiver of another Port at the other end. When a Fabric is present in the configuration, a fiber may attach to an N_Port and an F_Port. Patch panels or portions of the active Fabric may function as repeaters, concentrators or fibre converters.
Connectors
The LC assembly design incorporates an RJ-style latch in a connector body with half the footprint of a conventional SC
The SC connector is the standard connector for fiber optic cables. It is a push-pull connector
The LC assembly design incorporates an RJ-style latch in a connector body with half the footprint of a conventional SC. High-precision, 1.25 mm ceramic ferrules and engineered composites provide a durable package with consistent, repeatable performance. The SC connector is the standard connector for fiber optic cables. It is a push-pull connector and is favored over the ST connector (commonly used in patch panels). If the cable is pulled, the tip of the cable in the connector does not move out, which would result in loss of signal quality.
Multi-Mode Cable
Multi-mode transmitters send multiple short-wavelength signals through the same fiber The angle of entry is high and the signals tend to cancel each other out Multimode fiber has a larger diameter (62.5 or 50 microns) core than single-mode fiber
Multimode cable is dominant for short distances of 500 meters or less. Multimode has an inner diameter of 62.5 or 50 microns, allowing light to enter the cable in multiple modes, including straight and at different angles. The many light beams tend to lose shape as they move down the cable. This loss of shape is called modal dispersion and limits the distance.
Single-Mode Cable
Single-mode transmitters send one longer-wavelength signal down a much thinner-cored fiber The angle of entry is low (less bouncing) and there is little to muddy the signal, hence the much greater distances Single mode cable has an inner diameter of 9 microns
Single mode cable is used for long distance cable runs, only limited by the power of the laser at the transmitter and by the sensitivity of the receiver. Single mode cable has an inner diameter of 9 microns and is always used with a long wave laser, which limits the effects of modal dispersion. Therefore, with single mode cables the distance is greatly increased.
An HBA is an I/O adapter that sits between the host computer's bus and the Fibre Channel loop, and manages the transfer of information between the two channels. In order to minimize the impact on host processor performance, the host bus adapter performs many low-level interface functions automatically or with minimal processor involvement. In simple terms, a host bus adapter (HBA) provides I/O processing and physical connectivity between a server and storage. The storage may be attached using a variety of direct attached or storage networking technologies, including Fibre Channel, iSCSI, VI/IP, FICON, or SCSI. Host bus adapters provide critical server CPU off-load, freeing servers to perform application processing. As the only part of a storage area network that resides in a server, HBAs also provide a critical link between the SAN and the operating system and application software. In this role, the HBA enables a range of high-availability and storage management capabilities, including load balancing, fail-over, SAN administration, and storage management.
FC-1
Defines the method used to Encode data prior to transmission and decode the data upon reception A benefit of using 8b/10b encoding is that it defines a number of Special Characters
Fibre Channel Level 1 (FC1) Fibre Channel Level 1 (FC1) Fibre Channel Level 0 (FC0) Fibre Channel Level 0 (FC0)
Whenever the encoder creates a character with more ones than zeros, it remembers this by setting a single-bit variable called the Current Running Disparity (CRD) to positive. Whenever the encoder creates a character with more zeros than ones, it sets the CRD to negative. The CRD is fed back to the encoder to select the appropriate encoding in order to balance the number of ones and zero bits transmitted. As a series of characters are processed, the output alternates between positive and negative disparity.
Encoding Process
The encoding process transforms 8-bit input characters into 10-bit transmission characters having the desired attributes
The encoding process transforms 8-bit input characters into 10-bit transmission characters having the desired attributes. Not all of the possible 10-bit patterns are used. 10 bits allow 1024 different combinations. To prevent excessive DC (direct current) components and run length problems, only characters containing 6 ones and 4 zeros, 5 ones and 5 zeros, or 4 ones and 6 zeros are used. Any other weighting of bits is invalid. 5 ones and 5 zeros are considered to have neutral disparity 6 ones and 4 zeros are considered to have positive disparity 4 ones and 6 zeros are considered to have negative disparity All possible 8-bit characters have 2 possible 10-bit encodings This is the same method used to transmit a data stream in ESCON. The purpose of this is to break up the data stream in the serial environment and at the same time allow control characters to be embedded into the stream to speed up communications.
Ordered Sets
Two types of Transmission Characters (Data and Special) are defined
Data Field
IDLE
Header
Fill Bytes
Two types of Transmission Characters (Data and Special) are defined. Certain combinations of Transmission Characters, referred to as Ordered Sets, are designated to have special meaning. Ordered Sets are used to identify frame boundaries, transmit primitive function requests, and maintain proper link transmission characteristics during periods of inactivity.
FC-2
Defines the structure of the Fibre Channel frame The transported data is transparent to FC-2 and visible to FC-3 and above
Fibre Channel Level 2 (FC2) Fibre Channel Level 2 (FC2) Fibre Channel Level 1 (FC1) Fibre Channel Level 1 (FC1) Fibre Channel Level 0 (FC0) Fibre Channel Level 0 (FC0)
The FC-2 level serves as the transport mechanism of the Fibre Channel. The transported data is transparent to FC-2 and visible to FC-3 and above.
Frames
Frames are the basic building blocks of a Fibre Channel connection All information in Fibre Channel is passed in frames
Start-of-Frame
Data Field Header Optional Header Optional Header Payload (information Fill Bytes being Transported CRC
Frames are the basic building blocks of a Fibre Channel connection. The frames contain the information to be transmitted, the addresses of the source and destination ports, and link control information. All information in Fibre Channel is passed in frames. The maximum amount of data carried in a frame is 2112 bytes, with a total frame size of 2148 bytes. The general structure of a Frame is specific.
End-of-Frame
Header Contents
The header contains the Source and Destination Addresses which allows the frame to be routed to the correct port
Start-of-Frame End-of-Frame
Data Field Header Optional Header Optional Header Payload (information Fill Bytes being Transported CRC
DF_CTL OX_ID
Destination Address (D-1D) Source Address (S-ID) Frame Control (F_CNT) DF_CTL
The Header contains the Source and Destination Addresses, which allows the frame to be routed to the correct port. The Type field interpretation is dependent on whether the frame is a link control frame or a Fibre Channel data frame. For example, if the frame is a data frame, a 08 in the type field would indicate SCSI FCP information in the Data field.
An Exchange is a unidirectional or bi-directional set of non-concurrent Sequences. An Exchange is the largest construct understood by FC-2. FC-2 manages Exchanges that map directly to operations. FC-2 manages Sequences as unidirectional transfers of one or more frames. A Sequence is contained within an Exchange and is comprised of one or more Frames. FC-2 names each Sequence and tracks each Sequence to completion. The purpose of the Sequence is to reorder data when it is received at the other end.
Data Frame
Data Frame
R_RDY
R_RDY
The FC-2 provides flow control for buffer management. When nodes initialize on the Fabric they agree on operational parameters such as the number of buffers available (Buffer Credits). Transmitting nodes can continue to transmit as long as there are buffer credits. An R_RDY is sent to a transmitting node as buffers are cleared. An FC switch also performs buffering and flow control internally and externally.
Classes of Service
Classes of Service
Functions
Fabric discards Frames Communications type between ports Initial roundtrip delay Hunt group support Multicast support Broadcast support Stacked connect Request Dedicated Simplex Camp-on Buffered Class Unidirectional Connection End-to-End flow control Buffer-to-Buffer flow control Fabric can reject frames Fabric busies frames Delivery Order Guaranteed Nx_Port supports Class of service Fabric support Class of service
2005 EMC Corporation. All rights reserved.
Class 1
1:1
Class 2
Class 3
1 to Many Optional Optional Optional
Class 4
Class 5
1:1
Optional Optional
Introduction to Fibre Channel - 25
Classes of Service are different types of topology independent services provided by the Fabric and used by the communicating N_Ports destination. The allocation and retention method between the N_Ports and the level of delivery integrity required for an application distinguish classes of service. If the Fabric is not present, the service is provided as a special case of pointto-point. Fabrics and N_Ports are not required to support all Classes of service.
FC Switch
Fibre Channel addresses are used to designate the source and destination of frames in the Fibre Channel network. The Fibre Channel address field is 24 bits /3 bytes in length. Unlike Ethernet, these addresses are not burned in, but are assigned when the node either enters the loop or is connected to the switch. There are reserved addresses, which are used for services rather than interface addresses.
Addressing layout
The Physical Address is switch specific and dynamically generated in the Fabric Login (FLOGI) Each N_Port has a Fabric-unique identifier, with the following layout FC-SW
Switch Port in Switch
FC-AL
AL_PA
02 1F 00
2005 EMC Corporation. All rights reserved. Introduction to Fibre Channel - 27
The actual physical address is vendor and switch specific. Refer to the switch manufacturers documentation to determine how port addressing is configured. FLOGI will be discussed later in this lesson. Address identifiers are three bytes in length. The Frame Header contains two three-byte fields for address identifiers, the (D_ID) field and the (S_ID) field. Each N_Port has a Fabric-unique identifier, the N_Port Identifier, by which it is known. The source and destination N_Port Identifiers and alias address identifiers are used to route frames within the Fabric.
Addressing layout
1. Domain ID - Identifies the source or target switch inside the Fabric 2. Switch Port - Identifies the source or target port in the switch 3. AL_PA - Used in Private Loop environment, identifying the NL_Port (node loop port) FC-SW
Switch Port in Switch Switch
FC-AL
AL_PA
02 1F 12
2005 EMC Corporation. All rights reserved.
The most significant 8 bits of the Fibre Channel address contain the Domain ID, which basically identifies the switch. In the Fabric environment, this allows frames to be routed between switches. The middle 8 bits contain the area address, which has been implemented as the port address within the switch. In the Fabric environment, this allows frames to be routed between switches. In the Private Loop environment, the Domain and Area fields contain zeros, and the Port field contains the AL_PA (Arbitrated Loop Physical Address) for the NL_Port. NOTE: With a McDATA switch, the third byte will always be 13.
1 0 0 0 0 0 0 0 C 9 2 0 DC 4 0
Example: Emulex HBAs World Wide Name
2005 EMC Corporation. All rights reserved. Introduction to Fibre Channel - 29
A World Wide Name, or WWN, is a 64-bit address used in Fibre Channel networks to uniquely identify each element in the network. Assigned to a host bus adapter (HBA) or switch port by the vendor at the time of manufacture, it is similar to the MAC address in an Ethernet network. There are two designations of WWN; World Wide Port Name and World Wide Node Name. Both are globally unique 64-bit identifiers. The difference lies in where each value is physically assigned. For example, a server may have dual HBAs installed, thus having multiple ports or connections to a SAN. A WWPN is assigned to each physical port. The WWNN represents the entire server, which can be referred to as the node or node process, and is derived from one of the WWPNs. EMC uses the WWPN for all configurations Fibre Channel specifications allow for multiple formats of the World Wide Name. The example shown is that of the IEEE Registered Name Format. NAA is the Name Assignment Authority, which assigns the 24 bits (IEEE Company ID) to the specific vendor (i.e. EMC). Values for World Wide Name formats are based on the IEEE company_id. More information on these formats can be found at www.standards.ieee.org.
5 0 0 6 0 4 8 2E8 9 1 2B9 0
Example: Symmetrix FA World Wide Name
5 0 0 6 0 1 6 0 0 0 6 0 0 1B2
Example: CLARiiON SP World Wide Name
2005 EMC Corporation. All rights reserved. Introduction to Fibre Channel - 30
The WWNs for Symmetrix FA and CLARiiON SP ports are soft-assigned, so they will remain unchanged after a component replacement.
5 0 0 6 0 4 8 2E8 9 1 2B
0101 0000 0000 0110 0000 0100 EMC Company ID 24 bits 1000 0010 1110 1000 1001 0001 0010 1011 10 Symmetrix Serial Number 30 bits 001011101000100100010010101110 195183790
9
0 Port 15bA
0
1 1110 Side FA 4 bits 011110
Symmetrix DMX
5 0 0 6 0 4 8 A CC C 8 3 2
0101 0000 0000 0110 0000 0100 EMC Company ID 24 bits 1000 1 010 1100 1100 1100 1000 0011 0010 10 Half Symmetrix Serial Number 29 bits 01011001100110010000011001010 187900106
A
1 Port 2CB
Because the WWN of the FA is dependent on the Symmetrix Serial Number and the Slot rather than being burned in, the WWN stays constant if the FA fails and has to be replaced. To determine the WWN of an FA depends on the version of the Symmetrix being used. The original calculation was that the last 6 bits of the WWN were used. Bit position 6 specified the port (0 or 1), Bit position 5 the processor ( 0-A/1-B ) and Bit position 1-4 the slot (add 1 to get the adapter). On the DMX series, the ports are laid out as follows, from the top down: DB - processor 4, port 1 DA - processor 4, port 0 CB - processor 3, port 1 CA - processor 3, port 0 BB - processor 2, port 1 BA - processor 2, port 0 AB - processor 1, port 1 AA - processor 1, port 0
The new calculation had to be both backward compatible with previous Symmetrix versions, and at the same time allow for more addresses. The new calculation borrows a bit from the serial number. It was determined that we do not need 30 bits for a serial number, 29 gives us all the range we need. So, bit position 30 is now dubbed a "half bit" The half bit determines which pair of processor we are working with. It is set to 0 for processors A & B, and 1 for C & D. Then, as above, we make use of the processor" bit. This is used along with the half bit to determine exactly what processor. It breaks down like this: Half bit 0 / Processor bit 0 A Half bit 0 / Processor bit 1 B Half bit 1 / Processor bit 0 C Half bit 1 / Processor bit 1 - D Then, like the original calculation, we use the Port bit to determine the 0/A port or the 1/B port.
5 0 0 6 0 1 6 0 0 0 6 0 0 1 B2 5 0
0101 0000 0000 0110 0000 0001 EMC Company ID 24 bits 0110 0000 Port 0000 0000 0110 0000 0000 0001 1011 0010 CLARiiON seed 32 bits 0101 0000
CLARiiON - CX600 WWN seed - 00:60:01:b2 Resulting WWNN therefore is 50:06:01:60:80:60:01:b2 Resulting WWPNs therefore are: Storage Processor A Port 0 - 50:06:01:60:00:60:01:b2 Port 1 - 50:06:01:61:00:60:01:b2 Port 2 - 50:06:01:62:00:60:01:b2 Port 3 - 50:06:01:63:00:60:01:b2 Note : On the FC4500, we have the only real exception as the FC ports are in fact connected to a hub and share their WWPN. To get around this, only one port from each SP would be connected to the Fabric. Two ports existed per SP to allow for a dual cluster direct attach.
Frames routing
When N_Ports are connected to F_Ports, Fabric the Fabric Login begins, associating Sw 01 physical address and World Wide Name Port 07
Physical cable connected
Target
Source
Physical cable connected
ISL
Port 05
FA WWN: 50060482E8912B90
Sw 02
Introduction to Fibre Channel - 34
When N_Ports are connected to F_Ports, the Fabric Login begins, associating physical addresses and World Wide Names.
Target
Port 05
FA WWN: 50060482E8912B90
Sw 02
Introduction to Fibre Channel - 35
The WWN and Physical Address association is done by the switch and is stored using internal tables. The frames are routed using their Physical Address, same as happens in an FC-AL environment, using just the AL_PA.
Fabric Sw 01
Port 07
Physical cable connected
Target
ISL
WWN -> Switch/Port/AL_PA
Port 05
Physical cable connected HBA WWN: 10000000C920DC40
10000000C920DC40 -> 02
0600
FA WWN: 50060482E8912B90
Port 06
Sw 02
Introduction to Fibre Channel - 36
When nodes generate frames, they are routed by their addresses, not their WWNs. However, tables can be built which can associate WWNs to the destination addresses. A World Wide Name is a 64 bit value (16 characters).
Fabric
F_Port N_Port F_Port N_Port Fabric Login 1 1 Accept F_Port F_Port Port Login Accept Process Login Accept Fabric Login Accept N_Port N_Port 2 2 Process x Process x Process y Process y Process z Process z
There are three types of login supported in Fibre Channel: Fabric, Port and Process. All node ports must attempt to log in with the Fabric. This is typically done right after the link or the Loop has been initialized. It consists of the node port transmitting a Fabric Login (FLOGI) frame to the well-known Fabric address hex'FFFFFE'. The normal response is an Accept (ACC) frame from the Fabric back to the node port. Fabric Login accomplishes the following things: It determines the presence or absence of a Fabric. If a Fabric is present, it provides a specific set of operating characteristics associated with the entire Fabric, including which Classes of service are supported. If a Fabric is present, it will optionally assign or will confirm the native N_Port Identifier of the N_Port that initiated the Login. If a Fabric is present, it initializes the buffer-to-buffer credit. Before a node port can communicate with another node port, it must first perform N_Port Login with that node port. Similar to Fabric Login, the N_Port transmits a PLOGI frame to the destination node port. Again, the normal response is an ACC frame. Port Login accomplishes the following things: It provides a specific set of operating characteristics associated with the destination N_Port, including Class of service. With Class 3 services, buffer-to-buffer credit is initialized. PRLI is an acronym for Process login. Process logins establish sessions between related processes on a source and target N_Port. The processes are typically FC-4 layer applications.
Fabric Login
The Fabric Login (FLOGI) is used by an N_Port to determine the presence of a Fabric, then exchange service parameters. The N_Port performs a login to address FFFFFE (F_Port Server) using a source address of 000000. The Fabric Login service returns a frame which assigns an address to the port (24 bits)
When a Fibre Channel device is attached to a Fabric, it will begin a Fabric login (FLOGI). FLOGI is an extended link service command that sets up a session between two participants. With FLOGI, a session is created between an N_Port or NL_Port and a F_Port or FL_Port. An N_Port will send a FLOGI frame that contains its Node Name, its N_Port Name, and service parameters to address 0xFFFFFE.
FLOGI
HBA HBA
N_Port N_Port
FA
N_Port N_Port
Registration Request
Registration Request
Query Request
When the N_Port logs in, it uses a 24-bit port address of 0x000000. Because of this, the Fabric is allowed to assign the appropriate port address to that device, based on the Domain-Area-Port address format. The newly assigned address is contained in the ACC response frame.
An N_Port first opens the destination AL_PA 0x00 before issuing the FLOGI request. The switch accepts the login and returns an accept (ACC) frame to the sender. If some of the service parameters requested by the N_Port are not supported, the switch will set the appropriate bits in the ACC frame to indicate this. When the NL_Port logs in a similar process starts, except that the least significant byte is used to assign the AL_PA and the upper two bytes constitute a Fabric loop identifier. Before an NL_Port logs in, it will go through the LIP on the loop, which is started by the FL_Port, and from this process it has already derived an AL_PA. The switch then decides if it will accept this AL_PA for this device or not. If not, a new AL_PA is assigned to the NL_Port, which then causes the start of another LIP. This ensures that the switch assigned AL_PA does not conflict with any previously selected AL_PAs on the loop.
Once a port has logged in, it can query the Name Service database for information about all other logged in ports
After the node gets its Fabric address from FLOGI, it needs to register with the SNS with port login (PLOGI) on address 0xFFFFFC. The device may register values for all or just some database objects, but the most useful are: Port Identifier (ID) = S_ID Port Name(PN) = WWN of the N_Port Classes of Service (CS) = Class 3 currently FC-4 Types Supported (FT) = SCSI-3 Port Type (PT) = N_Port Once a port has logged in, it can query the Name Service database for information about all other logged in ports.
Some platforms require re-login when there is a long idle period (HP-UX for instance) Service parameters are exchanged between nodes before any upper level commands can be issued
2005 EMC Corporation. All rights reserved. Introduction to Fibre Channel - 42
When an N_Port logs in to another N_Port, a table can be built which will keep track of the WWN of the logged in port along with its Fibre Channel address. For example, when the NT and SUNs login to the Symmetrix, a table is created. This speeds data transfers and node-tonode communications. Some Host types (for example, HP-UX) may not maintain a connection while idle and would need to re-login. Port login is used to establish a session between two nodes, swapping service parameters and making themselves known to each other. Port logins exchange information such as: host address (SID) frame size (receive buffer size) flow control & version information (TOVs) port name (WWPN)
Process login is used to set up the environment between related processes on node ports. The Fibre Channel Protocol for SCSI-3 standard specifies that each SCSI task corresponds to Fibre Channel exchange. A Fibre Channel exchange consists of a single SCSI command or a group of linked SCSI commands. The FCP mapping of SCSI-3 to Fibre Channel defines four information sets that are transferred between SCSI initiator and target. The information sets are modeled after the SCSI-3 architecture defined protocols services. FCP_CMND - Corresponds to Send SCSI Command protocol service FCP_XFER_RDY Transports the offset and request byte count objects of the Send Data-In and Receive Data-Out protocol services FCP_DATA - Transports the data object of the Send Data-In and Receive Data-Out protocol services FCP_RSP corresponds to the Send Command Complete protocol service
FC-3
FC-3 was put into Fibre Channel as a placeholder In concept FC-4 would pass requests to FC-3 that would perform the desired service and then pass onto FC-2
Fibre Channel Level 3 (FC3) Fibre Channel Level 3 (FC3) Fibre Channel Level 2 (FC2) Fibre Channel Level 2 (FC2) Fibre Channel Level 1 (FC1) Fibre Channel Level 1 (FC1) Fibre Channel Level 0 (FC0) Fibre Channel Level 0 (FC0)
FC-3 was put into Fibre Channel as a placeholder. In concept, FC-4 would pass requests to FC-3 that would perform the desired service and then pass onto FC-2. Some of the things that have been identified as probably fitting into FC-3 are: Data Striping Multipathing Mirroring RAID Data encryption Data compression Data Translation
FC-4
The FC-4 level of Fibre Channel is designed to hand off to another protocol such as SCSI Fundamentally, the commands at FC-4 for SCSI allow SCSI initiator and target to communicate over Fibre Channel
Fibre Channel Level 4 (FC4) Fibre Channel Level 4 (FC4) Fibre Channel Level 3 (FC3) Fibre Channel Level 3 (FC3) Fibre Channel Level 2 (FC2) Fibre Channel Level 2 (FC2) Fibre Channel Level 1 (FC1) Fibre Channel Level 1 (FC1) Fibre Channel Level 0 (FC0) Fibre Channel Level 0 (FC0)
The FC-4 level consists of several standards documents describing how different upper-level protocols (ULPs) use the transport services provided by levels FC-2, FC-1 and FC-0. The purpose of an FC-4 protocol mapping is to make a logical connection between the ULPs and Fibre Channel's transport facilities.
ULP ULP Fibre Channel Level 4 (FC4) Fibre Channel Level 4 (FC4) Fibre Channel Level 3 (FC3) Fibre Channel Level 3 (FC3) Fibre Channel Level 2 (FC2) Fibre Channel Level 2 (FC2) Fibre Channel Level 1 (FC1) Fibre Channel Level 1 (FC1) Fibre Channel Level 0 (FC0) Fibre Channel Level 0 (FC0)
ULP (Upper level Protocol) is not actually part of Fibre Channel. Some examples of ULPs are: Small Computer System Interface (SCSI) Intelligent Peripheral Interface (IPI) High Performance Parallel Interface (HiPPI) Bus and Tag (FIPS) or ESCON IEEE 802.2 Logical Link Control (LLC)
The objectives for this lesson are shown here. Please take a moment to read them.
Logical Topology
Physical Topology
Windows Server
Storage
The Fibre Channel environment consists of a physical topology and a logical topology. The physical topology describes the physical interconnects among devices (servers, storage, and switch in the EMC-specific environment). The logical topology describes the logical paths established between the operating system device names and their associated storage ports and volumes. Logical topologies in the EMC/Fibre Channel switch environment can generally be described in terms of fan-in (into the EMC storage array) and fan-out (out of the EMC storage array).
Direct Connect is the simplest topology, where two devices are directly connected to each other
The ANSI Fibre Channel Standards define three topologies: Direct Connect, Arbitrated loop (FC-AL), and Switched Fabric (FC-SW). This slide shows a Fibre Channel topology where two devices are directly connected to each other. EMC specifies a 2 node FC-AL connection called Direct Connect. This Topology is the same as the standard Arbitrated Loop except that there are two nodes in the Loop and the FC-AL Hub is not used.
Arbitrated Loop (FC-AL) is a daisy-chain connecting up to 126 devices in a loop configuration over attachment points called L_Ports (loop ports). FC-AL is a low-cost connectivity solution because it does not require switches. FC-AL is a good choice for small to medium-sized configurations, and provides a growth path by allowing connection of a loop to a switched Fabric. Efficiency and connectivity is enhanced by incorporating one or more hubs into the loop. Routing traffic through a hub on each leg of a loop eliminates the loss of the entire loop, as happens in a hubless loop.
Switched Fabric (FC-SW) is one or more dynamic Fibre Channel switches connecting multiple devices. FC-SW involves a switching device (the Fabric) interconnecting two or more nodes. Rather than traveling around an entire loop, frames are routed between source and destination by the Fabric.
Fibre Channel hubs are used with Fibre Channel Arbitrated Loop (FC-AL) to increase server and storage connectivity. Using a hub, multiple servers can access multiple storage devices. A Hub can only have one Full Duplex connection between port pairs at any one time. Although every port is capable of full speed operation, only one pair of ports can be active at any one time. This is due to the fact that only two nodes in an FC-AL environment can be active at any one time, and they must be communicating with each other (pair). EMC Symmetrix storage systems are qualified with hubs in limited configurations with HP-UX, Sun Solaris, Windows NT, Windows 2000, Windows 2003, and Siemens servers.
Switches
A Fibre Channel switch is a device that routes data between host bus adapters and fibre adapters on storage systems A Fabric is a single switch or multiple switches that interconnect various N_Ports, and is capable of routing frames by using only the D_ID information in a Fibre Channel frame header Larger Fabric will be built by connecting multiple switches through E_Ports
Switched Fabrics offer strong advantages over hubs, in both performance and manageability. For example, a hub can have only one full-duplex connection between port pairs at one time. Although every port in a 10-port hub is capable of full-speed operation, only one pair of ports can be active at any one time. A switched Fabric, however, can have every port pair running full duplex simultaneously at 200 MB/s. In a switched topology, aggregate bandwidth of 200 MB/s times the Fabric port count is possible. The aggregate bandwidth of a hub is fixed, but the aggregate bandwidth of a Fabric scales as additional port pairs are added. This scaling capability is an important management feature, because it allows the administrator of the enterprise storage network to reallocate the bandwidth of the network among N_Ports without moving cables and reconfiguring the environment. A second management benefit of switched Fabrics over hubs is a Fabric-based service called the name server, which maintains a table of all logged-in devices. Used by the N_Ports for device discovery, the table is maintained by the Name Server during Fabric reconfigurations.
Lower number of ports Availability features High performance Web-based management features
Highest port count Highest availability Highest performance Web-based and/or console-based management features
In a SAN, implementation is necessary to select the director, switch or hybrid technology solution. The switch/director decision is often financial. Technically, directors are preferable to switches. If we consider the differences between them: Directors are easier to manage because the relationship between ports is equal and, therefore, it does not need to be considered in performance planning. A single high-availability Fabric is possible using directors, though switches require two Fabrics. Directors have scaled failure localization. This means the GBIC, power supply, etc. can each be replaced without bringing down the entire device, limiting risk of catastrophic failure. Switches offer some localization, at the GBIC or power supply, but in general, a physical failure results in whole device replacement. Directors are considered to be more highly available than switches. To be fully redundant, there is a requirement for multiple connections to more than one network, with complete routes to all devices throughout those networks. This improves resiliency of the Fabric, allowing the SAN to maintain its functionality in the even of failures. Because fault isolation in a director is at a more granular component level and the components are already dual redundant, directors have the least adverse impact on availability.
COST
MDS-9216i/A
Connectrix Switches
Redundant fans and power supplies High availability through redundant deployment Departmental and data-center deployment
DS-4100B
DS-8B3
SERVICE LEVEL
2005 EMC Corporation. All rights reserved. Introduction to Fibre Channel - 55
Only EMC offers a complete range of SAN productsfrom Connectrix directors for data-center deployments to Connectrix switches for data-center and departmental deployment. The Connectrix Family gives customers: The widest range of SAN service levels More choice Additional Fibre Channel functionality A pathway to IP SANs An additional platform for future network-hosted applications
Node port, a port at the end of a point-to-point link Node Loop port, a port which supports the arbitrated loop topology Fabric port, the access point of the Fabric which connects to a N_Port Fabric Loop port, a Fabric port which connects to a NL_Port Expansion port on a switch. Links multiple switches General port, a Connectix McDATA switch port with the ability to function as either an F_Port or an E_port Universal port, Connectrix B series equivalent to a G_Port
Introduction to Fibre Channel - 56
In an environment using Host Bus Adapters and Symmetrix FC Directors, ports are configured as either N-Ports or NL-Ports: N_Port - Node port, a port at the end of a point-to-point link NL_Port - Node Loop port, a port which supports the arbitrated loop topology Fibre Channel Switch ports are also configured for specific applications: F_Port - Fabric port, the access point of the Fabric which connects to a N_Port FL_Port - Fabric Loop port, a Fabric port which connects to a NL_Port E_Port - Expansion port on a switch. Links multiple switches G_Port - General port, a Connectix McDATA switch port with the ability to function as either an F_Port or an E_port U_Port - Universal port, Connectrix B series equivalent to a G_Port
Fabric
N_Port N_Port N_Port N_Port
N_Port N_Port
Fabric
E_Port E_Port F_Port F_Port ISL N_Port N_Port E_Port E_Port G_Port G_Port
A Fabric is a switch or a group of switches linked together. When looking at the graphic above you will notice that the cloud of the Fabric is actually two separate switches attached together via an ISL (Interswitch Link). Attached to the Fabric are several N ports which represent end port devices such as HBAs and Storage ports. The term Enterprise Director indicates a switch that has all major components redundant at the hardware level. If any major part fails, the switch will automatically fail over, maintaining operation during the failure. The Enterprise directors are commonly used as core switches in a core edge topology. A Departmental switch has less redundancy and is meant for smaller workgroups. With the lack of redundancy, the departmental switches are commonly used as edge switches for core edge designs with PowerPath in place for redundancy in case of a switch failure.
Zoning
Partitions a Fibre Channel switched Fabric into subsets of logical devices Zones contain a set of members that are permitted to access each other A member can be identified by its Source ID (SID), its World Wide Name (WWN), or a combination of both
Zoning is used to partition a Fibre Channel switched Fabric into subsets of logical devices. Each zone contains a set of members that are permitted to access each other. Members can be switch ports, HBAs, or storage ports. When zoning is enabled, members in the same zone can see and communicate with each other, but members in separate zones cannot. Ports and devices distributed across multiple switches in a Fabric may be grouped into the same zone. A zone member can be identified by its Source ID (SID), its World Wide Name (WWN), or a combination of both. Zones can be created using each of these forms of member identification. Thus, there are essentially three types of zoning: hard zoning, soft zoning, and mixed zoning.
What is Zoning?
10000000C920C4E4 10000000C920C321
50060482B8912B8E 50060482B8912B9E
50060482B8912B8F
200000E069000CFE
50060482B8912B9F
200000E0690014B8
200000E06900156D
500601684003491C
A Switched Fabric can be subdivided into a number of zones: A single zone typically includes two or more ports Zones can be created by Switch Port Number (Static) HBA WWN (Flexible) Custom Nickname/Alias EMC recommends that zones include only one HBA (Single HBA zoning), however an HBA may belong to multiple zones. Zones may include one or more EMC Symmetrix FA ports. FAs may belong to multiple zones.
Under single-HBA zoning, each HBA is configured with its own zone. The members of the zone consist of the HBA and one or more storage ports, such as a Symmetrix Fibre Adapter (FA) port, with the volumes that the HBA will use. This zoning practice provides a fast, efficient, and reliable means of controlling the HBA discovery/login process. Without zoning, the HBA will attempt to log in to all ports on the Fabric during discovery and during the HBAs response to a state change notification. With single-HBA zoning, the time and Fibre Channel bandwidth required to process discovery and the state change notification are minimized. Two VERY good reasons for Single HBA Zoning: Cuts down on the reset time for any change made in the state of the Fabric. Only the nodes within the same zone will be forced to log back into the Fabric after a RSCN (Registered State Change Notification) When a nodes state has changed in a Fabric (i.e. cable moved to another port), it will have to perform the Fabric Login process again before resuming normal communication with the other nodes it is zoned with. If there is only one SCSI Initiator in the zone (HBA), then the amount of disrupted communication is reduced. If you had a zone with two HBAs and one of them had a state change, then BOTH would be forced to log in again, causing disruption to the other HBA that did not have any change in its Fabric state. Performance can be severely impacted by this.
FC Switch
WWPN = 10:00:00:60:69:40:DD:A1 Domain ID = 25 Port = 3
Host
Fabric
FC Switch
WWPN = 50:06:04:82:E8:91:2B:9E
Storage
In general, zoning can be divided into two categories: Port zoning WWN zoning
WWN Zoning
WWPN = 10:00:00:60:69:40:8E:41 Domain ID = 21 Port = 1
WWPN = 10:00:00:00:C9:20:DC:40
FC Switch
WWPN = 10:00:00:60:69:40:DD:A1 Domain ID = 25 Port = 3
Host
Fabric
FC Switch
WWPN = 50:06:04:82:E8:91:2B:9E
Storage
WWN zoning creates zone sets using the WWNs of the attached nodes. This allows you to move nodes from one switch port to another without having to change the zone configuration. The zone set becomes independent of individual switch ports.
Port Zoning
WWPN = 10:00:00:60:69:40:8E:41 Domain ID = 21 Port = 1
WWPN = 10:00:00:00:C9:20:DC:40
FC Switch
WWPN = 10:00:00:60:69:40:DD:A1 Domain ID = 25 Port = 3
Host
Fabric
FC Switch
WWPN = 50:06:04:82:E8:91:2B:9E
Storage
Port zoning, based on physical topology, uses the domain ID and port number of the switch (in the format domainID:port number), not the port WWN of the attached device to determine the zone set. This method allows you to change the HBA without changing any zoning information. However, it does not give you the flexibility to physically move attached nodes between switch ports without having to redefine the port number in the zone set.
Zoning Components
Member #1 Zone #1 Member #2 Active Zone Set Member #3 Zone #2 Member #4 Member #5 Zone #3 Old Zone Set #1 Old Zone Set #2
2005 EMC Corporation. All rights reserved. Introduction to Fibre Channel - 64
Members are attached adapters which can be included in a zone. They include HBAs and Storage Fibre adapter ports. A Storage Fibre adapter is a front end port on a storage system. Examples include Symmetrix FA ports, CLARiiON SP ports and Compaq adapter ports. A zone contains a set of members that can access each other. A member can be a Storage Fibre adapter or a Host Bus Adapter (HBA) that is logged into a switch device. Devices spread throughout multiple switches in a multiswitch Fabric may be grouped into the same zone. Multiple zones can be assigned membership in one zone set. Zones are built and then included into a specific zone set based on the customers requirement to activate multiple zones simultaneously. Zone sets are then activated to simultaneously activate the zones in the set. Zoning a hosts HBA with a Symmetrix or CLARiiON Storage Fibre adapter port allows communication between those members. Volumes visible on that storage port will now be available to the host. Members of a zone can see each other; members in different zones cannot. A zone set is a group of zones that you can activate or deactivate as a single entity in either a single unit or a multi-unit Fabric. Only one zone set can be active at one time per Fabric. The active zone set is a single zone set that is currently enabled. When a zone set is active, all zones that are members of that zone set are active.
VSAN
VSANs are logical SANs providing isolation among physically connected devices.
VSANs are logical SANs over a common physical Fabric allowing logical segments in the network. Multiple independent SANs over a common physical infrastructure provide isolation among devices that are physically connected.
SONET
Long-distance solutions (1000s km) Smaller sites feeding into WDM
IP
Native IP for simplicity Bridged IP when WDM and SONET not available Reuse part of existing infrastructure
WDM Metro region (100s km) Shared infrastructure for many applications High bandwidth needs Limitation: Availability of dark fiber; Can be high cost SONET Long-distance solutions (1000s km) Smaller sites feeding into WDM Limitation: Cost varies (IP can be less expensive); Can be difficult to provision IP Native IP for simplicity Bridged IP when WDM and SONET not available Reuse part of existing infrastructure Limitation: Need a well-designed network; Limited interoperability between IP bridges
SAN Routing is a new technology development of the FC protocol. A study group within T11 was formed that is dedicated for the development of the standards to accommodate the solution.
SAN virtualization uses the storage network to redirect IOs on the fly.
iSCSI
Native TCP/IP protocol An IP-based protocol for establishing and managing connections between IPbased storage devices, hosts, and clients No Fibre Channel content
Today, there are three block storage over IP approaches: iSCSI, FCIP, and iFCP. iSCSI is a native IP-based protocol for establishing and managing Connections between IPbased storage devices, hosts, and clients. There is no Fibre Channel content.
FCIP
TCP/IP based tunneling/encapsulating protocol for connecting/extending Fibre Channel SANs More IP content, little Fibre Channel content
FCIP is a TCP/IP based tunneling/encapsulating protocol for connecting/extending Fibre Channel SANs. It contains more IP content, little Fibre Channel content.
iFCP
Gateway to gateway protocol for FC over IP Mapping natively in IP across Fibre Channel and IP An IP-based tunneling protocol for interconnecting Fibre Channel devices together in place of Fibre Channel switches More Fibre Channel content; when iFCP creates the IP packets, it inserts information that is readable by network devices and routable within the IP network. IFCP wraps Fibre Channel data in IP packets but maps IP addresses to individual Fibre Channel devices
2005 EMC Corporation. All rights reserved. Introduction to Fibre Channel - 71
iFCP is a gateway to gateway protocol for FC over IP, mapping natively in IP across Fibre Channel and IP. It is an IP-based tunneling protocol for interconnecting Fibre Channel devices together in place of Fibre Channel switches. It has more Fibre Channel content. When iFCP creates the IP packets, it inserts information that is readable by network devices and routable within the IP network. IFCP wraps Fibre Channel data in IP packets but maps IP addresses to individual Fibre Channel devices.
Native iSCSI allows for all communications using Ethernet. Initiators may be directly attached to iSCSI Targets or may be connected using standard Ethernet routers and switches.
Bridging architectures allow for the Initiators to exist in an Ethernet environment while the storage remains in a Fibre Channel SAN.
WAN
Extension architectures are most often used to provide connectivity across large distances. Either FCIP or iFCP bring the long distance benefits of IP to Fibre Channel.
Course Summary
Key points covered in this course: Fibre Channel Architecture Fibre Channel Layers Fibre Channel Topologies SAN concepts
These are the key points covered in this training. Please take a moment to review them. This concludes the training. In order to receive credit for this course, please proceed to the Course Completion slide to update your transcript and access the Assessment.