SII14 FCoE Introduction

sII14 - Fibre Channel over Ethernet:
An Introduction
Jim Blue
IBM SAN Central
STG Technical Conferences 2009
2009 IBM Corporation
Trademarks and Disclaimers

The following are trademarks of the International Business Machines Corporation in the United States and/or other countries. For a complete
list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:
The following are trademarks or registered trademarks of other companies:
Java and all Java based trademarks and logos are trademarks of Sun Microsystems, Inc., in the United States and other countries or both
Microsoft, Windows,Windows NT and the Windows logo are registered trademarks of Microsoft Corporation in the United States, other
countries, or both.
Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium
are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
UNIX is a registered trademark of The Open Group in the United States and other countries or both.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
NOTES:
Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are
dependent on many factors including system hardware configuration and software design and configuration. Some measurements quoted in
this document may have been made on development-level systems. There is no guarantee these measurements will be the same on
generally-available systems. Users of this document should verify the applicable data for their specific environment.
IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.
Information is provided AS IS without warranty of any kind.
All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have
used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending
on individual customer configurations and conditions.
This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other
countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the
product or services available in your area.
sII14 FCoE: An Introduction
Trademarks and Disclaimers (continued)

NOTES:
All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and
objectives only.
Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not
tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.
Prices are suggested US list prices and are subject to change without notice. Starting price may not include a hard drive, operating system
or other features. Contact your IBM representative or Business Partner for the most current pricing in your geography.
Any proposed use of claims in this presentation outside of the United States must be reviewed by local IBM country counsel prior to such
use.
The information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these
changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the
program(s) described in this publication at any time without notice.
Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an
endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web
sites is at your own risk.
IBM makes no representation or warranty regarding third-party products or services including those designated as ServerProven,
ClusterProven or BladeCenter Interoperability Program products. Support for these third-party (non-IBM) products is provided by non-IBM
Manufacturers.
IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not
give you any license to these patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive,
Armonk, NY 10504-1785 USA.
STG Technical conferences 2009
Agenda
Overview of Fibre Channel over Ethernet (FCoE) as a
product
Protocol Summary
Based on
Version 2 of FC-BB5 (ratified)
July 2009 versions of select IEEE standards drafts
FC-BB6 is already in development
What Is Fibre Channel over Ethernet?

A direct mapping of Fibre Channel upper layers onto
Ethernet as a transport mechanism
Aims to reduce cost through sharing of Ethernet
infrastructure
New standard specified by ANSI T11 Committee
within FC-BB-5
Ethernet enhancements are being defined by IEEE in
the Ethernet standards
Backed by virtually entire storage networking industry,
including IBM, Brocade, Cisco, QLogic, HP, and Intel
5
General Characteristics
Simple No complicated flow control or frame loss
detection
Entire standard 44 pages (FC-BB_E, Section 7)
Will make use of enhancements being added to

Ethernet standards
Alternative to native Fibre Channel
Generally intended for, but not limited to, 10Gb or
faster Ethernet
Basic FCoE Installation
Why use FCoE?

Large hosts generally use two networks right now:
Ethernet and Fibre Channel
FCoE eliminates need for additional Host Adapters
Reduces Cost
Reduces slot and bus space usage
Very helpful for Blades
Protocol / Stack Comparison
iSCSI
iFCP or FCIP
FCP
FCoE
FCoE vs. FCIP/iFCP

Not really equivalent
FCoE Direct replacement for FC-0, FC-1 and
limited portions of FC-2 layers while not using TCP/IP
FCIP/iFCP Generally used for Distance Extension
FCoE unsuitable for long distances or complex
networks (later version of standards may allow NAT
implementation)
No FCIP or iFCP storage devices or HBAs

FCoE capable storage devices are very likely
10
FCoE vs. iSCSI
11
FCoE
iSCSI
Local-area, lossless links,

no routing allowed
Allows many hops, lossy

connections and high latency
Simple encapsulation of
Fibre Channel
Substantial complexity on top of

TCP
Low overhead similar to

Fibre Channel
Overhead varies typically

higher
Storage administrators know

Fibre Channel well
TCP/IP well understood by most

IT staff
Frame loss can quickly

become catastrophic
Frame loss recovery built into

protocol stack
FCoE vs. iSCSI long-term

No overwhelming advantages for one vs. other at this time
for now
FCoE appears to be off to a better start than iSCSI
FCoE appropriate for existing SAN installs
No complex, gateway for non-iSCSI-native systems
FCoE expands the choices for new solutions

Can take advantage of existing IP infrastructure
Lossless Network needed for FCoE helpful to both

And the winner will be ????
12
?
Software vs. Hardware Implementations

FCoE is available as a software driver for many operating
systems
Primary use is proof-of-concept and interoperability testing
Virtually all production deployments expected to be

hardware based
New Converged Network Adapters (CNA)
In reality, FC HBAs with an Ethernet MAC layer or vice-versa
Will appear as another SCSI / RAID controller
13
FCoE Disadvantages
No flow control (not even buffer credits)
Relies on Ethernet extensions under development
802.1Qaz - Enhanced Transmission Selection (ETS)
Sponsor Ballot voted 07/2009 / full ratification by end of 2009 ???
802.1Qau -Congestion Notification

Sponsor Ballot target date scheduled for November 2009 and Forwarding
target date of March 2010
Required to deal with potential FC congestion, even if

Ethernet is not congested
14
FCoE Disadvantages
Currently limits transmission rate
802.3-2008 PAUSE mechanism
(all or nothing)
802.1Qbb Priority Flow Control (PFC)

(requires priority tagging to be in place)
Needs support for small jumbo frames to work

efficiently (frame size of approximately 2500 bytes)
Assumes high-quality network
Lossless Ethernet network required
15
Possible Industry Deployment
FCoE intended for 10Gb Ethernet links **

1Gb functional, not recommended
Blades with Ethernet/FCoE splitter on in-Chassis

switch **
Edge switches ** / Director blades
Entire Directors / Core Ethernet switches
Storage
EMC and NetApp have announced that they are working on products
** Products already announced and available

16
Summary of FCoE as a Product

FCoE, in summary:
A new standard designed to converge Fibre Channel
SANs and Ethernet networks
Simple in concept and design
Details still being developed
Fundamentally different from iFCP and FC/IP
In competition with iSCSI
Questions? (Information on protocol details to follow)
17
Protocol Terms and Concepts

FCoE Mapper (FCM) Provides translation between
Ethernet and Fibre Channel addresses
Present in NICs and switches
FCoE Forwarder (FCF) Ethernet switch that

understands Fibre Channel and FCoE encapsulation
Also provides FCoE discovery and login services
Does not need actual Fibre Channel ports
Provides same services as FC switch (FC-SW4)
FCoE ENode (ENode) Ethernet port w/ FCoE

abilities (software, hardware or both)
18
Terms and Concepts

FC-MAP Fabric ID
Usually 0E:FC:0z (z = a Fabric ID number, default is 0)
Used for first three bytes of Fabric-Assigned MAC
Addresses
Once Discovery and FLOGI (similar to normal FC

operations) are completed, FCoE_LEP (link end point)
is created at CNA and in the FCF for virtual ports and
regular FC operations continue.
19
Terms and Concepts
FCoE_LEP (Link End Point) FCoE component which handles FC frame

encapsulation or decapsulation as well as the transmission or reception of
encapsulated frames across a single virtual link. Each LEP has its own
MAC unique MAC address.
FCoE_LEP keeps up with 3 items as well as encapsulation duties:

MAC address of itself
Associated N_Port ID
MAC address of FCoE_LEP at other end of virtual link
20
Terms and Concepts

Virtual Ports The port types associated with
FCoE_LEP instances that provides the functionality
of physical FC port types. Valid types are:
VN_port (on the ENode)
VF_port (FCF port to VN_port)
VE_port (FCF to FCF)
21
FCoE Standard
Listed in ANSI T11, FC-BB-5, Chapter 7
Two protocols in one standard:
FCoE The encapsulation standard
FCoE Initialization Protocol (FIP) Protocol to provide
device discovery and login services
Use different EtherTypes
Facilitates FIP snooping by non FCF switches
22
FCoE and the Protocol Stack
23
FCoE Frame Format

2 new EtherTypes:
8906 FCoE type
8914 FCoE Initialization Protocol (FIP)
FC Frame Delimiters start with K28.5 which must be

changed in the FCoE header and trailer
Mini-Jumbo Ethernet frames strongly suggested to carry
maximum FC payload without needing to fragment
24
FCoE Frame Forwarding

Relies on Ethernet Spanning Tree Protocol (STP) or
variations of Spanning Tree mechanism
Rapid STP (RSTP)
Per VLAN STP (PVST) Cisco proprietary
Multiple STP (MSTP)
No built-in routing such as FSPF

Load balancing generally poor, but should improve as
standards are developed and vendors add unique
refinements/enhancements
25
More-Complex FCoE Topology is on the horizon

Not a redundant network design!
SAN A
SAN B
FCF
26
Ethernet
FCoE Network Without Much FC is also coming
27
FCoE Network Currently Supported and working
28
Addressing of Virtual ports

Two options
Server Provided MAC Address (SPMA)
Typical Ethernet standard globally unique address
Some HBA vendors support, no FCF vendors yet
Fabric Provided MAC Address (FPMA)

Format XX:XX:XZ:YY:YY:YY
XX:XX:XZ = FC-MAP (Z = Domain ID, 0E-FC-00 is default)
YY:YY:YY = FCID (or N_Port ID assigned during FLOGI)
No relation to MAC of the lossless Ethernet adapter
29
Addressing (cont.)
Multiple MACs per adapter expected
One for regular Ethernet traffic (i.e. TCP/IP)
ENode MAC Address
One for each occurance of FCoE_LEP

VN_Port MAC Address
NPIV supported
30
FCoE Discovery
Special Fabric Initialization Protocol (FIP) frames

Separate from regular FCoE frames
Used for FLOGI/FDISC, LOGO, ELP and FCoE
discovery solicitation/advertisement
Source MAC is the burned-in MAC Address of the
lossless Ethernet port
Initial Destination MAC is multicast MAC for all FCF
31
FCoE Discovery
32
Ethernet devices use MAC addresses so well-known

names will not work
Ethernet has the concept of Multicast (group of ports)
FCoE uses several multicast addresses

All FCoE MAC
01:10:18:01:00:00
All ENodes MAC
01:10:18:01:00:01
All FCF MAC
01:10:18:01:00:02
FCoE Discovery and FC Login Process

1. Discover available VLANs which support FCoE**
** Brocade defaults to VLAN 1002 only and ignores FIP VLAN Requests.
2. Send multi-cast Ethernet frame to All-FCF-MACs
multicast address with Discovery Solicitation
payload.
Contains:
33
MAC Address used to receive response

The hosts name
Maximum Receive Size (Max_FCoE_Size)
If an FCF itself, includes FC-MAP (Mapped Address
Prefix if using FPMA), similar to FC Domain ID
FCoE Discovery and FC Login Process (cont.)

3. FCF responds with Discovery Advertisement
Advertisement contains:
Priority 1-byte value used to determine which FCFMAC to use, if multiple respond
MAC Address of the FCF-MAC
If using FPMA, the FC-MAP of the FCF
Name of the Switch
Name of the Fabric
34
This response will be Max_FCoE_Size bytes to verify

that the network is capable of handling that size frame
FCoE Discovery and FC Login Process (cont.)

4. ENode sends FIP FLOGI
Just like FC FLOGI, only encapsulated within a FIP frame
Returned FCID also used to build FPMA MAC Address
For FCFs, FIP ELP sent instead
For NPIV, FIP FDISC is used
Works just like counterpart
5. The FCF responds with a FIP LS_ACC (or

ELP_SW_ACC)
35
At this point, VF_port at FCF and VN_port on ENode have

been created and are now ready to work with Name Server
to register itself and query zoning database.

1. FIP Request for FCoE VLAN(s)
using All FCF multicast MAC as SRC
FIP VLAN info (currently only Cisco)
ENode
36
FCF
L/E
L/E
MAC
MAC

ENode
FCF
2. FIP Discover Solicitation using

All FCF multicast MAC as Src
3. FIP Discover Advertisement
37
L/E
L/E
MAC
MAC

ENode
FCF

L/E
L/E
MAC
MAC
4. FIP FLOGI Request using
FCF MAC as Destination
38

ENode
FCF

VNPort
**
FCoE L/E
-LEP MAC
Virtual link established once

the FLOGI is accepted
L/E
FCoE
MAC -LEP
VFPort
4. FIP FLOGI Request using

FCF MAC as Destination
FCID: 04-0B-00
5. FIP FLOGI Response which returns

** FCID with new MAC **
MAC: 0E-FC-00-04-0B-00 All future FCoE traffic will now use this MAC address
39
Ethernet Enhancements for FCoE

Pause mechanism (described in 802.3x updates) is used to stop
and resume Ethernet traffic to prevent input buffer congestion. The
Receiving port sends the Pause frame to the transmitter.
ENode
FCF input
buffer utilization
100%
VNPort
**
FCoE L/E
-LEP MAC
FCF
No
frames
sent
L/E
FCoE
MAC -LEP
Pause
Time = xxxx
VFPort
Pause
Time = 0
0
Traffic being sent by ENode

40

Per Priority Pause Flow Control (PFC) in IEEE 802.1Qbb allows
the Pause mechanism to be used on an individual traffic class.
FCF
ENode
VNPort
**
41
FCoE L/E
-LEP MAC
0
1
2
3
4
5
6
7
L/E
FCoE
MAC -LEP
VFPort

Congestion Notification being standardized by IEEE 802.1Qau.
Congestion Point (CP) detects congestion and signals Reaction
Point (RP) to limit associated traffic.
42

Enhanced Transmission Selection (ETS) being defined in IEEE
802.1Qaz is adding QoS to Priority Groups based on 802.1Q VLAN tag
and priority. Priority levels (or class of service) can be mapped into
Groups and each priority group can then be allocated a portion of total
bandwidth.
43

IEEE 802.1Qaz also describes Data Center Bridge
Capability Exchange (DCBX which primarily enhances
Link Level Discovery Protocol (LLDP). Main
functionality inlcudes:
DCE peer discovery and link configuration
Detection of configuration mismatches
Negotiation Layer
Enable multipathing and load balancing where
possible
Provides link error detection
44
Summary
FCoE has two parts

FCoE frames simple Ethernet frame to transport a
Fibre Channel frame.
FIP protocol handles device discovery, fabric login
functions, and address assignment to allow traffic to
start
45
Special Session sSM60

"Meet The Storage Experts
Ludwig von Beethoven II
Thursday, 22 October
17:30 to 19:30
46
Thank you!
47

SII14 FCoE Introduction

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

SII14 FCoE Introduction

Загружено:

Авторское право:

Доступные форматы

sII14 - Fibre Channel over Ethernet:

STG Technical Conferences 2009

2009 IBM Corporation

STG Technical Conferences 2009

Trademarks and Disclaimers

sII14 FCoE: An Introduction

2009 IBM Corporation

STG Technical Conferences 2009

Trademarks and Disclaimers (continued)

sII14 FCoE: An Introduction

2009 IBM Corporation

STG Technical conferences 2009

sII14 FCoE: An Introduction

2009 IBM Corporation

STG Technical conferences 2009

What Is Fibre Channel over Ethernet?

sII14 FCoE: An Introduction

2009 IBM Corporation

STG Technical conferences 2009

Will make use of enhancements being added to

sII14 FCoE: An Introduction

2009 IBM Corporation

STG Technical conferences 2009

Basic FCoE Installation

sII14 FCoE: An Introduction

2009 IBM Corporation

STG Technical conferences 2009

Why use FCoE?

sII14 FCoE: An Introduction

2009 IBM Corporation

STG Technical conferences 2009

Protocol / Stack Comparison

sII14 FCoE: An Introduction

2009 IBM Corporation

STG Technical conferences 2009

FCoE vs. FCIP/iFCP

No FCIP or iFCP storage devices or HBAs

sII14 FCoE: An Introduction

2009 IBM Corporation

STG Technical conferences 2009

FCoE vs. iSCSI

Local-area, lossless links,

Allows many hops, lossy

Substantial complexity on top of

Low overhead similar to

Overhead varies typically

Storage administrators know

TCP/IP well understood by most

Frame loss can quickly

Frame loss recovery built into

sII14 FCoE: An Introduction

2009 IBM Corporation

STG Technical conferences 2009

FCoE vs. iSCSI long-term

FCoE expands the choices for new solutions

Lossless Network needed for FCoE helpful to both

sII14 FCoE: An Introduction

STG Technical conferences 2009

Software vs. Hardware Implementations

Virtually all production deployments expected to be

sII14 FCoE: An Introduction

2009 IBM Corporation

STG Technical conferences 2009

802.1Qau -Congestion Notification