Fibre Channel Storage Area Network Design: BRKSAN-2701

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 1
Fibre Channel Storage

Area Network Design
BRKSAN-2701
BRKSAN-2701
© 2006, Cisco Systems, Inc. All rights reserved. 1

Presentation_ID.scr
Agenda
Brief SAN Technology Overview

Fibre Channel Protocol
Virtual SAN (VSAN), Zoning
Port Channels, IOD
SAN Design Principles and Considerations

Design Factors
Design Types
Design Flexibility
SAN Security Design Considerations
Interoperability Design Considerations
BRKSAN-2701
SAN Technology
Overview
BRKSAN-2701

Presentation_ID.scr
SAN Technology Overview—Agenda

FC Communications
Port types, ISL
Addressing, Framing, Timers
Virtual SAN (VSAN), Zoning

Port Channels, IOD
Virtual Output Queuing (VOQ)
BRKSAN-2701
Fibre Channel Communications

Point-to-point oriented
Facilitated through device login
N_Port-to-N_Port connection
Logical node connection point
Flow controlled
Buffer-to-buffer credits and end-to-end basis
Acknowledged
For certain classes of traffic, none for others
Multiple connections allowed per device
Node Node
Transmitter Transmitter
N_Port N_Port
Receiver Receiver
Link
BRKSAN-2701

Presentation_ID.scr
FCP—Protocol Stack
SCSI FC
SCSI Applications (File SCSI Applications (File

Systems, Databases) Systems, Databases)
SCSI Block Commands SCSI Stream

Commands
SCSI Commands, Data, SCSI Commands, Data,

and Status and Status
Parallel SCSI Fibre Channel Protocol

Interfaces (FCP)
SCSI Bus
Fibre Channel
BRKSAN-2701
Fibre Channel Port Types

Fibre Channel Switch
Input Fabric Output
Port X Port
Fabric
Node NL_Port FL_Port E_Port E_Port Switch
G_Port F_Port N_Port Node

Node NL_Port

Node NL_Port
BRKSAN-2701

Presentation_ID.scr
Inter-Switch Link (ISL)
EISL
The interconnection between switches is called the ISL

E_Port to E_Port (‘Expansion port)
Supports all classes of service

Class 1, 2, 3, and a special Class F (switch-to-switch)
FC-PH permits consecutive frames of a sequence to be routed

over different ISL links for maximum throughput
Cisco’s implementation is to dedicate an FC_ID pair and/or a given
exchange to an ISL bundle member to guarantee in-order delivery
for exchange/sequence frames
Cisco Extended ISL (EISL, TE port)
BRKSAN-2701
Buffer to Buffer Credit Flow Control

BB_Credits and Distance
~ 2 km per Frame
1 Gbps FC
~ 1 km per Frame
2 Gbps FC
~ ½ km per Frame
4 Gbps FC
16 Km
BB_Credits are used to ensure enough FC frames in flight

A full (2112 byte) FC frame is approx 2 km long at 1 Gbps,
1 km long at 2 Gbps and ½ km long at 4 Gbps
As distance increases, the number of available BB_Credits
need to increase as well
Insufficient BB_Credits will throttle performance—no data will be
transmitted until R_RDY is returned
BRKSAN-2701

Presentation_ID.scr
FSPF Protocol
FSPF stands for Fabric Shortest Path First

Path selection protocol used in Fibre Channel
Based on link state protocol
Fibre Channel standard defined in FC-SW2
Conceptually based on Open Shortest Path First
(OSPF) Internet routing protocol
BRKSAN-2701
FC_ID Address Model

FC_ID address models help speed up routing
Switches assign FC_ID addresses to N_Ports
Some addresses are reserved for fabric services
Private loop devices only understand 8-bit address (0x0000xx)
FL_Port can provide proxy service for public address translation
Maximum switch domains = 239 (based on standard)
8 Bits 8 Bits 8 Bits
Switch Topology Switch

Area Device
Model Domain
Private Loop Device Arbitrated Loop

Address Model 00 00 Physical Address
(AL_PA)
Public Loop Device Arbitrated Loop
Switch
Address Model Area Physical Address
Domain
(AL_PA)
BRKSAN-2701

Presentation_ID.scr
Fibre Channel FC-2 Hierarchy
Multiple exchanges are initiated between initiators (hosts) and
targets (disks)
Each exchange consists of one or more bidirectional sequences
Each sequence consists of one or more frames
For the SCSI3 ULP, each exchange maps to a SCSI command
OX_ID and
RX_ID Exchange
SEQ_ID Sequence Sequence Sequence
SEQ_CNT Frame Frame Frame
Frame Fields
ULP Information Unit
BRKSAN-2701
Fibre Channel Timers

Receiver-Transmitter Time-Out (R_T_TOV)
Used to time events at the link level
Loss of synchronization
Times responses for link reset protocol
Default value: 100 ms
Error Detect Time-Out (E_D_TOV)
Timers for events and responses at the sequence level
Missing ACK or R_RDY when buffer credit has reached zero
N_Port logout
Timer value is set at fabric login to accommodate the network environment
based on delivery time of frames
Default value: 2 sec
Resource Allocation Time-Out (R_A_TOV)
Time-out value for how long to hold resources associated with a failed operation
Needed to free shared resources for reuse
Value to determine how long a port needs to keep responding to a link service request
before an error is detected
2xE_D_TOV in point-to-point and fabricwide from 1 to 231 – 1
Default value: 10 s
BRKSAN-2701

Presentation_ID.scr
VSANs, Zoning, IVR Zones
Fabric virtualization—VSAN
Provide independent (‘virtual’) fabric services on a single
physical switch
VSAN—Design Foundation
Zoning
Fabric routing (Inter-VSAN Routing—IVR)
Ability to provide selected connectivity between virtual fabrics
without merging them
Virtual Fabric Trunking (VSAN Trunking)
Ability to transport multiple virtual fabrics over a single ISL
or common group of ISLs
IVR zones
BRKSAN-2701
physical switch
VSAN—Design Foundation
Zoning
Fabric routing (Inter-VSAN Routing–IVR)
Virtual Fabric Trunking (VSAN Trunking)
IVR zones
BRKSAN-2701

Presentation_ID.scr
Virtual Storage Area Networks (VSAN)
VSANs address the limitations of
common SAN deployments
VSANs are virtual fabrics
SAN islands are virtualized onto a
common SAN infrastructure
Allocate ports within a physical fabric to Independent Physical SAN
Islands Are Virtualized onto a
create virtual fabrics Common SAN Infrastructure
Fabric disruption is limited to VSAN
Traffic statistics are gathered per VSAN
Features include:
Dynamic provisioning and resizing
Improved port utilization
Shared ISL bandwidth
BRKSAN-2701
Virtual SANs—VSANs
Production SAN Tape SAN Test SAN
FC
FC FC
FC
FC
SAN C SAN D SAN E SAN F

SAN B DomainID=3 DomainID=4 DomainID=5 Domain ID=6
SAN A
DomainID=1 DomainID=2
DomainID=7 DomainID=8
BRKSAN-2701

Presentation_ID.scr
Zoning
Zoneset
Devices within a zone can access
each other Zone A
Zoning establishes access control FC
FC Zone B
Limiting access prevents unauthorized access FC
FC
FC
FC
Soft zoning FC
Implemented in switch software and FC FC

enforced by name server FC
FC FC FC
Name server responds to discovery FC
queries with only devices found in Zone C FC
requestor’s zone or zones

“Soft zoning” used to be synonymous
with “WWN zoning”
Hard zoning Default Zone
Enforced by ACLs in port ASIC FC FC
FC FC
Applied to all data path traffic FC
FC
FC
FC
“Hard zoning” used to be synonymous FC
FC
with “port zoning” FC

FC
WWN zoning in hardware FC
Both at source and destination ports Devices Not Assigned to a Zone

Enhanced zoning Belong to the Default Zone
BRKSAN-2701
Zoning and VSANs

Hierarchical relationship Zones and VSANs Are Complementary
First assign physical ports to VSANs Physical Topology
Then configure independent VSAN 2 Active Zoneset A
zones per VSAN ZoneA
VSANs only change when ports Host1 Disk1
Disk3 Disk2
needed per virtual fabric
ZoneC
Zones can change frequently (e.g., backup) ZoneB
Default Host2
Zones provide added security and Zone Disk4
allow sharing of device ports

VSAN 7 Active Zoneset D
Zone membership is configured by:
Host4
Port World Wide Name (pWWN)—device
ZoneD
Fabric World Wide Name (fWWN)—fabric Default
Zone ZoneA
Disk5
Fibre Channel Identifier (FCID) Host3
Disk6
Fibre Channel Alias (FC_Alias)
IP address
Domain ID/port number One Active Zoneset per VSAN
Interface
BRKSAN-2701

Presentation_ID.scr
physical switch
VSAN—Design foundation
Zoning
Fabric routing (Inter-VSAN Routing—IVR)
Virtual fabric trunking (VSAN Trunking)
IVR zones
BRKSAN-2701
VSANs—Routed Connectivity
Common Physical Fabric
Sharing a Common
Resource such as Tape
Common Physical Fabric
Sales
SAN
HR
SAN
MS Sales Marketing
MS SAN
SAN MS
Marketing
SAN
SAN IP
MS Tape Production and or
SAN Extension
HR DR Interconnect Without Services FC
SAN Merging Fabrics
Tape Media Server Potentially Separate

MS Administrators per SAN
BRKSAN-2701

Presentation_ID.scr
Inter-VSAN Routing—IVR
Enables devices in different VSANs to communicate
Allows selective routing between specific members of two or
more VSANs
Traffic flow between selective devices
Resource sharing, i.e., tape libraries and disks
VSAN 10 IVR Zone VSAN 20

Media Server Tape Library
FC FC
FC
FC
FC FC
FC
FC
FC FC
FC
FC FC
FC FC
FC FC
FC
FC FC
FC
FC FC
FC FC FC FC
FC
FC FC FC
FC FC
BRKSAN-2701
IVR Zones
IVR zone Physical Topology
A container or access VSAN 2

control, containing two or Disk2 Disk3
more devices in different ZoneA Host1

Disk1
VSANs ZoneC
Standard zones are still Disk4 Host2

ZoneB
used to provide intra-
VSAN access
VSAN 3
IVR Zoneset Host4
ZoneD
A collection of IVR zones ZoneA
Disk5
that must be activated to be Host3
operational Disk6
Inter-VSAN Zone
BRKSAN-2701

Presentation_ID.scr
Port Channels
Port Aggregation Feature Used to Create a
Single Logical ISL from 1–16 Physical ISLs
Increases bandwidth and
availability
Very granular load balancing—
per exchange/src/dst or
per src/dst (policy on a per
VSAN basis)
Interfaces can both be
added and removed in a
nondisruptive manner in
4-Link Port Channel
production environments
EISL
Preserved FC guarantee
of in-order delivery (IOD)
BRKSAN-2701
Port Channel Protocol (PCP)

Exchange-based, in-order
load balancing
Mode 1: based on src/dst
FC_ID/OX_ID/RX_ID
Mode 2: based on src/dst FC_ID
Consistently detect misconfiguration Up to 160 Gbps
Port Channel
Transition misconfigured ports to with HA
isolated state so as to be able to
correct the misconfig
Synchronize bring up of ports in a
channel across peer switches
Provide the ability for the system to
automatically create Port Channels
among compatible ports
BRKSAN-2701

Presentation_ID.scr
VSANs, Trunking, Port Channels
Hierarchical relationship VSAN
10
Metric
100
VSAN Metric
10 50
Port Channels provide link aggregation to 20 50
yield virtual ISL (E_Port)
Single-link ISL or Port Channel ISL
can be configured to become EISL—
(TE_Port)
VSANs can be selective grafted or
pruned from EISL trunks 8 Gbps
Port Channel
All member links of a port channel p
Trunking
VS On
20 cku
must have same configuration prior E_Port
AN ly
E_Port AN Ba
VS 10
10
to creating channel (e.g., TE_Port (TE_Port) AN
VS
or E_Port, VSANs-enabled, etc.)
Port Channel technology provides Trunking E_Port
E_Port
high availability and fast recovery (TE_Port)
for VSAN trunk (EISL)
4-Link (8 Gbps) Port
Multiple Port Channels yield Channel Configured
multiple paths for custom traffic as EISL
engineering
BRKSAN-2701
IOD: In-Order Delivery—

Frames Are Delivered In-Order
Frames of a flow (SID,DID) are
received in the order they were sent
Some applications (FICON and HP CA Frames Left in Output
Queues/Buffers at Peer Switch
EVA) can not tolerate out of order frames
MDS HW always keeps frames Frames Left in Output 5 2
within same exchange on same Queues/Buffers 6 1
at Local Switch
path except during
FSPF route changes
4 3
Port Channel (PC) membership changes
7 8
With IOD the frame waits for 500ms
(switch latency value where frames
are dropped) to guarantee that all
frames have been drained from
Recovery Path Could Carry Frames
all the VOQs in the peer switch from Same Exchange Out-of-Order
Once the peer has performed
this operation, it responds with
an acknowledgement and traffic
forwarding can begin immediately
BRKSAN-2701

Presentation_ID.scr
Virtual Output Queuing (VOQ)
B C A C C C C C A B Input Output FC
A
Port Port
Input Output B
Port Port
Switch with no VOQ
HOL blocking FC
Input Output C
port Port
--------------------------------------------------------------------------------------------------------------------------------------------------------------
C C C C
Input Output FC
B C A C C C C C A B B Port Port A
A ARB
Input Output B
Port Port
Switch with VOQ support
FC
No HOL blocking
Input Output C
VOQ alleviates HOL Port Port
BRKSAN-2701
SAN Design Principles

and Considerations
BRKSAN-2701

Presentation_ID.scr
Agenda
Brief SAN Technology Overview

Zoning, Virtual SAN (VSAN)
Port Channels, IOD
SAN Design Principles and Considerations

Design Factors
Design Types
Design Optimization
SAN Security Design Considerations
Interoperability Design Considerations
BRKSAN-2701
SAN Design—Principles and

Considerations
Determine components to
Data Center Servers
be used and how they will
fit into your overall strategy
Creation of technical
infrastructure and how
the pieces will fit together Storage
Network
Determine how existing
infrastructure and new one
will be integrated
Creation of the processes Storage
and procedures that will Disk and Tape
guide personnel in how the

infrastructure is to be used
BRKSAN-2701

Presentation_ID.scr
Design Factors
BRKSAN-2701
Early SAN Designs

SAN Islands
1. First SANs hardly qualified Low Traffic
Across ISLs
as networks
4 Host
SAN islands of two to four switches Host Host
Fixed 8–16 port switches limited SAN growth 1 Few Switches
2. No fabric segmentation such

as VSANs
Host
3. Limited enhancements to FSPF
No Port Channeling
No equal cost load balancing
4. Traffic management was Switches

Managed
not needed Separately 3
No QoS because bandwidth 5 Single

was over provisioned Routes
5. Management tools focus Faults Impact

all Devices
on element management Host
Host Host Host Host
2
Not network management
BRKSAN-2701

Presentation_ID.scr
SAN Major Design Factors
1. Port density
High
How many now, how many Performance
Crossbar
later? topology Large Port
2
Count
2. Network performance QoS,
Directors
Congestion
What is acceptable? Control, 4
Unavoidable? Reduce FSPF 1
Routes
3. Traffic management 8 8 8 8 8 8 8 8 8 8 8 8
Preferential routing or
resource allocation
4. Fault isolation
Host Host Host Host Host Host
Consolidation while
maintaining isolation 3
Failure of One Device Has
5. Management No Impact on Others
Secure, simplified management

BRKSAN-2701
1. Scalability—Port Density,
Topology Requirements
Number of ports for end devices
High
How many ports are Performance
needed now? Crossbar
Large Port
2
Count
What is the expected Directors
life of the SAN? QoS,
Congestion
Control, 4
How many will be needed in Reduce FSPF 1
the future? Routes
8 8 8 8 8 8 8 8 8 8 8 8
Hierarchical SAN design
Best Practice
Design to cater for future
requirements Host Host Host Host Host Host
Doesn’t imply “build it all now,” 3

but means “cater for it” and Failure of One Device Has
avoids costly retrofits tomorrow No Impact on Others
BRKSAN-2701

Presentation_ID.scr
2. Network Performance—
Oversubscription Design Considerations
All SAN Designs Have Some
Degree of Oversubscription Disk Oversubscription
Disk Do Not Sustain
Without oversubscription, Wire-Rate I/O with
‘Realistic’ I/O Mixtures Tape Oversubscription Low
Sustained I/O Rates
SANs would be too costly Most Major Vendors
LTO-2 Native Transfer
Promote 12:1 host:disk
Fan-Out Rate ~ 60 MBps
Oversubscription is introduced
70 MBps 60 MBps
at multiple points Max/Port Max/Port
(Common) (Common)
Switches are rarely the bottleneck
in SAN implementations Typical
Oversubscription
Port Channels
Help Reduce
in Two-Tier Design Oversubscription
Device capabilities (peak and Can Approach 8:1,
Sometimes Even
While Maintaining
HA Requirements
sustained) must be considered Higher
along with network 7:1 O.S.

(Common)
oversubscription
Must consider oversubscription
during a network failure event
40 MBps Host Oversubscription
Remember, all traffic flows Max/HBA

(Common)
Most Hosts Suffer from PCI Bus
Limitations, OS, and Application
Limitations Thereby Limiting Maximum
towards targets—main Host I/O and Bandwidth Rate
bottlenecks
BRKSAN-2701
3. Traffic Management
Do different apps/servers
High
have different performance Performance
requirements? Crossbar
Large Port
2
Count
Should bandwidth be Directors
QoS,
reserved for specific Congestion 4
applications? Control,
Reduce FSPF 1
Is preferential treatment/ Routes
QoS necessary? 8 8 8 8 8 8 8 8 8 8 8 8
Given two alternate paths

for traffic between data
centers, should traffic use Host Host Host Host Host Host
one path in preference to
the other? 3
Failure of One Device Has
Preferential routes No Impact on Others
BRKSAN-2701

Presentation_ID.scr
4. Fault Isolation
Consolidation of Storage into
a Single Fabric = Increased
Storage Utilization + Reduced
Administration Overhead
Major Drawback Is That Faults
Are No Longer Isolated Physical SAN Islands Are
Virtualized onto Common
SAN Infrastructure
Technologies such as VSANs
enable consolidation and
scalability while maintaining
security and stability
Fabric
VSANs constrain fault impacts #3
Fabric
Faults in one virtual fabric (VSAN) #1 Fabric
#2
are contained and do not impact
other virtual fabrics
BRKSAN-2701
5. Management
Consolidation and Large Scale SAN Becomes
More Difficult to Manage. How Can It Be More
Secure? How Can the SAN Traffic Be Monitored
as Performance Requirements Increase?
FM Client FMS FM Clients
FC
SNMP FC
SNMP
SNMP
Tools and Features
RBAC on per VSAN basis
FC-SP for switch-to-switch or device-to-switch security
Fabric Manager Server
Online traffic monitoring through Device Manager
BRKSAN-2701

Presentation_ID.scr
Design Types
BRKSAN-2701
Core-Edge
Traditional SAN design for
growing SANs
High density directors in
core and fabric switches, A B
directors or blade switches
on edge
Predictable performance
A B
A B
Scalable growth up to core A B
and ISL capacity
BRKSAN-2701

Presentation_ID.scr
Large Core-Edge Example
Large Core/Edge
(2240 Usable Ports per Fabric) “A” Fabric Shown,
Repeat for “B” Fabric
Traditional Core-Edge Design
Is Ideal for Centralized
Services and Consistent
Host-Disk Performance 128 Storage
Regardless of Location; Ports at 2 G
Less Future Scalability 64 ISL to
Edge at 4 G
Requires third tier to scale
Port bandwidth
reservations guarantee 32 ISL to
Core at 4 G
performance for ISLs
496 Host
(edge side) Ports at 4 G
Ports Deployed: 2520

Used Ports: 2512
Storage Ports (2 G Dedicated): 256
Host Ports (4 G Shared): 1984
ISL Oversubscription (Ports): 15.5 : 1
Disk Oversubscription (Ports): 15.5 : 1
BRKSAN-2701
Scaling Traditional Core-Edge

with Three Tiers
Could Scale to ~ 40k Ports*
12 ISL to
Core at 4 G
124 Storage
Ports at 2 G
6 ISL to
Backbone at 4 G
64 ISL to
Edge at 4G
32 ISL to
Core at 4 G
496 Host
Ports at 4 G
15.5 : 1 Host
to Core
* No software testing has been done to this level of scalability

BRKSAN-2701

Presentation_ID.scr
Top of Rack Example—
9134 and 10 G ISLs
Top of Rack
(1088 Usable Ports per Fabric)
Top of Rack Design Utilizing
10 Gb ISLs; High Bandwidth
ISLs Provides Ample A B
Performance and 96 Storage
Reduces Cabling Ports at 2 G
28 ISL to Edge
at 10 G
2 ISL to
Core at 10 G
32 Host Ports
at 4 G
A
B
Used Ports: 1200
Storage Ports (2 G Dedicated): 192
14 Racks
Host Ports (4 G Shared): 896
32 Dual
ISL Oversubscription (Ports): 6.4 : 1 Attached
Disk Oversubscription (Ports): 9.3 : 1 Servers per
Rack
BRKSAN-2701
Blade Centers—4G ISLs

Top of Rack
(11200 Usable Ports per Fabric)
Blade Server Design Using 2
x 4 G ISL per Blade Switch;
Oversubscription Can Be A B
Reduced for Individual Blade 120 Storage
Centers by Adding Additional Ports at 2 G
ISLs as Needed 60 ISL to Edge
at 4 G
A B
2 ISL to
Core at 4G
16 Host
Used Ports: 1440 Ports at 4G Five Racks
Storage Ports (2 G Dedicated): 240 96 Dual
Host Ports (4 G Shared): 960 Attached
ISL Oversubscription (Ports): 8: 1 Blade Servers
Disk Oversubscription (Ports): 8: 1
per Rack
BRKSAN-2701

Presentation_ID.scr
Collapsed Core/Edge Design
Traditional Cisco MDS 9500 Director with
Core-Edge Design Collapsed-Core Configuration
Full Performance
(Non-Oversubscribed,
Non-Blocking)
Host Optimized
(Oversubscribed,
Non-Blocking)
Collapsed Core
Typically a lower oversubscription ratio
Room to grow—empty slots = future port count growth
While Director ports are more expensive than Fabric switch ports,
Collapsed Core design has no wasted ports for ISLs—similar
cost/usable port
BRKSAN-2701
Medium Scale Dual Fabric—

Collapsed Core Design
Dual Director Switches 96
(Up to 528 Ports per Fabric) Storage
Ports
Medium Scale Design Leveraging ..
48-Port Modules with Port
Bandwidth Reservations to
11 X 48-Port Modules
Provide High-Density, 528 Ports Total
Cost-Effective Solution 48 Ports for Storage
480 Ports for Hosts
VSAN support 10:1 Oversubscription
Port bandwidth reservations

guarantee performance for
those devices that require it 960 Host Ports
Port Channels with HA to other
switches for future growth,
scaling from a collapsed core
to core/edge design
Storage
Ports Deployed: 528 Within Each (1x or 2x Dedicated)
Usable Ports: 528 Port Group:
One or Two Ports
Usable (Available) Ports: 0
to Storage
Design Efficiency: 100% 11 or 10 Ports
End-to-End Oversubscription 10 : 1 to Hosts
(480 : 48) Port Bandwidth Hosts
Reservations Used (11x or 10x Shared)
BRKSAN-2701

Presentation_ID.scr
Top-of-Rack Consolidated I/O
I/O Consolidation at Access
LAN Core
SAN-A SAN-B
Distribution 8
MDS 9500
POD 1 POD N Access

Nexus 5000
10 GE/FCoE
CNA
Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N
BRKSAN-2701
iSCSI Design
Take advantage of what IP (IPv4, IPv6) and IPS have to offer
Low cost with many options
Proxy initiator
iSCSI Server Load Balancing (iSLB)
Initiator Configured to See
Targets at Virtual Address MDS9509-1
Real GigE Address
IP: 10.0.0.101 Storage Array
MAC: 0005.3000.aabf
IP Network
FC SAN
pWWN a
Virtual Address
10.1.40.163 IP: 10.1.10.100
iSCSI
MAC:
0000.5e00.0101
pWWN b
Real GigE Address

IP: 10.0.0.102
MAC: 000e.38c5.b82d Primary/Secondary pwwn
Initiator with Alias and Trespassing
NIC Teaming MDS9509-2
BRKSAN-2701

Presentation_ID.scr
SAN Extension—Design Considerations
Transport Primary Data Center
Optical or IP WAN/MAN
High availability
Application availability, IVR
Optimal performance: latency and
throughput
Application performance, tape and write
acceleration WAN/MAN
Resilience to WAN problems
WAN bandwidth: optimal use and
lowest cost
WAN bandwidth utilization, hardware compression
QoS: maintain and assure service
Traffic management, IVR, QoX, TCP tuning, IPv6
Data security in transit Backup Data Center
FCIP encryption and FC-SP auth
BRKSAN-2701
Design Optimization
BRKSAN-2701

Presentation_ID.scr
Blocking—Impact on
Design Performance
Performance can be adversely affected across an

entire multiswitch FC Fabric by a single blocking port
HOL is a transitory event (until some BB_Credits are returned
on the blocked port)
To help alleviate the blocking problem and enhance

the design performance
Virtual Output Queuing (VoQ) on all ports
Deep Buffers—255 BB_Credits per port
Fibre Channel Congestion-Control (FCC)—Detects congested
ports and throttles the port causing the congestion at its origin
BRKSAN-2701
Advanced Traffic Management

Department/ Department/
Port bandwidth reservation Customer ‘A’ Customer ‘B’
(Low Priority) (High Priority)
Dedicated mode—ports can act at
any dedicated rate including line rate
Shared mode
Enhance utilization
Oversubscription VSAN-
Enabled
Round robin fairness Fabric
Assured fairness
Potential VSAN
Port Channel to scale Bottlenecks Trunks
connectivity
Bundle ISLs between switches
Additional resiliency
In-order frame delivery

guarantee (IOD) Shared Storage
BRKSAN-2701

Presentation_ID.scr
Advanced Traffic Management
Department/ Department/
QoS allows traffic to be Customer ‘A’ Customer ‘B’
(Low Priority) (High Priority)
intelligently managed
Minimizes impact of
oversubscription
Allows more economical
topologies
Prioritize traffic by flow VSAN-
Enabled
VOQ for switch performance Fabric
Slow flows to not disrupt fast flows Potential VSAN

Bottlenecks Trunks
Non-blocking frame forwarding
FCC mechanism can throttle

back traffic at its origin
Manage congestion on ISLs
Shared Storage
BRKSAN-2701
Enhanced Quality of Service (QoS)

Arbiter-aware QoS Enabled within a switch or
Requires Supervisor 2 across the network
Allows QoS in a single switch
No longer requires FCC to configuration
operate
User definable DiffServ
DWRR Weight
Priority Absolute
Queue 2 60
Queue 3 10
Queue 4 30
PQ Transmit
Queue
DWRR 2
DWRR 3
DWRR 4
BRKSAN-2701

Presentation_ID.scr
Enhancing SAN Design
Extending the effective Increased distance
distance for remote Improved performance
applications
SAN acceleration—Local Ack
Write acceleration
Tape acceleration
SAN Extension with FC-WA
Reduces WAN-induced latency
Improves application performance
over distance Media
Servers
Backup design
performance enhancement SSM Application
SAN Servers
Serverless backup—NASB
Offloading the I/O and CPU work
from Media Server Tape Disk
BRKSAN-2701
N-Port Virtualizer (NPV)

Enabling Large-Scale Blade Server Deployments
Deployment Model—FC Switch Mode Deployment Model—HBA Mode
Blade System Blade System Blade Switch Blade System Blade System
Configured as
NPV (i.e.,
Blade N
Blade N
Blade N
Blade N
Blade 2
Blade 2
Blade 1
Blade 2
Blade 1
Blade 2
Blade 1
Blade 1
… HBA Mode)
…
FC Switch FC Switch NPV NPV
E-Port
NPV Enables Large Scale
N-Port
Blade Server Deployments By:
SAN Reducing Domain ID usage SAN
E-Port
F-Port
Addressing switch interop issues
Simplifying management
Storage Storage
Blade Switch Attribute

FC Switch Mode (E-Port) Deployment Model HBA Mode (N-Port)
None
One per FC Blade Switch # of Domain IDs Used
(Uses Domain ID of Core Switch
Interoperability Issues with
Yes No
Multivendor Core SAN Switch
Level of Management Coordination
Medium Between Server and SAN Low
Administrators
BRKSAN-2701

Presentation_ID.scr
SAN Security
BRKSAN-2701
SAN Design Security Challenges

SAN design security is often overlooked as an area of concern
Application integrity and security is addressed, but not back-end storage network carrying actual data
SAN extension solutions now push SANs outside datacenter boundaries
Not all compromises are intentional FC
Accidental breaches can still have the same consequences
SAN design security is only one part of complete data center solution
Host access security—one-time passwords, auditing, VPNs
Storage security—data-at-rest encryption, LUN security
Data center physical security
Theft
Privilege Escalation/
External Dos
Unintended Privilege
or Other Data
Unauthorized
Intrusion Tampering
Application Connections
FC Tampering (Internal)
FC
FC (Trojans, etc.)
FC
FC
FC
FC
FC SAN FC
FC
FC
FC
LAN
BRKSAN-2701

Presentation_ID.scr
SAN Security—Design Considerations
Data Integrity and Secrecy
Protecting data
SAN Fabric Protocol
Data integrity and encryption Security
in transit or at rest
Securing against
unauthorized user and
device access
Target
User/device authorization SAN Fabric
and authentication Host
Server and target

access controls
Target Access
Guarding against Security
malicious management Fabric Access
Security
misconfiguration iSCSI
IP Storage
Management access controls Security
SAN Management (iSCSI/FCIP)
Securing the SAN
Security
management information
BRKSAN-2701
Intelligent Fabric
Applications
BRKSAN-2701

Presentation_ID.scr
Intelligent Storage Applications
Delivered as a Transparent Fabric Service
Application Servers Extend storage services to

any device in the SAN
Transparent to applications
Nondisruptive deployment
No SAN reconfiguration
MSM-18/4 MSM-18/4
No rewiring to insert appliances
Highly scalable performance

Automatic load balance
Reliable, highly available
service
Wizard-based provisioning
Storage Array
BRKSAN-2701
Cisco Storage Media Encryption (SME)
Application
Encrypts storage media
Server (data at rest)
IEEE compliant AES-256 encryption
Name: XYZ
SSN: 1234567890
Amount: $123,456 Integrated as transparent fabric service
Status: Gold
Transparent Fabric Service
Key Management
Center Supports heterogeneous storage
SME SME IP arrays, tape devices, and VTLs
Compresses tape data
Offers secure, comprehensive key
@!$%!%!%!%%^&
management
*&^%$#&%$#$%*!^
@*%$*^^^^%$@*)
%#*@(*$%%%%#@
Allows offline media recovery
Built upon FIPS Level 3 system
Storage Tape architecture
Array Library
BRKSAN-2701

Presentation_ID.scr
Cisco Data Mobility Manager (DMM)
Application Servers Migrates data between storage

arrays for
Technology refreshes
Workload balancing
Storage consolidation
DMM offers
Online migration of heterogeneous arrays
Data Mobility Manager
Simultaneous migration of multiple LUNs
Unequal size LUN migration
Application Data Rate adjusted migration
I/O Migration
Verification of migrated data
Secure erase
Dual fabric support
Old Array New Array CLI and wizard-based management
with Cisco Fabric Manager
Utilizes Storage Services
Modules (SSM) Requires no SAN reconfiguration or
rewiring
BRKSAN-2701
Network Assisted Storage Applications

SAN Tap
Enables appliance-based
Initiator storage applications without
compromising SAN integrity
Initiator ÅÆ Target I/O About SAN Tap
SAN MDS delivers a copy of primary
I/O to an appliance
Appliance provides the storage
Copy of Appliance
Primary application
I/O
Examples of applications include
Continuous Data Protection
(CDP), replication, etc.
Target Key customer benefits

Preserve integrity, availability, and
performance of primary I/O
No service disruption
= SAN Tap
Investment protection
BRKSAN-2701

Presentation_ID.scr
Storage Virtualization for
Cisco MDS 9000
Independent
Control control path Services
Processor Program the Volume management
data path Data migration
Processes Point-in-time copies
exceptions
FAIS
High-performance fast path
SSM Integrated, HA architecture
Multiprotocol integration
Data Path
Comprehensive security
Troubleshooting and diags
Virtual to Physical Mapping
Data Traffic Control Traffic

BRKSAN-2701
SecureErase on Offline Storage Arrays

Configuration
Deploy a SSM
Create VIs
Zone VIs and target ports
Expose target LUNS to VI
Create and start SE sessions
Virtual Initiators on SSM Key Benefits

Cisco Fabric
Mgr
In-place SecureErase
FC FC No window of vulnerability
Vendor agnostic
Heterogeneous Storage Arrays
Compliance
DoD, RCMP, and Gutmann
Easy to deploy and cost effective

BRKSAN-2701

Presentation_ID.scr
Interoperability
BRKSAN-2701
Standard Fibre Channel Interoperability

Switch Interoperability Is Available Between the Cisco MDS
Platforms and Non-Cisco ‘Standards Compliant’ Switches
Provides way to redeploy smaller
edge switches
Both McDATA and Brocade switches
must be in ‘interoperability mode’
On both products this results in TE E
the loss of some functionality
No trunking
No port-based zoning Non-Cisco Switch in
No full zone-set exchanges ‘Interoperability’ Mode
Restricted # of domains and their ID
Cisco Brocade McDATA
Must also choose fabric timer
values that are the same and 5000- 4000- 1000-
RA_T_O_V
available across all vendors— 100000 ms 120000 ms 120000 ms
must be the same fabric-wide 1000- 1000- 200-
ED_T_O_V
DS_TOV ms 20000 ms 60000 ms
Must enable ‘interoperability’
mode on MDS 9000 v2.4.1 and v04.01.00 12
v3.0.1a
No loss of functionality on MDS 9000
Enabled per-VSAN—‘Interop mode 1’
BRKSAN-2701

Presentation_ID.scr
Third-Party Switch Native Mode
Enables MDS9000 family to
Cisco MDS 9000 Family
interoperate with legacy fabric
switches in ‘native mode’—
Brocade and McDATA
(SANOS3.1)
Reuse existing legacy fabric switches
as edge devices
No impairment to Cisco fabric—
all advanced services available VSAN 50 VSAN 40
No change required on legacy Legacy Brocade Legacy McDATA

switches—simply connect switches Switches Switches
Three additional modes Configurable on a VSAN-by-

Interop mode 2 VSAN Basis on MDS 9000
Interop mode 3
Interop mode 4
BRKSAN-2701
Closing Remarks
BRKSAN-2701

Presentation_ID.scr
Closing Remarks—SAN Design
Simple SAN design
Small port count
No interswitch links
Dual fabric
Minimal SAN/OS feature use
Scalable SAN design

Core/edge
Collapsed core/edge
Director class collapsed CORE SAN

Large port count
High availability design
Port Channels, FSPF routing
Multi-SAN/OS feature use
Core director SAN with blade server and edge switches

Multidomains
Interop
BRKSAN-2701
BRKSAN-2704: SAN Extension

BRKSAN-2719: From Storage Component Virtualization
to End-to-End Virtualization
BRKSAN-2705: iSCSI SAN Design and Operation
BRKSAN-2702: SAN Migration and Interoperability
BRKSAN-3707: Advanced SAN Design
BRKSAN-2701

Presentation_ID.scr
Additional Information
Cisco Storage Networking

http://www.cisco.com/go/storagenetworking
Cisco Data Center Networking

http://www.cisco.com/go/datacenter
Storage Network Industry Association (SNIA)

http://www.snia.org
Internet Engineering Task Force—IP Storage

http://www.ietf.org/html.charters/ips-charter.html
ANSI T11—Fibre Channel

http://www.t11.org/index.htm
BRKSAN-2701
Q and A
BRKSAN-2701

Presentation_ID.scr
Recommended Reading
Continue your Cisco Live

learning experience with further
reading from Cisco Press
Check the Recommended
Reading flyer for suggested
books
Available Onsite at the Cisco Company Store

BRKSAN-2701
Complete Your Online

Session Evaluation
Give us your feedback and you could win Don’t forget to activate
fabulous prizes. Winners announced daily. your Cisco Live virtual
account for access to
Receive 20 Passport points for each session all session material
evaluation you complete. on-demand and return
for our live virtual event
Complete your session evaluation online now in October 2008.
(open a browser through our wireless network Go to the Collaboration
to access our portal) or visit one of the Internet Zone in World of
stations throughout the Convention Center. Solutions or visit
www.cisco-live.com.
BRKSAN-2701

Presentation_ID.scr
BRKSAN-2701

Presentation_ID.scr

Fibre Channel Storage Area Network Design: BRKSAN-2701

Загружено:

Сведения о документе

Исходное описание:

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Fibre Channel Storage Area Network Design: BRKSAN-2701

Загружено:

Авторское право:

Доступные форматы

BRKSAN-2701

Fibre Channel Storage

© 2006, Cisco Systems, Inc. All rights reserved. 1

 Brief SAN Technology Overview

 SAN Design Principles and Considerations

© 2006, Cisco Systems, Inc. All rights reserved. 2

 Fibre Channel Protocol

 Virtual SAN (VSAN), Zoning

Fibre Channel Communications

© 2006, Cisco Systems, Inc. All rights reserved. 3

SCSI Applications (File SCSI Applications (File

SCSI Block Commands SCSI Stream

SCSI Commands, Data, SCSI Commands, Data,

Parallel SCSI Fibre Channel Protocol

Fibre Channel Port Types

G_Port F_Port N_Port Node

G_Port F_Port N_Port Node

G_Port F_Port N_Port Node

© 2006, Cisco Systems, Inc. All rights reserved. 4

 The interconnection between switches is called the ISL

 Supports all classes of service

 FC-PH permits consecutive frames of a sequence to be routed

Buffer to Buffer Credit Flow Control

 BB_Credits are used to ensure enough FC frames in flight

© 2006, Cisco Systems, Inc. All rights reserved. 5

 FSPF stands for Fabric Shortest Path First

FC_ID Address Model

Switch Topology Switch

Private Loop Device Arbitrated Loop

© 2006, Cisco Systems, Inc. All rights reserved. 6

SEQ_ID Sequence Sequence Sequence

SEQ_CNT Frame Frame Frame

Fibre Channel Timers

© 2006, Cisco Systems, Inc. All rights reserved. 7

VSANs, Zoning, IVR Zones

© 2006, Cisco Systems, Inc. All rights reserved. 8

SAN C SAN D SAN E SAN F

© 2006, Cisco Systems, Inc. All rights reserved. 9

Implemented in switch software and FC FC

queries with only devices found in Zone C FC

requestor’s zone or zones

with “port zoning” FC

 WWN zoning in hardware FC

Both at source and destination ports Devices Not Assigned to a Zone

Zoning and VSANs

allow sharing of device ports

© 2006, Cisco Systems, Inc. All rights reserved. 10

Tape Media Server Potentially Separate

© 2006, Cisco Systems, Inc. All rights reserved. 11

 Resource sharing, i.e., tape libraries and disks

VSAN 10 IVR Zone VSAN 20

A container or access VSAN 2

more devices in different ZoneA Host1

 Standard zones are still Disk4 Host2

 IVR Zoneset Host4

© 2006, Cisco Systems, Inc. All rights reserved. 12

Port Channel Protocol (PCP)

© 2006, Cisco Systems, Inc. All rights reserved. 13

IOD: In-Order Delivery—

© 2006, Cisco Systems, Inc. All rights reserved. 14

SAN Design Principles

© 2006, Cisco Systems, Inc. All rights reserved. 15

Brief SAN Technology Overview

SAN Design Principles and Considerations

Fibre Channel Protocol

Virtual SAN (VSAN), Zoning

The interconnection between switches is called the ISL

Supports all classes of service

FC-PH permits consecutive frames of a sequence to be routed

BB_Credits are used to ensure enough FC frames in flight

FSPF stands for Fabric Shortest Path First

WWN zoning in hardware FC

Resource sharing, i.e., tape libraries and disks

Standard zones are still Disk4 Host2

IVR Zoneset Host4

Brief SAN Technology Overview

SAN Design Principles and Considerations

Doesn’t imply “build it all now,” 3

Remember, all traffic flows Max/HBA

Given two alternate paths

Port bandwidth reservations

Performance can be adversely affected across an

To help alleviate the blocking problem and enhance

In-order frame delivery

FCC mechanism can throttle

Application Servers Extend storage services to

Highly scalable performance

Application Servers Migrates data between storage

Target Key customer benefits

Easy to deploy and cost effective

Three additional modes Configurable on a VSAN-by-

Scalable SAN design