Вы находитесь на странице: 1из 53

Cisco Training

• Speaker : Rahul Nadem/ Vignesh Shanbhag


• Date : 13th July 2012
Hardware Overview
Cisco MDS 9500 Series multilayer directors
− Cisco MDS 9513 multilayer director
− Cisco MDS 9509 multilayer director
− Cisco MDS 9506 multilayer director

Cisco MDS 9200 Series fabric switches


− Cisco MDS 9222i multilayer fabric switch
− Cisco MDS 9216i multilayer fabric switch

Cisco MDS 9100 Series fixed configuration fabric switches


− Cisco MDS 9134 multilayer switch
− Cisco MDS 9124 multilayer switch
− Cisco Fabric Switch for HP c-Class BladeSystem
− Cisco Fabric Switch for IBM BladeCenter
Cisco MDS 9500 Series Multilayer Directors
• Cisco MDS 9513 Director
− 13 slots
− slot 7 and slot 8 reserved Supervisor-2 modules
− 11 hot-pluggable switching or services modules.

• Cisco MDS 9509 Director


− 9 slots
− slot 5 and slot 6 reserved Supervisor-2 and Supervisor-1 modules
− 7 hot-pluggable switching or services modules
• Cisco MDS 9506 Director
− 6 slots
− slot 5 and slot 6 reserved Supervisor-2 and Supervisor-1 modules
− 4 hot-pluggable switching or services modules

• Management Access
− Gen-1 Modules - 10/100BASE-T Ethernet port + RS-232 serial port.
− Gen-2 Modules - 10/100/1000BASE-T Ethernet port + RS-232 serial port
− Additional USB port on Gen-2
Cisco MDS 9200
• The Cisco MDS 9200 Series includes the following
multilayer switches supporting multiprotocol capabilities

− Cisco MDS 9216i


• 2 Slots
• One Reserved for integrated Supervisor Module and other for switching or
services modules
• Supervisor Module - supervisor functions + 14 standard FC ports and two
multiprotocol ports
• Multiprotocol ports - support FCIP and iSCSI protocols simultaneously.

− Cisco MDS 9222i


• 2 Slots
• One Reserved for integrated Supervisor Module and other for switching or
services modules
• Supervisor Module - supervisor functions + 16 standard FC ports
Cisco MDS 9100 Fixed Configuration
• Cisco MDS 9134 with 34 ports
− 24-port base with 8-port license for growth
− two 10 Gbps ports can be activated independently in 24-port or 32-port configurations.
• On-demand port activation licensing
• Non disruptive upgrades

• Cisco MDS 9124 with 24 ports


− 8 base ports
− go upto 8 or 16 additional ports
• On-demand port activation licensing
• Non-disruptive upgrades

• Cisco Fabric Switch for HP c-Class BladeSystem (24 ports; 16 internal 2/4 Gbps, and 8 full-
rate ports)

• Cisco Fabric Switch for IBM BladeCenter (20 ports; 14 internal 2/4 Gbps, and 6 external full-
rate ports)

These fixed configuration switches are packaged in 1 RU enclosures and provide 1-Gbps, 2-
Gbps, 4-Gbps, or 10 Gbps autosensing Fibre Channel ports. Besides Telnet access, a
10/100BASE-T Ethernet port provides switch access.
Cisco MDS Modules
Modules 9500 Series 9216i 9222i
48-port 8-Gbps X
24-port 8-Gbps X
4/44-port 8-Gbps Host Optimized X X
48-port 4-Gbps X X X
24-port 4-Gbps X X X
12-port 4-Gbps X X X
4-port 10-Gbps X X X
32-port 2-Gbps * X X
18/4-port Multiservice module (MSM-18/4) X X
18/4-port Multiservice module FIPS X X
18-port 4-Gbps X
16-port 2-Gbps * X

14/2-port Multiprotocol Services (MPS-14/2) module X X

32 Storage Services Module (SSM) * X X X


8-port IP Storage Services (IPS-8) module X X
4-port IP Storage Services (IPS-4) module X

* - Gen1 Cards
9500 Supervisor Modules
Supervisor-1 Supervisor-2 Supervisor-2A

intelligent SAN services,


non disruptive software upgrades
stateful process restart and Supported Supported Supported
failover,
fully redundant

Port in Single Chassis 252 528 528

ports in a single rack 756 1584 1584

1.4 Tbps in 9509 / 9506 Chassis 1.4 Tbps in 9509 / 9506 Chassis
1.4 Tbps in 9509 /9506 Chassis 700 Gbps per Supervisor-2 Module 700 Gbps per Supervisor-2 Module
Switching Bandwidth
700 Gbps per Supervisor-1 Module 2.2 Tbps in 9513 Chassis 2.2 Tbps in 9513 Chassis
1.1 Tbps per Crossbar Module 1.1 Tbps per Crossbar Module

Port Density 1, 2, 4, and 10-Gbps 1, 2, 4, and 10-Gbps 1, 2, 4, 8, and 10-Gbps


(1) RS-232 RJ-45 console port
(1) RS-232 RJ-45 console port
(1) RS-232 RJ-45 Console Port (1) 10/100/1000 Ethernet Mgmt
(1) 10/100/1000 Ethernet Mgmt port
(1) 10/100 Ethernet Mgmt Port port
Interfaces (1) DB-9 COM port
(1) DB-9 COM Port (1) DB-9 COM port
(1) Compact Flash interface
(1) CompactFlash Interface (1) Compact Flash interface
(2) USB 2.0 port
(2) USB 2.0 port
NX-OS Version Prior to 4.1(1a) 3.x or later 3.x or later

Note: Must Replace Supervisor 1 modules with Supervisor 2 modules before upgrading to 4.1(1a) or
later.
Must Replace Gen 1 Line cards with Gen 2 Line cars before upgrading to 4.1(1a) or later.
4.1(1a) also doesn’t support MDS 9120, 9140, 9216,9216A switches
9500 Series Architecture
Cisco MDS 9000 Fibre Channel
Switching Modules
Oversubscription (Bandwidth Allocation)
• significant enhancements to the second-generation modules
− allows any port to perform like a line-rate interface.
− Used in conjunction with round robin fairness and data bursting capabilities,
− bandwidth allocation provides the capability to completely manage end-device
performance.
− Allocation of bandwidth is defined at the port level within a port group.
− A port group is defined by a series of ports that share back-end bandwidth
− Within a port group, port speed can be set to dedicated bandwidth or shared
bandwidth
− Bandwidth allocation is independent of the configured speed of the interface

Gen -1
− The 32-port module has a four-port port group. That is to say, four ports share the
back-end bandwidth to the backplane of the chassis.
Port Group Size per Module
Part Number Port Group Size Group Bandwidth
DS-X9032 4 2.5 Gbps
DS-X9124 6 12 Gbps
DS-X9148 12 12 Gbps
Oversubscription
• One characteristic that makes oversubscribed modules ideal for most data center
servers is their ability to respond to line-rate bursts of data
• Over subscribed Module
− First-generation 32-port module (DS-X9032)
− Second-generation DS-X9124 (24-port) and the DS-X9148 (48-port)

Data Burst and Fairness Capability in Oversubscribed Modules

Oversubscription Ratios per Module

Part
1 Gbps FC 2 Gbps FC 4 Gbps FC
Number
DS-X9032 1.6:1 3.2:1 -
DS-X9124 1:01 1:1 2:1
DS-X9148 1:01 2:1 4:1
Oversubscription (Gen-2 Module Example)
• In Figure the Fibre Channel interface speeds can be configured to 1 Gbps, 2 Gbps, or
4 Gbps
• The bandwidth dedicated to them might be above or below that
• With 12 Gbps of bandwidth available to a port group,
• the example explicitly reserves the amount of bandwidth required for ports 1, 2, 5, and
6.
• This allows the remaining 4 Gbps of bandwidth to be shared on ports 3 and 4 and
allows either port to burst to 4 Gbps.

Bandwidth Allocation also allows ISLs to be used on an oversubscribed module.


Because full line rate can be reserved for any given port, a port within the port group
can be set to full 4-Gbps speed, guaranteeing full performance for saturated ISLs in
the network. This feature makes a chassis with all oversubscribed modules a viable
configuration in many data centers.
Oversubscription …. Cont..
Gen-2 Example Gen-3 Example
ustlssnsbre001# sh module EMDC01PRODMDS031_A# sh mod
Mod Ports Module-Type Model Status
--- ----- -------------------------------- ------------------ ------------ Mod Ports Module-Type Model Status
1 48 1/2/4 Gbps FC Module DS-X9148 ok --- ----- -------------------------------- ------------------ ------------
2 24 1/2/4 Gbps FC Module DS-X9124 ok 1 24 1/2/4/8 Gbps 24-Port FC Module DS-X9224-96K9 ok
3 48 1/2/4 Gbps FC Module DS-X9148 ok 2 24 1/2/4/8 Gbps 24-Port FC Module DS-X9224-96K9 ok
4 24 1/2/4 Gbps FC Module DS-X9124 ok 3 24 1/2/4/8 Gbps 24-Port FC Module DS-X9224-96K9 ok
5 0 Supervisor/Fabric-2 DS-X9530-SF2-K9 active * 4 24 1/2/4/8 Gbps 24-Port FC Module DS-X9224-96K9 ok
6 0 Supervisor/Fabric-2 DS-X9530-SF2-K9 ha-standby 5 24 1/2/4/8 Gbps 24-Port FC Module DS-X9224-96K9 ok
7 48 1/2/4 Gbps FC Module DS-X9148 ok 6 24 1/2/4/8 Gbps 24-Port FC Module DS-X9224-96K9 ok
8 48 1/2/4 Gbps FC Module DS-X9148 ok 7 0 Supervisor/Fabric-2 DS-X9530-SF2-K9 ha-standby
9 48 1/2/4 Gbps FC Module DS-X9148 ok 8 0 Supervisor/Fabric-2 DS-X9530-SF2-K9 active *
9 24 1/2/4/8 Gbps 24-Port FC Module DS-X9224-96K9 ok
Mod Sw Hw World-Wide-Name(s) (WWN) 10 24 1/2/4/8 Gbps 24-Port FC Module DS-X9224-96K9 ok
--- -------------- ------ -------------------------------------------------- 11 24 1/2/4/8 Gbps 24-Port FC Module DS-X9224-96K9 ok
1 4.2(1a) 1.5 20:01:00:0d:ec:3f:1f:00 to 20:30:00:0d:ec:3f:1f:00 12 24 1/2/4/8 Gbps 24-Port FC Module DS-X9224-96K9 ok
2 4.2(1a) 1.5 20:41:00:0d:ec:3f:1f:00 to 20:58:00:0d:ec:3f:1f:00 13 48 1/2/4/8 Gbps 48-Port FC Module DS-X9248-96K9 ok
3 4.2(1a) 1.7 20:81:00:0d:ec:3f:1f:00 to 20:b0:00:0d:ec:3f:1f:00
4 4.2(1a) 1.7 20:c1:00:0d:ec:3f:1f:00 to 20:d8:00:0d:ec:3f:1f:00 Mod Sw Hw World-Wide-Name(s) (WWN)
5 4.2(1a) 1.4 -- --- -------------- ------ --------------------------------------------------
6 4.2(1a) 1.4 -- 1 4.2(7d) 1.1 20:01:00:0d:ec:3b:cc:40 to 20:18:00:0d:ec:3b:cc:40
7 4.2(1a) 1.6 21:81:00:0d:ec:3f:1f:00 to 21:b0:00:0d:ec:3f:1f:00 2 4.2(7d) 1.1 20:41:00:0d:ec:3b:cc:40 to 20:58:00:0d:ec:3b:cc:40
8 4.2(1a) 1.7 21:c1:00:0d:ec:3f:1f:00 to 21:f0:00:0d:ec:3f:1f:00 3 4.2(7d) 1.1 20:81:00:0d:ec:3b:cc:40 to 20:98:00:0d:ec:3b:cc:40
9 4.2(1a) 1.7 22:01:00:0d:ec:3f:1f:00 to 22:30:00:0d:ec:3f:1f:00 4 4.2(7d) 1.1 20:c1:00:0d:ec:3b:cc:40 to 20:d8:00:0d:ec:3b:cc:40
5 4.2(7d) 1.1 21:01:00:0d:ec:3b:cc:40 to 21:18:00:0d:ec:3b:cc:40
6 4.2(7d) 1.1 21:41:00:0d:ec:3b:cc:40 to 21:58:00:0d:ec:3b:cc:40
Mod MAC-Address(es) Serial-Num 7 4.2(7d) 1.4 --
--- -------------------------------------- ---------- 8 4.2(7d) 1.4 --
1 00-0d-ec-38-68-98 to 00-0d-ec-38-68-9c JAB103703MM 9 4.2(7d) 1.1 22:01:00:0d:ec:3b:cc:40 to 22:18:00:0d:ec:3b:cc:40
2 00-19-56-8f-70-54 to 00-19-56-8f-70-58 JAE11474A20 10 4.2(7d) 1.1 22:41:00:0d:ec:3b:cc:40 to 22:58:00:0d:ec:3b:cc:40
3 00-13-80-a8-e3-e8 to 00-13-80-a8-e3-ec JAE130201DT 11 4.2(7d) 1.1 22:81:00:0d:ec:3b:cc:40 to 22:98:00:0d:ec:3b:cc:40
4 00-0d-ec-75-80-c4 to 00-0d-ec-75-80-c8 JAE13072ZRE 12 4.2(7d) 1.1 22:c1:00:0d:ec:3b:cc:40 to 22:d8:00:0d:ec:3b:cc:40
5 00-19-56-3e-dc-18 to 00-19-56-3e-dc-1c JAB110700VK 13 4.2(7d) 1.1 23:01:00:0d:ec:3b:cc:40 to 23:30:00:0d:ec:3b:cc:40
6 00-19-56-3e-db-64 to 00-19-56-3e-db-68 JAB110700Z1
7 00-13-80-a8-38-a4 to 00-13-80-a8-38-a8 JAE1220HTGM
8 00-1e-f7-95-75-0c to 00-1e-f7-95-75-10 JAE1239VZI6
9 00-13-80-a8-a6-54 to 00-13-80-a8-a6-58 JAE1238V6FH
Oversubscription …. Cont..
Gen-2 Example Gen-3 Example
ustlssnsbre001# sho port-resources module 2 Module 2
Module 2 Available dedicated buffers for global buffer #0 [port-groups 1-4]
Available dedicated buffers are 4500 are 1873
Available dedicated buffers for global buffer #1 [port-groups 5-8]
Port-Group 1 are 5580
Total bandwidth is 12.8 Gbps
Total shared bandwidth is 10.8 Gbps Port-Group 1
Allocated dedicated bandwidth is 2.0 Gbps Total bandwidth is 12.8 Gbps
-------------------------------------------------------------------- Total shared bandwidth is 4.8 Gbps
Interfaces in the Port-Group B2B Credit Bandwidth Rate Allocated dedicated bandwidth is 8.0 Gbps
Mode --------------------------------------------------------------------
Buffers (Gbps) Interfaces in the Port-Group B2B Credit Bandwidth Rate
-------------------------------------------------------------------- Mode
fc2/1 16 4.0 shared Buffers (Gbps)
fc2/2 16 4.0 shared --------------------------------------------------------------------
fc2/3 16 4.0 shared fc2/1 500 4.0 dedicated
fc2/4 16 4.0 shared fc2/2 32 8.0 shared
fc2/5 16 4.0 shared fc2/3 500 4.0 dedicated
fc2/6 250 2.0 dedicated

Port-Group 2 Port-Group 2
****** ******
****** ******
Port-Group 4
Total bandwidth is 12.8 Gbps Port-Group 8
Total shared bandwidth is 10.8 Gbps Total bandwidth is 12.8 Gbps
Allocated dedicated bandwidth is 2.0 Gbps Total shared bandwidth is 12.8 Gbps
-------------------------------------------------------------------- Allocated dedicated bandwidth is 0.0 Gbps
Interfaces in the Port-Group B2B Credit Bandwidth Rate --------------------------------------------------------------------
Mode Interfaces in the Port-Group B2B Credit Bandwidth Rate
Buffers (Gbps) Mode
-------------------------------------------------------------------- Buffers (Gbps)
fc2/19 16 4.0 shared --------------------------------------------------------------------
fc2/20 16 4.0 shared fc2/22 32 8.0 shared
fc2/21 16 4.0 shared fc2/23 32 8.0 shared
fc2/22 16 4.0 shared fc2/24 32 8.0 shared
fc2/23 16 4.0 shared
fc2/24 250 2.0 dedicated
SAN Technology Overview
• Fibre Channel Protocol
− FC Communications
− Port types, ISL
− Addressing, Framing, Timers
− Virtual SAN (VSAN), Zoning
− Port Channels, IOD
− Virtual Output Queuing (VOQ)
Buffer to Buffer Credit Flow Control
• BB_Credits are used to ensure enough FC frames in flight
• A full (2112 byte) FC frame is approx 2 km long at 1 Gbps,
• 1 km long at 2 Gbps and ½ km long at 4 Gbps
• As distance increases, the number of available BB_Credits
• need to increase as well
• Insufficient BB_Credits will throttle performance—no data will be
• transmitted until R_RDY is returned

BB_Credits and Distance


FC_ID Address Model
• FC_ID address models help speed up routing
• Switches assign FC_ID addresses to N_Ports
• Some addresses are reserved for fabric services
• Private loop devices only understand 8-bit address (0x0000xx)
• FL_Port can provide proxy service for public address translation
• Maximum switch domains = 239 (based on standard)
Fibre Channel Timers
• Receiver-Transmitter Time-Out (R_T_TOV)
− Used to time events at the link level
− Loss of synchronization
− Times responses for link reset protocol
− Default value: 100 ms
• Error Detect Time-Out (E_D_TOV)
− Timers for events and responses at the sequence level
− Missing ACK or R_RDY when buffer credit has reached zero N_Port logout
− Timer value is set at fabric login to accommodate the network environment based on
delivery time of frames
− Default value: 2 sec
• Resource Allocation Time-Out (R_A_TOV)
− Time-out value for how long to hold resources associated with a failed operation Needed
to free shared resources for reuse
− Value to determine how long a port needs to keep responding to a link service request
before an error is detected
− 2xE_D_TOV in point-to-point and fabricwide from 1 to 231 – 1
− Default value: 10 s
SAN Design
Modern SAN design is about deploying ports and switches in a configuration
that provides flexibility and scalability.
It is also about making sure the network design and topology look as clean and
functional one, two, or five years later as the day they were first deployed.

• Principles of SAN Design


− Port density and topology requirements
• Number of ports required now and in the future

− Device performance and oversubscription ratios


• Determination of what is acceptable and what is unavoidable

− Traffic management
• Preferential routing or resource allocation

− Fault isolation
• Consolidation while maintaining isolation

− Control plane scalability


• Reduced routing complexity
Storage Network Evolution

Virtual Storage Area Network (VSAN)


• VSANs address the limitations of common SAN deployments
− VSANs are virtual fabrics
− SAN islands are virtualized onto a common SAN infrastructure
− Allocate ports within a physical fabric to create virtual fabrics
− Fabric disruption is limited to VSAN
− Traffic statistics are gathered per VSAN
• Features include:
− Dynamic provisioning and resizing
− Improved port utilization
− Shared ISL bandwidth
SAN Islands
• SAN islands are built to address several
technical and non-technical issues
− Maintains isolation from fabric events or
configuration errors
− Provides isolated and controlled management of
island infrastructure
− Driven by bad experiences of large multi- switch
fabrics
• However . . .
− Often over-provisioned port count for future
growth - wasteful and costly
− Very widespread issue today – some architects
still recommending islands
Fabric Virtualization
• A method to divide a common
physical fabric into virtual domains
• An infrastructure analogous to
VLANs in the Ethernet world
• A method to still isolate virtual
fabrics from one another for
− High availability
− Security
− Management

• A method to reduce wasted ports


as experienced in the island
approach
• A method to charge-back for used
resources from the physical fabric
3 Approaches to Fabric Virtualization
Switched Based Application Based Fabric Based

• Switch Line Card • Dedicated appliance • Fabric-wide


Partitioning provides routing virtualization via
• Island-level granularity hardware partitioning
• Island-level granularity
• No shared ISL’s • Port-level granularity
• No shared ISL’s
• Interconnection, but no • Fully shared ISL’s
• Interconnection, but no
consolidation • Drives consolidation
consolidation
Cisco’s Approach to Fabric Virtualization
• A VSAN provides a method to allocate ports
within a physical fabric to create
• virtual fabrics
• Analogous to VLANs in Ethernet
• Virtual fabrics created from larger cost-effective
physical fabric
• Reduces wasted ports with islands
• Fabric events are isolated per VSAN - maintains
HA (ie. RSCNs)
• Hardware-based isolation - traffic is explicitly
tagged across ISLs with VSAN membership info
• Statistics gathered per VSAN
• MDS 9500 Series
− Each port on the MDS 9000 family exists in VSAN
− Upto 256 VSAN in Single switch (Hardware can support
upto 4095)
− Logical configuration to move a port from one fabric to
another
− WWN-based VSANs can provide automated VSAN
membership
Virtual SAN’s - VSAN
Zoning

• Devices within a zone can access each other


− Zoning establishes access control
− Limiting access prevents unauthorized access

• Soft zoning
− Implemented in switch software and enforced by name
server
− Name server responds to discovery queries with only devices
found in requestor’s zone or zones
− “Soft zoning” used to be synonymous with “WWN zoning”

• Hard zoning
− Enforced by ACLs in port ASIC
− Applied to all data path traffic
− “Hard zoning” used to be synonymous with “port zoning”

• WWN zoning in hardware


− Both at source and destination ports
− Enhanced zoning
VSAN and Zoning
• Hierarchical relationship
− First assign physical ports to VSANs
− Then configure independent zones per VSAN
− VSANs only change when ports needed per
virtual fabric
− Zones can change frequently (e.g., backup)

• Zones provide added security and allow


sharing of device ports

• Zone membership is configured by:


− Port World Wide Name (pWWN)—device
− Fabric World Wide Name (fWWN)—fabric
− Fibre Channel Identifier (FCID)
− Fibre Channel Alias (FC_Alias)
− IP address
− Domain ID/port number
− Interface
Fibre Channel Communications
• Point-to-point oriented
• Facilitated through device login
• N_Port-to-N_Port connection
• Logical node connection point
• Flow controlled
• Buffer-to-buffer credits and end-to-end basis
• Acknowledged
• For certain classes of traffic, none for others
• Multiple connections allowed per device
Fibre Channel Port Types
N-Port Virtualization
Why Virtualize.

• Domain ID Limitation
− The Fibre Channel standard allows for a total of 239 port
addresses; however, qualification of such a fabric size is
nonexistent

• Interoperability with third-party switches

To address these concerns, two features, N-Port ID


Virtualization (NPIV) and N-Port Virtualizer, were
developed.

30 26 October 2018
N-Port Virtualization
N Port Virtualization
N port virtualization (NPV) reduces the number of Fibre Channel domain IDs in
SANs.
Switches operating in the NPV mode do not join a fabric. They pass traffic
between NPV core switch links and end devices, which eliminates the domain
IDs for these edge switches.

NPV is supported by the following Cisco MDS 9000 switches only:


• Cisco MDS 9124 Multilayer Fabric Switch
• Cisco MDS 9134 Fabric Switch
• Cisco Fabric Switch for HP c-Class BladeSystem
• Cisco Fabric Switch for IBM BladeCenter

31 26 October 2018
N-Port Virtualization
N-Port ID Virtualization

• NPIV allows a Fibre Channel host connection or N-Port, to be assigned multiple N-Port IDs or Fibre
Channel IDs (FCIDs) over a single link.
• All FCIDs assigned can now be managed on a Fibre Channel fabric as unique entities on the same
physical host.
• A host bus adapter (HBA) that supports the NPIV feature follows the standard login process.
• The initial connection and login to the fabric is performed through the standard F-Port login (FLOGI)
process.
• All subsequent logins for either virtual machines or logical part ions on a mainframe are transformed
into FDISC login commands.

EMDC01PRODMDS093_A# sh fl dat int fc2/11


--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
fc2/11 101 0x5d4a11 20:00:00:21:5a:2f:90:7e 10:00:00:21:5a:2f:90:7e
fc2/11 101 0x5d6100 50:06:0b:00:00:c2:67:c8 50:06:0b:00:00:c2:67:c9
fc2/11 101 0x5d9400 50:06:0b:00:00:c2:6b:18 50:06:0b:00:00:c2:6b:19

[BP1XEUAP1943] Total number of flogi = 3.

32 26 October 2018
VSAN – Routed Connectivity - IVR
Data traffic is transported between specific initiators and targets on different VSANs without merging
VSANs into a single logical fabric. Fibre Channel control traffic does not flow between VSANs, nor
can initiators access any resource across VSANs aside from the designated ones. Valuable
resources such as tape libraries are easily shared across VSANs without compromise
IVR is in compliance with Fibre Channel standards and incorporates third-party switches, however,
IVR-enabled VSANs may have to be configured in one of the interop modes
IVR Terminology
• Native VSAN
− The VSAN to which an end device logs on is the native VSAN for that end device.
• Inter-VSAN zone (IVR zone)
− A set of end devices that are allowed to communicate across VSANs within their interconnected SAN fabric. This definition is
based on their port world wide names (pWWNs) and their native VSAN associations. You can configure up to 2,000 IVR zones
and10,000 IVR zone members in the fabric from any switch in the Cisco MDS 9000 Family.
• Inter-VSAN zone sets (IVR zone sets)
− One or more IVR zones make up an IVR zone set. You can configure up to 32 IVR zone sets on any switch in the Cisco MDS
9000 Family. Only one IVR zone set can be active at any time.
• IVR path
− An IVR path is a set of switches and Inter-Switch Links (ISLs) through which a frame from one end-device in one VSAN can
reach another end-device in some other VSAN. Multiple paths can exist between two such end-devices.
• IVR-enabled switch
− A switch in which the IVR feature is enabled.
• Edge VSAN
− A VSAN that initiates (source edge-VSAN) or terminates (destination edge-VSAN) an IVR path. Edge VSANs may be adjacent
to each other or they may be connected by one or more transit VSANs
• Border switch
− An IVR-enabled switch that is a member of two or more VSANs. Border switches in Figure span two or more different color-
coded VSANs.
• Transit VSAN
− A VSAN that exists along an IVR path from the source edge VSAN of that path to the destination edge VSAN of that path.
• Autonomous fabric identifier (AFID)
− Allows you to configure more than one VSAN in the network with the same VSAN ID and avoid downtime when enabling IVR
between fabrics that contain VSANs with the same ID.
• IVR VSAN Topology
• IVR uses a configured IVR VSAN topology to determine how to route
traffic between the initiator and the target across the fabric. You can
configure this IVR VSAN topology manually on an IVR-enabled switch
and distribute the configuration using CFS in Cisco MDS SAN-OS
Release 2.0(1b) or later.
• SAN-OS Release 2.1(1a) or later, you can configure IVR topology in
auto mode. Prior to Cisco MDS SAN-OS Release 2.0(1b), you need
to manually copy the IVR VSAN topology to each switch in the fabric.
• Autonomous Fabric ID
• The autonomous fabric ID (AFID) distinguishes segmented VSANS
(that is, two VSANs that are logically and physically separate but have
the same VSAN number). Cisco MDS SAN-OS supports AFIDs from
1 through 64. AFIDs are used in conjunction with auto mode to allow
segmented VSANS in the IVR VSAN topology database. You can
configure up to 64 AFIDs.
• Transit VSAN Guidelines
− Consider the following guidelines for transit VSANs:
− Besides defining the IVR zone membership, you can choose to specify a set of transit VSANs to
provide connectivity between two edge VSANs:
− If two edge VSANs in an IVR zone overlap, then a transit VSAN is not required (though, not prohibited)
to provide connectivity.
− If two edge VSANs in an IVR zone do not overlap, you may need one or more transit VSANs to
provide connectivity. Two edge VSANs in an IVR zone will not overlap if IVR is not enabled on a switch
that is a member of both the source and destination edge VSANs.
− Traffic between the edge VSANs only traverses through the shortest IVR path.
− Transit VSAN information is common to all IVR zone sets. Sometimes, a transit VSAN can also act as
an edge VSAN in another IVR zone.

• Border Switch Guidelines


− Before configuring border switches, consider the following guidelines:
− Border switches require Cisco MDS SAN-OS Release 2.1(1a) or later.
− A border switch must be a member of two or more VSANs.
− A border switch that facilities IVR communications must be IVR enabled.
− IVR can (optionally) be enabled on additional border switches to provide redundant paths between
active IVR zone members.
− The VSAN topology configuration updates automatically when a border switch is added or removed.
IRV sample Config..

feature ivr
ivr distribute

ivr vsan-topology database


autonomous-fabric-id 1 switch-wwn 20:00:00:0d:ec:3b:ca:40 vsan-ranges 103,3001
autonomous-fabric-id 1 switch-wwn 20:00:00:0d:ec:3b:cc:40 vsan-ranges 101,3001

ivr vsan-topology activate


ivr vsan-topology database

autonomous-fabric-id 1 switch-wwn 20:00:00:05:73:ac:22:80 vsan-ranges 101,3001

ivr zone name IVR_Z_BP1XGBAP019_1_0_EMCDMX1397_07B0


member pwwn 50:06:04:84:52:a6:25:56 vsan 101
member pwwn 10:00:00:00:c9:6d:99:53 vsan 103

ivr zoneset name IVR_ZS_EMDC02_PROD_103_A_EMDC01_PROD_101_A


member IVR_Z_BP1XGBAP019_1_0_EMCDMX1397_07B0

ivr zoneset activate name IVR_ZS_EMDC02_PROD_103_A_EMDC01_PROD_101_A

ivr commit
FCIP Vs iFCP
• FCIP is a tunneling protocol that moves Fibre Channel
traffic over an IP network. It is mostly used for remote
connections between two Fibre Channel SANs over a
TCP/IP network

• iFCP is a protocol that uses the FCP protocol from the


Fibre Channel standard, but implements it on IP networks

• FCIP runs in some type of "junction box" like a router, so


there are no HBAs specifically for it. The function is
implemented in some type of network system..

39 26 October 2018
FCIP

40 26 October 2018
iFCP

41 26 October 2018
Fcdomain
About fcdomain Phases

• Principal switch selection—This phase guarantees the selection of a unique principal switch across
the fabric.
• Domain ID distribution—This phase guarantees each switch in the fabric obtains a unique domain
ID.
• FC ID allocation—This phase guarantees a unique FC ID assignment to each device attached to
the corresponding switch in the fabric.
• Fabric reconfiguration—This phase guarantees a resynchronization of all switches in the fabric to
ensure they simultaneously restart a new principal switch selection phase.

• The behavior of a subordinate switch depends on the allowed domain ID lists, the configured
domain ID, and the domain ID assigned by principal switch.
•When the received domain ID is not within the allowed list, the requested domain ID becomes the
runtime domain ID and all interfaces are isolated.
•When the assigned and requested domain IDs are the same, the preferred and static options are
not relevant, and the assigned domain ID becomes the runtime domain ID.
•When the assigned and requested domain IDs are different, the following cases apply:

–If the configured type is static, the assigned domain ID is discarded, all local interfaces are
isolated, and the local switch assigns itself the configured domain ID, which becomes the runtime
domain ID.
–If the configured type is preferred, the local switch accepts the domain ID assigned by the principal
switch, and the assigned domain ID becomes the runtime domain ID.
Output of “show fcdomain”
• VSAN 101
The local switch is a Subordinated Switch.
Local switch run time information:
State: Stable
Local switch WWN: 20:65:00:05:73:ac:22:c1
Running fabric name: 20:65:00:05:73:ac:22:81
Running priority: 99 ( Default priority of 128 )
Current domain ID: 0x63(99)
• ( all fcid assignments on this switch will start with 0x63)
Local switch configuration information:
State: Enabled
FCID persistence: Enabled
Auto-reconfiguration: Disabled
• ( the domain will not automatically start fabric reconfig in case of pricipal switch failure )
Contiguous-allocation: Disabled
Configured fabric name: 20:01:00:05:30:00:28:df
Optimize Mode: Disabled
Configured priority: 99
• ( manually assigned priority, 1 – 254 range. 1 Is the highest )
Configured domain ID: 0x63(99) (static)
• ( preferred / static , preferred will lead to new runtime domain id assigned when overlap is noted.
For static domain id , the switch will be segmented in case of overlap )
Principal switch run time information:
Running priority: 2
Interface Role RCF-reject
---------------- ------------- ------------
port-channel 1 Upstream Disabled
---------------- ------------- ------------
Output of “sh fcdomain domain-list”
• VSAN 101
Number of domains: 6 Depending on the vsans active
Domain ID WWN on the switches the principal
--------- -----------------------
0x03(3) 20:65:00:05:73:ac:22:81 [Principal] switch id can differ. Looking at
0x21(33) 20:65:00:05:73:ac:b7:81
0x1f(31) 20:65:00:0d:ec:3b:cc:41 the example, vsan 101 is local
0x63(99) 20:65:00:05:73:ac:22:c1 [Local]
0x5d(93) 20:65:00:0d:ec:3c:1c:81 while 1001 is across site and
0x5f(95) 20:65:00:0d:ec:b6:c4:41 hence the principal switch for
VSAN 1001 the vsans is different.
Number of domains: 11
Domain ID WWN
--------- -----------------------
0x01(1) 23:e9:00:0d:ec:3b:ca:41 [Principal]
0x1b(27) 23:e9:00:0d:ec:3b:cc:81
0x1d(29) 23:e9:00:0d:ec:3b:c1:c1
0x1f(31) 23:e9:00:0d:ec:3b:cc:41
0x03(3) 23:e9:00:05:73:ac:22:81
0x5d(93) 23:e9:00:0d:ec:3c:1c:81
0x5f(95) 23:e9:00:0d:ec:b6:c4:41
0x21(33) 23:e9:00:05:73:ac:b7:81
0x5b(91) 23:e9:00:0d:ec:3b:ca:c1
0x61(97) 23:e9:00:0d:ec:b7:43:01
0x63(99) 23:e9:00:05:73:ac:22:c1 [Local]
General issue’s on MDS switches
• Slow drain device - a device that cannot cope with the incoming traffic in a timely manner.
Slow drain devices can't free up their internal frame buffers and therefore don't allow the
connected port to regain their buffer credits quickly enough.
In NX-OS 4.2(7a) the slow drain policy for port monitor application is enabled by default.
SNMP traps can be used to alert admin/vendors on slow drain devices.
On older versions slow drain devices can be manually detected using commands below :
show hardware internal packet-flow dropped ( this will point us to the mod )
module-x show logging onboard timeout-drops ( this command can be used to isolate
the device down to the specific port)

• Congestion - a situation where the workload for a link exceeds its actual usable bandwidth.
Congestion happens due to overutilization or oversubscription. ( will need to check ISL
utilizations to verify if this is an issue using cisco performance manager or web client )

• Bottleneck - a link or component that is not able to transport all frames directed to or
through it in a timely manner. (e.g. because of buffer credit starvation or congestion)

• Link issue’s due to faulty hardware. Sometimes link can automatically go down if the error
count on that port is deemed too high.
Understanding
fc2/45 is up
details of show port o/p
Port description is REUXEUUS507_6
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
General commnds used to
Port WWN is 20:6d:00:0d:ec:3c:3a:80
Admin port mode is FX, trunk mode is off check port properties:
snmp link state traps are enabled Sh interface fcx/y
Port mode is F, FCID is 0x5e5981
Port vsan is 102 sh port interface fc x/y trans
Speed is 4 Gbps sh run interface fcx/y
Rate mode is shared
Transmit B2B Credit is 16
Receive B2B Credit is 16
Receive data field Size is 2112
Beacon is turned off
5 minutes input rate 30273160 bits/sec, 3784145 bytes/sec, 4450 frames/sec
5 minutes output rate 321758744 bits/sec, 40219843 bytes/sec, 22157 frames/sec
34631516316 frames input, 34268810545088 bytes
0 discards, 0 errors
0 CRC, 0 unknown class
0 too long, 0 too short
149240682528 frames output, 269415777626372 bytes
0 discards, 0 errors
2 input OLS, 2 LRR, 0 NOS, 2 loop inits
3 output OLS, 0 LRR, 1 NOS, 2 loop inits
16 receive B2B credit remaining
14 transmit B2B credit remaining
12 low priority transmit B2B credit remaining
Interface last changed at Thu Dec 29 20:36:02 2011
Port channels

A PortChannel has the following functionality:

•Provides a point-to-point connection over ISL (E ports) or EISL (TE ports). Multiple
links can be combined into a PortChannel.
•Increases the aggregate bandwidth on an ISL by distributing traffic among all functional
links in the channel.
•Load balances across multiple links and maintains optimum bandwidth utilization. Load
balancing is based on the source ID, destination ID, and exchange ID (OX ID).
* PortChannels may contain up to 16 physical links and may span multiple modules for
added high availability.

PortChanneling and trunking are used separately across an ISL:


•PortChanneling, which enables several links to be combined into one aggregated link,
can be done between E ports and TE ports.
•Trunking, which permits carrying traffic on multiple VSANs between switches, can be
done only between TE ports.
EMDC01PRODMDS031_A# sh interface port-channel 6
port-channel 6 is trunking
Port description is "Cross Site Port Channel To

Port channel EMDC02PRODMDS001_A"


Hardware is Fibre Channel
Port WWN is 24:06:00:0d:ec:3b:cc:40
outputs Admin port mode is auto, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 1
Speed is 10 Gbps
• EMDC01PRODMDS031_A# sh port-channel summary Trunk vsans (admin allowed and active) (1001,1301,3001)
---------------------------------------------------------------------------
-- Trunk vsans (up) (1001,1301,3001)
Interface Total Ports Oper Ports First Trunk vsans (isolated) ()
Oper Por Trunk vsans (initializing) ()
--------------------------------------------------------------------------- 5 minutes input rate 3206595784 bits/sec, 400824473
-- bytes/sec, 203281 frames/sec
port-channel 1 0 0 --
port-channel 2 12 12 fc2/1 5 minutes output rate 3539961920 bits/sec, 442495240
port-channel 6 5 5 fc2/8 bytes/sec, 222604 frames/sec
port-channel 10 16 16 fc5/5 12645170024674 frames input, 24954081409212588
port-channel 11 16 16 fc6/4 bytes
EMDC01PRODMDS031_A# sh port-channel database
0 discards, 152 errors
interface port-channel 6 152 CRC, 0 unknown class
port-channel 6 0 too long, 8 too short
Administrative channel mode is active 14388487132787 frames output, 28540941456794436
Operational channel mode is active bytes
Last membership update succeeded
First operational port is fc2/8 338112 discards, 0 errors
5 ports in total, 5 ports up 21 input OLS, 1148 LRR, 60 NOS, 4582 loop inits
Ports: fc12/6 [up] 4669 output OLS, 4352 LRR, 54 NOS, 2537 loop inits
fc1/8 [up] Member[1] : fc1/8
fc2/8 [up] * Member[2] : fc2/8
fc12/10 [up]
fc5/12 [up] Member[3] : fc5/12
Member[4] : fc12/6
Member[5] : fc12/10
Interface last changed at Wed Jun 27 20:37:04 2012
Checking switch logs
# sh logging logfile | last 50
2012 Jul 3 12:39:34 EMDC01PRODMDS099-A %PORT-5-IF_UP: %$VSAN 101%$ Interface fc2/21 is up in
mode F
2012 Jul 3 13:37:20 EMDC01PRODMDS099-A %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication
failed for user admiin from 149.189.130.239 - sshd[8645]
2012 Jul 3 13:37:20 EMDC01PRODMDS099-A %DAEMON-3-SYSTEM_MSG: error: PAM: Authentication
failure for illegal user admiin from 149.189.130.239 - sshd[8644]
2012 Jul 3 13:37:22 EMDC01PRODMDS099-A %DAEMON-3-SYSTEM_MSG: error: ssh_msg_send: write -
sshd[8646]
2012 Jul 3 13:41:47 EMDC01PRODMDS099-A %DAEMON-3-SYSTEM_MSG: error: ssh_msg_send: write -
sshd[8646]
2012 Jul 3 13:41:47 EMDC01PRODMDS099-A %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 101%$
Interface fc4/46 is down (Link failure)
2012 Jul 3 13:45:39 EMDC01PRODMDS099-A %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 101%$
Interface fc4/46 is down (Link failure)
2012 Jul 3 13:45:39 EMDC01PRODMDS099-A %PORT-5-IF_UP: %$VSAN 101%$ Interface fc4/46 is up in
mode F
2012 Jul 3 13:46:57 EMDC01PRODMDS099-A %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 101%$
Interface fc4/48 is down (Link failure)
2012 Jul 3 13:47:03 EMDC01PRODMDS099-A %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 101%$
Interface fc4/48 is down (Link failure)
2012 Jul 3 13:47:03 EMDC01PRODMDS099-A %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 101%$
Interface fc4/46 is down (Link failure)
2012 Jul 3 13:50:37 EMDC01PRODMDS099-A %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 101%$
Interface fc4/46 is down (Link failure)
2012 Jul 3 13:50:37 EMDC01PRODMDS099-A %PORT-5-IF_UP: %$VSAN 101%$ Interface fc4/46 is up in
mode F
Tools and Software's
• command-line interface (CLI)
− Telnet , SSH, Serial Connection
• Fabric Manager
− Java-based graphical user interface, using SNMP
Fabric Manager
• Fabric Manager Server
− advanced monitoring, troubleshooting, configuration for multiple fabrics
− accessed by up to 16 Fabric Manager Clients at a time
− must be start before Fabric Manager Client

• Fabric Manager Client


− real-time views of your network fabric
− manage the configuration of Cisco MDS 9000 Family devices and third-party switches.

• Device Manager—Presents two views


− Device View displays a continuously updated physical representation
− Summary View presents real-time performance statistics of all active interfaces

• Fabric Manager Web Services


− Allows operators to monitor MDS events, performance, and inventory, and perform
minor configuration tasks from a remote location using a web browser.

• Performance Manager
− Provides detailed traffic analysis by capturing data with SNMP. This data is
compiled into various graphs and charts that can be viewed with any web browser
using Fabric Manager Web Services.
Features

Logging Level

User Security
Roles
User

Switch Mgmt IP

Security
Snmp
User

IVR Features Callhome

Interface Config Ntp

Logging server VSANs


details Device Alias
Database
IVR Topology Domain Config

Zoneset FCID database


Distribute
Port Channel
Zone and interface
Zoneset VSAN
Database database
IVR Switch Mgmt
Zone and Gateway
Zoneset
Database
Interface Boot Image
Channel Group
Config
Technology for better business outcomes

© 2009 Hewlett-Packard Development Company, L.P.


The information contained herein is subject to change without notice.

Вам также может понравиться