Cisco MDS 32G HPE OSM Training - 062018 PDF

Cisco MDS 32G FC SAN Solutions
MDS 32G FC module & Entry Level 32G FC SAN Switch
Bhavin Yadav
Technical Marketing Engineer
May 15, 2018
Guidelines and Disclaimers
• All roadmaps are displayed in calendar year (CY) quarters.
• A roadmap presentation slide may not show a complete list of new features;
contact the account team for more details.
• Information contained in this document regarding future technology, future

features, future quality standards, or product or release availability is subject
to change and is not a binding commitment by Cisco.
• The information contained in this document is Cisco Partner Confidential

information and is provided to the recipient subject to the terms and
conditions of the applicable nondisclosure agreement between Cisco and
the recipient.
- For Partners and Customers with Nondisclosure Agreements
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Cisco MDS 32G FC Module
Agenda
Use case
Brief overview Hardware features Software features
topologies
Cisco MDS 32-Gbps Fibre Channel Module
Lock Screws
Eject Push Eject Lever

Button Port LEDs
• 48 x 32-Gbps Fibre Channel line-rate ports

• Common module for Cisco® MDS 9700 Series platform
• 4/8/16/32-Gbps FC supported speeds in SFP+ form
• 16 ports per port group
• Up to 1.5 Tbps front-panel bandwidth
• Cisco Part number: DS-X9648-1536K9
Port Numbering and Specifications
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47
2 4 6 8 10 12 14 16 18 20 22 24 10 28 30 32 34 36 38 40 42 44 46 48
• Top row: Odd-numbered ports

• Bottom row: Even-numbered ports
• Supports F, FL, E, TE, SD, and ST modes
• Supports FC TrustSec
• No support for 2G FC, 10G FC and Arbitrated Loop (AL) ports
Buffer to Buffer Credit Allocations
16-Member Port Group

Per Port
• Default 500 buffer credits per port for “E” port
• Default 32 buffer credits per port for “F” port
Port groups
• Three Port Groups, 16 ports in each port group
• Up to 8300 buffer credits among a single port group
• Up to 8270 buffer credits can be assigned to a port in port group
B2B Credits, Speed, and Distance Reference
Per-kilometer B2B credit requirements at different speeds and frame sizes
Frame Size 1 Gbps 2 Gbps 4 Gbps 8 Gbps 10 Gbps 16 Gbps 32 Gbps
512 bytes 2 B2B/km 4 B2B/km 8 B2B/km 16 B2B/km 24 B2B/km 32 B2B/km 64 B2B/km
1024 bytes 1 B2B/km 2 B2B/km 4 B2B/km 8 B2B/km 12 B2B/km 16 B2B/km 32 B2B/km
2112 bytes 0.5 B2B/km 1 B2B/km 2 B2B/km 4 B2B/km 6 B2B/km 8 B2B/km 16 B2B/km
With default buffer credit of 500 per port for 32-Gbps module
Speed and Frame Size 512 Bytes 1024 Bytes 2112 Bytes
2 Gbps 125 km 250 km 500 km
4 Gbps 62 km 125 km 250 km
8 Gbps 31 km 62 km 125 km
Using 8270 buffer credits per port for 32-Gbps module (with Enterprise license)
Speed and Frame Size 512 Bytes 1024 Bytes 2112 Bytes
2 Gbps 2050 km 4100 km 8200 km
4 Gbps 1025 km 2050 km 4100 km
8 Gbps 512 km 1025 km 2050 km
16 Gbps 256 km 512 km 1025 km
32 Gbps 128 km 256 km 512 km
Line-rate Port Configuration
Number of Fabric Modules in Front-Panel FC Front Panel FCoE Maximum Number of
Cisco® MDS 9700 Series Bandwidth per Slot Bandwidth per Slot Line-Rate Ports per Module
24 x 32G FC
3 768 Gbps 660 Gbps 48 x 16G FC
48 x 10GE FCoE
32 x 32G FC
4 1024 Gbps 880 Gbps 48 x 16G FC
48 x 10GE FCoE
40 x 32G FC
5 1280 Gbps 1100 Gbps 48 x 16G FC
48 x 10GE FCoE
48 x 32G FC
6 1536 Gbps 1320 Gbps 48 x 16G FC
48 x 10GE FCoE
SFP+ Optical Modules
Wavelength Cable Core Size Modal Bandwidth
SFP Cable Distance
(nm) Type (microns) (MHz-km)
32GFC - LW
• 3200-SM-LC-L
1310 SMF G.652 - 10 km
• 1600-SM-LC-L
• 800-SM-LC-L
32GFC - SW
• 3200-M5-SN-S 500 (OM2) 20m (OM2)
850 MMF G.651.1 2000 (OM3) 70m (OM3)
• 1600-M5-SN-S
4700 (OM4) 100m (OM4)
• 800-M5-SN-S
Functional Block Diagram
Central
Arbiter Spine 1 Spine 2 Spine 3 Spine 4 Spine 5 Spine 6
Crossbar ASIC -1 Crossbar ASIC -2
F32 ASIC -1 F32 ASIC -2 F32 ASIC -3
48 Ports
Temperature Sensors
• F32 ASIC has internal temperature sensor circuitry that is

accessed through internal registers
• No separate inlet or outlet temperature sensors
• Two external temperature sensors exist on module
- One for AXE3506 processor
- One for PEX8713 FPGA
LEDs
Module LED Description
Green • Operation is normal.
• The module is booting or running diagnostics.

Orange
• A minor temperature condition occurred.
• The diagnostics test failed.

• The module is not operational.
Red
• A fault occurred during the initialization sequence.
• An over-temperature condition occurred.
• The module is resetting.

• A major temperature threshold has been exceeded.
Blinking Red
• The module failed to download code and configuration information successfully
during the initial reset.
LEDs
Interface LEDs Description
Green The port is active.
Orange The module or port is disabled through the CLI command, or the module is initializing.
Blinking Orange The port is faulty and has been disabled.
Off The port is not active or the link is not connected.
ID LEDs Description
Blinking Blue The module is selected for service from the CLI.
Off The module is not selected.
Comparison: 16G and 32G FC Modules
16G FC (DS-X9448- 32G FC (DS-X9648-
Description
768K9) 1536K9)
Total Bandwidth 768G FC 1536G FC
Port-group size 4 ports 16 ports
Ports per ASIC 8 ports 16 ports
ASICs per module 6 3
Analytics capability Absent Present
Anti-Counterfeit Technology Absent Present
Secure Boot Absent Present
Use Case Topologies
32-Gbps Storage Edge
16- and 32-Gbps Flash
Storage Arrays
16- and 32-Gbps FC
Cisco® MDS 9700 Series Directors

with 32-Gbps FC Modules
16-Gbps FC
Cisco MDS 9700 Series Directors
2-, 4-, 8-, and
16-Gbps FC
32-Gbps Inter-Switch Links
16- and 32-Gbps Flash
Storage Arrays
16- and 32-Gbps FC

32-Gbps FC ISL
Cisco MDS 9700 Series Directors
with 16- and 32-Gbps FC Modules
2-, 4-, 8-, 16-, and
32-Gbps FC
High-Performance Storage
16- and 32-Gbps FC and NVMe
FC-SCSI NVMe-FC
Flash Storage
Array Array
16- and 32-Gbps FC

16- and 32-Gbps FC
FC-SCSI Servers NVMe-FC Servers
NVMe over FC and FCoE
Cisco UCS® C-Series Rack Servers
HBA
Traditional FC-SCSI-capable initiator
Traditional FC-SCSI-capable target
HBA
FC-NVMe-capable initiator FC or FCoE fabric built using Cisco® MDS FC-NVMe-capable target
9000 Family switches
• Increased performance: Unleash the power of all-flash arrays using NVMe over FC or FCoE
• Seamless insertion: Support on all existing 16-Gbps FC and new 32-Gbps FC Cisco MDS 9000 Family switches
• Phased transition: SCSI and NVMe initiators and targets continue to share same SAN
• Multiprotocol flexibility: SCSI or NVMe traffic over FC or FCoE transport
• Ecosystem support: Fully validated solution on Cisco UCS C-Series servers, Cisco MDS 9000 Family switches, and
Broadcom and Avago HBAs
Cisco MDS 9700 Series Modules
Bringing FC, FCoE, and FCIP Together
48-Port
32-Gbps
FC Module
• Flexibility
• Redundancy
24/10 SAN 48-Port
Extension 16-Gbps
Module FC Module
• Interchangeable
z
Cisco® Cisco Cisco
• Multiprotocol MDS MDS MDS
9710 9718 9706
• Licensing
48-Port 10 24-Port 40
GE FCoE GE FCoE
Module Module
Cisco MDS 9700 Series Directors Reference
Winning Points
• Multiprotocol flexibility
• Single OS (Cisco® NX-OS)
• Single management (Cisco DCNM)
14RU
• Line-rate performance 9RU
• Better Cisco UCS® interoperability
Hardware Feature Cisco MDS 9706 Cisco MDS 9710

Cisco part number DS-C9706-1K9 DS-C9710-1K9
Line-card slots 4 8
Line-rate 32-Gbps FC ports (DS-X9648-1536K9) 192 384
Line-rate 16-Gbps FC ports (DS-X9448-768K9) 192 384
Line-rate 10 GE FCoE ports (DS-X9848-480K9) 192 384
Line-rate 40 GE FCoE ports (DS-X9848-960K9) 96 192
Line-rate 2/4/8/10/16-Gbps FC ports (DS-X9334-K9) 96 192
Line-rate 1/10/40 GE FCIP ports (DS-X9334-K9) 32 8 64 16
Cisco MDS 9132T
Overview
Cisco MDS 9132T Overview
• Base Switch: 16 x 32G FC line-rate ports with one expansion slot

• 8-port active and 16-port active options available
• Logical Expansion Module (LEM): 16 x 32G FC line-rate add-on ports
• Supports 4/8/16/32G FC speeds
• Port side intake and Port side exhaust air-flow options
Cisco MDS 9132T Overview
• Dedicated Network Processor for SAN Analytics*
• Base chassis contains one IO-ASIC, CPU, NPU processor
• LEM module contains additional dedicated IO-ASIC
• Required NX-OS: 8.2(1) or later
• DCNM Management: Required DCNM version: 10.4(1) or later
*: Future Software Support

Physical Dimensions
17.3 inches
1.72 inches
Chassis dimensions:
Height: 1RU ; 1.72 inches
Width: 17.3inches Logical Expansion Module (LEM)
Depth: 20.12 inches
MDS 9132T Rear view
• Hot swappable PSUs

• 1 PSU option for 8-port base switch
• 1-2 PSU option for 16-32 port configurations
• Hot swappable redundant fan modules
• 2 Fan modules for 8-port base switch
• 2-4 Fan modules for 16 – 32 port configuration with 3+1 redundancy
• Color coded FAN and PSUs:
• Blue: Port side exhaust airflow
• Red: Port side intake airflow
• PSU and FAN status LEDs
© 2017 Cisco and/or its affiliates. All rights reserved.
Cisco Partner Confidential
Cisco MDS 9132T Chassis LEDs
• Switch Status (tri-color)
• Power Supply Status (tri-color)
• Fan Status (tri-color)
• Ethernet Management Port Link (bi-color)
• Ethernet management Port Activity (bi-color)
• Fibre Channel Port Link (one tri-color LED per port)
Management Ports
• One console port
• 2x 10/100/1000M-baseT Ethernet management ports

• Mgmt0 is a standard management port
• Mgmt1 is reserved for future use
Port Options
• Supports 4/8/16/32G FC speed
• Supported port modes: F, E, TE, SD, ST
• Up to 1024G FC front panel bandwidth
• Incremental port options

• 8-Port à 16-Port à 32-Port
• 8-Port à 24-Port à 32-port
• Supported SFPs: 32G FC SW/LW, 16G FC SW/LW, 8G FC SW/LW optics
• Based on same IO ASIC as 32G module of MDS 9700 platform
• SAN Analytics* supported on ANY port
* Future Software Support

Cisco MDS 9132T Port Numbering
Top Row: Odd number of ports
Bottom Row: Even number of ports
Console Port
Mgmt1 Eth Port*
1 15 17 31
2 16 18 32
Status Mgmt0 Eth Port

USB Port LEM Holding Screw
LEDs
PSU 1 FAN 1 FAN 2 FAN 3 FAN 4 PSU 2

*: For future use

Cisco MDS 9132T Rear LEDs
FAN Status Led
PSU Status Leds
• Power supply LEDs FAN handles
• Top LED (tri-color): Power Supply AC Status (1 tri-color per PS)

• Bottom LED (tri-color): PS Failure (AMBER), Temperature, Over Voltage, Fan Failure)
• Fan Status LEDs (tri-color)
Cisco MDS 9132T Port Options
Any 8 Ports
8 Port
+16 Port LEM +8 Ports on Demand
16 Port
Any 16 Ports
+16 Port LEM

24 Port
Expansion Module Slot +8 Ports on Demand
32 Port
Feature support
• POAP (Power On Auto Provisioning)
• Secure boot
• ISSU and ISSD
• FC Redirect capability
• Active power measurement
• Link level encryption (Cisco FC Trustsec) with AES-128 bit encryption
• Interoperability with Smartoptics xDWDM 8G/16G transceivers
• Per VL credit slicing and buffer allocation
Buffer to Buffer Credits
• Default buffer credits per port: 500
• 8300 buffer credits per 16-port port group
• Extended buffer credits:

• Maximum of 8270 credits on a single port
• Supports ~512km @ 32G FC Speed
Supported Optics
Part Number Product Name
DS-SFP-FC32G-SW 32G FC Shortwave Optics
DS-SFP-FC32G-LW 32G FC Longwave Optics

Power Supply Unit (PSU) and Fan Redundancy
• 650W Power supply unit
• N:N PSU redundancy (2 PSUs)
• Redundant Fan modules
• Airflow options (PSU and Fan):

• Port-Side Intake (PSI)
• Port-Side Exhaust (PSE)
Built-in PSU mis-match airflow protection
• Automatic detection of PSU hot-swap and airflow directions
• If PSU airflow is different from Fan-tray, a fatal error will be generated
• Switch will perform software initiated power down procedure in10 minutes
Fan-tray mismatch airflow protection
• Automatic detection of Fan during hot-swap and installation
• In case of mismatch airflow detection:

• Switch will power down the board in 20 seconds
• Light front panel status LED in slow flashing red
• Default shutdown timer: 20s
Troubleshooting
Front Panel LED Troubleshooting
LED Color State Description

Interface LED (Link) Solid green Link is up
Steady flashing green Link is up (beacon used to identify port)
Intermittent flashing green Link is up (traffic on port)
Solid orange Link is disabled by software
Flashing orange A fault condition exists
Off No Link
Ethernet LED
MGMT Eth1 (Left)
Solid Green Physical link
Off No Activity
Ethernet LED
MGMT Eth0 (Right)
BlinkingCisco
Green
© 2017 Cisco and/or its affiliates. All rights reserved. Partner Confidential Activity
LED Troubleshooting
Indicator Location Function Color Status State
Solid On Power supply is OK

Green
Front panel Chassis
Power LED Off The module is not receiving power
of chassis Power/Health
Red On Power supply failed
Green Solid on All diagnostics pass. The module is operational

The switch is booting or running diagnostics.
Orange
An over temperature condition occurred (minor)
Front panel
Status LED System status
of chassis Red Blinking Unmatched airflow direction for PSU/Fan-tray
The diagnostic test failed during initialization
Red Solid on sequence
An over temperature condition occurred (major)
LED Troubleshooting
Indicator Location Function Color Status State
Green Solid on Fan operating normally

Fan status Front panel Fan health
Red Solid on Fan failure
OFF No AC power to power supply
Green Solid On Power supply on and DC output OK
Blinking AC Power OK, DC output not enabled

PSU Status Faceplate of each
PSU Health
Indicators PSU Critical power supply failures, over voltage,
Solid On
over current, over temperature or fan failure
Amber Blinking Power supply warning, PSU is still operating
OFF Operating normally
Faceplate of each Fan tray heath Green Solid On Fan tray operating normally
Fan Status
Fan Module indicator
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential Amber Solid On Fan failure within the fan tray
Use Cases
Scalable Standalone Switch
Pay-as-You-Grow Flexibility
Fl
sha
St
or
ag
e
Backup
Storage
Application &
© 2017 Cisco and/or its affiliates. All rights reserved. Database Servers
Cisco Partner Confidential
Enterprise Core-Edge SAN
MDS 9132T as Middle of Row / Top of Rack
• MDS 97xx as Core switches Flash Storage

• Analytics visibility across Fabric (NVMe/SCSI)
• Easy to scale, Easy to Manage

Core MDS 9700s
ToR/MoR MDS 9132T
Compute Servers
(NVMe/SCSI)
Business Continuity in Metro-SAN
Business Continuity and Consolidation
Flash Storage
MAN / WAN
xWDM circuit
Backup Site
Application &
Database Servers
Primary Data Center

Software Features
Congestion Isolation and Slow Flow
• Credit-based flow control scheme using R_RDY protocol
• R_RDY protocol informs the transmitter that a new buffer is available at the
receiver end
• Normal flows and slow flows both consume the buffer credits on an ISL
• Slow flows hog the link credits on the ISL, reducing credits available for
normal flows leading to under-utilization of the ISL
PMON: Port monitor
VL: Virtual Link
R_RDY: Receiver Ready
ER_RDY: Extended Receiver Ready
HOL: Head On Line
Congestion-isolation Benefits
• Automatic detection of congestion at an interface level using PMON policy

• Configurable option to mark a device slow
• Configurable option to exclude a slow device
• Include and exclude config is supported for offline devices
• There is a de-isolate CLI option available at an interface level
Current Congestion Isolation overview
Credit-based flow control scheme using R_RDY protocol

Normal flows and slow flows both consume the buffer credits on an ISL, leading under-
utilization of the ISL
Improved Congestion Isolation overview
Automatic isolation of slow flows to a low priority VL
Congestion Isolation
Congestion Isolation using the ER_RDY & VL capability
• Supported on MDS 16G and 32G platform
• In a mixed fabric (8G Platform + 16G Platform + 32G module), feature will
not work effectively (Not disabled but will not function as expected)
• Allows different flows to be routed to different virtual links based on QoS
• Provides:
• A mechanism to isolate slow flows to a specified VL, there by the line rates of
normal flows are not affected.
• Automatic isolation of a slow flow to a low priority VL

PMON: Port monitor
• De-isolation of a slow flow using CLI mechanism VL: Virtual Link
R_RDY: Receiver Ready
ER_RDY: Extended Receiver Ready
HOL: Head On Line
Configuration Phase 1: Enabling ER_RDY
• Prepare fabric with all ISLs to come up in ER-RDY flow control mode
• CLI Command: “system fc flow-control er_rdy” (default is r_rdy (disabled))
• Followed by link flap for ISLs to to have VL(Virtual Links) configured

• Port-channel members can be flapped individually (less disruptive)
• Check the operational mode of the ISLs (R-RDY/ER-RDY)
• Confirm ER-RDY mode using CLI:
• CLI: show flow-control er_rdy
Configuration Phase 2: Congestion-Isolation
• Enabling indicates it will honor the slow-interface detected by
PMON
• PMON isolates the flow destined to the hosts behind slow-interface
• Active policy with port-guard action as ”cong-isolate” is needed
• PMON sends updates of the slow-interfaces to FCNS
• PMON policy monitors the counters in the policy
• When threshold is hit, PMON policy sends an update of the interface
to FCNS process
Configuration guidelines
Disabled by default
Turn ON before adding any new switch to the fabric working in R-RDY mode
Members of a PC can be flapped individually to change to ER_RDY mode
All configurations are per switch, not applied to the entire fabric
Slowdrain counters monitored by PMON
TX Credit Not Available: Transmit credit not available is the continuous no-
transmit of credit for 100ms
Credit Loss Recovery: When credits are not available for 1s or 1.5 sec (F or
E)
TX-Slowport-Oper-Delay: The total amount of time the port was in the zero-
transmit-credits, or the average credit delay experienced by the port
TXWait : Counter increments whenever TX B2B credits = 0 and frame(s) are
waiting in the egress queue
Design Considerations
Congestion-detection is based on the PMON counters which are monitored

at the interfaces level
Any device behind an interface, upon slow detection will be marked as a
slow-device
Upon logging back in, will detected as a slow device again on same interface
NPV/ FDISC: Devices connected to a slow interface will be marked slow
TF port-channel: All devices behind the TF port-channel will be marked slow
One of the VL is reserved for QoS based HBA, Zone QoS
Devices involved with IVR flows, FCR, FCIP and FCoE are not honored by this
feature
Anti-Counterfeit Technology
• Built-in ACT2 Chip to identify genuine Hardware
• Based on Secure Unique Device Identifier certificate
• SUDI (Secure Unique Device Identifier) is the combination of PID (Product ID)
+ SN (Serial Number) of module
• NX-OS reads PID and SN from ID-SPROM of module during boot up
• NX-OS authenticates the module’s PID & SN in ID-SPROM against the SUDI
in ACT2 chip
• Upon authentication failure, module will fail to load with error message:
ACT2_AUTH_FAIL: ACT2 test has failed on module 9 with
error : ACT2 authentication failure
Secure Boot
• Traditional mode: Boot Code Integrity not Validated
• Cisco Secure Boot: Boot Code Integrity is validated by Hardware
Traditional Boot Cisco Secure Boot
Step 1 Step 2 Step 1 Step 2 Step 3 Step 4

Boot Code CPU CPU Boot Code Hardware Anchor CPU CPU CPU
Bootloader NX-OS Micro Micro Bootloader NX-OS
Loader Loader
Bootloader OS Launched : Image Signing Micro Loader Bootloader OS Launched

Checks OS
checks boot loader Checks OS
: Integrity Checks
Secure Boot
• BIOS Chip is programmed with Secure Boot Key
• During Module boot up, board FPGA validates SB-Key
• Upon failure of SB-Key validation, Module will fail to boot
• BIOS tampering will invalidate the module
Diagnostic Enhancements
• Generate test traffic pattern from at MAC level
• Generate loopback at destination port to support D-port

and SYM-port functionality
• Ability to track packet drops on a SID/DID/VSAN basis
• Ability to track the packet latency in a switch
Smart Arbiter
• Enhances arbiter chip function

• Smart arbiter will implement a “smart” request-generation feature
in which generation of request to central arbiter is delayed until a
superframe forms or programmable time
RDP: Read Diagnostics
Parameter
What is Read Diagnostics Parameter (RDP) ?
• It is an ELS command which requests for diagnostics parameters for
an interface were the specified FCID is connected
• Communicates with LC module in identifying the SFP parameters for
the interface were the specified FCID is connected/logged in
• It consists of following parameters
• Control Frame
• ELS Frame
• Request-Response
• Diagnostics parameters for an N_Port_ID (FCID)
Read Diagnostics Parameter: The need
• Helps proactively identify
• Link Diagnostics
• Link Errors
• Degraded conditions
• Exchange issues
RDP Parameters Checklist
Port Speed
Port speed capabilities SFP diagnostics
Port operating speed Optical Elements Vcc
(Voltage, Temp, Current, Temp
Link Errors
Tx power, Rx Power) Tx Bias
Link failure
LOSync High Alarm/Warning Tx power
LOS Low Alarm/Warning Rx power

Primitive sequence FEC status
Optical Product Data
Invalid transmission word Wavelength
Vendor name
Invalid CRC SFP type
Model Number
Serial Number Port type
Buffer Credits
Switch B2B credit Revision Number FEC
Attached Device B2B Manufacture date Corrected blocks
credit Uncorrected blocks
Return Trip Time (RTT)
Use Cases
RDP
RDP
Case #2
FCID: Case #1
XX1122 S1 S2 FCID:
Dom: XX Dom: YY
ISL YY1177
FLOGI
H
PLOGI RDP Case #3
PRLI T
Case #1 Case#2
Case#3
(Local) (Remote) (Target)
SID XX1122 SID XX1122 SID XX1122
DID FFFFFE DID FFFCYY DID YY1177
Payload XX1122 Payload YY1177 Payload YY1177
PLOGI Local PLOGI Remote PLOGI End Device
Deployment Recommendations
• Included with Base License, enabled by default
• Supported on MDS 16G and 32G Platforms
• No Support for MDS 95xx, 9124, 9132, 9148, 9222i
RDP Control Flow Get
RDP
Duplicate Yes is PC Yes

Request ? or NPV?
No No
Send Get Err

is Device RJT Details Timeout
Logged? No
Yes Send
Resp
Get Err
Ifindex Timeout
Limitations
• HBA PLOGI is required before an RDP
• Not Supported:
• Port-Channel
• NPV Uplink
• FCoE / FCIP ports
Virtual Machine ID (VMID)
VM- VM- VM-
Why VMID ?
VM-
1 2 3 4
VN_Po
VN_Port Hypervisor rt
PN_Port
• Multiple VM’s share the same VN_port

• The fabric only knows the FCID associated with the VN_port
• Advantages:
• When VM moves, no new FCID is needed
• Possible Multipathing situation
• Granular VM visibility:
• Provide VM visibility within SAN fabric for VM’s behind hypervisor
• Enforce VM specific policies
VMID Overview
• Based on T11 standard
• Allows to mark VM originating from HBA
• CS_CTL field in the fc2 frame is leveraged to carry the 8 bit tag
• MDS 32G FC module can look up acl based on the incoming tag
VMID Flow process
• CS_CTL will be used with sid,did and vsan in the acl flow key
• HBA’s register the VM’s hosted behind a given FCID on the
hypervisor
• VMIS service on the switch maintains this per VM information as a
database
• Each VM is identified using a combination of Global VE ID, FCID and
Local VE ID
• Per FCID VM database is distributed across the fabric
Supported hardware and software platforms
• Supported across following platforms
• MDS 9700
• MDS 9132T
• MDS 9250i
• MDS 9148S
• Minimum NX-OS required: NX-OS 8.2(1)
Enabling VMID Feature
• Enable / Disable:
switch(config)# feature vmis
• Configure VMID range per vsan basis
vmis range <1-255> vsan <vsan-id>
show flogi database details
show flogi database details
show tech-support vmis
show vmis database [ domain | fcid | global-vmid |
interface | local | vem | vmotion | vsan ]
Important notes
• VMID range configuration is available only at per vsan basis
• Persistence is available only for local entries
• If vmis process crashes/restarts, will have to re-learn from the
switches in remote domains
• If FC Management security is enabled management station pwwn
needs to be configured for Unzoned name server read operation
• When a vmotion occurs VM de-instantiation will not occur
immediately
Buffer credit recovery
What is it - Buffer credit recovery?
• Lost credit:
• SoF (Start of Frame) is corrupted
• R_RDY is corrupted
• Detects and corrects the lost credits, non-disruptively

• Negotiated between two directly attached peer ports
• Enabled automatically during initial negotiation
• Per-hop feature
Functionality
• Can be used on any non-arbitrated loop link
• Most useful on unreliable links, such as MANs or WANs
• Can also help on shorter, high loss links, link with a faulty fiber
connection
• Process
• Local and port peer send checkpoint primitives during initial negotiation
• If frame loss is detected, sends R_RDY to replenish lost credits
• If R_RDY loss is detected, port internally replenishes lost credits
Implementation
• Recovery implementation
• Buffer-to-buffer state change SoF (BB_SCs) are transmitted every 2
BB_SC_N frames sent
• Enables attached port to determine if any frames are lost
• Upon frame loss detection, the receiver transmits the appropriate number
of R_RDYs to compensate for the lost frames
• Buffer-to-buffer state change R_RDY (BB_SCs) are transmitted every 2
R_RDY primitives
• Enables port to determine if any R_RDY primitives are lost
• If R_RDY primitive loss is detected, the receiver increments the number of
transmit credits to compensate for the lost R_RDYs
Limitations
• E Ports:
• Enabled by default on ISLs
• Works on an ISL between a Cisco MDS switch and a peer switch from
any vendor, provided this feature is supported on the peer switch.
• Supported only on links that are in R_RDY flow control mode
• F Port:
• Enabled by default
• Enabled on an F port between a Cisco MDS switch and a peer device
from any vendor
• N-Port ID Virtualization (NPIV) ports do not support buffer-to-buffer credit
recovery for Cisco N-Port Virtualizer (Cisco NPV) switch logins
Configuration
• Configuration example
• switch# configure terminal
• switch(config)# interface fc 1/1
• switch(config-if)# switchport fcbbscn
• This example shows how to configure default credits on an interface
• switch# configure terminal
• switch(config)# interface fc 1/1
• switch(config-if)# switchport fcrxbbcredit default
• switch(config-if)# switchport fcrxbbcredit 2000
Verification
switch# show flow-control r_rdy module 3
fc3/17
Fc3/18
switch# show interface fc2/17 bbcredit

fc2/17 is trunking
Transmit B2B Credit is 255
Receive B2B Credit is 12
Receive B2B Credit performance buffers is 375
12 receive B2B credit remaining
255 transmit B2B credit remaining
Troubleshooting
• Use the show logging onboard interrupt-stats command to view the
number of times a port sent extra R_RDYs or incremented transmit
buffer to buffer credits to restore credit count
switch# show logging onboard interrupt-stats
-------------------------------------------------------------------------------
INTERRUPT COUNTS INFORMATION FOR DEVICE: FCMAC
-------------------------------------------------------------------------------
Interface|| Interrupt Counter Name | Count | Time Stamp MM/DD/YY HH:MM:SS | | |
-------------------------------------------------------------------------------
fc1/1 |IP_FCMAC_INTR_ERR_BB_SCR_INCREMENT |1 |01/01/17 20:00:00
fc1/1 |IP_FCMAC_INTR_ERR_BB_SCS_RESEND |1 |01/01/17 10:00:00
Forward Error Correction
(FEC) enhancements
FEC/TTS description
• A technology used to correct errors in data transmission over unreliable or noisy environment
• Can correct upto 11bits in a 2112 bits frame
• Implemented at hardware level in MDS
• Optional @ 16G FC, Must for 32G FC
• Advantages of use:
• Allows limited error correction
• Reduce number of packet drops
• Proactively identify physical layer issues
• No Performance impact / No bit-rate change
FEC counters
• Noisy media increases FEC counters
• Indicates corruption either in FC primitives (or) FC frames.
• Counters to monitor:
• FCP_PCS_CNT_STATS_ERRORS_RX_FEC_CORRECTED_BLOCKS
• 32-bit counter that counts the number of blocks corrected by the FEC Decoder. (Hardware counter name:
fec_corrected_block_counter)
• FCP_PCS_CNT_STATS_ERRORS_RX_FEC_UNCORRECTED_BLOCKS
• 32-bit counter that counts the number of blocks the FEC Decoder was unable to correct (Hardware counter name:
fec_uncorrected_block_counter)
FEC configuration
configure t
interface fc1/1
switchport mode E (or) F
switchport speed 16000
switchport speed fec
switchport speed fec tts // FEC encryption/decryption enabled after TTS negotiated
Caveats:
• FEC doesn’t work with “switchport speed auto” configuration
• TTS-nego is mandatory with HBAs supporting FEC
• FEC & TTS-nego are optional on ISL ports
FEC Verification
switch(config-if)# sh int fc1/1 | i fc|fec|speed
fc1/1 is up
Speed is 16 Gbps
admin fec state is up
oper fec state is up
switch(config-if)# show interface fc7/15 cou details | i fc|fec

fc7/15
0 fec corrected blocks
0 fec uncorrected blocks
module-7# show hardware internal statistics device fcmac all port 17 | i FEC
FCP_PCS_CNT_STATS_ERRORS_RX_FEC_CORRECTED_BLOCKS 0000000000000002 17 -
FCP_PCS_CNT_STATS_ERRORS_RX_FEC_UNCORRECTED_BLOCKS 0000000000000007 17 –
switch(config-if)# show interface fc7/15 cou details | i fc|fec

0 fec corrected blocks
0 fec uncorrected blocks
FEC cli: counters
module-7# show hardware internal statistics device fcmac all port 17 | i FEC
FCP_PCS_CNT_STATS_ERRORS_RX_FEC_CORRECTED_BLOCKS 0000000000000002 17 -
FCP_PCS_CNT_STATS_ERRORS_RX_FEC_UNCORRECTED_BLOCKS 0000000000000007 17 –
module-1# sh ha int fcmac port 1 port-info | i FEC

FEC admin/oper mode : ADMIN UP/SUCCESS
FEC tts cfg/sta : ADMIN UP/SUCCESS
module-1# show port-config internal link-events

Time PortNo Speed Event Reason
---- ------ ----- ----- ------
Feb 6 04:39:04 2014 00018794 fc7/17 16G LR Success
Feb 6 04:39:03 2014 00258935 fc7/17 16G UP Not FL
Feb 6 04:39:03 2014 00258583 fc7/17 --- FEC_UP neg-succ
Feb 6 04:38:59 2014 00799923 fc7/17 --- DOWN Disable
Supported hardware and software releases
• 16G FC Module / MDS 24/10G FCIP Module
• Switchport fec
• Integrated from 6.2(7)
• Switchport fec tts
• Introduced in 6.2(11c) (Not available in 6.2(13))
• Available in all releases from 6.2(15)
• 32G FC Module
• Switchport fec [tts] integrated in NX-OS release 8.1(1) onwards
• MDS 32G 32-Port Switch
• NX-OS 8.2(1) onwards
• Supported HBAs
• Qlogic, Emulex, FICON host
HPE Product Offerings
Cisco MDS 32Gb FC Module & 32Gb SFPs
HPE P/N HPE Product Names (Long, Short) Cisco PN/PID Cisco Description
• HPE C-series 32 Gb Fibre Channel Short Wave SFP+ Transceiver
Q9D30A DS-SFP-FC32G-SW= 32 Gbps Fibre Channel SW SFP+, LC
• HPE C-series 32Gb FC SW SFP+ Transceiver
• HPE C-series 32 Gb Fibre Channel Long Wave SFP+ Transceiver

Q9D31A DS-SFP-FC32G-LW= 32 Gbps Fibre Channel LW SFP+, LC
• HPE C-series 32Gb FC LW SFP+ Transceiver
• HPE StoreFabric SN8500C 48-port 32Gb Fibre Channel Director Module MDS 9700 48-Port 32-Gbps Fibre
Q9D32A DS-X9648-1536HK9=
• HPE SN8500C 48-port 32Gb FC Dir Module Channel Switching Module, spare
Please refer to the HPE StoreFabric SN8500C Director QuickSpec for additional information
HPE Product Offerings – Cisco MDS 9132T Switch
(HPE StoreFabric SN6610C 32Gb FC Switch)
HPE P/N HPE Product Names (Long, Short) Cisco PN/PID Cisco Description
• HPE StoreFabric SN6610C 32Gb 8-port 16Gb Short Wave SFP+ Fibre MDS 9132T 32G FC switch, w/ 8 active ports
Q9D34A Channel Switch DS-C9132T-8PMESK9 + 8x16G SW Optics, 2 Fans, 1 PSUs, Port Side
• HPE SN6610C 32Gb 8p 16Gb SFP+ FC Swch Exhaust
• HPE StoreFabric SN6610C 32Gb 8-port 32Gb Short Wave SFP+ Fibre MDS 9132T 32G FC switch, w/ 8 active ports
Q9D35A Channel Switch DS-C9132T-8PMETK9 + 8x32G SW Optics, 2 Fans, 1 PSUs, Port Side
• HPE SN6610C 32Gb 8p 32Gb SFP+ FC Swch Exhaust
• HPE StoreFabric SN6610C 32Gb 24-port 16Gb Short Wave SFP+ Fibre MDS 9132T 32G FC Enterprise switch, w/ 24
Q9D36A Channel Enterprise Switch DS-C9132T-24PESK9 active ports + 24x16G SW Optics, Enterprise
• HPE SN6610C 32G 24p 16G SFP+ FC Ent Swch License, 4 Fans, 2 PSUs, Port Side Exhaust
• HPE StoreFabric SN6610C 32Gb 8-port Fibre Channel Upgrade E-LTU MDS 9132T 32G FC switch 8 Port Activation
Q9Z41AAE M9132T-PL8=
• HPE SN6610C 32Gb 8-port FC Upgrade E-LTU License for Base eDelivery, spare
• HPE StoreFabric SN6610C 16-port Fibre Channel Expansion Module MDS 32G FC Expansion module, w/ 16 active
Q9D33A M9XT-FC1632H=
• HPE SN6610C 16-port FC Expansion Module ports, Spare
• HPE StoreFabric SN6610C 650W Power Supply
Q9D37A DS-CAC-650W-E= AC PSU Port side Exhaust, Spare
• HPE SN6610C 650W Power Supply
• HPE StoreFabric SN6610C Fan Tray
Q9D38A DS-C32S-FAN-E= MDS 9132 FAN tray, port side Exhaust Spare
• HPE SN6610C Fan Tray
Please refer to the HPE StoreFabric SN6610C 32Gb FC Switch QuickSpec for additional information
HPE Product Spares – SFPs, Director Module, Switch*
HPE FRU P/N HPE FRU Description Cisco PN/PID Cisco Description
P06958-001 SPS-MDS 32Gb FC SW SFP+ Transceiver DS-SFP-FC32G-SW= 32 Gbps Fibre Channel SW SFP+, LC
P06959-001 SPS-MDS 32Gb FC LW SFP+ Transceiver DS-SFP-FC32G-LW= 32 Gbps Fibre Channel LW SFP+, LC
MDS 9700 48-Port 32-Gbps Fibre Channel Switching Module,
P06960-001 SPS-MDS SN8500C 48-p. 32Gb FC Dir Module DS-X9648-1536K9=
spare
P06993-001 SPS-MDS SN6610C 16-port FC Expan. Module M9XT-FC1632= MDS 32G FC Expansion module, w/ 16 active ports, Spare
MDS 9132T 32G 1 RU FC switch, w/ 8 active FC ports, 2
P06992-001 SPS-MDS SN6610C 32Gb 8a. ports FC Switch DS-C9132T-MEK9=
Fans, 1 PSU, Port Side Exhaust, Spare
P06994-001 SPS-MDS SN6610C 650W Power Supply DS-CAC-650W-E= AC PSU Port side Exhaust, Spare
P06995-001 SPS-MDS SN6610C Fan Tray DS-C32S-FAN-E= MDS 9132 FAN tray, port side Exhaust Spare
* All switch bundles share the same spare

p/n

Cisco MDS 32G HPE OSM Training - 062018 PDF

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Cisco MDS 32G HPE OSM Training - 062018 PDF

Загружено:

Авторское право:

Доступные форматы

Cisco MDS 32G FC SAN Solutions

MDS 32G FC module & Entry Level 32G FC SAN Switch

• All roadmaps are displayed in calendar year (CY) quarters.

• Information contained in this document regarding future technology, future

• The information contained in this document is Cisco Partner Confidential

- For Partners and Customers with Nondisclosure Agreements

Eject Push Eject Lever

• 48 x 32-Gbps Fibre Channel line-rate ports

• Top row: Odd-numbered ports

16-Member Port Group

Frame Size 1 Gbps 2 Gbps 4 Gbps 8 Gbps 10 Gbps 16 Gbps 32 Gbps

512 bytes 2 B2B/km 4 B2B/km 8 B2B/km 16 B2B/km 24 B2B/km 32 B2B/km 64 B2B/km

1024 bytes 1 B2B/km 2 B2B/km 4 B2B/km 8 B2B/km 12 B2B/km 16 B2B/km 32 B2B/km

2 Gbps 125 km 250 km 500 km

4 Gbps 62 km 125 km 250 km

2 Gbps 2050 km 4100 km 8200 km

4 Gbps 1025 km 2050 km 4100 km

8 Gbps 512 km 1025 km 2050 km

16 Gbps 256 km 512 km 1025 km

32 Gbps 128 km 256 km 512 km

Crossbar ASIC -1 Crossbar ASIC -2

F32 ASIC -1 F32 ASIC -2 F32 ASIC -3

• F32 ASIC has internal temperature sensor circuitry that is

Green • Operation is normal.

• The module is booting or running diagnostics.

• The diagnostics test failed.

• The module is resetting.

Green The port is active.

Blinking Orange The port is faulty and has been disabled.

Off The port is not active or the link is not connected.

Off The module is not selected.

Port-group size 4 ports 16 ports

Ports per ASIC 8 ports 16 ports

ASICs per module 6 3

Analytics capability Absent Present

Anti-Counterfeit Technology Absent Present

Secure Boot Absent Present

Cisco® MDS 9700 Series Directors

16- and 32-Gbps FC

Cisco® MDS 9700 Series Directors

Cisco® MDS 9700 Series Directors

16- and 32-Gbps FC

FC-SCSI Servers NVMe-FC Servers

Hardware Feature Cisco MDS 9706 Cisco MDS 9710

• Base Switch: 16 x 32G FC line-rate ports with one expansion slot

• Dedicated Network Processor for SAN Analytics*

• Base chassis contains one IO-ASIC, CPU, NPU processor

• LEM module contains additional dedicated IO-ASIC

• Required NX-OS: 8.2(1) or later

• DCNM Management: Required DCNM version: 10.4(1) or later

*: Future Software Support

• Hot swappable PSUs

• Switch Status (tri-color)

• Power Supply Status (tri-color)

• Fan Status (tri-color)

• Ethernet Management Port Link (bi-color)

• Ethernet management Port Activity (bi-color)

• Fibre Channel Port Link (one tri-color LED per port)

• 2x 10/100/1000M-baseT Ethernet management ports

• Supported port modes: F, E, TE, SD, ST

• Up to 1024G FC front panel bandwidth

• Incremental port options

• Based on same IO ASIC as 32G module of MDS 9700 platform