Вы находитесь на странице: 1из 117

In-depth and personal with the Cisco Nexus

2000 Fabric Extender Architectures, Features,


and Topologies
Pavan S Kaushik
Technical Marketing Engineer
BRKARC-3454
Never heard of FEX before ??
No Problem !!

FEX Gurus in the house ??


No Problem !!

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Session Goals

Brief Nexus 5000/6000/7000/9000 Parent platform introduction.


Discuss Nexus 2000 platform architectures, supported parent switches,
actual designs and best practices.
Nexus 2000 topologies, features and implementation.
vPC/FCoE/FC/FP/QoS/ACI Deep dive and troubleshooting will not be
covered in this session.

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
House Rules

We have Nexus 2300 switches in the Room !! Do take a look.

Keep the session interactive.

Highly detail-oriented questions will be taken offline(ex. Optics/Model


compatibility etc).

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Agenda
Introduction
Parent switches overview
Platform overview
Features deep-dive and packet-walk
Design and topologies
Configuration and Provisioning
Connectivity options and Optics
Summary
Introduction
Nexus 2000 / FEX
Cisco Nexus 2000 Fabric EXtenders

Modular Switch Parent Switch*


In-chassis Linecards With Remote Linecards (FEX)


* - Parent switch can be Fixed or Modular
BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
FEX: Virtualized Access Switching
Extending the boundaries of Data Center Fabric

Flexible and simplified top-of-rack (ToR) designs


Simplified Management, Centralized configuration
Multiple Parent Switch support N5K, N6K, N7K, N9K
Unified Ports provide flexible LAN and SAN deployment

...
Virtualized Switch

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Nexus 2000 Solution
Low TCO
Low OPEX
Low CAPEX

Simplified
Lower Devices to Manage
Operation
Central Point of Management
Nexus 2000 Architecture with Single Point
of Management
Traditional Architecture Architecture
Agnostic(ACI, FA) Flexibility

Enhanced Buffers
Low over subscription
Enhanced
High Port Density Performance

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Parent switches overview :
Nexus 5000/6000/7000/9000
Nexus 2000 - Parents Switches Family
Fixed Switching Platforms Modular Switching Platforms

Nexus 5600
Nexus 2200/ Nexus 6000 Nexus 7000 Nexus 7700
Nexus 2300

Nexus 5000/5500

Application Centric Infrastructure (ACI)


Nexus 9000 Standalone

Nexus 9000

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
NEXUS 5600 SERIES
Nexus 5672UP Nexus 56128P Nexus 5624Q Nexus 5648Q Nexus 5696Q
7.0(1)N1(1) 7.0(2)N1(1) 7.1(0)N1(1a) 7.1(1)N1(1) 7.0(4)N1(1)
Switch Fabric
1.44 Tbps 2.56 Tbps 1.92 Tbps 3.84 Tbps 7.68 Tbps
Throughput
PERFORMANCE

Port-to-Port Latency ~1.0 us

Layer 3 Capability Integrated Line-Rate

Switch Footprint 1RU 2RU 1RU 2RU 4RU


10 Gigabit Ethernet
72 128 96 192 384
Port
DENSITY

40 Gigabit Ethernet
6 8 24 48 96
Port
Unified Ports 16 48 N/A N/A 48

1/10 Gigabit Ethernet


1152 1152 1152 1152 2304
With FEX
SCALE

Packet Buffer 25MB per 3x40G (or 12x10G)

FEX Support Yes* Yes* Yes* Yes* Yes*

* Verify model-specific compatibility through DataSheets / ReleaseNotes / Hardware Installation Guides

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Nexus 7000 Product Family
Shipping up to 384x40G ports and 192x100G ports
Nexus 7000 Family
F3 Modules
768 x 10-Gbps Ports or
384 x 40-Gbps Ports or
768 x 10-Gbps Ports* or 192 x 100-Gbps Ports
192 x 40-Gbps Ports

384 x 10-Gbps Ports* or 384 x 10-Gbps Ports or


96 x 40-Gbps Ports 192 x 40-Gbps Ports or
96 x 100-Gbps Ports
336 x 10-Gbps Ports *
84 x 40-Gbps Ports
192 x 10-Gbps Ports or
96 x 40-Gbps Ports or
96 x 10-Gbps Ports * 48 x 100-Gbps Ports
24 x 40-Gbps Ports 48x 10-Gbps Ports or
24x 40-Gbps Ports or

Nexus 7010 Nexus 7018


Nexus 7009 Nexus 7718
Nexus 7004 Nexus 7710
Nexus 7706 Nexus 7002

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Nexus 7000 / 7700 Line Card Modules

M1 1G/10G M2 10G / 40G / 100G M3 1G / 10G / 40G

F2E 10G F3 40G


F1 10G F2 10G

F2E 10G
F3 10G / 40G / 100G

F3 closes the F/M feature gap!

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Nexus 7000/7700 Supervisor FEX support
Nexus 7000 SUP2 Nexus 7000 SUP2E Nexus 7700 SUP2E

CPU Quad-Core Xeon Dual Quad-Core Xeon Dual Quad-Core Xeon

Speed (GHz) 2.13 2.13 2.13

Memory (GB) 12 32 32

Max VDCs 4 (user) +1 (admin) 8 (user) +1 (admin) 8 (user) +1 (admin)

Max VLANs per FEX 2000 2000 2000

Max VLANs per FEX HIF 75 75 75

Max. no. of FEX/Ports 32 FEX /1536 Ports 64 FEX /3072 Ports 64 FEX /3072 Ports

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
NEXUS 7000 Line Card FEX support
M2-24XP M2-06FQ F2-48XP F2E-48XP F3-12FQ F2E-48XP F3-48XP F3-24FQ
Chassis
Nexus 7000 Nexus 7000 Nexus 7000 Nexus 7000 Nexus 7000 Nexus 7700 Nexus 7700 Nexus 7700
Supported
Ports and
24x10GE 6x40GE 48x1/10GE 48x1/10GE 12x40GE 48X1/10GE 48x1/10GE 24x40GE
Speed

Interfaces SFP+ QSFP+ SFP/SFP+ SFP/SFP+ QSFP+ SFP/SFP+ SFP/SFP+ QSFP+

Max Fabric
3 3 5 5 3 3 3 5
Cards and BW
240 240 480 480 480 480 480 960
(GBps)
Performance
120 720 720 720 720 720 1440
(Mpps) 120

vPC (Support) Yes Yes Yes Yes Yes Yes Yes Yes

FCoE, FP
No Yes Yes Yes Yes Yes Yes Yes
Support

Max. Power (W) 795 795 450 450 500 450 450 650

FEX Support Yes* Yes* Yes* Yes* Yes* Yes* Yes* Yes*

* Verify model-specific compatibility through DataSheets / Release Notes / Hardware Installation Guides

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Cisco Nexus 9500 Platform Switches
Density in DC Optimized Footprint
Cisco Nexus 9500
16-Slot

Fabric Bandwidth:
3.2 Tbps/ slot
8-Slot

4-Slot
Cloud Deployment Options:
21 RU

13 RU

7 RU
ACI Multi-Site, Telemetry,
Cloud Scale

Nexus 9516 Nexus 9508 Nexus 9504 Multi-Generation


Payload Slots 16 8 4 Investment Protection:
Capacity 57.6 Tbps 28.8 Tbps 14.4 Tbps No Mid-plane, Power Supply Headroom
100G Density 512 256 128 for 100/400G and integrated optics/
50G Density 1024 512 256 encryption
40G Density 576 288 144

10/25G Density 2304/2048 1152/1024 576/512

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Nexus 9500 Supervisor FEX support
Supervisor A Supervisor B

CPU 4 Core, 1.8GHz 6 Core, 2.2GHz

Memory (GB) 16 GB 24GB

Storage (SSD) 64 GB 256GB

Max. Power (W) 80 90

FEX Support 32 FEX /1536 Ports 32 FEX / 1536 Ports

VLANs per FEX 2000 2000

VLANs per FEX HIF 75 75

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
NEXUS 9500 Line Card FEX support
X9636PQ X9536PQ X9564PX X9432PQ X9464PX
Chassis Supported Nexus 9504/9508 Nexus 9500 Nexus 9500 Nexus 9500 Nexus 9500
48x1/10GB, 48x1/10GB,
Ports and Speed 36x40GB 36x40GB 32x40GB
4x40GB 4x40GB
SFP/SFP+ SFP/SFP+
Interfaces QSFP+ QSFP+ QSFP+
QSFP QSFP+
Performance
120 720 720 720 720
(Mpps)

vPC (Support) Yes Yes Yes Yes Yes

Buffer 36 MB 104 MB 104 MB 24 MB 12 MB

Max. Power (W) 795 450 450 310 500

FEX Support Yes* Yes* Yes* Yes* Yes*


* Verify model-specific compatibility through DataSheets / Release Notes / Hardware Installation Guides

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Nexus 9300 Fixed Series
ACI Leaf and NX-OS Switches
Analytics
Quad-core CPU
64 GB SSD
Integrated Uplink | CY14-15 24 GB Memory

9332 QSFP+ (32x40G)


32p QSFP Switch
Modular Uplink | 2HCY13 28p 40G + 4p 100G Q4CY16
24p 40G + 6p 100G

9396 SFP+(48x10+12x40G) 9372(E) SFP+ (48x10+6x40G)

93180YC-EX (48x25G+6x100G)

9372 (E) Copper (48x10+6x40G)


9396 Copper (48x10+12x40G) + NEW
93108TC-EX (48x10GT+6x100G) Q2CY16

93120 Copper (96x10+6x40G)


93128 Copper (96x10+8x40G)

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Nexus 9300 Switches FEX support
Nexus 9332PQ Nexus 9372PX Nexus 9396PX

Form Factor (RU) 1 1 2

Throughput(Tbps) 2.56 1.44 1.44

1/10GB Ports X 48 48

40 GB Ports 32 6 12

Latency (us) <2 <2 <2

Buffer 25 MB 25 MB 50 MB

Max. Power (W) 650 650 650

Applications ACI Leaf ACI Leaf ACI Leaf

FEX Support Yes Yes Yes


* Verify model-specific compatibility through DataSheets / Release Notes / Hardware Installation Guides

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
FEX Scalability
Nexus Nexus Nexus Nexus Nexus Nexus Nexus
5500 5600 5696 7000 7700 9300 9500
L3 16 24 32 64 64 16 32
L2 24 24 48 64 64 16 32
Cisco Parent Switch
Cisco Nexus 2000 FEX

16 - 64
FEX
+

Supports 3000+ ports per single point of management


2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Platform Overview:
Nexus 2000 / FEX
Nexus 2000 / FEX Naming Convention
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

N2K-C2232PP-10GE
13: Cisco Nexus 2000 Series Identifier
4: Dash
5: Chassis
6: Series Number
7: Series Generation
89: Number of Ports
10: Host Physical Interface Type (P = Requires SFP+, T = Copper, UP = Unified Port, F = Includes FET)
11: Uplink Physical Interface Type (P = Requires SFP+, Q = QSFP, T = Copper, F = Includes FET, M = Module)
12: Dash
1316: Host Connection Speed, Enhanced model or Bundle identifier (-BUN)

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
History of Cisco Nexus 2000 Series
Simplified Operations with Revolutionary Scale
Base-T FEX SFP/SFP+ FEX

N2K-C2348TQ / TQ-E
(40G Uplink, Enhanced Buffers)
N2K-C2348UPQ-1/10GE
(40GE Uplink, Enhanced Buffers)

N2K-C2332TQ-1/10/40GE
(40G Uplink, Enhanced Buffers)

N2K-C2248PQ 10GE
(QSFP Uplink)
B22 IBM
Blade FEX
N2K-C2232TM-E-10GE 1/10G FEX for
IBM Blade servers

N2K-C2232PP-10GE
B22 FTS FEX B22 Dell FEX
N2K-C2248TP-E-1GE 1/10G FEX for 1/10G FEX for
(Enhanced Buffers) B22 HP FEX Fujitsu Blade servers Dell Blade servers
1/10G FEX for
HP Blade servers
BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Nexus 2000 100M/1GBASE-T
Nexus 2224TP Nexus 2248TP Nexus 2248TP-E

Form Factor (RU) 1 1 1

Network Interfaces 2 (SFP+) 4 (SFP+) 4 (SFP+)

Host Interfaces 24 48 48

Performance 88Gbps 176Gbps 176Gbps

FCoE Support No No No

Buffer 8 MB 8 MB 32 MB

Max. Power (W) 80 W 95 W 95 W

Oversubscription 1.2:1 1.2:1 1.2:1

Nexus Parent Switches Nexus 5000, 6000, 7000, 9000 Nexus 5000, 6000, 7000, 9000 Nexus 5000, 6000, 7000, 9000

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Nexus 2000 1/10GBASE-T
Nexus 2232TM Nexus 2232TM-E Nexus 2332TQ Nexus 2348TQ/E*

Form Factor (RU) 1 1 1 1

Network
8 (SFP+) 8 (SFP+) 4 (QSFP+) 6 (QSFP+)
Interfaces

Host Interfaces 32 32 32 48

Performance 560Gbps 560Gbps 1.2Tbps 1.2Tbps

FCoE Support No No No Yes (FCoE)

Buffer 8 MB 8 MB 32 MB 32 MB

Max. Power (W) 350 W 240 W 350 W 350 W

Oversubscription 4:1 4:1 2:1 2:1

Nexus Parent Nexus 5000, 6000, 7000, Nexus 5000, 6000, 7000, Nexus 5000, 6000, Nexus 5000, 6000, 7000,
Switches 9000 9000 7000, 9000 9000
* Some N2348TQ-E items are on the roadmap

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Nexus 2000 10G (SFP+)
Nexus 2232PP Nexus 2248PQ Nexus 2348UPQ

Form Factor (RU) 1 1 1

Network Interfaces 8 (SFP+) 4 (QSFP+) 6 (QSFP+)

Host Interfaces 32 48 48

Performance 560Gbps 960Gbps 1.2Tbps

FCoE Support Yes Yes Yes

Buffer 8 MB 32 MB N2H, 8 MB H2N 32 MB

Max. Power (W) 210 W 175 W 350 W

Oversubscription 4:1 3:1 2:1

Nexus 5000, 6000, 7000, 9000


Nexus Parent Switches Nexus 5000, 6000, 7000, 9000 Nexus 5000, 6000, 7000, 9000
UCS FI

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Nexus 2348UPQ Overview
PORT SIDE OF FEX

48 Fixed 1/10G SFP+ Unified Capable Ports. 6x40G QSFP+ Ports


48x1/10G SFP+ Ports
Traditional Ethernet or FCoE
48 Unified Ports provide up to 16GFC
32M of Buffer with Lower Oversubscription ratio of 2:1
Parent Switch N5k, N6k, N7k, N9k.

Choice of Port Side


Intake/Exhaust Airflow
FAN SIDE OF FEX

Redundant 1+1 Redundant N+1 Fan


Modules
AC/DC Power Supplies Unit

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Nexus 2348UPQ Buffer Allocation
Total 32MB shared by 6 NIF(Network
Interface) and 48 HIF(Host Interface)

Dedicated buffer pool and shared buffer NIF 1-6x40GE

pool Each of the NIF or HIF can manage


all 32MB buffer dedicated
32MB
Queue-limits can be used to control shared
buffer-usage by bursty flows.
HIF 1-48

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Nexus 2000 Internal Architecture
Nexus 2248PQ vs. Nexus 2348UPQ

4x 40Gbe QSFP+ 6x 40Gbe QSFP+


(4x10G-Mode only) (40G or 4x10G-Mode)

QSFP QSFP QSFP QSFP QSFP QSFP QSFP QSFP QSFP QSFP

Uplink/Buffer Uplink/Buffer

3rd Generation ASIC

ASIC A ASIC B

Port 1 - 48
Port 1 - 24 Port 25 - 48
(Unified Ports: Ethernet/FCoE/FC)
(Ethernet/FCoE) (Ethernet/FCoE)
* For all N23XX : max. 8 x 10GE Uplink Port-Channel bundle

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Cisco Nexus 2348UPQ
Compared with other Nexus 2000 1/10GE Fabric Extender

Model Nexus 2232PP Nexus 2248PQ Nexus 2348UPQ

48x SFP/SFP+ (1/10G)


Host Ports 32x SFP/SFP+ (1/10G) 48 x SFP/SFP+ (1/10G)
(true 40G or 4x 10G-Mode)

4x 40Gbe QSFP+ 6x 40Gbe QSFP+*


Uplink Ports 8x 10GbE SFP+
(4x 10G-Mode only) (true 40G or 4x 10G-Mode)

FCoE
157.5KB per port
Buffer 157.5KB per port 32MB Shared Buffer
2x 16M Shared Buffer (Uplink)

* For all N23XX : max. 8 x 10GE Uplink Port-Channel bundle

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Fabric Extender 100M/1G/10G Copper
Nexus 2348TQ-E
48 port 3rd Generation Fabric Extender

Nexus 23XX FEX Uplink support


Cost optimized version of N2K-C2348TQ.
Max. 6 x 40GE Uplinks
Identical to N2K-C2348TQ in features.
or
48 x 100M/1G/10G host/downlink interfaces
Max. 8 x 10GE Uplinks
Large 32MB Buffer
(throught breakout cables)
Supports FCoE for LAN/SAN convergence.

48 100/1000 RJ45 Downlinks


BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Fabric Extender 100M/1GE
Nexus 2248TP-E-GE Overview
48 ports 100M/1G-BaseT
Design scenario:
Enhanced Fabric Extender
High density 100M/1G access
48x 100/1000M host/downlink Cost effective 100M/1G solution
interfaces.
Buffer optimized for specific Data Center
4x 10GE on network/uplink interfaces workloads such as Big Data, Hadoop and
Upstream N5K supports various FEX Distributed Storage
(mix-and-match)
Key differences from N2248TP:
6 user-defined QoS queues (8 QoS Optimized Buffers
queues) Enhanced Debug counters

48 100/1000 RJ45 Downlinks


BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Nexus 2248TP-E Buffer Allocation
Total 32MB shared by 4 NIF(Network
Interface) and 48 HIF(Host Interface)
Dedicated buffer pool and shared buffer
pool. Size of each pool is configurable by
software NIF 1-4x10GE

Current 128KB/port dedicated at NIF. dedicated


Rest (around 26MB) is shared 32MB
shared
Each of the NIF or HIF can manage all
32MB buffer
HIF 1-48

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Bigdata and Hadoop
NIF 1-4x10GE

dedicated
32MB Buffer
shared

N2248TP-E HIF 1-48


Cloudera
certified !!

Cisco Unified Fabric

N2248-TP-E

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/white_paper_c11-690561.html

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Nexus 2232PP-10GE Fabric Extender

8 SFP+ Uplinks
32 1G/10GE SFP+ Downlinks

32 ports 1/10G Fabric Extender Design Scenario

32x 1/10GE SFP+ host interfaces 1/10G Server Connectivity (SFP+ ToR)
8x SFP network interfaces
Connectivity Flexibility
Predictable low latency

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Nexus 2232TM-E
What is same as 2232TM?
Form Factor, ports, power supplies, etc.
Cabling options and distance supported
Port to port latency
What is different?
A newer and better 10GBase-T PHY (40nm)
Better BER characteristics expected qualification underway
Reduction in power consumption

8 x 10GE SFP+ Uplinks

32 1G/10GT Downlinks
BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Nexus 2232TM-E 10GBASE-T
IEEE 802.3.an standard
Structured Copper Cabling with RJ45 connectors
Supports 10Gbps throughput for up to 100 meters or 330 feet
Leverages Category 6, Category 6A, and Category 7 type cabling
Technology Cable Distance PHY Power Transceiver
(each side) Latency
10GBASE-T Cat6a/7 100m ~6W ~3ms
65nm Cat6a/7 30m ~4W ~3ms

10GBASE-T Cat6a/7 100m ~5W ~2.5ms


40nm Cat6a/7 30m ~3W ~2.5ms

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Nexus 2248PQ-10GE Fabric Extender

4 QSFP+ Uplinks

48 1G/10GE SFP+ Downlinks

48 ports 10/1G Fabric Extender Design Scenario

4x QSFP (16x10GE SFP+) on network interfaces High Density 10GE SFP+ ToR
Feature parity with Nexus 2232PP Connectivity Flexibility
Additional uplink buffers (2x16MB) Predictable low latency

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Nexus 2000 Overview
Redundant 1+1 AC/DC Power Supplies
Redundant N+1 Fan Modules
Color coding for both power supplies and Fan for port side exhaust
and port side intake airflow

Port Side Exhaust

2348UPQ/2348TQ PPT

Port Side Intake

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Nexus 2000 Airflow

Port Side Intake


(Cold Aisle-Network Rack Airflow Alignment)

Port Side Exhaust


(Hot Aisle-Server Rack Airflow Alignment)

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Features deep-dive
and Packet Walk
Nexus 5000/6000 and 2000
Packet Forwarding Overview
Nexus 2000 Nexus 5000/6000 Nexus 2000

1 Ingress Unified Egress


UPC Crossbar UPC
Nexus 2000 Nexus 2000
FEX ASIC
Fabric
FEX ASIC
6

2 3 4 5
3. Nexus 5000 UPC 4. If required
performs ingress 6. VNTag
egress queuing stripped and
1. Frame received on forwarding and and flow control
N2K HIF port queuing frame forwarded
out on N2K HIF
port
2. Nexus 2000
appends VNTag and 5. Nexus 5000 UPC
forwards frame to appends destination
fabric uplink VNTag and forwards
frame on fabric link

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Nexus 2000 Fabric Extender (FEX) VNTag
Nexus 2000 Fabric Extender operates as a remote line card and Logical Interface
does not support local switching (LIF) on the
ingress UPC is
All forwarding is performed on the Nexus 5500/5600/6000 UPC used to forward
VNTag is a Network Interface Virtualization (NIV) technology that the packet
extends (Logical Interface = LIF) to the Nexus 2000 VIF referred LIF
Packet is forwarded
to as a Host Interface (HIF) over fabric link using
a specific VNTag
VNTag is added/removed to the packet between Fabric
Extender and Nexus 5500/5600/6000/7000/9000
VNTAG is a 6Bytes Virtual NIC Tag used for forwarding DA[6]

decisions HIF SA[6]

VNTAG[6]

802.1Q[4]
VNTAG Ethertype d p destination virtual interface

l source virtual interface Frame


Payload
N2K ASIC maps
specific VNTag to
CRC[4]
HIF interface

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Nexus 2000 Port Channels
Nexus 2348/2248 Port Channels
Nexus 2000 series FEX support local port channels 1. Packet is
received and
A local port channel on the N2K is still seen as a single lookup forwards
extended port out a LIF (N2K)
interface
2. Packet is
Extended ports are each mapped to a specific VNTag forwarded over
fabric link using a
HW hashing occurs on the N2K ASIC specific VNTag for
the destination 3. N2K ASIC
Number of local port channels on each N2K is based on N2K LIF (port hashes locally and
the local ASIC channel) transmits packet
on one HIF
interface
22xx Support up to 24 local port channels of up to 8
interfaces each as well as vPC (total of 2 x 8 = 16 ports)

23xx Support up to 24 local port channels of up to 8


interfaces each as well as vPC (total of 2 x 8 = 16 ports)
VM VMK

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Pinning a HIF to a Port Channel

All server ports are pinned To the


Port Channel
Port Channel
Automatic load balancing
in case of a link failure

Definite traffic flow path


visibility and dedicated
bandwidth

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
FCoE and Unified Ports
Nexus 2000 flexible definition of port function

Fibre Channel over Ethernet Ethernet


(FCoE) allows encapsulation and Fibre
transport of Fibre Channel traffic Channel
Traffic
over a shared Ethernet network
SAN A / B isolation is maintained
across the network Nexus 5500 Ethernet and
Fibre Channel Switches
SAN SAN
Unified Ports may be configured to A B
support either native Fibre Channel F
C

or Ethernet Fibre Channel


Disk Array

FCoE Links

Servers with Converged


Network Adapters (CNAs)

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Nexus 2000 SPAN
Receiver Host FEX Host ports CAN be SPAN source

ports
(Except with F2 Line card in Nexus 7000
parent switch)

FEX Fabric ports CAN be SPAN source


ports with all parent switches

FEX Host ports CANNOT be SPAN


destination port with any parent switch

Sender Host Receiver Host

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Nexus 2000 QoS
Tuning the Port Buffers
Each Fabric Extender (FEX) has local port buffers (FEX leverages
a shared memory model) 10G Source (NFS)

10G NFS
You can control the queue limit for a specified Fabric Extender for
egress direction (from the network to the host) Gen 2 UPC

You can use a lower queue limit value on the Fabric Extender to
prevent one blocked receiver from affecting traffic that is sent to Unified Crossbar Fabric
other non-congested receivers ("head-of-line blocking)

Gen 2 UPC

# Disabling the per port tail drop threshold


N5672UP-1(config)# system qos
N5672UP-1(config-sys-qos)# no fex queue-limit
40G Fabric
N5672UP-1(config-sys-qos)#

# Tuning of the queue limit per FEX HIF port Nexus 2000
N5672UP-1(config)# fex 100 FEX ASIC
N5672UP-1(config-fex)# hardware N2248T queue-limit 356000

N5672UP-1(config-fex)# hardware N2248T queue-limit rx/tx ?


<CR> 1G Sink
<2560-652800> Queue limit in bytes

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Nexus 2348UPQ
32MB Shared Buffer 10G Attached Source (NAS Array)

NAS
Speed mismatch between 10G NAS and 1G server requires iSCSI

QoS tuning
32MB shared buffer to handle larger traffic bursts
Hadoop, AVID are examples of bursty applications

NFS
10G
You can control the queue limit for a specified Fabric
Extender for egress direction (from the network to the host)

VM VM VM
#2 #3 #4

1G Attached Server
N5672UP-1(config-fex)# hardware N2348UPQ queue-limit 4000000 rx
N5672UP-1(config-fex)# hardware N2348UPQ queue-limit 4000000 tx
Tune 2348UPQ to support a extremely large
N5672UP-1(config)#interface e110/1/1 burst (Hadoop, AVID, )
N5672UP-1(config-if)# hardware N2348UPQ queue-limit 4096000

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Nexus 2000 Offload
CDP
Efficient utilization of Nexus parent switch CPU
TCAM entries are installed on corresponding FEX
ASICs
LLDP
QoS offload is disabled by default
LACP
DCBX
QoS
N5672UP-1(config)# fex 100
N5672UP-1(config-fex)# hardware n2348uPQ qos-policy-offload

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Nexus 2000 QoS Offload Supported
N2224TP Fabric Extender 24x1G 2x10G SFP+ Module

N2232PP Fabric Extender 32x10G SFP+ 8x10G SFP+ Module

N2232TM Fabric Extender 32x10GBase-T 8x10G SFP+ Module

N2248T Fabric Extender 48x1G 4x10G SFP+ Module

N2248TP E Fabric Extender 48x1G 4x10G SFP+ Module

N2248PQ Fabric Extender 48x10G SFP+ 16x10G SFP+ Module

N2232TM-E Fabric Extender 32x10GBase-T 8x10G SFP+ Module

N23xx Fabric Extender 48x10G SFP+ 6x40G QSFP Module

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Nexus 2000 STP Design Considerations
N2K HIF ports have BPDU Guard enabled by default (it is not
possible to disable currently)
If a BPDU is received port will transition to err-disable
state
Global BPDU Filter compliments BPDU Guard
On link up port will send 11 BPDUs and then stop (in order to
reduce CPU load)
E E E

1. X-Connected
N5672UP-1#show spanning-tree int eth100/1/1 detail patch cable

Port 1025 (Ethernet100/1/1, vPC) of VLAN0190 is designated forwarding 2. BPDU Sent on


Port path cost 1, Port priority 128, Port Identifier 128.1025 Link-Up
Snip->>> 3. BPDU Guard
The port type is edge by port type edge trunk configuration err-disables
Link type is point-to-point by default edge port and
Bpdu guard is enabled
Bpdu filter is enabled by default prevents loop
4. BPDU are not
BPDU: sent 11, received 0
sent once link is
up and active

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Nexus 2000 : Routing Protocol Neighborships
Routing Protocol neighborship across FEX requires Priority
queueing for Control Plane traffic at FEX HIF, among others.
Currently this feature is only Supported on Nexus 7k starting
6.2(2), except for N2248PQ.

Note:
Nexus 5000/6000/9000 Series switches do NOT support this. Routing-Protocol
It is not recommended to connect high performance / critical Neighborship
servers to a FEX, rather use the Parent Port. HIF

On the N7k-FEX, this option may be used for non-critical low


bandwidth devices that need routing neighborships.

Router

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
FCoE Deployment
FEX as Unified TOR Ethernet
FC The fabric extenders (FEX) can
FCoE be used at the ToR
Converged
ToR FEX connect to the end or
LAN row switches which manages
the FEX

FEX work like external line cards


Nexus 5K FC/FCoE SAN to the end or row switches

The EoR switches splits the


Ethernet and FC traffic

Nexus 2K

B22 Dell FEX


B22F FEX 1/10G FEX for
1/10G FEX for Dell Blade servers
B22 HP FEX FTS Blade servers
1/10G FEX for B22 IBM FEX
HP Blade servers 1/10G FEX for
IBM Blade servers

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Nexus 7K - FEX FCoE Features Update

NXOS 7.3(0)N1(1) introduces


support for:

FCoE over FEX with F3 (N2K-


C2348UPQ-10GE, B22HP, N2K-
C2232PP-10GE)
F3 FCoE support for FEX with
physical port vPC and vPC+

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
FC Ports on Nexus 2348UPQ UP FEX
48x1/10G SFP+ Ports 48 Fixed 1/10G SFP+ Unified Capable Ports supporting Ethernet or
6x40G QSFP+ Ports
or 16G FC FCoE or FC
4x10G SFP+ Ports Flexibility
Unified Ports provide up to 24x 16GFC ports or 48x 8GFC ports
32M of Buffer with Lower Oversubscription ratio of 2:1
Parent Switch for Ethernet/FCoE: Nexus 5K, 6K, 7K and 9K
Parent Switch for FC: Nexus 5600

The FC ports on Nexus 2348UPQ are supported only on the Cisco Nexus 5600 switches.
Shipped with Ethernet/FCoE support in July 2014 Only F port is supported on FC-FEX
with 7.0(3)N1(1) NP Port type is not supported so NPV device cannot
Enabled FC support in March 2016 with 7.3(0)N1(1) connect to the FC FEX directly
Fibre Channel FEX supported number (ID) range is F-port trunking and F-port channel are not supported
from 100 to 130 on FC-FEX ports
Support native Fibre Channel speed 2/4/8/16G The default BB credits on each FC interface are 32;
Fibre Channel ports must be configured in groups of 4 and can be set to max 128
ports and starts from beginning Scale per parent switch: 24 FEX, MAX 96 FC ports

First FEX Solution Designed for All Storage Connectivity 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Configure 16G FC Ports on UP FEX

To support 16G FC, the fabric mode on the parent Nexus 5600 should be
in 40G mode.
Only odd-numbered HIF ports can be configured as the 16G FC.
When the odd-numbered HIF port is configured as 16G FC, the adjacent
even-numbered ports must be changed to the out-of-service state. Those
adjacent even-numbered ports next to it cannot be used.
When the speed is set to Auto, the 16G host port comes up at 8G speed.
For the ports to come up at 16G speed, use the switchport speed 16000
command to explicitly set the speed to 16G.
stg-5672-16G-1(config-vsan-db)# interface fc121/1/11
stg-5672-16G-1(config-if)# switchport speed 16000
fc121/1/11: (error) Config not allowed - required to change the mode of the next port to out-of-service
stg-5672-16G-1(config-if)#

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Extending ACI Policy Based Forwarding into Existing Data
Center Networks
Extend Policy Based Forwarding
Extend Visibility, Fault and Audit
Automated Device Management for extended Fabric nodes

ACI Enabled
Remote N9K

N5K N2K N6K


vSwitch AVS
vSwitch AVS HyperV OVS

Extended ACI Fabric

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Design and Topologies
Single Homed Nexus, Single Homed FEX Attached Host
Nexus 7000/5000 Port Channel
Simple and Easy to Configure
End to end
Nexus 7000
Traffic is automatically re-distributed
across remaining NIFs upon failure

Nexus 5000
Simple L2/L3 Flow visibility HIF/NIF to
next hop nexus switch
Nexus 2000
Single link to host: no redundancy at
Server or FEX level.

UCS / Hosts

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Dual Homed Nexus, Single Homed FEX Attached Host
Nexus 7000/5000 Virtual Port Channel
Easy to Configure FEX

Nexus 7000 Network Resiliency and High availability


between N7k & N5k with vPC / double-sided
vPC.
Nexus 5000
Simple L2/L3 visibility from HIF-NIF to N5k

Nexus 2000 Single link to host: no redundancy at Server


or FEX level.

UCS / Hosts

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Dual Homed Nexus/FEX,
Single Homed FEX Attached Host
Nexus 7000/5000/2000 Virtual Port Channel

High Availability and network


resiliency between N7k & N5k& FEX
Nexus 7000

No Blocking ports in STP when using


vPC.
Nexus 5000
L2/L3 Flows can be based on primary
or secondary vPC switch.
Nexus 2000

UCS / Hosts

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Dual Homed Nexus/FEX, Dual Homed FEX AA-Host

Nexus 7000/5600/2000 Virtual Port Channel


Network Resiliency and High
availability across network and
compute
Nexus 7000
Increased Network Bandwidth

Nexus 5600 Active NIC teaming can also be


used for further options.

Nexus 2000

UCS / Hosts

Active
BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Dual Homed Nexus/FEX, Dual Homed FEX AS-Host
Nexus 7000/5000/2000 Virtual Port Channel Network Resiliency and High
Availability across network and
compute
Nexus 7000
Increased Network Bandwidth

Nexus 5000 Active/Standby NIC teaming can


be dependent on CNA capabilities
Nexus 2000

UCS / Hosts

Active
Standby BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Nexus 2000 (FEX) Straight-Through Deployment with VPC

Port-channel connectivity from the server


Two Nexus switches bundled into a vPC S1 S2
pair Fabric Links
Suited for servers with Dual NIC and
capable of running Port-Channel

HIF Fex 100 HIF Fex 101

VPC

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Nexus 2000 (FEX)Active-Active Deployment with VPC
S1 S2
Fabric Extender connected to two Nexus
5X00 / 6000 or Nexus 7000/7700
Suited for servers with Single NIC or
Dual NIC not having port-channel
capability. Fabric Links
Scale implications of less FEX per
system and less VPC

Note : Fex 100


Fex 101
This design is currently not supported on Nexus 9X00 HIF HIF

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Nexus 2000 (FEX) Active-Active Scale &
Limitations (N7X00)
N7X00 can support up to 64 FEXs
N7X00 supports max. 15 Active-Active FEX in 7.2(0)D1(1)
N7X00 supports max. 32 Active-Active FEX in 7.2(0)D1(1)
Straight-Through FEX and Active-Active FEX cannot exist on the
same ASIC instance
Layer 3 HIF ports are not supported with Active-Active FEX
Active-Active FEX is not supported with vPC+

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Nexus 2000 (FEX) - Enhanced VPC

Port-channel connectivity to dual-homed


S1 S2
FEXs
From the server perspective a single access
switch with port-channel support each line
card supported by redundant supervisors
Ideal design for a combination of single Fabric Links
NIC and Dual NIC servers with port-
channel capability
Scale implications of less FEX per system
Fex 100 Fex 101
and less VPC
HIF HIF
Note :
This design is currently not supported on N7000 / N7700 and N9X00

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Physical Port vPC

vPC domain vPC domain

FEX101 FEX102 FEX101 FEX102


e101/1/1 VPC1 VPC1 e102/1/1 e101/1/1 VPC1 VPC1 e102/1/1
Po1 Po1

interface e101/1/1
Port-channel vPC switchport Physical port vPC

vpc 1
lacp mode active

vPC configuration on a physical Layer 2 port as opposed to a port-channel


Front panel ports and FEX ports connected to F2/F2e/F3 only
Improves scaling as separate PC interface not created for single-link VPC leg
Key benefit: more than 1000 host facing VPCs with FEX

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Nexus 2000 (FEX) Active-Active (Unsupported)

Reference : Supported and Unsupported FEX Topologies: N5000/6000/7000/9000


http://www.cisco.com/c/en/us/support/docs/switches/nexus-2000-series-fabric-extenders/200363-Nexus-2000-Fabric-Extenders-Supported-Un.html

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
VXLAN Termination on TOR (Nexus 5600)
Supported Combinations


VXLAN VXLAN VXLAN VXLAN

vtep vtep vtep vtep vtep vtep

Hypervisor Hypervisor Hypervisor Hypervisor

*Switch Local Port-Channel (non VPC) is supported


- TOR to Hypervisor
- TOR to FEX
- FEX to Hypervisor
- TOR to FEX to Hypervisor

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
VXLAN Termination on TOR (Nexus 5600)
Supported Combinations


VXLAN VXLAN VXLAN VXLAN

vtep vtep vtep vtep vtep vtep vtep

Hypervisor Hypervisor Hypervisor Hypervisor

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
VXLAN Termination on TOR (Nexus 5600)
Supported Combinations


VXLAN VXLAN VXLAN VXLAN

vtep vtep vtep vtep vtep vtep

vtep vtep vtep vtep


Hypervisor Hypervisor Hypervisor Hypervisor

*Switch Local Port-Channel (non VPC) is supported


- TOR to Hypervisor
- TOR to FEX
- FEX to Hypervisor
- TOR to FEX to Hypervisor

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
VXLAN Termination on TOR (Nexus 5600)
Supported Combinations


VXLAN VXLAN VXLAN VXLAN

vtep vtep vtep vtep


Hypervisor Hypervisor Hypervisor Hypervisor

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
VXLAN Termination on TOR (Nexus 5600)
Unupported Combinations


VXLAN VXLAN VXLAN VXLAN

vtep vtep vtep vtep vtep vtep vtep

vtep vtep vtep vtep


Hypervisor Hypervisor Hypervisor Hypervisor

*FEX connectivity with combination of Hypervisor


(Host originated VXLAN) and Bare-Metal Servers
is not supported

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
Configuration and Provisioning
Nexus 2000
Configuration
N5672UP-1(config)# feature fex
N5672UP-1(config-if)# int port-channel100
N5672UP-1(config-if)# switchport mode fex-fabric
N5672UP-1(config-if)# fex associate 100
N5672UP-1(config-if)# vpc 100
N5672UP-1(config-if)# no shut

Configuration
N5672UP-1(config)#show fex 100
FEX: 100 Description: FEX0100 state: Online
FEX version: 7.1(0)N1(1a) [Switch version: 7.1(0)N1(1a)]
Extender Serial: FOC1818R2M9
Extender Model: N2K-C2348UPQ, Part No: 73-15489-03
Pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth2/5
FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state:
Po100 - Interface Up. State: Active
Eth2/5 - Interface Up. State: Active
Eth2/6 - Interface Up. State: Active

Verification

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Pre-Provisioning Configuration
N5672UP-1(config)# slot 101
N5672UP-1(config-slot)# provision model N2K-C2348UPQ

Provision

N5672-1(config-slot)# show run int eth101/1/1

!Command: show running-config interface Ethernet103/1/1


!Time: Mon Mar 23 10:58:01 2015

version 7.1(0)N1(1a)

interface Ethernet101/1/1

N5672-1(config-if)# int eth101/1/1


N5672-1(config-if)# switchport access vlan 190
N5672-1(config-if)# no shut
Configuration & Verification

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
Nexus 2248TP-E Enhanced Counters
N5672UP-1(config-if)# sh queuing interface eth100/1/1
Ethernet110/1/1 queuing information:
Input buffer allocation:
Qos-group: 0
frh: 2
drop-type: drop
cos: 0 1 2 3 4 5 6
xon xoff buffer-size
---------+---------+-----------
19200 24320 48640 Ingress queue limit(Configurable)
Queueing:
queue qos-group cos priority bandwidth mtu
--------+------------+--------------+---------+---------+---- Egress queues:
2 0 0 1 2 3 4 5 6 WRR 100 1600
CoS to queue mapping
Queue limit: 66560 bytes Bandwidth allocation
MTU
Queue Statistics:
queue rx tx Egress queue limit(Configurable)
------+---------------+---------------
2 24434 3066978

Port Statistics: Drop counters


rx drop rx mcast drop rx error tx drop mux ovflow
---------------+---------------+---------------+---------------+--------------
0 0 0 0 InActive

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
Nexus 2348UPQ Enhanced Counters
N5672UP-1(config-if)# show queuing interface ethernet 100/1/1
if_slot 32, ifidx 0x1f630000
Ethernet100/1/1 queuing information:
Input buffer allocation:
Qos-group: 0
frh: 2
drop-type: drop
cos: 0 1 2 3 4 5 6
xon xoff buffer-size
---------+---------+----------- Ingress queue limit
5120 15360 71680

Queueing: Egress queues:


queue qos-group cos priority bandwidth mtu CoS to queue mapping
--------+------------+--------------+---------+---------+----
2 0 0 1 2 3 4 5 6 WRR 100 1600 Bandwidth allocation
MTU
Queue limit: 2088960 bytes

Queue Statistics: Egress queue limit


queue rx tx
------+---------------+---------------
2 126 49
Drop counters
Port Statistics:
rx drop rx mcast drop rx error tx drop mux ovflow
---------------+---------------+---------------+---------------+--------------
0 0 0 0 InActive

Priority-flow-control enabled: no

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
Ethernet Flow Control (802.3x) Configuration
N5672UP-1(config-if)# show interface ethernet 100/1/1 flowcontrol

--------------------------------------------------------------------------------
Port Send FlowControl Receive FlowControl RxPause TxPause
admin oper admin oper
---------------------------------------------------------------------
Eth100/1/1 on off off off 0 0

N5672UP-1(config-if)# interface ethernet 100/1/1


N5672UP-1(config-if)# flowcontrol send on/off
N5672UP-1(config-if)# flowcontrol receive on/off

Why do I care to honor flow control?

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
FCoE configuration
N5672UP-1(config)# feature fcoe (If FCoE not enabled)
FC license checked out successfully
fc_plugin extracted successfully
FC plugin loaded successfully
FCoE manager enabled successfully
FC enabled on all modules successfully
Enabled FCoE QoS policies successfully

N5672UP-1(config)# vlan 100


N5672UP-1(config)# fcoe vsan 100
N5672UP-1(config)# vsan database
N5672UP-1(config)# vsan 100
N5672UP-1(config)# fex 100
N5672UP-1(config)# fcoe
N5672UP-1(config)# interface vfc 1011
N5672UP-1(config)# bind interface ethernet 100/1/1
N5672UP-1(config)# switchport trunk allowed vsan 100
N5672UP-1(config)# no shut

Is it not Fairly easy to configure FCoE?

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
Configuration Verification
N5k-1#show fex 100 detail
FEX: 100 Description: FEX0100 state: Online
FEX version: 7.1(0)N1(1a) [Switch version: 7.1(0)N1(1a)]
FEX Interim version: 7.1(0)N1(1a)
Switch Interim version: 7.1(0)N1(1a)
Extender Serial: FOC1818R2M9
Extender Model: N2K-C2348UPQ, Part No: 73-15489-03
Snip->>>>>>
Logs:
03/18/2015 09:52:21.289192: Module register received
03/23/2015 09:18:18.516214: Module Offline
03/23/2015 09:18:18.561004: Deleting route to FEX
03/23/2015 09:18:18.568532: Module disconnected
03/23/2015 09:21:53.581415: Module register received
03/23/2015 09:21:53.583664: Registration response sent
03/23/2015 09:21:54.634249: create module inserted event.
03/23/2015 09:21:54.635044: Module Online Sequence
03/23/2015 09:22:03.006536: Module Online

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
Configuring FEX with ACI

* - Verify model specific FEX support with ACI


BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
Configuring FEX with ACI

* - Verify model specific FEX support with ACI


BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
Configuring FEX with ACI

* - Verify model specific FEX support with ACI


BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
Configuring FEX with ACI

* - Verify model specific FEX support with ACI


BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
Configuring FEX with ACI

* - Verify model specific FEX support with ACI


BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
Connectivity Options
Nexus 2000
Nexus 2300 Connectivity Options
10G SFP+ Connectivity 40G QSFP+ Connectivity

Nexus Parent Switch Nexus Parent Switch

Nexus 2300 Nexus 2300

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
Nexus 2300 Connectivity Options
Network Interface (NIF)
The Network Interface Fabric Ports can operate 10G/40G mode
FET-40G, QSFP-Bidi, QSFP+, Active Twinax, including AoC cables can be used to connect to a parent switch
while operating in 40G mode

Host Interface (HIF)


SFP+ Host Interface will support SR, LR, GLC-T, GLC-SX-MM, AoC
Supported Twinax: 1, 3, 5M (passive) and 7, 10M (active)

*Check the parent switch documentation for the latest supported cable options Twinax Passive cable is
Not supported on NIF

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
OM3-MM Fiber Cable

For QSFP-40G-SR4 and


QSFP-40G-CSR4

12-Fiber MPO Connector


MPO/MTP connector with 12
pins
Use 4 fiber pairs
4 TX and 4 RX allows for 40 GE
4 unused fibers in the center

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
QSFP-40G-SR4

40G-SR4 40G-SR4

MPO-12 MPO-12

Support 100M with OM3 MMF


and 150M with OM4 MMF
10G-SR
10G-SR
40G-SR4 10G-SR
10G-SR

MPO-12 LC Breakout
BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
QSFP 40G Bidirectional Transceiver 40G
deployment with 10G cabling
Ciscos innovation has created a new QSFP pluggable 40Gbit/s
Bidirectional (BiDi) Transceiver that works with existing infrastructure:
Multi-mode fiber
Using the existing 10Gbit/s Multimode infrastructure
QSFP-40G-SR-BD has dual LC Connector
40Gbit/s BiDi transceiver has two 20Gbit/s channels each transmitted and
received simultaneously on two wavelengths

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
QSFP-40G-CSR4
Supports 300M with OM3 MMF
and 400M with OM4 MMF
Compatible with Cisco QSFP-
40G-CSR4 40G-SR4/CSR4
40G-SR4

MPO-12 MPO-12
Compatible with 10G SFP-SR

10G-SR
10G-SR
40G-CSR4 10G-SR
10G-SR

MPO-12 LC Breakout
BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
QSFP-40G-SR-BD Supports 100m with OM3 with
MMF and 150m with OM4
(OM4+) MMF
Multimode
LC Duplex LC duplex connectors
40G-BD 40G-BD
Wavelength: ~850nm and
~900nm
Pull-Tab Color: grey

Multimode 10G-SR
10G-SR
40G-BD LC Duplex 10G-SR
10G-SR

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
Physical Connection Choices
40G QSFP+ 40G QSFP+
QSFP-LR4 QSFP-LR4

40G QSFP+ 40G QSFP+


QSFP-SR-BD QSFP-SR-BD

40G QSFP+ 40G QSFP+


QSFP-SR4/CSR4 QSFP-SR4/CSR4

1M, 3M, 5M Passive 40G QSFP+


40G QSFP+ 7M, 10M Active

40G QSFP+ 10G SFP+


QSFP-SR4/CSR4

40G QSFP+ QSFP-4x10G 10G SFP+

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
FET-40G (aka QSFP+ FET)
Low-cost QSFP optical transceiver connecting FEX to Cisco Nexus 6004
Supported on Cisco Nexus 5500/N5600/6000/N7000/N9000 and Nexus 2248PQ-10G/Nexus 2300
Interoperable with FET-10G
Support for 100m distance with OM3
Cisco Nexus 5600/6000 Cisco Nexus 5600/6000
FET-40G
FET-40G

FET-10G
FET-40G Cisco Nexus 2232PP and 2232TM-E 2232TM
Cisco Nexus 2248TP-E 2248TP
Cisco Nexus 2248PQ-10G/N2300

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
Convert 40GE Interface to Four 10 GE Interfaces
Convert the QSFP Interfaces to Four 10 GE Interfaces When Connecting to a SFP+ Port or to the QSFP Interfaces That
Operate in Four 10 GE Mode, such as the QSFP Uplink of N2248PQ-10G or the Cisco Nexus 5500 QSFP GEM.

Apply global CLI to change interface types N6004(config)# interface breakout slot
1 1 port 1-6 map 10g-4x
to 10 GE
Every three contiguous QSFP interfaces resides
on one Port ASIC.
The port range specified in the CLI has to
include all ports on the ASIC N6004(config)# poweroff module ?
<1-8> Please enter module number
2 Power off the affected modules N6004(config)# poweroff module 1
Every group of 12 QSFP interfaces is managed
as one module, even for the fixed interfaces

3 Power on the affected modules N6004(config)# no poweroff module 1


N6004(config)#

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 102
QSA-QSFP+ to SFP Adapter
Convert one QSFP port to one SFP+ port.
Provides 1G/10G transceiver support
Long reach connection between for QSFP+ devices and SFP+ devices
Following 1G/10G transceivers are qualified with QSA in release
10G-LR
1G GLC-T, 1G SX, LX

Requires a mode change on QSFP+ port to 4x10G


Only first 10G interface can be used. Remaining three 10G shown as Link Not Connected

1G SFP or 10G SFP+


QSFP+ Adapter Transceiver

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
Summary
Nexus 2000
Key Takeaways
Understand FEX Architectures & Capabilities
Realize the benefits offered by a FEX design
Evaluate FEX for YOUR network requirements.

Simplify your Flexible and Scalable ToR design provides Large number of Ports,
Network with low number of management points.

Lower CAPEX Easy to Provision, manage, troubleshoot.


OPEX Lowest Cost option for connectivity

Unified
Network Flexible Ethernet, FCoE and FC solutions with FEX

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
Acronyms and Abbreviations
HIF Host Interface SoC System on Chip / Switch on Chip
NIF Network Interface FC Fiber Channel
CIF CPU Interface FCoE Fiber Channel over Ethernet
VNTag Virtual NIC Tag UP Unified Port
H2N Host to Network UCF Unified Crossbar Fabric
N2H Network to Host UPC Unified Port Controller
ASIC Application-Specific Integrated Circuit IGMP Internet Group Management Protocol
vPC Virtual Port-Channel ALE Application Leaf Engine
FEX Fabric Extender NFE Network Forwarding Engine
VXLAN Virtual Extensible LAN TCAM Ternary Content Addressable Memory
ACI Application Centric Infrastructure FIB Forwarding Information Base
APIC Application Policy Infrastructure Controller HPC High-Performance Computing
OTV Overlay Transport Virtualization Mroute Multicast Route
STP Spanning-Tree LC Line Card
LPM Longest Prefix Match EOBC Ethernet Out-of-Band Channel
CPU Central Processing Unit

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 106
Complete Your Online Session Evaluation
Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 Amazon gift card.
Complete your session surveys
through the Cisco Live mobile
app or from the Session Catalog
on CiscoLive.com/us.

Dont forget: Cisco Live sessions will be available


for viewing on-demand after the event at
CiscoLive.com/Online

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 107
Continue Your Education
Demos in the Cisco campus
Walk-in Self-Paced Labs
Lunch & Learn
Meet the Engineer 1:1 meetings
Related sessions

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 108
Please join us for the Service Provider Innovation Talk featuring:
Yvette Kanouff | Senior Vice President and General Manager, SP Business
Joe Cozzolino | Senior Vice President, Cisco Services

Thursday, July 14th, 2016


11:30 am - 12:30 pm, In the Oceanside A room

What to expect from this innovation talk


Insights on market trends and forecasts
Preview of key technologies and capabilities
Innovative demonstrations of the latest and greatest products
Better understanding of how Cisco can help you succeed

Register to attend the session live now or


watch the broadcast on cisco.com
Thank you
Appendix
9 Reference Slides
10G MAC Tx
Host to VNTag Rewr
8
802.1q Rewr

Network 7
6 Read Mux Tx Arb

Output Queue Time Stamp

5
37 read 4
Shared Packet Buffer 32MB Input port
Accounting
37 write Flow Cntr

3
VNTag 802.1q VNTag
Forwarding Rewrite Rewrite Svif Check
HIF and HIF
2 ACL Data
10G MAC Rx 10G MAC Tx

1
BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 113
1 Reference Slides
Network to 10G MAC Rx 10G MAC Tx

Host 2 Unicast FW
Multicast FW
VNTag 802.1q VNTag
NIF Forwarding Rewrite Rewrite

3
37 read 4
Shared Packet Buffer 32MB Input port
Accounting
37 write Flow Cntr

5
Output Queue
6
Read Mux Tx Arb

7
VNTag Rewr 802.1q Rewr
HIF
8 10G MAC Tx
9
BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 114
Reference Slides
Typical UP FEX Topologies
Straight-Through FEX Active-Active FEX

Nexus Nexus
5600 5600
VP VP
C FPC 40G FCoE C
FPC
40G FCoE 40GE
Nexus Nexus
2348UPQ 2348UPQ

VSAN A 10GE VSAN B VSAN A 10GE VSAN B

8/16G FC 8/16G FC

10GE NIC + FC HBA 10GE NIC + FC HBA


FC-HIF ports configuration must be the same in both the The Active-Active FEX is supported, but only one FC FEX
primary and the secondary VPC switches. can be associated with a parent switch in the pair.
For the Active-Active FEX, the port-conversion CLI should switch1(config)# fex 101
be configured on both the vPC peer switches. switch1(config-fex)# fcoe

10G FCoE CNA


BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 115
Reference Slides

Configuration of 16G FC Ports on UP FEX


To support 16G FC, the fabric mode on the parent Nexus 5600 should be
in 40G mode.
Only odd-numbered HIF ports can be configured as the 16G FC.
When the odd-numbered HIF port is configured as 16G FC, the adjacent
even-numbered ports must be changed to the out-of-service state. Those
adjacent even-numbered ports next to it cannot be used.
When the speed is set to Auto, the 16G host port comes up at 8G speed.
For the ports to come up at 16G speed, use the switchport speed 16000
command to explicitly set the speed to 16G.
stg-5672-16G-1(config-vsan-db)# interface fc121/1/11
stg-5672-16G-1(config-if)# switchport speed 16000
fc121/1/11: (error) Config not allowed - required to change the mode of the next port to out-of-service
stg-5672-16G-1(config-if)#

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 116
VFC Creation on 4-Port (HIF) VPC Reference Slides

Supported from Cisco NX-OS release 7.2(0)N1(1)

Only supported on N5600/N6000 platforms (Bigsur ASIC)


N5600-1 N5600-2 Only supported on Nexus 2300 FEX (Tiburon ASIC)
Not supported for NPV mode

Straight-Through FEX N5600-1 N5600-2


vfc1 vfc2 vfc3 vfc4
FEX-1 FEX-2
vfc1 vfc2 vfc3 vfc4 vPC

CNA
vPC
Host
CNA

Host
Switch(config)# int port-channel 500
Switch(config-if)# fcoe multi-vfc enable the feature

BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 117