Вы находитесь на странице: 1из 71

Cisco Nexus 7000 Hardware Architecture

BRKARC-3470

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Session Goal
To provide you with a thorough understanding of the Cisco Nexus 7000 switching architecture, supervisor, fabric, and I/O module design, packet flows, and key forwarding engine functions

This session will not examine NX-OS software architecture or other Nexus platform architectures
Related sessions:
BRKARC-3471: Cisco NX-OS Software Architecture

BRKDCT-2951: Deploying Nexus 7000 in Data Center Networks


BRKDCT-2081: Cisco FabricPath Technology and Design BRKDCT-2048: Deploying Virtual Port Channel in NX-OS BRKDCT-2121: VDC Design and Implementation Considerations with Nexus 7000 BRKARC-3472: NX-OS Routing & Layer 3 Switching BRKCRS-3144: Troubleshooting Cisco Nexus 7000 Series Switches TECDCT-4125: Cisco FabricPath (4h Techtorial) LTRDCT-4047: Deploying Nexus 7000/NX-OS Hands-on Lab (Lab) LTRCRT-5205: Configuring Nexus 7000 Virtualization (Lab)
BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

What Is Nexus 7000?


Data-center class Ethernet switch designed to deliver high-availability, system scale, usability, investment protection
Supervisor Engines I/O Modules
Chassis

Fabrics

Forwarding Engines
BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Agenda
Chassis Architecture

Supervisor Engine and I/O Module Architecture


Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding

Classification Policies NetFlow Summary

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Nexus 7010 Chassis


Integrated cable management with cover

System status LEDs

Front-toback airflow

Air exhaust

Optional front door

Fan trays

21RU

Supervisor slots (5-6) I/O module slots (1-4, 7-10)

Two chassis per 7 rack Crossbar fabric modules

Power supplies Air intake with optional filter


BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

Front

N7K-C7010Cisco Public

Rear

Supported in NX-OS release 4.1(2) and later

Nexus 7018 Chassis


Integrated cable management

System status LEDs


Optional front door Side-to-side airflow

Fan trays

Supervisor slots (9-10)

25RU

Crossbar fabric modules

I/O module slots (1-8, 11-18)

Power supply air intake


BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

Power supplies

Front

N7K-C7018 Cisco Public

Rear

Agenda
Chassis Architecture

Supervisor Engine and I/O Module Architecture


Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding

Classification Policies NetFlow Summary

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Supervisor Engine
Performs control plane and management functions
Dual-core 1.66GHz x86 processor with 8GB DRAM
2MB NVRAM, 2GB internal bootdisk, compact flash slots, USB

Console, aux, and out-of-band management interfaces Interfaces with I/O modules via 1G switched EOBC Houses dedicated central arbiter ASIC that controls VOQ admission/fabric access via dedicated arbitration path to I/O modules

N7K-SUP1

ID LED
Status LEDs

AUX Port Console Port


BRKARC-3470

USB Ports Management Ethernet Compact Flash Slots


Cisco Public

CMP Ethernet

Reset Button
8

2011 Cisco and/or its affiliates. All rights reserved.

Nexus 7000 I/O Module Families M1 and F1


M family L2/L3/L4 with large forwarding tables and rich feature set
N7K-M148GT-11/N7K-M148GT-11L

N7K-M108X2-12L

N7K-M148GS-11/N7K-M148GS-11L

N7K-M132XP-12/ N7K-M132XP-12L

F family Low-cost, high performance, low latency, low power and streamlined feature set

N7K-F132XP-15
BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Supported in NX-OS release 5.0(2a) and later

8-Port 10GE M1 I/O Module


N7K-M108X2-12L

8-port 10G with X2 transceivers

80G full-duplex fabric connectivity Two integrated forwarding engines (120Mpps)


Support for XL forwarding tables (licensed feature)

Distributed L3 multicast replication 802.1AE LinkSec


N7K-M108X2-12L

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

11

8-Port 10G XL M1 I/O Module Architecture


N7K-M108X2-12L
EOBC To Central Arbiter
To Fabric Modules

Fabric ASIC

LC CPU

VOQs

Forwarding Engine
Replication Engine

Forwarding Engine
Replication Engine

VOQs

Replication Engine

Replication Engine

10G MAC Linksec


1

10G MAC Linksec


2

10G MAC Linksec


3

10G MAC Linksec


4

10G MAC Linksec


5

10G MAC Linksec


6

10G MAC Linksec


7

10G MAC Linksec


8

Front Panel Ports


BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

12

N7K-M132XP-12 Supported in all releases N7K-M132XP-12L Supported in NX-OS release 5.1(1) and later

32-Port 10GE M1 I/O Modules


N7K-M132XP-12, N7K-M132XP-12L 32-port 10G with SFP+ transceivers

80G full-duplex fabric connectivity Integrated 60Mpps forwarding engine


XL forwarding engine on L version

Oversubscription option for higher density (up to 4:1) Supports Nexus 2000 (FEX) connections Distributed L3 multicast replication 802.1AE LinkSec
N7K-M132XP-12/ N7K-M132XP-12L

BRKARC-3470

2011 Cisco and/or its affiliates. All Public reserved. Cisco rights

Cisco Public

13

Shared vs. Dedicated Mode


To Fabric
rate-mode shared (default) 10G

11

13

15

Shared mode
Four interfaces in port group share 10G bandwidth Port group group of contiguous even or odd ports that share 10G of bandwidth (e.g., ports 1,3,5,7)

To Fabric

rate-mode dedicated

10G

Dedicated mode
9 11 13 15
First interface in port group gets 10G bandwidth Other three interfaces in port group disabled
BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

14

32-Port 10G M1 I/O Module Architecture


N7K-M132XP-12, N7K-M132XP-12L
EOBC To Central Arbiter
To Fabric Modules

Fabric ASIC

LC CPU

VOQs

Forwarding Engine

VOQs

Replication Engine Replication Engine

Replication Engine Replication Engine

10G MAC 4:1 Mux + Linksec


1 3 5 7

10G MAC 4:1 Mux + Linksec


9 11 13 15

10G MAC 4:1 Mux + Linksec


17 19 21 23

10G MAC 4:1 Mux + Linksec


25 27 29 31

10G MAC 4:1 Mux + Linksec


2 4 6 8

10G MAC 4:1 Mux + Linksec

10G MAC 4:1 Mux + Linksec

10G MAC 4:1 Mux + Linksec

10 12 14 16 18 20 22 24 26 28 30 32

Front Panel Ports


BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

15

48-Port 1G M1 I/O Modules

N7K-M148GT-11, N7K-M148GS-11, N7K-M148GS-11L, N7K-M148GT-11L

Four 1G I/O module options:


48 10/100/1000 RJ-45 ports (N7K-M148GT-11) 48 1G SFP ports (N7K-M148GS-11) 48 1G SFP ports with XL forwarding engine (N7K-M148GS-11L) 48 10/100/1000 RJ-45 ports with XL forwarding engine (N7K-M148GT-11L)

N7K-M148GT-11
All releases

N7K-M148GS-11
Release 4.1(2) and later

Integrated 60Mpps forwarding engine 46G full duplex fabric connectivity


Line rate on 48-ports with some local switching
N7K-M148GS-11L
Release 5.0(2a) and later

Distributed L3 multicast replication 802.1AE LinkSec

N7K-M148GT-11L
BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Release 5.1(1) and later


16

48-Port 1G M1 I/O Modules Architecture


N7K-M148GT-11, N7K-M148GS-11, N7K-M148GS-11L, N7K-M148GT-11L
EOBC To Fabric Modules

To Central Arbiter

Fabric ASIC

LC CPU

VOQs

Replication Engine

Forwarding Engine

Replication Engine

12 x 1G MAC

12 x 1G MAC

12 x 1G MAC

12 x 1G MAC

Linksec

Linksec

Linksec

Linksec

Linksec

Linksec

1-12

13-24
Front Panel Ports

25-36

37-48

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

17

Supported in NX-OS release 5.1(1) and later

32-Port 1G/10GE F1 I/O Module


N7K-F132XP-15 32-port 1G/10G with SFP/SFP+ transceivers

230G full-duplex fabric connectivity (320G local switching) System-on-chip (SoC) forwarding engine design
16 independent SoC ASICs

Layer 2 forwarding with L3/L4 services (ACL/QoS) Multi-protocol Classic Ethernet, FabricPath, DCB, FCoE
N7K-F132XP-15

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

sometimes called switch-on-chip

18

32-Port 1G/10G F1 I/O Module Architecture


N7K-F132XP-15
EOBC To Fabric Modules To Central Arbiter

LC CPU Fabric ASIC Fabric ASIC

Arbitration Aggregator

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC
1 2 3 4 5 6

2 X 10G SoC
7 8

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

9 10 11 12 13 14 15 16

17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

Front Panel Ports

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

19

Agenda
Chassis Architecture

Supervisor Engine and I/O Module Architecture


Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding

Classification Policies NetFlow Summary

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

21

M1 Forwarding Engine Hardware


Hardware forwarding engine(s) integrated on every I/O module 60Mpps per forwarding engine Layer 2 bridging with hardware MAC learning 60Mpps per forwarding engine IPv4 and 30Mpps IPv6 unicast IPv4 and IPv6 multicast support (SM, SSM, bidir)
Hardware Table FIB TCAM Classification TCAM (ACL/QoS) MAC Address Table NetFlow Table
BRKARC-3470

RACL/VACL/PACLs

QoS remarking and policing policies


Policy-based routing (PBR)

Unicast RPF check and IP source guard


Ingress and egress NetFlow (full and sampled)
M1-XL Modules without License 128K 64K M1-XL Modules with License 900K 128K

M1 Modules 128K 64K

128K 512K
Cisco Public

128K 512K

128K 512K
22

2011 Cisco and/or its affiliates. All rights reserved.

M1 Forwarding Engine Architecture


FE Daughter Card

Ingress policing

Ingress Pipeline Egress Pipeline

Ingress NetFlow collection FIB TCAM and adjacency table lookups for Layer 3 forwarding ECMP hashing Multicast RPF check

Ingress ACL and QoS classification


Unicast RPF check

Layer 3 Engine

Egress NetFlow collection

Egress policing

Egress ACL and QoS classification


Ingress MAC table lookups IGMP snooping lookups IGMP snooping redirection

Layer 2 Engine

Egress MAC lookups IGMP snooping lookups


Final lookup result to I/O Module Replication Engine
Cisco Public

Packet Headers from I/O Module Replication Engine


BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

23

F1 Forwarding Engine Hardware


SoC forwarding engine services two front-panel 10G ports
16 SoCs per I/O module

VACL/PACLs

QoS remarking policies FabricPath forwarding Priority Flow-Control (PFC) and Enhanced Transmission Selection (ETS)

480Mpps Layer 2 bridging with hardware MAC learning per I/O module
30Mpps per forwarding engine

Hardware Table

Per SoC 16K 1K in/1K out

Per F1 Module 256K 16K in/16K out

MAC Address Table Classification TCAM (ACL/QoS)

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

24

F1 Forwarding Engine Architecture


To/From Central Arbiter
To Fabric

Virtual output queues

From Fabric

Egress forwarding decision (MAC lookup, egress ACL) 2 X 10G SoC

Ingress Buffer (VOQ) Ingress forwarding decision (MAC lookup, FP lookup, ingress ACL/QoS

Forwarding Engine (Egress)

Forwarding Engine (Ingress) Pre-Forwarding Ingress Buffer

MAC Table, ACL, QoS

Egress fabric receive buffer Egress Buffer

Skid buffer Accommodates pause reaction time

Forwarding tables

1G and 10G capable interface MAC

1G/10G MAC

1G/10G MAC

Port A 1G/10G
BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

Two front-panel interfaces per ASIC


Cisco Public

Port B 1G/10G
25

Agenda
Chassis Architecture

Supervisor Engine and I/O Module Architecture


Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding

Classification Policies NetFlow Summary

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

26

Crossbar Switch Fabric Module


Each fabric module provides 46Gbps per I/O module slot
Up to 230Gbps per slot with 5 fabric modules

Different I/O modules leverage different amount of fabric bandwidth


80G per slot with 10G M1 modules
230G per slot with 10G F1 modules

Access to fabric controlled using QoS-aware central arbitration with VOQ

N7K-C7010-FAB-1 N7K-C7018-FAB-1
BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

27

Multistage Crossbar
Nexus 7000 implements 3-stage crossbar switch fabric
Stages 1 and 3 on I/O modules Stage 2 on fabric modules Fabric Modules

2nd stage
2 x 23Gbps per I/O slot per fabric module

Crossbar Fabric ASIC

Crossbar Fabric ASIC

Crossbar Fabric ASIC

Crossbar Fabric ASIC

Crossbar Fabric ASIC

Up to 230Gbps per I/O module with 5 fabric modules installed

20 x 23Gbps channels per fabric module


Crossbar Fabric ASIC Crossbar Fabric ASIC

1st stage

Ingress Module
Cisco Public

Egress Module

3rd stage

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

28

M1 I/O Module Capacity


1G modules
Require 1 fabric for full bandwidth Require 2 fabrics for N+1 redundancy
46Gbps/slot 46Gbps/slot

Fabric Modules
Crossbar Fabric ASICs

Crossbar Fabric ASICs

230Gbps 46Gbps 184Gbps 138Gbps 92Gbps


per slot bandwidth 4th and 5th fabric modules provide additional redundancy and future-proofing

46Gbps/slot

Crossbar Fabric ASICs

46Gbps/slot

Crossbar Fabric ASICs

10G modules
Require 2 fabrics for full bandwidth Require 3 fabrics for N+1 redundancy
BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

46Gbps/slot

Crossbar Fabric ASICs

Cisco Public

30

F1 I/O Module Capacity


F1 SFP+ module
Operates with any number of fabrics Requires 5 fabrics for maximum bandwidth Redundancy model is graceful bandwidth derating

Fabric Modules
46Gbps/slot
Crossbar Fabric ASICs

230G

46Gbps/slot

Crossbar Fabric ASICs

230Gbps 46Gbps 184Gbps 138Gbps 92Gbps


per slot bandwidth

46Gbps/slot

Crossbar Fabric ASICs

230G

46Gbps/slot

Crossbar Fabric ASICs

46Gbps/slot

Crossbar Fabric ASICs

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

31

Access to Fabric Bandwidth


Access to fabric controlled using central arbitration
Arbiter ASIC on Supervisor Engine provides fabric arbitration via dedicated arbitration path

Virtual Output Queues (VOQs) at ingress to fabric represent bandwidth availability on egress modules
Four levels of priority per VOQ destination Central arbiter controls admission to VOQ based on bandwidth availability and priority

Buffer credits represent bandwidth availability for each VOQ destination at each priority level
Credits requested by ingress I/O modules with traffic to send into fabric Credits granted by central arbiter based on bandwidth availability Credits returned to the pool by egress I/O modules after receiving traffic from fabric

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

32

Benefits of Central Arbitration with VOQ


Ensures priority traffic takes precedence over besteffort traffic across fabric
Four levels of priority for each VOQ destination

Ensures fair access to bandwidth for multiple ingress ports transmitting to one egress port
Central arbiter ensures all traffic sources get appropriate access to fabric bandwidth, even with traffic sources on different modules

Prevents congested egress ports from blocking ingress traffic destined to other ports
Mitigates head-of-line blocking by providing independent queues for individual destinations across the fabric

Enables lossless service for some traffic classes across the fabric
Can provide strict priority and backpressure (blocking instead of dropping) for certain traffic classes, such as FCoE traffic
BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

34

Agenda
Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture

I/O Module Queuing Layer 2 Forwarding


IP Forwarding IP Multicast Forwarding Classification Policies

NetFlow Summary

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

35

I/O Module Queuing


All configuration through Modular QoS CLI (MQC)
Queuing parameters applied using class-maps/policymaps/service-policies

Queuing and scheduling behavior based on I/O module capabilities


Default queuing on all modules enables two classes including egress priority queue

Buffering model varies by I/O module family


M1 modules: hybrid model combining egress port buffered with ingress VOQ buffered architecture F1 modules: pure ingress VOQ buffered architecture

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

36

Hybrid Ingress/Egress Buffered Model (M1 I/O Modules)


Ingress port buffer manages congestion in ingress forwarding/replication engines only

Ingress VOQ buffer manages congestion toward egress interface over fabric Egress FIFO buffer just enough to catch frames in flight and keep pipe full
Egress port buffer manages congestion at egress interface
Ingress port buffer Ingress VOQ buffer

Ingress Module

Egress FIFO buffer

Egress port buffer

Ingress Module

Crossbar Fabric

Egress Module

Ingress Module
BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

37

Ingress Buffered Model (F1 I/O Modules)


Ingress skid buffer just enough to absorb packets received after external flow control asserted Ingress VOQ buffer manages congestion toward egress interfaces Egress FIFO buffer just enough to catch frames in flight and keep pipe full
Ingress skid buffer Ingress VOQ buffer

Ingress Module

Egress FIFO buffer

Ingress Module

Crossbar Fabric

Egress Module

Ingress Module
BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

38

M1 I/O Module Buffering


Hybrid ingress/egress buffered architecture on 8-port 10G M1 I/O module

To fabric

Egress
LC CPU

Fabric ASIC

FIFO
Shared VOQs

VOQs

Forwarding Engine
Replication Engine

Forwarding Engine
Replication Engine

VOQs

1234

VOQ 16MB
10G MAC

Replication Engine

Replication Engine

1p3q1t
Replication Engine

10G MAC

10G MAC

10G MAC

10G MAC

10G MAC

10G MAC

10G MAC

Linksec

Linksec

Linksec

Linksec

Linksec

Linksec

Linksec

Linksec

5
Front Panel Ports

12345678

Egress 80MB

1p7q4t
Per-Port 10G MAC 10G MAC

12345678

Ingress 96MB

Port 1

Port 2

8q2t
BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

40

F1 I/O Module Buffering


Pure ingress buffered architecture on 32-port 10G F1 I/O module

To fabric

Egress
LC CPU Fabric ASIC Fabric ASIC Arbitration Aggregator

FIFO
1234
VOQ 1.25MB

Per-Port

1p3q1t
1
Skid

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC 2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC 2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

2 X 10G SoC

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 Front Panel Ports

FIFO

Port 1

Port 2

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

41

Agenda
Chassis Architecture

Supervisor Engine and I/O Module Architecture


Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding

Classification Policies NetFlow Summary

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

43

Layer 2 Forwarding
MAC table size depends on module type:
M1 MAC table is 128K entries F1 MAC table is 256K entries (16K entries per SoC)

Hardware MAC learning


CPU not directly involved in learning

Forwarding engine(s) on each module have copy of MAC table


New learns communicated to other forwarding engines via hardware flood to fabric mechanism
Software process ensures continuous MAC table sync

Spanning tree (PVRST or MST), Virtual Port Channel (VPC), or FabricPath ensures loop-free Layer 2 topology

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

44

Hardware Layer 2 Forwarding Process


In Classic Ethernet and FabricPath edge switches, MAC table lookup drives Layer 2 forwarding
Source MAC and destination MAC lookups performed for each frame, based on {VLAN,MAC} pairs

Source MAC lookup drives new learns and refreshes aging timers
Destination MAC lookup dictates outgoing switchport (CE/FabricPath local) or destination Switch ID (FabricPath remote)

In FabricPath core switches, Switch ID (routing) table lookup drives Layer 2 forwarding
Destination SID lookup dictates outgoing FabricPath interface and next hop

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

46

HDR

= Packet Headers

DATA

= Packet Data

CTRL

= Internal Signaling

M1 L2 Packet Flow

Credit grant for fabric access

Supervisor Engine
Central Arbiter

Return credit to pool

12

Fabric Module 1
Fabric ASIC

Fabric Module 2
Fabric ASIC

Fabric Module 3
Fabric ASIC

Transmit to fabric

10

Receive from fabric Return buffer credit

VOQ arbitration and queuing

11

8
Fabric ASIC

ACL/QoS/ NetFlow lookups

Layer 3 Engine
Layer 2 Engine
Forwarding Engine

Layer 3 Engine
Layer 2 Engine
Forwarding Engine

Fabric ASIC

VOQs

VOQs

Submit packet headers for 4 lookup Ingress port QoS

Replication Engine
10G MAC

L2 SMAC/ DMAC lookups Return result

Replication Engine
10G MAC

LinkSec decryption

Receive packet from wire

Linksec

Module 1

2011 Cisco and/or its affiliates. All rights reserved.

13 Module 2
Transmit 15 packet on Cisco Public wire

Egress port QoS

Linksec

LinkSec encryption

14
47

e1/1

e2/1

BRKARC-3470

HDR

= Packet Headers

DATA

= Packet Data

CTRL

= Internal Signaling

F1 L2 Packet Flow

Credit grant for fabric access

Supervisor Engine
Central Arbiter

Return credit to pool

Fabric Module 1
Fabric ASIC

Fabric Module 2
Fabric ASIC

Fabric Module 3
Fabric ASIC

Transmit to fabric

VOQ arbitration and queuing

Fabric ASIC

Fabric ASIC

Receive from fabric Return buffer credit

2 SoC

Ingress L2 & ACL/QoS lookups

Receive packet from wire

1
e1/1

Module 1
e1/2

SoC
Transmit packet on wire

Module 2
e2/2

e2/1

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

48

Agenda
Chassis Architecture

Supervisor Engine and I/O Module Architecture


Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding

Classification Policies NetFlow Summary

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

49

IP Forwarding
Nexus 7000 decouples control plane and data plane

Forwarding tables built on control plane using routing protocols or static configuration
OSPF, EIGRP, IS-IS, RIP, BGP for dynamic routing

Tables downloaded to forwarding engine hardware for data plane forwarding

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

50

IP Forwarding Architecture
Routing protocol processes learn routing information from neighbors IPv4 and IPv6 unicast RIBs calculate routing/next-hop information
Unicast Forwarding Distribution Manager (UFDM) interfaces between URIBs on supervisor and IP FIB on I/O modules IP FIB process programs forwarding engine hardware on I/O modules
FIB TCAM contains IP prefixes

Supervisor Engine

BGP

OSPF

ISIS

RIP

EIGRP

URIB/U6RIB UFDM

IP FIB
Hardware

IP FIB
Hardware

IP FIB
Hardware

Adjacency table contains next-hop information


n7010# PID 20944 n7010# 3573 3574 n7010# 3836 sh processes Runtime(ms) 93 sh processes 117 150 sh processes 1272 cpu | egrep ospf|PID Invoked uSecs 1Sec 33386880 0 0 cpu | egrep u.?rib 44722390 0 0 34200830 0 0 cpu | egrep ufdm 743933460 0 0

Process ospf u6rib urib ufdm

I/O Module

I/O Module

I/O Module

Hardware FIB TCAM ADJ Table

module-9# sh processes cpu | egrep fib 1534 80042 330725 242 0.0 module-9#
BRKARC-3470

ipfib
Cisco Public

2011 Cisco and/or its affiliates. All rights reserved.

51

Hardware IP Forwarding Process


FIB TCAM lookup based on destination prefix (longest-match)

FIB hit returns adjacency, adjacency contains rewrite information (next-hop) Pipelined forwarding engine architecture also performs ACL, QoS, and NetFlow lookups, affecting final forwarding result

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

52

IPv4 FIB TCAM Lookup


Generate Lookup Key 10.1.1.10
10.1.1.2 10.1.1.3 10.1.1.4 10.10.0.10
Generate TCAM lookup key (destination IP address)

Compare lookup key

Ingress unicast IPv4 packet header


Flow Data

Forwarding Engine
Next-hop 1 (IF, MAC) Next-hop 2 (IF, MAC)

Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops

Load-Sharing Hash

10.10.0.100 10.10.0.33
10.1.1.xx 10.1.2.xx 10.1.3.xx 10.10.100.xx
HIT!

Offset

Next-hop 3 (IF, MAC)


Return lookup result

Index, # next-hops
Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops
Hit in FIB Index, returns result# next-hops in FIB DRAM Adjacency index identifies ADJ block
Cisco Public

Next-hop 4 (IF, MAC) # nexthops Adj Index Next-hop 5 (IF, MAC) Next-hop 6 (IF, MAC) Next-hop 7 (IF, MAC)

10.1.1.xx 10.100.1.xx 10.10.0.xx 10.100.1.xx


FIB TCAM
BRKARC-3470

Result

FIB DRAM
2011 Cisco and/or its affiliates. All rights reserved.

Hash selects exact next hop entry

Adjacency Table
53

ECMP Load Sharing


Up to 16 hardware load-sharing paths per prefix
10.10.0.0/16

Use maximum-paths command in routing protocols to control number of load-sharing paths Load-sharing is per-IP flow Configure load-sharing hash options with global ip load-sharing command:
Source and Destination IP addresses

Source and Destination IP addresses plus L4 ports (default) Destination IP address and L4 port

Additional randomized number added to hash prevents polarization


Automatically generated or user configurable value

10.10.0.0/16 via Rtr-A via Rtr-B

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

57

HDR

= Packet Headers

DATA

= Packet Data

CTRL

= Internal Signaling
Return credit to pool

M1 L3 Packet Flow
Fabric Module 1
Fabric ASIC
Transmit to fabric VOQ arbitration and queuing

Credit grant for fabric access

Supervisor Engine 9
Central Arbiter

12

Fabric Module 2
Fabric ASIC

Fabric Module 3
Fabric ASIC

10

Receive from fabric Return buffer credit

8
Fabric ASIC

6
Layer 3 Engine
Layer 2 Engine
Forwarding Engine

L3 FIB/ADJ lookup Ingress and egress ACL/QoS/ NetFlow lookups

11

Layer 3 Engine

Fabric ASIC

VOQs
Submit packet headers for 4 lookup

Replication Engine
10G MAC

L2 ingress and egress SMAC/ DMAC lookups Return result

Layer 2 Engine
Forwarding Engine

VOQs

Replication Engine
10G MAC

Ingress port QoS

3
Receive packet from wire

Linksec

LinkSec decryption

Module 1
2011 Cisco and/or its affiliates. All rights reserved.

13 Module 2
Transmit packet on 15 Cisco Public wire

Egress port QoS

Linksec

LinkSec encryption

14
60

e1/1

e2/1

BRKARC-3470

Layer 3 Forwarding with F1 I/O Modules


F1 modules do not natively provide Layer 3 switching
Cannot inter-VLAN route on their own

However, one or more M1/M1-XL modules can provide proxy Layer 3 services
M1 ports can proxy route for F1 modules Proxy L3 forwarding enabled by default when VDC in mixedmodule mode

Packets destined to router MAC forwarded to M1 modules for Layer 3 via internal Router Port-Channel
Selection of which port on which M1 module based on EtherChannel hash function Traffic requiring L3 from F1 modules traverses the fabric, vectoring toward M1 ports enabled for proxy L3 M1 module receiving such packets programmed to perform full ingress/egress L3 lookups

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

61

Proxy L3 Forwarding
VLAN 10 DMAC router_mac

10.1.10.100 vlan 10 e2/1 e2/2


SoC

10.1.20.100 vlan 20 e3/1 e3/2


SoC

Dest Port internal channel

2 3

F1
interface vlan 10 ip address 10.1.10.1/24 ! interface vlan 20 ip address 10.1.20.1/24

Fabric ASIC

F1

Fabric ASIC

Fabric Module

Fabric

M1
Fabric
Routing: DIP Server 2
MAC: VLAN 20 Next Hop 10.1.20.100

VOQs

6
DMAC server_2

Forwarding Engine

Forwarding Engine

VOQs

Dest Port e3/1

Replication Engine Replication Engine


7

Replication Engine 5

Replication Engine

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

e1/1 e1/2 e1/3

Cisco Public

e1/4

e1/5 e1/6

e1/7 e1/8
62

Agenda
Chassis Architecture

Supervisor Engine and I/O Module Architecture


Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding

Classification Policies NetFlow Summary

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

64

IP Multicast Forwarding
Forwarding tables built on control plane using multicast protocols
PIM-SM, PIM-SSM, PIM-Bidir, IGMP, MLD

Tables downloaded to:


Forwarding engine hardware for data plane forwarding Replication engines for data plane packet replication

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

65

IP Multicast Forwarding Architecture


Multicast routing processes learn routing information from neighbors/hosts IPv4 and IPv6 multicast RIBs calculate multicast routing/RP/RPF/OIL information

Supervisor Engine
PIM IGMP PIM6 ICMP6 BGP MSDP MRIB/M6RIB MFDM

Multicast Forwarding Distribution Manager (MFDM) interfaces between MRIBs on supervisor and IP FIB on I/O modules IP FIB process programs hardware:
FIB TCAM

IP FIB

IP FIB

IP FIB

Adjacency table
Multicast Expansion Table (MET) n7010# PID 3842 3850 n7010# 3843 3847 n7010# 3846 sh processes Runtime(ms) 109 133 sh processes 177 115 sh processes 2442 cpu | egrep pim|igmp|PID Invoked uSecs 1Sec Process 32911620 0 0 pim 33279940 0 0 igmp cpu | egrep m.?rib 33436550 0 0 mrib 47169180 0 0 m6rib cpu | egrep mfdm 743581240 0 0 mfdm

Hardware
I/O Module

Hardware
I/O Module

Hardware
I/O Module

Hardware
FIB TCAM

MET
ADJ Table

module-9# sh processes cpu | egrep fib 1534 80153 330725 242 0.0 module-9#
BRKARC-3470

ipfib

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

66

IPv4 Multicast FIB TCAM Lookup


Generate TCAM lookup key (source and group IP address)

Ingress multicast packet header


Compare lookup key

Generate Lookup Key 10.1.1.10, 239.1.1.1

Forwarding Engine
RPF, ADJ Index RPF, ADJ Index
RPF, ADJ Index RPF, ADJ Index RPF, ADJ Index FIB DRAM

10.1.1.12, 239.1.1.1 10.1.1.10, 232.1.2.3


10.4.7.10, 225.8.8.8
HIT! 10.1.1.10, 239.1.1.1

MET Index MET Index MET Index


MET Index
Adj Index

10.6.6.10, 239.44.2.1
FIB TCAM
Hit in FIB returns result in FIB DRAM

MET Index
Adjacency Table

Result

Identifies multicast adjacency entry

Return lookup result

Replication Engine
Replication for each OIF in MET block

OIFs OIFs OIFs OIFs


MET
Cisco Public

Replicate

MET index used to find OIFs for replication

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

67

Egress Replication
IIF

Distributes multicast replication load among replication engines of all I/O modules with OIFs

Local OIF

Module 1
Replication MET Engine

Input packets get lookup on ingress forwarding engine For OIFs on ingress module, ingress replication engine performs the replication For OIFs on other modules, ingress replication engine replicates a single copy of packet over fabric to those egress modules
Each egress forwarding engine performs lookup to drive replication
2 Fabric Module

Fabric ASIC

Fabric Copy

Fabric ASIC

Fabric ASIC

Fabric ASIC

Fabric ASIC

Replication MET Engine

Replication MET Engine

Replication MET Engine

Replication engine on egress module performs replication for local OIFs


Local OIFs
BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

Local OIFs

Local OIFs
71

Cisco Public

HDR

= Packet Headers

DATA

= Packet Data

L3 Multicast Packet Flow


Fabric Module 1
Fabric ASIC
Transmit to fabric

Fabric Module 2
Fabric ASIC

Fabric Module 3
Fabric ASIC
Dequeue multicast distribution copy from fabric

11

VOQ queuing

10
Fabric ASIC
Transmit multicast fabric distribution packet 9 Submit packet headers for 4 lookup Ingress port QoS

6
Layer 3 Engine Layer 2 Engine
Forwarding Engine

L3 multicast FIB lookup Ingress ACL/QoS/ NetFlow lookups Egress ACL/QoS/ NetFlow lookups L2 ingress snooping lookup

12

Layer 3 Engine

Fabric ASIC
Replicate for local OIF delivery

VOQs
Replication Engine 10G MAC

15

Layer 2 Engine

VOQs

13

Return MET result

Receive packet from wire

Linksec

Replicate for fabric delivery

Forwarding Engine
Egress port QoS

Replication Engine
10G MAC

Submit packet headers for egress lookups

14
LinkSec encryption

Module 1
LinkSec decryption

16

L2 egress snooping lookup

17 Module 2
Transmit packet on wire

Linksec

18

e1/1

2
19

e2/1

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

72

Agenda
Chassis Architecture

Supervisor Engine and I/O Module Architecture


Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding

Classification Policies NetFlow Summary

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

73

Classification Policies
Enforce security and QoS policies based on Layer 2, Layer 3, and Layer 4 information Classification TCAM (CL TCAM) provides ACL lookups in forwarding engine
64K(non-XL) or 128K (XL) hardware entries in M1 modules 1K ingress/1K egress hardware entries per SoC in F1 modules (16K/16K entries per module)

Security policies
Router ACL (RACL) for IPv4, IPv6, ARP VLAN ACLs (VACLs) for IPv4, MAC Port ACLs (PACLs) for IPv4, MAC Secure Group Tag ACLs (SGACLs) for Cisco TrustSec

QoS classification policies for IPv4 and IPv6


Ingress (Per-L2 switchport, per-VLAN, per-interface routed/subinterface) Egress (Per-VLAN, per-interface routed/subinterface)

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

74

CL TCAM Lookup (ACL)


Packet header: SIP: 10.1.1.1 DIP: 10.2.2.2 Protocol: TCP SPORT: 33992 DPORT: 80
Compare lookup key
Generate TCAM lookup key (source/dest IPs, protocol, L4 ports, etc.)

Security ACL ip access-list example permit ip any host 10.1.2.100


deny ip any host 10.1.68.44 deny ip any host 10.33.2.25 permit tcp any any eq 22 deny tcp any any eq 23

deny udp any any eq 514

Generate Lookup Key

SIP | DIP | Protocol | SPORT | DPORT

permit tcp any any eq 80 permit udp any any eq 161

10.1.1.1 | 10.2.2.2 | 06 | 84C8 | 0050

Forwarding Engine
Permit Deny

xxxxxxx | 10.2.2.2 xx | xxx | xxx xxxxxxx | 10.1.2.100| |xx | xxx | xxx

xxxxxxx | 10.1.68.44 | xx | xxx | xxx xxxxxxx | 10.33.2.25 | xx | xxx | xxx

Deny
Permit Deny Deny Permit Result

X=Mask

xxxxxxx | xxxxxxx | |06 | |xxx | 0050 xxxxxxx xxxxxxx 06 xxx | 0016


xxxxxxx | xxxxxxx | 06 | xxx | 0017 xxxxxxx | xxxxxxx | 11 | xxx | 0202

HIT! xxxxxxx | xxxxxxx | 06 | xxx | 0050


xxxxxxx | xxxxxxx | 11 | xxx | 00A1 CL TCAM
Hit in CL TCAM returns result in CL SRAM
Cisco Public

Permit
CL SRAM

Result affects final packet handling

Return lookup result

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

76

Agenda
Chassis Architecture

Supervisor Engine and I/O Module Architecture


Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding

Classification Policies NetFlow Summary

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

78

NetFlow
NetFlow support provided by M1 I/O modules
No support for F1F1 or F1M1 proxy-routed flows

NetFlow table is 512K entries, shared between ingress/egress NetFlow Hardware NetFlow entry creation
CPU not involved in NetFlow entry creation/update

All modules have independent NetFlow table Full and sampled NetFlow supported by hardware

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

79

Full vs. Sampled NetFlow


NetFlow configured per-direction and per-interface
Ingress and/or egress on per-interface basis

Each interface can collect full or sampled flow data Full NetFlow: Accounts for every packet of every flow on interface, up to capacity of NetFlow table Sampled NetFlow: Accounts for M in N packets on interface, up to capacity of NetFlow table

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

83

Sampled NetFlow
Random packet-based sampling

M:N sampling: Out of N consecutive packets, select M consecutive packets and account only for those flows in the hardware NetFlow table
Sampled flows aged and exported from NetFlow table normally Advantages
Reduces NetFlow table utilization Reduces CPU load on switch and collector

Disadvantages
Accuracy may be sacrificedcollector or user must extrapolate total traffic load based on configured sampling rate
BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

84

NetFlow Data Export


To NetFlow Collector

Generate NetFlow v5 or v9 export packets

I/O Module
Fabric ASIC via Inband VOQs

Supervisor Engine

LC CPU

NetFlow Table

Aged Flows

Forwarding Engine I/O Module


LC CPU
NetFlow Table

Hardware Flow Creation

Main CPU

Switched EOBC

Aged Flows

Forwarding Engine I/O Module


LC CPU
NetFlow Table

Hardware Flow Creation

via mgmt0

Mgmt Enet

To NetFlow Collector

Aged Flows

Forwarding Engine

Hardware Flow Creation

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

87

Agenda
Chassis Architecture

Supervisor Engine and I/O Module Architecture


Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding

Classification Policies NetFlow Summary

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

89

Nexus 7000 Architecture Summary


Variety of front-panel interface and transceiver types with LinkSec, VOQ, and other advanced hardware features
Future-proofed chassis designs with density and airflow options

Control plane protocols, system and network management

Supervisor Engines

I/O Modules

Chassis

Fabrics

Forwarding Engines
BRKARC-3470

Hardware services, including unicast/multicast, bridging/routing, ACL/QoS classification, and NetFlow statistics
Cisco Public

Lossless-capable fabric with 230G/slot bandwidth to interconnect I/O modules and provide investment protection
90

2011 Cisco and/or its affiliates. All rights reserved.

Conclusion
You should now have a thorough understanding of the Nexus 7000 switching architecture, I/O module design, packet flows, and key forwarding engine functions

Any questions?

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

91

Complete Your Online Session Evaluation


Receive 25 Cisco Preferred Access points for each session evaluation you complete. Give us your feedback and you could win fabulous prizes. Points are calculated on a daily basis. Winners will be notified by email after July 22nd.

Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center. Dont forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and ondemand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

93

Visit the Cisco Store for Related Titles http://theciscostores.com

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

94

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

95

Thank you.

BRKARC-3470

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

96