Cisco ASR 9000 System Architecture

Yongzhong Peng, Technical Marketing Engineer, SP Routing Infrastructure

CCIE #13669, R&S, SP and SAN
BRKARC-2003
Agenda
ASR9000 Products Overview
ASR9000 Distributed System Architecture and RP/RSP Details
ASR9000 Line Card Details
Common Line Card Components
Typhoon Hardware Architecture
Tomahawk Hardware Architecture

ASR9000 Switch Fabric

ASR9000 Packet Flow and Control Plane Architecture
Packet Flow Unicast/Multicast Transit Traffic and Local Control Traffic
L2/L3 Control Plane Architecture

IOS-XR
IOS-XR Architecture Overview
Virtualized IOS-XR XR 64 bit: the foundation for NG-ISSU

Summary
ASR9000 Products Overview
 Edge Routing -- Cisco ASR9000 Market Roles
1. High-End Aggregation &
Transport

Carrier Ethernet Cable/MSO 1. Mobile Backhaul

2. L2/Metro Aggregation
Mobile Backhaul 3. CMTS Aggregation
4. Video Distribution &
Services

Web/OTT
Multiservice Edge 2. DC Gateway Router

1. DC Interconnect
DC gateway 2. DC WAN Edge
Broadband 3. WEB/OTT
Gateway
3. Services Router
Large Enterprise 1. Business Services
WAN 2. Residential Broadband
3. Converged Edge/Core
4. Enterprise WAN
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Cisco ASR 9000 System Comprehensive Portfolio
Compact & Powerful High Density Service Edge
Flexible Service Edge
Access/Aggregation and Core
Small footprint with full IOS-XR Optimized for ESE and MSE with high Scalable, ultra high density service
for distributed environments M-D scale for medium to large sites routers for large, high-growth sites
ASR 9922

One Platform, One OS, One Family

ASR 9912
nV Satellites
ASR 9000v, 901, 920
ASR 9010 ASR 9910

ASR 9006
ASR 9904
ASRASR
9001

Fixed 2RU 2 LC/6RU 4 LC/10RU 8 LC/21RU 8 LC/21RU 10 LC/30RU 20 LC/44RU

240 Gbps 16 Tbps 7 Tbps 14 Tbps 64 Tbps 80 Tbps 160 Tbps

MSE E-MSE Peering P/PE CE Mobility Broadband

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
 ASR9K 100GE Readiness w/ Investment Protection
991
0
100

3.4T/slot
32 991
Full Duplex, Line Rate Capacity (Tbps)

992 2 0.5 Watts/Gbps

2 680x100G/chassis
990 1.2T/slot 160x400G/chassis
4
16
800G/slot
1.5 Watts/Gbps
1600x10G/chassis
901 160x100G/chassis
7 0
900
6 360G/slot
2.5 Watts/Gbps
1 720x10G/chassis Over an 8 Year Period:
40x100G/chassis
80G/slot 300x Capacity Increase
8 Watts/Gbps 95% Power Reduction
64x10G/chassis
40G/slot
10 Watts/Gbps
32x10G/chassis

2009 2012 Time (Year) 2015 2018

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
ASR 9001/ASR 9001-S Compact Chassis
Side-to-Side airflow Front-to-back air flow with air flow
2RU baffles, 4RU, require V2 fan
Sub-slot 0 with MPA* Sub-slot 1 with MPA*

Supported MPAs:
Redundant Fixed 4x10G
(AC or DC) 20x1GE
Power Supplies 2x10GE
SFP+ ports
Field Replaceable 4x10GE Fan Tray
1x40GE
Field Replaceable

*MPA: Modular Port Adapter

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
 ASR 9001-S Compact Chassis
Side-to-Side airflow Front-to-back air flow with air flow
2RU baffles, 4RU, require V2 fan

Supported MPAs: Pay As You Grow

20x1GE Low entry cost
2x10GE SW License upgradable to full 9001
4x10GE
1x40GE

Sub-slot 0 with MPA Sub-slot 1 with MPA

60G bandwidth are disabled by

software. SW license to enable it
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Cisco ASR 9006 Overview

Front-to-back air flow Feature Description

with air flow baffles, Total Capacity 3.68T
13RU, vertical Capacity per Slot 920G

Side-to-side airflow, 10 RU Slots 6 slots - 4 Line Cards and 2 RSPs

Rack size 10RU
Power 1 Power Shelf, 4 Power Modules
2.1 KW DC / 3.0 KW AC supplies
Fan: Side to Side Airflow
Optional Baffle for Front-to-Back Airflow
2 Fan Trays, FRU
RSPs Integrated Fabric, 1+1 Redundancy
Line cards Tomahawk
Typhoon
VSM
SIP700 & SPAs

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Cisco ASR 9010 Overview

Feature Description
Total Capacity 7.36T
Capacity per Slot 920G
Slots 10 slots - 8 Line Cards and 2 RSPs
Rack size 21RU
Power: 2 Power Trays
2.1 KW DC / 3.0 KW AC supplies
4.4 KW DC / 6.0 KW AC supplies
Fan: Front to Back Airflow
2 Fan Trays, FRU
RSPs Integrated Fabric, 1+1 Redundancy
Line cards Tomahawk
Typhoon
VSM
SIP700 & SPAs

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
 Cisco ASR 9904 Overview

Feature Description
Front-to-back air flow with air flow Total Capacity 6T
baffles, 10RU Capacity per Slot 3T
Slots 4 slots - 2 Line Cards and 2 RSPs
Rack size 6RU
Power 1 Power Trays, 4 Power Modules
2.1 KW DC / 3.0 KW AC supplies
Fan Side to Side Airflow, Front-to-Back Optional
1 Fan Tray, FRU
RSPs Integrated Fabric, 1+1 Redundancy
Line cards Tomahawk
Typhoon
VSM

SW XR 5.1.0 September 2013

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Cisco ASR 9912 Overview

Features Description
Total Capacity 30T
Capacity per Slot 3T
Slots 10 slot chassis
Rack Size 30 RU
Power 4 Power Trays
2.1 KW DC / 3.0 KW AC supplies
4.4 KW DC / 6.0 KW AC supplies
Fan 2 Fan Trays
Front to back airflow
RP 1+1 RP redundancy
Fabric (SFC) 6+1 fabric redundancy
SW XR 4.3.2 Shipping

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Cisco ASR 9922 Overview

Features Description
Total Capacity 60T
Capacity per Slot 3T
Slots 20 Line cards, 2 RP, 7 SFC
Rack Size 44 RU (Full Rack)

Power 4 Power Trays

2.1 KW DC / 3.0 KW AC supplies
4.4 KW DC / 6.0 KW AC supplies
Fan 4 Fan Trays
Front to back airflow
RP 1+1 RP redundancy
Fabric (SFC) 6+1 fabric redundancy
Line cards Tomahawk
Typhoon, VSM

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Cisco ASR 9910 Overview
Feature Description
Total Capacity 24T
Capacity per Slot 3T
Slots 10 slots - 8 Line Cards and 2 RSPs
Rack size 21RU
Power: 2 Power Trays
4.4 KW DC supplies
6.0 KW AC supplies
Fan: Front to Back Airflow
2 Fan Trays, FRU
RSPs Integrated Fabric, 1+1 Redundancy
Fabric Cards 5 Fabric cards on rear of chassis for additional capacity
230G per FC at FCS
Up to 6+1 Redundancy using RSPs integrated fabric
Line cards Tomahawk
Typhoon
VSM

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
 Cisco ASR 9910 Details

Greater Capacity, Greater Flexibility:

2 Fan Trays
Start with 2 RSPs, then add fabric
cards to scale beyond 460G per slot
ASR 9910 comes prepared with
8 Line Card Slots
sufficient power and cooling to
support high density 100G line cards. 2 RSPs
Available with IOS XR 6.1 with the 5 Fabric Cards
same feature parity as the rest of the
ASR 9000 family.

2 Power Trays

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
ASR-9910 Mid-plane Architecture
New Mid-plane Architecture for Greater Flexibility

RSP0 230G FT0

M FT1
1G/10G/40G/100G
LC0 I
D

P
L Fabric card 0 230G
1G/10G/40G/100G A
LC7
N
E
RSP1 230G Fabric card 4 230G

Power Tray 0 Fabric

Control
Power Tray 1
Line Side

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
ASR 9000 Hardware Components RP/RSP
ASR 9922

ASR 9010 ASR 9910

ASR 9912

ASR 9006

ASR 9904

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
ASR 9900 Route Processor
Common for ASR 9912 and ASR 9922
ASR 9922 Built for massive control place scale
Ultra High Speed Control Plane with Multi-Core Intel CPU
Huge Scale through High Memory options
Time and Synchronization Support
ASR 9912 RP1 RP2
Availability Q1CY12 Q1CY15
Processor Four Cores Eight Cores
2.1GHz 2.2GHz
NPU Bandwidth 60G 240G
Fabric Planes 5 7
Fabric Capacity 5x230Gbps 7x230Gbps
Memory 6G for TR 16G for TR
12G for SE 32G for SE
SSD 2x 16GB Slim SATA 2x 32GB Slim SATA
LC Support Typhoon Tomahawk/Typhoon
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
ASR 9000 Route Switch Processor
Common for ASR 9904, ASR 9010, and ASR 9010
Common internal HW with RP for feature parity on IOS XR
Integrated Multi-Stage Switch Fabric
ASR 9010 TR and SE Memory options
Time and Synchronization Support

RPS440 RSP880

ASR 9006
Availability Q1CY12 Q1CY15
Processor Four Cores - 2.1GHz Eight Cores - 2.2GHz
NPU Bandwidth 60G 240G
Fabric Capacity 440G 880G
Memory 6G for TR 16G for TR
ASR 9904
12G for SE 32G for SE
SSD 2x 16GB Slim SATA 2x 32GB Slim SATA
LC Support Typhoon Tomahawk/Typhoon

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
ASR 9910 Route Switch Processor
Specific for ASR 9910
Common internal HW with RP for feature parity on IOS XR
Integrated Multi-Stage Switch Fabric
TR and SE Memory options
Time and Synchronization Support
ASR 9910

A99-RSP
Processor Eight Cores 1.9GHz
NPU Bandwidth 240G
Fabric Planes 2
Fabric Capacity 230G
Memory 16G for TR
32G for SE
SSD 2x 32GB Slim SATA
LC Support Tomahawk/Typhoon

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
ASR 9900 - Switch Fabric Cards
ASR 9922
Common for ASR 9912 and ASR 9922
SFC-S for ASR 9910
7 Fabric Card Slots
5 Fabric Card Slots for ASR 9910 (2 on RSPs => total 7)
Decoupled, multi-stage switch fabric hardware
ASR 9912
True HW separation between control and data plane
Add bandwidth per slot easily & independently
ASR 9910

SFC110 SFC2/SFC-S
Fabric Capacity per 110G 230G
SFC
Fabric Capacity 660G N+1 1.38T N+1
Per Line Card Slot 770G N+0 1.61T N+0
Fabric Redundancy N+1 N+1

LC Support Typhoon Tomahawk/Typhoon

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
ASR 9000 - Fans
ASR 9922-FAN-V2

ASR 9006-FAN-V2

V2 Fan provides higher cooling capacity for ultra high density cards.
New material capable of dissipating more heat
Dynamic Fan Speed Adjustment (DFSA) based on temperature
reading from multiple sensors on the board.
Fans are grouped into multiple groups with each group mapped to a
set of fixed slots for temperature control.
In service upgrade

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
ASR 9000 - PAYG Power
N+1 Redundancy for DC
N+N Redundancy for AC
Typhoon and Tomahawk LCs supported with V2/V3

Line Card Used BW Consumption at 27C Consumption at 40C

Typhoon 360G / Slot 2.8 Watts/G 3.5 Watts/G

Tomahawk 1T / Slot 1.6 Watts/G 1.9 Watts/G

Version AC DC Chassis

V2 3.0KW 2.1KW ASR9006, ASR9010,

ASR9904, ASR9910, ASR9912, ASR9922
V3 6.0KW 4.4KW ASR9010, ASR9910, ASR9912, ASR9922

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
ASR9000 Distributed System
Architecture and RP/RSP Details
 ASR 9000 System Architecture
Distributed CPU
Fully Distributed Control Plane for High
Performance and Multi-dimensional RSP
Line Card
Scale Distributed Arbiter
CPUs on RP/RSP and LC RSP
CPU
Control Plane Split among RP/RSP and CP
BITS/DTI
LC CPU U
Arbiter CPU FIA
Arbiter
Fully Distributed Data Traffic Processing 6x10GE
SFP +
NP FIA Arbiter CPU FIC
for High Throughput and Data Plane Scale 6x10GE
SFP +
6x10GE
NP
NP
FIA
FIA
SFP +
6x10GE
Multi-stage Switch Fabric Architecture SFP +
6x10GE
NP
NP
FIA
FIA

SFP + Swit
6x10GE ch
Distributed NP for Data Traffic Processing
NP FIA
SFP + Fabri
6x10GE FIA Switch
NP c
SFP + Fabric
6x10GE FIA
NP
Distributed Arbitration SFP +
6x10GE
SFP +
NP FIA
6x10GE FIA
NP
SFP +
6x10GE
NP FIA
SFP +
6x10GE
NP FIA
SFP +

Network Processor Integrated Switch Fabric

Switch Fabric
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
 ASR 9000 System Architecture Dedicated SFC
Similarities: Line Card RP
Fully Distributed Control Plane for High
Performance and High Multi-dimensional
Control Plane Scale RSP
Fully Distributed Data Traffic Processing for CPU
CP
High Throughput and Data Plane Scale Arbiter CPU U
BITS/DTI

6x10GE
SFP +
NP FIA Arbiter CPU
FIA
Arbiter
6x10GE
NP
Benefits: 6x10GE
SFP +
SFP +
NP FIA
FIA

6x10GE
NP FIA
Separated Switch Fabric Cards SFP +
6x10GE
SFP +
NP FIA
Swit
6x10GE ch
NP FIA
Easy Switch Fabric Upgrading SFP +
6x10GE
SFP +
NP FIA
Fabri
cSwitch
Fabric
6x10GE FIA
NP
Easy Bandwidth Capacity Expansion SFP +
6x10GE
NP FIA
SFP +
6x10GE FIA
NP

SFP +
Provide N+1 Switch Fabric Redundancy 6x10GE
NP FIA
SFP +
6x10GE
NP FIA
SFP +

Dedicated Switch Fabric

Switch Fabric
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
ASR9000 Fully Distributed Control Plane
CPU
LPTS (local packet transport service):
control plane policing
RP
CPU Punt
FPGA FIA

Switch Fabric Switch Fabric

Punt Switch

Control 3x10GE
SFP +
Typhoo
LPTS
n
FIA
packet 3x10GE
SFP +
NP
RP CPU: Routing, MPLS, IGMP, PIM,
3x10G
E NP HSRP/VRRP, etc

Fabric ASIC
Switch
SFP + FIA
3x10GE
SFP +
NP
3x10GE
NP
LC CPU: ARP, ICMP, BFD, NetFlow,
SFP +
3x10GE
NP
FIA OAM/CFM, L2 Protocols etc
SFP +
3x10GE
SFP +
NP
3x10GE
NP
FIA NP Offloading: BFD
SFP +

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
RSP/RP Engine Architecture
Timing Time
Clock
FPGA
RAM MEM Timing Domain

SDD
Ext. USB Up to 4
Mgt Eth 10GE Front Panel
CPU
Ether SFP+
Mgt Eth Switch
EOBC 10GE SFP+
Console
Aux eUSB Punt Fabric Arbitration Arbitration
FPGA Interface

I/O FPGA
Crossbar
Fabric
Alarm ASIC

NVRAM Boot Flash Crossbar

Fabric
Front Panel CPU Complex ASIC

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
RP Engine Architecture
Timing Time
Clock
FPGA
RAM MEM Timing Domain

SDD
Ext. USB Up to 4
Mgt Eth
CPU
Ether SFP+ External Panel
10GE
Mgt Eth Switch
EOBC 10GE SFP+
Console
Aux eUSB Punt Fabric Switch To Switch
FPGA Interface Fabric Fabric Cards
I/O FPGA

Alarm

NVRAM Boot Flash

External Panel CPU Complex

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
 ASR-9001 System Architecture Overview
MPAs
2,4x10GE
20xGE
1x40GE
Typhoon
FIA

Switch Fabric
On-board SFP+ 10GE
2x10
SFP+ SFP+ 10GE LC RP

ASIC
ports Internal CPU CPU
On-board SFP+ 10GE EOBC
2x10
SFP+ SFP+ 10GE
ports
MPAs Typhoon
2,4x10GE
20xGE
FIA
1x40GE

RP CPU and linecard CPU same arch. as the larger systems

Single Crossbar Switch Fabric ASIC

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
 ASR-9001S System Architecture Overview
MPAs
2,4x10GE
20xGE
1x40GE
Typhoon
FIA

Switch Fabric
On-board SFP+ 10GE
2x10
SFP+ SFP+ 10GE LC RP

ASIC
ports Internal CPU CPU
On-board SFP+ 10GE EOBC
2x10
SFP+ SFP+ 10GE
ports
MPAs Typhoon
2,4x10GE
20xGE
FIA
1x40GE
Disabled by Default
Upgradable via License

Half of the System Disabled by Default

License Upgradable to Full Capacity
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
ASR9000 Line Card Details
Edge Linecard Silicon Slice Evolution
1st Gen
Trident
Class Trident Octopus Santa Cruz PowerPC
120G 90nm 130nm 130nm Dual Core
15 Gbps 60 Gbps 90 Gbps 1.2 Ghz

2nd Gen
Typhoon
Class Typhoon Skytrain Sacramento PowerPC
360G 55nm 65nm 65nm Quad Core
60 Gbps 60 Gbps 220 Gbps 1.5 Ghz

3rd Gen
Tomahawk
Class Tomahawk Tigershark SM15 X86
800G 28nm 28nm 28nm 6 Core
240 Gbps 200 Gbps 1.20 Tbps 2 Ghz

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
 ASR 9K Ethernet Line Card Overview
2nd LC Typhoon
NPU: 60Gbps,
~45Mpps

A9K-MOD80 A9K-MOD160

A9K-36x10GE A9K-2x100GE A9K-24x10GE MPAs

20x1GE
2x10GE
4x10GE
1x40GE
A9K-40GE 2x40GE
A9K-4T16GE

3rd LC Tomahawk MPAs

1x100GE
NPU: 240Gbps, 2x100GE
~150Mpps A9K-MOD400/MOD200 20x10GE
+
Typhoon
A9K-400G-DWDM MPAs
A9K-12x100GE A9K-8x100GE A9K-4x100GE
-TR, -SE
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 Runs distributed control plane
Line Card Components protocols for increased scale
BFD, CFM, ARP
Main forwarding engine L2 and L3 lookups
Receive FIB table from RP and
Multicast replication toward Optics program hardware forwarding table
User level QoS and Security features

CPU

NPU P1
PHY P1
FIA
P2
P3 Switch
TM
P2
P3
BE Fabric
BE ASIC

Dedicated queue ASIC TM (traffic Provides data connection to switch fabric

manager) per NPU for QoS functions
Manage VoQ, Superframe and loadbalancing
User Configurable Queue on TM. data traffic across switch fabric
Default Port Queue Always Created. Mcast replication table for replication toward
NPs
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Network Processor Architecture Details
TR and SE has different
TR and SE has same
memory size
memory size NPU Complex

STATS MEMORY
FIB MAC
LOOKUP Forwarding chip (multi core) FRAME MEMORY
MEMORY TCAM
-

TCAM: VLAN tag, QoS and ACL classification

Stats memory: interface statistics, forwarding statistics etc
Frame memory: buffer, Queues
Lookup Memory: forwarding tables, FIB, MAC, ADJ
TR/SE
Different TCAM/frame/stats memory size for different per-LC QoS, ACL, logical interface scale
Same lookup memory for same system wide scale mixing different variation of LCs doesnt impact system wide scale

-TR: transport optimized, -SE: Service edge optimized

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Typhoon Forwarding Engine
60Gbps bidirectional/120 Gbps unidirectional throughput
Feature rich and highly programmable
Enhanced Video Monitoring
Tunneling services optimized
Buffering for lossless Video support
Sophisticated Traffic Manager implementation
256k queues (child)
32k schedulers (parent)
4K schedulers (grand parent)
Scalable WRED
256k policers
Coupled Policer capable

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Typhoon LC: 24x10G Ports Architecture
CPU
3x 10G
3x10GE Typhoon 30G
60G
SFP + (NP)
3x 10G FIA 60G RSP440 Switch
3x10GE Typhoon 30G Fabric
SFP + (NP)
3x 10G
3x10GE Typhoon 30G 60G
SFP + (NP)
3x 10G FIA 60G
Linecard RSP 0
3x10GE Typhoon 30G
SFP + (NP) Local
3x 10G Switch 8x55G
3x10GE Typhoon 30G
60G Fabric
SFP + (NP)
FIA Complex
3x 10G 60G
3x10GE Typhoon 30G
SFP + (NP) RSP 1
3x 10G
3x10GE Typhoon 30G 60G
SFP + (NP)
3x 10G FIA 60G
3x10GE Typhoon 30G
SFP + (NP)

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Typhoon LC: 24x10G Ports Architecture
CPU
3x 10G
3x10GE Typhoon 30G
60G
SFP + (NP)
3x 10G FIA 60G RSP440 Switch
3x10GE Typhoon 30G Fabric
SFP + (NP) No Congestion
3x10GE
3x 10G
Typhoon
Fully Non-Blocking
30G 60G
SFP + (NP)
3x 10G FIA 60G
Linecard RSP 0
3x10GE Typhoon 30G
SFP + (NP) Local
3x 10G Switch 8x55G
3x10GE Typhoon 30G
60G Fabric
SFP + Typhoon
(NP) NPU BW and Performance
FIA Complex

3x10GE 120G and3x90Mpps
10G
Uni-directional
Typhoon 30G
60G

+ 60G
SFP and 45Mpps full-duplex
(NP) (each direction ingress/egress) RSP 1
3x 10G
3x10GE Typhoon 30G 60G
SFP + (NP)
3x 10G FIA 60G
3x10GE Typhoon 30G
SFP + (NP)

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Typhoon Line Cards Architectures
36x10G Port LC 2x100G Port LC
CPU CPU
6x10GE
SFP + NP FIA iNP FIA
6x10GE 100GE
SFP + NP FIA CFP
eNP FIA
RSP 0 RSP 0
6x10GE
SFP + NP FIA
Switch 8x55 Switch 8x55
Fabric G Fabric G
6x10GE
SFP + NP FIA iNP FIA
6x10GE
RSP 1 100GE eN RSP 1
NP FIA CFP FIA
SFP + P
6x10GE
SFP + NP FIA

MOD80 LC MOD160 LC
CPU CPU

NP FIA NP FIA
MPA MPA
Bay0 Bay0
NP FIA
RSP 0 RSP 0

Switch 8x55 Switch 8x55

Fabric G Fabric G
NP FIA NP FIA
MPA RSP 1 RSP 1
MPA
Bay1 Bay1 NP FIA

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Virtualized Services Module (VSM) Overview
Service-3 Service-4
ASR 9000 VSM
Service-1 Service-2 Data Center Compute:
VM-4 VM-3
VM-1 VM-2 4 x Intel 10-core x86 CPU
VMM 2 Typhoon NPU for hardware network processing
OS / Hypervisor
120 Gbps of Raw processing throughput
HW Acceleration
40 Gbps of hardware assisted Crypto throughput
Hardware assist for Reg-Ex matching
Virtualization Hypervisor (KVM)
Service VM life cycle management integrated into IOS-XR

CGN Anti-DDOS

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
 VSM Details
RSP/RP
Switch Fabric

DDR3
DDR3 DIMM Crypto Typhoon NPU
DIMM Crypto

Linecard
10GE/FCoE SFP+

local Fabric
Complex
RSP/
Data
RP 0
Path
Switch
Typhoon NPU

DDR3 Crypto
DIMM DDR3 Crypto RSP
DIMM /RP1

Virtualized Services Sub- Module Router Infrastructure Sub-Module

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Tomahawk Forwarding Engine and LC
High Density/Scale
240 Gbps throughput/ 150 MPPS+ for
line rate 100GE ports
Increased scale for multi-dimensional
profiles
2M MAC addresses(6M Capacity)
10M v4 or 5M v6 routes
1M queues/NP
Flexible Interface Support
Supports dense 100GE/40GE/10GE
Mix and match interface types on the
same LC
CPAK Advantage
Pluggable CPAK based on customer
demand (1x100GE, 2x40GE, 10x10GE)
In-line Security
MACSec support in-line on LC at line rate
100G (800G/LC)
Green Design
Ability to power down components for power
savings
High Availability Customized Silicon Hitless
FPD Upgrades
Hardware Offload of CPU Intensive Protocols
Elimination of loopback for most feature paths

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Tomahawk NPU Architecture
TCAM
Tomahawk NPU 4 TM loopback/replication ports 4x36Gbps

TM0
Line port Input bypass Line port Output

WRED
I/F x12 I/F x12
ICU & Pre-parse Line port Output

TOP Resolve
TOP Search
Line port Input

TOP Modify
TOP Parse
I/F x12

SPri WRR
I/F x12
ICFDQ
per flow queuing
TM1 FIA Output I/F
FIA Input I/F
bypass x16

WRED
x16

FIA Input I/F FIA Output I/F

x16 x16

per flow queuing

TOPs bypass

Internal MEM, Cache

DRAM Frame Buffers
Packet classification, preparse complete 64 groups x 4 CoS queues
frame header fields Strict priority round robin scheduling
HW flow hashing to map traffic flows to HW based EFD for low priority pkt
ICFDQ group priority queues ICFDQ multicast replication w/o pps limitation
(Typhoon ICFC repl <25mpps)
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Tomahawk LC: 8x100GE Architecture
Slice Based Architecture Ivy Bridge CPU
VoQ buffering, Fabric credits,
mcast hashing, scheduler for fabric
FPOE, Auto-Spread, Complex
L2/L3/L4 lookups, all VPN types, all DWRR, RBH, replication
CPAK: Macsec Suite B+, G.709, OTN, and egress port
feature processing, mcast
100G, 40G, 10G Clocking
replication, QoS, ACL, etc

240G 240G
PHY
Tomahawk Tigershark
Up to
NP FIA 14x115G

240G 240G

Separated Switch Fabric

PHY
Tomahawk Tigershark
NP FIA

(SM15)
XBAR
240G 240G
PHY
Tomahawk Tigershark
NP FIA

240G 240G
Per Slice Power Management:
PHY
Tomahawk
(100-200W Power Savings)
Tigershark
PE1(admin-config)# hw-moduleNP FIA
power disable slice [0-3] location

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Tomahawk LC: 8x100GE with RSP880
240G
240G
Tomahawk
PHY
(NP)
FIA RSP880 Switch
Fabric

240G 240G
Tomahawk
PHY
(NP)
FIA Linecard RSP 0
Local
Switch
240G
240G Fabric
Tomahawk Complex
PHY
(NP) FIA

RSP 1
240G 240G
Tomahawk
PHY
(NP) FIA

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Tomahawk Line Card Architectures
8x100G/80x10G Port LC 4x100G/40x10G Port LC
CPU CPU
CPA CPA
K PHY NP FIA K PHY NP FIA
CPA
K
CPA CPA
K K FIA
CPA PHY NP FIA PHY NP
Switch Up to Switch
K
CPA Fabric 14x115G CPA Fabric Up to
K K
CPA PHY NP FIA PHY NP FIA 14x115G

K
CPA CPA
K K
CPA PHY NP FIA PHY NP FIA
K

MOD400 LC MOD200 LC
CPU CPU
P P
Bay H
Y
NP FIA Bay H
Y

SerDes
XBAR
Switch
Fabric Up to
SerDes
XBAR
NP FIA Switch
Fabric Up to
14x115G 14x115G

P P
Bay H
Y
NP FIA Bay H
Y

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Tomahawk Modular LCs Mod400 & Mod200
MPAs Supported:
MPA #1 2 x 100G 2x100GE CPAK. 1x100GE CPAK. 20x10GE SFP+
Mod400 LC

(2x100G) CPAKs All Typhoon MPA

Flexibility to use CPAK 10G/40G/100G optics on 100G MPAs

MPA #2 20 x 10G 2 Flavors for Flexibility

(20x10G) SFP+ MOD400 has 2 Tomahawk ASICS
MOD200 has 1 Tomahawk ASIC

MOD200 Comb 1 Comb 2 Comb 3 MOD400 Comb 1 Comb 2 Combo 3 Comb 4

Support Matrix Support
Matrix
EP0 2x100G-MPA 20x10G- 1x100G or any
EP0 2x100G-MPA 20x10G-MPA 2x100G-MPA 1x100G or any
MPA Typhoon MPAs
Typhoon MPAs

EP1 None None 1x100G or any

Typhoon MPAs EP1 2x100G-MPA 20x10G-MPA 20x10G-MPA 2x100G-MPA
OR
20x10G-MPA

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Skyhammer 12x100GE Line Card at a Glance

Skyhammer is a 12 port 100GE line card for ASR 99xx routers

Contains six Tomahawk slices with no external TCAM
7 fabrics
Based on QSFP28 optics
Optimized for lowest power consumption per Gigabit
Full software feature parity with existing 100G line cards (lower scales)
Targeted mainly towards OTT/Web LSR/LER requirements

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Skyhammer LC: 12x100GE Architecture
240G
Tomahawk
PHY
(NP) FIA
240G
Tomahawk 14x115G
PHY
(NP)
FIA
240G

Separated Switch Fabric

Tomahawk Linecard
PHY
(NP) FIA
Local
240G
Switch
Tomahawk Fabric
PHY
(NP)
FIA Complex
240G
Tomahawk
PHY
(NP)
FIA
240G
Tomahawk
PHY
(NP) FIA

Slice: per slice power management (100-200W Power Savings)

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
 Flex 100G CPAK

Investment
Protection
Start with 10 GE
and upgrade to
100 GE in the
future
100 GE LR4 10x10 GE 2x40 GE 100 GE SR10

CPAK Options
CPAK 100 GE ER4 CPAK 100 GE LR4 CPAK 100 GE SR10 CPAK
10x10-LR

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
CPAK Optical Interface Support
Standards Compliant
Meets IEEE 802.3ba 100GBASE-SR10 / LR4 / ER4 requirements
OTN OTU4 compliant, Electrical interface OIF compliant

Highest Density
1/3 the size of CFP
20% smaller than CFP2
Enables 10+ Pluggable 100G Ports (1Tb/s) per card

Low Power Consumption

Max power dissipation <7.5W for all reaches
CPAK 100G-LR4 nominally dissipates < 5.5W (<1/3 of CFP)
Enabled by Ciscos CMOS Photonics Technology
CMOS Photonics leverages massive industry investment in CMOS mfg

In Production and Shipping since 2013

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
CPAK Interface Flexibility
DUPLEX SC TO
Configurable 100GE Interconnect LC /SC/ST SM
Options for 10GE interfaces: LGX
Panel
Interface HunGigE 0/x/y/z
CPAK-100G-LR4

Breakout Interface Convention:

Rack/Slot/Bay/Port (phy)/Breakout# MPO24 TO 10X
DUPLEX
10GE Interconnect Options LC/SC/ST MM LGX
Panel
hw-module 0/x/cpu0 port z breakout 10xTenGigE CPAK-100G-SR10
LGX
Interface TenGigE 0/x/y/z/0 Panel
Interface TenGigE 0/x/y/z/1 MPO24 TO 10X
CPAK-10X10G-LR DUPLEX LC
Interface TenGigE 0/x/y/z/9 /SC/ST SM

40GE Interconnect Options

hw-module 0/x/cpu0 port z breakout 2xFortyGigE LGX
Panel
Interface FortyGigE 0/x/y/z/0 CPAK-2X40G-LR4 LC TO
Interface FortyGigE 0/x/y/z/1 DUPLEX
LC/SC/ST
SM
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Tomahawk Flexible MAC PHY
Configurable MAC PHY between optical interface to NPU:

10GE/40GE/100GE LANPHY
100G LR4/ER4 (4x25.78G) 1xCAUI (100GE, 802.3ba)
100G SR10 (10x 10G) 2x CAUI or 2x10x 10GE XFIs
10x 10G LR 10x 10GE XFIs
2x 40GE LR4/SR4 2x 40GE XLAUIs (2x40GE, 802.3ba)
10GE WANPHY
CPU
10x 10GE WANPY (9.954Gbps WIS) 10x 10GE XFIs CPA
K PHY NP FIA
OTN: OTUk wrapper/mapping for 10GE/100GE MAC PCS CPA
K
OTL4.4 (4x 27.95G) 2x CAUI (2x 100GE) CPA
K
CPA PHY NP FIA
OTL4.10 (10x 11.18G) 2x 10x 10GE XFIs K Switch
CPA Fabric
OTU4, OTU-2 (10.7Gbps), OTU-2e (11.096Gbps) K
CPA PHY NP FIA
G.709 framing, EFC, EFEC (G.975 I.4) and UFEC (I.7) K
CPA
More features: K
CPA PHY NP FIA
802.1qbb Priority based Ethernet flow control K

Integrated 1588v2 and SyncE timing support

MACSec 802.1ae at 10GE, 40Ge, 100Ge line-rate
128-bit, 256-bit encryption key, 10,240 octet MTU

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Tomahawk Per Slice MACSEC PHY Capability
MACSEC Security Standards Compliant with:
IEEE 802.1EA-2006
IEEE 802.1AEbn- 2011 (256-bit key)
IEEE 802.1AEbw-2013 (extended packet
numbering)
Security Suites Supported:
AES-GCM-128, 128-bit key (32 bits)
AES-GCM-256, 256-bit key (32 bits)
AES-GCM-XPN-128, provides extended packet
number counter (64 bits)
AES-GCM-XPN-256, provides extended packet
number counter (64 bits)
Unique Security Attributes Per Security Association
(SA):
10G port = 32 SA
40G port = 128 SA
100G port = 256 SA
Per Slice Port Combination Supported (CPAK)
2x100G, 20x10G, 4x40G, 1x100G + 10x10G, 2x40G
+ 10x10G, 2x40G + 1x100G
Most Tomahawk LC variations are MACSEC capable
8x100G, 4x100G, MOD-400, MOD-200
12x100GE does not support MACSec
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
MACSec Use Cases
Usecase #1: Link MACSEC in MPLS/IP Topology Usecase #2: Link MACSEC over LAG members

ASR9k MACSEC Member link

CE PE P PE CE on LAG Inheritance

MACSEC Links
MACSEC Links

Usecase #3 CE Port Mode MACSEC over L2VPN Usecase #4 VLAN Clear Tags MACSEC over L2VPN

ASR9k MKA ASR9k ASR9k MKA ASR9k

DCI/CE DCI/CE DCI/CE DCI/CE
port port vlan vlan
mode mode clear-tags clear-tags
L2VPN L2VPN
CE/WAN CE/WAN

MACSEC Links MACSEC Links

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Tomahawk Vs Typhoon Hardware Scale
Metric Tomahawk - SE Scale Typhoon SE Scale
MPLS Labels 1M 1M
MAC Addresses 2M 2M

FIB Routes (v4/v6) Search

10M(v4) / 5M(v6) 4M(v4) / 2M(v6)
Memory

VRF 8K/LC 8K/LC

Bridge Domains 64K/LC 64K/LC
TCAM TCAM (80Mb) TCAM (40Mb)
12GB/NPU 2GB/NPU
Packet Buffer
200ms 100ms
EFPs 256K/LC (64K/NP) 64K/LC
L3 Subif (incl. BNG) 128K (64K/NP) 20k/LC (sys)

Egress Queues 1M/NPU (4M for 8x100GE) 256k/NPU

Policers 256k/NPU 256k

-Note: Actual available LC/NPU capacity depends on software release and subject to change!
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Tomahawk 8x100G and 12x100G
High Level Differences
8x100GE Octane 12x100GE Skyhammer
8-port 100GE with 4 NPU slices 12-port 100GE with 6 NPU slices
CPAK Optics QSFP28 Optics
5-Fabrics and 7-Fabric versions 7-Fabric card
Compatible with all 99xx-series and 90xx-series chassis Support only for 99xx-series chassis
Full L3VPN/L2VPN feature support Only L3 features in Ph-1, L2 support in Ph-2
Has external TCAM for High Qos/ACL scale Has no external TCAM . Will use only 5Mb internal TCAM
Total TCAM entries: 192K Total TCAM entries: 32K
32K sub-interfaces 1K sub-interfaces
10M V4 route, 5M V6 routes 10M V4 route, 5M V6 routes

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Tomahawk Scale Differences
Tomahawk Tomahawk Skyhammer
Metric
-TR Scale -SE Scale Scale
MPLS Labels 1M
MAC Addresses 2M
FIB Routes (v4/v6) Search
10M(v4) / 5M(v6)
Memory
Mroute/MFIB (v4/v6) 128k/32k
VRF 8K
Bridge Domains 64K
TCAM (acl space v4/v6) TCAM 1/4 -SE TCAM (80Mbit) Internal TCAM (5Mbit)
Packet Buffer 100ms Packet Buffer 200ms
Packet Buffer Packet Buffer 100ms
(6G/NPU) (12G/NPU)
EFPs 16K/LC 128K/LC (64K/NP) N/A
L3 Subif (incl. BNG) 8K 128K (64K/NP) 1K
IP/PPP/LAC subscriber
16K 256K (64K/NP) N/A
sessions per LC
Egress Queues 8 Queues / port + nV Sat Qs 1M/NPU (4M for 8x100GE!) 8 Queues / port
Policers 32K/NPU 512K/NPU
QOS/ACL (v4/v6) 16k v4 or 4k v6 ACEs/LC 98k v4 / 16k v6 ACEs 24K v4/1.5K v6

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
ASR9k Optical Solution Before IPoDWDM LC
ASR9k + NCS2k Optical Shelf
Integrated nV Optical System

ASR9000 NCS2k

Low Cost
Interconnect

SR10
CFP/CPAK SR10
transceiver CXP/CPAK
100GE Linecard Coherent Transponder

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
400G IPoDWDM LC Overview (Tomahawk Based LC)

Tomahawk based Linecard

Feature and scale parity with other TR and SE Tomahawk cards
2xCFP2 based DWDM ports (100G, 200G)
BPSK, QPSK, 16QAM modulation options
96 channels, ITU-T 50GHz spacing
FlexSpectrum support
HD FEC, SD FEC (3000+ km w/o regen)
20x10GE SFPP ports (SR, LR, ZR, CWDM, DWDM)
Flexible port options up to 400 Gbps total capacity
2 x 200G DWDM (CFP2) or
2 x 100G DWDM (CFP2) + 20 x 10G (SFP+) or
1 x 100G + 1x200G DWDM (CFP2) + 10 x 10G (SFP+)
OTN and pre-FEC FRR

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
 400G IPoDWDM LC Specific HW components
Ivy Bridge
NPMEM
CPU Complex
VOQ

Tomahaw
Coherent HD-FEC
CFP2 Etna X240 k
FPGA FIA
200G capable

TCAM

#1 10GE SFP+

X240 SerDes
#20 10GE SFP+ Mux
SM15

NPMEM VOQ

Tomahaw
Coherent HD-FEC
CFP2 Etna X240 k
FPGA FIA
200G capable

TCAM
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
 400Gbs IPoDWDM LC Internals Ivy Bridge
NPMEM
CPU Complex
VOQ

Tomahaw
#0
Coherent HD-FEC
CFP2 Etna X240 k
FPGA FIA
200G capable

TCAM

#1 10GE SFP+

X240 SerDes
#20 10GE SFP+ Mux
SM15

NPMEM VOQ

Tomahaw
Coherent HD-FEC
#1
CFP2 Etna X240 k
FPGA FIA
200G capable

TCAM
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Switch Fabric Architecture
 ASR 9000 Switch Fabric Overview

Separated fabric card

Fabric is integrated on RSP Integrated + Separated 6+1 redundancy
1+1 redundancy Fabric
6+1 redundancy
Integrated fabric/RP/LC
9904
RSP880: 805G+805G/slot
9001, 2RU, 120G

9910
9001-S, 2RU, 60G
1.38Tb+230G /slot
9912 9922

9006 9010 1.38Tb+230G /slot

RSP880: 450G+450G /slot

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
 ASR 9000 Switch Fabric High-Level Architecture
3-Stage Non-Blocking Fabric (Separate Unicast and Multicast Crossbars)
Fabric frame format:
Super-frame Stage 1 Stage 2 Stage 3
Fabric load balancing: Active-Active
Unicast is per-packet Fabric
Multicast is per-flow Unicast
Virtual Output Queue Crossbar
Arbitration fabric

fabric fabric
Arbiter FIA
FIA Multicast FIA
FIA RSP0
Crossbar FIA
FIA
Tomahawk
Typhoon LC Arbiter
Arbiter fabric
Egress Linecard
Ingress Linecard
Arbiter

RSP1
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Fabric Super-framing Mechanism
Multiple unicast frames from/to same destinations aggregated into one super
frame
Super frame is created if there are frames waiting in queue, up to 32 frames
or when min threshold met, can be aggregated into one super frame
Super frame only apply to unicast, not multicast
Super-framing significantly improves total fabric throughput

Packet 1 No super-framing
Packet 2 Packet 1 Min reached
Packet 3 Packet 2 Packet 1 Max reached

Packet 1 Jumbo
Max Super-frame Min Super-frame 0 (Empty)
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Fabric Load Balancing Unicast
Fabric
ASIC

Fabric
ASIC

Fabric ASIC Arbitration Fabric ASIC

and VOQ SFC or RSP0 and VOQ 4 3 2 1

Fabric
ASIC
Fabric
ASIC
Arbitration
SFC or RSP1

Unicast traffic sent across first available fabric link to destination (maximizes efficiency)
Each frame (or super frame) contains sequencing information
All destination fabric ASIC have re-sequencing logic
Additional re-sequencing latency is measured in nanoseconds

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Fabric Load Balancing Multicast
Fabric
ASIC

Fabric
ASIC

Fabric ASIC Arbitration Fabric ASIC C B A B A A

and VOQ SFC or RSP0 and VOQ 1 2 3 1 2 1

Fabric
Flows exit in-order
ASIC
Fabric
ASIC
Arbitration
SFC or RSP1

Multicast traffic hashed based on (S,G) info to maintain flow integrity

Very large set of multicast destinations preclude re-sequencing
Multicast traffic is non arbitrated sent across a different fabric plane

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Switch Fabric Arbitration
5: credit return
Fabric
1: Fabric Request ASIC

Fabric
ASIC

Arbitration
Fabric ASIC Fabric ASIC
and VOQ SFC or RSP0 2: Arbitration and VOQ

Fabric
ASIC
3: Fabric Grant
Fabric
ASIC

4: load-balanced Arbitration
transmission SFC or RSP1
across fabric links

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Virtual Queue Identifier (VQI)
Each Physical Interface is Identified by VQI. VQI is used to load balance traffic across fabric
One VQI Comprise 4 VOQ Priority 1, 2, 3 and Normal
On 8x100G&4x100G Tomahawk LCs:
12 VQI each physical interface(9922, 9912, 9910, 9904)
On 12x100G Tomahawk LCs:
MIX mode: 12 VQI each physical interface(9922, 9912, 9910, 9904)
High Bandwidth Mode: 6 VQI each physical interface(9922, 9912, 9910)
VOQ
On Typhoon LCs, NP is designed for multiple 10G, 40G, and 100G ports.
VOQ
Each 10G port is 1:1 mapped to one VQI VQI
Each 40G port is mapped to 8 VQI VOQ

Each 100G port is mapped to 16 VQI VOQ

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
ASR9k Virtual Output Queuing (VoQ) System Architecture

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
ASR 9000 Traffic Priority Across Fabric
End-to-End priority (P1,P2, P3 and Best-effort) propagation
Unicast VOQ and back pressure
Unicast and Multicast separation

Ingress side of LC Egress side of LC

CPU CPU 4
NP PHY
P1
2 3 P1
P2 P2 NP PHY
P3
FIA
BE FIA P3
BE
PHY NP P1 Switch
P2
P3 Fabric
PHY NP1 BE

1 2
Ingress Port 4 VOQ per each SFP virtual
QoS port in the entire system
Up to 8K VOQs per TSK FIA
(vs 4k per SKT FIA)
3 4
4 Priority Egress Destination Egress Port
Queues (DQs) per port, QoS
aggregated at egress port
rate

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
 Backpressure and VoQ Mechanism (simplified)
Ingress LC Fabric Interface Asic has Virtual Output Queues (VOQs)
One set of VOQs per egress NPU for all LCs in the system
VOQ set: 1 P1/ 1 P2/ 1 P3/ and 1 BE queues
Packets to different egress NPU are in different VOQ
no Head of Line Blocking
1: 100 Gbps Hi Backpressure triggered when priority queues are congested
2: 100 Gbps Hi
CPU CPU
1 CPA
K
P1
P2 P1 CPA
K
CPA PHY NP FIA
P3
Fabric FIA P2
P3
NP PHY
CPA
BE
K BE K
2 CPA
K
P1
P2
ASIC CPA
K
CPA PHY NP FIA
P3 FIA NP PHY
CPA
K BE Switch Fabric Switch K
CPA Fabric
ASIC Fabric P1 CPA
3 K
CPA PHY NP
P1
FIA
P2
FIA P2
NP PHY K
P3 P3 CPA
K BE
Arbitration BE K
CPA CPA
K K
CPA PHY NP FIA FIA NP PHY CPA
K K

Packet going to different egress NP put into different VoQ set Congestion on one NP wont block
the packet going to different NP
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
ASR90xx RSP880 and Mixed LC
Stage 1 Stage 2 Stage 3

Egress Linecard
Ingress Linecard Fabric 8x115Gbps
SM15

Fabric
Arbiter FIA
FIA Fabric FIA
FIA RSP880 SM15 FIA
FIA

Tomahawk Line
Typhoon Linecard Card
Fabric
SM15

Fabric bandwidth: 8x55Gbps Fabric bandwidth:

Arbiter
8x55Gbps =440Gbps/slot with dual RSP 8x115Gbps ~ 900Gbps/slot with dual RSP
RSP880
4x55Gbps =220Gbps/slot with single RSP 4x115Gbps ~ 450Gbps/slot with single RSP

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
ASR90xx RSP440 and Mixed LC
Stage 1 Stage 2 Stage 3

Egress Linecard
Ingress Linecard 8x55Gbps
fabric

fabric Arbiter FIA

FIA Fabric FIA
FIA RSP0 SM15 FIA
FIA

Tomahawk Line
Typhoon Linecard Card
fabric

Fabric bandwidth: Arbiter

8x55Gbps Fabric bandwidth:
8x55Gbps =440Gbps/slot with dual RSP
4x55Gbps =220Gbps/slot with single RSP RSP1 8x55Gbps =440Gbps/slot with dual RSP
RSP440 4x55Gbps =220Gbps/slot with single RSP

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
 ASR9904 with RSP880
Stage 1 Stage 2 Stage 3

2x 7x 115Gbps ~
1.61Tbps
Fabric
ASIC
(SM15)
Ingress Egress
Linecard Fabric ASIC Arbiter Fabric ASIC Linecard
(SM15) (SM15)
and VOQ RSP880 and VOQ

Tomahawk LC Tomahawk LC
5-Fabric 7-Fabric

2x 5 x 115Gbps ~ Fabric
1.1Tbps ASIC
(SM15)
5-Fabric LC Bandwidth: 7-Fabric LC Bandwidth:
10x 115Gbps ~1.1Tbps/slot with dual RSP Arbiter 14x 115Gbps ~1.61Tbps/slot with dual RSP
5 x 115Gbps ~575Gbps/slot with single RSP 7 x 115Gbps ~ 805Gbps/slot with single RSP
RSP880

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
ASR99xx Switch Fabric Card (FC2)
6+1 All Active 3-Stage Fabric Planes, Scale to 1.6Tbps/LC, Separated Ucast/Mcast Crossbars

5x 2x115G bi-directional 7x2x115G bi-directional

= 1.15Tbps = 1.6Tbps

5-Fabric LC 7-Fabric LC

FIA
FIA Fabric FIA
FIA Fabric
SM15 FIA
SM15
FIA

Tomahawk Line Tomahawk Line

Card Card

Fabric frame format:

Super-frame Fabric bandwidth:
Fabric load balancing: 10x115Gbps ~ 1.15 Tbps/slot with 5x FC2
Unicast is per-packet SFC v2
Multicast is per-flow 14x115Gbps ~ 1.6 Tbps/slot with 7x FC2

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
ASR9922/12/10 SFC2 and Mixed Gen LCs

(5-1)x2x115G bi-directional
= 920Gbps (protected)

FIA
FIA FIA
FIA Fabric
FIA
SM15
fabric FIA

Typhoon Linecard Tomahawk Line

Card

(5-1)x2x55G bi-directional Fabric Lanes 6 & 7 are only used towards

= 440Gbps (protected) 3rd Gen 7-fabric plane linecards
SFC v2

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
 ASR99xx 5-Fab & 7-Fab Skyhammer

100G 100G
83G
200G
5-Fabric 200G 7-Fabric
LC LC
SM15
SM15

1T with 5 Fabrics
800G w/ 4 Fabrics
800G linerate /LC (protected) capacity all
Typhoon, Tomahawk LCs (except
Skyhammer) interoperate at full throughput
Fabric 6 and 7 not used
SFC2
All 7-Fabric LCs are rate-limited to 1T
aggregate Fabric Bandwidth Access

7-fabric LCs uses 5 fabrics to interoperate with other 5-fab LCs

7-Fabric LCs rate-limited to 1T aggregate bandwidth to avoid fabric oversubscription

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
Packet Flow and Control Plane
Architecture
 Transit Traffic Two-Stage Forwarding Overview
Ingress lookup yields packet egress port and applies ingress features

Egress lookup performs packet-rewrite and applies egress features

CPU CPU
CPAK 0 CPAK 0
1
PHY NP FIA FIA NP PHY
CPAK 1
CPAK 1

Switch CPAK 2
CPAK 2
Fabric
PHY NP FIA FIA 2
NP PHY
CPAK 3 CPAK 3
Switch Switch

Fabric Fabric
CPAK 4 CPAK 4
FIA
(SM15) (SM15)
PHY NP FIA NP PHY
CPAK 5
CPAK 5
Up to Up to
14x120G 14x120
CPAK 6 1 Switch G CPAK 6
PHY NP 2 FIA Fabric FIA NP PHY
CPAK 7 CPAK 7

Uniform packet flow for simplicity and predictable performance

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
Feature Order on ASR 9000 NP
TCAM
Ingress linecard
From wire I/F ACL QOS
Fwd lookup
classification classification classification

IFIB action QoS action L2 rewrite ACL action IFIB lookup

To fabric From fabric

egress linecard
QOS ACL
ACL action L2 rewrite Fwd lookup
classification classification

To wire
QoS action
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
Feature Order on ASR 9000 NP QoS Action

Ingress linecard
From wire I/F ACL QOS
Fwd lookup
classification classification classification

IFIB action QoS action L2 rewrite ACL action IFIB lookup

To fabric From fabric

QoS Action

WRED classifies on marked/remarked values

egress linecard
QOS ACL
ACL action L2 rewrite Fwd lookup
classification classification Queue/Shape/
Police Mark WRED
To wire
QoS action
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
L3 Unicast Forwarding
Packet Flow (Simplified)
from wire Rx LAG hashing
lookup key LAGID SFP Packet rewrite
LAG
L3: (VRF-ID, IP DA) System headers added
TCAM Ing-IDB L3FIB rx-adj rewrite
SFP
Packet Source L3 FIB lookup Next-hop
classification interface info Switch Fabric Port SFP
ACL and QoS Lookup (egress NPU)
Ingress NPU also happen in parallel
Tx LAG hashing Fabric
LAG

rewrite Egr-IDB tx-adj L3FIB

destination Next-hop L3 FIB
interface info lookup
ACL and QoS Lookup
Egress NPU happens before rewrite

to wire ECH type: tell egress NPU type of lookup it should execute

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
Tomahawk MultiStage Multicast Replication Overview
7. MGID lookup for NPU 4. 22-bit FGID for slot bitmask
to egress OIF interface based unique fabric LC 6. MGID in FIA DI table lookup
replication 5. MGID lookup into FPOE for unique NPU replication
replications and FPOE lookup
table (256k entries) for Flow based load balance
unique FIA replications

7 SFP+
6 SFP+
FIA NP SFP+
SFP+
SFP+
SFP+
SFP+
CPAK 0 5 FIA NP SFP+
SFP+
SFP+
7 6FIA
PHY NP Fabric 6 7 SFP+
SFP+
CPAK 1 SM15 FIA NP SFP+
SFP+
SFP+
SFP+
5
4
FIA NP SFP+
SFP+
CPAK 2 7 SFP+
SFP+
PHY NP 6FIA
RSP880 Tomahawk Line Card
CPAK 3
Switch SFP+
Fabric SFP+
CPAK 4 (SM15) FIA NP SFP+
SFP+
SFP+
PHY NP FIA
SFP+
CPAK 5 3 FIA NP
SFP+
SFP+
SFP+
SFP+
Up to Fabric SFP+
2 SFP+
CPAK 6
14x120 SM15 FIA NP SFP+
SFP+
PHY NP 1 FIA G SFP+
CPAK 7 SFP+
RSP880 FIA NP SFP+
SFP+
SFP+
SFP+
Tomahawk Line Card
1. 512k MGID/FGID lookup
2 & 3. Flow based multi link
Flow based multi link load
load balance
balance
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
 ASR9000 QoS Architecture
Dedicated Traffic Manager(TM) for Traffic Queuing
User Configurable QoS Policy on Ingress/Egress NP
End-to-End priority propagation Guarantee bandwidth, low latency for high priority traffic
Unicast VOQ and back pressure
Ingress side of LC Egress side of LC

CPU CPU 4
P1 NP0 PHY
P2
P3 3 P1
P1
P2
BE
2
P2
P3 NP1 PHY
CPAK
0 PH
TM P3
BE
FIA BE

CPAK Y
NP FIA Switch NP2 PHY
1
1 Fabric P1
NP3
TM
P2
P3
PHY
BE

1 2 3
4
Ingress (sub-)interface 4xVOQ per VQI 4x Egress Destination Qs per Port,
QoS Queues Egress (sub-)interface
Up to 8K VOQs per TSK FIA aggregated at egress port rate
User Configurable (vs 4k per SKT FIA) QoS Queues
Ingress QoS Policy Configure with Egress MQC

Configure with Ingress MQC

Implicit Configuration
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
5 Level Hierarchy QoS Queuing Overview

L0 L1 L2 L3 L4 L0, L1 level schedulers are automated

PG Port Logical Sub- Classes by TM, not user configurable
Port interfaces

L2, L3 and L4 can be flexibly mapped

to parent level scheduler
Port Group

Class
EFP1 C-VLAN Class Hierarchy levels used are determined
(S-Vlan
or Class
by how many nested levels a policy-
Vlans) C-VLAN Class map is configured for and applied to a
Port

given sub-interface
Class
EFP2 C-VLAN Class
(S-Vlan
or
Up to 16 classes per child/grandchild
Class
Vlans) C-VLAN Class level (L4)

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
ASR9000 Ingress Control Plane Packet

LPTS (local packet transport service):

control plane policing
RP
Punt
CPU FPGA
FIA

CPU
LC1
CPAK 0 iFIB Switch Fabric Switch Fabric
Control CPAK 1
PHY NP
LPTS FIA
packet CPAK 2
PHY NP FIA
CPAK 3
Switch

Fabric
CPAK 4
(SM15)
PHY NP FIA
CPAK 5
Up to
14x120G
CPAK 6
PHY NP FIA
CPAK 7

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
Layer 3 Control Plane Overview

BGP OSPF
LDP RSVP-TE
Static
ISIS EIGRP

LSD RIB RP

Over internal EOBC

ARP Hierarchical FIB table

FIB Adjacency
SW FIB structure for prefix
AIB LC NPU independent
convergence: TE/FRR,
IP/FRR, BGP, Link
LC CPU AIB: Adjacency Information Base
RIB: Routing Information Base
bundle
FIB: Forwarding Information Base
LSD: Label Switch Database

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
 MAC Learning and Sync
Hardware based MAC learning: ~4Mpps/NP
1 NP learn MAC address in hardware (around
4M pps)
RP
2 NP flood MAC notification (data plane) Punt
CPU
message to all other NPs in the system to sync FPGA FIA
up the MAC address system-wide. MAC
notification and MAC sync are all done in
hardware
Switch Fabric Switch Fabric

CPU LC1
LC2
CPU
Data 3x10GE
SFP + 1NP 2
packet 3x10GE
SFP + NP
FIA 3x10GE
SFP +
NP
FIA
3x10GE
3x10GE
SFP +
NP 2

Fabric ASIC
Switch
SFP + NP
FIA 3x10GE
3x10GE NP

Fabric ASIC
Switch
NP SFP +
SFP + FIA
3x10GE
3x10GE
SFP +
NP
SFP + NP
FIA 3x10GE
3x10GE
SFP +
NP
SFP + NP FIA
3x10GE
3x10GE
SFP +
NP
SFP + NP
3x10GE FIA 3x10GE
NP
NP SFP +
SFP + FIA
3x10GE
SFP +
NP

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
L3 Multicast Control Plane

1. Incoming joins (IGMP, PIM, MLD, MLDP, etc )

2. NPU punts joins to RP directly bypassing LC CPU
3. Each protocol updates multicast info (VRF, route, olist, etc ) to MRIB
4. MRIB downloads all multicast info (VRF, route, olist, etc ) to MFIB in each LC
5. MFIB programs HW with multicast info: NPU, FIA and LC Fabric

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
L2 Multicast Control Plane

1. Incoming joins (IGMP, PIM, MLD, etc )

2. NPU punts joins to RP directly bypassing LC CPU
3. Each snooping protocol updates L2FIB (mrouter, port-list, input port, BD, etc )
4. RP L2FIB downloads L2 multicast info to each LC L2FIB
5. LC L2FIB programs HW with multicast info: NPU, FIA and LC Fabric

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
IOS-XR Architecture
Cisco IOS A Recap
Cisco IOS Cisco IOS-XE Classic IOS-XR Virtual IOS-XR
Control Data Mgmt
Plane Plane Plane
XR Code v1 XR Code v2

Hosted App 1

Hosted App 2
IOSd

NetFlow
SNMP
OSPF

LPTS

XML
BGP

NetFlow
QoS

NetFlow
ACL
PIM

SNMP
SNMP

OSPF
OSPF

LPTS
LPTS

BGP
BGP

XML
QoS
XML
QoS

ACL
ACL

PIM
PIM
System
IOS Blob Admin

Operational Infra Distributed Infra Distributed Infra Distributed Infra

Kernel Kernel Kernel Kernel Kernel

Linux-BinOS QNX, 32bit Linux, 64bit Linux, 64bit Linux, 64bit

Virtualization Layer

1990s 2000s 2003-04 Present Day

Incremental Development, with Industry leading investment protection

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
Carrier Class OS IOS-XR
A Fully Distributed, Microkernel-Based, Modular Architecture
Modular Components Applications Architecture
MPLS Multicast IS-IS Multicast
RIP BGP
Routing Distributed Middleware
OSPF ISIS

Manageability QoS BGP

Security
Host Forwarding Distributed Service Separation
Base
OAM MAC OAM MAC OAM MAC
Admin
VLAN Subs VLAN Subs VLAN Subs
Line Card
LC1 LC2 LCn

Designed for Scale, HA, and Performance

For IP NGN Applications
Microkernel-based design Scale through distribution
Highly modular, highly extensible Unique address tables per linecard
Service-enabled blade architecture Process-level, stateful HA

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
System HA Architecture Overview
Checkpoint to shared memory
for process restart

Active RP Centralized control plane: Routing,

PIM, IGMP, HSRP/VRRP, etc Checkpoint
between two RPs
Standby RP Centralized control plane: Routing, for RP failover
PIM, IGMP, etc
Checkpoint to shared memory
for process restart
Distributed Control Plane: BFD,
RP failover has zero
ARP, CFM, STP
impact on the
LC1

LC2

LC3

forwarding plane
Fully distributed forwarding plane and no impact on
the distributed
control planes
Data Forwarding Continue

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
Bug Fixes - SMUs
Scenario Traffic Impact Comments
PI processes No Warm restart
Data plane device driver
No Warm restart
code change (fast path)
Control plane process code Impacts in-flight control
Possible
change (slow path) packets
LC reload (*FPD upgrade
Data plane device driver
Yes does not require reload in
data structure change
Tomahawk)

Most SMUs do not require Line Card reload

Effects may be further minimized with network-layer redundancy
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
IOS XR 64 bit: Architecture Overview
IOS-XR

Admin Plane
IOS-XR Separate Admin Plane
Routing
Apps
RP

RP
Routing
System Control Plane 64-bit IOS XR.
Admin

QNX Linux
IOS XR IOS-XR

Admin Plane
Line Card

Line Card
Linux Container/VM

LC-CPU LC-CPUs

QNX 64 bit Linux Kernel

Linux

Classic IOS XR Virtualized IOS XR

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
IOS XR 64 bit: Architecture Evolution Benefits
OS: 32-bit QNX to 64-bit Yocto Wind River Linux
Architecturally ready for larger scale (IPv4: 10M; IPv6: 5M)
Open to Standard Linux Applications (Application Hosting)
Modularized Software Packaging
Easy Adaption for Merchant x86 Hardware (Virtual Router)

Virtualization Standard Linux Based

NG ISSU
Improved Time for Bootup, Process Restart and RSP Failover

Flexible Linux Style RPM Packaging

Feature-Level RPM Packaging

Admin Plane and Control Plane Separation

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
System Architectural Support for ISSU
Different Version IOS XR Simultaneously Running

LC-CPU

Admin
IOS XR IOS XR

Admin
Admin

IOS IOS V1 V2
XR V1 XR V2

Linux

Linux Linux Slice 1 Slice 2 Slice 3 Slice 4

V1
V2
Active
Standby Standby
Active

RP (ZTL/MTL) LC (ZPL/MPL)
Tomahawk only
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 102
IOS XR 64 bit: More Than Just 64 bit

Telemetry Network Visibility

Application Hosting
1. Standard Linux Application
2. Customized Cisco Application

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
 App Hosting

Application Hosting
Apps Built for Device Extension with off the shelf applications
Cisco OS
Multi-level API access
Linux Based Native Linux APIs

Apps

Apps
Applications
ASIC SDK level APIs
Apps

OS level APIs
Apps

Cisco OS Complete tool chain integration

Easy to use and integrate with customer build

FWD ASIC SDK

environments
Industry standard delivery mechanisms (RPMs)
Linux
Hardware

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
 Telemetry

Subscribe to Data Increased Visibility

Pool of Network Data

Data Consumer

sensing &
measurement
Controller, NMS
and Analysis

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
 Telemetry

Increased Network Visibility

Information Scope Information Delivery

Telemetry
Instant Monitoring of Network
State
Control Periodic: 30 10 1 sec
Plane SysDB MIB
Data
Event driven

Data Minimal impact on system

Low Level
Plane
Data
QoS
Data
performance as receiver
grows
YANG based Model Driven Telemetry

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 106

SDN Ready Infrastructure
Data as a Service
Telemetry data consumers
subscribe to receive data
Telemetry consumers
dynamically control:
What data to get
The format of the data
Frequency of data delivery
Telemetry
Network Application Centric
Complementing SNMP, which
is network device centric
Operational Simplicity
No additional housekeeping on
systems to get additional MIB data
BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 107
Summary
ASR9000 - Truly Carrier-Class Edge Router Provides:
Rich Features, Flexible Service Capability
Variety of Hardware to Meet Different Capacity Requirements

Fully Distributed Architecture for High Performance

and System Scales
Uniform and Modularized Software Architecture for
Simple Operation and Efficient Failure Protection
Complete Your Online Session Evaluation
Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 Amazon gift card.
Complete your session surveys
through the Cisco Live mobile
app or from the Session Catalog
on CiscoLive.com/us.

Dont forget: Cisco Live sessions will be available

for viewing on-demand after the event at
CiscoLive.com/Online

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 109
Continue Your Education
Demos in the Cisco campus
Walk-in Self-Paced Labs
Lunch & Learn
Meet the Engineer 1:1 meetings
Related sessions

BRKARC-2003 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 110
Please join us for the Service Provider Innovation Talk featuring:
Yvette Kanouff | Senior Vice President and General Manager, SP Business
Joe Cozzolino | Senior Vice President, Cisco Services

Thursday, July 14th, 2016

11:30 am - 12:30 pm, In the Oceanside A room

What to expect from this innovation talk

Insights on market trends and forecasts
Preview of key technologies and capabilities
Innovative demonstrations of the latest and greatest products
Better understanding of how Cisco can help you succeed

Register to attend the session live now or

watch the broadcast on cisco.com
SP Sessions & Team Meeting
Session ID Session Title
BRKSPG-2062 SDN-enabled Carrier Ethernet Networks by Dennis Cai

BRKARC-2003 Cisco ASR 9000 Architecture by Yong

BRKSPG-3012 SP Security - Leveraging BGP FlowSpec to protect your infrastructure - Nicolas Fevrier

BRKSPG-2210 Designing Programmable Access Networks by Ahmed Abeer & Nicolas Breton

TECSPG-2300 Network Function Virtualization Seminar by Prasanna & team

Meet SP Team
At the World of Solutions (WoS) - Products & Solutions for Service Providers
Meet SP Team (Whisper Suite & Meet the Expert)

BRKSPG-2518 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 112
Thank you