Advance IOS XE

Advanced Troubleshooting of
IOS-XE Key Features

BRKARC-2021
Frederic Detienne, Distinguished Engineer

Olivier Pelerin, Technical Leader
Agenda
• Platform and Hardware Architecture
• Software Architecture
• Day in the Life of a Normal Packet
• Day in the Life of an IPsec Packet
• Debugging strategies
• Data Plane Debugging with the Packet Tracer
• Understanding and Extracting ESP Logs
• Resource Consumption Monitoring
• Wrapping up...
BRKCRS-2021 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Objectives
• Understand the ASR 1K and ISR 4x5y architecture
– software
– hardware
– relationship between these two
• Understand how features process packets through IOS-XE
• Understand how to easily debug the platform
– long journey
– presentation of recent serviceability enhancements
– spare memorizing – focus on understanding
– Not “tips & tricks” but debugging strategy and tools
BRKCRS-2021 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
ASR Series Hardware Architecture
Cisco ASR 1000 Series Routers: Overview
Compact, Powerful Router Business-Critical Resiliency Instant-on Services Delivery
Instant On
Service Delivery
 Line-rate performance 2.5G to  Fully separated control and  Integrated firewall, VPN,
200G+ with services enabled forwarding planes encryption, DPI, CUBE
 Investment protection with modular  Hardware and software redundancy  Scalable on-chip service
engines, IOS CLI and SPAs for I/O provisioning through software
 In-service software upgrades
licensing
 Hardware based QoS engine with
up to 472K queues
One IOS-XE Feature Set

ASR 1013
ASR 1006
ASR 1004
ASR 1001 / 1001-X ASR 1002-X
2.5-5 5-36 10-40 10-100 40-200

Gbps Gbps
Gbps Gbps Gbps
ASR1K Building Blocks
ESP RP RP ESP
Active
FECP CPU CPU FECP
Stby
Active
Stby
QFP interconn. GE switch interconn. GE switch QFP
Crypto Crypto
QFP
Assist. PPE BQS Assist. PPE BQS
Route Processor
interconn. Handles control plane traffic interconn.
Manages system
Embedded Service Processor

Handles forwarding plane traffic Midplane
SIP interconn. SIP interconn. SIP interconn.
SPA SPA SPA

IOCP IOCP IOCP
Aggreg. Aggreg. Aggreg.
SPA SPA SPA SPA SPA SPA
SPA Interface Processor

Houses SPA’s
Queues packets in & out (FIFO)
System Architecture Control Plane
ESP RP RP ESP
Active
FECP CPU CPU FECP
Stby
Active
Stby
Crypto Crypto
Assist. PPE BQS EOBC switch in Assist. PPE BQS
RP
interconn. interconn.
Midplane
Ethernet Out of Band Channel
(aka EOBC)
Inter Integrated Circuit (I2C) Bus 1Gbps Ethernet bus
SIP interconn. SIP interconn. SIP interconn. Used by RP to program system
Slow (few kbps)
Used for system monitoring Used by system to notify RP
(temp., OIR, fan speed,…)
SPA SPA SPA
IOCP IOCP IOCP
SPA Control Link

Works between the SPA’s and SIP
System Architecture Forwarding Plane
Hypertransport
10 Gbps Ethernet
ESP RP RP ESP
Active
FECP CPU CPU FECP
Stby
Active
Stby
Crypto Crypto
Assist. PPE BQS Assist. PPE BQS
Embedded Service Interconnect
aka ESI Bus
interconn. 11.2 – 40 Gbps Forwarding Bus interconn.
Centralized Architecture Midplane

All traffic flows through ESP
SIP interconn. SIP interconn. SIP interconn.
SPA SPA SPA

IOCP IOCP IOCP
RP
CPU
Route Processor Architecture interconn. GE switch

Highly Scalable Control Plane Processor
Route Processor
System Logging
Manages all chassis functions Not a traffic interface!
Core Dumps
Runs IOS Management only
Mgmt Console BITS

USB 2.5’’
Ethernet & Aux (input & output)
Hard disk
Card Infrastructure
Runs IOS, Linux OS
Manages boards and chassis
33MB
IOS Memory: RIB, FIB & NVRAM
other processes RP1: 1GB
Determines BGP routing CPU CPU Bootdisk RP2: 2GB
table size
RP1: 4GB Memory (1.5 – 2.66 GHz Dual-core)
RP2: 8&16GB Stratum-3 Network
clock circuit GE, 1Gbps
I2C
Chassis I2C
Management Bus ESI EOBC SPA Control
Interconnect Gig Eth Switch Output Input SPA Bus
clocks clocks
ESI, 11.2-40 Gbps
SPA-SPI, 11.2Gbps
Hypertransport, 10Gbps
Other
SIPs ESPs RP Misc ESPs SIPs ESPs RP SIPs SIPs RP
Ctrl
ESP
FECP
ESP10 Block Diagram Crypto

Assist.
QFP
PPE BQS
intercon.
Reset / Pwr Ctrl Packet Buffer

TCAM Resource DRAM Part Len / BW
DRAM
(10Mbit) (512MB) SRAM
Temp Sensor (128MB)
EEPROM
QFP
DDRAM Packet Processor Engine BQS
Boot Flash PPE1 PPE2 PPE3 PPE4 PPE5

(OBFL,…) FECP E-CSR
JTAG Ctrl PPE6 PPE7 PPE8 … PPE40
PCI* E-RP*
Dispatcher
GE, 1Gbps
Packet Buffer
I2C
SPA Control
Crypto SPA Bus
(Nitrox-II CN2430) SPI Mux
Reset / Pwr Ctrl ESI, 11.2Gbps
SA table SPA-SPI, 11.2Gbps
DRAM Interconnect Interconnect Hypertransport, 10Gbps
Other
RPs RPs ESP RPs SIPs

ESP
FECP
ESP10 Block Diagram (comments) Crypto

Assist.
QFP
PPE BQS
intercon.
Forwarding Engine Control Quantum Flow Processor
ProcessorReset / Pwr Ctrl OverallDRAM Packet Buffer
packet forwarding
TCAM Resource Part Len / BW
Manages board (10Mbit) (512MB)
DRAM
SRAM
Programs QBS,
TempPPE,
SensorCrypto (128MB)
Linux Kernel
EEPROM
QFP
Buffering Queuing & Scheduling
DDRAM Packet Processor Engine BQS
Executes complex QoS scheduling
(shapers, LLQ’s,…)
Boot Flash PPE1 PPE2 PPE3 PPE4 PPE5 Queues and schedules packets in
(OBFL,…) FECP E-CSR
due time
JTAG Ctrl PPE6 PPE7 PPE8 … PPE40
Packet Processor Engine

PCI* E-RP* Multicore CPU
Dispatcher Routes and applies features to
GE, 1Gbps
Packet Buffer packets
I2C
SPA Control
Crypto SPA Bus
(Nitrox-II CN2430) SPI Mux
DRAM Interconnect Interconnect Hypertransport, 10Gbps
Other

Embedded Services Processor – The Real Thing
Interconnect ASIC
SPI MUX
TCAM Crypto
Engine
FECP
QFP Subsystem CPU
PPE + BQS
FECP
DRAM
PPE BQS
DRAM Packet
DRAM
Cisco “Quantum Flow Processor”
• Packet Processing Engine (QFP-PPE)

– 40-64 Packet Processors with 4 threads per core
– Up to 1.2GHz Tensilica ISA processors + DRAM packet memory
– Single TCAM4 I/F; can cascade 1-4 devices
Multi-Core (40) Packet Processor
– C-language for feature development; extensive development
support tools
– HW assist for flow-locks, look-ups, stats, WRED, policers, range
lookup, crypto, CRC
• Buffer/queue subsystem (QFP-BQS)

– HW hierarchical 3-parameter (min, max & excess) scheduler
– Fully configurable # of layers based on HQF
– Priority propagation through the multiple layers
Traffic Manager (BQS)
ESP200 Block Diagram
Packet Buffer Packet Buffer
TCAM Resource DRAM Resource DRAM
DRAM DRAM
(80Mbit) (2GB) (2GB)
(512MB) (512MB)
Reset / Pwr Ctrl QFP QFP

Packet Processor Engine BQS Packet Processor Engine BQS
Temp Sensor PPE PPE PPE PPE PPE PPE PPE PPE PPE PPE
1 2 3 4 5 1 2 3 4 5
EEPROM PPE PPE PPE … PPE PPE PPE PPE … PPE

6 7 8 40 6 7 8 40
Dispatcher Dispatcher
DDRAM Packet Buffer Packet Buffer
Boot Flash TCAM Resource DRAM

Packet Buffer
Resource DRAM
Packet Buffer GE, 1Gbps
(OBFL,…) FECP (80Mbit) (2GB)
DRAM
(512MB)
(2GB)
DRAM
(512MB)
I2C
SPA Control
SPA Bus
JTAG Ctrl QFP QFP ESI, 11.5 or 23Gbps
Packet Processor Engine BQS Packet Processor Engine BQS
SPA-SPI, 11.2Gbps
PPE PPE PPE PPE PPE PPE PPE PPE PPE PPE Hypertransport, 10Gbps
1 2 3 4 5 1 2 3 4 5
Other
PPE PPE PPE … PPE PPE PPE PPE … PPE
6 7 8 40 6 7 8 40
Dispatcher Dispatcher
Packet Buffer Packet Buffer
Reset / Pwr Ctrl

Memory Crypto Pkt Re-
Dispatcher Interconnect order Logic
Memory Crypto
RPs RPs SIPs
ESP RPs
SIP intercon.
SPA
IOCP
Aggreg.
SIP10 Block Diagram SPA SPA
ESPs RPs RPs
Reset / Pwr Ctrl Interconnect

EV-RP
Temp Sensor EV-FC In ref
clocks
EEPROM
DDRAM Egress
Ingress
Buffer
Boot Flash IOCP Scheduler
Status
(OBFL,…) (SC854x SOC)
JTAG Ctrl
SPA Aggregation Network
clock
ASIC (Marmot) … distribution
…
Ingress buffers Egress buffers

(per port) (per port)
Network
clocks
Ingress Classifier
Reset / Pwr Ctrl SPA Agg. C2W
ESI, 11.2 Gbps GE, 1Gbps

RPs RPs SPA-SPI, 11.2Gbps 4 SPAs 4 SPAs I2C 4 SPAs 4 SPAs 4 SPAs
Hypertransport, 10Gbps SPA Control
Other SPA Bus
SIP intercon.
SPA
IOCP
Aggreg.
SIP10 Block Diagram (comments) SPA SPA
ESPs RPs RPs

EV-RP
clocks
EEPROM
DDRAM Egress
Ingress
Buffer
Boot Flash IOCP Scheduler SPA Aggregation
Status
(OBFL,…) (SC854x SOC) Queues packets in & out
Uses Ingress and Egress buffers
JTAG Ctrl
IO Control Processor clock
Manages SPA OIR & drivers
…
Linux Kernel Egress Packet Buffers
Ingress buffers Egress buffers Holds packets if SPA backpressures
(per port) (per port) (e.g. Pause frames)
Network
clocks
Ingress Packet Buffers Ingress Classifier
Holds packets to ESP
Hi & Lo priority queues (1K only)
ESI, 11.2 Gbps GE, 1Gbps
RPs RPs SPA-SPI, 11.2Gbps 4 SPAs 4 SPAs I2C 4 SPAs 4 SPAs 4 SPAs
Hypertransport, 10Gbps SPA Control
Other SPA Bus
ISR Series Hardware Architecture
ISR 4451-X Hardware Diagram
DDR3 Control Plane 4xPCIe Data Plane 4xSGMI

DRAM (4 cores) (10 core) FPGE
Ctrl SVC1 PPE1 PPE2 PPE3 PPE4 PPE5
DDR3
DRAM
SVC2 SVC3 PPE6 PPE7 PPE8 PPE9 PPE10
10 Gbps XAUI
System 1xSGMI
FPGA DSP
Mgmt Ethernet Multi Gigabit
Fabric
Console / Aux 10 Gbps/slot
Peripheral SM-X
Interconnect SM-X
USB
2Gb/slot
Flash
NIM
NIM
NIM
ISR 4451-X Hardware Diagram (comments)
10 Cores, 1 thread / core Inline Cryptography
5 fwd cores by default No Crypto Assist chip
4 remaining cores license Crypto “locks” core
activated True run-to-completion
DDR3 Control Plane 4xPCIe Data Plane 4xSGMI

DRAM (4 cores) (10 core)
BQS onFPGE
a core
One Core dedicated to BQS
1 Control Plane Core Ctrl SVC1 PPE1 PPE2 PPE3 PPE4 PPE5 Always active
RP and FECP-like roles DDR3(5+1 or 9+1 cores)
DRAM
SVC2 SVC3 PPE6 PPE7 PPE8 PPE9 PPE10
3 Services Core
No hardware TCAM
10 Gbps XAUI
System 1xSGMI
FPGA DSP
Mgmt Ethernet Multi Gigabit
Fabric
Console / Aux 10 Gbps/slot
Peripheral SM-X
Interconnect SM-X
USB
2Gb/slot
Flash
NIM
NIM
NIM
ISR 44xx System Layout (2RU Platform)
Dataplane DIMM (left)
and
Controlplane DIMM (2x right)
6 or 10 core
Dataplane 4 Cores Control and
Services Plane
Compact Flash
Multi Gig
Ethernet Fabric
1 SW-NIM or Dual HDD
Configurable Slot
Integrated
(@ factory only)
Services Card
(e.g. DSP)
4431 & 4451
Front panel PoE

power
The ISR 43xx Series
4351 Hardware Diagram (aka Utah)
Rangeley CPU mSATA

DRAM
(MO-300)
PPE1 PPE2 PPE3 PPE4 PPE5 Mgmt Ethernet Console, Aux & USB
Front Panel Ethernet System Glue Logic Console
Front Panel Ethernet
Front Panel Ethernet FPGA
PPE6 PPE7 PPE8
I2C to Modules
SPI Flash GE Switch
USB Host Ports PCIe Switch
eMMc
USB-to-SD
NIM Slots x 2 NGSM Slots x 2
NIM Slots x 2 NGSM slots x 2
4351 Hardware Diagram (aka Utah)
8 Cores @ 2.4 Ghz / 1 thread per core
1 core for RP/IOSd
1 core acting for Crypto & QoS
4 cores @ 1 thread/core for features
2 service cores
1 core
Rangeley as
CPU RP hosting IOSd mSATA
DRAM
(MO-300)
PPE1 PPE2 PPE3 PPE4 PPE5 2 service cores

Mgmt Ethernet Console, Aux & USB
Front Panel Ethernet System Glue Logic Console
Front Panel Ethernet
Front Panel Ethernet FPGA
PPE6 PPE7 PPE8
I2C to Modules
SPI Flash 1 core as Crypto and BQS GE Switch
2 cores QFP
2 cores QFP license activated
USB Host Ports PCIe Switch
eMMc
USB-to-SD
NIM Slots x 2 NGSM Slots x 2 4331 and 4321 are similar; just less cores
NIM Slots x 2 NGSM slots x 2
and expansion slots
ESP
FECP
Generic ESP Block Diagram Crypto

Assist.
QFP
PPE BQS
intercon.
Reset / Pwr Ctrl Packet Buffer Part Len / BW

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP
DDRAM Packet Processor EngineComplex BQS

(OBFL,…) FECP
JTAG Ctrl PPE6 PPE7 PPE8 … PPEN
Dispatcher
GE, 1Gbps
Packet Buffer
I2C
SPA Control
Crypto SPA Bus
SPI Mux
DRAM Interconnect Hypertransport, 10Gbps
Other

Acronyms
• RP – Route Processor
• FP – Forwarding Processor = ESP (Embedded Service Processor)
• CPP – Cisco Packet Processor Compex= QFP (Quantum Flow Processor)
• PPE – Packet Processing Engine
• IOCP – I/O Control Processor
• FECP – Forwarding Engine Control Processor
• SPA – Shared Port Adapter
• SIP – SPA Interface Processor
• IOSd – IOS image that runs as a process on the RP
• FMAN – Forwarding manager (FMAN-RP, FMAN-FP)
• Scbac – FW Session Control Block
• EOBC = Ethernet Out of Band Channels – Packet Interface for Card to Card Control Traffic
• IOS-XE (BinOS) = Linux Based Software Infrastructure That Executes on MCP
Software Architecture
ASR1K Software Architecture
RP
CPU
ESI (10-40 Gbps)

interconn. GE switch
ESP
EOBC (1 Gbps) FECP
I2C
Crypto
QFP
Assist.
interconn.
ESI (10-40 Gbps)
SIP interconn.
SPA
IOCP
Aggreg.
SPA SPA
RP
CPU
Chassis Manager
IOS
ESI (10-40 Gbps)

Forwarding Manager
Linux Kernel
ESP
EOBC (1 Gbps) FECP
I2C
Crypto
QFP
Assist.
interconn.
ESI (10-40 Gbps)
SIP interconn.
SPA
IOCP
Aggreg.
SPA SPA
RP
CPU
Chassis Manager
IOS
ESI (10-40 Gbps)

Forwarding Manager
Linux Kernel
ESP FECP Chassis Manager

EOBC (1 Gbps)
Drivers Forwarding Manager
Drivers
I2C Drivers
Linux Kernel
QFP
µ µµ Crypto
µ BQS
µ µ Assist.
ESI (10-40 Gbps)
SIP interconn.
SPA
IOCP
Aggreg.
SPA SPA
RP
CPU
Chassis Manager
IOS
ESI (10-40 Gbps)

Forwarding Manager
Linux Kernel

EOBC (1 Gbps)
Drivers
I2C Drivers
Linux Kernel
QFP
µ µµ Crypto
µ BQS
µ µ Assist.
ESI (10-40 Gbps)
SIP
IOCP
SPA Driver Chassis
SPA Driver
SPA Driver Manager
Linux Kernel
SPA SPA SPA
Chassis Manager (CM)
RP • CM on RP communicates with CM processes
CPU on ESP and SIP
ESI (10-40 Gbps)
IOS
Chassis Manager
– Distributed function
Forwarding Manager
• Initializes hardware and boots other processes
Linux Kernel – CM on SIP queries SPA type and load SPA
drivers
• Manages hardware components
– Manages EOBC on RP
EOBC (1 Gbps)
Drivers
Drivers
Forwarding Manager
– Manages ESI links on RP/ESP/SIP
Drivers
–
I2C
Linux Kernel Manages timing circuitry on RP

– Reset and power-down on RP/ESP/SIP
QFP
µ µµ Crypto
µ
µ µ
BQS
Assist. • Communicates IOS hardware components
– Static & OIR
ESI (10-40 Gbps)
SIP
IOCP
SPA Driver Chassis
• Monitors environmental variables and alarms
SPA Driver
SPA Driver Manager
• Selects active/standby RP or ESP
Linux Kernel – Coordinates switchover in case of failure or
operator command
SPA SPA SPA
Forwarding Manager (FMAN)
RP
CPU FMAN-RP
• FMAN on RP communicates with
Chassis Manager FMAN process on ESP
IOS
– Distributed function
ESI (10-40 Gbps)
Forwarding Manager
Linux Kernel • Propagates control plane ops. to ESP

FMAN-FP – CEF tables, ACL’s, NAT, SA’s,…
ESP FECP ESP aka Forwarding Plane
• FMAN-FP communicates information
Chassis Manager
EOBC (1 Gbps)
Drivers
Drivers
Forwarding Manager back to FMAN-RP
Drivers
I2C
Linux Kernel – e.g. statistics

QFP – FMAN-RP pushes info back to IOS
µ µµ Crypto
µ BQS
µ µ Assist.
• FMAN on active RP maintains state for
ESI (10-40 Gbps)
SIP
both active & standby ESP’s
IOCP
SPA Driver
SPA Driver
Chassis
Manager
– Facilitates NSF after re-start with bulk
SPA Driver
download of state information
Linux Kernel
SPA SPA SPA
PPE Microcode
RP
CPU • Written in C
ESI (10-40 Gbps)
IOS
Chassis Manager
– proper features, no hack
Forwarding Manager
Linux Kernel • Runs on each thread of the PPE
ESP FECP
• Processes packets
Chassis Manager
– run to completion
EOBC (1 Gbps)

Drivers
PPE Microcode runs here
Drivers – assisted by various memories
I2C
Linux Kernel
QFP
– TCAM, DRAM,… various speeds
µ QFP
Packet Processor Engine BQS
µ
PPE PPE PPE PPE PPE
Crypto
µ µ… BQS
• Features applied via FIA
1 2 3 4 5
PPE PPE PPE PPE

6
µ µ
7 8 N
Assist.
Dispatcher
Packet Buffer
ESI (10-40 Gbps)
SIP
– Feature Invocation Array
IOCP
SPA Driver Chassis
• FIA per interface
SPA Driver
SPA Driver Manager
Linux Kernel
– input FIA, output FIA
SPA SPA SPA – drop FIA (Null interface)
Day in Life of Normal Packet
SIP intercon.
SPA
IOCP
Aggreg.
Ingress Packet Through SIP SPA SPA
ESPs

EV-RP
clocks
EEPROM
DDRAM Egress
Ingress
Buffer
Status
…
JTAG Ctrl
clock
…

Network
clocks
Ingress Classifier
SPA
ESP
FECP
Ingress Packet Through ESP Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP

(OBFL,…) FECP
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
Packet Dispatched to PPE Core Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP

(OBFL,…) FECP
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
Packet Dispatched to PPE Core Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP
Boot Flash PPE1 PPE3 PPE4 PPE5

FECP
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
Packet Dispatched to PPE Thread Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP

FECP
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
FIA’s Applied on Packet by PPE Thread Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP

FECP
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP

FECP
Thread 1
Thread 2
Thread 4
(OBFL,…)
Dispatcher
Packet Buffer
PPE2
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP

Assist. PPE
QFP
BQS
intercon.
X-ConnectReset / Pwr Ctrl

L2 Switch IPv4 IPv6 MPLS
Packet Buffer Part Len / BW
TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP

FECP
Thread 1
Thread 2
Thread 4
(OBFL,…)
Dispatcher
Packet Buffer
PPE2
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
Input FIA
PPE2 QFP
Netflow
Input ACL
Boot Flash NBAR Classify PPE1 PPE2 PPE3 PPE4 PPE5

FECP
Thread 1
Thread 2
Thread 4
(OBFL,…)
MQC Classify
JTAG Ctrl … PPE6 PPE7 PPE8 … PPEN
NAT
PBR
Dispatcher
Dialer IDLE Rst Packet Buffer
PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
Input FIA
PPE2 QFP
Netflow
Input ACL

FECP
Thread 1
Thread 2
Thread 4
(OBFL,…)
MQC Classify IP Unicast
NAT IP Multicast
PBR
Dispatcher
Dialer IDLE Rst Packet For
Packet Buffer
Us PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
Input FIA Output FIA
PPE2 QFP
Netflow Netflow
Input ACL
NAT
FECP
Thread 1
Thread 2
Thread 4
(OBFL,…) NBAR Classify
JTAG Ctrl … PPE6 PPE7 PPE8 … … N
PPE
NAT IP Multicast MQC Policing

PBR MAC Accounting
Dispatcher
Packet Buffer
Us Output ACL PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP
Netflow Netflow
Input ACL
NAT
FECP
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…) NBAR Classify
JTAG Ctrl … PPE6 PPE7 PPE8 … … N
PPE

PBR MAC Accounting
Dispatcher
Packet Buffer
Us Output ACL PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
Leaving the PPE Thread Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP
Netflow Netflow
BGP Accounting
NAT
FECP NBAR Classify
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
JTAG Ctrl … PPE6 PPE7 PPE8 … …N
PPE

PBR
Dispatcher WRED
Packet Buffer
Us Output ACL PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
Packet proceeding to BQS then SIP Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP

(OBFL,…) FECP
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

SIP intercon.
SPA
IOCP
Aggreg.
Egress Packet Through SIP SPA SPA
ESPs

EV-RP
clocks
EEPROM
DDRAM Egress
Ingress
Buffer
Status
JTAG Ctrl
clock
…

Network
clocks
Ingress Classifier
SPA
Debugging strategies
Everyday situations
Traffic did not reach its target !

What happened to that packet ?
Why did that happen ?
Everyday situations
IPsec ZBF NAT
WAAS SNMP
OTV
First Routing
Which feature went wrong ?
What went wrong in the feature ?

Memory
Config performance Ordering
Bug
Traffic Ambiguity
issue
Everyday situations
Second
What went wrong in the feature ?

Performanc Memory
Config e Ordering
Bug
Traffic Ambiguity
issue
Using statistics for troubleshooting packet drops
Not easy… not very practical either.
Let’s dig deeper before making it simpler
• SPA • ESP
• show platform hardware slot {f0|f1} serdes statistics

• show interfaces <interface-name>
• show platform hardware slot {f0|f1} serdes statistics internal
• show interfaces <interface-name> accounting
• show platform hardware qfp active bqs 0 ipm mapping
• show interfaces <interface-name> stats
• show platform hardware qfp active bqs 0 ipm statistics channel all
• SIP
• show platform hardware qfp active bqs 0 opm mapping
• show platform hardware port <slot/card/port> plim statistics
• show platform hardware qfp active bqs 0 opm statistics channel all
• show platform hardware subslot {slot/card} plim statistics

• show platform hardware qfp active statistics drop [detail]
• show platform hardware slot {slot} plim statistics • show platform hardware qfp active interface if-name <Interface-name> statistics
• show platform hardware slot {0|1|2} plim status internal • show platform hardware qfp active infrastructure punt statistics type per-cause | exclude _0_
• show platform hardware slot {0|1|2} serdes statistics • show platform hardware qfp active infrastructure punt statistics type punt-drop | exclude _0_
• show platform hardware qfp active infrastructure punt statistics type inject-drop | exclude _0_
• RP
• show platform hardware qfp active infrastructure punt statistics type global-drop | exclude _0_
• show platform hardware slot {r0|r1} serdes statistics
• show platform hardware qfp active infrastructure bqs queue output default all
• show platform software infrastructure lsmpi
• show platform hardware qfp active infrastructure bqs queue output recycle all
Debugging Strategies to Date
Top Down
IOS Control Plane
Well Known
• ACL + show access-list,…
• show interface / ip route / bgp …
Rock bottom
Platform Control Plane

Very Difficult • ESP “stuff”
• e.g. show platform … hard to Let’s
remember change
that!!
Data Plane
• ESP “stuff”
• More arcane show platform …
The Road to Simplification:
The Packet Tracer
69
IOS 3.7
The Embedded Packet Capture
One way of capturing packets…
Device# monitor capture mycap start

Device# monitor capture mycap access-list v4acl
Device# monitor capture mycap limit duration 1000
Device# monitor capture mycap interface GigabitEthernet 0/0/1 both
Device# monitor capture mycap buffer circular size 10
Device# monitor capture mycap start Shows whether packets have been received or sent
Device# monitor capture mycap export tftp://10.1.88.9/mycap.pcap
Device# monitor capture mycap stop
Shows what packets look like
Requires hex dump analysis or export to decoder (sniffer)
Does not tell us what happened to the packet
Device# show monitor capture mycap buffer dump
0
0000: 01005E00 00020000 0C07AC1D 080045C0 ..^...........E.
0010: 00300000 00000111 CFDC091D 0002E000 .0..............
0020: 000207C1 07C1001C 802A0000 10030AFA .........*......
0030: 1D006369 73636F00 0000091D 0001 ..example....... Excellent tool but insufficient in many cases
1
0000: 01005E00 0002001B 2BF69280 080046C0 ..^.....+.....F.
0010: 00200000 00000102 44170000 0000E000 . ......D.......
0020: 00019404 00001700 E8FF0000 0000 .............. http://www.cisco.com/en/US/docs/ios-
xml/ios/epc/configuration/xe-3s/asr1000/nm-packet-capture-
2 xe.html
0000: 01005E00 0002001B 2BF68680 080045C0 ..^.....+.....E.
0010: 00300000 00000111 CFDB091D 0003E000 .0..............
0020: 000207C1 07C1001C 88B50000 08030A6E ...............n
0030: 1D006369 73636F00 0000091D 0001 ..example.......
IOS 3.10
The Packet Tracer and FIA Debugger
TCAM Resource DRAM
Packet Buffer Part Len / BW Packet # 16
Condition determines packets DRAM SRAM
Temp Sensor
to be traced Input ACL
EEPROM
Optionally match on the
Pak Match ? PPE2 QFP egress FIA MQC Classify
DDRAM Input ACL Packet Processor EngineComplex BQS
Output ACL NAT
FECP
MQC
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…) PBR
Classify NAT
Encaps Output ACL
NAT IP Unicast
Statistics and final action will be NAT
Dispatcher Crypto
collected (matched packets dropped,
Packet Buffer punted to RP, forwarded to output
PBR PPE2 Encaps
interface …)
Crypto Thread 3 Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
Optionally, FIA actions can logged per packet
System can capture several packets flows
RPs RPs ESP RPs SIPs Packet flows can be reviewed in show commands
Packet Tracer Demonstration
Demo Network Diagram – Problem 1
This extranet site can not connect to the server in
the DMZ.
192.168.3.1
10.0.101.0/24 Spoke 1
Spoke 2
Spoke 3 IPv4 Internet
Spoke … Ipv6 Internet
GE 1 ASR1000 GE 2
10.0.112.0/24 Spoke 12 GE 3 2003:1::1
192.168.3.12
Spoke 13
2005:1::1/64
10.0.0.254
DMZ
Demo Network Diagram – Problem 2
Plenty of drops… who, what, why ?

192.168.3.1
10.0.101.0/24 Spoke 1
Spoke 2
GE 1 ASR1000 GE 2
10.0.112.0/24 Spoke 12 GE 3 2003:1::1
192.168.3.12
Spoke 13
2005:1::1/64
10.0.0.254
DMZ
IPsec Packet Forwarding
ESP
FECP
Egress IPsec Packet Flow (I) Crypto

Assist. PPE
QFP
BQS
Look up IPsec proxy-identities Lookup SA Handler by class-group ID

Obtain Crypto SA ctx ID intercon.
Obtain class-group ID
TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP

(OBFL,…) FECP
Uses Crypto Context identified

by Context ID Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
Egress IPsec Packet Flow (II) Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP
DDRAM packetProcessor EngineComplex
PPE may be different butPacket BQS
processing continues where it
Boot Flash
stopped (right after crypto)
PPE1 PPE2 PPE3 PPE4 PPE5
(OBFL,…) FECP
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

Simplifying the IPsec show commands
One show command to rule them all
------------------ show platform software ipsec fp active flow identifier 34130 ------------------
…
interface: Virtual-Access1002
Crypto map tag: Virtual-Access1002-head-0, local addr 172.18.0.1 ------------------ show platform hardware qfp active feature ipsec sa 1427 ------------------
…
protected vrf: (none)
local ident (addr/mask/prot/port): (172.18.0.1/255.255.255.255/47/0) ------------------ show platform software ipsec fp active encryption-processor context 6502aa4f ------------------
remote ident (addr/mask/prot/port): (17.0.0.26/255.255.255.255/47/0)
current_peer 17.0.0.26 port 500
…
PERMIT, flags={origin_is_acl,} ------------------ show platform software ipsec fp active flow identifier 34129 ------------------
#pkts encaps: 25227, #pkts encrypt: 25227, #pkts digest: 25227
#pkts decaps: 25237, #pkts decrypt: 25237, #pkts verify: 25237 …
#pkts compressed: 0, #pkts decompressed: 0 ------------------ show platform hardware qfp active feature ipsec sa 1867 ------------------
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0 …
#send errors 0, #recv errors 0 ------------------ show platform software ipsec fp active encryption-processor context 2e02aa4e -----------------
local crypto endpt.: 172.18.0.1, remote crypto endpt.: 17.0.0.26 …
plaintext mtu 1458, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0/2
current outbound spi: 0xA7B61FE5(2813730789)
PFS (Y/N): N, DH group: none
inbound esp sas:

spi: 0xA222F391(2720199569)
transform: esp-aes esp-sha-hmac ,
in use settings ={Transport, }
conn id: 36130, flow_id: HW:34130, sibling_flags FFFFFFFF80000008, crypto map: Virtual-
Access1002-head-0 show crypto ipsec sa interface virtual-access 1002 platform
sa timing: remaining key lifetime (k/sec): (4607974/2137)
IV size: 16 bytes
replay detection support: Y replay window size: 512
or
…
Status: ACTIVE(ACTIVE)
show crypto ipsec sa peer 17.0.0.26 platform
outbound esp sas:
spi: 0xA7B61FE5(2813730789)
Access1002-head-0
IV size: 16 bytes
…
Zone Based Firewall
ZBF Quick Recap
Sample Config
zone security inside Zone Definition
zone security outside ip access-list extended ipacl
deny ip 10.0.0.0 0.0.0.255 172.16.0.0 0.0.0.127
interface GigabitEthernet0/0/2 deny ip 10.0.0.0 0.0.0.255 172.16.0.128 0.0.0.127
ip address 172.18.25.254 255.255.255.0 permit tcp any any
Apply zone to interfaces deny udp 10.0.0.0 0.0.0.255 172.16.0.0 0.0.0.127
zone-member security inside
permit udp any any
interface GigabitEthernet0/0/3 permit icmp any any
ip address 172.19.25.254 255.255.255.0
zone-member security outside
Class map to match traffic
class-map type inspect match-all ipv4acl
match access-group name ipacl
Policy map to determine
policy-map type inspect in2out action on matched traffic
class type inspect ipv4acl
inspect
class class-default
Apply policy between two
zone-pair security in2out source inside destination outside zones
service-policy type inspect in2out
Simplifying the ZBF show commands
Three Commands for ZBF under the sky
Collects everything
show policy-firewall config platform but TCAM
--show platform software firewall FP active bindings--
--show platform software firewall RP active bindings--
--show platform software firewall FP active pairs--
--show platform software firewall RP active pairs--
--show platform software firewall FP active parameter-maps--
--show platform software firewall RP active parameter-maps--
--show platform software firewall FP active zones--
--show platform software firewall RP active zones--
show policy-firewall sessions platform

show tech firewall
--show platform hardware qfp active feature firewall datapath scb any any any any any all any --
show policy-firewall stats platform
--show platform software firewall FP active statistics--

--show platform software firewall RP active statistics--
--show platform hardware qfp active feature firewall runtime--
--show platform hardware qfp active feature firewall memory--
--show platform hardware qfp active feature firewall drop--
--show platform hardware qfp active feature firewall client statistics--
ESP
FECP
ZBF Packet Flow Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP

(OBFL,…) FECP
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
FIA’s Applied on Packet by PPE thread Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP

FECP
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP
Netflow
DDRAM Packet Processor EngineComplex
... BQS
Input ACL
OUTPUT_INSPECT
FECP
Thread 1
Thread 2
Thread 4
(OBFL,…)
MQC Classify
NAT IP Unicast
PBR
Dispatcher
PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
Inside Output Threat Inspect Crypto

Assist. PPE
QFP
BQS
intercon.

Packet Buffer Part Len / BW TCAM
TCAM Resource DRAM
DRAM SRAM
Input
Temp Sensor
Policy Selection Session Lookup Miss Classify Traffic
(precise + imprecise)
EEPROM
Hit
PPE2 QFP Pass
Netflow
... BQSDrop Inspect
Input ACL
OUTPUT_INSPECT
Create Session
FECP Session DB
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify Drop
NAT IP Unicast L4 Inspection

PBR
Pass Dispatcher
Dialer IDLE Rst Packet Buffer L7 Parse
PPE2
URD
Crypto Imprecise Channel Thread 3
SPI Mux L7 Inspection
Creation
Reset / Pwr Ctrl
SA table
Output DRAM Interconnect IPV4 OUTPUT
INSPECT
ESP
FECP

Assist. PPE
QFP
BQS
Using Session DB in DRAM

Imprecise lookup only for initial Match each class-map in intercon.
µIDB input+output  Zone Pair  Policy
packets (syn…) policy (ACL’s in TCAM)
TCAM Resource DRAM
DRAM SRAM
Input
Temp Sensor
Policy Selection Session Lookup Miss Classify Traffic
EEPROM
Hit
PPE2 QFP Pass
If Action = Inspect, create
Netflow
... BQSDrop Inspect
session flow in DB
Input ACL
OUTPUT_INSPECT
Create Session
FECP Session DB
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify Drop

PDU reassembly, parsing
PBR
Dispatcher (HTTP GET, POST,…)
Pass
PPE2
URD
Creation
Reset / Pwr Ctrl
SA table
Action Mapping
Child session creation (data flow INSPECT

RPs RPs ESP RPs from FTP, RTP flow from SIP,…)
SIPs
ESP
FECP

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Miss
Input
Temp Sensor
Policy Selection Session Lookup Classify Traffic
EEPROM
Hit
PPE2 QFP Pass
Netflow
... BQSDrop Inspect
Input ACL
OUTPUT_INSPECT
Create Session
FECP Session DB
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify Drop

show policy-firewall session platform
--show platform
hardware qfp active feature firewall datapath scb any any any any any all any --
[s=session i=imprecise channel c=control channel d=data channel]
172.18.25.66
PBR 58513 10.0.0.1 1967 proto 6 (0:0)[sc]
172.18.25.66 59869 10.0.0.1 1967 proto Pass Dispatcher
17 (0:0)[sc]
Dialer IDLE
172.18.25.66 Packet Buffer
Rst10.2.6.254 1967 proto 6 (0:0)[sc]
59824 L7 Parse
172.18.25.66 56338 10.11.32.15 6665 proto 17 (0:0)[sd] 2 PPE
URD
… Crypto Imprecise Channel Thread 3
Creation
Reset / Pwr Ctrl
SA table
THREAT
RPs RPs ESP RPs SIPs INSPECT
ESP
FECP

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Miss
Input
Temp Sensor
EEPROM
Hit
PPE2 QFP Pass
Netflow
... BQSDrop Inspect
Input ACL
OUTPUT_INSPECT
Create Session
FECP Session DB
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify Drop
JTAG Ctrl …
show policy-firewall PPE6 PPE7
session platform tcp destination-port PPE8
80 detail
… PPEN
--show platform hardware qfp active feature firewall datapath scb any any any 80 6 L4 allInspection
any detail--
NAT IP Unicast
[s=session i=imprecise channel c=control channel d=data channel]
172.18.25.66
PBR 53471 213.94.72.66 80 proto 6 (0:0)[sc]
… Pass Dispatcher
nxt_timeout: 100,Rst
Dialer IDLE refcnt: 1, ha nak cnt: 0,Packet
rg: Buffer
0, sess id: 32584 L7 Parse
… 2 PPE
URD
ingress/egress intf: GigabitEthernet0/0/2 (1021), GigabitEthernet0/0/3 (65526)
Crypto
current time 1384744571498 create tstamp:
Imprecise
1384690046997
Channel
last access: 1384690179236L7 Inspection
Thread 3
SPI Mux
… Creation
Reset / Pwr Ctrl syncookie fixup: 0x0
SA table
Output … DRAM Interconnect IPV4 OUTPUT
THREAT
ESP
FECP
Post ZBF FIA Continuation Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Miss
Input
Temp Sensor
EEPROM
Hit
PPE2 QFP Pass
Netflow
... BQSDrop Inspect
Input ACL
OUTPUT_INSPECT
Create Session
FECP Session DB
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify Drop

PBR
Pass Dispatcher
PPE2
URD
Creation
Reset / Pwr Ctrl
SA table
THREAT
ESP
FECP
Post ZBF FIA Continuation Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP
Netflow
... BQS
Input ACL
OUTPUT_INSPECT
FECP …
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify
VFR_REFRAG
L2_REWRITE
NAT IP Unicast
PBR
Dispatcher
PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
ZBF Packet Flow (cont.) Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP

(OBFL,…) FECP
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

Network Address Translation
NAT Quick Recap
Sample Config
interface GigabitEthernet0/0/2
ip nat inside
Apply role to interfaces
interface GigabitEthernet0/0/3
ip nat outside
Static NAT configuration
ip nat inside source static 172.16.89.32 10.0.0.1
Dynamic NAT configuration
ip nat inside source list pat interface GigabitEthernet0/0/3 overload (NAT Overload aka PAT)
ip access-list extended pat Match traffic to PAT

permit ip 172.18.25.0 0.0.0.255 any
ESP
FECP

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP
DDRAM
Netflow Packet Processor EngineComplex
... BQS
Input ACL OUTPUT_NAT

FECP
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…) INPUT_VFR
JTAG Ctrl
MQC Classify
PPE6 PPE7 PPE8 … PPEN
…
IP Unicast
PBR
Dispatcher
URD Packet Buffer
PPE2
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
NAT In  Out – OUTPUT_NAT Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor Miss Hit
Input In  Out Check Session Lookup Child Session Lookup
EEPROM
Input FIA Output FIA Miss
Hit
PPE2 QFP
Netflow Hit
... Static NATBQS
Lookup
Input ACL
OUTPUT_NAT Miss
Boot Flash NBAR Classify Session DB PPE1 PPE2 PPE3
Door
PPE
DB PPE
FECP 4 5
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify Classification ACL
JTAG Ctrl … PPE6 PPE7 PPE8 … PPEN then Route-Map
Bind DB
NAT IP Unicast
PBR
Dispatcher Allocate Addr Drop

PPE2
URD
Output L7Crypto
Translation Alg
SPI Mux L3/L4 Translation SessionThread
Create 3
Reset / Pwr Ctrl Untranslated
SA table
DRAM Interconnect IPV4 OUTPUT
NAT
ESP
FECP
NAT In  Out – OUTPUT_NAT Crypto

Assist. PPE
QFP
BQS
Using Session DB Half Open (embryonic) intercon.

packet from NAT inside  outside ?
in DRAM L7 Child Sessions Connection checks
TCAM Resource DRAM
DRAM SRAM
EEPROM
Hit
PPE2 QFP
Netflow Hit
... Static NATBQS
Lookup
Input ACL
OUTPUT_INSPECT Miss
Door
PPE
DB PPE
FECP 4 5
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…) Use config  use CGM and
TCAM to match packet
JTAG Ctrl … PPE6 PPE7 PPE8 … PPEN then Route-Map against configured rule
Bind DB
NAT Descriptor modification
SIP Media IP Unicast
+ RTPPBR
pinhole creation, FTP data
session fixup,… Dispatcher Allocate Addr Drop

PPE2
URD
Output L7Crypto
Translation Alg
Create 3
SA table
NAT
ESP
FECP
Post NAT FIA Continuation Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP
Netflow
... BQS
Input ACL
OUTPUT_INSPECT
OUTPUT_NAT
FECP …
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify
VFR_REFRAG
L2_REWRITE
NAT IP Unicast
PBR
Dispatcher
PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

Understanding and Extracting ESP Logs
108
ESP Tracing aka Logging
TEMP RAM FS
RP RP logs are first written
CPU
Chassis Manager
here (efficiency)
IOS
NFS Shared Disk
ESI (10-40 Gbps)

Forwarding Manager
Hard disk is really here
Linux Kernel
ESP FECP
TEMP RAM FS
Chassis Manager
ESP logs are first written
EOBC (1 Gbps)
here (efficiency)
Drivers
I2C Drivers
Linux Kernel
QFP Mounted NFS

µ µµ Crypto
µ BQS
Assist. ESP logs are committed
µ µ
here at regular intervals
ESI (10-40 Gbps)
SIP
IOCP
SPA Driver Chassis
SPA Driver
SPA Driver Manager
Linux Kernel
SPA SPA SPA
Important logs
RP
CPU
Chassis Manager
fman_rp_R[0|1]-0.log
IOS
ESI (10-40 Gbps)

Forwarding Manager Under /harddisk/tracelogs
fman_rp_R[0|1]-0.log.<timestamp>
Linux Kernel
fman-fp_R0.log.<timestamp>
cpp_cp_F[0|1]-0.log.<timestamp>
EOBC (1 Gbps) fman_fp_F[0|1]-0.log
Drivers Forwarding Manager cpp_cp_F[0|1]-0.log
Drivers
I2C Drivers
Linux Kernel
QFP Under /harddisk/tracelogs/

µ µµ Crypto fman-fp_R0.log.<timestamp>
µ BQS
µ µ Assist.
cpp_cp_F[0|1]-0.log.<timestamp>
ESI (10-40 Gbps)
SIP
IOCP
SPA Driver Chassis
SPA Driver
SPA Driver Manager
Linux Kernel
SPA SPA SPA
What log files are important?
• Important log files to get for security issues:
– fman_rp_R[0|1].log (under /tmp/rp/trace directory on RP)
– fman-fp_F[0|1]-0.log (under /tmp/fp/trace directory on ESP
– cpp_cp_F[0|1]-0.log (under /tmp/fp/trace directory on ESP)
• All these logs get rotated and are copied to /harddisk/tracelogs directory
on active RP.
• Look for the relevant log files depending on the time of the failure
• By default, all ERR messages are logged, these should be the first things
to look for
Example log files
The timestamp…
My-ASR1000-2#dir harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/tracelogs/
3768365 -rwx 1048934 Jan 6 2014 18:20:16 +00:00 cpp_cp_F0-0.log.7133.20140106182015
3768330 -rwx 551643 Jan 7 2014 09:27:51 +00:00 cpp_cp_F0-0.log.7133.20140107092751
3768335 -rwx 1048901 Jan 7 2014 08:56:44 +00:00 cpp_cp_F0-
0.log.7133.2014010708564339313059840 bytes total (30680653824 bytes free)
Rotating the log files
My-ASR1000-2#dir harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/traceMy-ASR1000-2#test platform software trace slot rp active forwarding-manager
rotate
Rotated file from: /tmp/rp/trace/stage/fman_rp_R0-0.log.13836.20140107094754, Bytes: 0, Messages:
6535
My-ASR1000-2#test platform software trace slot FP active cpp-control-process rotate
Rotated file from: /tmp/fp/trace/stage/cpp_cp_F0-0.log.7133.20140107093650, Bytes: 154027, Messages:
786
My-ASR1000-2#test platform software trace slot FP active forwarding-manager rotate
Rotated file from: /tmp/fp/trace/stage/fman-fp_F0-0.log.8247.20140107093738, Bytes: 20170, Messages:
210
OR use
My-ASR1000-2#request platform software trace rotate all Does not show the rotated file names w/
time stamp  have to hunt them down
Conditional Feature Debugging
114
IOS 3.10
The Packet Tracer and Conditional Debugger
TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP

FECP
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

IOS 3.10
TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
Input FIA
PPE2 QFP

FECP
Thread 1
Thread 2
Thread 4
(OBFL,…)
Dispatcher
Packet Buffer
PPE2
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

IOS 3.10
TCAM Resource DRAM
Temp Sensor
to be traced
EEPROM
Input FIA
Ingress Match ? PPE2 QFP

FECP
Thread 1
Thread 2
Thread 4
(OBFL,…)
Dispatcher
Packet Buffer
PPE2
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

IOS 3.10
TCAM Resource DRAM
Temp Sensor
to be traced
EEPROM
Input FIA

FECP
Thread 1
Thread 2
Thread 4
(OBFL,…)
Dispatcher
Packet Buffer
PPE2
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
The packet tracer collects statistics
and final action (matched packets
dropped, punted to RP, forwarded to
RPs RPs ESP RPs SIPs output interface …)
IOS 3.10
TCAM Resource DRAM
Temp Sensor
to be traced
Input FIA Output FIA Input ACL
EEPROM
MQC Classify
Output ACL NAT
FECP
MQC
Thread 1
Thread 2
Thread 4
(OBFL,…)
NAT PBR
Classify
Encaps Output ACL
NAT IP Unicast
NAT
Dispatcher Crypto
Packet Buffer
PBR PPE2 Encaps
Crypto Thread 3 Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
IOS 3.10
TCAM Resource DRAM
Temp Sensor
to be traced
EEPROM
MQC Classify
Output ACL NAT
FECP
MQC
Thread 1
Thread 2
Thread 4
(OBFL,…)
NAT PBR
Classify
Encaps Output ACL
NAT IP Unicast
NAT
Dispatcher Crypto
Packet Buffer
PBR If feature conditional debugger is PPE2 Encaps
activated, these blocks will be
Crypto
debugged
Thread 3 Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
Our focus now
Conditional Debugger Demonstration
Demo Network Diagram – ZBF Problem
192.168.3.1
10.0.101.0/24 Spoke 1
Spoke 2
GE 1 ASR1000 GE 2
10.0.112.0/24 Spoke 12 GE 3 2003:1::1
192.168.3.12
Spoke 13
2005:1::1/64
PC’s located in Spoke 12 are not able to reach
webserver in DMZ
10.0.0.254
DMZ
Interesting ZBF Commands (summary)
• debug platform condition feature fw dataplane submode detail layer4 policy drop
– DO NOT use “event” – non conditional – HIGH CPU IMPACT
• In 3.14+: debug platform condition feature dw dataplane submode all
– In this release, “event” will be conditional  “all” will be safe too
Resource Monitoring
In IOS-XE near you now…
129
The vital signs…
RP
CPU
Chassis Manager
IOS
Forwarding Manager
Control Plane CPU’s
Linux Kernel

Where does it hurt ?
Drivers
Drivers
Linux Kernel
µ QFP
Data Plane CPU’s µ
µ µµ BQS
Crypto
µ Assist.
SIP
IOCP
SPA Driver Chassis
SPA Driver
SPA Driver Manager
Linux Kernel
SPA SPA SPA
Example: IOS Memory Usage vs IOSd RP Utilization
asr-1k#show mem stat
Load for five secs: 6%/1%; one minute: 5%; five minutes: 3% RP
Time source is NTP, 22:18:08.111 EDT Sat Apr 19 2014 CPU
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b) Chassis Manager
Processor 300AE008 1713127140 564269356 1148857784 1066242316 992444168 IOS
lsmpi_io 963791D0 6295088 6294120 968 968 968
Forwarding Manager
asr-1k#show process mem | inc BGP
523 0 2333028 51368 389076 313 313 BGP Router
Linux Kernel
asr-1k#show process cpu
…
Complex CLI, platform specific.
Additional information require connecting to the Linux shell

asr-1k#sh platform software process list RP active summary
…
Architecture : ppc
Memory (kB)
Physical : 4127744
Total : 3874516
Used : 2095636
Free : 1778880
asr-1k#sh platform software process list RP active | inc fman

fman_rp 29015 27992 29015 S 20 136847360
asr-1k#show platform hardware qfp active infrastructure exmem statistics
QFP Memory Utilization QFP exmem statistics
Type: Name: DRAM, QFP: 0

Total: 1073741824
It is getting worse… InUse: 219466752
Free: 854275072
Lowest free water mark: 854005760
Type: Name: IRAM, QFP: 0
Total: 134217728
InUse: 8728576
Free: 125489152
Type: Name: SRAM, QFP: 0
Total: 32768
InUse: 15088
Free: 17680
asr-1k#show platform hardware qfp active infrastructure exmem statistics user

…
10 279092 284672 CEF
40 36441494 36458496 NAT

Drivers
asr-1k#show platform hardware qfp active tcam resource-manager usage Drivers
Load for five secs: 0%/0%; one minute: 1%; five minutes: 1% Linux Kernel
Time source is NTP, 09:43:55.075 EDT Fri Apr 25 2014
QFP TCAM Usage Information

QFP
µ µ µ
<snip>
Crypto
Total TCAM Cell Usage Information µ BQS
---------------------------------- µ µ Assist.
Name : TCAM #0 on CPP #0
Total number of regions : 3
Total tcam used cell entries : 28 TCAM DRAM DRAM
Total tcam free cell entries : 524260
Threshold status : below critical limit
IOS 3.14
Starting point: the simplified view
RP
CPU
Chassis Manager
asr-1k# show platform resources IOS
Resource Usage Max Warning Critical State Forwarding Manager
RP0 (active) H
Linux Kernel
Memory 1814MB 3783MB 90% 95% H
CPU 5.80% 100% 90% 95% H
FP0 (active) H
Memory 683MB 1962MB 90% 95% H
CPU 19.89% 100% 90% 95% H ESP FECP Chassis Manager
QFP H
DRAM 76244KB 524288KB 80% 90% H
IRAM 8817KB 131072KB 80% 90% H Drivers Forwarding Manager
SRAM 14KB 32KB 80% 90% H
Drivers
Drivers
TCAM 28cells 131072cells 80% 90% H Linux Kernel
FP1 (standby) H
Memory 683MB 1962MB 90% 95% H
CPU 19.89% 100% 90% 95% H µ QFP
µ Crypto
QFP H µ µµ BQS
Assist.
DRAM 76244KB 524288KB 80% 90% H µ
IRAM 8817KB 131072KB 80% 90% H
SRAM 14KB 32KB 80% 90% H
TCAM 28cells 131072cells 80% 90% H SIP
SIP0 (active) H IOCP
SPA Driver Chassis
Memory 307MB 460MB 90% 95% H SPA Driver
CPU 4.10% 100% 90% 95% H SPA Driver Manager
SIP1 (standby) H
Memory 160MB 460MB 90% 95% H
Linux Kernel
CPU 1.10% 100% 90% 95% H
**State Acronym: H - Healthy, W - Warning, C – Critical
SPA SPA SPA
IOS 3.14
Other show commands improved too
Improves interaction with TAC RP
CPU
Chassis Manager
IOS
Forwarding Manager
Control Plane CPU’s
Linux Kernel

Drivers
Drivers
Linux Kernel
µ QFP
Data Plane CPU’s µ
µ µµ BQS
Crypto
µ Assist.
show process cpu or show memory or show process memory SIP

IOCP
SPA Driver Chassis
SPA Driver
SPA Driver Manager
This command only shows processes inside the IOS daemon. Linux Kernel
Please use 'show <something> platform'
to show processes from the underlying operating system. SPA SPA SPA
Wrapping up…
135
ESP
FECP
QoS Mark/Police
General Feature Dependencies NAT sessions
IPSec SA
Crypto
Assist.
QFP
PPE BQS
Netflow Cache
Per session data (FW, Netflow) QoS Queuing intercon.
Class/Policy Maps: QoS, DPI NAT VFR re-assembly
ACL/ACE storage
Reset / Pwr Ctrl IPSec headers
IPSec Traffic Selectors, classes, rules TCAM Resource DRAM
DRAM SRAM
NAT lists Temp Sensor
EEPROM Cores execute packet processing

QFP
All features handled from here
CPU
Packet horsepower
Processor is here…
Engine Complex BQS
DDRAM

(OBFL,…)
Memory for FECP FECP
QFP client / driver
OBFL
BQS offloads queuing and scheduling from
QoS Class maps
Crypto Assist chip offloads crypto cores.
FM FP
from the PPE cores 16000 Queues on ASR1001 & ESP 5
Statistics
Dispatcher 128000 Queues on ESP10+
ACL ACEs copy GE, 1Gbps
Packet Buffer 4x118,000 Queues on ESP 200+
NAT config objects 2IC
IPSec/IKE SA SPA Control
NF config data Crypto System Bandwidth SPA Bus
ZB-FW config objects SPI Mux 5, 10, 20, 40, 100, 200 Gbps
DRAM Interconnect Hypertransport, 10Gbps
Other

New Debugging Strategy
IOS Control Plane
Well Known
• show interface, show ip route, show bgp …
• Feature debugging
Platform Control Plane
Still Difficult • Unified show commands
(not overly) • Platform show commands
• Future: control plane conditional debugging
Data Plane
Easy!! • Packet Tracer
• Forwarding plane conditional debugging
• Embedded Packet Capture
Call to Action
• Visit the World of Solutions for
– Cisco
– Walk in Labs
– Technical Solution Clinics
• Meet the Engineer
• Lunch time Table Topics
• DevNet zone related labs and sessions
• Recommended Reading: for reading material and further resources for this
session, please visit www.pearson-books.com/CLMilan2015
Complete Your Online Session Evaluation
• Please complete your online session
evaluations after each session.
Complete 4 session evaluations
& the Overall Conference Evaluation
(available from Thursday)
to receive your Cisco Live T-shirt.
• All surveys can be completed via

the Cisco Live Mobile App or the
Communication Stations
Feature Invocation Array (FIA)
• A per protocol array of functions/features to be executed in sequence
• The FIA’s are executed in PPE for every packet
• Input interface  Input FIA ; Output interface  Output FIA
Example Input FIA Example Output FIA Example “Punt” FIA
Dst Lookup Consume Output Inspect
Output Inspect
For Us Martian WCCP RP seens as an external device
VFR Refrag from the ESP… connected to a
RPF NAT special interface. This interface
has its own FIA.
Security ACL (in) Refragment Drop Policy
RPF Checks MQC Classify Internal Transmit Pkt

NAT Lawful Intercept
PBR Security ACL (out)
Input Lookup Process Tunnel Encapsulation

These are simple examples. Real
IP Options Process Crypto (tunnel protection) FIA’s can be somewhat arcane...
Zone Based Firewall
ZBF Quick Recap
Sample Config
zone security inside Zone Definition
zone security outside ip access-list extended ipacl
deny ip 10.0.0.0 0.0.0.255 172.16.0.0 0.0.0.127
interface GigabitEthernet0/0/2 deny ip 10.0.0.0 0.0.0.255 172.16.0.128 0.0.0.127
ip address 172.18.25.254 255.255.255.0 permit tcp any any
Apply zone to interfaces deny udp 10.0.0.0 0.0.0.255 172.16.0.0 0.0.0.127
zone-member security inside
permit udp any any
interface GigabitEthernet0/0/3 permit icmp any any
ip address 172.19.25.254 255.255.255.0
zone-member security outside
Class map to match traffic
class-map type inspect match-all ipv4acl
Policy map to determine
policy-map type inspect in2out action on matched traffic
inspect
class class-default
Apply policy between two
zone-pair security in2out source inside destination outside zones
Zone Creation
RP
CPU
zone security inside
zone security outside Chassis Manager
IOS
Forwarding Manager
Linux Kernel
sh platform software firewall fp active zones
EOBC (1 Gbps)
Forwarding Manager Firewall Zone Configurations
Zone Name: inside, parameter-map: (null), Obj-id 1

Zone Name: outside, parameter-map: (null), Obj-id 2
Zones are created (# of zones increase) but Zone Name: self, parameter-map: (null), Obj-id 65535
zone details and names can not be seen.
# of zones always includes zone “self”.
Drivers
Drivers
Linux Kernel
show platform hardware qfp active feature firewall client statistics
…
QFP
Number of vrf interfaces with zone: 0 µ µ Crypto
Number of zoned interfaces: 0 µ µµ BQS
Assist.
µ
Number of zones: 3
Number of zone pairs with policy: 0
TCAM DRAM DRAM
Zone Binding
interface GigabitEthernet0/0/2 RP
zone-member security inside CPU
Chassis Manager
interface GigabitEthernet0/0/3 IOS
zone-member security outside Forwarding Manager
Linux Kernel
sh platform software firewall f0 bindings

Output feature activated on ALL ESP Interface Handle (µIDB) Forwarding Manager Zone Binding Table
interfaces since interface is not
EOBC (1 Gbps)
known when packet enters. Interface Interface Na Zone Name
show platform hardware qfp active interface if-name Gig0/0/0 --------------------------------------------
… 10 GigabitEthernet0/0/2 inside
11 GigabitEthernet0/0/3 outside
Protocol 1 - ipv4_output
FIA handle - CP:0x108f90e4 DP:0x8080e200
IPV4_OUTPUT_VFR ESP FECP Chassis Manager
IPV4_OUTPUT_INSPECT
IPV4_OUTPUT_THREAT_DEFENSE Drivers Forwarding Manager
… Drivers
Drivers
Linux Kernel
show platform hardware qfp active feature firewall client QFP

statisticsNumber of vrf interfaces with zone: 0 µ µ Crypto
µ µµ BQS
Number of zoned interfaces: 2 µ Assist.
Number of zones: 3
Number of zone pairs with policy: 0 TCAM DRAM DRAM
Classes, Policies and Zone Pair Assignment (I)
ip access-list extended ipacl RP
CPU
deny ip 10.0.0.0 0.0.0.255 172.16.0.0 0.0.0.127
Chassis Manager
… IOS
class-map type inspect match-all ipv4acl Forwarding Manager
Linux Kernel
policy-map type inspect in2out
inspect
class class-default
zone-pair security in2out source inside destination outside
EOBC (1 Gbps)

Drivers
Drivers
Linux Kernel
show platform hardware qfp active feature firewall client statistics
…
Number of vrf interfaces with zone: 0 µ QFP
µ Crypto
Number of zoned interfaces: 2 µ µµ BQS
Assist.
µ
Number of zones: 3
Number of zone pairs with policy: 1
TCAM DRAM DRAM
Classes, Policies and Zone Pair Assignment (II)
CPU
deny ip 10.0.0.0 0.0.0.255 172.16.0.0 0.0.0.127
Chassis Manager
… IOS
sh platform software
Linux Kernel firewall fp active zones
policy-map type inspect in2out Forwarding Manager Firewall Zone Configurations
Zone Name: inside, parameter-map: (null), Obj-id 1
inspect
Zone Name: outside, parameter-map: (null), Obj-id 2
class class-default Zone Name: self, parameter-map: (null), Obj-id 65535
EOBC (1 Gbps)
service-policy type inspect in2out show platform software firewall f0 pairs
Zone-Pair Name Source Zone Destination Zone Obj-id
-----------------------------------------------------------
in2out inside outside 2

show platform hardware qfp active feature firewall zonepair 2
Zonepair name:in2out | id:2
Source zone name:inside | id:1 Drivers Forwarding Manager
Destination zone name:outside | id:2 Drivers
Drivers
Class group name:in2out | id:8719920 Linux Kernel
…
Class name:ipv4acl | id:15565841 QFP
… µ µ Crypto
µ µµ BQS
Assist.
µ
Class name:class-default | id:1593
…
TCAM DRAM DRAM
Classes, Policies and Zone Pair Assignment (III)
CPU
deny ip 10.0.0.0 0.0.0.255 172.16.0.0 0.0.0.127
Chassis Manager
… IOS
Linux Kernel
inspect
class class-default
EOBC (1 Gbps)

Drivers
…
… µ µ Crypto
µ µµ BQS
Assist.
µ
…
TCAM DRAM DRAM
CPU
deny ip 10.0.0.0 0.0.0.255 172.16.0.0 0.0.0.127
Chassis Manager
… IOS
Linux Kernel
class type inspect ipv4acl show platform hardware qfp active classification feature-manager class-group tcam cce 8719920 zone-pair 2 detail
inspect QFP classification class group TCAM
class-group [cce-cg:8719920] (classes: 2, total number of vmrs: 11)
class class-default
key name: 160_01 value size: 160 result size: 16
region id: 1 vmr id: 3 number of vmrs: 11 tcam id: TCAM0
Value: : 0a000000 00000000 00000000 ac100000 00040001
EOBC (1 Gbps)
service-policy type inspect in2out Mask: : ffffff00 00000000 00000000 ffffff80 ffff0009
Result: : 81000000 89435400 00000000 00000000
…

Drivers
…
… µ µ Crypto
µ µµ BQS
Assist.
µ
…
TCAM DRAM DRAM
CPU
deny ip 10.0.0.0 0.0.0.255 172.16.0.0 0.0.0.127
Chassis Manager
… IOS
Linux Kernel
class class-default
Value: : 0a000000 00000000 00000000 ac100000 00040001
EOBC (1 Gbps)
Result: : 81000000 89435400 00000000 00000000
…

Drivers
…
… µ µ Crypto
µ µµ BQS
Assist.
µ
…
TCAM DRAM DRAM
CPU
deny ip 10.0.0.0 0.0.0.255 172.16.0.0 0.0.0.127
Chassis Manager
… IOS
Linux Kernel
class class-default
Value: : 0a000000 00000000 00000000 ac100000 00040001
EOBC (1 Gbps)
Result: : 81000000 89435400 00000000 00000000
…

Drivers
…
… µ µ Crypto
µ µµ BQS
Assist.
µ
…
TCAM DRAM DRAM
The Road to Simplification – Part II
Control Plane Unified Show Commands
154
Simplifying the IPsec show commands
One show command to rule them all
------------------ show platform software ipsec fp active flow identifier 34130 ------------------
…
interface: Virtual-Access1002
Crypto map tag: Virtual-Access1002-head-0, local addr 172.18.0.1 ------------------ show platform hardware qfp active feature ipsec sa 1427 ------------------
…
local ident (addr/mask/prot/port): (172.18.0.1/255.255.255.255/47/0) ------------------ show platform software ipsec fp active encryption-processor context 6502aa4f ------------------
current_peer 17.0.0.26 port 500
…
PERMIT, flags={origin_is_acl,} ------------------ show platform software ipsec fp active flow identifier 34129 ------------------
#pkts encaps: 25227, #pkts encrypt: 25227, #pkts digest: 25227
#pkts decaps: 25237, #pkts decrypt: 25237, #pkts verify: 25237 …
#pkts compressed: 0, #pkts decompressed: 0 ------------------ show platform hardware qfp active feature ipsec sa 1867 ------------------
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0 …
#send errors 0, #recv errors 0 ------------------ show platform software ipsec fp active encryption-processor context 2e02aa4e -----------------
local crypto endpt.: 172.18.0.1, remote crypto endpt.: 17.0.0.26 …
plaintext mtu 1458, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0/2
current outbound spi: 0xA7B61FE5(2813730789)
PFS (Y/N): N, DH group: none
inbound esp sas:

spi: 0xA222F391(2720199569)
Access1002-head-0 show crypto ipsec sa interface virtual-access 1002 platform
IV size: 16 bytes
or
…
show crypto ipsec sa peer 17.0.0.26 platform
outbound esp sas:
spi: 0xA7B61FE5(2813730789)
Access1002-head-0
IV size: 16 bytes
…
Simplifying the ZBF show commands
Three Commands for ZBF under the sky
show policy-firewall config platform
--show platform software firewall FP active bindings--

--show platform software firewall RP active bindings--
--show platform software firewall FP active pairs--
--show platform software firewall RP active pairs--
--show platform software firewall FP active parameter-maps--
--show platform software firewall RP active parameter-maps--
--show platform software firewall FP active zones--
--show platform software firewall RP active zones--
show policy-firewall sessions platform
--show platform hardware qfp active feature firewall datapath scb any any any any any all any --
show policy-firewall stats platform
--show platform software firewall FP active statistics--

--show platform software firewall RP active statistics--
--show platform hardware qfp active feature firewall runtime--
--show platform hardware qfp active feature firewall memory--
--show platform hardware qfp active feature firewall drop--
--show platform hardware qfp active feature firewall client statistics--
An Advanced Example:
IPsec control plane programming
IPsec SA – from IOS to FMAN-FP
show crypto ipsec sa interface virtual-access 1002
RP
interface: Virtual-Access1002 CPU
Crypto map tag: Virtual-Access1002-head-0, local addr 172.18.0.1 Chassis Manager
IOS
local ident (addr/mask/prot/port): (172.18.0.1/255.255.255.255/47/0) Forwarding Manager
current_peer 17.0.0.26 port 500 Linuxplatform
Kernel software ipsec fp active flow identifier <flow_id>
… show
inbound esp sas:
spi: 0x956A1B11(2506758929) SPD id: 1008
… …
conn id: 30214, flow_id: HW:28214, sibling_flags QFP SA handle: 1892
FFFFFFFF80000008, crypto map: Virtual-Access1002-head-0 …
Crypto SA ctx id: 0x000000002e02b9b6
outbound esp sas:
EOBC (1 Gbps)
spi: 0x51E3FC8E(1373895822)
…
conn id: 30213, flow_id: HW:28213, sibling_flags
FFFFFFFF80000008, crypto map: Virtual-Access1002-head-0
…

Drivers
Drivers
Linux Kernel
µ QFP
µ Crypto
µ µµ BQS
Assist.
µ
TCAM DRAM DRAM

IPsec SA – from FMAN-FP to QFP TCAM
RP
IOS
… show
inbound esp sas:
spi: 0x956A1B11(2506758929) SPD id: 1008
… …
outbound esp sas:
EOBC (1 Gbps)
spi: 0x51E3FC8E(1373895822)
…
conn id: 30213, flow_id: HW:28213, sibling_flags
…
show platform hardware qfp active classification feature-manager ESP FECP Chassis Manager
class-group tcam ipsec <SPD-id> global detail
class-group [ipsec-cg:1008] (classes: 1, total number of vmrs: 4) Drivers
key name: 160_03 value size: 160 result size: 16 Drivers
region id: 1 vmr id: 698 number of vmrs: 4 tcam id: TCAM0 Linux Kernel
Value: : ac120001 2f000000 00000000 1100001a 12d70000
Mask: : ffffffff ff000000 00000000 ffffffff ffff0000
Result: : 40000000 905a0400 00000000 00000000 µ QFP
µ Crypto
µ µµ BQS
Assist.
Value: : ac120001 2f000000 00000000 1100001a 12d70000 µ
Mask: : ffffffff ff000000 00000000 ffffffff ffff0000
Result: : 20000000 8d458860 00000000 00000000
TCAM DRAM DRAM
…
IPsec SA – from FMAN-FP to Crypto Engine
RP
IOS
… show
inbound esp sas:
spi: 0x956A1B11(2506758929) SPD id: 1008
… …
outbound esp sas:
EOBC (1 Gbps)
spi: 0x51E3FC8E(1373895822)
… show platform software ipsec fp active
conn id: 30213, flow_id: HW:28213, sibling_flags encryption-processor context 2e02b9b6
… =======Context id: 0x02b249
…
SA word 0: 0x5ae0460fc201aa5
action bits: 0x001f84
show platform hardware qfp active classification feature-manager ESP FECP direction: outbound
Chassis Manager
class-group tcam ipsec <SPD-id> global detail mode: transport
protocol: esp
Drivers Forwarding Manager authentication: SHA-1
key name: 160_03 value size: 160 result size: 16 Drivers confidentiality: AES-128
region id: 1 vmr id: 698 number of vmrs: 4 tcam id: TCAM0 Linux Kernel …
Value: : ac120001 2f000000 00000000 1100001a 12d70000 mfs: 1454
Mask: : ffffffff ff000000 00000000 ffffffff ffff0000 …
QFP sequence number: 306
Result: : 40000000 905a0400 00000000 00000000 µ µµ Crypto
…
µ BQS
µ µ Assist. byte count: 25704
Value: : ac120001 2f000000 00000000 1100001a 12d70000
Mask: : ffffffff ff000000 00000000 ffffffff ffff0000 packet count: 306
Result: : 20000000 8d458860 00000000 00000000
TCAM DRAM DRAM
…
IPsec SA – from FMAN-FP to QFP DRAM
RP
IOS
… show
inbound esp sas:
spi: 0x956A1B11(2506758929) SPD id: 1008
… …
outbound esp sas:
EOBC (1 Gbps)
spi: 0x51E3FC8E(1373895822)
… show platform software ipsec fp active
conn id: 30213, flow_id: HW:28213, sibling_flags encryption-processor context 2e02b9b6
… =======Context id: 0x2e02b9b6
…
SA word 0: 0x5ae0460fc201aa5
action bits: 0x001f84
Chassis Manager
protocol: esp
Value: : ac120001 2f000000 00000000 1100001a 12d70000 mfs: 1454
…
µ BQS
Value: : ac120001 2f000000 00000000 1100001a 12d70000
Result: : 20000000 8d458860 00000000 00000000
TCAM DRAM DRAM
…
RP
IOS
… show
inbound esp sas:
spi: 0x956A1B11(2506758929) SPD id: 1008
… …
show platform hardware qfp active feature ipsec sa <qfp_sa_handle> …
outbound esp sas: QFP ipsec sa Information
EOBC (1 Gbps)
spi: 0x51E3FC8E(1373895822)
… QFP sa id: 3623 show platform software ipsec fp active
conn id: 30213, flow_id: HW:28213, sibling_flags pal sa id: 32085 encryption-processor context 2e02b9b6
FFFFFFFF80000008, crypto map: Virtual-Access1002-head-0 QFP spd id: 3398
… QFP sp id: 1066 ======= Context id: 0x2e02b9b6
QFP spi: 0x51E3FC8E(1373895822) …
crypto ctx: 0x000000002e02b9b6 SA word 0: 0x5ae0460fc201aa5
… action bits: 0x001f84
Chassis Manager
protocol: esp
Value: : ac120001 2f000000 00000000 1100001a 12d70000 mfs: 1454
…
µ BQS
Value: : ac120001 2f000000 00000000 1100001a 12d70000
Result: : 20000000 8d458860 00000000 00000000
TCAM DRAM DRAM
…
RP
IOS
… show
inbound esp sas:
spi: 0x956A1B11(2506758929) SPD id: 1008
… …
EOBC (1 Gbps)
spi: 0x51E3FC8E(1373895822) Indexed by class-group
QFP spi: 0x51E3FC8E(1373895822) …
Chassis Manager
protocol: esp
Value: : ac120001 2f000000 00000000 1100001a 12d70000 mfs: 1454
…
µ BQS
Value: : ac120001 2f000000 00000000 1100001a 12d70000
Result: : 20000000 8d458860 00000000 00000000
TCAM DRAM DRAM
…
RP
IOS
… show
inbound esp sas:
spi: 0x956A1B11(2506758929) SPD id: 1008
… …
EOBC (1 Gbps)
QFP spi: 0x51E3FC8E(1373895822) …
Chassis Manager
protocol: esp
Value: : ac120001 2f000000 00000000 1100001a 12d70000 mfs: 1454
…
µ BQS
Value: : ac120001 2f000000 00000000 1100001a 12d70000
Result: : 20000000 8d458860 00000000 00000000
TCAM DRAM DRAM
…
RP
IOS
… show
inbound esp sas:
spi: 0x956A1B11(2506758929) SPD id: 1008 FMAN-FP remembers
… … everything
EOBC (1 Gbps)
QFP spi: 0x51E3FC8E(1373895822) …
Chassis Manager
protocol: esp
Value: : ac120001 2f000000 00000000 1100001a 12d70000 mfs: 1454
…
µ BQS
Value: : ac120001 2f000000 00000000 1100001a 12d70000
Result: : 20000000 8d458860 00000000 00000000
TCAM DRAM DRAM
…
The Road to Simplification
Part III, Deep Data Plane Debugging
167
IOS 3.10
The Feature Conditional Debugger
TCAM Resource DRAM
DRAM SRAM
Temp Sensor

EEPROM
Pak Match ? PPE2 QFP
MQC Classify
Output ACL NAT
FECP
MQC
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
NAT PBR
Classify
Encaps Output ACL
NAT IP Unicast
NAT
Dispatcher Crypto
Packet Buffer Cond Dbg ?
PBR PPE2 Encaps
Crypto If packet matches condition is on for Thread 3 Crypto
SPI Mux feature AND if packet needs to be
Reset / Pwr Ctrl
traced… feature will log its action step
SA table by step in cpp_cp_f0-0.log !!
DRAM Interconnect Logs will go straight to IOS
syslogs in the future – no
RPs RPs ESP RPs need to rotate logs anymore!
SIPs
Feature Debugging
Step 2 – Define feature(s) to troubleshoot
asr-1k# debug platform condition feature ?

acl ACL feature
alg ALG feature
fw FW feature
ipsec IPSEC feature
nat NAT feature Many features are supported but
nat64 NAT64 feature focus is on NAT, ZBF and FW at the
… moment
acl ACL feature cxsc CXSC feature nat64 NAT64 feature

alg ALG feature dpss DPSS feature nbar NBAR feature
appnav AppNav feature evc EVC feature overlay overlay feature
atm ATM feature fw FW feature qos QOS feature
atom ATOM feature ipsec IPSEC feature subscriber Subscriber feature
bridge-domain Layer2 bridging feature lisp LISP feature tcp TCP feature
cent CENT feature multicast multicast feature vpls VPLS feature
cft CFT feature nat NAT feature
Feature Debugging (cont.)
Step 2 (cont.) – Define feature submodes to be troubleshot
asr-1k# debug platform condition feature fw dataplane submode ?

alg-inspect Debug firewall ALG inspect information
all Debug firewall all information
detail Debug firewall detail
drop Debug firewall drop information
event Debug firewall event information
ha Debug firewall HA information
layer4 Debug firewall Layer 4 information
level Debug level information
policy Debug firewall policy information
asr-1k# debug platform condition feature fw dataplane submode drop layer4 policy
Multiple submodes can be active at once
Conditional Debugger Demonstration
Platform Conditional Debugging
BGL.D.16-ASR1000-1# debug platform condition feature ?
atm ATM feature
atom ATOM feature
bridge-domain Layer2 bridging feature
cft CFT feature
cxsc CXSC feature
evc EVC feature
fw FW feature Debugs get populated in cpp_cp_F0-0.log
ipsec IPSEC feature
nbar NBAR feature
otv OTV feature
subscriber Subscriber feature
vpls VPLS feature
Same match statement as
packet tracer…
BGL.D.16-ASR1000-1#debug platform condition ipv4 172.19.2.1/32 ingress
BGL.D.16-ASR1000-1#debug platform condition feature ipsec dataplane submode cce level info
BGL.D.16-ASR1000-1#debug platform condition start
Tells which feature to
debug
Start and stop debugging
NAT Control plane – config push RPESP
173
Interface Role Assignment
ip nat inside CPU
Chassis Manager
ip nat outside Forwarding Manager
Linux Kernel
EOBC (1 Gbps)

Drivers
Drivers
Linux Kernel
µ QFP
µ Crypto
µ µµ BQS
Assist.
µ
TCAM DRAM DRAM

ip nat inside CPU
Chassis Manager
Linux Kernel
EOBC (1 Gbps)

Drivers
Drivers
Linux Kernel
µ QFP
µ Crypto
µ µµ BQS
Assist.
µ
TCAM DRAM DRAM

ip nat inside CPU
Chassis Manager
Linux Kernel
sh platform software nat f0 interface

Dump NAT interface config
Name: GigabitEthernet0/0/2, Inteface handle: 10
EOBC (1 Gbps)
Domain: DOMAIN_INSIDE, Static-host allowed: No
QFP handle: 9

Domain: DOMAIN_OUTSIDE, Static-host allowed: No
QFP handle: 10

Drivers
Drivers
Linux Kernel
µ QFP
µ Crypto
µ µµ BQS
Assist.
µ
TCAM DRAM DRAM

ip nat inside CPU
Chassis Manager
Linux Kernel

EOBC (1 Gbps)
Domain: DOMAIN_INSIDE, Static-host allowed: No
QFP handle: 9

sh platform hardware qfp active interface if-name gig0/0/2 Domain: DOMAIN_OUTSIDE, Static-host allowed: No
Protocol 0 - ipv4_input QFP handle: 10
FIA handle - CP:0x108f8fac DP:0x8080fd00
… ESP FECP Chassis Manager
IPV4_INPUT_VFR
…
sh platform hardware qfp active interface if-name gig0/0/3 Drivers
Protocol 0 - ipv4_input Drivers
Linux Kernel
…
IPV4_INPUT_VFR
IPV4_NAT_INPUT_FIA µ QFP
µ Crypto
… µ µµ BQS
Assist.
Protocol 1 - ipv4_output µ
IPV4_OUTPUT_VFR
IPV4_NAT_OUTPUT_FIA TCAM DRAM DRAM
ip nat inside CPU
Chassis Manager
Linux Kernel

Features get installed in FIA when NAT roles Name: GigabitEthernet0/0/2, Inteface handle: 10
EOBC (1 Gbps)
are applied. Domain: DOMAIN_INSIDE, Static-host allowed: No
QFP handle: 9

sh platform hardware qfp active interface if-name gig0/0/2 Domain: DOMAIN_OUTSIDE, Static-host allowed: No
Protocol 0 - ipv4_input QFP handle: 10
FIA handle - CP:0x108f8fac DP:0x8080fd00
… ESP FECP Chassis Manager
IPV4_INPUT_VFR
…
sh platform hardware qfp active interface if-name gig0/0/3 Drivers
Protocol 0 - ipv4_input Drivers
Linux Kernel
…
IPV4_INPUT_VFR
IPV4_NAT_INPUT_FIA µ QFP
µ Crypto
… µ µµ BQS
Assist.
Protocol 1 - ipv4_output µ
IPV4_OUTPUT_VFR
IPV4_NAT_OUTPUT_FIA TCAM DRAM DRAM
ip nat inside source list pat interface GigabitEthernet0/0/3 overload RP
CPU
ip access-list extended pat Chassis Manager

permit ip 172.18.25.0 0.0.0.255 any IOS
Forwarding Manager
Linux Kernel
EOBC (1 Gbps)

Drivers
Drivers
Linux Kernel
µ QFP
µ Crypto
µ µµ BQS
Assist.
µ
TCAM DRAM DRAM

CPU

permit ip 172.18.25.0 0.0.0.255 any IOS
Forwarding Manager
show platform software nat f0Linux Kernel

mapping dynamic
Dump NAT dynamic mapping config
Mapping id: 8
Domain: INSIDE, Lookup: LOOKUP_LOCAL
Flags: overload
Interface name: GigabitEthernet0/0/3, IP address: 213.94.72.254
EOBC (1 Gbps)
IF handle: 11, QFP handle: 10
ACL name: pat
CGM class group: INSIDE_SRC_CG, CGM class id: 8
Dynamic PAT: No
Last stats update: 04/01 19:39:42.799
Last refcount value: 0

Drivers
Drivers
Linux Kernel
µ QFP
µ Crypto
µ µµ BQS
Assist.
µ
TCAM DRAM DRAM

CPU

permit ip 172.18.25.0 0.0.0.255 any IOS
Forwarding Manager

mapping dynamic
Mapping id: 8
Flags: overload
EOBC (1 Gbps)
ACL name: pat
CGM class group: INSIDE_SRC_CG, CGM class id: 8
Dynamic PAT: No
sh platform software nat f0 cgm Last refcount value: 0
Dump NAT CGM information
Current CGM tid: 0
Class group: INSIDE_SRC_CG (1), num_classes: 1, last class id: 8
First class created: Yes, first EA CGM attach pending: No Drivers Forwarding Manager
Drivers
Class id: 8, match_type: Match-ACL, acl: pat, refcnt: 1, in CGD: Yes Drivers
Class group: INSIDE_SRC_STATIC_CG (2), num_classes: 0, last class id: 0 Linux Kernel
First class created: No, first EA CGM attach pending: No
Class group: OUTSIDE_SRC_CG (3), num_classes: 0, last class id: 0 µ QFP
First class created: No, first EA CGM attach pending: No µ Crypto
µ µµ BQS
Assist.
Class group: INSIDE_DST_CG (4), num_classes: 0, last class id: 0 µ
… TCAM DRAM DRAM
CPU

permit ip 172.18.25.0 0.0.0.255 any IOS
Forwarding Manager

mapping dynamic
Mapping id: 8
Flags: overload
These are the Class Groups for various NAT types.
EOBC (1 Gbps)
Class groups are “Hardware ACL’s” (sorta) and ACL name: pat
have a TCAM representation CGM class group: INSIDE_SRC_CG, CGM class id: 8
Dynamic PAT: No
sh platform software nat f0 cgm Last refcount value: 0
Current CGM tid: 0
Drivers
µ µµ BQS
Assist.
… TCAM DRAM DRAM
CPU

permit ip 172.18.25.0 0.0.0.255 any IOS
Forwarding Manager
Linux Kernel
EOBC (1 Gbps)

Drivers
Drivers
Linux Kernel
µ QFP
µ Crypto
µ µµ BQS
Assist.
µ
TCAM DRAM DRAM

CPU

permit ip 172.18.25.0 0.0.0.255 any IOS
Forwarding Manager
Linux Kernel
EOBC (1 Gbps)

Drivers
Drivers
Linux Kernel
µ QFP
µ Crypto
µ µµ BQS
Assist.
µ
TCAM DRAM DRAM

CPU

permit ip 172.18.25.0 0.0.0.255 any IOS
Forwarding Manager
Linux Kernel
EOBC (1 Gbps)
sh platform software nat f0 cgm
Current CGM tid: 0
Drivers
µ µµ BQS
Assist.
… TCAM DRAM DRAM
CPU
ip access-list extended pat Chassis

sh platform hardware qfp active classification Manager
class-group-manager class-group client nat 1001
permit ip 172.18.25.0 0.0.0.255 any class-group [nat-cg:1001] IOS
(classes: 1)
clients: Forwarding Manager
fields: ipv4_src:1 ipv4_dst:1 vrf:1 (0:c000:0:00000200)
( class: logical-expression [1001.8] (filters: 3)
Linux Kernel
lexp: LOG-EXP: ,
(1) filter: generic [1001.8.1] (rules: 1)
(10) rule: generic [1001.8.1.3] (permit)
match ipv4_src 172.18.25.0 0.0.0.255
This is the TCAM representation of NAT ACL’s match ipv4_dst any
EOBC (1 Gbps)
match vrf 0
(3) filter: logical operator [1001.8.3] (rules: 0)
logical operation: type: AND, id: 3

Current CGM tid: 0
Drivers
µ µµ BQS
Assist.
… TCAM DRAM DRAM
CPU

(classes: 1)
Linux Kernel
lexp: LOG-EXP: ,
match ipv4_src 172.18.25.0 0.0.0.255
EOBC (1 Gbps)
match vrf 0

Current CGM tid: 0
Drivers
µ µµ BQS
Assist.
… TCAM DRAM DRAM
CPU

(classes: 1)
Linux Kernel
lexp: LOG-EXP: ,
match ipv4_src 172.18.25.0 0.0.0.255
(2) filter: generic [1001.8.2] (rules: 1) +1000
EOBC (1 Gbps)
match vrf 0

Current CGM tid: 0
Drivers
µ µµ BQS
Assist.
… TCAM DRAM DRAM
CPU

(classes: 1)
Linux Kernel
lexp: LOG-EXP: ,
match ipv4_src 172.18.25.0 0.0.0.255
match ipv4_dst any
EOBC (1 Gbps)
match vrf 0
sh platform hardware qfp active classification feature-manager class-group tcam nat 1001 detail
QFP classification class group TCAM
class-group [nat-cg:1001] (classes: 1, total number of vmrs: 2)
key name: NAT_01 value size: 160 result size: 16 Drivers Forwarding Manager
region id: 1 vmr id: 4 number of vmrs: 2 tcam id: TCAM0 Drivers
Drivers
Value: : ac121900 00000000 00000000 00000000 00140000 Linux Kernel
Mask: : ffffff00 00000000 00000000 00000000 ffffffff
Result: : 00000001 00000000 00000000 00000000
µ QFP
µ Crypto
Value: : 00000000 00000000 00000000 00000000 00140000 µ µµ BQS
Assist.
Mask: : 00000000 00000000 00000000 00000000 ffff0000 µ
Result: : 00000000 00000000 00000000 00000000
TCAM DRAM DRAM
NAT Pool Management
RP
CPU
Chassis Manager
IOS
Forwarding Manager
Linux Kernel
EOBC (1 Gbps)

Drivers
Drivers
Linux Kernel
µ QFP
µ Crypto
µ µµ BQS
Assist.
µ
TCAM DRAM DRAM

NAT Pool Management
RP
CPU
Chassis Manager
IOS
Forwarding Manager
Linux Kernel
EOBC (1 Gbps)

Drivers
Drivers
Linux Kernel
µ QFP
µ Crypto
µ µµ BQS
Assist.
µ
TCAM DRAM DRAM

NAT Pool Management
RP f0 pool-stats id 1
show platform software nat CPU
NAT Pool Statistics Chassis Manager

IOS
Forwarding Manager
Pool name test, id 1
Linux Kernel
Assigned Available
Addresses 0 6
UDP Low Ports 0 3072
TCP Low Ports 0 3072
UDP High Ports 0 387066
TCP High Ports 0 387066
(Low ports are less than 1024. High ports are greater than or equal to 1024.)
EOBC (1 Gbps)

Drivers
Drivers
Linux Kernel
µ QFP
µ Crypto
µ µµ BQS
Assist.
µ
TCAM DRAM DRAM

NAT Pool Management

IOS
Forwarding Manager
Linux Kernel
Assigned Available
Addresses 0 6
UDP High Ports 0 387066
TCP High Ports 0 387066
(Low ports are less than 1024. High ports are greater than or equal to 1024.)
EOBC (1 Gbps)
sh platform hardware qfp active feature nat datapath pool
pool_id 1 type 1 addroute 0 mask 0xfffffff0 allocated 0 misses 0 rotary idx
0x0 ahash sz 8 size 6 next 0x0 hash_index 0xd7, hilo ports 0x0 pool mem
0x8de7f860 flags 0x1 pat_wl 0 no_ports_wl 0 num_maps 0
Conf block info Drivers Forwarding Manager
start 172.18.25.10 end 172.18.25.15 flags 0x0 next 0x0 Drivers
Drivers
Free block info Linux Kernel
start 172.18.25.10 end 172.18.25.15 flags 0x0 next 0x0
TCP PAT block info QFP
UDP PAT block info µ µ Crypto
ICMP PAT block info µ µµ BQS
Assist.
µ
GRE PAT block info
Alloced addr info
TCAM DRAM DRAM
NAT Pool Management

IOS
Forwarding Manager
show platform software nat f0 pool Linux Kernel
Assigned Available
Dump NAT pool config Addresses 0 6
ID: 1, Name: test, Type: Generic, Mask: 255.255.255.240 UDP High Ports 0 387066
Acct name: TCP High Ports 0 387066
Address range blocks: 1 (Low ports are less than 1024. High ports are greater than or equal to 1024.)
EOBC (1 Gbps)
Start: 172.18.25.10, End: 172.18.25.15
Last refcount value: 0
sh platform hardware qfp active feature nat datapath pool

pool_id 1 type 1 addroute 0 mask 0xfffffff0 allocated 0 misses 0 rotary idx
0x0 ahash sz 8 size 6 next 0x0 hash_index 0xd7, hilo ports 0x0 pool mem
0x8de7f860 flags 0x1 pat_wl 0 no_ports_wl 0 num_maps 0
Conf block info Drivers Forwarding Manager
start 172.18.25.10 end 172.18.25.15 flags 0x0 next 0x0 Drivers
Drivers
Free block info Linux Kernel
start 172.18.25.10 end 172.18.25.15 flags 0x0 next 0x0
TCP PAT block info QFP
UDP PAT block info µ µ Crypto
ICMP PAT block info µ µµ BQS
Assist.
µ
GRE PAT block info
Alloced addr info
TCAM DRAM DRAM
ESP
FECP
(ANIMATED) NAT In  Out – OUTPUT_NAT Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP
Netflow
... BQS
Input ACL
OUTPUT_NAT
FECP
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify
NAT IP Unicast
PBR
Dispatcher
PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
(ANIMATED) NAT In  Out – OUTPUT_NAT Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
EEPROM
Hit
PPE2 QFP
Netflow Hit
... Static NATBQS
Lookup
Input ACL
OUTPUT_NAT Miss
Door
PPE
DB PPE
FECP 4 5
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
JTAG Ctrl … PPE6 PPE7 PPE8 … PPEN then Route-Map
Bind DB
NAT IP Unicast
PBR
Dispatcher Allocate Addr Drop

PPE2
URD
Output L7Crypto
Translation Alg
Create 3
SA table
NAT
Resource Monitoring
202
IOS 3.14
Unified show memory platform summary
show memory platform summary
• Simplified memory consumption
Total number of processes: 134 • Currently limited to RP memory
Virtual memory : 2822197248
Pages resident : 360197 • This is a Linux level view; no
Major page faults: 1921 process details
Minor page faults: 1290831
Memory (kB)
Physical : 4127744
Total : 3874992
Used : 2231964
Free : 1643028
Active : 1438412
Inactive : 694176
…
IOS 3.14
Unified show process memory platform
Will display control plane memory usage
ASR1006# show proc memory platform ?

detailed Show process detailed memory information
location FRU location Check other CPU’s
sorted Sort based on total memory used by platform processes
By default, show the RP CPU

ASR1006# show proc memory platform
System memory: 16342752K total, 4862204K used, 11480548K free
Lowest: 12794824K
Pid Text Data Stack Dynamic RSS Total Name
---------------------------------------------------------------------------
1 24 516 84 132 516 1820 init
386 635 2924 84 1720 2924 4164 pman.sh
740 29 6620 84 676 6620 17484 psd
761 635 2952 84 1744 2952 4188 pman.sh
1158 4 1660 84 132 1660 4816 rotee
1259 329 15020 84 1156 15020 183996 zlPtpRxTask
1365 4 1656 84 132 1656 4816 rotee
…
Linux memory mapping
Not a sinecure
Source: https://techtalk.intersec.com/2013/07/memory-part-2-understanding-process-memory/
IOS 3.14
View of a Linux process memory
ASR-1006# show proc memory platform detailed process-id 14364

PID: : 14364
Name: : oom.sh
VM Size: : 2932
VM Text : 635
VM Stack : 84
VM Heap : 488
RSS : 1684
PSS : 662
Shared clean : 1044
Shared dirty : 0
Private clean: 0
Private dirty: 640
Referenced : 1684
Swap : 0
IOS 3.14
Process memory mapping
ASR-1006# show proc memory platform detailed process-id 14364 smaps

smaps for process 14364:
address perms offset dev inode pathname
08048000-080e7000 r-xp 00000000 00:01 716 /bin/bash
Size: 636 kB
Rss: 452 kB
Pss: 12 kB
Shared_Clean: 452 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 452 kB
Swap: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
080e7000-080ec000 rw-p 0009f000 00:01 716 /bin/bash
……
IOS 3.14
Unified show process CPU platform
ASR-1006# sh processes cpu platform By default, show the RP CPU

CPU utilization for five seconds: 9%, one minute: 5%, five minutes: 4%
Pid PPid 5Sec 1Min 5Min Status Size Name
--------------------------------------------------------------------------------
1 0 0% 0% 0% S 1863680 init
2 0 0% 0% 0% S 0 kthreadd
3 2 0% 0% 0% S 0 migration/0
4 2 0% 0% 0% S 0 sirq-high/0
5 2 0% 0% 0% S 0 sirq-timer/0
6 2 0% 0% 0% S 0 sirq-net-tx/0
7 2 0% 0% 0% S 0 sirq-net-rx/0
8 2 0% 0% 0% S 0 sirq-block/0
…
Check other CPU’s
ASR-1006# show processes cpu platform location <your CPU of choice>

ASR-1006# show processes cpu platform location fp active
Here: the FECP
A Unix “top” command
IOS 3.14
By default, show the RP CPU

ASR-1006# sh proc cpu platform monitor
bsns-asr1006-6#sh proc cpu platform monitor
top - 10:49:37 up 33 days, 21:18, 0 users, load average: 0.00, 0.00, 0.00
Tasks: 199 total, 1 running, 198 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.4%us, 1.1%sy, 0.0%ni, 96.4%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 16342752k total, 3549108k used, 12793644k free, 245848k buffers
Swap: 0k total, 0k used, 0k free, 1821664k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

31231 root 20 0 5072m 1.0g 340m S 4 6.6 1951:53 linux_iosd-imag
29852 root 20 0 1041m 106m 97m S 2 0.7 778:45.64 fman_rp
1 root 20 0 1820 516 440 S 0 0.0 0:21.46 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0 0.0 0:11.18 migration/0
4 root -50 0 0 0 0 S 0 0.0 0:00.00 sirq-high/0
5 root -50 0 0 0 0 S 0 0.0 49:39.36 sirq-timer/0
/0
13 root -50 0 0 0 0 S 0 0.0 54:40.89 sirq-rcu/0
14 root RT 0 0 0 0 S 0 0.0 0:00.10 watchdog/0
15 root 10 -10 0 0 0 S 0 0.0 0:09.53 desched/0
16 root RT 0 0 0 0 S 0 0.0 0:10.41 migration/1
Monitor other CPU’s
ASR-1006# show processes cpu platform monitor location <the CPU you want>
ASR-1006# show processes cpu platform monitor location fp active
Here: the FECP

Advance IOS XE

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Advance IOS XE

Загружено:

Авторское право:

Доступные форматы

Advanced Troubleshooting of

IOS-XE Key Features

Frederic Detienne, Distinguished Engineer

One IOS-XE Feature Set

2.5-5 5-36 10-40 10-100 40-200

Embedded Service Processor

SIP interconn. SIP interconn. SIP interconn.

SPA SPA SPA

SPA SPA SPA SPA SPA SPA

SPA Interface Processor

SPA SPA SPA SPA SPA SPA

SPA Control Link

Centralized Architecture Midplane

SIP interconn. SIP interconn. SIP interconn.

SPA SPA SPA

SPA SPA SPA SPA SPA SPA

Route Processor Architecture interconn. GE switch

Mgmt Console BITS

ESP10 Block Diagram Crypto

Reset / Pwr Ctrl Packet Buffer

Boot Flash PPE1 PPE2 PPE3 PPE4 PPE5

JTAG Ctrl PPE6 PPE7 PPE8 … PPE40

RPs RPs ESP RPs SIPs

ESP10 Block Diagram (comments) Crypto

JTAG Ctrl PPE6 PPE7 PPE8 … PPE40

Packet Processor Engine

RPs RPs ESP RPs SIPs

• Packet Processing Engine (QFP-PPE)

• Buffer/queue subsystem (QFP-BQS)

Reset / Pwr Ctrl QFP QFP

EEPROM PPE PPE PPE … PPE PPE PPE PPE … PPE

Boot Flash TCAM Resource DRAM

Reset / Pwr Ctrl

SIP10 Block Diagram SPA SPA

ESPs RPs RPs

Reset / Pwr Ctrl Interconnect

Ingress buffers Egress buffers

ESI, 11.2 Gbps GE, 1Gbps

SIP10 Block Diagram (comments) SPA SPA

ESPs RPs RPs

Reset / Pwr Ctrl Interconnect

DDR3 Control Plane 4xPCIe Data Plane 4xSGMI

Ctrl SVC1 PPE1 PPE2 PPE3 PPE4 PPE5

DDR3 Control Plane 4xPCIe Data Plane 4xSGMI

Front panel PoE

Rangeley CPU mSATA

USB Host Ports PCIe Switch

PPE1 PPE2 PPE3 PPE4 PPE5 2 service cores

USB Host Ports PCIe Switch

Generic ESP Block Diagram Crypto

Reset / Pwr Ctrl Packet Buffer Part Len / BW

Boot Flash PPE1 PPE2 PPE3 PPE4 PPE5

RPs RPs ESP RPs SIPs

ESI (10-40 Gbps)

ESI (10-40 Gbps)

ESI (10-40 Gbps)

ESP FECP Chassis Manager

ESI (10-40 Gbps)

ESP FECP Chassis Manager

SPA SPA SPA

Linux Kernel Manages timing circuitry on RP

Linux Kernel • Propagates control plane ops. to ESP

Linux Kernel – e.g. statistics

SPA SPA SPA