BRKRST 3143

Troubleshooting Cisco
Catalyst 6500 Series

Switches
BRKRST-3143
BRKRST-3143
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 2
© 2006, Cisco Systems, Inc. All rights reserved. 1

14664_05_2008_c2.scr
Agenda
Sup720 Architecture (A Quick Look)

Layer 2 and Layer 3 Unicast Troubleshooting
Multicast Troubleshooting
Virtual Switch System Troubleshooting
BRKRST-3143
Agenda

BRKRST-3143

14664_05_2008_c2.scr
Sup720 Architecture
18 - 20 Gbps Conns EARL-DBUS
EARL-RBUS
Multicast Expansion
Fabric ASIC Table (MET)
Switch and
Fabric Replication
Engine
PFC
L2 L2 CAM
Engine
CPU Card
LC-RBUS
LC-DBUS
NetFlow Table
RP
Controller FIB TCAM
CPU
EOBC
Port L3 Adj CAM

ASIC Engine
SP QoS TCAM
Controller
CPU ACLTCAM
Supervisor Engine 720

Port 1 Port 2
BRKRST-3143
Agenda

BRKRST-3143

14664_05_2008_c2.scr
Troubleshooting Unicast Forwarding
Typical Problems?
(Some) packets don’t get through (drops, incorrect
forwarding)
What platform specific counters and tables to check ?
Unwanted flooding
Do we learn MAC, are L2 tables in sync ?
High CPU due to SW path forwarding

How do we find out what packets hit the CPU ?
BRKRST-3143

Unicast L2 and L3 Traffic: What to Check ?
Test topology network diagram
Quick sanity checklist (Layer 2/Layer 3)
Detailed L2 packet flow troubleshooting
Which counters and tables to look at

Some useful troubleshooting tools
BRKRST-3143

14664_05_2008_c2.scr
Test Topology Network Diagram
R1 DUT R2
DUT is the Device Under Test we are troubleshooting

DUT is a 6509 with Supervisor 720
R1/R2 are neighboring devices
Connections are respectively a 5 x 1 Gigabit Ethernet
links and 2 x 1 Ten Gigabit Ethernet port channel
After normal network troubleshooting, conclusion
is that DUT has a problem: (some) unicast packets
don’t go through ….. Where do we go from there ??
BRKRST-3143
Quick Sanity Check

Quickly Understand Situation/Topology/Traffic Flow
If no up-to-date topology diagram, confirm the connections
between DUT and (relevant) neighbors … “show cdp neighbor”
can be a good tool
Check for the obvious:
Are all modules on line and OK, are links up ?
What does “show proc cpu” say ?
Any log messages ?
Any recent changes in configuration or topology ?
Can we ping the neighboring hops (L3) ?
Do we learn (neighbor) MAC addresses (L2), routes (L3) ?
If nothing obvious, identify traffic flows that are impacted and
*should* go through the DUT
Verify the path for impacted flow through DUT
BRKRST-3143

14664_05_2008_c2.scr
L2 Unicast Traffic Network Configuration
Po1 Po2
Gig5/2 Gig7/2 Po1 Po2
Gig8/2 Gig7/3
Ten8/1
Gig8/1 Gig7/4
Gig8/3 Gig7/5 Ten8/3 Ten8/3
Gig8/4 Gig7/6
Host1 R1 DUT R2 Host2

7.0.1.1 Vlan700 7.0.1.2
Find MAC address of Host 1
host1#sh ip arp 7.0.1.1 (using router as host; depending on
Protocol Address Age (min) Hardware Addr Type Interface
host OS, you can use e.g. arp -a)
Internet 7.0.1.1 - 000b.fca2.fe0a ARPA Vlan700
host1#
Find MAC address of
host2#sh ip arp 7.0.1.2 Host 2
Internet 7.0.1.2 - 0011.bced.e400 ARPA GigabitEthernet2/3
host2#
BRKRST-3143
Sanity Check for L2 Unicast Traffic

Network Path Verification: Result
Po1 Po2
Gig8/2 Gig7/3
Ten8/1 Ten8/1
Gig8/1 Gig7/4
Gig8/4 Gig7/6
7.0.1.1 Vlan700 7.0.1.2
Po1 Po2
Gig8/2 Gig7/3
Ten8/1 Ten8/1
Gig8/1 Gig7/4
Host1
Gig8/4 Gig7/6 Host2
R1 DUT R2
7.0.1.1 Vlan700 7.0.1.2
Each direction can use different links in the bundles !!
BRKRST-3143

14664_05_2008_c2.scr
Network Path Verification: mac Address Table Check
DUT#show mac-address-table address 000b.fca2.fe0a vlan 700 all
Check MAC addresses are
Legend: * - primary entry
present in all Forwarding
age - seconds since last seen
Engines in the system
n/a - not available (PFC/DFC) … if not, possibly
vlan mac address type learn age ports flooding !!
------+----------------+--------+-----+----------+--------------------------
Module 1:
700 000b.fca2.fe0a dynamic Yes 170 Po2
Active Supervisor: Primary entry: the MAC is
700 000b.fca2.fe0a dynamic Yes 170 Po2 learned on an interface tied to the
Standby Supervisor: L2 Forwarding Engine (module 7
700 000b.fca2.fe0a dynamic Yes 170 Po2 is ingress line card for packets
Module 7[FE 1]: coming from this MAC); if ingress
line card is CFC (doesn’t have
* 700 000b.fca2.fe0a dynamic Yes 50 Po2
local FE), ingress FE is the PFC
Module 7[FE 2]:
of the active supervisor
* 700 000b.fca2.fe0a dynamic Yes 170 Po2
Module 8[FE 1]:
700 000b.fca2.fe0a dynamic Yes 170 Po2
Module 8[FE 2]: Which physical link in the
700 000b.fca2.fe0a dynamic Yes 170 Po2 port channel really receives
the flow ?
DUT#sh interface po2 | i Members
Members in this channel: Gi7/2 Gi7/3 Gi7/4 Gi7/5 Gi7/6
Repeat this for MAC address of Host2
BRKRST-3143

Network Path Verification: mac Address Table Check
DUT#show mac-address-table address 0011.bced.e400 vlan 700 all
…
vlan mac address type learn age ports
------+----------------+--------+-----+----------+--------------------------
Active Supervisor:
700 0011.bced.e400 dynamic Yes 265 Po1
Standby Supervisor:
Module 7[FE 1]:
Module 7[FE 2]:
Module 8[FE 1]:
* 700 0011.bced.e400 dynamic Yes 230 Po1
Module 8[FE 2]:
DUT#sh interface po1 | i Members
Members in this channel: Te8/1 Te8/3
BRKRST-3143

14664_05_2008_c2.scr
Network Path Verification: Which EtherChannel Links?
Po1 Po2
Gig8/2 Gig7/3
Ten8/1 Ten8/1
Gig8/1 Gig7/4
Gig8/4 Gig7/6

7.0.1.1 Vlan700 7.0.1.2
R1#show etherchannel load-balance Check load balancing
configuration used; default
EtherChannel Load-Balancing Configuration: is src-dst-ip Check load balancing
dst-ip configuration used
mpls label-ip
EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Destination MAC address Mode is “dst-ip”, so only destination IP as argument. As of
IPv4: Destination IP address 12.2(33)SXH, new CLI added to RP: show etherchannel load-
IPv6: Destination IP address balance hash-result … (same arguments); one can use remote
login switch (instead of remote command);
MPLS: Label or IP
R1#remote command switch test etherchannel load-balance interface po1 ip 7.0.1.2
Computed RBH: 0x1
Link selected is Gi8/1 in Po1 of R1 for traffic to 7.0.1.2 leaving R1
Would select Gi8/1 of Po1
Repeat same steps for finding links used in Po2, Po1 on DUT and Po2 on R2 in
both directions (to 7.0.1.2 and to 7.0.1.1)
BRKRST-3143

Network Path Verification: Which EtherChannel Links?
DUT#show etherchannel load-balance
EtherChannel Load-Balancing Configuration:
dst-ip
mpls label-ip
EtherChannel Load-Balancing Addresses Used Per-Protocol:

Non-IP: Destination MAC address
IPv4: Destination IP address
IPv6: Destination IP address
MPLS: Label or IP
DUT#remote command switch test etherchannel load-balance int po1 ip 7.0.1.2
Computed RBH: 0x1
Would select Te8/3 of Po1

Computed RBH: 0x2
R2#show etherchannel load-balance

…
R2#remote command switch test etherchannel load-balance int po2 ip 7.0.1.1
Computed RBH: 0x2
Would select Te8/1 of Po2
BRKRST-3143

14664_05_2008_c2.scr
Po1 Po2
Gig8/2 Gig7/3
Ten8/1 Ten8/1
Gig8/1 Gig7/4
Gig8/4 Gig7/6
7.0.1.1 Vlan700 7.0.1.2
Po1 Po2
Gig8/2 Gig7/3
Ten8/1 Ten8/1
Gig8/1 Gig7/4
Host1
Gig8/4 Gig7/6 Host2
R1 DUT R2
7.0.1.1 Vlan700 7.0.1.2
Each direction can use different links in the bundles !!
BRKRST-3143
L3 Unicast Traffic Network Configuration

Po1 Po2
Gig5/2 Gig7/2
Gig8/2 Gig7/3
Gig8/3 Gig7/5
Ten8/3 Ten8/3
Gig8/4 Gig7/6
8.0.1.1 9.0.1.2

Connections are respectively a 5 x 1 Gigabit L2 Ethernet Port
Channel carrying VLAN’s 701 to 705 and 2 x 1 L3 Ten Gigabit
links
Running equal cost multi path routing with respectively 5 and 2
equal cost paths
DUT has a problem: (some) unicast packets don’t go through …..
BRKRST-3143

14664_05_2008_c2.scr
Network Path Verification: Which L3 Next Hop/L2 Link ?
R1#sh ip route 9.0.1.0 | i via Which next hop will be actually used by the traffic
Known via "eigrp 700", distance 90, metric 3328, type internal flow in case of Equal Cost Multi Path routing?
Redistributing via eigrp 700
* 7.2.1.2, from 7.2.1.2, 00:21:58 ago, via Vlan702
7.5.1.2, from 7.5.1.2, 00:21:58 ago, via Vlan705 Check next hop used for
7.4.1.2, from 7.4.1.2, 00:21:58 ago, via Vlan704 SW based CEF (SW
7.3.1.2, from 7.3.1.2, 00:21:58 ago, via Vlan703 forwarding data path) for
7.1.1.2, from 7.1.1.2, 00:21:58 ago, via Vlan701 flows 8.0.1.1 -> 9.0.1.2
R1#sh ip cef exact-route 8.0.1.1 9.0.1.2
8.0.1.1 -> 9.0.1.2 : Vlan701 (next hop 7.1.1.2) Check next hop used for HW based CEF (SW
R1#show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 forwarding data path) for flows 8.0.1.1 -> 9.0.1.2;
source and destination port 0 as test flow was
Interface: Vl705, Next Hop: 7.5.1.2, Vlan: 705, Destination Mac: 0050.f0f8.7400
ICMP echo request/replies in example
R1#remote command switch test etherchannel load-balance int po1 ip 9.0.1.2 Check which link between R1 and
Computed RBH: 0x7 DUT in 5 port etherchannel, based on
Would select Gi8/2 of Po1 etherchannel loadbalance
Traffic flow 8.0.1.1 -> 9.0.1.2 leaves R1 on Gi8/2 link, in vlan 705, to next
hop 7.5.1.2 for HW CEF switched packets; for SW CEF switched packets,
same link, but in vlan 701, to next hop 7.1.1.2
Repeat the same steps for finding L3 next hops and links on DUT, and R2,
in both directions
BRKRST-3143

Network Path Verification: Which L3 Next Hop?
DUT#sh ip route 9.0.1.0 | i via Look at ingress line card L3
Known via "eigrp 700", distance 90, metric 3072, type internal tables: all of the L3 tables should
be in sync, but the lookup happens
Redistributing via eigrp 700 at the ingress DFC/PFC. In case the
* 7.7.1.2, from 7.7.1.2, 00:07:33 ago, via TenGigabitEthernet8/3
ingress module doesn’t have DFC,
ingress forwarding engine is the
7.6.1.2, from 7.6.1.2, 00:07:33 ago, via TenGigabitEthernet8/1
PFC of the active supervisor
DUT#sh ip cef exact-route 8.0.1.1 9.0.1.2
8.0.1.1 -> 9.0.1.2 => IP adj out of TenGigabitEthernet8/1, addr 7.6.1.2
DUT#show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 mod 7
Interface: Te8/3, Next Hop: 7.7.1.2, Vlan: 1090, Destination Mac: 000f.f8e4.d000
DUT#sh vlan internal usage | i 1090 Next hop for L3 interface is linked to
1090 TenGigabitEthernet8/3 internal vlan; check internal VLAN
matches physical interface
Traffic flow 8.0.1.1 -> 9.0.1.2 leaves DUT on Ten8/3 link

Repeat the same steps for DUT and R2 (both directions)
BRKRST-3143

14664_05_2008_c2.scr
Network Path Verification: Which L3 Next Hop/L2 Link?
DUT#sh ip route 8.0.1.0 | i via
Known via "eigrp 700", distance 90, metric 3072, type internal
* 7.5.1.1, from 7.5.1.1, 00:15:49 ago, via Vlan705
7.4.1.1, from 7.4.1.1, 00:15:49 ago, via Vlan704
7.3.1.1, from 7.3.1.1, 00:15:49 ago, via Vlan703
7.2.1.1, from 7.2.1.1, 00:15:49 ago, via Vlan702
7.1.1.1, from 7.1.1.1, 00:15:49 ago, via Vlan701

9.0.1.2 -> 8.0.1.1 => IP adj out of Vlan701, addr 7.1.1.1
DUT#show mls cef exact-route 9.0.1.2 0 8.0.1.1 0 mod 8
Interface: Vl705, Next Hop: 7.5.1.1, Vlan: 705, Destination Mac: 0011.bc75.9c00

Computed RBH: 0x4
Traffic flow 9.0.1.2 -> 8.0.1.1 leaves DUT on Gi7/6 link, in vlan 705
BRKRST-3143

Network Path Verification: Which L3 Next Hop?
R2#sh ip route 8.0.1.0 | i via
* 7.7.1.1, from 7.7.1.1, 00:32:01 ago, via TenGigabitEthernet8/3
7.6.1.1, from 7.6.1.1, 00:32:01 ago, via TenGigabitEthernet8/1
R2#sh mls cef exact-route 9.0.1.2 0 8.0.1.1 0
Interface: Te8/3, Next Hop: 7.7.1.1, Vlan: 4043, Destination Mac: 0050.f0f8.7400
Traffic flow 9.0.1.2 -> 8.0.1.1 leaves R2 on Ten8/3 link
BRKRST-3143

14664_05_2008_c2.scr
Po1 Po2
Gig5/2 Gig7/2
Gig8/2 Gig7/3
Gig8/3 Gig7/5
Ten8/3 Ten8/3
Gig8/4 Gig7/6

8.0.1.1 9.0.1.2
Po1 Po2
Gig5/2 Gig7/2
Gig8/2 Gig7/3
Gig8/3 Gig7/5
Ten8/3 Ten8/3
Gig8/4 Gig7/6

8.0.1.1 9.0.1.2
Each direction can use different links !!
BRKRST-3143
What Did We Get from Path Verification?
The physical links the specific traffic flow should come

in and leave the DUT, as well as the exact L3 next hops
Caveat:
Flapping links in port channel, can change the bundle hash
mapping, and change physical path of traffic
Clearing routes can as well change the order in which the L3
adjacencies get re-programmed, and in case of ECMP hence
change the physical path of the traffic
=> any of these happen, you need to re-verify the path
BRKRST-3143

14664_05_2008_c2.scr

BRKRST-3143
Detailed L2 Packet Flow Troubleshooting

“Verify the Traffic Path in the Switch”
Ten8/3 Ten8/1
Port Port 4 x 1x10GE port asic
EOBC
Port Port
ASIC ASIC ASIC ASIC
WS-X6704
Fabric Layer 2 Layer 2 Fabric Module 8
Interface & Interface &
MET Engine Engine MET
Replication Replication
Engine L3/4 Engine
DFC3 Engine
Switch Fabric
WS-X6748
Engine L3/4 Engine
Port Port DFC3 Engine Port Port
ASIC ASIC ASIC ASIC
4 x 12xGE port asic
Gig7/4
Identify path in the switch

Check counters
Verifying L2 forwarding tables (HW/SW)
BRKRST-3143

14664_05_2008_c2.scr
Identify the “Traffic Path in the Switch”: Which Fabric Channels?
Ten8/3 Ten8/1
DUT#sh fabric fpoe interface Gi 7/4 Mod 8

L2 L2
fpoe for GigabitEthernet7/4 is 15
DUT#sh fabric fpoe interface ten 8/1 L3/4
Slot 8, channel 0 Slot 8, channel 1
fpoe for TenGigabitEthernet8/1 is 16
DUT#sh fabric fpoe interface ten 8/3 Slot 7, channel 1 ? Fabric
Mod 7
fpoe for TenGigabitEthernet8/3 is 7 L2 L2
DUT#sh fabric fpoe map
slot channel fpoe
L3/4
… … … Gig7/4
7 0 6
7 1 15 For each in/egress interface identified in the path
verification, find Fabric Port Of Exit (FPOE) the
8 0 7 interface maps to
8 1 16
… … … Find what fabric channel the relevant FPOE’s map
to, and from previous command, what fabric channel
maps to what interface
Gig7/4 maps to slot 7, fabric channel 1, Ten8/1 maps to

slot 8, channel 1, Ten 8/3 to slot 8, channel 0
BRKRST-3143

Counters and L2 Tables Overview
EOBC
Ten8/3 Ten8/1
Port counters Port Port Port counters Port Port 4 x 1x10GE port asic
ASIC ASIC ASIC ASIC
WS-X6704
L2 Engine counters
Fabric Layer& 2Tables
Layer 2 Fabric Module 8
Engine L3/4 Engine
DFC3 Engine
Fabric counters Channel0 Channel1 Fabric counters
Switch Fabric
Fabric counters
Channel1 WS-X6748
Fabric L2 Engine counters Fabric
Layer& 2 Layer 2
Tables Module 7
Engine L3/4 Engine
Port counters Port Port DFC3 Engine Port Port
ASIC ASIC ASIC ASIC
4 x 12xGE port asic
Gig7/4
BRKRST-3143

14664_05_2008_c2.scr
Verify L2 Counters: Interface Counters (Port asic)
DUT#clear counters Cleared interface counters (port level) just for illustration
DUT#clear vlan 700 counters

Cleared L2 Forwarding Engine VLAN counters just for illustration
Did a ping (2000 packets/100 bytes per packet) from 7.0.1.1 -> 7.0.1.2,
verify interface counters relevant to the path did move sufficiently !!
DUT#sh int gi 7/4 count
Port InOctets InUcastPkts InMcastPkts InBcastPkts
Gi7/4 249784 2000 8 40
Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
Gi7/4 245614 2000 6 0
DUT#sh int ten 8/3 count
Ten8/3 4 x 10GE port asic Ten8/1
Port InOctets InUcastPkts InMcastPkts InBcastPkts
Te8/3 10590 18 28 0 ? ? Mod 8
Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
L2 L2
Te8/3 246449 2000 10 0
DUT#sh int ten 8/1 count
L3/4
Port InOctets InUcastPkts InMcastPkts InBcastPkts Fabric
Te8/1 273441 2032 174 0
Mod 7
Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts L2 L2
Te8/1 2890 0 11 0
L3/4
DUT#
Gig7/4
?
4 x 12xGE port asic
BRKRST-3143

Verify L2 Counters: L2 Forwarding Engine Vlan Count
DUT#sh vlan id 700 counters
* Multicast counters include broadcast packets
Vlan Id : 700
L2 Unicast Packets : 4000
L2 Unicast Octets : 472000 VLAN is bidirectional, so
counts both directions of
L3 Input Unicast Packets : 0
the flow (7.0.1.1 <-> 7.0.1.2
L3 Input Unicast Octets : 0
L3 Output Unicast Packets : 0
L3 Output Unicast Octets : 0 Ten8/3 4 x 10GE port asic Ten8/1
L3 Output Multicast Packets : 0
L3 Output Multicast Octets : 0 Mod 8
L3 Input Multicast Packets : 0 ? ?
L2 L2
L3 Input Multicast Octets : 0 L3/4
L2 Multicast Packets : 0
L2 Multicast Octets : 0 Fabric
Mod 7
DUT#sh interface <interface> counter errors
DUT#sh counters interface <interface>
?L2 L2?
L3/4
Gig7/4
4 x 12xGE port asic
SNMP like interface counters

Interface level errors (e.g. OutDiscards …)
BRKRST-3143

14664_05_2008_c2.scr
Verify L2 Forwarding Engines Counters
Do the counters move ? Check all
relevant forwarding engines/modules.
DUT#remote command mod 7 show platform hardware earl statistics
Superman 0 Forwarding statistics: L2 Engine 0 on module 7
Forwarded Frames = 0x0000000016E3E0E8 (384033000)
Frames fwd'ed to Tycho = 0x000000000BA62E47 (195440199) Amount of frames forwarded by L2 Engine
L3 results rcvd = 0x000000000BA62E47 (195440199)
. . .
Amount of frames that required L3 lookup
Amount of L3 lookup results received from L3 Forwarding Engine
. . . Increases per new learn (source mac lookup miss) Increases per flooded packet (destination mac lookup miss)
Src Mac misses = 0x000000000425D50C (69588236)
Dst Mac misses = 0x0000000005340140 (87294272)
line full encountered during New l = 0x0000000000000000 (0) Unable to learn because all hash buckets full
. . .
correctable errors in bank 0 = 0x0000000000000000 (0)
Correctable ECC errors upon reading entry in L2 table
uncorrectable errors in bank 0 = 0x0000000000000000 (0)
correctable errors in bank 1 = 0x0000000000000000 (0)
uncorrectable errors in bank 1 = 0x0000000000000000 (0) Uncorrectable ECC errors upon reading entry in L2 table .. HW
DBus Header Checksum errors = 0x0000000000000000 (0)
address of the line full = 0x00000204
L2 Engine sees bad CRC DBUS header
address of the last error in Bank0 = 0x00004022
address of the last error in Bank1 = 0x00002040
Superman 1 Forwarding statistics:
L2 Engine 1 on module 7
BRKRST-3143

Verify L2 Counters: Switching Fabric Utilization
DUT#sh fabric status 7
Check status of fabric channels is OK Ten8/3 4 x 10GE port asic Ten8/1
slot channel speed module fabric hotStandby Standby Standby
status status support module fabric
Mod 8
7 0 20G OK OK Y(not-hot)
L2 L2
L3/4
DUT#sh fabric status 8 ? ?
slot channel speed module
status
fabric
status
hotStandby
support
Standby
module
Standby
fabric ? ? Fabric
Mod 7
8 0 20G OK OK Y(not-hot) L2 L2
L3/4
DUT#sh fabric utilization detail
Fabric utilization: Ingress Egress Gig7/4
4 x 12xGE port asic
Module Chanl Speed rate peak rate peak
Check utilization (current and last peak
1 0 20G 0% 14% @18:34 17Dec07 0% 13% @14:42 03Jan08
value) for relevant fabric channels … did any
4 0 8G 0% 86% @23:20 17Dec07 0% 100% @10:58 21Dec07
peak coincide with moment of drops ?
5 0 20G 0% 7% @00:43 18Dec07 0% 27% @10:42 21Dec07
6 0 8G 0% 9% @15:23 17Dec07 0% 16% @16:58 17Dec07
Gig7/4
7 0 20G 0% 1% @04:54 22Feb08 0% 1% @02:34 22Feb08
7 1 20G 0% 1% @15:47 21Feb08 0% 6% @18:35 20Mar08
Gig8/3
8 0 20G 0% 5% @13:12 21Mar08 0% 6% @16:58 17Dec07
8 1 20G 0% 43% @15:11 26Dec07 0% 29% @13:44 21Dec07
Gig8/1
BRKRST-3143

14664_05_2008_c2.scr
Verify L2 Counters: Relevant Fabric Channels
line card fabric ASIC reports bad packets:
card inserted properly ? A few incrementing unable to send packets from fabric to line card:
‘rxErrors', which is not correlated to any Check traffic levels, line card OK ?
network events, is OK & acceptable
DUT#sh fabric channel-counters 7
slot channel rxErrors txErrors txDrops lbusDrops fabric interface unable to send packets
7 0 0 0 0 0 from local bus to fabric (Supervisor and
65XX modules only – not 67XX, 67XX
7 1 0 0 0 0 will report Overruns in “show interface”):
DUT#sh fabric errors 7 check traffic levels, congestion ?
Module errors:
slot channel crc hbeat sync DDR sync
7 0 0 0 0 0
7 1 0 0 0 0 fabric serial link bit errors (8 serial links in each
Fabric errors: fabric channel), reported as soon as 2 fabric
serial link interrupts within 100ms; can result in
slot channel sync buffer timeout
rxErrors/txErrors; check card inserted OK ?
7 0 0 0 0
7 1 0 0 0
DUT#sh fabric channel-counters 8 …
DUT#sh fabric errors 8 … fabric ASIC unable to send traffic to the fabric
enabled module for last +3 seconds
BRKRST-3143

Verify L2 Tables Check all L2 Forwarding Engines MAC
DUT#show mac-address-table address 0011.bced.e400 vlan 700 all tables are in sync (if not … possibly flooding)
Legend: * - primary entry and correct
age - seconds since last seen n/a - not available

------+----------------+--------+-----+----------+--------------------------
Module 1: Ten8/3 4 x 10GE port asic Ten8/1
Active Supervisor:
Mod 8
700 0011.bced.e400
Standby Supervisor:
dynamic Yes 40 Po1
? ?
L2 L2
L3/4
Module 7[FE 1]: Fabric
700 0011.bced.e400 dynamic Yes 95 Po1 Mod 7
Module 7[FE 2]: ?L2 L2?
L3/4
Module 8[FE 1]:
Gig7/4
* 700 0011.bced.e400 dynamic Yes 30 Po1 4 x 12xGE port asic
Module 8[FE 2]: <-
2 Layer 2 Engines on module 8

Primary entry: indicates ingress module for
this MAC address is module 8
BRKRST-3143

14664_05_2008_c2.scr
Verify L2 Tables
DUT#show mac-address-table learning vlan 700 Is learning on for the VLAN in all L2 Engines
VLAN Mod1 Mod5 Mod6 Mod7 Mod8 … if not, possible flooding
---- ------------------------------
700 yes yes yes yes yes
If flooding and MAC address tables not in sync across
DUT#show mac-address-table synchronize statistics DFC/PFC’s: check if extra EOBC L2 table SW sync feature
MAC Entry Out-of-band Synchronization Feature Statistics: (complements HW L2 synchronization) is on, if not try turning
on: “mac-address-table synchronize” (sup720 only)
---------------------------------------------------------
Module [7]
-----------
Module Status:
Statistics collected from module : 7
Number of L2 asics in this module : 2 Off by default, except on
WS-X6708 it is on by
Global Status: default
Status of feature enabled on the switch : on
Default activity time : 160
Default value is 160 seconds; normal aging
Configured current activity time : 160 timer should be at least 3x activity interval …
Statistics from ASIC 0 when last activity timer expired: so with default of 160 seconds, change aging
. . . timer to 480 seconds or more
Number of active entries read : 41295

Number of entries ignored with update to age byte : 16251
Number of entries updated with age byte : 20217
Number of entries created new : 227 Number of entries that were synced by
SW sync feature
Statistics from ASIC 1 when last activity timer expired: …
BRKRST-3143


BRKRST-3143

14664_05_2008_c2.scr
L3 Unicast Traffic Network Refresh
Po1 Po2
Gig5/2 Gig7/2
Gig8/2 Gig7/3
Ten8/1 Ten8/1
Gig8/1 Gig7/4
Gig8/4 Gig7/6
R1 DUT R2
8.0.1.1 9.0.1.2

Connections are respectively a 5 x 1 Gigabit Ethernet Port
Channel and 2 x 1 Ten Gigabit links,
Running equal cost multi path routing with respectively 5 (Vlans
701 – 705) and 2 (L3 Te8/1 and Ten8/3) equal cost paths
BRKRST-3143

“In the Switch Path”, L3 Counters and Tables
Ten8/3
EOBC
Port Port Port counters Port Port 4 x 1x10GE port asic
ASIC ASIC ASIC ASIC
L2 Engine counters
& tables
WS-X6704
Engine L3/4 Engine
DFC3 Engine
L3 Engine counters
Channel1 & Tables Channel0
Fabric counters
Switch Fabric
Fabric counters
Channel1 L2 Engine counters
WS-X6748
Fabric Layer& 2tables
Layer 2 Fabric Module 7
Engine L3/4 Engine
Port counters Port Port DFC3 Engine Port Port
L3 Engine counters
ASIC ASIC & Tables ASIC ASIC
4 x 12xGE port asic
Gig7/3 Gig7/6
Similar to L2 check port counters, relevant fabric channels, L2 Engine counters and
tables;
Additionally: check L3 Engine counters and tables
BRKRST-3143

14664_05_2008_c2.scr
L3 Engine in Detail: Counters and Tables FIB contains IPv4/IPv6
ADJ prefixes and MPLS
QoS TCAM entries
contains QoS contains
ACL entries rewrite info
Hardware for ACL

NetFlow table TCAM counters
for stats and
features
ACE
4 x 10GE port asic Counter
Ten8/3
QoS Adj FIB ACL
NetFlow TCAM TCAM TCAM ACL TCAM
TCAM contains
Mod 8 security and
L2 L2 feature ACL
entries
?
L3/4 L3/4 Engine
Fabric
Mod 7
L2 L2 PFC3/DFC3
L3/4
? L2 Engine L2 CAM
Gig7/3 contains
4 x 12xGE port asic L2 CAM (64K) MAC entries
Gig7/6
L3 forwarding tables get programmed by SW: copy of SW forwarding tables in HW

EOBC is used for communication between modules and RP, and program L3 tables
BRKRST-3143

Verify L2 Tables for Router mac Address
DUT#sh int vlan 702 | i address
Hardware is EtherSVI, address is 0050.f0f8.7400 (bia 0050.f0f8.7400)
Check all L2 Forwarding Engines MAC
Internet address is 7.2.1.2/24 tables have the routed interface mac
DUT#show mac-address-table address 0050.f0f8.7400 vlan 702 all programmed as a router MAC; if not,
possibly no HW switching
Legend: * - primary entry
age - seconds since last seen
n/a - not available 4 x 10GE port asic Ten8/3
------+----------------+--------+-----+----------+--------------------------
Mod 8
Module 1:
* 702 0050.f0f8.7400 static No - Router
? ?
L2 L2
L3/4
Active Supervisor:
* 702 0050.f0f8.7400 static No - Router Fabric
Standby Supervisor: Mod 7
* 702 0050.f0f8.7400 static No - Router ?L2 L2?
Module 7[FE 1]:
L3/4
Gig7/3
Module 7[FE 2]: 4 x 12xGE port asic
Gig7/6
Module 8[FE 1]:
* 702 0050.f0f8.7400 static No - Router Tagged as router MAC, so packets going to
Module 8[FE 2]: that mac address will be L3 HW switched
based on PFC3/DFC3 HW content
BRKRST-3143

14664_05_2008_c2.scr
L3 FIB Table Programming Flow
show ip arp
<next hop ip
address>
show ip route
<ip address> Verify Layer 3 Verify Layer 2 rewrite
IOS Routing Table (RP) IOS ARP Cache Table (RP) show ip cef
adjacency
show ip cef < ip <interface>
address> <next hop ip
IOS FIB Table (RP) IOS Adjacency Table (RP) address>
IOS FIB Table (SP/DFC) IOS Adjacency Table (SP/DFC)

remote
command MLS FIB Table (SP/DFC) MLS Adjacency Table (SP/DFC)
module <mod>
show ip cef < ip remote command
address>… module <mod> show
adjacency <interface>
show mls cef lookup <next hop ip address>
<ip address> <mod> show mls cef adjacency detail
entry <index> module
<mod>
BRKRST-3143

L3 FIB Table and Counters
DUT#sh ip cef 9.0.1.2 <- SW FIB
9.0.0.0/8 Which one of the 2 is being used ?
SW
nexthop 7.6.1.2 TenGigabitEthernet8/1
Exact path for SW switched packets
8.0.1.1 -> 9.0.1.2 => IP adj out of TenGigabitEthernet8/1, addr 7.6.1.2
DUT#sh ip cef adjacency TenGigabitEthernet 8/1 7.6.1.2
7.6.1.2/32 SW adjacency
attached to TenGigabitEthernet8/1
9.0.0.0/8
DUT#sh mls cef lookup 9.0.1.2 mod 7
Check HW FIB table on ingress DFC/PFC (module 7 HW
in this case): finds the longest prefix match in HW for
Codes: decap - Decapsulation, + - Push Label … is it consistent with the SW information ?
Index Prefix Adjacency
Which adjacency is used ?
108749 9.0.0.0/8 Te8/1 , 000f.f8e4.d000 (Hash: 007F)
Te8/3 , 000f.f8e4.d000 (Hash: 7F80)
Displays exact HW load sharing
DUT#sh mls cef exact-route 8.0.1.1 0 9.0.1.2 0 module 7
path for the flow … if not UDP or
Interface: Te8/3, Next Hop: 7.7.1.2, Vlan: 1090, Destination Mac: 000f.f8e4.d000 TCP, use port numbers 0, else use
correct port numbers !
DUT#show vlan internal usage | i 1090
L3 Interface map internally to a “1-
1090 TenGigabitEthernet8/3 port” VLAN
BRKRST-3143

14664_05_2008_c2.scr
DUT#show adjacency ten 8/3 7.7.1.2 detail
Aggregate HW adjacency statistics (SW
collects it from all DFC/PFC’s for all prefixes
Protocol Interface Address linked to this adjacency): do they move ?
IP TenGigabitEthernet8/3 7.7.1.2(17)
2001 packets, 228114 bytes
epoch 0
Rewrite information (Dmac|Smac|0800): verify
sourced in sev-epoch 774 it is conform with next hop rewrite info
Encap length 14
000FF8E4D0000050F0F874000800
ARP
DUT#show mls cef lookup 9.0.1.2 detail mod 7

To get HW adjacency statistic for this prefix on this module
Codes: M - mask entry, V - value entry, A - adjacency index, P - priority bit

D - full don't switch, m - load balancing modnumber, B - BGP Bucket sel
V0 - Vlan 0,C0 - don't comp bit 0,V1 - Vlan 1,C1 - don't comp bit 1
RVTEN - RPF Vlan table enable, RVTSEL - RPF Vlan table select
Format: IPV4_DA - (8 | xtag vpn pi cr recirc tos prefix) Start adjacency pointer is
294933, 14 + 1 = 15 adjacencies
Format: IPV4_SA - (9 | xtag vpn pi cr recirc prefix)
linked to the prefix
M(108749 ): E | 1 FFF 0 0 0 0 255.0.0.0
V(108749 ): 8 | 1 0 0 0 0 0 9.0.0.0 (A:294933 ,P:1,D:0,m:14,B:0 )
BRKRST-3143

DUT#show mls cef adjacency entry 294933 to 294947 mod 7
Index: 294933 smac: 0050.f0f8.7400, dmac: 000f.f8e4.d000
mtu: 9234, vlan: 1091, dindex: 0x0, l3rw_vld: 1
packets: 0, bytes: 0 15 HW adjacencies linked to this prefix:
which one is really used ?
. . .
packets: 0, bytes: 0
DUT#show mls cef adjacency entry 294933 to 294947 mod 7 | i packets
th
The 8 one … as SW polls this clear on read
packets: 0, bytes: 0 counters, hard to capture … check if adjacency
packets: 0, bytes: 0 moves
. . .
DUT#show mls cef adjacency entry 294940 det mod 7
… th
8 one reset to 0: SW polls this clear on read counters for “show adjacency”
For other direction (to 8.0.1.1), completely similar commands
BRKRST-3143

14664_05_2008_c2.scr
L3 FIB Table Special Entries/Adjacencies
Default entry: 0.0.0.0/0 (“match all”)
ALWAYS at bottom of FIB TCAM, if no default route, punt to drop adjacency,
DUT#sh ip route 0.0.0.0 0.0.0.0
No default route present
% Network not in table
DUT#sh mls cef lookup 123.0.1.1
Codes: decap - Decapsulation, + - Push Label
Match-all entry links to drop adjacency,
134368 0.0.0.0/0 drop which is subject to rate limiter "ICMP
UNREAC. NO-ROUTE": in profile packets
DUT# get punted to CPU … so possible reason
for packets hitting CPU
else default route linked to HW adjacency
After adding default route to Vlan1200,
Codes: decap - Decapsulation, + - Push Label adjacency points to next hop, all switched in
HW
134368 0.0.0.0/0 Vl1200 , 0011.bc75.9c00
DUT#
BRKRST-3143

Drop adjacency (route to Null0): subject to rate limiter "ICMP UNREAC. NO-
ROUTE"
FIB receive (local IP address): subject to rate limiter “CEF RECEIVE”
If not present, packets for local IP addresses
Index Prefix Adjacency don’t get to RP (SW)
343 7.1.1.2/32 receive
CEF Glean: subject to rate limiter “CEF GLEAN”

DUT#sh ip route 5.0.1.0
Routing entry for 5.0.1.0/24
…
* directly connected, via Vlan1000
Route metric is 0, traffic share count is 1
DUT#sh ip arp 5.0.1.123
DUT#show mls cef lookup 5.0.1.123 Unresolved directly connected host
If not present, packets to unresolved IP
3212 5.0.1.0/24 glean addresses for directly connected hosts/routers
will not get punted to RP (SW) to trigger ARP
resolution
BRKRST-3143

14664_05_2008_c2.scr
CEF Glean: subject to rate limiter “CEF GLEAN” (continued)
DUT#sh ip route 77.0.0.0
Known via "static", distance 1, metric 0 (connected)
Redistributing via ospf 100
Routing Descriptor Blocks: Another example where we need CEF
glean: static route with next hop
* directly connected, via TenGigabitEthernet8/3 specified as interface relies on proxy arp
Route metric is 0, traffic share count is on next hop to resolve next hop
Index Prefix Adjacency Not yet resolved: first packet hits glean entry, goes to RP,
108750 77.0.0.0/8 glean triggers ARP resolution; no glean entry present: we keep
hitting SW
DUT#sh ip arp 77.0.0.1
DUT#sh ip arp 77.0.0.1 Resolved via ARP
Internet 77.0.0.1 0 000f.f8e4.d000 ARPA TenGigabitEthernet8/3
Host entry for destination based on proxy arp
resolution. Static routes like this can use up lots
165 77.0.0.1/32 Te8/3 , 000f.f8e4.d000 of FIB table entries !!
BRKRST-3143

L3 FIB Table HW Rate Limiters
Rate limiters: HW rate limit packets pointed to Route Processor, no counter !!
For L3 Unicast:
DUT#sh mls rate-limit
Sharing Codes: S - static, D - dynamic
Codes dynamic sharing: H - owner (head) of the group, g - guest of the group
Rate Limiter Type Status Packets/s Burst Sharing

Truncated output: only listed relevant ones
--------------------- ---------- --------- ----- ------- for Ip Unicast
....
IP FEATURES Off - - -
CEF RECEIVE Off - - -
CEF GLEAN Off - - -
IP RPF FAILURE On 100 10 Group:0 S
TTL FAILURE Off - - -
ICMP UNREAC. NO-ROUTE On 100 10 Group:0 S
ICMP UNREAC. ACL-DROP On 100 10 Group:0 S
ICMP REDIRECT Off - - - Shared (same group) indicates packets
matching these types, will be subject to the
MTU FAILURE Off - - -
same HW rate limiter at 100 pps aggregate
UCAST IP OPTION Off - - - per DFC/PFC !!
IP ERRORS On 100 10 Group:0 S
...
BRKRST-3143

14664_05_2008_c2.scr
L3 Engine Counters Check for all modules that have DFC/PFC;
DUT#sh mls statistics mod 8 lookup is at ingress DFC/PFC
Statistics for Earl in Module 8

L2 Forwarding Engine Refer to earlier L2 counters
Total packets Switched : 1453743937845
L3 Forwarding Engine Total packets and current Packet-
Per-Second seen
Total packets L3 Switched : 1251810539335 @ 0 pps
Total Packets Bridged : 200667165283 L2 Switched packets
Total Packets FIB Switched : 1251810539334
Total Packets ACL Routed : 0 Forwarded based on FIB TCAM table result,
Total Packets Netflow Switched : 1
ACL TCAM table result or Netflow TCAM
table result
…
Total packets dropped by ACL : 2
Security ACL drops
Total packets dropped by Policing : 0
Total packets exceeding CIR : 0
Total packets exceeding PIR : 0 QOS ACL drops
Errors
MAC/IP length inconsistencies : 0
Errors pointed to route processor,
Short IP packets received : 0
subject to HW rate limiters
IP header checksum errors : 0
TTL failures : 7852668
MTU failures : 200209207135
BRKRST-3143

L3 FIB Table uRPF and VRF
Checking uRPF:
DUT#sh mls cef rpf 9.0.1.1 Verify unicast RPF check is performed in HW;
RPF information for prefix 9.0.1.1
uRPF check performed in the hardware for interfaces:
TenGigabitEthernet8/1
TenGigabitEthernet8/3
uRPF check disabled for interfaces:
Checking VRF’s Check interface (vlan 701) is in the correct VRF (VPN
DUT#remote com sw sh mls vlan-ram 701 end 701 value 0: default routing table)
TYCHO Vlan RAM
Key: * => Set, - => Clear
vlan eom nf-vpn mpls mc-base siteid stats rpf vpn-num bgp-grp l2-metro rpf-pbr-ovr
----+---+------+----+-------+------+-----+---+-------+-------+--------+-----------
701 - - * 0 0 - - 0 0 - *
DUT(config)#int vlan 701
Illustration: move to different VRF, and how to
DUT(config-if)#ip vrf forwarding customer-1 check this got programmed in HW …
DUT#remote com sw sh mls vlan-ram 701 end 701 sometimes issues seen with interface staying
TYCHO Vlan RAM in default VRF; check this on each DFC/PFC !!
Key: * => Set, - => Clear
vlan eom nf-vpn mpls mc-base siteid stats rpf vpn-num bgp-grp l2-metro rpf-pbr-ovr
----+---+------+----+-------+------+-----+---+-------+-------+--------+-----------
701 - - * 0 0 - - 256 0 - *
DUT#show mls cef exact-route vrf ?
Use further same commands as with default
WORD VPN Routing/Forwarding instance name
FIB, specifying VPN with vrf key word
BRKRST-3143

14664_05_2008_c2.scr
What Have We (Not Yet) Looked at ?
FIB contains IPv4/IPv6
QoS TCAM entries
Hardware for ACL

for stats and
features
ACE?
Counter
V
?
QoS Adj FIB
NetFlow TCAM TCAM TCAM
ACL
TCAM
V ? ACL TCAM
contains
security and
feature ACL
entries
L3/4 Engine
PFC3/DFC3
L2 Engine L2 CAM
contains
V
L2 CAM (64K) MAC entries
Verified already at FIB and adjacency tables, as well as L2 CAM table

Still to look at: ACL and Netflow Table
BRKRST-3143

L3 ACL Table and Counters
After configuring Vlan Access Map on vlan 701 to 705
DUT#sh vlan access-map DenyHost1<->Host2
Vlan access-map "DenyHost1<->Host2" 10
Configuration of VLAN access map
match: ip address DenyHost1<->Host2

action: drop
Vlan access-map "DenyHost1<->Host2" 20
Verify the (correct) ACL is present in the HW; remember flow
match: ip address MatchAll 8.0.1.1 -> 9.0.1.2 came in via Vlan 705, ingress interface gi7/3 but
action: forward VACL is bidirectional, hence similar out(bound) entry should be
present,
DUT#sh tcam int vlan 705 acl in ip mod 7
* Global Defaults not shared
Entries from Bank 0
deny ip host 8.0.1.1 host 9.0.1.2 (87 matches)

permit ip any any (1167 matches)
Entries from Bank 1 ACL drop counter, PFC2/PFC3A don’t have

this counter, can get cleared by SW, so look at
Deny/permit keywords; other possibilities: trend while debugging
-redirect: redirect to a specific interface (can be RP, central rewrite …)
- punt: point to CPU
- policy-route
Useful to find out why packets get punted to SW (RP)
BRKRST-3143

14664_05_2008_c2.scr
After adding Policy Based Routing on Vlan 705
DUT# show route-map PolicyRouteTo9.0.1.2
route-map PolicyRouteTo9.0.1.2, permit, sequence 10
Match clauses: Policy based routing configuration
ip address (access-lists): Select9.0.1.2
Set clauses:
ip next-hop 7.1.1.1
Policy routing matches: 0 packets, 0 bytes
DUT#sh tcam int vlan 705 acl in ip mod 7
* Global Defaults not shared
Entries from Bank 0
deny ip host 8.0.1.1 host 9.0.1.2
Inbound VACL entry
permit ip any any

Entries from Bank 1
policy-route ip any host 9.0.1.2
permit ip any any (84 matches)
(Inbound) PBR entry
What does this effectively mean … ?
BRKRST-3143

L3 ACL Table: ACL TCAM Structure
Higher and lower Bank 144 bits, Packet information
Lookup in BOTH banks,

generates 2 results 0 16K
Result with highest priority wins, HI bank LOW bank

TCAM TCAM
if both results have high priority, (hi priority) (low priority)
HI bank wins, if both low, LO
bank wins
16K 32K
Single lookup mode: equivalent
with single bank result
RSLT1 RSLT2
Serial lookup mode: only apply priority1 priority2
LO Bank lookup if HI Bank result
says permit
BRKRST-3143

14664_05_2008_c2.scr
Understanding what is programmed in the ACL
DUT#sh tcam int vlan 705 acl in ip detail mod 7
Look at ingress module
…
Interface: 705 label: 3585 lookup_type: 0
protocol: IP packet-type: 0 flow 8.0.1.1 -> 9.0.1.2 first match
+-+-----+---------------+---------------+---------------+---------------+-------+---+----+-+---+--+---+---+
lookup result in Hi bank: Index 17789,
priority set (to high) as indicated by (*)
|T|Index| Dest Ip Addr | Source Ip Addr| DPort | SPort | TCP-F |Pro|MRFM|X|TOS|TN|COD|F-P|
+-+-----+---------------+---------------+---------------+---------------+-------+---+----+-+---+--+---+---+
V 17789 9.0.1.2 8.0.1.1 P=0 P=0 ------ 0 ---- 0 0 -- C-- 0-0
M 17792 255.255.255.255 255.255.255.255 0 0 ------ 0 ---- 0 0
R rslt: L2_L3_DENY_RESULT (*) rtr_rslt: L2_L3_DENY_RESULT (*) hit_cnt=0
V 17840 0.0.0.0 0.0.0.0 P=0 P=0 ------ 0 ---- 0 0 -- C-- 0-0
M 17846 0.0.0.0 0.0.0.0 0 0 ------
Apply rule from 0previous
---- 0 0
slide: if first match in Lo and Hi Bank have
R rslt: PERMIT_RESULT rtr_rslt: PERMIT_RESULT hit_cnt=0
both priority set (to <- Match
high) , Hi Bank all
result entry
wins =>Hi Bank
V 18396 0.0.0.0 0.0.0.0 P=0 P=0 ------
L2_L3_DENY_RESULT0 ---- 0 (deny)
0 -- --- 0-0
M 18404 0.0.0.0 0.0.0.0 0 0 ------ 0 ---- 0 0
Similarly, applying rules, for flow 8.0.1.2 -> 9.0.1.2, entry 31642 will
R rslt: L3_DENY_RESULT rtr_rslt: L3_DENY_RESULT hit_cnt=0 Hi Bank
win, REDIRECT_ADJACENCY with index 0x7F803 (policy routing)
V 31642 9.0.1.2 0.0.0.0 P=0 P=0 ------ 0 ---- 1 0 -- C-- 0-0

M 31643 255.255.255.255 0.0.0.0 0 0 ------ 0 ---- 1 0
R rslt: REDIRECT_ADJACENCY (*) rtr_rslt: PERMIT_RESULT indx: 0x7F803 hit_cnt=0
V 36293 0.0.0.0 0.0.0.0 P=0 P=0 ------ 0 ---- 0 0 -- C-- 0-0 <-
M 36296 0.0.0.0 0.0.0.0 0 0 ------ 0 ---- 0 0 flow 8.0.1.1
<- -> 9.0.1.2 first match
R rslt: PERMIT_RESULT rtr_rslt: PERMIT_RESULT hit_cnt=95
lookup result in Lo bank: Index 31642,
<- <- Match all entry Lo Bank
priority set (to high) as indicated by (*)
V 36828 0.0.0.0 0.0.0.0 P=0 P=0 ------ 0 ---- 0 0 -- --- 0-0
M 36836 0.0.0.0 0.0.0.0 0 0 ------ 0 ---- 0 0
R rslt: L3_DENY_RESULT (*) rtr_rslt: L3_DENY_RESULT (*) hit_cnt=17 Lo Bank
BRKRST-3143

What Have We (Not Yet) Looked at ?
FIB contains IPv4/IPv6
QoS TCAM entries
Hardware for ACL

for stats and
features
ACE
V
Counter
QoS Adj FIB ACLV V V
?
NetFlow TCAM TCAM TCAM
TCAM
ACL TCAM
contains
security and
feature ACL
entries
L3/4 Engine
PFC3/DFC3
L2 Engine L2 CAM
contains
V
L2 CAM (64K) MAC entries
Verified already at FIB and adjacency tables, as well as L2 CAM table, ACL
TCAM/counters
Still to look at: Netflow Table
BRKRST-3143

14664_05_2008_c2.scr
L3 ACL and NetFlow Table: Programming the Table
Feature Interaction Engine process takes care of selecting a correct
strategy to program Hi/Lo Bank in case multiple ACL based features are
combined on same interface :
HW based like PBR and Security ACL’s
HW assisted like NAT, SLB, TCP intercept, Reflexive ACL … using ACL’s to
select traffic that needs to be punted to CPU, SW installs netflow entry in HW to
forward consecutive packets for same flow
SW based features: ACL used to punt packets that require SW processing
to CPU
Feature Manager process transforms all ACL’s into Value-Mask-Result,
and calls merge algorithms to combine multiple ACL based features,
outcome programmed into ACL TCAM
If no success full strategy to combine features in FIE (feature conflicts,
flow mask conflicts), FIE will move one of the features to SW and re-
attempt to find a strategy for the remaining ones
HW assisted/SW features: ACL TCAM identifies packets that needs to
be punted to SW for HW assisted (Netflow based) or SW forwarding
BRKRST-3143
L3 Unicast Traffic Network Refresh

Po1 Po2
Gig5/2 Gig7/2
Gig8/2 Gig7/3
Ten8/1 Ten8/1
Gig8/1 Gig7/4
Gig8/4 Gig7/6
R1 Outside DUT Inside R2
8.0.1.2 9.0.1.1
10.0.1.3/9.0.1.1
Running equal cost multi path routing with respectively 5 (Vlan 701
to 705) and 2 (Ten8/1 and Ten8/3) equal cost paths
Doing NAT on DUT between Vlan701-705 and Ten8/1,Ten8/3
BRKRST-3143

14664_05_2008_c2.scr
L3 ACL and NetFlow Tables: HW Assisted Features
NAT SW configuration status: configure NAT between Vlan701-705 and Ten8/1,Ten8/3
DUT#sh ip nat stat
Total active translations: 0 (0 static, 0 dynamic; 0 extended)
Outside interfaces:
Vlan701, Vlan702, Vlan703, Vlan704, Vlan705
Inside interfaces:
TenGigabitEthernet8/1, TenGigabitEthernet8/3
Hits: 10 Misses: 0
CEF Translated packets: 10, CEF Punted packets: 0
Expired translations: 2
Dynamic mappings:
-- Inside Source
[Id: 4] access-list FromR2 pool TowardsR1 refcount 0
pool TowardsR1: netmask 255.255.255.0
start 10.0.1.1 end 10.0.1.255
type generic, total addresses 255, allocated 0 (0%), misses 0
longest chain in pool: TowardsR1's addr-hash: 0, average len 0,chains 0/256
DUT#sh ip nat translations
Pro Inside global Inside local Outside local Outside global
tcp 10.0.1.3:19343 9.0.1.1:19343 8.0.1.2:23 8.0.1.2:23
--- 10.0.1.3 9.0.1.1 ---
BRKRST-3143

L3 ACL Table and Hardware Assisted Features
HW assisted forwarding: SW based creation of Netflow entries
DUT#sh mls netflow ip sw-installed mod 7 Based on ACL TCAM, first packet matching NAT criteria
Displaying Netflow entries in EARL in module 7 gets punted to CPU, and SW NAT’ed; SW then installes
Netflow Entry with correct NAT rewrite info into Netflow
DstIP SrcIP Prot:SrcPort:DstPort Src i/f TCAM; subsequent
:AdjPtrpackets wil hit this one and get
forwarded in HW. Check ACL TCAM similar to previous
-----------------------------------------------------------------------------
slides (if not OK, no NAT), check presence of
Pkts Bytes Age LastSeen Attributes
SwInstalled Netflow entry HW acceleration of specific
--------------------------------------------------- NAT translation … (if not, OK, possible reason for
8.0.1.2 9.0.1.1 tcp :19343 :telnet Te8/3 high CPU) :0x8000A
0 0 36 18:47:42 L3 - SwInstalled
10.0.1.3 8.0.1.2 tcp :telnet :19343 Vl701 :0x8000B
5 230 36 18:47:46 L3 – SwInstalled
DUT#sh mls netflow ip sw-installed mod 8
Displaying Netflow entries in EARL in module 8 Adjacency pointer to adjacency with NAT rewrite info
DstIP SrcIP Prot:SrcPort:DstPort Src i/f :AdjPtr

-----------------------------------------------------------------------------
Adjacency pointer to adjacency with NAT rewrite info
Pkts Bytes Age LastSeen Attributes
---------------------------------------------------
8.0.1.2 9.0.1.1 tcp :19343 :telnet Te8/3 :0x80028
7 322 38 18:47:47 L3 – SwInstalled
10.0.1.3 8.0.1.2 tcp :telnet :19343 Vl701 :0x80029
0 0 38 18:47:43 L3 - SwInstalled
BRKRST-3143

14664_05_2008_c2.scr
Adjacencies linked to SW installed Netflow
DUT#show mls cef adj entry 0x8000B det mod 7 Adjacency contains NAT rewrite info; index is only
Index: 524298 smac: 0050.f0f8.7400, dmac: 000f.f8e4.d000 significant per PFC/DFC !!
format: MAC_IPV4, flags: 0x4008418
ip_sa: 0.0.0.0, ip_da: 9.0.1.1
Rewrite info says modify IP destination Address
DUT#show mls cef adj entry 0x80028 det mod 8
Index: 524329 smac: 0050.f0f8.7400, dmac: 0011.bc75.9c00
format: MAC_IPV4, flags: 0x2008418
ip_sa: 10.0.1.3, ip_da: 0.0.0.0 Rewrite info says modify IP source Address
“show tcam acl” commands on in/egress interface explain what traffic gets punted
to SW because of NAT (first packet(s), till SW installs Netflow entry), follow same
logic as in ACL TCAM when interpreting the output
Similar HW assisted features: Reflexive ACL, SLB, TCP intercept … look at the
SW installed Netflow entries, as well as the ACL TCAM content for the relevant
interfaces
BRKRST-3143

Checking for feature conflicts (this can cause packet punt to CPU), truncated output
DUT#sh fm fie interface vlan 705
Check both in/egress L3 interface, both in/egress
Interface Vl705: direction, e.g. Vlans 701 to 705 and Ten8/1 and Ten8/3 in
Feature interaction state created: Yes our example
Flowmask conflict status for protocol IP : FIE_FLOWMASK_STATUS_SUCCESS
No flow mask conflict, OK !!
Flowmask conflict status for protocol OTHER : FIE_FLOWMASK_STATUS_SUCCESS
E.g. other possibilities:
Interface Vl705 [Ingress]:
FLOWMASK_CONFLICT
FIE Result for protocol IP : FIE_SUCCESS_NO_CONFLICT
Features Configured : VACL NAT PBR - Protocol : IP FLOWMASK_REDUCED
FM Label when FIE was invoked : 355 If conflicting flow mask
requirements, traffic on this interface
. . .
will be sent to software …..Redefine
Interface Vl705 [Egress]: and reapply or deconfigure one or
FIE Result for protocol IP : FIE_SUCCESS_NO_CONFLICT more features to avoid the conflict.
No flow mask
Features Configured : NAT VACL - Protocol : IP conflict, OK !!
FM Label when FIE was invoked : 370
. . .
No feature conflict, OK !! No feature conflict, OK !! No HW

acceleration support for multiple flow
based features on the same flow since
we do not build an adjacency that will do
the combine operation of all the features
in one pass.
BRKRST-3143

14664_05_2008_c2.scr
Detailed Packet Flow Troubleshooting
Other Useful Commands
HW/SW inconsistencies: run consistency checker on demand
show mls cef inconsistency now module <module>
Capture all info at once for a particular L3 Prefix:

show platform tech-support unicast <destination> <mask>
Are we running out of L2/L3 Engine Resources (FIB, ACL, Netflow TCAM full …) ?
show platform hardware capacity forwarding
BRKRST-3143

Unicast L2 and L3 traffic: what to check ?
Which counters and (forwarding) tables to look at

BRKRST-3143

14664_05_2008_c2.scr
Some Useful Troubleshooting (Tools)
What Packets Are (Not) Hitting the CPU ?
Reference:
http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800a70f2.shtml
Which interface(s) on RP are seeing the packets ?
87% interrupts handling = packets hit the CPU,
DUT#show proc cpu | i CPU otherwise, look at what process uses CPU
CPU utilization for five seconds: 99%/87%; one minute: 5%; five minutes: 4%
DUT#show interface stats
Look at switching statistics on each interface … which interface has high count (and still
... increasing) for Process and/or SW CEF for incoming packets ?
Vlan701
Switching path Pkts In Chars In Pkts Out Chars Out Process switching
Processor 1635 124010 1701 128085
SW CEF switching
Route cache 14431965 1731835800 29499 3539880
Distributed cache 508013546 62991421254 215068498 25808219760
HW CEF (distributed CEF) switching
Total 522447146 64723381064 215099698 25811887725
...
Shorter output (faster to
DUT#show interface stats | i (^Giga|^Fast|^Port|^Vlan|Processor|Route cache) analyze, 3 lines/interface)
DUT#show ibc
Interface information:
(Truncated output)Overall InBand Channel (IBC)
Interface IBC0/0(idb 0x4784157C)
traffic statistics: whatever gets punted to CPU goes
Hardware is Mistral IBC (revision 5) through IBC; check the rates
5 minute rx rate 66000 bits/sec, 38 packets/sec
5 minute tx rate 64000 bits/sec, 40 packets/sec 8135 Inband input packet drops
229904 Packets CEF Switched, 28956320 Packets Fast Switched
BRKRST-3143 Potential/Actual paks copied to process level 106180293/108080293(4293067296 dropped,2120 spd drops)
Some Useful Troubleshooting Tools

What Packets Are (Not) Hitting the CPU ?
What/why Process (not SW CEF) switched packets are hitting the RP ?
DUT#show ip cef switching statistics
DUT#show ip traffic
Information on Process (not SW CEF) switched
packets; (if no SW CEF, no HW CEF either)
DUT#show buffers input-interface <interface>

In case of Process (not SW CEF) switching, if input
queue is filling up, what packets are in queue ?
What/why SW CEF (not Process) switched packets are hitting the RP ?

DUT#show ip cef summary
DUT#show mls cef summary
DUT#show platform capacity forwarding Are we running out of HW resources (HW FIB full), compare
DUT#show mls cef exception status detail amount of entries between SW (IP) and HW (MLS) CEF ?
Does the platform HW capacity get exceeded ? Any FIB
exceptions ?
DUT#show tcam interface <interface> acl in …
DUT#show fm fie interface <interface> … Refer to earlier slides: on the “culprit” interface any HW assisted
features or SW features enabled … TCAM will be used to point
packets to CPU, any flow mask or feature conflicts on the interface ?
monitor session 1 type local
source cpu <rp|sp>
destination interface …
Extra tool in 12.2(33)SXH and higher: CPU SPAN can be used to
quickly see with sniffer what packets are sent to RP (or SP) CPU ,
then check the tables (L2/L3 etc. ..)
BRKRST-3143

14664_05_2008_c2.scr
Does the CPU Inband Driver See the Packet ?
Extra tool: debug netdr (use with caution … check with TAC)
DUT#debug netdr capture ?
Be as specific as possible; on
acl (11) Capture packets matching an acl SP, remote login switch, then
and-filter (3) Apply filters in an and function: all must match same set of commands)
continuous (1) Capture packets continuously: cyclic overwrite
destination-ip-address (10) Capture all packets matching ip dst address
dstindex (7) Capture all packets matching destination index
ethertype (8) Capture all packets matching ethertype
interface (4) Capture packets related to this interface
or-filter (3) Apply filters in an or function: only one must match
rx (2) Capture incoming packets only
source-ip-address (9) Capture all packets matching ip src address
srcindex (6) Capture all packets matching source index
tx (2) Capture outgoing packets only
vlan (5) Capture packets matching this vlan number
<cr>
BRKRST-3143

Does the CPU Inband Driver See the Packet ?
DUT#sh netdr captured-packets
A total of 289 packets have been captured
The capture buffer wrapped 0 times
Total capture capacity: 4096 packets
------- dump of incoming inband packet -------
interface Vl1000, routine mistral_process_rx_packet_inlin E.g.: ARP packet came in on Vlan1000 of RP
Inband Driver
dbus info: src_vlan 0x3E8(1000), src_indx 0x45(69), len 0x40(64)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x43E8(17384)
80000401 03E80400 00450000 40800000 E0000000 00000000 00000008 43E80000
mistral hdr: req_token 0x0(0), src_index 0x45(69), rx_offset 0x76(118)
requeue 0, obl_pkt 0, vlan 0x3E8(1000)
destmac FF.FF.FF.FF.FF.FF, srcmac 00.A0.CC.21.94.C4, protocol 0806
layer 3 data: 00010800 06040001 00A0CC21 94C40500 01660000 00000000
05000102 00000000 00000000 00000000 00000000 000001FE
00000006 00000000 000003E8
..
Make sure to turn it off afterwards
DUT#undebug netdr
DUT#debug netdr clear-capture
Make sure to clear memory used up by captured packets
BRKRST-3143

14664_05_2008_c2.scr
Problems We’ve Looked at
(Some) packets don’t get through (drops, incorrect
forwarding)
Checked platform specific counters and tables
Unwanted flooding
Check we learn MAC, L2 tables are in sync
High CPU due to SW path forwarding

Finding out (quickly) what packets hit the CPU
Troubleshoot step-by-step, no steps skipping !!
BRKRST-3143
Agenda

BRKRST-3143

14664_05_2008_c2.scr
Terminology
OIF: Outgoing Interface
OIL: Outgoing Interface List
IGMP: Internet Group Management Protocol
Multicast FIB: Contains the (*,G) and (S,G) entries as well as RPF-
VLAN
Adjacency Table: Contains the rewrite information and MET index
LTL: Local Target Logic - forwarding logic for the Catalyst 6500
MET: Multicast Expansion Table - Hardware table that contains the
OIFs for for the (*,G) and (S,G) entries
BRKRST-3143
Local Target Logic (LTL)
Every valid packet that ingresses the Catalyst 6500 will

be sent to a forwarding engine (FE) within the system
(DFC or the PFC on the supervisor)
The FE makes the decision about where to forward the
packet or to drop the packet
Part of the result of the forwarding decision is a
destination LTL index (or destination index)
The destination index is used to select the physical
port(s) that will forward the packet
For multicast, another important part of the forwarding
decision is the MET index
BRKRST-3143

14664_05_2008_c2.scr
Multicast Expansion Table (MET)
The MET is memory where the list of OIFs for the

multicast entries are stored
Each replication engine in the chassis has a separate
MET
Read using the MET index from the CEF adjacency
MET block contains the list of OIFs and the
corresponding destination LTL index
BRKRST-3143
Multicast Expansion Table (MET)
Entry
OIF VLAN LTL
ID
Index 0x26 from ADJ 0x26 100 0x942

101 0x943 MET Block
102 0x945
Index 0x8A from ADJ 0x8A 100 0x960

1019 0x961
Index 0x8B from ADJ 0x8B 700 0x919

4030 0x920
4031 0x921
4032 0x933
BRKRST-3143

14664_05_2008_c2.scr
Multicast Replication
Replication: Process of creating copies of packets

L2 Replication: Creating copies of a packet within a
single VLAN
(e.g., Forwarding a single broadcast packet out all ports within a
VLAN)
Does not require a replication engine
L3 Replication: Creating copies of a multicast packet for

forwarding out each of the interfaces in an OIL.
Requires a replication engine
For this multicast discussion, the term Replication will

mean L3 Replication
BRKRST-3143
Multicast Replication Modes
Replication mode refers to where in the system

multicast replication occurs
In classic system, replication always occurs centrally on
the supervisor engine
In a fabric-enabled system, there are two possible
replication modes:
Ingress replication mode
Egress replication mode
BRKRST-3143

14664_05_2008_c2.scr
Ingress Replication Mode
Replication engine on ingress module performs replication
for all OIFs
One copy of the original packet is forwarded across the
fabric for each of the OIFs
All fabric enabled modules are capable of ingress mode
Only 6516A and 6700 series modules are capable of egress
mode
System will default to ingress mode when at least one
module not capable of egress mode is present
Packets ingress on a module without a replication engine
(classic module) will be replicated by the supervisor’s
replication engine
BRKRST-3143
Egress Replication Mode

Each Replication engine performs replication for local OIFs* only
Only a single copy of the original packet is sent across the fabric
for all interfaces in the OIL
Requirements:
1. Supervisor Engine 720
2. All cards in chassis must be 6700 series or 6516A
System will default to egress mode whenever possible
System can be forced to either mode with the command
mls ip multicast replication-mode [ingress|egress]
* Local OIF: Any OIF local to the replication engine.

BRKRST-3143

14664_05_2008_c2.scr
Local OIF Example
VLAN O
VLAN B R
Card 1 S
Card 2
R Port Port S Port Port
ASIC ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
R
R VLAN P CFC
VLAN G
Replication Replication Replication Replication

Engine & Engine & Engine & Engine &
Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC
EARL-DBUS
EARL-RBUS
Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
R Port Fabric ASIC L2 L3
& Engine Engine
R ASIC
VLAN P R Replication PFC
Supervisor Engine
BRKRST-3143
Diagram for troubleshooting example
Group
225.10.10.10 Gi9/1 Gi5/2
L3 Network VLAN 20 VLAN 10
Source Receiver
172.16.25.1 10.10.10.100
Gi9/4
VLAN 20
Receiver
20.20.20.100
BRKRST-3143

14664_05_2008_c2.scr
Packet Walk Components Replication Engine/Fabric
ASIC: Transmits & receives
Centralized Forwarding Card (CFC): packets from the switch fabric.
Port ASIC: Handles
Serves as BUS ASIC. Transmits & Responsible for SPAN and
packets to and from
receives packets from the EARL- multicast replication. Performs
physical interfaces.
Module 1 DBUS and EARL-RBUS Module 2 all MET lookups using indices
Applies and
Port Port Port from L3 lookups. Does packet
Port
removes any
rewrites for packets sent across
ASIC
trunking encap/tags, ASIC ASIC ASIC
the fabric
applies ingress and CFC CFC
egress QoS

EARL-DBUS
Replication Engine/Fabric
EARL-RBUS ASIC (Supervisor): Transmits &
receives packets from the switch
fabric. Responsible for SPAN
and multicast replication.
PFC/Switching Engine:
Performs all MET lookups using
indices from L3 lookups. Does
Switch L2
Fabric
engine: L2 lookups
L3 engine: L3 FIB & Adj
packet rewrites for packets sent
lookups; NetFlow lookups;
across the fabric. Also serves as
RACL, VACL & QoS lookups
LC-DBUS
SP RP BUS ASIC
LC-RBUS
CPU CPU
Fabric ASIC L2 L3
Port
& Engine Engine
ASIC
Replication PFC
Supervisor Engine
BRKRST-3143
Ingress Replication Packet Walk 4. DBUS ASIC on

module 2 receives
2. Port ASIC sends to fabric ASIC
packet and discards it.
VLAN B R
Module 1 VLAN O Module 2
R Port Port Port Port
ASIC ASIC
S
S R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC CFC
VLAN P
3. BUS ASIC sends 1. Source S

DBUS packet over Engine & Engine & sends packets Engine & Engine &
EARL-DBUS Fabric ASIC Fabric ASIC in VLAN O Fabric ASIC Fabric ASIC
EARL-DBUS
EARL-RBUS
7. L3 engine performs ACL,
5. Supervisor DBUS 6. L2 engine performs
VACL and QoS lookup in
ASIC receives DBUS L2 lookup in ingress
ingress VLAN and performs an
packet and accepts it VLAN and forwards
RPF check
and forwards to PFC headers to L3 engine Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
& Engine Engine
R ASIC
Supervisor Engine
BRKRST-3143

14664_05_2008_c2.scr
Ingress Replication Packet Walk
VLAN B R
ASIC ASIC
S
S R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC CFC
VLAN P
11. ASIC originating

the DBUS packet
Replication Replication accepts the result, all Replication Replication
Engine & Engine & others discard Engine & Engine &
EARL-DBUS
EARL-RBUS
9. L2 engine
sends final result
10. Fabric ASIC on over LC-RBUS to
Supervisor Fabric ASIC
forwards result Switch Fabric
onto E-RBUS
LC-DBUS
SP RP
LC-RBUS
CPU CPU
8. L3 engine returns result

VLAN G R
to L2 engine. Result
& Engine Engine
R ASIC contains LTL index for
VLAN P R Replication PFC forwarding in the ingress
Supervisor Engine VLAN as well as indices
for MET lookup
BRKRST-3143
Ingress Replication Packet Walk 15. Port ASIC receives

the rewritten packet
and result and forwards
to receiver in the
ingress VLAN
VLAN B R
ASIC ASIC
S
S R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC CFC
VLAN P
12. Fabric ASIC on ingress
card rewrites the packet
according to the result, builds
Engine & Engine & a fabric packet containing theEngine & Engine &
Fabric ASIC Fabric ASIC rewritten packet and the result
Fabric ASIC Fabric ASIC 14. Fabric ASIC
and forwards to the fabric
on egress
module receives
packet from the
EARL-DBUS 13. Switch Fabric uses the
switch fabric and
FPOE in the fabric packet and
EARL-RBUS forwards to port
forwards only to channels that
ASIC
have receivers or mrouters in
the ingress VLAN (VLAN O)
Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
& Engine Engine
R ASIC
Supervisor Engine
BRKRST-3143

14664_05_2008_c2.scr
16. Replication Engine (RE) performs
a lookup in the MET using the MET
indices from the result received in step
VLAN B R
Module 1 11 toVLAN Oof the OIF’s in the OIL
get all Module 2
ASIC ASIC
S
S R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC CFC
VLAN P

17. RE makes one copy of the
original packet for each of the
OIF’s in the OIL and sends a
EARL-DBUS
corresponding DBUS packet
for each to the switching 19. L3 engine
EARL-RBUS
engine over the DBUS (only receives
showing one packet here for appropriate
brevity) 18. L2 engine does headers from the
no lookup and L2 engine and
forwards appropriate performs egress
headers to L3 engine
Switch Fabric ACL, VACL and
QoS lookup.
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
& Engine Engine
R ASIC
Supervisor Engine
BRKRST-3143
VLAN B R
ASIC ASIC
S
S R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC CFC
VLAN P
22. ASIC
Replication Replication originating the Replication Replication
Engine & Engine & DBUS packets Engine & Engine &
Fabric ASIC Fabric ASIC accepts the results, Fabric ASIC Fabric ASIC
all others discard
EARL-DBUS
EARL-RBUS
21. BUS ASIC

forwards result
onto EARL-RBUS 20. L3 engine forwards
Switch Fabric
result to L2 engine and
L2 engine forwards
result onto LC-RBUS
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
& Engine Engine
R ASIC
Supervisor Engine
BRKRST-3143

14664_05_2008_c2.scr
Ingress Replication Packet Walk 26. Port ASIC receives
the rewritten packet
and result and forwards
to receiver in the
egress VLAN
VLAN B R
ASIC ASIC
S
S R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC CFC
VLAN P
25. Fabric Replication Replication Replication Replication

ASICs on Engine & Engine & Engine & Engine &
Fabric ASIC Fabric ASIC 23. Fabric ASIC on Fabric ASIC Fabric ASIC
egress
ingress card rewrites the
modules
packets according to the
receive
results, builds a fabric
packet from
EARL-DBUS
the switch
packet for each containing
the rewritten packet and
fabric and
EARL-RBUS the result and forwards to
forward to
the fabric
port ASIC
24. Switch Fabric uses the
Switch Fabric
the egress VLANs
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
& Engine Engine
R ASIC
Supervisor Engine
BRKRST-3143
Egress Replication Packet Walk 4. DBUS ASIC on module 2

receives packet and
2. Port ASIC sends to fabric ASIC discards it.
VLAN O
VLAN B R
Module 1 S
Module 2
ASIC ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
R
R VLAN P CFC
VLAN G
3. Fabric ASIC sends 1. Source

DBUS packet over Engine & Engine & sends in Engine & Engine &
EARL-DBUS Fabric ASIC Fabric ASIC VLAN O Fabric ASIC Fabric ASIC
EARL-DBUS
EARL-RBUS
5. Supervisor DBUS ASIC receives

6. DBUS packet
DBUS packet and accepts it
forwarded to L2 engine
for L2 lookup
Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU 7. L3 engine performs
lookup using the
primary CEF entry.
L3 engine also does
VLAN G R
R Port Fabric ASIC L2 L3 ACL, VACL and QoS
& Engine Engine lookup in ingress
R ASIC
VLAN P R Replication PFC VLAN and RPF check
Supervisor Engine
BRKRST-3143

14664_05_2008_c2.scr
Egress Replication Packet Walk
VLAN O
VLAN B R
Module 1 S
Module 2
ASIC ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
R
R VLAN P CFC
VLAN G
11. ASIC originating
the DBUS packet
accepts the result, Replication Replication Replication Replication
all others discard Engine & Engine & Engine & Engine &
EARL-DBUS
EARL-RBUS
9. L2 engine
sends final result
10. Fabric ASIC on over LC-RBUS to
Supervisor forwards Fabric ASIC
result onto E-RBUS Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
8. L3 engine
returns result to L2
engine. Result
VLAN G R
R Port Fabric ASIC L2 L3 contains LTL index
& Engine Engine for forwarding in
R ASIC
VLAN P R Replication PFC the ingress VLAN
as well as indices
Supervisor Engine for MET lookup
BRKRST-3143
Egress Replication Packet Walk 15. Port ASIC

receives the rewritten
packet and result and
forwards to receiver in
the ingress VLAN
VLAN O
VLAN B R
Module 1 S
Module 2
ASIC ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
R
R VLAN P CFC
VLAN G
12. Fabric ASIC on ingress
card rewrites the packet
according to the result, builds a
Engine & Engine & fabric packet containing the Engine & Engine &
Fabric ASIC Fabric ASIC rewritten packet and the result
Fabric ASIC Fabric ASIC 14. Fabric
and forwards to the fabric
ASIC on
egress
module
EARL-DBUS 13. Switch Fabric uses the
receives
EARL-RBUS packet from
the switch
fabric and
the ingress VLAN (VLAN O)
forwards to
port ASIC
Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
& Engine Engine
R ASIC
Supervisor Engine
BRKRST-3143

14664_05_2008_c2.scr
Egress Replication Packet Walk 16. RE does a lookup in
the MET using the MET3
index from the result
received in step 11 to get
all of the OIF’s in the OIL VLAN O
VLAN B R
Module 1
that are local to this RE.
S
Module 2
R Port There is one receiver inPort S Port Port
ASIC VLAN G local to the RE ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
R
R VLAN P CFC
VLAN G

EARL-DBUS
EARL-RBUS
17. RE copies the original
packet onto VLAN G and 18. L2 engine performs
Fabric ASIC sends DBUS no lookup and
packet to the switching forwards appropriate
engine for egress headers to L3 engine
Switch Fabric
processing
LC-DBUS
SP RP
LC-RBUS
CPU CPU 19. L3 engine
receives appropriate
headers from the L2
engine and performs
VLAN G R
R Port Fabric ASIC L2 L3 ACL, VACL and QoS
& Engine Engine lookup for VLAN G.
R ASIC
VLAN P R Replication PFC Result is forwarded to
L2 engine
Supervisor Engine
BRKRST-3143

VLAN O
VLAN B R
Module 1 S
Module 2
ASIC ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
R
R VLAN P CFC
VLAN G
Replication Replication 21. Port ASIC Replication Replication

Engine & Engine & forwards packet Engine & Engine &
Fabric ASIC Fabric ASIC to receiver in Fabric ASIC Fabric ASIC
VLAN G based on
destination index
20. Fabric ASIC in the result
EARL-DBUS receives result &
EARL-RBUS forwards copy of
packet and result to
port ASIC
Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
& Engine Engine
R ASIC
Supervisor Engine
BRKRST-3143

14664_05_2008_c2.scr
22. RE does a second
lookup in the MET using
the MET2 index from the
VLAN O
previous result. This will
Module 1 Module 2
VLAN B R
R Port yield an egress Port S
S Port Port
ASIC
replication VLAN ID and
ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R a destination index.
CFC
R
R VLAN P CFC
VLAN G

EARL-DBUS
EARL-RBUS
24. L2 lookup indicates

23. RE copies the
bridging to all other
packet onto the egress
modules with receivers
replication VLAN and Switch Fabric
and flagging the packet so
Fabric ASIC & sends a
that it’s replicated by the
corresponding DBUS
receiving module
packet to the switching LC-DBUS
engine. SP RP
LC-RBUS
CPU CPU
25. No RACL,
VACL or QoS
lookups done
VLAN G R
on this packet.
& Engine Engine
R ASIC Result is
VLAN P R Replication PFC forwarded to
Supervisor Engine L2 engine
BRKRST-3143

VLAN O
VLAN B R
Module 1 S
Module 2
ASIC ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
R
R VLAN P CFC
VLAN G

EARL-DBUS
EARL-RBUS
26. Result received by
fabric ASIC on the
module 1. All others
discard result
Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
& Engine Engine
R ASIC
Supervisor Engine
BRKRST-3143

14664_05_2008_c2.scr
27. Fabric ASIC sets
FPOE to forward to all
cards with local
VLAN O
Module 1 receivers or mrouters in Module 2
VLAN B R
R Port Port S
S the egress VLANsPort and Port
ASIC ASIC
sendsR
Rthe packetASIC
and
ASIC
R
R VLAN O
VLAN O R
R CFC
R
R result
VLAN P
to the fabric in the
CFC
VLAN Gegress replication VLAN

EARL-DBUS
EARL-RBUS
29. Fabric ASICs on

egress modules
Switch Fabric
receive the packet on
the internal replication
28. Switch Fabric uses the LC-DBUS
VLAN and hand packet SP RP FPOE in the fabric packet
over to the RE LC-RBUS
CPU CPU to forward only to
channels that have local
receivers or mrouters in
the on any OIF
VLAN G R
& Engine Engine
R ASIC
Supervisor Engine
BRKRST-3143

30. Fabric ASIC
that received the VLAN O
VLAN B R
Module 1 on the
packet
S
Module 2
R Port Port
internal replication S Port Port
ASIC VLAN sends ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFCpacket to the
R
R VLAN P CFC
forwarding engine VLAN G
for CEF lookup

EARL-DBUS
EARL-RBUS
31. L2 engine
Note:
Note: Steps
Steps 3030 -- 32
32 are
are repeated
repeated for for each
each of
of the
the recognizes packet is
fabric
fabric ASICs
ASICs that
that received
received the the packet
packet on on the
the flagged for egress
internal replication VLAN. Each
internal replication VLAN. Each needs the needs the Switch Fabric
replication and
result
result of
of the
the CEF
CEF lookup
lookup (i.e.,
(i.e., the
the index
index for
for the
the forwards headers to
MET
MET lookup
lookup toto get
get thethe OIL
OIL forfor all
all the
the local
local L3 engine
receivers
receivers and
and mrouters)
mrouters) LC-DBUS
SP RP
LC-RBUS
CPU CPU
32. L3 engine performs
CEF lookup using
secondary entry.
VLAN G R
R Port Fabric ASIC L2 L3 Lookup yields MET
& Engine Engine index for replication to
R ASIC
VLAN P R Replication PFC all local OIFs. Result is
forwarded to the L2
Supervisor Engine engine
BRKRST-3143

14664_05_2008_c2.scr
33. Result received
by fabric ASIC on
VLAN O
Module 1the module 1. All Module 2
VLAN B R
R Port others discard Port S
S Port Port
ASIC
result
ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
R
R VLAN P CFC
VLAN G

EARL-DBUS
EARL-RBUS
Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
& Engine Engine
R ASIC
Supervisor Engine
BRKRST-3143

VLAN O
VLAN B R
Module 1 S
Module 2
ASIC ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
35. Packet R R VLAN P CFC
is forwarded
VLAN G
over the
DBUS to the
34. RE performs a
forwarding
MET lookup using Replication Replication Replication Replication
the MET index Engine & Engine engine
& for Engine & Engine &
an egress
from the result and Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC
lookup.
replicates packet
onto VLAN B
EARL-DBUS
EARL-RBUS
Switch Fabric
36. L2 engine
performs L2 lookup
in egress VLAN
(VLAN B) and LC-DBUS
SP RP forwards headers to
LC-RBUS
CPU CPU L3 engine 37. L3 engine
performs RACL,
VACL and QoS
lookups for
VLAN G R
R Port Fabric ASIC L2 L3 egress VLAN and
& Engine Engine forwards result to
R ASIC
VLAN P R Replication PFC L2 engine
Supervisor Engine
BRKRST-3143

14664_05_2008_c2.scr
39. Fabric ASIC
forwards a copy of
VLAN O
Module 1packet and result to Module 2
VLAN B R
R Port port ASIC Port S
S Port Port
ASIC ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
R
R VLAN P CFC
VLAN G
40. Port ASIC
forwards packet to
receiver in VLAN B Engine & Engine & Engine & Engine &
based on destination Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC
index in the result
38. Result received

EARL-DBUS by fabric ASIC on
EARL-RBUS the module 1. All
others discard
result
Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
& Engine Engine
R ASIC
Supervisor Engine
BRKRST-3143
Egress Replication Packet Walk 41. RE on module 2

performs a MET
lookup using the MET
index from the result
and replicates the
packet onto VLAN P
VLAN O
VLAN B R
Module 1 S
Module 2
ASIC ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
R
R VLAN P CFC
VLAN G

Engine & Engine & Engine & Engine & 42. Packet is
Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC sent to the
forwarding
engine over the
DBUS for an
EARL-DBUS egress lookup.
EARL-RBUS
Switch Fabric
43. L2 engine
performs L2 lookup
in egress VLAN
(VLAN P) and LC-DBUS
LC-RBUS
performs RACL,
VACL and QoS
lookups for
VLAN G R
R ASIC
Supervisor Engine
BRKRST-3143

14664_05_2008_c2.scr
46. Fabric ASIC
forwards a copy of
VLAN O
Module 1 packet and result to
Module 2
VLAN B R
R Port Port S
S Port port ASIC Port
ASIC ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
R
R VLAN P CFC
VLAN G
45. Result received
by fabric ASIC on
47. Port ASIC the module 1. All
Engine & Engine & forwards packet to Engine & Engineothers
& discard
Fabric ASIC Fabric ASIC receiver in VLAN P Fabric ASIC result
Fabric ASIC
based on destination
index in the result
EARL-DBUS
EARL-RBUS
Switch Fabric
LC-DBUS
SP RP
LC-RBUS
CPU CPU
VLAN G R
& Engine Engine
R ASIC
Supervisor Engine
BRKRST-3143

VLAN O
VLAN B R
Module 1 S
Module 2
ASIC ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
R
R VLAN P CFC
VLAN G

EARL-DBUS 48. RE on supervisor 49. Packet is

performs a MET lookup forwarded over
EARL-RBUS using the MET index the LC-DBUS to
from the result and the forwarding
replicates packet onto engine for an
VLAN G egress lookup.
Switch Fabric
50. L2 engine
performs L2 lookup
in egress VLAN
(VLAN G) and LC-DBUS
LC-RBUS
performs RACL,
VACL and QoS
lookups for
VLAN G R
R ASIC
Supervisor Engine
BRKRST-3143

14664_05_2008_c2.scr
VLAN O
VLAN B R
Module 1 S
Module 2
ASIC ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
R
R VLAN P CFC
VLAN G

EARL-DBUS
EARL-RBUS
52. Result received by

fabric ASIC on the
supervisor. Result is not
54. Port ASIC sent over the EARL-RBUS
Switch Fabric
53. Fabric ASIC
forwards packet to
forwards a copy of
receiver in VLAN G
packet and result to LC-DBUS
based on destination SP RP port ASIC
index in the result LC-RBUS
CPU CPU
VLAN G R
& Engine Engine
R ASIC
Supervisor Engine
BRKRST-3143

VLAN O
VLAN B R
Module 1 S
Module 2
ASIC ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
R
R VLAN P CFC
VLAN G

56. Packet
EARL-DBUS 55. RE on supervisor is forwarded
performs a MET lookup over the
EARL-RBUS using the MET index DBUS to the
from the result and forwarding
replicates packet onto engine for
VLAN P an egress
lookup.
57. L2 engineSwitch Fabric
performs L2 lookup
in egress VLAN
(VLAN P) and LC-DBUS
LC-RBUS
performs RACL,
VACL and QoS
lookups for
VLAN G R
R ASIC
Supervisor Engine
BRKRST-3143

14664_05_2008_c2.scr
VLAN O
VLAN B R
Module 1 S
Module 2
ASIC ASIC
R
R ASIC ASIC
R
R VLAN O
VLAN O R
R CFC
R
R VLAN P CFC
VLAN G

EARL-DBUS
EARL-RBUS
52. Result received

by fabric ASIC on
the module 1. All
54. Port ASIC others discard
Switch Fabric
53. Fabric ASIC
forwards packet to result
forwards a copy of
receiver in VLAN P
packet and result to LC-DBUS
based on destination SP RP port ASIC
index in the result LC-RBUS
CPU CPU
VLAN G R
& Engine Engine
R ASIC
Supervisor Engine
BRKRST-3143
Verify L1/L2
Use…
Show interfaces
Show interfaces counters
Show interfaces counters errors
and look for any physical layer errors

Follow the L2 troubleshooting steps from the previous
section IP unicast troubleshooting
BRKRST-3143

14664_05_2008_c2.scr
Diagram for troubleshooting example
Group
225.10.10.10 Gi9/1 Gi5/2
L3 Network VLAN 20 VLAN 10
Source Receiver
172.16.25.1 10.10.10.100
Gi9/4
VLAN 20
Receiver
20.20.20.100
BRKRST-3143
Receiving IGMP membership reports ?

Use show ip igmp groups [group] to verify that the receivers’
membership reports are received by the switch
Shows both
receivers in the
correct VLANs
Cat6K#show ip igmp groups 225.10.10.10
IGMP Connected Group Membership
Group Address Interface Uptime Expires Last Reporter
225.10.10.10 Vlan20 00:17:08 00:02:10 20.20.20.100
225.10.10.10 Vlan10 3d04h 00:02:30 10.10.10.100
Note:
Note: TheThe output
output only
only shows
shows thethe last
last reporter,
reporter,
so
so aa given
given host
host may
may not
not show
show upup in
in the
the output
output ifif
there
there are
are other
other receivers
receivers on
on the
the same
same interface.
interface.
Make
Make sure
sure that
that the
the OIF
OIF shows
shows up up in
in the
the
interface
interface column.
column.
BRKRST-3143

14664_05_2008_c2.scr
Is IGMP Snooping actually snooping ?
Use show mac-address-table multicast igmp-snooping to display the
IGMP Snooping L2 forwarding table
Gi9/1 is the incoming interface
also an mrouter port
Cat6K#show mac-address-table multicast igmp-snooping

Gi9/4 contains receiver
20.20.20.100 in VLAN 20
vlan mac address type learn qos ports
-----+---------------+--------+-----+---+--------------------------
20 0100.5e0a.0a0a static Yes - Gi9/1,Gi9/4,Router
10 0100.5e0a.0a0a static Yes - Gi5/2,Router
20 0100.5e00.0127 static Yes - Gi9/1,Gi9/4,Router
10 0100.5e00.0127 static Yes - Router Router indicates
Router indicates that the MSFC
10 0100.5e00.0128 static Yes - Router is a router port
20 0100.5e00.0128 static Yes - Gi9/1,Gi9/4,Router
Gi5/2 contains receiver

10.10.10.100 in VLAN 10
BRKRST-3143
Correct reverse path back to the source?

Use show ip route [source] to identify ingress or RPF interface for the
multicast traffic
Cat6K#show ip route 172.16.25.1

Known via "ospf 100", distance 110, metric 2, type inter area
Last update from 20.20.20.2 on Vlan20, 02:15:43 ago
Routing Descriptor Blocks:
* 20.20.20.2, from 20.20.20.2, 02:15:43 ago, via Vlan20
Route metric is 2, traffic share count is 1
BRKRST-3143

14664_05_2008_c2.scr
Does (S,G) exist? Is it installed in HW?
Use show ip mroute [group] [source] to verify that an mroute entry
exists
Cat6K#show ip mroute 225.10.10.10
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
L - Local, P - Pruned, R - RP-bit set, F - Register flag,
T - SPT-bit set, J - Join SPT, M - MSDP created entry,
X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
U - URD, I - Received Source Specific Host Report, Z - Multicast Tunnel
Y - Joined MDT-data group, y - Sending to MDT-data group
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode
(*, 225.10.10.10), 01:21:15/00:02:55, RP 100.100.100.100, flags: SJC

Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
(S,G) Vlan20, Forward/Sparse-Dense, 00:08:28/00:02:55 RPF neighbor
Vlan10, Forward/Sparse-Dense, 01:21:15/00:02:10
(172.16.25.1, 225.10.10.10), 01:21:15/00:02:50, flags: T

Incoming interface: Vlan20, RPF nbr 20.20.20.2, RPF-MFD
Outgoing interface list: RPF-MFD - Reverse Path
Vlan10, Forward/Sparse-Dense, 01:21:15/00:02:10, H Forwarding-Multicast Fast
Drop: when a multicast entry is
installed in the hardware, the
OIL entry is flagged with the RPF-
MFD flag. This flag ensures that
H-Flag: multicast entry is
RPF VLAN multicast traffic that is switched
installed in hardware
within a VLAN and non-rpf traffic
are not bridged to the RP.
BRKRST-3143
Are mcast packets being forwarded?

Use show ip mroute [source] [group] count to verify packets are
being forwarded for the mroute entry
Cat6K#show ip mroute 225.10.10.10 count

IP Multicast Statistics
5 routes using 3620 bytes of memory
3 groups, 0.66 average sources per group
Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kilobits per second
Other counts: Total/RPF failed/Other drops(OIF-null, rate-limit etc)
Group: 225.10.10.10, Source count: 1, Packets forwarded: 350, Packets received: 350
RP-tree: Forwarding: 0/0/0/0, Other: 0/0/0
Source: 172.16.25.1/32, Forwarding: 350/1/975/2, Other: 350/0/0
Make sure that drops

Make sure that are not incrementing
forwarding packet
counts are incrementing
BRKRST-3143

14664_05_2008_c2.scr
Are they forwarded in HW?
Use the show mls ip multicast group [group]

command to verify a hardware entry exists for the
group and that packets are being forwarded for that
entry.
Cat6K#show mls ip multicast group 225.10.10.10

Multicast hardware switched flows:
(172.16.25.1, 225.10.10.10) Incoming interface: Vlan20, Packets switched: 361
Hardware switched outgoing interfaces:
Vlan10
RPF-MFD installed
Verifies RPF interface
and shows that packets
are being switched and
the correct OIF
BRKRST-3143
Which forwarding mode is being used?
Use show mls ip multicast capability to show which

forwarding mode is being used.
Shows that the global
mode is Ingress
Cat6K#show mls ip multicast capability
Current mode of replication is Ingress
Configured replication mode is Auto
Slot Multicast replication capability

5 Egress
9 Ingress
One card in the chassis

only capable of ingress
mode cause the mode to
move to ingress
BRKRST-3143

14664_05_2008_c2.scr
Is there a CEF entry in the HW?
Use show mls cef ip multicast group [group] to get adjacency pointer
for (S,G) entry in hardware. Also note the rewrite index, met3 index and
LTL indices for OIF’s
Cat6K#rem comm sw show mls cef ip multicast group 225.10.10.10 detail
Multicast CEF Entries for VPN#0

(172.16.25.1, 225.10.10.10)
IOSVPN:0 (1) PI:1 (1) CR:0 (1) Recirc:0 (1)
Vlan:20 AdjPtr:30 FibRpfNf:1 FibRpfDf:1 FibAddr:0x30080
rwvlans:20 rwindex:0x9BD adjmac:001d.a29a.1f00 rdt:1 E:0 CAP1:0
fmt:Mcast l3rwvld:1 DM:0 mtu:1518 rwtype:L3 met2:0x0 met3:0x26
packets:0000000000063 bytes:000000000000007434
Starting Offset: 0x0026
V E C: 10 I:0x009BF
BRKRST-3143
A closer look at the CEF entry

Pointer to location in
adjacency table where
(S,G) rewrite, LTL index and
LTL index used to
MET indices are stored
forward to all receivers
and mrouters in the
Multicast CEF Entries for VPN#0 ingress VLAN
(172.16.25.1, 225.10.10.10)
RPF VLAN
rwvlans:20 rwindex:0x9BD adjmac:001d.a29a.1f00 rdt:1 E:0 CAP1:0
fmt:Mcast l3rwvld:1 DM:0 mtu:1518 rwtype:L3 met2:0x0 met3:0x26
Egress VLAN packets:0000000000063 bytes:000000000000007434
Starting Offset: 0x0026
V E C: 10 I:0x009BF MET index used to
derive the LTL indices
to forward to all
receivers and mrouters
in all of the OIF’s in the
Result of the MET lookup OIL
using the met3 index. We
have only one OIF: VLAN 10.
0x9BF is the LTL index used
to forward to all receivers and
mrouters in VLAN 10
BRKRST-3143

14664_05_2008_c2.scr
Check adjacency for consistency
Check output of show mls cef adjacency using pointer address
from previous command to verify consistency in rewrite index and
met3 index
Cat6K#show mls cef adjacency multicast detail | begin 30
Index: 30 smac: 001d.a29a.1f00, dmac: 0000.0000.0000
mtu: 1518, vlan: 20, dindex: 0x9BD, l3rw_vld: 1
format: MULTICAST, flags: 0x2608
met2: 0, met3: 38 Same as rwindex from
show mls cef ip
multicast command
Same as met3 index

from show mls cef ip
multicast command.
Output here is in
decimal format
BRKRST-3143
Do the indices point to the correct OIF’s?
Use test platform mcast ltl index [index] to verify

correct forwarding ports in the ingress VLAN and all
OIF’s Gi9/1 is an mrouter port
in the ingress VLAN
and Gi9/4 is a receiver
Cat6K#rem comm sw test mcast ltl index 9bd port in the ingress
VLAN
index 0x9BD contain ports 5/T1, 9/1,4,T1,T2
The Tn (n=1,2,…)
Cat6K#rem comm sw test mcast ltl index 9bf entries refer to the
replication engine on
index 0x9BF contain ports 5/2,T1, 9/T1,T2 the module specified
Gi5/2 is a receiver port

in the egress VLAN,
VLAN 10
BRKRST-3143

14664_05_2008_c2.scr
Egress mode troubleshooting
In egress mode you will have two CEF entries: a primary entry PI:1 and a
non-primary or secondary entry PI:0
Cat6K#rem comm sw show mls cef ip multicast group 225.10.10.10 detail
Multicast CEF Entries for VPN#0

Primary entry
(172.16.25.1, 225.10.10.10)
rwvlans:20 rwindex:0x939 adjmac:001d.a29a.1f00 rdt:1 E:0 CAP1:0
fmt:Mcast l3rwvld:1 DM:0 mtu:1518 rwtype:L2&L3 met2:0x8A met3:0x8B
packets:0000000000049 bytes:000000000000005782
Starting Offset: 0x008A
V E L0 C:1015 I:0x0080B
Starting Offset: 0x008B
secondary entry
V E C: 10 I:0x0091B

rwvlans:1015 rwindex:0x7FFA adjmac:001d.a29a.1f00 rdt:1 E:0 CAP1:0
fmt:Mcast l3rwvld:1 DM:0 mtu:1518 rwtype:L3 met2:0x0 met3:0x8B
packets:0000000000000 bytes:000000000000000000
V E C: 10 I:0x0091B
BRKRST-3143
The primary entry is used by the ingress forwarding

engine for:
Forwarding to all receivers & mrouters in the ingress VLAN
Forwarding to all “local” receivers & mrouters on all OIF’s in the OIL
The non-primary entry is used by the egress

forwarding engines for:
Forwarding to all “local” receivers & mrouters on all OIF’s in the OIL
BRKRST-3143

14664_05_2008_c2.scr
A closer look at the primary entry
MET index used to retrieve
the LTL indices for receivers
LTL index used to and mrouters local to the
The primary entry forward to all receivers ingress replication engine.
and mrouters in the One LTL index per OIF in the
(S,G) ingress VLAN OIL
(172.16.25.1, 225.10.10.10)
RPF VLAN
rwvlans:20 rwindex:0x939 adjmac:001d.a29a.1f00 rdt:1 E:0 CAP1:0
fmt:Mcast l3rwvld:1 DM:0 mtu:1518 rwtype:L2&L3 met2:0x8A met3:0x8B
Egress VLAN
packets:0000000000049 bytes:000000000000005782
met2 block Starting Offset: 0x008A Packet and byte counts
V E L0 C:1015 I:0x0080B
should increment with MET index used to retrieve the
Starting Offset: 0x008B packets forwarded
met3 block egress replication VLAN and the
V E C: 10 I:0x0091B
LTL index used to forward a
single copy of the multicast
packet across the fabric in the
egress replication VLAN
Met3 lookup result.
Show egress VLAN 10 Met2 lookup result.
and LTL index 0x91B Shows egress
replication VLAN 1015
and LTL index 0x80B
BRKRST-3143
A closer look at the non-primary entry
The non-primary entry LTL index used to get

Egress replication
the packet to the
VLAN 1015
replication engine on
the egress module
rwvlans:1015 rwindex:0x7FFA adjmac:001d.a29a.1f00 rdt:1 E:0 CAP1:0
RPF VLAN fmt:Mcast l3rwvld:1 DM:0 mtu:1518 rwtype:L3 met2:0x0 met3:0x8B
packets:0000000000000 bytes:000000000000000000
V E C: 10 I:0x0091B MET index used by the egress
replication engine to retrieve the
LTL indices for receivers and
Met3 lookup result. mrouters local to the egress
Shows egress VLAN 10 Packet and byte counts replication engine. One LTL index
and LTL index 0x91B will always be zero on per OIF in the OIL
secondary entry
BRKRST-3143

14664_05_2008_c2.scr
Egress Mode Troubleshooting
Can check for consistency by reading the MET directly

with test mcast rd-met command
Cat6K#rem comm sw test mcast rd-met slot 9 addr 8a end 8b
Met 0x008A V E L0 C: 1015 I: 0x0080B

Met 0x008B V E C: 10 I: 0x0091B
***The slot number will be the slot with the ingress replication engine if looking at the
primary entry and the egress slot if looking at the non-primary entry
BRKRST-3143
Use test platform mcast ltl index [index] to verify

correct forwarding ports for both entries
contains only the
Cat6K#rem comm sw test mcast ltl index 80b replication engine on
the egress module
index 0x80B contain ports 5/T1
Cat6K#rem comm sw test mcast ltl index 91b contains only the port
on the egress module
index 0x91B contain ports 5/2 where the receiver in
VLAN 10 lives
What about the receiver in the ingress VLAN 20 on Gi9/4?

Remember, that’s a different LTL index. Shows Gi9/4 and Gi9/1
which is an mrouter port
Cat6K#rem comm sw test mcast ltl index 939 in VLAN 20
index 0x939 contain ports 5/T1, 9/1,4,T1,T2
BRKRST-3143

14664_05_2008_c2.scr
What if there are DFCs in the system?
In ingress replication mode, all lookups are performed by the DFC
on the ingress module
In egress replication mode:
Lookups for the original packet and all those replicated by the ingress
replication engine are performed by the DFC on the ingress module
Lookups for all packets replicated by the egress replication engine are
performed by the DFC on the egress module
For the supervisor and modules without a DFC, lookups are
performed by the PFC on the active supervisor
Troubleshooting is the same as outlined above and show
commands are the same as shown, however…
Instead of remote command switch use remote command
module [slot#]
BRKRST-3143
Show Platform Tech ipmulticast

Use show platform tech-support ipmulticast [group] [source]
Cat6K#show platform tech-support ipmulticast 225.10.10.10 172.16.25.1
show version
show running-config
show interface Vlan20 counters
show ip igmp group 225.10.10.10
show ip igmp interface Vlan20
show ip mroute 225.10.10.10
show ip mroute 225.10.10.10 count
show mls ip multicast group 225.10.10.10 source 172.16.25.1
show mls ip multicast connected
show mls ip multicast rp-mapping
remote command switch show mac address 0100.5e0a.0a0a vlan 20
remote command switch show mmls igmp process
remote command switch show mls cef ip multicast source 172.16.25.1 group 225.10.10.10 detail
remote command switch show table cbl slot 5 vlan 20
remote command switch show table fpoe slot 5 start 0x938 end 0x939
remote command switch show table fpoe slot 5 start 0x938 end 0x939 sw
remote command switch show table fpoe slot 5 start 0x80B end 0x80B
remote command switch show table fpoe slot 5 start 0x80B end 0x80B sw
remote command switch show table fpoe slot 5 start 2331 end 2331
remote command switch show table fpoe slot 5 start 2331 end 2331 sw
remote command switch test mcast ltl index 938
remote command switch test mcast ltl index 939
remote command switch test mcast ltl index 91B
----- AND MORE -----
BRKRST-3143

14664_05_2008_c2.scr
Agenda

BRKRST-3143
VSS Specific Troubleshooting

VSS: What to Check ?
VSS test topology network diagram
VSS system control plane debugs
VSS specific L2/L3 packet flow troubleshooting
BRKRST-3143

14664_05_2008_c2.scr
VSS Test Topology Network Diagram
Po1 Po2
Gig5/9 Gig2/6/12
Gig4/16 Gig1/9/36
Gig2/2 Gig2/9/15 Ten1/3/2 Ten1/1
Gig5/2 Gig1/6/2
Ten2/2/7 Ten1/4
Gig2/4 Gig1/5/1
R1 R2
8.0.1.1 DUT 9.0.1.2

DUT is a Virtual Switch System, consisting of 2 6509’s with
supervisor VS-S720-10G-3C(XL)
Running equal cost multi path routing with respectively 5 (Vlan 701
to 705) and 2 (Ten8/1 and Ten8/3) equal cost paths
BRKRST-3143
VSS System Control Plane Debugs

VSS Specific Protocols Overview
Virtual Switch Link (VSL) is special port channel
required to bundle 2 physical switches into 1 virtual
switch
VSL Protocol (VSLP) runs between active and standby
switch over the VSL, and has 2 components:
Link Maintenance Protocol (LMP): runs over each individual
link in VSL bundle
Role Resolution Protocol (RRP): runs on each side of
the VSL port channel between the 2 physical switches
Enhanced PAgP: PAgP protocol enhanced with extra

Type-Length-Value (TLV) fields
BRKRST-3143

14664_05_2008_c2.scr
VSS Quick Configuration Sanity Check
DUT#show switch virtual
Switch mode : Virtual Switch Unique domain number for each VSS
Virtual switch domain number : 1
Local switch number : 1
Local switch operational role: Virtual Switch Active
Check status for each link in VSL port channel is P
Peer switch number : 2
Peer switch operational role : Virtual Switch Standby
DUT#show switch virtual link port-channel
Flags: D - down P - bundled in port-channel
. . . Interfaces identified by <switchNr>/<modNr>/<portNr>
Group Port-channel Protocol Ports
------+-------------+-----------+-------------------
256 Po256(RU) - Te1/3/3(P) Te1/3/4(P) Te1/3/6(P)
Switch id 1 side of the VSL
Te1/5/4(P)
255 Po255(RU) - Te2/2/3(P) Te2/2/6(P) Te2/2/8(P)
Switch id 2 side of the VSL
Te2/5/4(P)
DUT#show switch virtual role
Switch Switch Status Preempt Priority Role Session ID
Number Oper(Conf) Oper(Conf) Local Remote
------------------------------------------------------------------
Switch id 1 is active, 2
LOCAL 1 UP TRUE (Y*) 200(200) ACTIVE 0 0 is standby, both are up
REMOTE 2 UP TRUE (Y*) 100(100) STANDBY 2977 3643
Standby configured preempt timer(switch 2): 5 minutes
Active configured preempt timer(switch 1): 5 minutes
Switch is not in dual active recovery mode
In dual-active recovery mode: No
BRKRST-3143

VSS: Looking at LMP
DUT#show switch virtual link
VSL Status : UP How long are we up … did VSL go down ?
VSL Uptime : 18 hours, 12 minutes
VSL SCP Ping : Pass
VSL ICC Ping : Pass
Carries EOBC and IBC control
VSL Control Link : Te1/5/4
messages (SCP and ICC/IPC)
DUT#show switch virtual link port
LMP summary
Link info: Configured: 4 Operational: 4
Peer Peer Peer Peer Timer(s)running
Interface Flag State Flag MAC Switch Interface (Time remaining)
--------------------------------------------------------------------------------
Te1/5/4 vfs operational vfs 0011.bc75.4400 2 Te2/5/4 T4(220ms)
T5(175s)
Te1/3/3 vfs operational vfs 0011.bc75.4400 2 Te2/2/6 T4(220ms) Check LMP state and Flags (vf) of
T5(175s) the links in the VSL bundle
T5(175s)
T5(175s)
Flags: v - Valid flag set f - Bi-directional flag set
s - Negotiation flag set
Timers: T4 - Hello Tx Timer T5 - Hello Rx Timer
BRKRST-3143

14664_05_2008_c2.scr
VSS: Looking at LMP
DUT#show switch virtual link port (continued)
. . .
LMP Status
Last operational Current packet Last Diag Time since
Interface Failure state State Result Last Diag
-------------------------------------------------------------------------------
Te1/5/4 No failure Hello bidir Never ran --
LMP hello timer <- LMP timer values
Any link failures detected by LMP in the past ?
Hello Tx (T4) ms Hello Rx (T5*) ms
Interface State Cfg Cur Rem Cfg Cur Rem
-------------------------------------------------------------------------
Te1/5/4 operational 5000 5000 220 180000 180000 175548
Te1/3/3 operational 5000 5000 220 180000 180000 175548
Te1/3/4 operational 5000 5000 220 180000 180000 175548
Te1/3/6 operational 5000 5000 768 180000 180000 175548
*T5 = min_rx * multiplier

Cfg : Configured Time
Cur : Current Time
Rem : Remaining Time
BRKRST-3143

VSS: Looking at LMP Problem sending Packets received
DUT#show vslp lmp counters from VSL peer
LMP packet to
Instance #1: with our info,
LMP packets tx’ed the VSL peer ? Problem
LMP counters
proving the link is
to the VSL peer bidirectional receiving LMP
packet to the VSL
Tx Rx peer ?
Interface OK Fail Bidir Uni Fail Bad
-------------------------------------------------------------------- Receiving
Te1/5/4 12649 0 12675 1 0 0 incorrect LMP
Te1/3/3 12000 0 12024 0 0 0 packets ?
Te1/3/4 11999 0 12024 0 0 0
Te1/3/6 12001 0 12025 0 0 0 Packets received from VSL
peer without our info, proving
Rx error details the link is unidirectional at
Interface My info My info Bad MAC Bad switch Domain id Peer info that moment; when the first
mismatch absent Address id mismatch mismatch link comes up, the first packet
will always be a unidir packet
-------------------------------------------------------------------------------
Te1/5/4 0 1 0 0 0 0
Te1/3/3 0 0 0 0 0 0
Te1/3/4 0 0 0 0 0 0 Why receiving
Te1/3/6 0 0 0 0 0 0 incorrect LMP
packets ……
DUT#clear vslp lmp counters ? configuration error ?
interface Interface OK, 1 unidirectional
<cr> packet when first link in
VSL came up
BRKRST-3143

14664_05_2008_c2.scr
VSS: Looking at RRP
DUT#show switch virtual role detail
------------------------------------------------------------------
LOCAL 1 UP TRUE (Y*) 200(200) ACTIVE 0 0
REMOTE 2 UP TRUE (Y*) 100(100) STANDBY 2977 3643
Standby configured preempt timer(switch 2): 5 minutes Check 1 is active,
Active configured preempt timer(switch 1): 5 minutes 1 is standby
RRP Counters:
--------------------------------------------------------------------
Inst. Peer Direction Req Acc Est Rsugg Racc
---------------------------------------------------------------------- State machine info on RRP
1 1 Tx 0 2 0 2 6 protocol; current state is “role
1 1 Rx 2 0 2 0 6 resolved”
RRP FSM info
----------------------------------------------------------------------
sm(vslp_rrp RRP SM information for Instance 1, Peer 1), running yes, state role_res
Last transition recorded: (req)-> hold (srt_exp)-> hold (est)-> role_neg (srt_exp)-> role_neg (est)->
role_neg (racc)-> role_res (srt
_. . .
In dual-active recovery mode: No Switch is not in dual active recovery mode
DUT# show vslp rrp ?
counters Counters
detail detail information Same information
fsm Finit State Machine (FSM) information as in “show
summary Summary information virtual role detail”
BRKRST-3143

VSS: Enhanced PAgP Based Dual Active Detection
What if VSL fails ?
2 separate physical switches with identical configuration on same network =
trouble
Solution: dual active detection mechanisms will put make sure only one “active”
switch has interfaces up, other active switch will be in recovery mode (all of it’s
interfaces down)
Enhanced PAgP based:
New “Dual Active” Type-Length-Value field in PAgP is used to insert info on
which physical switch is active in the VSS in PAgP packets to/from remote
switch attached to VSS
Requires:
Multi-chassis Ether Channel (MEC) with at least one interface member from
both switches in the VS, must be running the PAgP protocol, with at least one
side’s mode configured as desirable.
PAgP dual-active detection mechanism must be enabled (“dual-active detection
pagp” command)
The specific port channel must be “trusted” to be used for dual-active detection
(“dual-active detection pagp trust channel-group” command
The MEC neighbor switch must be running an image capable of supporting the
BRKRST-3143 enhanced PAgP dual-active tlvs. 136
14664_05_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

14664_05_2008_c2.scr
VSS: BFD Based Dual Active Detection
BFD based: Bfd dual-active detection requires that the pair of interfaces
being used for this method be directly connected via a cable.
Requires:
An ip address must be configured on the interface.
The two interfaces must be on a different subnet.
Bfd interval parameters must be configured on the interfaces.
The bfd dual-active detection mechanism must be enabled – this is configured
using the “dual-active detection bfd” command.
The pair of interfaces to be used in the detection mechanism must be specified
using the “dual-active pair interface” command.
The BFD neighbors are not created until the VSL fails, BFD neighbor
establishment is the trigger of the dual-active detection !!
BRKRST-3143

VSS: dual active detection setup
BFD direct connection
Gig1/6/1-Gig2/9/1
Gig1/6/1
Gig2/9/1
Po1 Po2
Gig5/9 Gig2/6/12
Ten1/3/2 Ten1/1
Gig4/16 Gig1/9/36
Gig2/2 Gig2/9/15 Ten2/2/7 Ten1/4
Gig5/2 Gig1/6/2
Gig2/4 Gig1/5/1 R2
8.0.1.1 R1 Trusted Port-channel2 DUT 9.0.1.2
Port channel 2 is the only port channel trusted for dual active detection
BFD direct connection is between Gig1/6/1 and Gig2/9/1
Both mechanisms can be on simultaneously
BRKRST-3143

14664_05_2008_c2.scr
VSS: Dual Active Detection troubleshooting
Truncated display
DUT#sh startup-config | b switch virtual
switch virtual domain 1
switch mode virtual Port channel 2 is trusted for Enhanced
PAgP dual active detection Interface pair for BFD
... dual active detection
dual-active detection pagp trust channel-group 2
dual-active pair interface GigabitEthernet1/6/1 interface GigabitEthernet2/9/1 bfd
dual-active exclude interface GigabitEthernet1/5/3
Interfaces excluded from recovery
dual-active exclude interface GigabitEthernet2/5/3
mode, they will not go down in case
! the switch ends up in recovery mode
interface GigabitEthernet1/6/1
no switchport
ip address 100.10.10.9 255.255.255.252
bfd interval 50 min_rx 50 multiplier 3 BFD configuration on interface pair for
BFD dual active detection; notice they
end
are in different subnets !! Exclude these
interface GigabitEthernet2/9/1 from redistribution in routing protocols.
no switchport
ip address 100.10.10.13 255.255.255.252
bfd interval 50 min_rx 50 multiplier 3
Automatically added with “dual-active pair”
...
command, required to be present as the
ip route 100.10.10.8 255.255.255.252 GigabitEthernet2/9/1 directly connected interfaces are in
ip route 100.10.10.12 255.255.255.252 GigabitEthernet1/6/1 different subnets !! Don’t redistribute these
static routes.
BRKRST-3143

VSS: Dual Active Detection Troubleshooting
DUT#show switch virtual dual-active summary
Pagp dual-active detection enabled: Yes
Bfd dual-active detection enabled: Yes
Interfaces excluded from shutdown in recovery mode:
Gi1/5/3
Gi2/5/3
DUT#show switch virtual dual-active pagp
PAgP dual-active detection enabled: Yes
PAgP dual-active version: 1.1
Port channel 2 is trusted for Enhanced
PAGP dual active detection … at least 1
Channel group 2 dual-active detect capability w/nbrs trusted port channel needed !!
Dual-Active trusted group: Yes
Dual-Active Partner Partner Partner
Port Detect Capable Name Port Version
Check that the neighbor runs a SW version
Gi1/5/1 Yes R1 Gi2/4 1.1
that supports Enhanced PAGP … if not, no
Gi1/6/2 Yes R1 Gi5/2 1.1 dual active detection !!
Gi1/9/36 Yes R1 Gi4/16 1.1
Gi2/6/12 Yes R1 Gi5/9 1.1
At least 1 port channel member on each
Gi2/9/15 Yes R1 Gi2/2 1.1
switch id !!
Channel group 3 dual-active detect capability w/nbrs
Dual-Active trusted group: No
. . . Port channel 3 is not trusted for Enhanced PAGP dual active detection
BRKRST-3143

14664_05_2008_c2.scr
DUT#show switch virt dual-active bfd
Bfd dual-active detection enabled: Yes
Bfd dual-active interface pairs configured:
interface-1 Gi1/6/1 interface-2 Gi2/9/1
DUT#
Triggering dual active situation by shutdown of VSL (switch id 2 side)

DUT#show int po 255 | i Member Truncated display
Members in this channel: Te2/2/3 Te2/2/6 Te2/2/8 Te2/5/4
DUT#conf t
Enter configuration commands, one per line. End with CNTL/Z.
DUT(config)#int range ten 2/2/3 , Te2/2/6 , Te2/2/8 , Te2/5/4
DUT(config-if-range)#shutdown
*Apr 1 12:40:22.885 CET: %PAGP_DUAL_ACTIVE-SW1_SP-1-RECOVERY: PAgP running on
Gi1/5/1 triggered dual-active recovery: active id 0011.bc75.4400 received, expected
0011.5d54.6800
Enhanced PAGP detected both switch id’s were active at the same time
*Apr 1 12:40:22.945 CET: %DUAL_ACTIVE-SW1_SP-1-DETECTION: Dual-active condition

detected: all non-VSL and non-excluded interfaces have been shut down
Originally active switch id (switch 1 in example) goes into recovery mode

BRKRST-3143

DUT#show switch virtual role On switch id 1: originally active, now in recovery mode
Number Oper(Conf) Oper(Conf) Local Remote Doesn’t see switch id 2
(as VSL is still down)
------------------------------------------------------------------
In dual-active recovery mode: Yes
Triggered by: PAgP detection Mechanism that detected dual active was
Triggered on interface: Gi1/5/1 Enhanced PAgP, via link 1/5/1
Received id: 0011.bc75.4400

Expected id: 0011.5d54.6800
DUT#
Alternative: “show switch virtual dual-active summary”

On switch in recovery mode, all interfaces except for the ones excluded from
recovery mode should be down: quick check via “show ip interface brief | i up” that
only the ones allowed are up
BRKRST-3143

14664_05_2008_c2.scr
Trying to bring the system back up via “no shutdown” of VSL port channel: you need
to do this on both sides (active switch in recovery mode as well as the real active
switch id 2 at this point in time)
DUT#conf t
DUT(config-if-range)#no sh
DUT(config-if-range)#
DUT#
*Apr 1 12:49:29.513 CET: %DUAL_ACTIVE-1-VSL_RECOVERED: VSL has recovered during
dual-active situation: Reloading switch 1
Switch in recover mode should reload
and come back up as standby !!
*Apr 1 12:49:29.513 CET: %VS_GENERIC-5-VS_CONFIG_DIRTY: Configuration has changed.
Ignored reload request until configuration is saved
Configuration has been modified (sh/no sh), it needs to be saved before it will recover; if not and
configurations are possibly out of sync (any “conf t” has been issued without saving while the VSS was still
up), standby mode will be RPR+ until we manually save/sync the configuration and reset standby;
BRKRST-3143

For reference: console logs on switch id 2 (standby -> active upon VSL failure)
*Apr 1 12:40:08.032 CET: %VSLP-SW2_SPSTBY-3-VSLP_LMP_FAIL_REASON: Te2/2/6: Link down
*Apr 1 12:40:12.825 CET: %VSLP-SW2_SPSTBY-3-VSLP_LMP_FAIL_REASON: Te2/2/8: Link down
*Apr 1 12:40:20.096 CET: %VSLP-SW2_SPSTBY-2-VSL_DOWN: Last VSL interface Te2/5/4
went down
*Apr 1 12:40:20.096 CET: %VSLP-SW2_SPSTBY-2-VSL_DOWN: All VSL links went down
while switch is in Standby role
*Apr 1 12:40:20.096 CET: %DUAL_ACTIVE-SW2_SPSTBY-1-VSL_DOWN: VSL is down switchover,
or possible dual-active situation has occurred
*Apr 1 12:40:20.100 CET: %PFREDUN-SW2_SPSTBY-6-ACTIVE: Initializing as Virtual
Switch ACTIVE processor
DUT#show switch virtual role
------------------------------------------------------------------
Switch id 2 is now the “real” active
switch, and doesn’t see switch id 2 as
long as the VSL is down !!
BRKRST-3143

14664_05_2008_c2.scr
For reference: console logs on switch id 2 (standby -> active upon VSL failure)
DUT#conf t
DUT(config-if-range)#no sh “Unshutting” the VSL links on both switch id’s
*Apr 1 12:49:32.781 CET: %LINK-SW2_SP-3-UPDOWN: Interface TenGigabitEthernet2/5/4
changed state to up
*Apr 1 12:49:49.128 CET: %VSLP-SW2_SP-5-VSL_UP: Ready for Role Resolution with
Switch=1, MAC=0011.5d54.6800 over Te2/2/6
*Apr 1 12:49:50.320 CET: Initializing as Virtual Switch ACTIVE processor
*Apr 1 12:49:52.140 CET: %VSLP-SW2_SP-5-RRP_MSG: Peer Switch with unsaved
configurations needs to be reloaded.
Please save relevant configurations on the peer switch and reload it.
BFD based mechanism: similar

If no dual-active detection method was enabled, and VSL recovers, RRP determines
which switch stays active and which reloads to become standby, based on the switch
number and priority configurations. In case of a “dirty configuration” (if any “conf t”
command is issued), it will put the “to-become-standby” switch into recovery and wait
for manual reload command.
BRKRST-3143

VSS: Dual Active Debug Commands
For reference, and use with caution:
debug switch virtual dual-active detect bfd events (on RP ONLY)
debug switch virtual dual-active detect summary
general dual-active debugging when going into recovery
mode
debug pagp dual-active (SP ONLY)
enable PAgP dual-active specific debugging
BRKRST-3143

14664_05_2008_c2.scr
BRKRST-3143
VSS L2/L3 Forwarding (Data Plane)

VSS Data Plane Design: Minimal Load on the VSL
Multi-chassis Ether Channel (MEC):
Modify the hash so that links on local physical switch get
preferred to transmit packet, instead of links on remote switch
Equal Cost Multi Path (ECMP):

Modify adjacency table to prefer next hops attached on local
switch; only select paths on remote switch if no local paths are
available
Knowing this, similar commands (enhanced with option

to give switch id as input)/steps can be used as in
standalone;
BRKRST-3143

14664_05_2008_c2.scr
VSS L2/L3 Forwarding Setup
VSS: Data Path Test Setup
Po1 Po2
Gig5/9 Gig2/6/12
Gig4/16 Gig1/9/36
Ten1/3/2 Ten1/1
Gig2/2 Gig2/9/15
Gig5/2 Gig1/6/2 Ten2/2/7 Ten1/4
Gig2/4 Gig1/5/1
R1 R2
8.0.1.1 DUT 9.0.1.2
DUT learns 8.0.1.0/24 via ECMP on VLAN’s 701 to 705 over port
channel 2
DUT learns 9.0.1.0/24 via ECMP on L3 interfaces Ten1/3/2 and
Ten2/2/7
Launching ping from 8.0.1.1 to 9.0.1.2
Using similar commands/steps as in Unicast L3
troubleshooting to find out the path, only VSS specifics are
highlighted in next slides
BRKRST-3143

VSS Data Plane Troubleshooting L2 MEC
Verify the load-balance algorithm used
DUT#show etherchannel load-balance switch 2 mod 2 What type of etherchannel load balancing is being used on this module ?
EtherChannel Load-Balancing Configuration:
src-dst-ip enhanced
mpls label-ip
EtherChannel Load-Balancing Addresses Used Per-Protocol: Important: depending on the type of load balancing used,
use different arguments, e.g. in case of dst-ip, only give
Non-IP: Source XOR Destination MAC address
the destination ip as argument … otherwise command
IPv4: Source XOR Destination IP address
doesn’t work correctly
IPv6: Source XOR Destination IP address
MPLS: Label or IP
Identify the physical interface flow to host 1 (out of Port-channel 2) will use
DUT#show etherchannel load-balance hash-result interface Port-channel 2 switch 2 ip 9.0.1.2 8.0.1.1
Computed RBH: 0x3
Would select Gi2/9/15 of Po2 Packet coming in on switch id 2, needing to go
out on Po2 will select Gi2/9/15
DUT#show etherchannel load-balance hash-result interface Port-channel 2 switch 1 ip 9.0.1.2 8.0.1.1
Computed RBH: 0x3
Would select Gi1/6/2 of Po2 Packet coming in on switch id 1, needing to go
out on Po2 will select Gi1/6/2
For MEC, load-balance should prefer physical interfaces local to the switch the packet was
received on
BRKRST-3143

14664_05_2008_c2.scr
VSS Data Plane Troubleshooting ECMP
Routing table shows 2 Equal Cost Paths to 9.0.1.0/24
DUT#show ip route 9.0.0.0 | i via
7.7.1.2, from 7.7.1.2, 1d00h ago, via TenGigabitEthernet2/2/7
* 7.6.1.2, from 7.6.1.2, 1d00h ago, via TenGigabitEthernet1/3/2
Looking at the HW table shows next hop directly attached to local switch is
preferred
DUT#show mls cef lookup 9.0.1.0 switch 1 mod 3
Codes: decap - Decapsulation, + - Push Label Packet coming in on switch 1 module 3, for 9.0.0.0/8
Index Prefix Adjacency prefers next hop attached to local switch id 1
108775 9.0.0.0/8 Te1/3/2 , 000f.35ed.7c00
DUT#show mls cef lookup 9.0.1.0 switch 2 mod 2
Codes: decap - Decapsulation, + - Push Label Packet coming in on switch 2 module 2, for 9.0.0.0/8
Index Prefix Adjacency prefers next hop attached to local switch id 2
108775 9.0.0.0/8 Te2/2/7 , 000f.35ed.7c00
DUT#show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 switch 1 mod 3
? ... show vlan internal usage | I 4064
Interface: Te1/3/2, Next Hop: 7.6.1.2, Vlan: 4064, Destination Mac: 000f.35ed.7c00
DUT#show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 switch 2 mod 2
Interface: Te2/2/7, Next Hop: 7.7.1.2, Vlan: 4056, Destination Mac: 000f.35ed.7c00
Further, use similar commands (enhanced with extra argument of switch id) as in
standalone switch
BRKRST-3143

BRKRST-3143

14664_05_2008_c2.scr
VSS Troubleshooting Tools
VSS Additional Useful Commands
Virtual Slot Numbers: some log messages can display virtual slot numbers, to
identify matching switch id/module number:
DUT#show switch virtual slot-map
Virtual Slot to Remote Switch/Physical Slot Mapping Table:
Virtual Remote Physical Module

Slot No Switch No Slot No Uptime
---------+-----------+----------+----------
No module present in switch id 1, slot 1
17 1 0 -
18 1 0 -
19 1 3 1d01h
Module present in virtual slot id 19, maps
20 1 0 - to switch id 1, slot 3,
Capture all info specific to VSS:

show switch virtual troubleshooting all
BRKRST-3143
Troubleshooting VSS
Problems We’ve Looked at
VSS control plane issues: VSS doesn’t form, dual
active, dirty configuration …
Checked using VSS specific commands
VSS data plane forwarding (L2/L3)

Checked what is different in VSS …
Troubleshooting VSS data plane is pretty much the

same as standalone, step-by-step, no steps
skipping !!
BRKRST-3143

14664_05_2008_c2.scr
Now What … ?
BRKRST-3143
What Did We Just Talk About?
Path verification. “Get oriented’

Great time to have good diagrams.
Looking at counters and HW forwarding tables for IP

unicast, multicast, VSS ?
Check HW/SW consistency …
There is more reference material in the appendices on:
QOS, WS-Sup32P, Modular IOS, HW Health monitoring
Still … I need TAC assistance …how about a cheat

sheet?
BRKRST-3143

14664_05_2008_c2.scr
Open Case with Cisco TAC
Collect any syslogs or tacacs logs Log your
P1/P2 Phone only! telnet to switch session to
P3/P4 – email your Desktop!!
Include:
1. Brief Description On RP
2. Bridge number terminal length 0
Catalyst 6500
3. Hostname and IP
show log Sup720 Native IOS
show clock
Supervisor show tech
Troubleshooting
Failover Show tech platform Procedure
On Route Processor (RP)
show scp accounting
show scp counters Determine Problem Type
show eobc
show ibc Routing
show ipc status Multicast
show ipc ports On RP On SP
show heartbeat Show platform tech unicast <..> show mls cef ip detail
show fabric errors show ip arp show mls cef inconsistency On RP
show fabric utilization show ip cef show platform tech-support ipmulticast <..>
show mls cef summary
show fabric channel show adjacency detail show tech ipmulticast
show ip route show ip mroute
On Switch Processor (SP) show ip ospf statistics Module show mls ip multi connected
show mls ip multi statistics
show scp accounting show ip ospf data data
show scp counters show ip ospf neigh show mls ip multi sum
show eobc On RP show mls ip multi group <ip> source <ip>
show ip bgp neighbor
show ibc show module <mod> show mls rp ip
show ip bgp summary
show earl status show idprom all detail
show ip eigrp neighbor
show earl statistics show power
show ip eigrp topology
show diagnostic result <mod>
show fabric errors traceroute <w.x.y.z> On SP
show fabric timeout show mls cef summary show mmls v g g
show ipc status show mls cef show mls cef ip multicast detail
show ipc ports show mls cef adjacency
show heartbeat
sh platform hard superman config
show platform hard tycho interrupt
BRKRST-3143
14664_05_2008_c2
Send data
© 2008 Cisco Systems, Inc. All rights reserved.
to Cisco TAC and attach to case
Cisco Public 157
Q and A
BRKRST-3143

14664_05_2008_c2.scr
Recommended Reading
Continue your Cisco Live learning experience with

further reading from Cisco Press
Check the Recommended Reading flyer for suggested
books:
Cisco LAN Switching Fundamentals (by David Barnes, Basir Sakandar)
Cisco Catalyst QoS: Quality of Service in Campus Networks (by Richard
Froom, Mike Flannagan, Kevin Turek)
Available Onsite at the Cisco Company Store

BRKRST-3143
Complete Your Online

Session Evaluation
Give us your feedback and you could win Don’t forget to activate
fabulous prizes. Winners announced daily. your Cisco Live virtual
account for access to
Receive 20 Passport points for each session all session material
evaluation you complete. on-demand and return
for our live virtual event
Complete your session evaluation online now in October 2008.
(open a browser through our wireless network Go to the Collaboration
to access our portal) or visit one of the Internet Zone in World of
stations throughout the Convention Center. Solutions or visit
www.cisco-live.com.
BRKRST-3143

14664_05_2008_c2.scr
BRKRST-3143
Appendices: Reference Materials
QoS troubleshooting
WS-SUP32P (PISA) troubleshooting
Modular IOS troubleshooting
Monitoring the health of the system (GOLD/EEM)
BRKRST-3143

14664_05_2008_c2.scr
QoS troubleshooting
BRKRST-3143
QoS Common Issues
Marking and Policing

Flow mask conflicts (micro-flow policing)
Unsupported configurations
QoS Gotcha’s
BRKRST-3143

14664_05_2008_c2.scr
Policing and Marking
Use Appropriate Traffic to Test Policing
Use traffic that will produce reliable results with a Policer
TCP traffic will yield rates below the CIR due to the slow-start algorithm
and retransmissions.
A traffic generator should be used to source and receive the traffic.
This allows complete control of the ingress rate and an accurate
measure of the egress rate.
UDP traffic can also be used since it does not use the slow-start
algorithm or suffer from retransmissions.
Any type of traffic should be OK for testing marking
BRKRST-3143

How Do I Verify Correct Policing or Marking ?
Make sure the desired traffic is passing through the
interface to which the policer is applied
Use a packet sniffer to capture the ingress traffic and check the
destination MAC address on the frames
Make sure the desired traffic is in accord with the match

criteria in the class-map
Examine the packets in the sniffer trace to see if they match
BRKRST-3143

14664_05_2008_c2.scr
Match Criteria
Are you using a supported match criteria in your
class-map?
Supported Not Supported

match precedence match cos
match dscp match class-map
match access-group match source-address
match destination-address
match input-interface
match qos group
BRKRST-3143

Port-Based and VLAN-Based QOS
Make sure the ingress interface is configured for port-
based or VLAN-based QoS in agreement with how the
service-policy is applied.
!
policy-map police-host-to-host interface GigabitEthernet9/1
class host-to-host switchport
police cir 9000000 bc 281250 be 281250 switchport access vlan 20
conform-action set-dscp-transmit cs5 switchport mode access
exceed-action drop violate-action drop mls qos vlan-based
!
interface Vlan20 Cat6K#show mls qos
ip address 20.20.20.1 255.255.255.0 QoS is enabled globally
ip pim sparse-dense-mode Policy marking depends on port_trust
load-interval 30 QoS ip packet dscp rewrite enabled globally
service-policy input police-host-to-host Input mode for GRE Tunnel is Pipe mode
Input mode for MPLS is Pipe mode
QoS is vlan-based on the following interfaces:

Gi9/1
BRKRST-3143

14664_05_2008_c2.scr
How Do I Verify the Correct Policing or Marking ?
Use show mls qos last [module [slot]] to see a snapshot of the last packet
switched by the hardware.
Cat6K#show mls qos last
----- Module [5] -----

QoS last packet policing information:
Shows that the last
---------------------------------------------------------------------
packet was dropped
Packet was dropped Shows input and output
Packet L3 Prot: 0, packet length: 1518, dont_plc: No
CoS, ToS and DSCP
Input COS: 0, TOS/DSCP: 0x0/0
Output TOS/DSCP: 0xA0/40[rewritten] Output COS: 5[rewritten] for the packet
Output MPLS EXP (if outgoing packet is MPLS): 5
---------------------------------------------------------------------
Aggregate policer index: Input - 1, Output - 0(none)
thr_hi_ip: 0x44D leak_hi_ip: 0x233 drop_ena_ag_ip: Yes
thr_lo_ip: 0x44D leak_lo_ip: 0x233
thr_hi_op: 0x0 leak_hi_op: 0x3FF drop_ena_ag_op: No
thr_lo_op: 0x0 leak_lo_op: 0x3FF
---------------------------------------------------------------------
Microflow policer index: Input - 0(none), Output - 0(none)
--------------------------------------------------------------------- Shows last packet
Netflow policer: nf_hit: Yes, nf_addr: 0x83, snap-shot matches Source ==> destination
NT&NS: l3_prot: 1(0), 172.16.25.1.0x0000 ==> 10.10.10.100.0x0000
along with L4 port
numbers
Note: May be difficult to catch the interesting traffic as this will show the last packet
switched in hardware.
BRKRST-3143
Flow Mask Conflicts

Netflow
The NetFlow feature collects traffic statistics about the
packets that flow through the switch and stores the
statistics in the NetFlow table.
NetFlow Table: Resides on the PFC and is divided
into two pieces
Netflow Key Table: Stores the actual flows
Netflow Statistics Table: Stores the flow information like
number of packets and number of bytes switched per flow
BRKRST-3143

14664_05_2008_c2.scr
Flow Mask Conflicts
What Is a Flow ?
A flow is identified using the following fields:
Source IP Address
Destination IP Address
Source TCP/UDP Port Number
Destination TCP/UDP Port Number
IP Protocol Type
Input VLAN
Which fields are used to identify and store flows in the
NetFlow table?
The PFC uses a flow mask to identify which of the
fields are used to identify and store the flow
BRKRST-3143
Flow Mask Conflicts

Flow Masks
IP
VLAN SRC IP DST IP Protocol Src Port Dst Port Full-Interface
IP
VLAN SRC IP DST IP Protocol Src Port Dst Port Full
IP
VLAN SRC IP DST IP Protocol Src Port Dst Port Destination-Source-Interface
IP
VLAN SRC IP DST IP Protocol Src Port Dst Port Source-only
IP
VLAN SRC IP DST IP Protocol Src Port Dst Port Destination
IP
VLAN SRC IP DST IP Protocol Src Port Dst Port Destination-Source
BRKRST-3143

14664_05_2008_c2.scr
Flow Mask Conflicts
Netflow Features
Several Catalyst 6500 features use the NetFlow Table
for their operation. These include:
Micro-flow Policing TCP-Intercept

NetFlow Data Export (NDE) WCCP
IOS-SLB CBAC
Reflexive ACLs NAT/PAT
Each feature requires a specific flow mask when

configured
These requirements can conflict with one another
BRKRST-3143
Flow Mask Conflicts

Conflicts with Micro-Flow Policing
If any of the NetFlow features are configured along with
a micro-flow service policy on a given interface, the flow
mask for the micro-flow policer must be Full
When a conflict occurs with NDE, the first feature
configured will take precedence and the later will get a
flow mask conflict
When a conflict occurs with one of the other NetFlow
features, micro-flow policing will take precedence and
the other feature will be processed in software
BRKRST-3143

14664_05_2008_c2.scr
Flow Mask Conflicts
How Do I Know There Is a Conflict ?
When you configure the second feature that causes
conflict with a previously existing feature a log message
will be generated.
Cat6K(config)#int vlan 20
Cat6K(config-if)#service-policy input police-host-to-host
Cat6K(config-if)#
QoS-ERROR: QoS policy on interface Vl20 cannot be successfully installed
due to the interaction with other feature configuration
Failure reason is Unresolvable flowmask conflict with other features
QoS-ERROR: installation of policy on Vl20 failed
5w2d: %FM-2-FLOWMASK_CONFLICT: Features configured on interface Vlan20

have conflicting flowmask requirements, traffic may be switched in
software
BRKRST-3143
Flow Mask Conflicts

How Can I Tell What Feature Is the Cause of the
Problem ?
Cat6K#show fm interface vlan 20
Interface: Vlan20 IP is enabled
hw_state[INGRESS] = not reduced, hw_state[EGRESS] = not reduced
mcast = 1
priority = 0
flags = 0x0
parent[INGRESS] = none
inbound label: 35
Feature FM_GUARDIAN:
Features Bumping the flowmask on the interface:
NDE
Feature NAT_INGRESS:
In this case
it’s NDE
BRKRST-3143

14664_05_2008_c2.scr
Unsupported Features
The following are unsupported policy map class

commands for PFC/DFC QoS
Bandwidth
Priority
Queue-limit
Random-detect
Set qos-group
Service policy (nested policies are not supported)
BRKRST-3143
Unsupported Features
None of the following are supported on Ethernet

modules
CBWFQ
LLQ
WRED
Class-based Shaping
Hierarchical Traffic Shaping
***All are supported on OSM’s, FlexWAN and Enhanced FlexWAN. See QoS
configuration guide for OSM and FlexWAN modules for specific Caveats
BRKRST-3143

14664_05_2008_c2.scr
QOS Gotchas
Remember the Following Points !!
Each PFC and DFC polices independently
This will affect policers applied to port-channels and SVI’s
Egress policing is applied at the ingress interface

The ingress PFC/DFC makes the policing decision NOT the
egress PFC/DFC
Ingress and Egress policing applied to the same traffic

must have the same policy
Both must mark down or both must drop
Egress ACL uses ingress marking by default
BRKRST-3143
QoS troubleshooting
BRKRST-3143

14664_05_2008_c2.scr
WS-Sup32P – PISA
Programmable IP Services Accelerator (PISA)
Generic L2/L3 troubleshooting is similar to
previous sections
PISA specific troubleshooting focus is on packets
going to/coming from the PISA daughter card
What tables to look at
What counters to look at
BRKRST-3143
WS-Sup32P – PISA
Programmable IP Services Accelerator (PISA)
Packet back to EARL for FIB
CDE redirects packets to NP or RP lookup (ingress case) or L2
lookup /VACL (egress case)
GE Uplinks
Supervisor Engine 32
Baseboard
DRAM
DRAM
512 SP
SP CPU
CPU 1 Gbps
512 MB
MB
Port counters L3/4 Engine
DRAM
DRAM 1 Gbps
RP
RP CPU
CPU Classification Port ASIC
11 GB
GB Classification
10G and
and Dispatch
Dispatch L2/L3 Engine counters
1-3G & tables
Network Engine
Engine PISA
PISA
NP counters PISA Channel
Process
DRAM
DRAM
Micro Engines
or CDE counters
L2 Engine
768
768 MB
MB PFC3B
Replication
32M
32M Daught
Engine
SRAM
SRAM PISA Daughter er
CPU Card
Card
Bus
Up to 3 Gbps internal
Network Processor EtherChannel interface
Accelerates NBAR and Incoming packet on bus gets redirected to
for PISA connection
FPM at up to 2 Gbps PISA based on PFC3B ACL redirect
(Po256)
BRKRST-3143 (ingress) or modified FIB entry (egress)

14664_05_2008_c2.scr
WS-Sup32P – PISA
Troubleshooting Sequence
Verify the internal PISA port channel (Po256 is up)
Verify L2/L3 Forwarding Engine on PFC3B redirects
packets correctly to PISA
Look at the internal port channel counters (both port
ASIC as CDE side) and PISA specific tables
Some useful commands and tools
BRKRST-3143
WS-Sup32P
Is the Internal Port Channel to PISA OK ?
DUT#sh int po 256 Truncated output
Port-channel256 is up, line protocol is up (connected)
input flow-control is on, output flow-control is on
Members in this channel: Gi6/8 Gi6/10
DUT#sh running-config Truncated output
interface Port-channel256
mtu 4160
Verify flowcontrol, MTU and pisa-
…
channel configuration on port channel
flowcontrol receive on interface, and its physical members;
flowcontrol send on
pisa-channel
interface GigabitEthernet6/8
mtu 4160
…
flowcontrol receive on
flowcontrol send on
no cdp enable
channel-group 256 mode on
Extra command: show etherchannel 256 detail

BRKRST-3143

14664_05_2008_c2.scr
WS-Sup32P
Do the Packets Get Punted to PISA ?
DUT#show class-map TELNET-traffic
Class Map match-all TELNET-traffic (id 2)
Match protocol telnet
DUT#show policy-map Configuration just for illustration
Policy Map Vlan701
Class TELNET-traffic
set dscp af42
DUT#conf t
DUT(config)#int vlan 701
Applied service policy to Vlan 701
DUT(config-if)#service-policy input Vlan701
DUT(config-if)#
04:24:24: %PISA-6-NBAR_ENABLED: feature accelerated on input direction of: Vlan701
04:24:24: %PISA-6-NBAR_ENABLED: feature accelerated on output direction of: Vlan701^Z
DUT#
DUT#sh tcam interface vlan 701 acl in ip
* Global Defaults shared There is a “policy-route” type entry programmed in TCAM, meaning
packet matching this will get redirected to PISA module
Entries from Bank 0
Entries from Bank 1
permit ip any 224.0.0.0 15.255.255.255 (105 matches)
policy-route ip any any (110 matches)
deny ip any any
BRKRST-3143
WS-Sup32P
Do the Packets Get Punted to PISA ?
DUT#sh tcam interface vlan 701 acl in ip detail
Interface: 701 label: 1537 lookup_type: 0 Truncated output
protocol: IP packet-type: 0
+-+-----+---------------+---------------+---------------+---------------+-------+---+----+-+---+--+---+---+
|T|Index| Dest Ip Addr | Source Ip Addr| DPort | SPort | TCP-F |Pro|MRFM|X|TOS|TN|COD|F-P|
+-+-----+---------------+---------------+---------------+---------------+-------+---+----+-+---+--+---+---+
V 36250 0.0.0.0 0.0.0.0 P=0 P=0 ------ 0 ---- 1 0 -- C-- 0-0 <-
M 36251 0.0.0.0 0.0.0.0 0 0 0 ---- 1 0 <-
R rslt: REDIRECT_ADJACENCY (*) rtr_rslt: PERMIT_RESULT (*) indx: 0x7E03 hit_cnt=118 <-
DUT#show mls cef adjacency entry 0x7F803 detail Calculate redirect index: 0x7E03 – 0x7E00 + 0x7F800 = 0x7F803
Index: 522243 mtu: 65535, vlan: 0, dindex: 0x340, l3rw_vld: 1
format: RECIR, flags: 0xA0000001000E00
packets: 140, bytes: 8960 Gets redirected to internal port index 0x340,
matching the port channel 256 to PISA module
DUT#show table ltl module 6 start 0x340 end 0x340
LTL indexes from: 0x340 to 0x340 - slot: 6
Index Ports
Index 0x340 maps to interfaces 8 and 10 on module 6 (so
---------+----------------------------------------------------
Gi6/8 and Gi6/10), matches Po256 Members !! …. If not
0x00340 8,10 correct, packets won’t get punted to NP
DUT#show interface Port-channel 256 | i Members
Members in this channel: Gi6/8 Gi6/10
BRKRST-3143

14664_05_2008_c2.scr
WS-Sup32P – PISA
Troubleshooting Sequence
Verify the internal PISA port channel (Po256 is up)
Verify L2/L3 Forwarding Engine on PFC3B redirects
packets correctly to PISA
Look at the internal port channel counters (both
port ASIC as CDE side) and PISA specific tables
Some useful commands and tools
BRKRST-3143
WS-Sup32P
Do the Packets Get Out of Port Channels to PISA?
DUT#show interface port-channel 256 counters
DUT#show interface port-channel 256 counters errors Check counters on internal port channel
DUT#show interface port-channel 256 are moving, any errors … ?
…
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max) Indication CDE is flow controlling towards the port
30 second input rate 266000 bits/sec, 211 packets/sec ASIC, because e.g. NP is too busy
30 second output rate 273000 bits/sec, 211 packets/sec
DUT#show flowcontrol interface gi6/8

Check the flow controlling status of the individual
links in the internal port channel, if we get flow
Port Send FlowControl Receive FlowControl RxPause TxPause controlled, it indicates the CDE is backed up
admin oper admin oper because e.g. NP is too busy
----- -------- -------- -------- -------- ------- -------
Gi6/8 on on on on 0 0
DUT#show flowcontrol interface gi6/10

Port Send FlowControl Receive FlowControl RxPause TxPause
admin oper admin oper
----- -------- -------- -------- -------- ------- -------
Gi6/10 on on on on 0 0
BRKRST-3143

14664_05_2008_c2.scr
WS-Sup32P
Do the Packets Get into the CDE ?
DUT#show platform hardware pisa cde counters
GIGMAC0 RX-Counter = 221348961 Truncated output
GIGMAC1 RX-Counter = 0
GIGMAC2 RX-Counter = 1813
Interface counters for GIGMAC’s on CDE side of
GIGMAC3 RX-Counter = 79673 internal port channel .. Do they move ?
GIGMAC0 TX-Counter = 221348817

…
GIGMAC0 RX-DRP-CNT = 0
…
GIGMAC0 RX-UNSZ-CNT = 0
CDE to NP (IXP 2800 complex) and IXP to CDE
count … Do they move ? Errors ?
…
GIGMAC0 RX-OVSZ-CNT = 0
…
GIGMAC0 TX-DRP-CNT = 0
…
SPI-CDP-TO-IXP TX-Cnt = 221256824
SPI-IXP-TO-CDP TX Cnt = 221256823 RP to CDE and CDE to RP count … Do they move ? Errors ?
SPI-IXP Chan0-ERR-Cnt = 0
SPI-IXP Chan8-ERR-Cnt = 0
RP-TO-CDEP-FIFO-Cnt = 1386
CDEP-TO-RP-FIFO-Cnt = 79930
CDEP-RP-FIFO-CRC-Cnt = 0
BRKRST-3143
WS-Sup32P
Looking at the NP Counters ?
DUT#show platform hardware pisa np ?
ME ME Counters Truncated output
acl Access-list
all All
RX = what comes in from CDE, TX, what goes back to
fpm Flexible Packet Matching Info CDE
mqc Modular QoS CLI Info
nbar Network Based Application Recognition Info
rx Receive Engine Info
tx Transmit Engine Info
DUT#show platform hardware pisa np nbar counters
NBAR Statistics(ME2)
In our example, we did reclassification based on
NBAR for telnet
--------------------
NBAR Pkts Received : 325
NBAR Pkts Classified: 325
PD Pkts Received : 0
NBAR Pkts Out : 325
NBAR Debug 0 : 82
NBAR Debug 1 : 81
NBAR Debug 2 : 81
NBAR Debug 3 : 81
BRKRST-3143

14664_05_2008_c2.scr
WS-Sup32P
Looking at the Split VLAN … Packets leaving NP will be in a VLAN
different from ingress VLAN, but
DUT#sh platform software pisa split-vlan interface vlan 701 associated with ingress VLAN: split
Codes: P - NBAR PD, N - NBAR, F - FPM, 0x380 - RP, 0x340 - IXP VLAN
Interface Vlan PisaVlan InFeat OutFeat DestIndex State
-------------------------------------------------------------------------------------
Vlan701 701 1022 N - - N - - 0x340 up
DUT#show mac-address-table vlan 1022
Legend: * - primary entry Verify the split VLAN has the router MAC programmed,
age - seconds since last seen After this the modified packet will get L3 switched by PFC3B
n/a - not available

------+----------------+--------+-----+----------+--------------------------
* --- 0006.52b4.8000 static No - Router
For egress side features, the only difference in the troubleshooting

step is the fact that will be no Redirect ACL, but an internal vlan to
which the FIB entry for the egress interface points, that is used to
send the packet to the PISA channel.The egress feature is then
applied and packt goes back to EARL for applying L2 features (like
VACL) and forwarding. I.e. adjacency info for next hop will rewrite into split VLAN to send the
egress (from PFC viewpoint) packet to PISA to apply egress feature
BRKRST-3143
WS-Sup32P Troubleshooting Tools

Usefull Commands and Tools
Providing info w.r.t. PISA HW status:
show platform hardware pisa health
Get an idea on overall load on PISA:

show platform hardware pisa np all
In case of a performance problem, e.g. BGP is flapping due to PISA

channel congestion, or to simply skip certain type of traffic from being
sent to PISA, user can enable the “SKIP ACL’ feature:
DUT# show run int vlan 701
interface Vlan1
ip address 7.1.1.1 255.255.255.0
no ip redirects
platform ip features pisa access-group skip_bgp
service-policy input Vlan701 Use deny acl entry to skip BGP, match
all entry at bottom of access list is
DUT# show access-list skip_bgp
needed to send all the rest to PISA;
Extended IP access list skip_bgp remember, this is a debug tool !!
10 deny tcp any any eq bgp
20 permit ip any any
BRKRST-3143

14664_05_2008_c2.scr
QoS troubleshooting
BRKRST-3143
Typical Problems with Modular IOS
Process Crash
Memory Leak
High CPU utilization
BRKRST-3143

14664_05_2008_c2.scr
Crashes
Crashes will require TAC involvment

Open a TAC service request and collect the following
info:
1. Crashinfo file
2. Core file (if configured so)
3. Show tech-support
4. What you were doing that made it crash!!
BRKRST-3143
Example of Process Crash Output

Crashing process ID Crashing process name
00:05:29: %DUMPER-3-PROCINFO: pid = 16427: (sbin/tcp.proc), terminated due to signal SIGTRAP,

trace trap (not reset when caught) (Signal from user)
00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: zero at v0 v1
00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: R0 00000000 00000000 00000004 00000000
00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: a0 a1 a2 a3
00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: R4 7BC22298 00000000 00000000 00000000
00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: t0 t1 t2 t3
00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: t4 t5 t6 t7
00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: s0 s1 s2 s3
00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: R16 00FDDFA0 00000000 00000000 00000000
00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: s4 s5 s6 s7
00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: t8 t9 k0 k1
00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: R24 00000000 722B3F4C 00000000 00000000
00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: gp sp s8 ra Crashinfo
00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: R28 7828FF90 00FDDF60 00000000 72297450
filename
00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: sr lo hi bad
00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: R32 1001FC73 00000000 00000000 78288970 and location
00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: cause pc epc
00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: R36 00800020 722B3F5C 00000000
00:05:29: %DUMPER-3-TRACE_BACK_INFO: 16427: (libc.so+0x2EF5C) (libc.so+0x12450) (s72033_rp-
adventerprisek9_wan-58-dso-p.so+0x17C00) (libc.so+0x127AC)
00:05:30: %DUMPER-3-CRASHINFO_FILE_NAME: 16427: Crashinfo for process sbin/tcp.proc at
Core
bootflash:/crashinfo_tcp.proc-20050910-012841
00:05:30: %DUMPER-3-CORE_FILE_NAME: 16427: Core for process sbin/tcp.proc at filename
disk0:/tcp.proc.012842.dmp.Z and location
00:05:31: %DUMPER-5-DUMP_SUCCESS: 16427: Core dump success
00:05:31: %SYSMGR-3-ABNORMTERM: tcp.proc:1 (jid 91) abnormally terminated, restarted scheduled
BRKRST-3143

14664_05_2008_c2.scr
Example of What Files to Collect After Crash
For previous slide tcp.proc process crash you need to

collect the following files:
Cat6K#dir bootflash:
Directory of bootflash:/
4 -rw- 139528 Sep 9 2008 19:28:42 -06:00 crashinfo_tcp.proc-20050910-012841
65536000 bytes total (64979832 bytes free)
Cat6K#dir disk0:
Directory of disk0:/
1 -rw- 111923344 Sep 1 2008 10:26:54 -06:00 s72033-adventerprisek9_wan_dbg-

vz.PP_R31_INTEG_050829
2 -rw- 112078968 Sep 9 2008 14:50:54 -06:00 s72033-adventerprisek9_wan_dbg-
vz.pikespeak_r31_0908_1
3 -rw- 107608208 Sep 9 2008 18:50:04 -06:00 s72033-adventerprisek9_wan-vz.122-
99.SX1010
4 -rw- 131517 Sep 9 2008 19:28:42 -06:00 tcp.proc.012842.dmp.Z
512040960 bytes total (180281344 bytes free)
Both
filenames
Crashinfo
encode the
filename
process that
and location
crashed
BRKRST-3143
Restarting a Process
To restart a process use the command process restart [process]
Cat6K#process restart tcp.proc

Restarting process tcp.proc
Cat6K#
03:47:08: %SYSMGR-6-RESPAWN: Process tcp.proc:1 has been respawned :
sysmgr.proc : (PID=20498, TID=14) : -Traceback=(s72033_rp-
ipservices_wan-57-dso-p.so+0x11364) ([36:0]+0x134FC) ([36:0]+0xB418)
([25:-9]1+0x167C) ([35:0]+0x39B4) ([35:0]+0x3F48) ([0:-
3]libc+0x252D4) ([7:0]+0x127AC)
Cat6K#
Restarting the process produces a log message stating that the process has been
respawned and a traceback
Use show processes detailed [process] to see that a process has been restarted
BRKRST-3143

14664_05_2008_c2.scr
Restarting a Process
New
Process ID
Cat6K#show processes detailed tcp.proc
Job Id: 97
PID: 45097 Process
Executable name: tcp.proc name
Executable Path: sbin/tcp.proc
Instance ID: 1
Respawn: ON Number of
Respawn count: 4 times
process has
Respawn since last patch: 4
restarted
Max. spawns per minute: 30
Last started: Tue Apr 8 23:58:45 2008
Process state: Run
Process Redundancy State: Active (last exit status : 2)
Core: SHAREDMEM MAINMEM
Max. core: 0
Mandatory: ON User who restarted the
Last restart userid: user1 process. Requires AAA or
local login enabled
BRKRST-3143
Configuring a Core Dump
Use the exception flash command to enable a core file

collection process
Cat6K(config)#exception flash ?
bootflash: Device name
disk0: Device name
disk1: Device name
sup-bootflash: Device name
Up to 3 choices for file location are supported

Will try each location, in order, until saved or runs out of
choices
BRKRST-3143

14664_05_2008_c2.scr
Memory Leak
Memory leaks will also require TAC involvment

Open a TAC service request and collect the following
info (several interations):
1. Show clock*
2. Show memory
3. Show process memory detailed
Do several iterations of the above commands
* Show clock will give an indication of the leak rate
BRKRST-3143
Show Memory
Show memory gives a high level view of the leak
Cat6K#show clock
*01:39:31.399 UTC Wed Apr 9 2008
Cat6K#show memory
System Memory: 524288K total, 282464K used, 241824K free, 1000K kernel reserved
Lowest(b) : 233308160
Look the used and free memory as a first indication of

a problem
BRKRST-3143

14664_05_2008_c2.scr
Show Process Memory Detailed
Show process memory detailed gives a more granular

view
Cat6K#show processes memory detailed
System Memory : 524288K total, 282464K used, 241824K free, 1000K kernel reserved
Lowest(b) : 233308160
<SNIP>
Process sbin/ios-base, type IOS, PID = 24600
156592K total, 59376K text, 31412K data, 76K stack, 65728K dynamic
Heap : 67108864 total, 42759560 used, 24349304 free
Task TTY Allocated Freed Holding Getbufs Retbufs TaskName
0 0 50898384 7511824 40820144 0 0 *Init*
Task ID
for CEF 0 0 45294808 44231528 1021560 0 0 *Neutrino*
process 182 0 913424 94112 934656 0 0 FM core
0 0 14614384 13934376 658832 4267800 0 *Dead*
170 0 466944 20456 432288 0 0 CEF process
31 0 261024 288 270752 120600 0 EEM ED Syslog
2 0 7067400 6980032 145408 0 0 Service Task
274 0 84672 18024 127328 0 0 QM Process on R
29 0 191888 2168 101112 0 0 IPC Seat Manage
19 0 122136 29608 99576 0 0 Entity MIB API
43 0 174464 52016 92584 0 0 rf proxy rp age
10 0 23066464 22885360 74328 0 0 Exec
140 0 22752 27848 61016 0 0 HWIF QoS Proces
276 0 192 192 61016 0 0 QM Timer ACL Pr
<SNIP>
BRKRST-3143
Show Process Memory Detailed
Using the Task ID from the previous output allows us to

drill down further to get the program counter value
Cat6K#show processes memory detailed ios-base taskid 170
System Memory : 524288K total, 282464K used, 241824K free, 1000K kernel reserved
Lowest(b) : 233308160
156592K total, 59376K text, 31412K data, 76K stack, 65728K dynamic
Memory Summary for TaskID = 170

Holding = 432288
PC 0x75A6C430 is the 320056
largest contributer
PC Size Count PC value has to be
93290
0x75A6C430 320056 1 interpreted by the TAC 6744
0x75A5CE54 93280 20 6184
0x75CBBDC8 6744 1 3056
0x73D12644 6184 1 Sum of sizes 1600
0x75A69EDC 3056 1 equals holding 640
0x75A5CE20 1600 20 value 208
0x73D3479C 640 1 192
0x75CBBCFC 208 1 168
0x73D15324 192 1 + 160
0x73D40F48 168 1
432288
0x73CA404C 160 2
BRKRST-3143

14664_05_2008_c2.scr
High CPU Utilization
Check high level CPU with show process cpu*
Cat6K#show process cpu | exclude 0.0

CPU utilization for five seconds: 63%; one minute: 54%; five minutes: 50%
PID 5Sec 1Min 5Min Process
1 0.1% 0.3% 1.1% kernel
24600 55.6% 47.6% 43.2% ios-base
24615 7.1% 6.0% 5.2% raw_ip.proc
ios-base process is taking the majority
* Use pipe option with exclude 0.0 to eliminate the irrelevant output
BRKRST-3143
High CPU Utilization
Now use show processes cpu detailed [process] to

narrow down further
Cat6K#show processes cpu detailed ios-base | exclude 0.0
CPU utilization for five seconds: 61%; one minute: 57%; five minutes: 53%
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
24600 52.7% 49.1% 45.2% ios-base 17m38s
1 1.9% 1.9% 1.7% 10 Receive 29.961
4 4.9% 7.0% 6.7% 10 Receive 53.240
7 15.6% 14.9% 13.9% 21 Intr 4m10s
8 0.2% 0.2% 0.2% 22 Intr 85.812
12 10.8% 7.5% 4.6% 10 Reply 84.788
13 8.2% 8.0% 6.5% 10 Receive 2m11s
16 0.1% 1.3% 2.3% 10 Receive 88.128
17 11.0% 8.0% 6.7% 10 Receive 98.316
CPU utilization for five seconds: 27%/25%; one minute: 24%; five minutes: 21%
Task Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Task Name
2 76645 2597348 29 4.07% 4.01% 3.64% 0 Service Task
3 176849 2489254 71 13.19% 11.80% 10.34% 0 Service Task
11 40279 3829 10519 0.37% 0.13% 0.12% 0 Check heaps
126 126079 1311720 96 9.01% 7.95% 7.14% 0 IP Input
BRKRST-3143

14664_05_2008_c2.scr
QoS troubleshooting
BRKRST-3143
HW Installs, Moves, Changes
Deploying new hardware?

Hardware troubles most common during changes.
Weekend chassis install.
Weekend config changes.
Late night line card replacement.
What can we do to make these evolutions less painful?
BRKRST-3143

14664_05_2008_c2.scr
Generic Online Diagnostics
What Is Gold?
Gold defines a common framework for
diagnostics operations across Cisco
platforms running Cisco IOS Software
Goal: check the health of hardware components
and verify proper operation of the system data
plane and control plane at run-time and boot-time Si
Provides a common CLI and scheduling for field
diagnostics including:
Bootup Tests (includes online insertion)

Health Monitoring Tests (background non-disruptive)
On-Demand Tests (disruptive and Non-disruptive)
User Scheduled Tests (disruptive and Non-disruptive)
CLI access to data via Management Interface
BRKRST-3143

How Does Gold Work?
Diagnostic packet switching
tests verify that the system
is operating correctly: Forwarding
Engine
Is the supervisor control plane and
forwarding plane functioning properly? Line
Card
Is the standby supervisor ready to
Fabric
take over? Forwarding
Engine CPU
Are line cards forwarding packets
properly?
Active Supervisor
Are all ports working?
Is the backplane connection working? Standby Supervisor
Other types of diagnostics tests

including memory and error
correlation tests are also available Line
Card
BRKRST-3143

14664_05_2008_c2.scr
What Type of Failure Does Gold Detect?
Diagnostics capabilities
built in hardware
Depending on hardware,
Gold can catch:
Port failure
Bent backplane connector
Bad fabric connection
Malfunctioning forwarding engines
Stuck control plane
Bad memory
—
BRKRST-3143

Diagnostic Operation
Boot-Up Diagnostics Run During System Bootup, Line

Card OIR Or Supervisor Switchover
Switch(config)# diagnostic bootup level complete Makes Sure Faulty Hardware Is
Taken out of Service
Runtime Diagnostics
Health-Monitoring
Switch(config)# diagnostic monitor module 5 test 2 Non-Disruptive Tests Run
Switch(config)# diagnostic monitor interval module 5 test 2 00:00:15 in the Background
Serves As HA Trigger
On-Demand
Switch# diagnostic start module 4 test 8
Module 4: Running test(s) 8 may disrupt normal system
operation
Do you want to continue? [no]: y All diagnostics tests can be run
Switch# diagnostic stop module 4
on demand, for troubleshooting
purposes. It can also be used as a
Scheduled pre-deployment tool.
Switch(config)# diagnostic schedule module 4 test 1
port 3 on Jan 3 2005 23:32
Switch(config)# diagnostic schedule module 4 test 2 Schedule Diagnostics Tests, for
daily 14:45
Verification and Troubleshooting
Purposes
BRKRST-3143

14664_05_2008_c2.scr
Using Diagnostics as a Pre-Deployment Tool
The Order in Which Tests Are Run Matters
Run diagnostics first on line cards, then on supervisors
Run packet switching tests first, run memory tests after
Switch# diagnostic start module 6 test all

Module 6: Running test(s) 8 will require resetting the line card after the test has completed
Module 6: Running test(s) 1-2,5-9 may disrupt normal system operation
Do you want to continue? [no]: yes
*Mar 25 22:43:16: %DIAG-SP-6-TEST_RUNNING: Module 6: Running TestTransceiverIntegrity{ID=1} ...
*Mar 25 22:43:16: %DIAG-SP-3-TEST_SKIPPED: Module 6: TestTransceiverIntegrity{ID=1} is skipped
*Mar 25 22:43:16: %LINK-5-CHANGED: Interface GigabitEthernet6/1, changed state to administratively down
*Mar 25 22:43:16: %DIAG-SP-6-TEST_RUNNING: Module 6: Running TestLoopback{ID=2} ...
*Mar 25 22:43:16: %DIAG-SP-6-TEST_RUNNING: Module 6: Running TestAsicMemory{ID=8} ...
*Mar 25 22:43:16: SP: ******************************************************************
*Mar 25 22:43:16: SP: * WARNING:
*Mar 25 22:43:16: SP: * ASIC Memory test on module 6 may take up to 2hr 30min.
*Mar 25 22:43:16: SP: * During this time, please DO NOT perform any packet switching.
*Mar 25 22:43:16: SP: ******************************************************************
<snip>
Switch# diagnostic start module 5 test all
Module 5: Running test(s) 27-30 will power-down line cards and standby supervisor should be power-down
manually and supervisor should be reset after the test
Module 5: Running test(s) 26 will shut down the ports of all linecards and supervisor should be reset
after the test
Module 5: Running test(s) 3,5,8-10,19,22-23,26-31 may disrupt normal system operation
Do you want to continue? [no]: yes
<snip>
BRKRST-3143

Catalyst Gold Operation Example
Switch# show diagnostic content mod 5
Module 5: Supervisor Engine 720 (Active)‫‏‬
<snip>
Testing Interval
ID Test Name Attributes (day hh:mm:ss.ms)‫‏‬
==== ================================== ============ =================
1) TestScratchRegister -------------> ***N****A*** 000 00:00:30.00
2) TestSPRPInbandPing --------------> ***N****A*** 000 00:00:15.00
3) TestTransceiverIntegrity --------> **PD****I*** not configured
4) TestActiveToStandbyLoopback -----> M*PDS***I*** not configured
5) TestLoopback --------------------> M*PD****I*** not configured
6) TestNewIndexLearn ---------------> M**N****I*** not configured
Diagnostics test suite attributes:
7) TestDontConditionalLearn --------> M**N****I*** not configured
M/C/* - Minimal bootup level test / Complete bootup
8) TestBadBpduTrap -----------------> M**D****I***
level not
testconfigured
/ NA
9) TestMatchCapture ----------------> M**D****I*** B/*not configured
- Basic ondemand test / NA
10) TestProtocolMatchChannel --------> M**D****I***
P/V/*not configured
- Per port test / Per device test / NA
11) TestFibDevices ------------------> M**N****I***
D/N/*not configured test / Non-disruptive test / NA
- Disruptive
12) TestIPv4FibShortcut -------------> M**N****I*** S/*not configured
- Only applicable to standby unit / NA
13) TestL3Capture2 ------------------> M**N****I*** X/*not configured
- Not a health monitoring test / NA
14) TestIPv6FibShortcut -------------> M**N****I*** F/*not configured
- Fixed monitoring interval test / NA
15) TestMPLSFibShortcut -------------> M**N****I*** E/*not configured
- Always enabled monitoring test / NA
16) TestNATFibShortcut --------------> M**N****I*** A/Inot configured is active / Monitoring is
- Monitoring
inactive
17) TestAclPermit -------------------> M**N****I*** not configured
R/* - Power-down line cards and need reset
18) TestAclDeny ---------------------> M**N****A***
supervisor00:00:05.00
000 / NA
19) TestQoSTcam ---------------------> M**D****I*** K/*not configured
- Require resetting the line card after the
<snip> test has completed / NA
T/* - Shut down all ports and need reset
supervisor / NA
BRKRST-3143

14664_05_2008_c2.scr
Catalyst Gold Operation Example (Cont.)
20) TestL3VlanMet -------------------> M**N****I*** not configured n/a
21) TestIngressSpan -----------------> M**N****I*** not configured n/a
22) TestEgressSpan ------------------> M**D****I*** not configured n/a
23) TestNetflowInlineRewrite --------> C*PD****I*** not configured n/a
24) TestFabricSnakeForward ----------> M**N****I*** not configured n/a
25) TestFabricSnakeBackward ---------> M**N****I*** not configured n/a
26) TestTrafficStress ---------------> ***D****I**T not configured n/a
27) TestFibTcamSSRAM ----------------> ***D*X**IR** not configured n/a
28) TestAsicMemory ------------------> ***D*X**IR** not configured n/a
29) TestNetflowTcam -----------------> ***D*X**IR** not configured n/a
30) ScheduleSwitchover --------------> ***D****I*** not configured n/a
31) TestFirmwareDiagStatus ----------> M**N****I*** not configured
Diagnostics n/a
test suite attributes:
32) TestAsicSync --------------------> ***N****A***
M/C/*000 00:00:15.00
- Minimal 10 level test / Complete bootup
bootup
level test / NA
B/* - Basic ondemand test / NA
P/V/* - Per port test / Per device test / NA
D/N/* - Disruptive test / Non-disruptive test / NA
S/* - Only applicable to standby unit / NA
X/* - Not a health monitoring test / NA
F/* - Fixed monitoring interval test / NA
Pay extra attention to Memory tests:
E/* - Always enabled monitoring test / NA
Memory tests can take hours to A/I - Monitoring is active / Monitoring is
inactive
complete and a reset is required
R/* - Power-down line cards and need reset
after running these tests! supervisor / NA
K/* - Require resetting the line card after the
test has completed / NA
T/* - Shut down all ports and need reset
supervisor / NA
BRKRST-3143

Switch# show diagnostic result mod 7
Current bootup diagnostic level: complete
Module 7: CEF720 24 port 1000mb SFP
Overall Diagnostic Result for Module 7 : MINOR ERROR

Diagnostic level at card bootup: complete
Test results: (. = Pass, F = Fail, U = Untested)‫‏‬
1) TestTransceiverIntegrity:
Port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
----------------------------------------------------------------------------
Test results: (. = Pass, F = Fail, U = Untested)‫‏‬
U U . U . . U U . . U U . . U U U U U U U U U U
2) TestLoopback:
Port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
----------------------------------------------------------------------------
. . . . . . . . . . . . F . . . . . . . . . . .
3) TestScratchRegister -------------> .
4) TestSynchedFabChannel -----------> .
<snip>
BRKRST-3143

14664_05_2008_c2.scr
r1# show diagnostic description module 5 test ?
<1-33> Test ID Number
ID Test Name [On-Demand Test Attributes]
--- -------------------------------------------
1 TestScratchRegister [***N****]
2 TestSPRPInbandPing [***N****]
3 TestTransceiverIntegrity [**PD****]
4 TestActiveToStandbyLoopback [M*PDS***]
5 TestLoopback [M*PD****]
6 TestNewIndexLearn [M**N****]
<snip>
r1# show diagnostic description module 5 test 2
TestSPRPInbandPing :
By default, this test is enabled as health-monitoring test.
The SP-RP Inband test catches most of the runtime software driver
and hardware issues on supervisors. This is done by using diagnostic
packet tests exercising the layer 2 forwarding engine, the L3-4
forwarding engine, and the replication engine along the path from
the Switch Processor to the Route Processor.
Packets are sent at an interval of 15 seconds and 10 consecutive
failures of the SP-RP Inband test result in failover to the
redundant supervisor (default).
BRKRST-3143

Recommendations
Bootup diagnostics:
Set level to complete
On demand diagnostics:
Use as a pre-deployment tool: run complete diagnostics
before putting hardware into production environment
Use as a troubleshooting tool when suspecting
Si
hardware failure
Scheduled diagnostics:
Schedule key diagnostics tests periodically
Schedule all non-disruptive tests periodically
Health-monitoring diagnostics:
Key tests running by default
Enable additional non-disruptive tests for specific functionalities
enabled in your network: IPv6, MPLS, NAT
BRKRST-3143

14664_05_2008_c2.scr

BRKRST 3143

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

BRKRST 3143

Загружено:

Авторское право:

Доступные форматы

Troubleshooting Cisco

Catalyst 6500 Series

© 2006, Cisco Systems, Inc. All rights reserved. 1

 Sup720 Architecture (A Quick Look)

 Sup720 Architecture (A Quick Look)

© 2006, Cisco Systems, Inc. All rights reserved. 2

Port L3 Adj CAM

Supervisor Engine 720

 Sup720 Architecture (A Quick Look)

© 2006, Cisco Systems, Inc. All rights reserved. 3

 High CPU due to SW path forwarding

Troubleshooting Unicast Forwarding

 Detailed L3 packet flow troubleshooting

 Some useful troubleshooting tools

© 2006, Cisco Systems, Inc. All rights reserved. 4

 DUT is the Device Under Test we are troubleshooting

Quick Sanity Check

© 2006, Cisco Systems, Inc. All rights reserved. 5

Host1 R1 DUT R2 Host2

Sanity Check for L2 Unicast Traffic

 Each direction can use different links in the bundles !!

© 2006, Cisco Systems, Inc. All rights reserved. 6

 Repeat this for MAC address of Host2

Sanity Check for L2 Unicast Traffic

© 2006, Cisco Systems, Inc. All rights reserved. 7

Host1 R1 DUT R2 Host2

Sanity Check for L2 Unicast Traffic

EtherChannel Load-Balancing Addresses Used Per-Protocol:

DUT#remote command switch test etherchannel load-balance int po2 ip 7.0.1.1

R2#show etherchannel load-balance

© 2006, Cisco Systems, Inc. All rights reserved. 8

 Each direction can use different links in the bundles !!

L3 Unicast Traffic Network Configuration

 DUT is the Device Under Test we are troubleshooting

© 2006, Cisco Systems, Inc. All rights reserved. 9

Sanity Check for L3 Unicast Traffic

 Traffic flow 8.0.1.1 -> 9.0.1.2 leaves DUT on Ten8/3 link

© 2006, Cisco Systems, Inc. All rights reserved. 10

DUT#sh ip cef exact-route 9.0.1.2 8.0.1.1

DUT#remote command switch test etherchannel load-balance int po2 ip 8.0.1.1

Sanity Check for L3 Unicast Traffic

 Traffic flow 9.0.1.2 -> 8.0.1.1 leaves R2 on Ten8/3 link

© 2006, Cisco Systems, Inc. All rights reserved. 11

Host1 R1 DUT R2 Host2

Host1 R1 DUT R2 Host2

 Each direction can use different links !!

What Did We Get from Path Verification?

 The physical links the specific traffic flow should come

© 2006, Cisco Systems, Inc. All rights reserved. 12

 Detailed L3 packet flow troubleshooting

 Some useful troubleshooting tools

Detailed L2 Packet Flow Troubleshooting

 Identify path in the switch

© 2006, Cisco Systems, Inc. All rights reserved. 13

DUT#sh fabric fpoe interface Gi 7/4 Mod 8

 Gig7/4 maps to slot 7, fabric channel 1, Ten8/1 maps to

Detailed L2 Packet Flow Troubleshooting

© 2006, Cisco Systems, Inc. All rights reserved. 14

DUT#clear vlan 700 counters

Detailed L2 Packet Flow Troubleshooting

SNMP like interface counters

© 2006, Cisco Systems, Inc. All rights reserved. 15

Detailed L2 Packet Flow Troubleshooting

Sup720 Architecture (A Quick Look)

Sup720 Architecture (A Quick Look)

Sup720 Architecture (A Quick Look)

High CPU due to SW path forwarding

Detailed L3 packet flow troubleshooting

Some useful troubleshooting tools

DUT is the Device Under Test we are troubleshooting

Each direction can use different links in the bundles !!

Repeat this for MAC address of Host2

Each direction can use different links in the bundles !!

DUT is the Device Under Test we are troubleshooting

Traffic flow 8.0.1.1 -> 9.0.1.2 leaves DUT on Ten8/3 link

Traffic flow 9.0.1.2 -> 8.0.1.1 leaves R2 on Ten8/3 link

Each direction can use different links !!

The physical links the specific traffic flow should come

Detailed L3 packet flow troubleshooting

Some useful troubleshooting tools

Identify path in the switch

Gig7/4 maps to slot 7, fabric channel 1, Ten8/1 maps to

Detailed L3 packet flow troubleshooting

Some useful troubleshooting tools

DUT is the Device Under Test we are troubleshooting

L3 forwarding tables get programmed by SW: copy of SW forwarding tables in HW

For other direction (to 8.0.1.1), completely similar commands

CEF Glean: subject to rate limiter “CEF GLEAN”

Verified already at FIB and adjacency tables, as well as L2 CAM table

What does this effectively mean … ?

Lookup in BOTH banks,

Result with highest priority wins, HI bank LOW bank

Capture all info at once for a particular L3 Prefix:

Detailed L3 packet flow troubleshooting

Some useful troubleshooting tools

What/why SW CEF (not Process) switched packets are hitting the RP ?