Академический Документы
Профессиональный Документы
Культура Документы
Home
**Note that these methods can be used to determine the cause of high CPU on
a RSP720, Sup32 and VS-S720, due to common architecture.
You can view the CPU utilization using the following command:
SP CPU Util:
OR
This will be traffic that needs to be processed for layer 3 operations, such
as ARP, HSRP, forwarding traffic in software.
Below I will go over
troubleshooting steps when seeing high CPU on the IP Input/ARP input process
as well as CPU utilization caused by interrupt switched traffic on the RP
CPU.
You can view the CPU utilization using the following command:
RP CPU Util:
There are two type of CPU utilization within IoS, interrupt and process.
This error message is received when the amount of available space in the
TCAM is exceeded. This results in high CPU. This is a FIB TCAM limitation.
Once TCAM is full, a flag will be set and FIB TCAM exception is received.
This stops from adding new routes to the TCAM. Therefore, everything will be
software switched. The removal of routes does not help resume hardware
switching. Once the TCAM enters the exception state, the system must be
reloaded to get out of that state. You can view if you have hit a FIB TCAM
exception with the following command:
6500-2#sh mls cef exception status
This issue is common when trying to route a full BGP table on PFC-3A or a
PFC-3B.
**Note a failover of the supervisors in dual supervisor system will not recover this exception, even
through the show mls cef exception status will no longer indicate a FIB exception. A full
reload of the switch is required.
ICMP redirects - If traffic is taking a path that is not efficient, an ICMP
redirect will be sent out to inform the host of a better next-hop.
This
will cause the packet to be punted in order to trigger the MSFC to send the
ICMP redirect to the host. This can be seen when performing a netdr
capture.
An example of using netdr can be seen in the Tools used to
determine the source of the CPU utilization: section
Turn off icmp redirects to stop this traffic from being punted.
However
this is an indication of network inefficiency that was attempting to be
dynamically resolved.
User interaction is needed in order to track down
this inefficiency.
If you need assistance in determining why ICMP redirects are being generated
please open a TAC case
To work around this problem do not have both an ACL based and Netflow based
feature configured on the same interface, matching the same traffic.
You can read more about troubleshooting feature conflict issues via the
following link:
https://supportforums.cisco.com/docs/DOC-15670
Directed Broadcast traffic All broadcast traffic must be sent to the MSFC on
a vlan when a layer 3 interface is configured within that vlan.
This
includes directed broadcast traffic.
Use multicast instead of directed
broadcast.
Bridging loop - If a bridging loop occurs on the network, this could cause
high CPU on the MSFC. All broadcast traffic must be sent to the MSFC on a
vlan when a layer 3 interface is configured within that vlan.
You can determine what traffic is hitting the CPU by using a netdr capture
to track down the source interface of the loop (See Using Netdr to determine
traffic punted to the CPU section).
GRE with non-unique tunnel source - On the sup720, tunnel sources must be
unique for all tunnels. Tunnels with a non-unique source will be software
switched. The workaround for this limitation is to use either unique
loopback interfaces for every GRE tunnel OR
use secondary addresses on a
loopback interface for the tunnel source addresses.
For more
information see CSCdy72539.
You may also see the following error:
%Warning: Using same source IP for more than one IP/GRE tunnels may cause software
switching packets for tunnels using this address. If possible, use a unique tunnel source
for Interface Tunnel <tun#>
The following features/traffic types are common features that are not
supported by the 6500 and will cause high CPU if implemented:
**Note this is not an exhausted list and there may be unsupported features not listed below.
NBAR
Traffic with IP options field set.
Multicast RPF drops
RSVP (INTSERV QOS) *can be used for tunnels
CEF accounting
The following link will give a larger list of all unsupported features and
commands on the sup720-PFC3:
http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/release/n
otes/features.html#wp3691673
**Note** you will only be able to see traffic in the interface buffers on a
layer 3 interface if the traffic is being processed switched (see
Determining type of CPU utilization above).
This will not work when
traffic is being interrupt switched.
In the case of interrupt switched
traffic use the netdr capture instead.
One of the quickest ways to determine the layer 3 interface that is the
source of traffic that is causing high CPU is to see which interface has a
large amount of drops flushes on the interfaces input queue.
The input
queue on a layer 3 interface is the CPU queue for that interface on the
sup720.
If we ever see packets/drops on the input queue on the sup720 it
is always due to traffic that is being sent towards the CPU.
You can
narrow down the location of such an interface with the following commands:
is up|drop
We can see that SVI (Switched Virtual Interface) 10 has 74 packets in its
buffer, whose queue size is 75 packets. This demonstrates that a large
amount of traffic is being punted on this interface to the RP CPU, since
this queue is full.
Now that we can see a large amount of traffic within this queue, we can
look at what is in this queue with the command "show buffers input-interface
vlan 10 header".
This command will display the IP header of the packet so
we can attempt to determine the source.
If you want to look at the entire
packet you can use the command
"show buffers input-interface vlan 10
packet".
Above we can see the basic information about this traffic that is included
in the IP header, including the TOS, TTL and protocol encapsulated within
the IP header.
..G&{.........E.
........6Q.....d
e.............P.
..&\........}
Since at this point we are unsure why this traffic is being punted, we can
look at show ip traffic" statistics to see why this traffic is being punted
to the CPU. First start by clearing the IP traffic statistics. We can then
see what is incrementing in these counters to see what would be the cause:
6500-2#clear ip traffic
Clear "show ip traffic" counters [confirm]
6500-2#sh ip traffic
IP statistics:
Rcvd:
0 generated, 0 forwarded
Drop:
ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 0 unreachable
0 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 0 timestamp, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
0 time exceeded, 0 timestamp replies, 0 info replies
Sent: 0 redirects, 0 unreachable, 0 echo, 0 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp
0 info reply, 58464 time exceeded, 0 parameter problem
can also the 6500 is sending ICMP TTL expired messages as well.
<---- We
<snip>
Looking at the traffic statistics we can see that the bad hop count counter
is incrementing and the switch is sending ICMP time exceeded messages.
On the 6500 all traffic with a TTL of 1 is punted to the CPU so that an
ICMP TTL expired message can be sent to the host who sent this traffic.
Also, the first packet in the buffer can be seen to have TTL of 1, which is
why this traffic is punted.
We can see that the 2nd packet is sourced from
10.10.10.1 (SVI 10) sent to 10.10.10.2. This packet is an ICMP TTL expired
message.
and-filter
continuous
destination-ip-address
dstindex
ethertype
interface
or-filter
match
rx
source-ip-address
srcindex
tx
vlan
<cr>
OPTIONS:
Using the continuous option, the switch will capture packets on the RPinband continuously fill the entire capture buffer (4096 packets) and then
start to overwrite the buffer in a FIFO fashion.
The tx and rx options will capture packets coming from the MSFC and going
to the MSFC respectivey.
The and-filter and the or-filter specify that an and or an or will be
applied respectively to all of the options that follow. For example, if
you use the syntax below, then both option #1 and option #2 must match for
the packet to be captured. Similarly, if the or-filter is used either
option #1 or option #2 or both must match for the packet to be captuered.
------- dump of incoming inband packet ------interface Vl10, routine mistral_process_rx_packet_inlin, timestamp 00:00:11
dbus info: src_vlan 0xA(10), src_indx 0xC0(192), len 0x40(64)
bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)
10020400 000A0000 00C00000 40080000 00060468 0E000040 00000000 03800000
mistral hdr: req_token 0x0(0), src_index 0xC0(192), rx_offset 0x76(118)
requeue 0, obl_pkt 0, vlan 0xA(10)
destmac 00.15.C7.26.FB.80, srcmac 00.00.01.00.06.00, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 46, identifier 0
df 0, mf 0, fo 0, ttl 100, src 10.10.10.2, dst 10.100.101.10
tcp src 0, dst 0, seq 0, ack 0, win 0 off 5 checksum 0x265C
Purple = IP Header
You can use this information to track down the source of the traffic being
punted.
Please refer to Troubleshooting with a NETDR capture on a
sup720/6500 documention for a further explanation of how to interpret this
data.
Please open a TAC case if you need further assistance interpreting this
data.
configuration for an
RP Console:
SP Console:
-OR-
On the 12.2(33)SXH train and later, this is the configuration for an inband
sp->rp span session:
<rp|sp> <tx|rx|both>
http://cco.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configura
tion/guide/span.html#wp1109488
Once this information is collected you can then use the source MAC/source IP
information to determine the source of the traffic.
action 0.0 syslog msg "High CPU DETECTED. Please wait - logging
Information to <file system>:high_cpu.txt"
action 0.1 cli command "enable"
action 0.2 cli command "show clock | append <file system>:high_cpu.txt"
action 1.2 cli command "term length 0"
action 1.3 cli command "show process cpu sorted | append <file
system>:high_cpu.txt"
Please fill in <file system> with the location of the file system without
"<" or ">".
If you need further assistance in determining the cause of your high CPU
please open a Cisco TAC case.
Rating
1
2
3
4
5
Overall Rating: 5 (7 ratings)
Comments
Collapse all
Recent replies last
See More
See More
See More
See More
See More
https://supportforums.cisco.com/document/59926/troubleshooting-high-cpu-6500-sup720