CH A P T E R 5

Troubleshooting Bundles and Load Balancing

This chapter explains the procedures for troubleshooting link bundles and load balancing on the
Cisco ASR 9000 Aggregation Services Router.
A link bundle is a group of ports that are bundled together and act as a single link. The advantages of
link bundles are:
• Multiple links can span several LCs to form a single interface; thus, the failure of a single link does
not cause a loss of connectivity.
• Bundled interfaces increase bandwidth availability, because traffic is forwarded over all available
members of the bundle. Therefore, traffic can move onto another link if one of the links within a
bundle fails. This allows you to add or remove bandwidth without interrupting packet flow.
This chapter contains the following sections:
• Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing, page 5-115
• Troubleshooting Problems with Link Bundles, page 5-118
• Troubleshooting Layer 2 Bundles and Load Balancing, page 5-122
• Troubleshooting Layer 3 Bundles and Load Balancing, page 5-124

Troubleshooting Routing and CEF Issues Related to Bundles and

Load Balancing
Cisco Express Forwarding (CEF) uses the path information in the IP routing table to balance traffic over
link bundles. For this reason, verifying correct load balancing with CEF begins with confirming the
contents of the IP routing table and CEF database.
• Verifying Routing Table Entries for Parallel Links, page 5-115
• Verifying the CEF Database and Measuring Flows, page 5-117

Verifying Routing Table Entries for Parallel Links

Perform this procedure to verify the contents of the IP routing table.

SUMMARY STEPS

1. show route destination-address

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 5-115
 Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing

2. configure
3. router ospf process
4. maximum paths number
5. end
6. show route destination-address
7. show ospf process interface brief
8. show running-config router ospf process

Command or Action Purpose

Step 1 show route destination-address Displays the routes to a destination address. Use a
destination address on another host that is reachable
through the parallel links.
Example:
RP/0/RSP0/CPU0:router# show route 10.1.2.1 Verify that number of routes in the routing table equals the
number of parallel links. If you have fewer routes than
expected, continue with this procedure.
Step 2 configure Enters global configuration mode.

Example:
RP/0/RSP0/CPU0:router# configure
Step 3 router ospf process Enters configuration mode for the OSPF process.

Example:
RP/0/RSP0/CPU0:router# router ospf 200
Step 4 maximum paths number Configures the maximum number of paths over which to
load balance. By default, OSPF balances up to 4 equal-cost
paths.
Example:
RP/0/RSP0/CPU0:router# maximum paths 3
Step 5 end Ends the configuration process. Enter yes at the prompt to
commit the changes.
Example:
RP/0/RSP0/CPU0:router# end
Step 6 show route destination-address Displays the routes to a destination address.
Verify that number of routes in the routing table equals the
Example: number of parallel links. If you have fewer routes than
RP/0/RSP0/CPU0:router# show route 10.1.2.1 expected, continue with this procedure.

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
5-116 OL-23591-02
 Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing

Command or Action Purpose

Step 7 show ospf process interface brief Shows interface information for all routes to the destination
address, which displays the cost metric. OSPF balances
loads over equal-cost routes only, so verify that the
Example:
RP/0/RSP0/CPU0:router# show ospf 200 interface
interfaces have equal costs. To load balance over unequal
brief paths, use Enhanced Interior Gateway Routing Protocol or
Interior Gateway Routing Protocol (EIGRP/IGRP) as the
IGP instead.
Step 8 show running-config router ospf process Displays the running configuration for the OSPF process.
This is another way to determine if the interfaces have
different costs.
Example:
RP/0/RSP0/CPU0:router# show
running-configuration router ospf process

Verifying the CEF Database and Measuring Flows

Perform this procedure to verify the contents of the CEF database.

SUMMARY STEPS

1. show cef ipv4 [prefix [mask]] | interface-type interface-path-id] [detail] [location node-id]
2. show cef [ipv4 | ipv6] exact-route source-address destination address [protocol type]
[source-port source-port] [destination-port destination-port] [ingress-interface type
interface-path-id] [policy-class value] [detail | location node-id]
3. show interfaces [type interface-path-id | all | local | location node-id] [accounting | brief | detail |
summary]

DETAILED STEPS

Command or Action Purpose

Step 1 show cef ipv4 [prefix [mask]] | interface-type Displays the CEF forwarding table. Verify that it contains
interface-path-id] [detail] [location node-id] the same interfaces that the routing table has for this
destination.
Example:
RP/0/RSP0/CPU0:router# show cef ipv4 10.1.2.1
detail

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 5-117
 Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Problems with Link Bundles

Command or Action Purpose

Step 2 show cef [ipv4 | ipv6] exact-route Displays the exact route that a specific flow would take,
source-address destination address [protocol including the egress interface for a specific source and
type] [source-port source-port]
[destination-port destination-port]
destination IP. Use this command for several flows to verify
[ingress-interface type interface-path-id] that they are distributed equally over the parallel interfaces.
[policy-class value] [detail | location
node-id]

Example:
RP/0/RSP0/CPU0:router# show cef exact-route
192.168.254.1 10.1.2.1 protocol ospf
source-port 5500 destination-port 80
ingress-interface gi0/6/5/4
Step 3 show interfaces [type interface-path-id | all | Displays the traffic rates by interface. Use this command to
local | location node-id] [accounting | brief | verify that the simulated traffic takes the expected egress
detail | summary]
interface.

Example:
RP/0/RSP0/CPU0:router# show interfaces
accounting rates

Troubleshooting Problems with Link Bundles

This section explains how to troubleshoot problems with link bundles. It contains the following
subsections:
• Bundle Does Not Come Up, page 5-118
• Bundle Member Not Distributing, page 5-119
• Bundle Not Using MAC-Address From Backplane, page 5-119
• Layer 3 Data Traffic Not Flowing, page 5-120
• Ping Failed over Bundle, page 5-120
• Layer 3 Packets Not Synching Over Bundle, page 5-121
• Layer 2 Traffic Not Flowing, page 5-121
• Bundle Statistics, page 5-122

Bundle Does Not Come Up

Step 1 Ensure that the member port is not “shutdown”. Ensure that the MAC burned-in address (BIA) of the
port is valid.
RP/0/RSP0/CPU0:router# show interface

Step 2 If running Link Aggregation Control Protocol (LACP), ensure that LACP packets are able to send and
receive accordingly. If LACP packets are not able to send and receive accordingly, check interface
counters to identify at what stage packets are dropped.
RP/0/RSP0/CPU0:router# show lacp counters

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
5-118 OL-23591-02
 Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Problems with Link Bundles

Step 3 View LACP statistics.

RP/0/RSP0/CPU0:router# show lacp

Step 4 Ensure that the other side of the link is up (bundle and members).
RP/0/RSP0/CPU0:router# show bundle

Bundle Member Not Distributing

Step 1 Ensure that the member is up.

RP/0/RSP0/CPU0:router# show interface node-id

Step 2 Ensure that the remote side is up.

Step 3 Ensure that the LACP parameters are the same on both sides.
a. If LACP is enabled, check its status.
RP/0/RSP0/CPU0:router# show lacp bundle

b. Ensure that bundle members have the same characteristics.

RP/0/RSP0/CPU0:router# show running interface interface-name

c. If the bundle members have different characteristics, make them all the same.
d. Ensure that LACP packets are transmitted and received.
RP/0/RSP0/CPU0:router# debug bundlemgr local packets port node-id

Workaround

If the bundle with LACP cannot come up, use one side of the bundle in passive mode and the other in
active mode. At least one side must be active.

Bundle Not Using MAC-Address From Backplane

Step 1 Ensure that the backplane MAC is programmed. Note that this command has to be run in the admin
mode.
RP/0/RSP0/CPU0:router(admin)# show diag chassis eeprom-info

Step 2 Display the backplane information.

RP/0/RSP0/CPU0:router# show controllers backplane bpe-trace

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 5-119
 Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Problems with Link Bundles

Layer 3 Data Traffic Not Flowing

Regular Interface (No Subinterfaces)

Step 1 View the Address Resolution Protocol (ARP).

RP/0/RSP0/CPU0:router# show arp

Step 2 Verify that the lag table is programmed properly in the hardware.

RP/0/RSP0/CPU0:router# show interface bundle-ether bundle-id

Step 3 View the running configuration information.

RP/0/RSP0/CPU0:router# show running-config

Step 4 View information about packets forwarded by CEF.

RP/0/RSP0/CPU0:router# show cef

Step 5 RP/0/RSP0/CPU0:router# show cef hardware ingress location node-id
Step 6 RP/0/RSP0/CPU0:router# show cef hardware egress location node-id

Subinterface

Step 1 Troubleshoot Layer 3 IPv4 traffic.

Step 2 Ensure that VLAN traffic coming in matches that on the incoming interface.

Ping Failed over Bundle

Step 1 View the ARP.

RP/0/RSP0/CPU0:router# show arp

Step 2 View the ARP information on the particular LC or RSP.

RP/0/RSP0/CPU0:router# show arp location node-id

Step 3 RP/0/RSP0/CPU0:router# show cef hardware detail location node-id ingress
Step 4 RP/0/RSP0/CPU0:router# show interface
Step 5 Use the hash calculator to determine which bundle member (interface) to test.
Step 6 Remove the interface from the bundle.
Step 7 Assign an IP address to the interface.
Step 8 Ping the interface.
Step 9 Ensure that the ARP is resolved between the router and the node being pinged.

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
5-120 OL-23591-02
 Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Problems with Link Bundles

Step 10 Ensure that the MAC address in the ARP table of the other side corresponds to that on the router.
Step 11 Ensure that the MAC address of the bundle is valid.
Step 12 Ensure that the routing and hardware routing table has an entry to the next hop.
Step 13 Check the interface counters to see if ping packets are transmitted and being received on the router
member port of the bundle.
Step 14 Check the ucode counters to see where packets are dropped on the incoming or outgoing member of the
bundle.
Step 15 Make sure that the table lookup (TLU) entries are allocated and bundle adjacency information is properly
programmed.
show cef adajacency bundle-type bundle-number hardware egress detail location location-id

show cef adajacency bundle-type bundle-number hardware ingress remote detail location
location-id

Workaround

Try a different port.

Layer 3 Packets Not Synching Over Bundle

Step 1 View the interface information:

RP/0/RSP0/CPU0:router# show interface

Step 2 Turn on the debug of that protocol or look at the protocol counters to see if the protocol packets are being
sent and received.
Step 3 If the protocol packets are not being sent or received, check the interface counters to see if interface
indicates packets in and out.
Step 4 If the interface level indicates that packets are coming in and out but not reaching protocol, check the
ucode counters to see if there are any drops.

Layer 2 Traffic Not Flowing

VPLS

Step 1 Verify that the AC is up.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain

Step 2 Verify that the bridge domain is up.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 5-121
 Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Layer 2 Bundles and Load Balancing

Step 3 Look for MTU mismatches.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detail

VPWS

Step 1 View brief information on configured cross-connects.

RP/0/RSP0/CPU0:router# show l2vpn xconnect summary

Step 2 RP/0/RSP0/CPU0:router# show l2vpn xconnect state
Step 3 RP/0/RSP0/CPU0:router# show controllers bundle bundle-ether bundle-id location node-id

Bundle Statistics
Layer 2 statistics are not supported in the show interface accounting command for bundle interfaces in
the current release.

Troubleshooting Layer 2 Bundles and Load Balancing

This section describes how to troubleshoot Layer 2 bundles and load balancing. It includes the following
topics:
• Verifying the Bundle Status, IGP Route, and CEF Database, page 5-122
• Viewing the Expected Paths and Measuring the Flows, page 5-123

Verifying the Bundle Status, IGP Route, and CEF Database

CEF uses the path information in the IP routing table to balance traffic over multiple links. For this
reason, confirming proper CEF load balancing begins with confirming the contents of the IP routing
table. When troubleshooting a bundle, verify that the bundle is up and that the IGP route to the desired
destination includes the bundle interface.

SUMMARY STEPS

1. show running-config
2. show bundle {Bundle-Ether | Bundle-POS} interface-path-id
3. show interface {Bundle-Ether | Bundle-POS} bundle-id
4. show arm router-id
5. show controllers bundle {Bundle-Ether | Bundle-POS} bundle-id location node-id
6. show route destination-address
7. show cef ipv4 prefix

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
5-122 OL-23591-02
 Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Layer 2 Bundles and Load Balancing

DETAILED STEPS

Command or Action Purpose

Step 1 show running-config Displays the running configuration.
Verify that the configuration related to the bundles are
Example: correct.
RP/0/RSP0/CPU0:router# show running-config
Step 2 show bundle {Bundle-Ether | Bundle-POS} Displays the bundle status.
interface-path-id
Verify that the bundle has the expected number of links. If
not, troubleshoot the bundle first.
Example:
RP/0/RSP0/CPU0:router# show bundle
bundle-ether 12
Step 3 show interface {Bundle-Ether | Bundle-POS} Displays the interface status.
bundle-id
Verify that the interface is assigned to the bundle.

Example:
RP/0/RSP0/CPU0:router# show interface
bundle-ether 12
Step 4 show arm router-id —

Example:
RP/0/RSP0/CPU0:router# show arm router-id
Step 5 show controllers bundle {Bundle-Ether | —
Bundle-POS} bundle-id location node-id

Example:
RP/0/RSP0/CPU0:router# show controllers bundle
bundle-ether 12 location 0/4/CPU0
Step 6 show route destination-address Displays the routes to a destination address. Use a
destination address on another host that is reachable
through the bundle.
Example:
RP/0/RSP0/CPU0:router# show route 10.1.2.1 Verify that the route to the desalination address includes the
bundle interface. If not, make sure that the bundle interface
is included in the IGP process configuration.
Step 7 show cef ipv4 prefix Displays the CEF forwarding table. Verify that it contains
the same bundle interface that the routing table has for this
subnet prefix.
Example:
RP/0/RSP0/CPU0:router# show cef ipv4 10.1.2.1

Viewing the Expected Paths and Measuring the Flows

Cisco IOS XR provides a bundle utility that predicts how Layer 2 loads are balanced across member
links. This is an interactive tool prompts for the information that the load balancing algorithm uses to
allocate flows to member links.

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 5-123
 Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Layer 3 Bundles and Load Balancing

SUMMARY STEPS

1. bundle-hash {Bundle-Ether | Bundle-Pos} interface-path-id

2. show interfaces [type interface-path-id | all | local | location node-id] [accounting | brief | detail |
summary]

DETAILED STEPS

Command or Action Purpose

Step 1 bundle-hash {Bundle-Ether | Bundle-Pos} Launches the bundle-hash utility. This is an interactive
interface-path-id utility that prompts for the necessary information.

Example:
RP/0/RSP0/CPU0:router# bundle-hash
bundle-ether 12
Step 2 show interfaces [type interface-path-id | all | Displays interface information, which includes the traffic
local | location node-id] [accounting | brief | rates. Use this command for each link in the bundle to verify
detail | summary]
that the simulated traffic takes the expected link. Use clear
counters to make it easier to view the traffic allocation.
Example:
RP/0/RSP0/CPU0:router# show interfaces tenGigE
0/5/0/1

Troubleshooting Layer 3 Bundles and Load Balancing

This section provides commands for troubleshooting Layer 3 bundles and load balancing.

Step 1 RP/0/RSP0/CPU0:router# show arm router-id

Step 2 RP/0/RSP0/CPU0:router# show bundle
Step 3 RP/0/RSP0/CPU0:router# show interface bundle-ether bundle-id
Step 4 RP/0/RSP0/CPU0:router# show running-config
Step 5 RP/0/RSP0/CPU0:router# show arm router-ids
Step 6 Find out which member is carrying the traffic out.

RP/0/RSP0/CPU0:router# bundle-hash bundle-ether bundle-id

Step 7 View each member of the bundle to see which member is actually carrying the traffic out.

RP/0/RSP0/CPU0:router# show interface

Step 8 Display the exact route, including the egress interface for a specific source and destination IP. Use this
command for several flows to verify that they are distributed equally over the parallel interfaces.
show cef [ipv4 | ipv6] exact-route source-address destination address [protocol type]
[source-port source-port] [destination-port destination-port] [ingress-interface type
interface-path-id] [policy-class value] [detail | location node-id]

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
5-124 OL-23591-02