Вы находитесь на странице: 1из 286
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router Cisco IOS
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router Cisco IOS

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

Cisco IOS XR Software, Release 4.0 April, 2011

Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com

Tel:

408 526-4000

Fax:

800 553-NETS (6387) 408 527-0883

Text Part Number: OL-23591-02

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1005R)

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router © 2010 Cisco Systems, Inc. All rights reserved.

Preface xi Changes to This Document xi CONTENTS   Obtaining Documentation and Submitting a Service

Preface

xi

Changes to This Document

xi

CONTENTS

 

Obtaining Documentation and Submitting a Service Request

xii

CHAPTER

1

General Troubleshooting Procedures

1-1

 

Prerequisite Documentation for Troubleshooting

1-1

Verifying and Troubleshooting CLI Access

1-2

General CLI Access Information

1-2

User Access Privileges Cisco-support Task ID

1-2

1-3

CLI

Access Through a Console Port

1-3

CLI

Access Through a Terminal Server

1-3

 

CLI

Access Through the Management Ethernet Interface

1-4

Validating and Troubleshooting Installation of the Cisco IOS XR Software Package

Verifying the Software Version

Validating the Installation

1-10

1-8

Validating and Troubleshooting Cisco IOS XR Software Configuration

Local and Global Configurations Collecting Configuration Information Verifying the Running Configuration

Using the show configuration failed Command

1-16

1-19

1-20

1-24

Verifying the System

Troubleshooting the Backplane Ethernet Control System

Basic Cisco IOS XR Verification and Troubleshooting Commands

1-26

1-41

man Command

describe Command

show platform Command

top

show context Command show users Command

show history Command

show configuration Command

1-46

1-49

1-49

1-50

1-52

1-52

Command

1-50

1-53

Displaying ASIC Errors

Using Trace Commands

1-54

1-56

1-46

1-16

1-7

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

1-56 1-46 1-16 1-7 Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services

OL-23591-02

iii

Contents

MIB Location

Gathering Information Before You Call Cisco TAC

1-57

1-58

Gathering Information about Crashes and Core Dumps

Capturing Logs

Using Debug Commands

Using Diagnostic Commands

Commands Used to Display Process and Thread Details

1-58

1-59

1-59

1-58

1-59

CHAPTER

2

Verifying and Troubleshooting Interface Status

2-61

 
 

Verifying and Troubleshooting Gigabit Ethernet Interfaces

2-61

Verifying and Troubleshooting Pluggable Optical Line Card Interfaces

2-68

CHAPTER

3

Troubleshooting Interface Connectivity

3-75

 

Troubleshooting Ping and ARP Connectivity

3-75

Troubleshooting Bidirectional Forwarding Detection

3-81

 

Using show and debug Commands

3-82

BFD Sessions in Down State

3-83

BFD Sessions Flap

3-83

BFD Sessions Down on Neighboring Router BFD Sessions Are Not Created on the LC

3-85

3-85

 

Troubleshooting Ethernet CFM

3-85

Using show and debug Commands

3-87

MEPs Are Not Created

3-88

 

MIPs Are Not Created

3-88

No CCMs are Received at the MEP or Peer MEPs Are Not Seen

3-89

Peer MEP Defects and Mismatches Are Seen

3-90

 

Remote Defect Indication Received

3-91

Peer MEP Times Out But No Alarm Or Action Occurs

3-92

No Debugs or Counters for Higher-Level Packets at a MEP or MIP

3-92

Dropped CFM PDUs

3-92

CFM ping Or traceroute Returns a “not found” Error

3-93

AIS Messages Are Not Sent

3-93

CHAPTER

4

Troubleshooting Packet Forwarding

4-95

 

Understanding IPv4 CEF

Troubleshooting IPv4 CEF

4-95

4-96

Troubleshooting Adjacency Information

4-101

Troubleshooting Transient Traffic Drop

4-106

4-101 Troubleshooting Transient Traffic Drop 4-106 Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

iv

OL-23591-02

Contents

Troubleshooting Packet Drop in the Fabric

4-109

Troubleshooting Control Plane Information

4-109

CHAPTER

5

Troubleshooting Bundles and Load Balancing

5-115

 

Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing

5-115

Verifying Routing Table Entries for Parallel Links Verifying the CEF Database and Measuring Flows

5-115

5-117

Troubleshooting Problems with Link Bundles

5-118

Bundle Does Not Come Up

5-118

Bundle Member Not Distributing

5-119

 

Bundle Not Using MAC-Address From Backplane

5-119

Layer 3 Data Traffic Not Flowing

5-120

 

Ping Failed over Bundle

5-120

Layer 3 Packets Not Synching Over Bundle

5-121

Layer 2 Traffic Not Flowing

5-121

Bundle Statistics

5-122

Troubleshooting Layer 2 Bundles and Load Balancing

5-122

Verifying the Bundle Status, IGP Route, and CEF Database Viewing the Expected Paths and Measuring the Flows

5-122

5-123

Troubleshooting Layer 3 Bundles and Load Balancing

5-124

CHAPTER

6

Troubleshooting Layer 3 Connectivity

6-125

 
 

Using show and debug Commands

6-125

Traffic Loss

6-128

Packets Are Punted and Switched in Software

6-129

Traceroute Fails

6-130

 

Adding Routes Fails

6-131

Continuous Tracebacks

6-133

fib_mgr Does Not Come Up During LC Reload or After Multiple Process Restarts

6-134

CEF Entries Out of Sync

fib_mgr Crashes

Tracebacks Appearing

Traffic Loss Because of Changing encap on a Subinterface

Traffic Loss during RSP Failover

Troubleshooting Virtual Router Redundancy Protocol

6-135

6-136

6-136

6-138

6-138

Using show and debug Commands VRRP Fails to Reach Active State

6-139

6-140

6-137

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

6-139 6-140 6-137 Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

OL-23591-02

v

Contents

Tracked Interface Failing, Router State Not Changed

VRRP State Flapping

6-140

More Than One VRRP Router Active

VRRP Active Router Not Forwarding Traffic

Traffic Loss or Unexpected VRRP State After Interface shut/no shut

6-140

6-141

6-141

Additional Information On Routing Configuration Commands

6-142

6-142

CHAPTER

7

Troubleshooting Router Switch Fabric and Data Path

7-143

 

Understanding Switch Fabric Architecture

7-143

 

Getting Started with Fabric Troubleshooting

7-145

Troubleshooting Packet Drops

7-146

Displaying Traffic Status in Line Cards and RSP Cards

7-147

Locating Packet Drops by Examining Counters

7-148

Locating Drops of Punted Packets

7-155

Packet Drop from LC to LC

7-157

Packet Drop Between RSP and LC

7-158

Packet Drop After Certain Actions

7-160

Packet Drop After a Redundancy Switchover

7-161

Packet Drop with Unknown Reason

7-163

Troubleshooting RSP and LC Crashes

7-165

7-165

Active RSP Is Crashing Standby RSP Is Crashing

7-166

LC Is Crashing

7-167

Troubleshooting Complete Loss of Traffic

7-168

 

No Traffic from LC to LC

7-169

No Traffic Between RSP and LC

7-170

Gathering Fabric Information Before Calling TAC

7-172

CHAPTER

8

Troubleshooting MPLS Services

8-173

Verifying MPLS PIE Activation and MPLS Configuration

8-173

Troubleshooting Connectivity Over MPLS

8-174

Using show and debug Commands

8-174

IP Packets Not Forwarded to LSP

8-175

IP Packets Not Forwarded to MPLS TE Tunnel

8-176

 

MPLS Packets Not Forwarded to MPLS TE Tunnel

8-176

MPLS TE Tunnels Do Not Come Up

8-176

FRR-Protected Tunnel Goes Down After Triggering FRR

8-177

MPLS TE FRR Database Not Built

8-178

Triggering FRR 8-177 MPLS TE FRR Database Not Built 8-178 Cisco IOS XR Troubleshooting Guide for

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

vi

OL-23591-02

Contents

CHAPTER

9

MPLS FRR Switch Time Debugging

8-178

Troubleshooting L2VPN and Ethernet Services

9-181

Troubleshooting VLAN Traffic and L2 TCAM Classification

9-181

Understanding Problems with VLAN Traffic and L2 TCAM Classification

Verifying the Configuration Is Correct

9-182

9-182

Verifying Interfaces, Subinterfaces, and Packet Forwarding

9-183

Troubleshooting Multipoint Layer 2 Services

9-190

Basic Bridging: Example

9-190

Verifying MAC Address Updates

9-192

Troubleshooting Multipoint Layer 2 Bridging Services (VPLS)

9-195

Troubleshooting Bridge Domains That Use BGP-AD

9-201

Troubleshooting Point-to-Point Layer 2 Services Example of Point-To-Point Layer 2 Deployment

9-206

9-206

Using show and debug Commands

AC Is Down

9-211

Pseudowire Is Down

9-212

9-210

VPWS Not Forwarding Traffic from AC to Pseudowire

Pseudowire Up but Ping Fails

Traffic Loss

Traffic Loss During RSP Fail Over

Preferred Path Not Working

Troubleshooting Specific Outage Scenarios In Layer 2 Services

9-212

9-213

9-213

9-213

9-214

9-214

Using show and debug Commands L2VPN Discovery Not Working

AC Is Down

9-218

Pseudowire Is Down

VPLS Not Forwarding Flooding Traffic

9-215

9-217

9-219

9-220

VPLS Not Forwarding Flooding Traffic from AC to Pseudowire

9-224

VPLS Not Forwarding Flooding Traffic from Pseudowire to AC

9-224

VPLS Not Forwarding Unicast Traffic from AC to AC

9-225

VPLS Not Forwarding Unicast Traffic from AC to Pseudowire

9-225

VPLS Not Forwarding Flooding Traffic from Pseudowire to AC

9-225

Pseudowire Up but Ping Fails

Traffic Loss

Pseudowire Flap Causing Traffic Loss Traffic Loss During RSP Fail Over

Preferred Path Not Working

9-226

9-226

9-226

9-227

9-227

Troubleshooting Dynamic Host Configuration Protocol Snooping

9-227

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

Snooping 9-227 Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router OL-23591-02

OL-23591-02

vii

Contents

Show Commands

9-228

Trace Commands

9-228

Syslog Commands

9-228

Tech-support Commands

9-229

Action Commands

9-229

L2VPN Commands

9-229

L2Snoop Commands

9-229

Interface Controller Commands

9-230

Troubleshooting Multiple Spanning Tree Using show and debug Commands

Troubleshooting Multiple Spanning Tree Using show and debug Commands

9-230

9-230

MSTP Incorrectly or Inconsistently Formed

MSTP Correctly Formed, but Traffic Flooding Packet Forwarding Does Not Match MSTP State

9-230

9-231

9-231

MSTAG Access Network Does Not Recognize MSTAG Node as Root

9-231

Traffic Not Switching Through MSTAG Node(s)

9-232

Additional References—Command Reference and Configuration Guides

9-232

CHAPTER

10

Troubleshooting Quality of Service and Access Control Lists

Using show and debug Commands

Service-Policy Configuration Is Rejected

Packets are Incorrectly Classified

Packets in Wrong Queue

10-234

10-235

10-235

10-236

10-233

Packets Incorrectly Marked

10-236

Packets Incorrectly Policed

10-237

Shaping Incorrect

10-237

Weighted Random Early Detection Incorrect

Bandwidth Not Guaranteed

Bandwidth Ratio Not Working

Non-zero Queue(conform) and Queue(exceed) Counters In show policy-map Commands

Unable to Modify or Delete policy-map or class-map

Unable to Modify or Delete class-map ACL

Unable to Delete service-policy

After QoS EA Restarts, show policy-map interface Fails

After QoS EA Restarts, service-policy config Fails

show policy-map interface Output Error

Bundle Members Not Configured with service-policy

Troubleshooting Access Control Lists

10-237

10-238

10-238

10-240

10-240

10-240

10-240

10-241

10-241

10-241

10-241

10-239

10-240 10-240 10-240 10-241 10-241 10-241 10-241 10-239 Cisco IOS XR Troubleshooting Guide for the Cisco

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

viii

OL-23591-02

Contents

Using show and debug Commands ACL Messages Not Appearing

Fragmented Packets Being Accepted Egress Counter Incorrect or Not Working

10-242

10-243

10-243

10-244

10-244

ACL Interface Bind Rejected Single ACE Using Many TCAMs

10-244

ACL Using Varying TCAM Space

10-245

ACL Logs Not Working for Ethernet Services

Ethernet Services ACL Bind on Interface Rejected

Changing ACL Exhausts TCAM

Cannot Delete ACL DF Bit Not Supported

Max ACL Limit Reached

Unsupported Combinations in ACL

No Statistics Counters TCAMs Out of Resources

10-246

10-245

10-245

10-245

10-246

10-246

10-246

10-246

10-246

CHAPTER

11

Troubleshooting Multicast Services

11-247

 

Troubleshooting IGMP Snooping (Layer 2 Multicast)

11-247

Using show Commands

11-247

Using the debug, trace, and show tech-support Commands Troubleshooting Missing Routes and Forwarding Errors

11-249

11-250

Troubleshooting Native Multicast Routing (Layer 3)

11-256

11-256

Using show and debug Commands Multicast PIE Installation Fails

11-262

 

Multicast CLI Unavailable Although PIE Is Installed

11-263

“This command not authorized” Error Message

11-263

Dynamic IGMP Failure

11-263

Traffic Fails on Some Interfaces

11-267

Traffic Fails on Some Interfaces—MGID Throughput Loss at Receiver Interfaces

11-268

11-268

Reverse Path Forwarding IP Address Problems

11-268

I NDEX

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

11-268 I NDEX Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

OL-23591-02

ix

Contents

Contents Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router x OL-23591-02

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

x

OL-23591-02

Preface This guide describes how to troubleshooting a router using the Cisco IOS XR software.

Preface

Preface This guide describes how to troubleshooting a router using the Cisco IOS XR software. This

This guide describes how to troubleshooting a router using the Cisco IOS XR software.

This preface contains the following sections:

Changes to This Document, page xi

Obtaining Documentation and Submitting a Service Request, page xii

Changes to This Document

Table 1 lists the technical changes made to this document since it was first printed.

Table 1

Changes to This Document

Revision

Date

Change Summary

OL-23591-02

April, 2011

Added details for the following topics:

Chapter 1, “General Troubleshooting Procedures”—Added information on prerequisite documentation for troubleshooting, Cisco-support task ID, show tech-support command, displaying ASIC errors, gathering logs and system information. Modified the information on diagnostics.

Divided interface troubleshooting into two separate chapters—Chapter 2, “Verifying and Troubleshooting Interface Status”and Chapter 3, “Troubleshooting Interface Connectivity.”

Chapter 3, “Troubleshooting Interface Connectivity”—Added information on connectivity fault management (CFM).

Chapter 6, “Troubleshooting Layer 3 Connectivity”—Added information on CEF and interface accounting.

Chapter 7, “Troubleshooting Router Switch Fabric and Data Path”—Added information on NP counters.

Chapter 8, “Troubleshooting MPLS Services”—Corrected syntax of several commands.

continued

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

continued Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router OL-23591-02 xi

OL-23591-02

xi

Preface

Table 1

Changes to This Document (continued)

Revision

Date

Change Summary

OL-23591-02

April, 2011

continued

Chapter 9, “Troubleshooting L2VPN and Ethernet Services”—Enhanced information about VLAN verification, included sample VLAN and PW configurations, added a section on Verifying MAC Address Updates, enhanced information about multipoint Layer 2 services (VPLS), moved information on DHCP snooping to this chapter, enhanced information on MST access gateways (MSTAGs). Reorganized the chapter for ease of use.

Chapter 10, “Troubleshooting Quality of Service and Access Control Lists”—Added information on queue conform and queue exceed counters displayed by the policy-map command.

Chapter 11, “Troubleshooting Multicast Services”—Reorganized

this

chapter to highlight IGMP snooping (Layer 2 MC) and native

MC

(Layer 3), and added information to each of these sections.

OL-23591-01

November

(Initial release of this document as a multichapter book.) The content was

2010

reorganized for usability and updated to reflect Release 3.9 and 4.0 features.

OL-20794-01

December

Initial release of this document as a single module.

2009

Obtaining Documentation and Submitting a Service Request

For information on obtaining documentation, submitting a service request, and gathering additional information, see the monthly What’s New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at:

Subscribe to the What’s New in Cisco Product Documentation as a Really Simple Syndication (RSS) feed and set content to be delivered directly to your desktop using a reader application. The RSS feeds are a free service and Cisco currently supports RSS version 2.0.

a free service and Cisco currently supports RSS version 2.0. Cisco IOS XR Troubleshooting Guide for

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

xii

OL-23591-02

CHAPTER 1 General Troubleshooting Procedures This chapter describes general troubleshooting techniques you can use to

CHAPTER

1

General Troubleshooting Procedures

CHAPTER 1 General Troubleshooting Procedures This chapter describes general troubleshooting techniques you can use to

This chapter describes general troubleshooting techniques you can use to troubleshoot the Cisco ASR 9000 Aggregation Series Router. It includes the following sections:

Prerequisite Documentation for Troubleshooting, page 1-1

Verifying and Troubleshooting CLI Access, page 1-2

Validating and Troubleshooting Installation of the Cisco IOS XR Software Package, page 1-7

Validating and Troubleshooting Cisco IOS XR Software Configuration, page 1-16

Verifying the System, page 1-26

Troubleshooting the Backplane Ethernet Control System, page 1-41

Basic Cisco IOS XR Verification and Troubleshooting Commands, page 1-46

Displaying ASIC Errors, page 1-54

Using Trace Commands, page 1-56

MIB Location, page 1-57

Gathering Information Before You Call Cisco TAC, page 1-58

Prerequisite Documentation for Troubleshooting

As a starting point for troubleshooting, we strongly recommend that you have a system of maintaining and accessing detailed information about your network and ASR 9000 router. This should include:

Current documentation about the system, including chassis numbers, serial numbers, installed cards, and location of chassis details.

Diagrams illustrating the connectivity of the router control plane Ethernet network.

Detailed documentation about the network, including the following:

Up-to-date internetwork map that outlines the physical location of all the devices on the network and how they are connected, as well as a logical map of interfaces, network addresses, network numbers, subnetworks, and so on

List of all network protocols implemented in your network; and for each of the protocols implemented, a list of the network numbers, subnetworks, zones, areas, and so on that are associated with them

All points of contact to external networks

Routing protocol for each external network connection

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

network connection Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router OL-23591-02

OL-23591-02

1-1

Verifying and Troubleshooting CLI Access

Chapter 1

General Troubleshooting Procedures

Established baseline for your network, that is, the normal network behavior and performance at different times of the day so that you can compare any problems with a baseline

Name of the device that is the spanning tree root bridge for the system control plane Ethernet network

Captured output of all commands

Verifying and Troubleshooting CLI Access

Ensure that the system has been booted. If the system has not booted, see Cisco IOS XR Getting Started Guide for the Cisco ASR 9000 Aggregation Services Router for information on booting a router running Cisco IOS XR software. The following CLI access troubleshooting information is provided:

General CLI Access Information, page 1-2

User Access Privileges, page 1-2

Cisco-support Task ID, page 1-3

CLI Access Through a Console Port, page 1-3

CLI Access Through a Terminal Server, page 1-3

CLI Access Through the Management Ethernet Interface, page 1-4

General CLI Access Information

The following CLI access information applies to a console port, terminal server, and management Ethernet interface connections.

Once the terminal emulation software is started and you press Enter, a router prompt should appear. If no prompt appears, verify the physical connection to the console port and press Enter again. If the prompt still does not appear, contact Cisco Technical Support. See the “Obtaining Documentation and Submitting a Service Request” section on page xii for Cisco Technical Support contact information.

If a prompt appears, indicating that the CLI is accessible, but your login username and password are invalid, you are prevented from accessing the router. Verify that you have the correct username and password. If you have the correct username and password, but are locked out of the router, you may need to perform password recovery to access the system again. See Cisco ASR 9000 Aggregation Services Router ROM Monitor Guide for password recovery procedures.

User Access Privileges

When you log on to the router, use a username that is associated with a valid user group that has the authorization to execute the required commands. If you suspect user group assignment is preventing you from using a command, contact your AAA administrator for assistance.

See Cisco ASR 9000 Aggregation Services Router System Security Command Reference and Cisco ASR 9000 Aggregation Services Router System Security Configuration Guide for information on users, usernames, and user groups.

Guide for information on users, usernames, and user groups. Cisco IOS XR Troubleshooting Guide for the

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

1-2

OL-23591-02

Chapter 1

General Troubleshooting Procedures

Verifying and Troubleshooting CLI Access

Cisco-support Task ID

Many of the troubleshooting commands can be performed only by users who are assigned to a user group that includes the cisco-support task ID. Users without the cisco-support task ID receive a “This command is not authorized” response if they attempt to use those commands. The cisco-support commands are normally reserved for use by Cisco Technical Support personnel, because there is some risk that they may cause performance or other issues.

some risk that they may cause pe rformance or other issues. Caution These Cisco support commands

Caution

that they may cause pe rformance or other issues. Caution These Cisco support commands are normally
that they may cause pe rformance or other issues. Caution These Cisco support commands are normally

These Cisco support commands are normally reserved for use by Cisco Technical Support personnel only. There is some risk that they may cause performance or other issues that impact products without proper usage, and we highly recommend that you contact Cisco Technical Support prior to using any of these commands. See the “Obtaining Documentation and Submitting a Service Request” section on

page xii for information on contacting Cisco TAC.

on page xii for information on contacting Cisco TAC. CLI Access Through a Console Port The

CLI

Access Through a Console Port

The first time a router is started, you must use a direct connection to the console port to connect to the router and enter the initial configuration. See Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for information on connecting to the router through a console port. When you use

direct connection to the Console port, CLI commands are entered at a terminal or at a computer running terminal emulation software. A direct Console port connection is useful for entering initial configurations and performing some debugging tasks.

a

CLI

Access Through a Terminal Server

A terminal server connection provides a way to access the Console port from a remote location. A

terminal server connection is used when you need to perform tasks that require Console port access from

a remote location.

Connecting to a router through a terminal server is similar to directly connecting through the Console port. For both connection types, the physical connection takes place through the Console port. The difference is that the terminal server connects directly to the Console port, and you must use a Telnet session to establish communications through the terminal server to the router.

If you are unable to access the CLI through a terminal server, perform the following procedure.

Step 1

Disable flow control (XON/XOFF) on the Terminal Server.

Step 2

Disable local echo mode on the Terminal Server.

Step 3

Verify the router name configured using the hostname command.

Step 4

Check whether the port address is configured correctly.

Step 5

Verify whether the address (interface) used for the reverse Telnet is up/up. The output of the show interfaces brief command provides this information. Cisco recommends you to use loopbacks because they are always up.

Step 6

Ensure that you have the correct type of cabling. For example, you must not use a crossover cable to extend the length.

Step 7

Establish a Telnet connection to the IP address port to test direct connectivity. You must Telnet from both an external device and the terminal server. For example, telnet 172.21.1.1 2003.

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

172.21.1.1 2003 . Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

OL-23591-02

1-3

Verifying and Troubleshooting CLI Access

Chapter 1

General Troubleshooting Procedures

Step 8

Ensure that you have the transport input telnet command under the line for the target device. The target device is the device that is connected to the terminal server.

Step 9

Use a PC/dumb terminal to connect directly to the console of the target router. The target router is the device connected to the terminal server. This step helps you identify the presence of a port issue.

Step 10

If you are disconnected, check timeouts. You can remove or adjust timeouts.

check timeouts. You can remove or adjust timeouts. Note If you encounter authentication failures, remember
check timeouts. You can remove or adjust timeouts. Note If you encounter authentication failures, remember

Note

If you encounter authentication failures, remember that the terminal server performs the first authentication (if configured), while the device to which you try to connect performs the second authentication (if configured). Verify whether AAA is configured correctly on both the terminal server

and the connecting device.

and the connecting device.

Step 11

Contact Cisco Technical Support. See the “Obtaining Documentation and Submitting a Service Request” section on page xii for Cisco Technical Support contact information.

CLI Access Through the Management Ethernet Interface

The Management Ethernet interface allows you to manage the router using a network connection. Before you can use the Management Ethernet interface, the interface must be configured. See Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for information on configuring the interface.

Once configured, the network connection takes place between client software on a workstation computer and a server process within the router. The type of client software you use depends on the server process you use. See Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for information on the client and server services supported by the Cisco IOS XR software.

If you are unable to access the CLI through a management Ethernet interface, perform the following procedure.

SUMMARY STEPS

1. show interface MgmtEth interface-instance

2. show arp MgmtEth interface-instance

3. show ipv4 interface type instance

4. ping

5. Contact Cisco Technical Support if the problem is not resolved

Cisco Technical Support if the problem is not resolved Cisco IOS XR Troubleshooting Guide for the

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

1-4

OL-23591-02

Chapter 1

General Troubleshooting Procedures

Verifying and Troubleshooting CLI Access

DETAILED STEPS

Step 1

Step 2

Step 3

Command or Action

Purpose

show interfaces MgmtEth interface-instance

Displays statistics for all interfaces configured on the router.

Example:

Check the following:

RP/0/RSP0/CPU0:router# show interfaces MgmtEth

MgmtEth interface is up

0/RSP0/CPU0/0

Line protocol (state of the Layer 2 line protocol) is up

Number of input and output errors

If an interface is administratively down, use the no shutdown command to enable the interface.

If an interface is down (operationally down), input or output errors are not within an acceptable range, the management Ethernet interface is not enabled when the no shutdown command is used, or the line protocol is down, see Chapter 2, “Verifying and Troubleshooting Interface Status,” for detailed information on troubleshooting interfaces.

If the interface is up and the input and output errors are within an acceptable range, proceed to Step 2.

show arp MgmtEth interface-instance

Displays the Address Resolution Protocol (ARP) table for the management Ethernet interface.

Example:

Ensure that the expected ARP entries exist for the management Ethernet interface.

RP/0/RSP0/CPU0:router# show arp MgmtEth

0/RSP0/CPU0/0

 

If the expected ARP entries exist, proceed to Step 3.

If the expected ARP entries do not exist, verify the physical layer Ethernet interface connectivity. Use the show arp trace command to display the ARP entries in the buffer. See the Chapter 2, “Verifying and Troubleshooting Interface Status,” for more information on troubleshooting interfaces.

show ipv4 interface type instance

Displays the usability status of interfaces configured for

IPv4.

Example:

If the interface is in the expected state, proceed to Step 4.

RP/0/RSP0/CPU0:router# show ipv4 interface MgmtEth 0/RSP0/CPU0/0

If the status of the interface is not as expected, see Chapter 2, “Verifying and Troubleshooting Interface Status,” for more information on troubleshooting interfaces.

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

interfaces. Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router OL-23591-02 1-5

OL-23591-02

1-5

Verifying and Troubleshooting CLI Access

Chapter 1

General Troubleshooting Procedures

Step 4

Step 5

Command or Action

Purpose

ping

Checks host reachability and network connectivity on the IP network.

Example:

Note

Enter a specific IP address or follow the prompts to send the ping message to the target address.

RP/0/RSP0/CPU0:router# ping

If no problems are detected, proceed to Step 5.

Contact Cisco Technical Support.

If the problem is not resolved, contact Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request” section on page xii.

Examples

The output from the show interfaces MgmtEth command displays the status of the management Ethernet interface. In the following example, the management Ethernet interface is up, and there are 20 input errors and 8 output errors.

RP/0/RSP0/CPU0:router# show interface MgmtEth 0/RSP0/CPU0/0 Tue Sep 14 14:21:07.496 DST MgmtEth0/RSP0/CPU0/0 is up, line protocol is up Interface state transitions: 1

Hardware is Management Ethernet, address is 001b.53ff.4a62 (bia 001b.53ff.4a62) Description: Connected to Lab LAN Internet address is 172.29.52.137/24 MTU 1514 bytes, BW 100000 Kbit (Max: 100000 Kbit) reliability 73/255, txload 0/255, rxload 0/255 Encapsulation ARPA, Half-duplex, 100Mb/s, THD, link type is autonegotiation output flow control is off, input flow control is off loopback not set, ARP type ARPA, ARP timeout 04:00:00 Last input 00:00:00, output 00:00:00 Last clearing of "show interface" counters never

5

minute input rate 2000 bits/sec, 3 packets/sec

5

minute output rate 0 bits/sec, 0 packets/sec 373082 packets input, 51028824 bytes, 239105 total input drops

62028 drops for unrecognized upper-level protocol

Received 2601 broadcast packets, 194653 multicast packets 10 runts, 0 giants, 0 throttles, 0 parity

20 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

45232 packets output, 3042775 bytes, 0 total output drops

Output 24 broadcast packets, 0 multicast packets 8 output errors, 0 underruns, 0 applique, 0 resets

0 output buffer failures, 0 output buffers swapped out

1 carrier transitions

The output from the show arp MgmtEth 0/RSP0/CPU0/0 command displays the ARP table for the management Ethernet interface. Use the output from this command to verify that there are dynamic ARP addresses in the table and that ARP is functioning over the interface. The output shows that ARP is functioning over the management Ethernet interface 0/RSP0/CPU0/0.

RP/0/RSP0/CPU0:router# show arp MgmtEth 0/RSP0/CPU0/0 Tue Sep 14 14:24:03.962 DST

-------------------------------------------------------------------------------

0/RSP0/CPU0

-------------------------------------------------------------------------------

Address

Age

Hardware Addr

State

Type Interface

Address Age Hardware Addr State Type Interface Cisco IOS XR Troubleshooting Guide for the Cisco ASR

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

1-6

OL-23591-02

Chapter 1

General Troubleshooting Procedures

Validating and Troubleshooting Installation of the Cisco IOS XR Software Package

172.29.52.1

01:44:00

0000.0c07.ac01 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.13

01:16:59

0010.79e9.6038 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.21

01:40:25

0022.0d5a.a6c4 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.27

02:18:16

0012.7fd6.ba08 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.28

02:05:29

0012.7fd6.ba09 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.32

01:42:16

0022.0d26.3bc5 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.36

02:39:34

0026.527c.5341 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.46

01:36:50

0012.7fd6.b9aa Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.47

01:36:39

0012.7fd6.b9ab Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.60

01:35:20

0003.a099.8000 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.69

00:00:00

001b.7852.4bd1 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.70

01:23:38

0011.93ef.e8e6 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.71

02:00:47

0011.93ef.e8fe Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.75

01:44:59

5a59.0000.0202 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.76

01:41:10

0011.93ef.e8ea Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.81

00:15:35

001a.6c40.d89c Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.83

00:21:05

001a.6c40.d89c Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.127

01:43:38

0013.c4cb.a200 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.134

01:15:53

001f.6c26.7fc0 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.135

01:01:46

001f.6c25.c480 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.136

00:43:39

0022.5560.8840 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.137

-

001b.53ff.4a62 Interface ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.138

-

001b.53ff.4a62 Interface ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.161

01:32:12

0019.aaa3.3d48 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.171

00:16:12

001c.5838.5b28 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.172

00:17:47

001c.5838.5b29 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.173

01:57:04

0015.c75f.09f8 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.180

01:26:50

0015.c75f.0800 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.217

01:16:21

0019.aaa3.b5ff Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.226

01:28:32

0010.f60e.8400 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

172.29.52.243

01:48:25

001e.79c1.e0c1 Dynamic

ARPA MgmtEth0/RSP0/CPU0/0

The ping command checks to see if the neighbor is reachable.

RP/0/RSP0/CPU0:router# ping 172.16.52.28 count 10 Tue Sep 14 14:36:52.441 DST Type escape sequence to abort. Sending 10, 100-byte ICMP Echos to 172.16.52.28, timeout is 2 seconds:

!!!!!!!!!! Success rate is 100 percent (10/10), round-trip min/avg/max = 1/1/2 ms

Validating and Troubleshooting Installation of the Cisco IOS XR Software Package

The Cisco IOS XR software is divided into software packages allowing you to select which features run on your router. Each package contains the components to perform a specific set of router functions, such as routing, security, or Modular Services Card (MSC) support. Bundles are groups of packages that can be downloaded as a set. For example, the Unicast Routing Core Bundle provides six packages for use on every router.

This section provides information on how to validate and troubleshoot the Cisco IOS XR software package installation. The following sections are provided:

Verifying the Software Version, page 1-8

Validating the Installation, page 1-10

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

page 1-10 Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router OL-23591-02

OL-23591-02

1-7

Validating and Troubleshooting Installation of the Cisco IOS XR Software Package

Chapter 1

General Troubleshooting Procedures

Verifying the Software Version

To verify the Cisco IOS XR software version, perform the following procedure.

SUMMARY STEPS

 

1. show version

2. show install

DETAILED STEPS

Step 1

Step 2

Command or Action

Purpose

show version

Displays a variety of system information, including hardware and software version, router uptime, boot settings (configuration register), and active software.

Example:

RP/0/RSP0/CPU0:router# show version

Determine if all expected packages are installed and the current software versions are the expected versions.

If the expected packages are not installed or are not the expected version, install the correct package. See Cisco ASR 9000 Aggregation Series Router Getting Started Guide for information on installing and upgrading Cisco IOS XR software packages.

show install

Displays a list of all installed and active packages on each node.

Example:

Determine if the expected packages are installed on each node.

RP/0/RSP0/CPU0:router# show install

If the software or active package versions are not as expected for a node, the package is not compatible with the node for which it is being activated, or the package being activated is not compatible with the current active software set, install the correct software or package on the node. See Cisco ASR 9000 Aggregation Series Router Getting Started Guide for information on installing and upgrading Cisco IOS XR software packages.

The following example shows that the Cisco IOS XR software and active packages are version 4.0.0.

RP/0/RSP0/CPU0:router# show version

Cisco IOS XR Software, Version 4.0.0 Copyright (c) 2010 by cisco Systems, Inc.

ROM: System Bootstrap, Version 1.04(20100216:021454) [ASR9K ROMMON],

router uptime is 1 day, 18 hours, 34 minutes System image file is "bootflash:disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm"

cisco ASR9K Series (MPC8641D) processor with 4194304K bytes of memory. MPC8641D processor at 1333MHz, Revision 2.2

2 Management Ethernet

processor at 1333MHz, Revision 2.2 2 Management Ethernet Cisco IOS XR Troubleshooting Guide for the Cisco

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

1-8

OL-23591-02

Chapter 1

General Troubleshooting Procedures

Validating and Troubleshooting Installation of the Cisco IOS XR Software Package

12

DWDM controller(s)

12

TenGigE

40

GigabitEthernet

2

SONET/SDH

2

Packet over SONET/SDH

219k bytes of non-volatile configuration memory. 975M bytes of compact flash card. 33994M bytes of hard disk. 1605616k bytes of disk0: (Sector size 512 bytes). 1605616k bytes of disk1: (Sector size 512 bytes).

Configuration register on node 0/RSP0/CPU0 is 0x0 Boot device on node 0/RSP0/CPU0 is disk0:

Package active on node 0/RSP0/CPU0:

asr9k-optics-supp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-optics-supp-4.0.0

Built on Wed Sep

By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie

8 16:17:30 DST 2010

asr9k-fwding, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-fwding-4.0.0

Built on Wed Sep

By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie

8 16:12:40 DST 2010

asr9k-cpp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-cpp-4.0.0

Built on Wed Sep

By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie

8 16:13:28 DST 2010

asr9K-doc-supp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9K-doc-supp-4.0.0

Built on Wed Sep

By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie

8 16:16:57 DST 2010

asr9k-scfclient, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-scfclient-4.0.0

Built on Wed Sep

--More--

.

.

.

8 16:13:26 DST 2010

The following example shows that the Cisco IOS XR software and active packages are version 4.0.0. If there is an expected package missing or an active package is not an expected package, install and activate the missing package or upgrade the unexpected package to the appropriate package. See Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for details on installing, activating, and upgrading software packages.

RP/0/RSP0/CPU0:router# show install

Node 0/RSP0/CPU0 [RP] [SDR: Owner] Boot Device: disk0:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm Active Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-doc-p-4.0.0

disk0:asr9k-k9sec-p-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mgbl-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/1/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Active Packages:

disk0:asr9k-mini-p-4.0.0

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router OL-23591-02 1-9

OL-23591-02

1-9

Validating and Troubleshooting Installation of the Cisco IOS XR Software Package

Chapter 1

General Troubleshooting Procedures

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/2/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Active Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/4/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Active Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/6/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Active Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Validating the Installation

Validate the Cisco IOS XR software package installation to ensure the packages were installed correctly. The following commands are used to validate the currently installed software packages:

install verify Command, page 1-10

show install active Command, page 1-12

show install committed Command, page 1-14

install verify Command

Use the install verify command to verify the consistency of a previously installed software set with the package file from which it originated.

This command can be used as a debugging tool to verify the validity of the files that constitute the packages to determine if there are any corrupted files. The command is also used to check that the install infrastructure is up and running and to determine if all files are expected. If there are corrupted files, see Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for information on deactivating and removing software packages and adding and activating software packages.

packages and adding and activating software packages. Cisco IOS XR Troubleshooting Guide for the Cisco ASR

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

1-10

OL-23591-02

Chapter 1

General Troubleshooting Procedures

Validating and Troubleshooting Installation of the Cisco IOS XR Software Package

Instal lation of the Cisco IOS XR Software Package Note The install verify command can take

Note

The install verify command can take up to two minutes per package to process.

command can take up to two minutes per package to process. Note The install verify command
command can take up to two minutes per package to process. Note The install verify command

Note

The install verify command ignores secure domain router (SDR) boundaries and performs the operation

in global scope.

(SDR) boundaries and performs the operation in global scope. The following example shows the output of

The following example shows the output of the install verify command. The output is used to verify the consistency of a previously installed software set with the package file from which it originated.

RP/0/RSP0/CPU0:router(admin)# install verify

Sat Sep 25 08:18:14.077 DST Install operation 3 '(admin) install verify packages' started by user_A 'dwolman-r' via CLI at 08:18:14 DST Sat Sep 25 2010. The install operation will continue asynchronously.

RP/0/RSP0/CPU0:router(admin)#Info:

being verified.

This operation can take up to 2 minutes per package

Info:

Please be patient.

Info:

0/0/CPU0 [LC] [SDR: Owner]

Info:

meta-data: [SUCCESS] Verification Successful.

Info:

/install/asr9k-optics-supp-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/asr9k-fwding-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/asr9k-cpp-4.0.0: [SUCCESS] Verification Successful.

Info:

/install/asr9k-scfclient-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/iosxr-video-adv-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/iosxr-mpls-4.0.0: [SUCCESS] Verification Successful.

Info:

/install/iosxr-mcast-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/iosxr-routing-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/iosxr-infra-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/iosxr-fwding-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/iosxr-diags-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/asr9k-adv-video-supp-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/asr9k-diags-supp-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/asr9k-mcast-supp-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/asr9k-base-4.0.0: [SUCCESS] Verification Successful.

Info:

0/6/CPU0 [LC] [SDR: Owner]

Info:

meta-data: [SUCCESS] Verification Successful.

Info:

/install/asr9k-optics-supp-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/asr9k-fwding-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/asr9k-cpp-4.0.0: [SUCCESS] Verification Successful.

Info:

/install/asr9k-scfclient-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/iosxr-video-adv-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/iosxr-mpls-4.0.0: [SUCCESS] Verification Successful.

Info:

/install/iosxr-mcast-4.0.0: [SUCCESS] Verification

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

Verification Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router OL-23591-02 1-11

OL-23591-02

1-11

Validating and Troubleshooting Installation of the Cisco IOS XR Software Package

Chapter 1

General Troubleshooting Procedures

Info:

Successful.

Info:

/install/iosxr-routing-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/iosxr-infra-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/iosxr-fwding-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/iosxr-diags-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/asr9k-adv-video-supp-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/asr9k-diags-supp-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/asr9k-mcast-supp-4.0.0: [SUCCESS] Verification

Info:

Successful.

Info:

/install/asr9k-base-4.0.0: [SUCCESS] Verification Successful.

Info:

0/5/CPU0 [LC] [SDR: Owner]

.

.

.

Info:

Verification Summary:

Info:

0/0/CPU0: SUCCESSFUL. No anomalies found.

Info:

0/6/CPU0: SUCCESSFUL. No anomalies found.

Info:

0/5/CPU0: SUCCESSFUL. No anomalies found.

Info:

0/7/CPU0: SUCCESSFUL. No anomalies found.

Info:

0/1/CPU0: SUCCESSFUL. No anomalies found.

Info:

0/4/CPU0: SUCCESSFUL. No anomalies found.

Info:

0/2/CPU0: SUCCESSFUL. No anomalies found.

Info:

0/RSP0/CPU0: SUCCESSFUL. No anomalies found.

Info:

The system needs no repair.

Install operation 3 completed successfully at 08:19:48 DST Sat Sep 25 2010.

show install active Command

Use the show install active command to display active software packages. Verify that the command output matches the output of the show install committed command. If the output does not match, when you reload the router, the software displayed in the show install committed command output is the software that will be loaded. For example, the following output shows two different software package versions, one is the active version and the other is the committed version, so when the router reloads, the 3.9.1 version will be loaded even though 4.0.0 is the currently active version on 0/RSP0/CPU0.

RP/0/RSP0/CPU0:router(admin)# show install active location 0/RSP0/cpu0

Node 0/RSP0/CPU0 [RP] [SDR: Owner] Boot Device: disk0:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm << 4.0.0 is active, not committed Active Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-doc-p-4.0.0

disk0:asr9k-k9sec-p-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mgbl-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

RP/0/RSP0/CPU0:router(admin)# show install committed location 0/RSP0/cpu0

Node 0/RSP0/CPU0 [RP] [SDR: Owner] Boot Device: disk0:

Boot Image: /disk0/asr9k-os-mbi-3.9.1/mbiasr9k-rp.vm<< 3.9.1 is committed

<< 3.9.1 is committed Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

1-12

OL-23591-02

Chapter 1

General Troubleshooting Procedures

Validating and Troubleshooting Installation of the Cisco IOS XR Software Package

Committed Packages:

disk0:asr9k-mini-p-3.9.1

disk0:asr9k-optic-3.9.1

disk0:asr9k-doc-p-3.9.1

disk0:asr9k-k9sec-p-3.9.1

disk0:asr9k-video-p-3.9.1

disk0:asr9k-mpls-p-3.9.1

disk0:asr9k-mgbl-p-3.9.1

disk0:asr9k-mcast-p-3.9.1

If the expected active software packages are not displayed, install the packages (if required) and activate the packages. See Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for information on installing and activating Cisco IOS XR software packages. The following example output shows the active packages for all cards in a router. The output displays the disk on which each package is located.

RP/0/RSP0/CPU0:router# show install active

Node 0/RSP0/CPU0 [RP] [SDR: Owner] Boot Device: disk0:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm Active Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-doc-p-4.0.0

disk0:asr9k-k9sec-p-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mgbl-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/0/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Active Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/1/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Active Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/2/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Active Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/4/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

Boot Device: mem: Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

OL-23591-02

1-13

Validating and Troubleshooting Installation of the Cisco IOS XR Software Package

Chapter 1

General Troubleshooting Procedures

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Active Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/5/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Active Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/6/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Active Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/7/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Active Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

The output shows the name of the disk on which the packages are located. In the above example, the active packages for each node are on disk0, and for all nodes, the composite package asr9k-os-mbi-4.0.0 is active. Additional packages shown are optional packages that have been activated after the initial loading of the Cisco ASR 9000 Aggregation Series Router Unicast Routing Core Bundle.

show install committed Command

Use the show install committed command to display committed software packages. The committed software packages are the software packages that will be booted on a router reload.

Committed packages are the packages that are persistent across router reloads. If you install and activate

a package, it remains active until the next router reload. If you commit a package set, all packages in that set remain active across router reloads until the package set is replaced with another committed package set. The show install committed command is useful to ensure software is installed and committed after

a router reload. If the expected software is not installed and committed, see

Cisco ASR 9000 Aggregation Services Router Getting Started Guide for information on installing and

committing Cisco IOS XR software packages.

The following command output shows the committed software packages on all cards in the router. The output displays the disk on which each package is located.

RP/0/RSP0/CPU0:router# show install committed

Node 0/RSP0/CPU0 [RP] [SDR: Owner]

show install committed Node 0/RSP0/CPU0 [RP] [SDR: Owner] Cisco IOS XR Troubleshooting Guide for the Cisco

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

1-14

OL-23591-02

Chapter 1

General Troubleshooting Procedures

Validating and Troubleshooting Installation of the Cisco IOS XR Software Package

Boot Device: disk0:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm Committed Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-doc-p-4.0.0

disk0:asr9k-k9sec-p-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mgbl-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/0/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Committed Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/1/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Committed Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/2/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Committed Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/4/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Committed Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/5/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Committed Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/6/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

Boot Device: mem: Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

OL-23591-02

1-15

Validating and Troubleshooting Cisco IOS XR Software Configuration

Chapter 1

General Troubleshooting Procedures

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Committed Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

Node 0/7/CPU0 [LC] [SDR: Owner] Boot Device: mem:

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Committed Packages:

disk0:asr9k-mini-p-4.0.0

disk0:asr9k-optic-4.0.0

disk0:asr9k-video-p-4.0.0

disk0:asr9k-mpls-p-4.0.0

disk0:asr9k-mcast-p-4.0.0

The output shows the name of the disk on which the packages are located. In the above example, the committed packages for each node are on disk0, and for all nodes, the composite package asr9k-os-mbi-4.0.0 is committed. Additional packages shown are optional packages that have been committed after the initial loading of the Cisco ASR 9000 Aggregation Series Router Unicast Routing Core Bundle.

Validating and Troubleshooting Cisco IOS XR Software Configuration

Validating the Cisco IOS XR software configuration includes collecting configuration information on the router to determine configuration changes and verifying the current running configuration. When a configuration fails during a commit, the failed configuration can be viewed to help determine why the configuration was not committed.

The following sections are provided:

Local and Global Configurations, page 1-16

Collecting Configuration Information, page 1-19

Verifying the Running Configuration, page 1-20

Using the show configuration failed Command, page 1-24

Local and Global Configurations

To troubleshoot configurations, you need to determine whether the problem is in the local configuration or the shared (global) configuration.

The local configuration is specific to the individual LC or RP to which it belongs. Every LC and RP has a data store containing the local data for that node, including configuration and operational data for the local interfaces. An example of a local configuration is the port designations on a particular LC.

The shared (global) configuration applies to the entire router, and is shared with all of the LCs and RPs. An example of a shared configuration is the routing protocol parameters.

To view the local configuration, use the show running-config interface * command. The output displays all the configured interfaces on the node.

output displays all the configured interfaces on the node. Cisco IOS XR Troubleshooting Guide for the

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

1-16

OL-23591-02

Chapter 1

General Troubleshooting Procedures

Validating and Troubleshooting Cisco IOS XR Software Configuration

RP/0/RSP0/CPU0:router# show running-config interface *

interface Bundle-Ether16 description Connect to router-S Port-Ch 16 mtu 9216 bundle maximum-active links 1

!

interface Bundle-Ether16.160 l2transport description Connect to router-S Port-Ch 16 Service Instance 160 encapsulation dot1q 160

!

.

.

.

!

interface Loopback0 ipv4 address 10.144.144.144 255.255.255.255

!

interface tunnel-ip10

!

interface tunnel-te44190 description Primary GE Tunnel from router-S to router-T ipv4 unnumbered Loopback0 priority 0 0 autoroute announce signalled-bandwidth 100000 destination 10.19.19.19 fast-reroute record-route path-option 1 explicit name Primary_GE_Path_to_router-T ospf 100 area 0

!

.

.

.

interface MgmtEth0/RSP0/CPU0/0 description Connected to LAN ipv4 address 172.29.52.137 255.255.255.0

!

interface MgmtEth0/RSP0/CPU0/1

shutdown

!

interface GigabitEthernet0/0/0/0

shutdown

!

interface GigabitEthernet0/0/0/1

shutdown

!

interface GigabitEthernet0/0/0/2

shutdown

!

.

.

.

interface TenGigE0/7/0/1

shutdown

!

interface TenGigE0/7/0/2

shutdown

!

interface TenGigE0/7/0/3

shutdown

!

interface POS0/2/0/0 description Connected to PE_router-2 POS 0/2/0/0

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

POS 0/2/0/0 Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router OL-23591-02

OL-23591-02

1-17

Validating and Troubleshooting Cisco IOS XR Software Configuration

Chapter 1

General Troubleshooting Procedures

!

interface POS0/2/0/1 description Connected to PE_router-3 POS 0/2/0/1

!

controller SONET0/2/0/0

clock source internal

!

controller SONET0/2/0/1

clock source internal

!

Use the show sysdb trace commands to display the contents of the system database after a configuration change. The trace information includes a history of any changes to the running configuration. You can specify either a local node or the shared plane.

The following example output shows the contents of the local database, that is, for a specific location (node):

RP/0/RSP0/CPU0:router# show sysdb trace verification location 0/5/cpu0 reverse

.

.

.

Timestamp

nid

reqid

jid

tid reg_hndl connid

action

path

432 wrapping entries (8192 possible, 158 filtered, 590 total)

Sep 23 04:35:39.969

0/RSP0/CPU0 8168

354

1

94

4483

apply

reply

'--'

Sep 23 04:35:39.960

0/RSP0/CPU0 8168

354

1

94

4505

Apply

called

'cfg/if/act/tunnel-ip10/v'

 

Sep 23 04:35:39.960 reply: accept

0/RSP0/CPU0 8168 '--'

354

1

94

4505

verify

Sep 23 04:35:39.685

0/RSP0/CPU0

8168

354

1

94

4505

Verify

called

'cfg/if/act/tunnel-ip10/v'

 

Sep 23 04:35:39.678

0/RSP0/CPU0 0

354

1

94

4505

register

'cfg/if/act/tunnel-ip[0-9]*/mtu/tunnel-ip'

Sep 23 04:35:39.678

0/RSP0/CPU0 0

354

1

94

4505

register

'cfg/if/act/tunnel-ip[0-9]*/im/bw'

 

Sep 23 04:35:39.678

0/RSP0/CPU0 0

354

1

94

4505

register

'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/keepalive'

 

Sep 23 04:35:39.678

0/RSP0/CPU0 0

354

1

94

4505

register

'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/dfbit_disable'

Sep 23 04:35:39.678

0/RSP0/CPU0 0

354

1

94

4505

register

'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/ttl'

Sep 23 04:35:39.678

0/RSP0/CPU0 0

354

1

94

4505

register

'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/tos'

Sep 23 04:35:39.678

0/RSP0/CPU0 0

354

1

94

4505

register

'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/mode'

.

.

.

The following example output shows the contents of the shared database, that is, the configuration data that is shared with all LC and RP in the router:

RP/0/RSP0/CPU0:router# show sysdb trace verification shared-plane reverse

Config Shared Server ==================== Timestamp nid reqid jid tid reg_hndl connid action path 2259 wrapping entries (4096 possible, 0 filtered, 2259 total)

wrapping entries (4096 possible, 0 filtered, 2259 total) Cisco IOS XR Troubleshooting Guide for the Cisco

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

1-18

OL-23591-02

Chapter 1

General Troubleshooting Procedures

Validating and Troubleshooting Cisco IOS XR Software Configuration

Sep 23 19:34:40.202

0/3/CPU0

0

241

8