Layer by Layer Troubleshooting with a Cisco Router by David Davis - January 8, 2009 Every network admin is going to have

trouble with network links on a Cisco router, at one point or another. The best way to troubleshoot any networking issues is to use the OSI model and go layer by layer. In my article How to use the OSI Model to Troubleshoot Networks, we talked about the different troubleshooting approaches and how to use them to troubleshoot your network, in general. In this article, you will find out how to use the OSI model to troubleshoot, bottom up, using a Cisco router. Troubleshooting Apps? Locate the Exact Root Cause. Fast. Time is money. Save both when you get to the core of application issues using Network Instruments' advanced monitoring technology. End the blame game between app and network teams. Benefits include:

- Application Insights - Troubleshooting & Performance Optimization - Baselines for Proactive Control - Total Transparency - High-Level Dashboards of Application Health Download our FREE White Paper by industry analyst Jim Metzler.

OSI Model - Bottom Up Troubleshooting

If you will recall, the OSI model starts with the physical layer (layer 1) and goes up to layer 7 (application). When troubleshooting with a Cisco router, much of your time will be spent working in layers 1-3. They are: Layer 3 - Network Layer 2 - Data Link Layer 1 - Physical

Because these layers build on each other, Layer 1 is most critical, without layer 1, layer 2 will not function. Without layer 1 & 2, layer 3 will not function, and so on. For this reason, I start troubleshooting at layer 1, physical, and move on up from there. Router Troubleshooting at OSI Layer 1 & 2 - Physical & Data link

Remember, if Layer 1 isn't up, nothing else will work so make sure you start here. Examples of layer 1 are your T1 circuit or your Ethernet cable - physical connectivity. I usually troubleshoot layer 1 and layer 2 in union because they are so closely paired. Examples of layer 2 - data link - are your line protocol (such as Ethernet, ATM, 802.11, PPP, frame-relay, HDLC, or PPP).

To troubleshoot at these layers, the first thing I would do on your router is a show interface. Here is an example of a LAN Gigabit Ethernet circuit: Router# show interface GigabitEthernet0/0 is up, line protocol is up Hardware is BCM1125 Internal MAC, address is 0015.2b46.5000 (bia 0015.2b46.5000) Description: LAN Connection to Data center Internet address is 10.20.100.1/16 MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 1000Mb/s, link type is autonegotiation, media type is RJ45 output flow-control is XON, input flow-control is XON ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: weighted fair

Output queue: 0/1000/64/0 (size/max total/threshold/drops) Conversations 0/2/256 (active/max active/max total) Reserved Conversations 0/0 (allocated/max allocated) Available Bandwidth 750000 kilobits/sec 5 minute input rate 3218000 bits/sec, 1715 packets/sec 5 minute output rate 1390000 bits/sec, 2129 packets/sec 1416888620 packets input, 15402720 bytes, 0 no buffer Received 0 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 1556005 multicast, 0 pause input 0 input packets with dribble condition detected 1666663097 packets output, 573841802 bytes, 0 underruns 19 output errors, 0 collisions, 3 interface resets 0 babbles, 0 late collision, 0 deferred 19 lost carrier, 0 no carrier, 0 pause output 0 output buffer failures, 0 output buffers swapped out

Here is what a WAN T1or T3 circuit might look like: Routerl# show interface serial 3/0 Serial3/0 is up, line protocol is up Hardware is DSXPNM Serial Description: Sprint T3 Internet address is 10.2.100.2/30 MTU 4470 bytes, BW 9000 Kbit, DLY 200 usec, reliability 255/255, txload 77/255, rxload 26/255

Encapsulation HDLC, crc 16, loopback not set Keepalive set (10 sec) Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 18394 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 927000 bits/sec, 1914 packets/sec 5 minute output rate 2752000 bits/sec, 1504 packets/sec 1560997932 packets input, 3254680247 bytes, 0 no buffer Received 255480 broadcasts, 1 runts, 1 giants, 0 throttles 1567 input errors, 1567 CRC, 976 frame, 496 overrun, 0 ignored, 908 abort 1303636803 packets output, 3737276508 bytes, 0 underruns 0 output errors, 0 collisions, 3 interface resets 0 output buffer failures, 0 output buffers swapped out 1 carrier transitions DSU mode 1, bandwidth 9000, real bandwidth 9000, scramble 0

Here is the quick version: Router# show ip interface brief Interface IP-Address OK? Method Status Protocol GigabitEthernet0/0 10.20.100.1 YES NVRAM up up Serial3/0 10.2.100.2 YES NVRAM up up

Here is what you look for:

Is the interface UP? Is the line protocol UP? If both the interface and line protocol are NOT up, your connection is never going to work. To resolve a line down, I look at the cable or the keepalives To resolve a line protocol down, check to make sure that the protocols match on each side of the connection(notice the "line protocol" on each of the interfaces above). Are you taking input, CRC, framing, or other errors on the line (notice how the serial interface above does show errors)? If so, check your cable or contact your provider.

In general, verify that you have a good cable on each side, verify that line protocols match, and that clocking settings are correct.

If this is an Ethernet connection, is there a link light on the switch?

If this is a serial connection, do you have an external CSU/DSU? If it is an external CSU, check that the Carrier Detect (CD) light & data terminal ready (DTR) lights are on. If not, contact your provider. This also applies if you have an internal Cisco WIC CSU card. If that is the case, take a look at this Cisco link on understanding the lights on that card.

You can, of course, use the Cisco IOS test commands to test your network interfaces with internal staff and with your telecommunications providers.

Do not proceed to upper level layers until your Physical interface on the router shows as being UP and your line protocol is UP. Until then, don't worry about IP addressing, pinging, access-lists or anything like that. Router Troubleshooting at OSI Layer 3 - Network Once you have Layers 1 & 2 working (your show interface command shows the line is "UP & UP", it is time to move on to layer 3 - the OSI Network layer. The easiest thing to do here to see if layer 3 is working is to ping the remote side of the LAN or WAN link from this router. Make sure you ping as close as possible to the router you are trying to communication with - from one side across to the other side.