IP Telephony Troubleshooting

by Ronald Trunk rtrunk@netcraftsmen.net

Page 1

IP Telephony Troubleshooting
By Ron Trunk
Now that IP telephony systems are becoming commonplace, network engineers need to be familiar with IP telephony operations and troubleshooting procedures. IP telephony is almost always a mission-critical application, so keeping the system running smoothly is an important task. A thorough understanding of the interaction between the various components of the IP telephony system will help you quickly identify and resolve most common problems. Your users expect the IP telephony system to be as reliable as a traditional PBX, so resolving problems quickly is important if your users are going to accept the system. If youre responsible for maintaining the network infrastructure (routers, switches, etc.) you need to be able to tell whether the problem is network-related, or whether it is a problem with the PBX configuration. Often, the staff responsible for the network infrastructure is not the same that is responsible for operating and maintaining the PBX functions (adding users, maintaining voicemail, etc.) So its important to get the right resources working to solve the problem (Note: thats not the same as finding someone to blame!). In a traditional PBX, each telephone is directly connected to a physical port on the PBX. The phone communicates directly with the PBX for all the signaling, audio and power. This makes troubleshooting fairly easy either the phone is communicating with the PBX or its not. If it is, the problem is probably a PBX configuration problem. If it isnt, the problem is often simply a physical connection problem between the phone and PBX. In IP telephony, things get a little more complicated. There are a number of components that all have to work together correctly. The connections between them are logical, not physical and transitory at that so troubleshooting problems becomes a lot more involved. You need to understand the interactions between the components in order to find the problem quickly. The components of an IP phone system can be divided into two groups: The phone components themselves and the network infrastructure components. The phone components are the IP phones themselves, the IP PBX, various gateways that provide the interface between the IP network and the telephone network (the PSTN), and other devices such as conference bridges, gatekeepers, voice mail servers, etc. The IP PBX is a server running a PBX application such as Ciscos Call Manager, Avayas Communications Manager, or Nortels Succession. Theres even an open source product called Asterisk. The network components include the infrastructure itself (the routers, switches, wiring, etc.) and also ancillary services that are important to IP telephony: DHCP servers and TFTP servers (Note: if you have more than a handful of PCs and are still using static IP addresses instead of DHCP, we need to talk!).

IP Telephony Troubleshooting

by Ronald Trunk rtrunk@netcraftsmen.net

Page 2

Understanding the Boot Process

To begin, you should be familiar with the boot process of the IP telephone. A failure to boot properly is often a result of some network-related problem. Understanding how a phone initializes will help you see what can go wrong and how to fix it. We will look at the three major vendors of IP phones: Cisco, Avaya and Nortel. At a high level, all three follow the same basic steps: 1. The phone requests an IP address from a DHCP server 2. The DHCP server gives the phone an IP address and additional information, such as the address of the TFTP server and/or IP PBX. 3. The phone connects to the TFTP server and receives configuration information and/or updated firmware. 4. The phone connects to the IP PBX and registers itself with it. 5. The IP PBX configures the phones operational features, namely the phone number, programmable buttons and more. When a phone has successfully registered with the IP PBX, the phones display will show the extension number and usually the time of day. If you see this on the phone you know a couple of things: 1. The phone received a valid IP address from the DHCP server 2. The phone was able to exchange data with the TFTP server 3. The phone was able to exchange data with IP PBX and the PBX was configured to recognize the phone. If a phone fails any of these steps you will not see the normal display. Instead, you will see various status or error messages indicating that one of the above things did not happen. To understand what went wrong, lets look at each of the vendors phone systems in a little more detail:

Cisco
When a Cisco phone is powered up, you will see the mute, headset and speakerphone indicators light up. Next, the phone and switch exchange Cisco Discovery Protocol (CDP) data. CDP is a Cisco proprietary protocol that lets interconnected Cisco devices learn about each other. The switch uses CDP to tell the phone which VLAN will be used for voice traffic (Note: if you are not using separate VLANs for voice and data traffic, we need to talk!). During this process the phone will display Configuring VLAN.

IP Telephony Troubleshooting

by Ronald Trunk rtrunk@netcraftsmen.net

Page 3

Cisco IP Telephone Boot Process

TFTP Server

3 1

IP Telephone
2

Call Manager (IP PBX)

DHCP Server
1 2 3 4

Cisco switch tells phone which VLAN to use via CDP protocol Phone requests IP address from DHCP server. Server offers address and TFTP server address Phone downloads configuration from TFTP server, including address of Call Manager(s) Phone registers with Call Manager

Once the phone knows which VLAN to use, it will tag all of its data frames with the appropriate 802.1q tag and request an address from the DHCP server. The phone will display Configuring IP. The DHCP server will offer an IP address as well as the address of the TFTP server. The phone now connects to the TFTP server and downloads a configuration file that lists the Cisco Call Manager (IP PBX) address as well as some other phone configuration info. Now the phone will attempt to register with the Call Manager. If the configuration file from the TFTP server listed more than one Call Manager, the phone will try each address until it can register with one. During this step, the phone will display Configuring CM list, then Opening a.b.c.d which is the address of the Call Manager. If the Call Manager accepts the phones registration request, the phone will display the extension and time. You should be able to pick up the phone and dial a number. At this point, it doesnt matter if the call goes through or not. Whether you hear a ringing tone, fast busy, or reorder tone, you know that the phone has successfully registered with the Call Manager and the two can communicate. If the call cannot be completed, it is probably a problem with the dial plan or codec selection.

IP Telephony Troubleshooting

by Ronald Trunk rtrunk@netcraftsmen.net

Page 4

Avaya
When an Avaya phone powers up, it shows a test pattern on the phones display and the hold lamp is lit. Not surprisingly, Avaya does not use the Cisco Discovery Protocol to determine the voice VLAN. Instead, the phone requests an IP address from the DHCP server on the native (data) VLAN. The DHCP server offers an address and also a string of configuration information that among other things, tells the phone which VLAN to use. The phone then reboots and begins tagging Ethernet frames with the proper VLAN information. The phone will display DHCP xxx

Avaya IP Telephone Boot Process

TFTP Server

IP Telephone
1

Communications Manager (IP PBX)

DHCP Server
1 2 3 4

Phone requests IP address from DHCP server (on native VLAN). Server offers address and VLAN information Phone reboots and requests IP address on voice VLAN. Server offers new address, TFTP server and C-LAN address Phone downloads new firmware from TFTP server (if available) Phone registers with Communication Manager (via C-LAN interface)

Next, the phone requests an address from the DHCP server, this time on the voice VLAN. The DHCP server responds with an IP address on the voice VLAN, and additional configuration information including the address of the TFTP server and the C-LAN (IP PBX interface) card. The phone will display DHCP Now the phone connects to the TFTP server, downloads configuration information and firmware upgrade (if needed). The phone will display TFTP server a.b.c.d [Verify]. Finally, the phone will connect to the C-LAN card and attempt to register with the Communication Manager (IP PBX application). If the DHCP server offered several C-LAN addresses, the phone will try them in order until registration is successful. During this process, the phone will display Discovering When the phone connects to the PBX, it will either display the extension and time, or it will display the prompt Ext=, meaning it is waiting to be told what extension to use. In either case, the phone has successfully completed the boot process. From a networking perspective, everything is working properly.

IP Telephony Troubleshooting

by Ronald Trunk rtrunk@netcraftsmen.net

Page 5

Nortel
Like Avaya, Nortel phones do not use the Cisco Discovery Protocol to determine the voice VLAN, but instead relies on configuration information supplied by the DHCP server. When the Nortel phone boots, it displays a test pattern and the speakerphone and message waiting lamps light. The phone requests an IP address (on the native VLAN) from the DHCP server. The server responds with an IP address and a VLAN ID. The phone reboots and requests a new IP address on the voice VLAN by tagging all Ethernet frames with the VLAN ID.

Nortel IP Telephone Boot Process

IP Telephone
1

Succession Call Server (IP PBX)

DHCP Server
1 2 3 4

Phone requests IP address from DHCP server (on native VLAN). Server offers address and VLAN information Phone reboots and requests IP address on voice VLAN. Server offers new address and call server address. Phone downloads new firmware from call server (if available) Phone registers with Call Server

The DHCP server responds with a new IP address and a list of call server (IP PBX) addresses. The phone displays Locating Server, connects to the call server, optionally downloads updated firmware and attempts to register. If successful, you will either see a normal phone display or the prompt Node: xx? Either one indicates that the phone can communicate with the call server. From a networking perspective, everything is working properly.

Troubleshooting the Boot Process

Now that you have a good understanding of the boot process, you can isolate problems and identify them more quickly. To begin with, all the phones have some rudimentary diagnostic display that tells you what parameters the phone is working with. On Cisco phones, press the [options?, network config?] keys. You can then scroll though a list of parameters such as IP address, DHCP server address, call manager address, etc. On Avaya phones, press <mute> 8439 # (8439 spells view on the phone keypad), and you can scroll through a similar list. On Nortel phones, press <Services> <Options> <SetInfo> to scroll through the list.

IP Telephony Troubleshooting

by Ronald Trunk rtrunk@netcraftsmen.net

Page 6

For instance, if a Cisco phone is stuck at the prompt Configuring VLAN, check that the auxiliary VLAN is configured correctly and that CDP is enabled on that port. By default it is, but it can be turned off on individual parts as well as globally. If the phone displays Configuring IP address, or an Avaya phone displays DHCP xxx (xxx is a counter if it increases much beyond 15, theres usually a problem), you need to troubleshoot the DHCP server. Since the phone is an IP device, you can troubleshoot the DHCP problems just like any other network device. Figure [x] shows a list of general troubleshooting steps. If the phone indicates a problem accessing the TFTP server, check to see if the server is reachable from the phone. Phones in general do not allow you to ping other devices. Instead, try pinging the phone from the server. Verify that the TFTP service is running, and that there are no file or directory permissions getting in the way. Use a TFTP client on a PC to verify that the files can be read or written. DHCP Troubleshooting Tips for IP Telephony Finally, if the phone indicates that it is trying to register with the IP PBX but cant, follow the same troubleshooting steps as you did for the TFTP server. Can you ping the phone from the PBX? Is the PBX running? Check the phones diagnostic to verify that it has the correct address for the PBX. Remember that from a networking perspective, if the PBX rejects the phones attempt to register, that is just as good as if it accepted it. It means that the phone and the PBX can communicate. The problem lies elsewhere, most likely in the PBX configuration. As you can see, by understanding some of the details how IP phones communicate with other components of the IP telephony system during boot up, troubleshooting the process becomes much simpler. Next time, Ill explain how calls between phones (and gateways) are set up so you can quickly troubleshoot problems with them as well.
6. 1. 2. 3. Is the DCHP service running? Can other devices such as PCs get addresses? Is there a DHCP scope set up (and activated) for the voice VLAN? Is the switch port that the phone is plugged into configured for trunking, and are the correct VLANs enabled? Are the DHCP options correct? Doublecheck the gateway address, mask, etc. Is the DHCP server reachable from the voice VLAN? If the DHCP server is on a different VLAN than the phone, do you have ip helper-address (or equivalent) configured on the router interface? Are there access lists that may be blocking DHCP packets? You can test this by configuring a switch port on the voice VLAN (without trunking) and connecting another device, such as a PC to it. You can also use a packet sniffer to monitor the DHCP exchange. If you are using Avaya phones, carefully check the syntax of the DHCP option 176. This option is an ASCII string that configures the voice VLAN, TFTP server, and C-LAN (IP PBX interface) addresses. The syntax is very unforgiving of typos a mistake anywhere in the string will affect all the parameters. Check that the phone received all the correct parameters from the DHCP server. Use the phones diagnostic display to verify.

4. 5.

7.

IP Telephony Troubleshooting

by Ronald Trunk rtrunk@netcraftsmen.net

Page 7

Troubleshooting IP Telephones I. Understanding the phone boot process a. Components: Phone, DHCP, TFTP, PBX b. Boot process i. Cisco ii. Avaya iii. Nortel II. Troubleshooting the process a. VLAN problems b. Addressing problems c. TFTP problems d. Registration problems III. Troubleshooting call problems a. Understanding call setup i. Phone to phone 1. cisco 2. Avaya/nortel ii. Phone to gateway 1. Cisco/Avaya/nortel b. Reorder tone or fast busy c. No or one-way audio Quality problems a. Garbled, choppy, distorted audio i. Symptom ii. Causes iii. Solution b. Echo i. Symptom ii. Causes iii. Solution c. Static, hissing, popping i. Symptom ii. Causes iii. Solution

IV.

Copyright 2005, Ron Trunk