CIS 188 CCNP TSHOOT (Troubleshooting)

Ch. 6 Troubleshooting Addressing Services

Rick Graziani Cabrillo College graziani@cabrillo.edu Fall 2010

Materials
Book: Troubleshooting and Maintaining Cisco IP Networks (TSHOOT) Foundation Learning Guide: Foundation learning for the CCNP TSHOOT 642-832 By Amir Ranjbar Book ISBN-10: 1-58705-876-6 ISBN-13: 978-1-58705-876-9 eBook ISBN-10: 1-58714-170-1 ISBN-13: 978-1-58714-170-6

NAT/PAT Operation

Network Address Translation (NAT) was designed for IP version 4 (IPv4) Used for: Address conservation Address hiding NAT usually operates at the border of a network Translates the source address of the exiting IP packets that are private addresses to public addresses before packets are forwarded.
3

NAT Example
1 2

DA 128.23.2.2

SA 10.0.0.3 IP Header .... Data

DA 128.23.2.2

SA 179.9.8.80 IP Header .... Data

The translation from Private source IP address to Public source IP address.

4

NAT Example
4 3

DA 10.0.0.3

SA 128.23.2.2 IP Header .... Data

DA 179.9.8.80

SA 128.23.2.2 IP Header .... Data

Translation back, from Public destination IP address to Private destination IP address.

PAT Example

Note: The router will try to maintain the original TCP/UDP source port number if possible

NAT/PAT table maintains translation of: DA, SA, SP, Converted SP

DA 128.23.2.2 SA 10.0.0.3 IP Header DP 80 SP 1331 Data DA SA DP 80 SP 3333 Data

128.23.2.2 179.9.8.80

1
DA 128.23.2.2

TCP/UDP Header DP 80 SP 1555 Data

2
DA

IP Header

TCP/UDP Header DP 80 SP 2222 Data

SA 10.0.0.2 IP Header

SA

128.23.2.2 179.9.8.80 IP Header

TCP/UDP Header

TCP/UDP Header

PAT Example

NAT/PAT table maintains translation of:

SA (DA), DA (SA), DP (SP)

DA 10.0.0.3 SA 128.23.2.2 IP Header DP 1331 SP 80 Data DA SA DP 3333 SP 80 Data

179.9.8.80 128.23.2.2

4
DA 10.0.0.2

TCP/UDP Header DP 1555 SP 80 Data

3
DA 179.9.8.80

IP Header

TCP/UDP Header DP 2222 SP 80 Data

SA 128.23.2.2 IP Header

SA 128.23.2.2 IP Header

TCP/UDP Header

TCP/UDP Header

 Static NAT: Local and global addresses are mapped one to one. Dynamic NAT: Local addresses are translated to a group or pool of global addresses. Still dealing with one-to-one translation once a global address has been selected. NAT overloading: Special type of dynamic NAT in which addresses are translated in a many-to-many fashion. Also known as PAT, or Port Address Translation

Dynamic NAT
SA 172.16.10.1

Example: We want NAT to allow the first 31 hosts from each subnet on the inside to communicate with devices on the outside using a public IP address. Note: We will use the 172.16.10.0/24 network to simulate a public address Range of public addresses: 172.16.10.1 through 172.16.10.63

Dynamic NAT (one-to-one)

interface ethernet 0 ip address 10.10.10.1 255.255.255.0 ip nat inside interface ethernet 1 ip address 10.10.20.1 255.255.255.0 ip nat inside interface serial 0 ip address 172.16.10.64 255.255.255.0 ip nat outside ip nat inside source list 7 pool no-overload !--- Indicates that any packets received on the inside interface that !--- are permitted by access-list 7 !--- will have the source address translated to an address out of the !--- NAT pool "no-overload". access-list 7 permit 10.10.10.0 0.0.0.31 access-list 7 permit 10.10.20.0 0.0.0.31 !--- Access-list 7 permits packets with source addresses ranging from !--- 10.10.10.0 through 10.10.10.31 and 10.10.20.0 through 10.10.20.31. ip nat pool no-overload 172.16.10.1 172.16.10.63 prefix 24 ! !--- Defines a NAT pool named no-overload with a range of addresses !--- 172.16.10.1 - 172.16.10.63

172.16.10.1 172.16.10.2 172.16.10.3 172.16.10.4

10

NAT Overloading (many-to-one)

interface ethernet 0 ip address 10.10.10.1 255.255.255.0 ip nat inside interface ethernet 1 ip address 10.10.20.1 255.255.255.0 ip nat inside interface serial 0 ip address 172.16.10.64 255.255.255.0 ip nat outside ip nat inside source list 7 pool ovrld overload ! !--- Indicates that any packets received on the inside interface that !--- are permitted by access-list 7 will have the source address !--- translated to an address out of the NAT pool named ovrld. !--- Translations will be overloaded which will allow multiple inside !--- devices to be translated to the same valid IP address. access-list 7 permit 10.10.10.0 0.0.0.31 access-list 7 permit 10.10.20.0 0.0.0.31 !--- Access-list 7 permits packets with source addresses ranging from !--- 10.10.10.0 through 10.10.10.31 and 10.10.20.0 through 10.10.20.31 ip nat pool ovrld 172.16.10.1 172.16.10.1 prefix 24 ! !--- Defines a NAT pool named ovrld with a range of a single IP !--- address, 172.16.10.1. 172.16.10.1

11

Static and NAT Overloading (many-to-one)

172.16.131.1

172.16.131.2

interface e 0 ip address 10.10.10.254 255.255.255.0 ip nat inside interface s 0 ip address 172.16.130.2 255.255.255.0 ip nat outside

ip nat inside source list 7 pool test overload ip nat inside source static 10.10.10.1 172.16.131.1 access-list 7 permit 10.10.10.0 0.0.0.255 ip nat pool test 172.16.131.2 172.16.131.2 netmask 255.255.255.0

12

 Other examples http://www.cisco.com/en/US/tech/tk648/tk361/technologies_tech_note09186 a0080094e77.shtml#topic1

13

Advantages and disadvantages of implementing NAT

14

 Some applications or protocols have direct conflict with Network Address Translation (NAT) or Port Address Translation (PAT). Example: IPsec Virtual Private Networks (VPN) IPsec protocols encapsulate the original IP packet The protocol type on the IP header changes (to ESP or AH) There is no TCP or UDP header next to the IP header. Means that there is no port number for NAT/PAT to translate. Some mechanisms have been invented to allow IPsec and NAT to coexist. Those mechanisms include: NAT Transparency or NAT traversal IPsec over TCP IPsec over UDP. In certain cases you may still be required to disable NAT for VPN traffic, or create exceptions for it.

15

Troubleshooting Common NAT/PAT Issues

Some of the important NAT issues and considerations to keep in mind are: Having a diagram for the NAT configuration ACLs are used to tell the NAT device: "what source IP addresses are to be translated IP NAT pools are used to specify: "to what those addresses translate Marking the IP NAT inside interfaces and the IP NAT outside interfaces correctly is very important NAT packets still have to obey routing protocols and reachability rules, so make sure that every router knows how to reach the desired destinations.
16

Troubleshooting NAT problems: clear ip nat translations show ip nat translations show ip nat statistics debug ip nat debug ip packet [access-list] debug condition interface interface show debug condition

17

Troubleshooting Example: NAT/PAT Problems

18

 R1 can ping R4 But router R1 cannot ping R3

19

Static Translation

Looks good Looks good

Static NAT in operation

20

 Typical issues with static translations occur when: there is no route back to the statically translated address when the statically selected global address overlaps with an available address in the dynamic address pool The next step is to verify whether packets leaving R1 actually reach R3 or not.
21

?
R1# ping 172.11.6.3
p

Ping R3 from R1 and observe the out of the debug on R3. (pings fail at R1) ICMP echo requests reach R3, but R3s ICMP echo replies do not reach R1. Conclude that the NAT translation is working but there is a routing issue on R3 toward the 172.16.6.0 destination.
22

Because there are no routing protocols in use, we can only fix the routing problem by entering a static route in R3s routing table

23

 Success!

24

Troubleshooting Example: NAT Problem Caused by an Inaccurate Access List

25

SSH ICMP

ICMP

SSH

We are told that administrators are unable to use Secure Shell (SSH) from the 10.10.10.0/24 network to routers R3 or R4 But they can accomplish connectivity (ping, etc) from the R1 loopbacks. The routing protocol used is single area OSPF Our mission is to restore end-to-end connectivity and make sure SSH is operational to support management processes.

26

 Is SSH traffic is being filtered somewhere? Or is this simply be a routing problem - traffic is either not reaching the destination, or it is not coming back? Not a routing problem
27

 Our follow-the-path strategy points to looking at the intermediate routers all the way up to the destination (R3). We will use a clever tool to discover the potential filter: ip tcp debug transactions Try the SSH connection again and observe the output of debug. The results indicate that the attempt made by R1 to setup a TCP session with R3 failed because the remote device (R3) responded with a TCP RST (reset).
28

 Focus on R3: the output of show ip int serial 0/1/0 shows that an access-list called FIREWALL-INBOUND is applied to serial 0/1/0 interface on the inbound direction. show access-lists command and the access list looks correct: statement number 30 permits TCP connection to 172.16.11.3 TCP port number 22 (SSH).

29

 Cautiously make use of debug ip packet on R3. Re-attempt the SSH session from R1 to R3 and observe the debug output on R3. The SSH attempt from R1 fails again, but: Security message (%SEC-6-IPACCESSLOGP) on R3 states that the denied TCP with: Source IP address 10.10.10.1 and source port number 29832 Destination IP address 172.16.11.3 and destination port number 2222 Now we know why the packet is denied, destination port is 2222 instead of 22.

30

 We have to find out which device has translated the port number from 22 to 2222. Prime suspect is NAT on R2. Cautiously, we use debug ip nat on R2 Re-attempt SSH from R1 to R3. To confirm our findings, we also enter the show ip nat translations command on R2.

31

 We are using TCP 2222 (destination port not the NAT source port); but the access-list on R3 is permitting TCP 22. We correct the FIREWALL-INBOUND on R3

32

We re-attempt SSH from R1 to R3. SSH attempt is successful now. The problem was not the NAT configuration. The configuration on R2 was doing port mapping to a custom port (2222) but the access-list configuration on R3 did not consider or account for the custom port.
33

Reviewing DHCP Operation

These commands are most helpful to know during the troubleshooting process, for example. DHCPDECLINE client-to-server message - The client through some other means has discovered that the IP address is already in use. DHCPNAK - DHCP server refuses the request for a certain configuration 34 parameter.

Three roles router takes in DHCP

The most common reasons for problems are configuration issues. This can result in a multitude of symptoms, such as: Clients not obtaining IP information from the server Client requests not reaching the server across a DHCP relay agent Clients failing to obtain DHCP options and extensions.

35

Router(config-if)# ip helper-address address Another issue related to DHCP Relay Agent is that enabling a router interface with the ip helper-address interface command Allows you to control which broadcast packets and which protocols are forwarded. Address: Destination broadcast or host address to be used when forwarding UDP broadcasts. Forwards six protocols (not just DHCP) TFTP (port 69) DNS (port 53) Time Service (port 37) NetBIOS Name Service and Datagram Service (ports 137 and 138) TACACS (port 49) DHCP/BOOTP Client and Server (ports 67 and 68) If other protocols do not require this service, forwarding their requests should be disabled using the no ip forward-protocol udp port-number global configuration mode command.

36

 DHCP options deliver parameters in addition to the traditional IP address, subnet mask, default-gateway, and DNS server address. 82 - Used for 911 purposes, in order to know the switch port that an IP phone connects to the 911 callers exact location.

37

Some of the Cisco IOS commands that can be helpful for DHCP troubleshooting are: show ip dhcp server show ip dhcp binding show ip dhcp conflict show ip dhcp database show ip dhcp pool show ip dhcp limit lease debug ip udp debug dhcp detail debug ip dhcp server [packet | event] clear ip dhcp binding clear ip dhcp conflict

38

ip dhcp excluded-address 172.16.1.100 172.16.1.103 ip dhcp excluded-address 172.16.2.100 172.16.2.103 ! ip dhcp pool 0 network 172.16.0.0 /16 domain-name cisco.com dns-server 172.16.1.102 172.16.2.102 netbios-name-server 172.16.1.103 172.16.2.103 netbios-node-type h-node ! ip dhcp pool 1 network 172.16.1.0 /24 default-router 172.16.1.100 172.16.1.101 lease 30 ! ip dhcp pool 2 network 172.16.2.0 /24 default-router 172.16.2.100 172.16.2.101 lease 30

Three DHCP address pools are created: network 172.16.0.0, subnet 172.16.1.0 and subnet 172.16.2.0 Attributes from network 172.16.0.0 - such as the domain name, DNS server, NetBIOS name server, and NetBIOS node type are inherited in subnetworks 172.16.1.0 and 172.16.2.0. In each pool, clients are granted 30-day leases and all addresses in each subnetwork, except the excluded addresses, are available to the DHCP Server for assigning to clients. Excluded addresses include default-gateway, DNS and NETBIOS servers

39

Troubleshooting Example: DHCP Problems

40

 R1 provides DHCP services to the clients in the 10.1.1.0 subnet. (Not R4) The DHCP clients in this example are routers R2 and R3. It is reported that R1 is no longer providing reliable DHCP services: The clients are unable to renew their IP addresses.

41

 Check R2 and R3 to make sure that they are configured as a DHCP clients. The output of the show ip interfaces brief command shows that interface fa0/0 is configured as a DHCP client and it shows an unassigned IP address. Because multiple clients are having the same problem, it is reasonable to suspect the problem originates elsewhere.

42

 debug dhcp detail command on R3: Finds that the DHCP DISCOVER messages are generated out of interface fa0/0, but no DHCP offers are received back from the DHCP server. Timed out Selecting state message, followed by No allocation possible.

43

 shows that one address pool configured as part of the service and very few DHCP messages sent and received. So it looks to be in good shape, just not very active.

44

 Most likely the IP address scope, known as the address pool in Cisco IOS DHCP, is either misconfigured or exhausted We check the DHCP address pool using the show ip dhcp pool command. We see that the number of leased addresses are 0, meaning that there are no leased addresses. We still have 254 addresses in this pool, but none are allocated at the moment.
45

 We use the show ip socket command to see the active ports on R1, the DHCP server. The show ip socket command is not frequently used by network administrators, but it is very handy in monitoring the open ports on a router We would need to see UDP 67 if the DHCP service was running. This is certainly a problem.

46

 We enable the DHCP service using the service dhcp command. After enabling this service you must wait a few seconds, because DHCP clients retry at different intervals. Use the show ip socket command again and now see port 67 as an active port on R1. Finally, we check routers R2 and R3 (the DHCP clients) and see that they successfully obtained IP addresses.

47

DHCP Troubleshooting Example: Duplicate Client IP Addresses

48

 The IP address of router R1 on the fast Ethernet interface was changed from 10.1.1.100 to 10.1.1.1 to comply with the new addressing scheme and policies of the network. This policy states that all branch routers will have the first IP address on any subnet that is being assigned to a network segment. After the change, some DHCP clients are reporting duplicated IP addresses. Clients state that this happens sporadically, a few times a week.

49

 One piece of information we have is that the IP address duplication happens sporadically, and one host at a time. Knowing that, perhaps our first order of business is to look at the lease times, and see if they match the frequency of the symptoms. The Cisco IOS DHCP default values is 1 day. The output of this command shows the vlan10 DHCP pool with a lease time of 3 days. The DHCP pool seems to be correct.
50

 show ip dhcp conflict command will tell us whether the DHCP server has found overlap or duplication in the IP addresses that it has assigned. One of the many conflicting addresses is 10.1.1.1, which is the new ip address of router R1 (the dhcp server itself) on interface fa0/0. However, we know that the DHCP server will not provide its own IP address to its clients!

51

 Many devices such as servers and printers are usually configured as DHCP clients and have static IP addresses. If their addresses are not excluded from the DHCP dynamic pool, there will definitely be conflict problems. We must check and verify which IP addresses are being excluded on R1, the DHCP server. We do that using the show running | include excluded command. The only IP address excluded from the DHCP dynamic pool is 10.1.1.100, which is R1s old address.

52

 We need to exclude the range of addresses that are meant to be statically assigned. This range is 10.1.1.1 to 10.1.1.20 To ensure that the users will receive unique addresses from the DHCP server and will not incur any more address conflicts, we must renew IP address leases on all DHCP clients, especially those that have experienced conflicts before. (Success!)
53

DHCP Troubleshooting Example: Relay Agent Issue

54

? ?

The DHCP clients in network segment 10.1.1.0 are unable to obtain IP address and other parameters from the central DHCP server. R2 is a DHCP client that is having trouble acquiring IP address R1 is the router that is supposed to act as a relay agent and forward DHCP messages between local clients and the DHCP server (R4). There are several possible causes: The clients could be misconfigured or faulty The relay agent could be not configured or misconfigured The server could be misconfigured or exhausted (or faulty/disabled) There could be network problems or filtering/security barriers.

55

 Because multiple clients are having the same problem, it is possible that they are all misconfigured, but that is not likely. It is simpler to check the relay agent. If we find no problem with the relay agent, we can then check the DHCP server.

56

 One of the quickest ways to verify DHCP relay agent operations is using the debug ip udp command. R1 is certainly receiving DHCP requests. The UDP/IP packets shown have a source address of 0.0.0.0, destination address of 255.255.255.255 with source UDP port of 68 (DHCP client) and destination UDP port of 67 (DHCP server). The problem could be that the fa 0/0 interface facing the DHCP client is missing the ip helper-address command pointing to 192.168.1.4.
57

R1(config)# interface fa0/0 R1(config-if)# ip helper-address 192.168.1.4

Checking the configuration reveals that this command is indeed missing, so adding this command is the first thing we have to do. debug ip udp command on R4, shows the DHCP Requests are being received and DHCP information is being returned. Finally, we verify the status of the DHCP clients, such as R2, in the 10.1.1.0 subnet, and see that they are acquiring IP address and other parameters from the DHCP server.

58

CIS 188 CCNP TSHOOT (Troubleshooting)

Ch. 6 Troubleshooting Addressing Services

Rick Graziani Cabrillo College graziani@cabrillo.edu