FWSM: Troubleshoot Traffic Failures Due to Wrong Xlates

Document ID: 115010 Contributed by Michael Robertson and Jay Johnston, Cisco TAC Engineers. Nov 02, 2012

Contents
Introduction Prerequisites Requirements Components Used Conventions Symptoms Logical Topology Relevant Configuration Observed Behaviors Triggers Solutions Resolve Incorrect Routing Configurations Disable samesecuritytraffic permit intrainterface Drop Packets that Arrive on an Incorrect Interface (ACLs or uRPF) Enable xlatebypass Summary Related Information

Introduction
Due to the design of the Firewall Services Module's (FWSM) packet processing, xlates built by incorrectly routed packets can cause traffic failures for connections through the firewall. In order to select an egress interface for an inbound packet, the FWSM first checks to see if the destination IP of the inbound packet matches any existing global IP/Network in a NAT translation (xlate) for that interface in its xlate table. If a match is found, the egress interface is simply chosen based on the local interface in the xlate entry and the firewall does not consult the routing table to make the egress interface decision. The default behavior of the FWSM is to build an xlate entry for the source IP of any permitted packet that is received on one of its interfaces. If a packet is routed through the network incorrectly (for any number of reasons) and arrives inbound on the wrong interface of the FWSM, an xlate is built to reflect this. When this occurs, entries in the xlate table can override entries in the routing table and cause traffic failures for the affected destinations. This document describes the symptoms and triggers for this issue, how to diagnose it, and provides solutions for preventing it from occurring.

Prerequisites

Requirements
Cisco recommends that you have knowledge of FWSMs.

Components Used
This document is not restricted to specific software and hardware versions.

Conventions
Refer to Cisco Technical Tips Conventions for more information on document conventions.

Symptoms
Logical Topology

Relevant Configuration
interface Vlan1 nameif outside securitylevel 0 ip address 192.168.100.50 255.255.255.0 ! interface Vlan10 nameif inside securitylevel 100 ip address 10.10.1.50 255.255.255.0 ! interface Vlan20 nameif dmz

securitylevel 50 ip address 10.20.1.50 255.255.255.0 ! samesecuritytraffic permit intrainterface accesslist outside_in extended permit tcp any host 10.30.1.1 eq www accesslist inside_in extended permit ip any any accessgroup inside_in in interface inside accessgroup outside_in in interface outside route outside 0.0.0.0 0.0.0.0 192.168.100.254 route dmz 10.30.1.0 255.255.255.0 10.20.1.254

Observed Behaviors
Connections from the client PC at 172.16.1.10 to the web server at 10.30.1.1 fail. A packet capture on the outside interface shows a TCP SYN from the client PC arriving at the FWSM's interface.
FWSM# show capture outside 3 packets seen, 3 packets captured 1: 13:58:09.280752960 802.1Q vlan#1 P0 172.16.1.10.57389 > 10.30.1.1.80: S 918518428:918518428(0) win 8192 <mss 1380,nop,nop,sackOK> 2: 13:58:12.280755950 802.1Q vlan#1 P0 172.16.1.10.57389 > 10.30.1.1.80: S 918518428:918518428(0) win 8192 <mss 1380,nop,nop,sackOK> 3: 13:58:18.280761960 802.1Q vlan#1 P0 172.16.1.10.57389 > 10.30.1.1.80: S 918518428:918518428(0) win 8192 <mss 1380,nop,nop,sackOK> 3 packets shown

A packet capture on the dmz interface does not show that packet leaving the firewall.
FWSM# show capture dmz 0 packet seen, 0 packet captured 0 packet shown

No entry is built in the FWSM's connection table and syslogs do not show any information related to the client or server IP addresses.

Triggers
At a fundamental level, this issue is caused by an entry in the FWSM's xlate table that was built by an incorrectly routed packet. Because of the way the FWSM's packet processing is designed, the firewall checks the xlate table before it checks the routing table to determine the egress interface. As a result, if a packet matches an existing xlate the egress interface will be selected based on that entry, even if the entry conflicts with what is listed in the routing table. In other words, the xlate table takes precedence over the routing table. In order to diagnose this issue, check the output of the show xlate debug command:

FWSM# show xlate debug Flags: D DNS, d dump, I identity, i inside, n no random, o outside, r portmap, s static 3 in use, 3 most used NAT from inside:10.30.1.1 to outside:10.30.1.1 flags Ii idle 0:00:00 timeout 3:00:00 connec NAT from inside:10.30.1.1 to inside:10.30.1.1 flags Ii idle 0:00:07 timeout 3:00:00 connect

NAT from dmz:10.30.1.1 to outside:10.30.1.1 flags Ii idle 0:00:10 timeout 3:00:00 connectio

Note: The debug keyword in the show xlate is crucial. Without it, the xlate entries will not include the interface names that the entry is associated with.

The xlate table shows that there are 3 xlates built for the web server. The first xlate is built between the inside interface and the outside interface. The second xlate is built as a hairpinned or uturned xlate on the inside interface. The third xlate is built between the dmz and the outside interface. The I flag indicates that this is an identity xlate and the IP is not actually being translated. The first interface listed in the entry is the "real" or "local" interface where the IP is supposed to actually exist. The second interface listed is the "mapped" or "global" interface where the IP is being translated. Neither of these xlates shown are correct. This is because the web server (10.30.1.1) actually exists behind the dmz interface. The third xlate is correct for this network design. The connection failure occurs because of the first xlate listed in the table. When the client's TCP SYN packet arrives on the outside interface destined to 10.30.1.1, the FWSM checks the xlate table and matches the first entry. This entry indicates that the packet should egress on the inside interface, which is incorrect, and the packet is blackholed. By default, the FWSM will automatically build an identity xlate for any traffic that does not match an explicitly configured NAT rule. Because of this, even if a packet erroneously arrives on an incorrect interface, an xlate will be built. Specifically for this case, packets sourced from 10.30.1.1 arrived inbound on the inside interface instead of arriving on the dmz interface as is expected. The first xlate (inside > outside) was built when the web server tried to ping a nonexistent IP address (10.199.199.1). The echo request left the web server destined to its default gateway (the DMZ router). The DMZ router forwarded the packet toward the inside router, per its static route:
S 10.0.0.0/8 [1/0] via 10.50.1.254

Because the 10.199.199.0/24 network does not actually exist anywhere, the inside router simply follows its default route and sends the packet to the FWSM's inside interface:
S* 0.0.0.0/0 [1/0] via 10.20.1.50

Likewise, the FWSM also does not have a route for the destination network. Therefore, it selects the outside interface as the egress interface and builds an identity xlate from inside > outside:
S 0.0.0.0 0.0.0.0 [1/0] via 192.168.100.254, outside

The second xlate (inside > inside) was built when the web server tried to access the DNS server while the inside router's 10.40.1.254 interface was temporarily down due to a link flap. The DNS request left the web server destined to its default gateway (the DMZ router). The DMZ router forwarded the packet toward the inside router, per its static route:
S 10.0.0.0/8 [1/0] via 10.50.1.254

However, the inside router's interface connected to the 10.40.1.0/24 network was temporarily down and its directly connected route for this network was missing. Therefore, the only matching route in the routing table was the default route back toward the FWSM:
S* 0.0.0.0/0 [1/0] via 10.20.1.50

The packet was routed to the FWSM's inside interface. The FWSM's routing table indicated that the destination network of 10.40.1.0/24 existed behind the same inside interface:
S 10.40.1.0 255.255.255.0 [1/0] via 10.10.1.254, inside

Because the samesecuritytraffic permit intrainterface command is enabled, the FWSM will allow the uturned xlate to be built. To summarize, the first xlate was triggered by: A broad 10.0.0.0/8 route configured on the DMZ router A permit ip any any ACL configured on the FWSM's inside interface The second xlate was triggered by: A flapping interface on the Inside router samesecuritytraffic permit intrainterface configured on the FWSM

Solutions
There are many different possible solutions to this problem. First and foremost, deleting the xlate from the table should allow traffic to start working again until the xlate is rebuilt. This can be done with the clear xlate command. For example:
FWSM# clear xlate interface inside local 10.30.1.1 global 10.30.1.1

Note: Any connections that are using the deleted xlate(s) will also be torn down. Once that is complete, the focus should be on preventing the xlates from returning. Often times, the most preferred way to do this is to fix the routing configuration in the environment to prevent traffic from arriving on the wrong FWSM interface. The FWSM also offers a handful of configuration options to address these issues.

Resolve Incorrect Routing Configurations

This solution takes careful planning and a deep understanding of the network environment. In the first example above, the 10.0.0.0/8 route on the DMZ router is technically incorrect since the entire /8 network does not exist beyond its 10.50.1.253 interface. Instead, some options that exist are: Eliminate the 10.50.1.0/24 network all together and simply route all traffic through the FWSM. This also provides better segmentation and security between the Inside and DMZ networks. Configure a static route on the DMZ for only 10.40.1.0/24 and remove the 10.0.0.0/8 route. Use a dynamic routing protocol between the Inside and DMZ routers to correctly advertise only the networks that actually exist. There are often many possibilities for adjusting the routing configuration, but the end goal is to ensure that traffic from a given host is able to arrive only on a single FWSM interface.

Disable samesecuritytraffic permit intrainterface

The samesecuritytraffic permit intrainterface command allows the FWSM to uturn or hairpin traffic on an interface. This means that a packet can enter the firewall on the same interface it leaves on. This functionality is disabled by default and has very little use in most FWSM designs. Because the FWSM uses VLAN interfaces, traffic that stays within the same VLAN should never be processed by the FWSM. In the second example above, the samesecuritytraffic permit intrainterface command allowed a packet to both enter and leave the inside interface. Disabling samesecuritytraffic permit intrainterface would prevent this behavior and drop the packet before an xlate was ever built:

FWSM(config)# no samesecuritytraffic permit intrainterface

Drop Packets that Arrive on an Incorrect Interface (ACLs or uRPF)

In both examples above, the xlates were built when a packet from the web server incorrectly arrived on the inside interface. In order to prevent the problem all together, the FWSM can be configured to drop packets that arrive on the wrong interface. The FWSM requires that all traffic be permitted by an ACL before it can pass. Therefore, this functionality can be achieved by only permitting traffic from appropriate source networks on each interface. In the examples above, the inside interface permits all IP traffic:
accesslist inside_in extended permit ip any any

Instead, this should be changed to only permit traffic from the 10.10.1.0/24 and 10.40.1.0/24 subnets:
accesslist inside_in extended permit ip 10.10.1.0 255.255.255.0 any accesslist inside_in extended permit ip 10.40.1.0 255.255.255.0 any

In some environments, this is not a feasible option due to the size and/or scale of the different networks passing through the FWSM. However, this functionality can be achieved more simply using a feature called Unicast Reverse Path Forwarding (uRPF). When the uRPF feature is enabled, the FWSM will compare the source IP address of the first packet of every connection against its routing table. If the route that is found does not match up with the interface that the packet arrived on, that packet will be dropped due to a RPF failure. In the example above, the FWSM has a static route that uses the dmz interface to reach the 10.30.1.0/24 network. Therefore, if uRPF is enabled on the inside interface, packets sourced from the web server (10.30.1.1) that arrive incorrectly on the inside interface will be dropped. In order to enable uRPF, apply the ip verify reversepath command to each interface in question. For example:
FWSM(config)# ip verify reversepath interface inside

Enable xlatebypass
In both of the examples above, the xlates are created with the Ii flags. These flags indicate that the xlate is an identity translation (I) that originated on a high security (i) interface. By default, the FWSM will build these xlates for any traffic that does not match an explicit NAT/PAT rule. In order to disable this behavior, the xlatebypass command can be enabled in FWSM 3.2(1) and later:
FWSM(config)# xlatebypass

This feature will prevent the FWSM from building identity xlates in the first place. Thus, the traffic in the examples above would not be redirected to an incorrect interface due to an xlate table entry. However, the traffic will still pass through the FWSM untranslated.

Summary
In order to determine the egress interface for a packet, the FWSM will always consult its xlate table before looking at its routing table. If that packet matches an existing xlate, the egress interface is selected based on

the xlate's associated interface. This happens regardless of any contradictions that might be found in the routing table. In this way, the xlate table takes precedence over the routing table. Because the FWSM will always build an xlate entry for all new connections by default, this can cause traffic failures in cases where incorrectly routed packets cause the FWSM to build an xlate. As outlined above, there are many possible scenarios where this can occur but all relate back to a packet being received on an incorrect interface. This document covered these possible issues: A broad routing config sends packets in an incorrect direction The FWSM is configured to permit traffic from incorrect source networks The FWSM is configured to hairpin/uturn traffic In order to quickly restore connectivity for connections that fail due to a wrong xlate, delete the entry with the clear xlate command. This document also covered multiple solutions for preventing these xlates from returning in the future, including: Resolve incorrect routing configurations using more specific routes Disable samesecuritytraffic permit intrainterface Drop packets that arrive on an incorrect interface using ACLs or uRPF Enable xlatebypass

Related Information
Command Reference: ip verify reversepath Command Reference: xlatebypass Technical Support & Documentation Cisco Systems

Contacts & Feedback | Help | Site Map 2012 2013 Cisco Systems, Inc. All rights reserved. Terms & Conditions | Privacy Statement | Cookie Policy | Trademarks of Cisco Systems, Inc.

Updated: Nov 02, 2012

Document ID: 115010