Brkapp 3003

2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public BRKAPP-3003

1
Advanced Troubleshooting
the Cisco Application
Control Engine
BRKAPP-3003
2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003
2
Core Message
Understanding the architecture and flow management
will help troubleshoot the Application Control Engine
3
Session Objective
ACE Architecture
Understand the ACE architecture and connectivity through ACE
Verify software images, licenses and image recovery
Use the real-time TCP-DUMP command
Understand access list and ACL merge on ACE
Flow Management
Understand the difference between L4 and L7 processing
Check for possible asymmetric flows
Provide layer 7 troubleshooting
Ability to monitor performance and troubleshoot resources
Understand high availability
At the End of the Session, You Will Be Able To:
4
Session Agenda
ACE Architecture
Discuss the Architecture
Functions of control plane and data plane
Common debugging commands
Packet Capturing and logging
Traffic Forwarding on ACE
Admin Context and ACL Merge
Flow Management
Connection Handling on ACE
Layer 4/7 Troubleshooting and Performance
Health Monitoring on ACE
High Availability on ACE
55
ACE Architecture
6
ACE20 Module Hardware Architecture
Switch
Fabric
Interface
16G
Daughter
Card 1
Daughter
Card 2
8G
8G
SSL
Crypto
10G
2G
Console
port
Sup
Connect
100M
Control
Plane
Network
Processor 1
Network
Processor 2
10G 10G
Classification
Distribution
Engine
(CDE)
7
ACE30 Module Hardware Architecture
Switch
Fabric
Interface
16G
2G
Console
port
Sup
Connect
100M
Control
Plane
8G
Daughter Card 1
NP1 NP2
8G
Daughter Card 2
NP1 NP2
Classification
Distribution
Engine
(CDE)
8
2x 700MHz MIPS
1 GB Memory
Control Plane Software
Supervisor
Connection
DBUS
16 Gbps
Bus
RBUS
EOBC
Cisco
ASIC
100 Mbps
8 Gbps
8 Gbps
1 Gbps
ACSW OS
60Gbps switching Capacity
IPv4, IPv6 Classifications
TCP Checksum
Generation/Verification
Variable Load Distribution
Daughter Card 1
16 Gbps
CEF720 Linecard
20 Gbps
20 Gbps
Switch Fabric
ACE30 Detailed Hardware Architecture
CPU
Classification Distribution
Engine (CDE)
Network
Processor
1
Verni FPGA
DRAM
4 GB
DRAM
4 GB
Network
Processor
2
shared memory
Daughter Card 2
Network
Processor
3
Verni FPGA
DRAM
4 GB
DRAM
4 GB Network
Processor
4
shared memory
Cavium Octeon CN5860 (OcteonPlus)
16 core, 600 MHz CPUs with 4G DRAM 32k iCache, 16k dCache, 2MB L2 cache
On chip support for Encryption/Decryption Coprocessors for Compression/Decompression
9
Data Traffic vs. Management Traffic
ACE30 Control plane architecture is very similar to
ACE20
Device control
Configuration manager (CLI, XML API, SSH, )
Server health monitoring (native probes, TCL scripts)
Syslog's, SNMP,
ARP, DHCP relay
High-Availability
ACL Compilation
ACE30 data plane architecture is
very similar to ACE 4710
Connection management
TCP termination
Access lists
NAT
SSL Offload
Regular expression matching
Load Balancing & forwarding
10 10
Common Debugging
11
Common Debugging
12
Common Debugging
Show commands on the Catalyst 6500 Supervisor
show version
show clock
show module
show power
show asic slot <n>
show interface TenGigabitEthernet <n>/1
show interface TenGigabitEthernet <n>/1 trunk
show svclc vlan-group
[no] power enable <module>
show svclc module <n> traffic
Make sure the module status is OK
VLANs used by ACE must be
configured in the MSFC
13
Common Debugging
Show commands available on ACE
show version
show cde health
show ft group status
show ip int br
show int vlan <n>
show arp
show service-policy
show serverfarm
show rserver
show probe
show conn
show stat
show ip traffic
show resource usage
show np 1 me-stats -s norm
show np 1 me-stats -s norm M1
System Information
L2, L3
Performance,
Resources
Debugging
Flows
L4, L7
This provides the DELTA
If incorrect version, check bootparameter
14
Show Module from the
Catalyst 6500 Supervisor
cat6k#show mod
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 1 Application Control Engine 10G Module ACE20-MOD-K9 SAD12345678
2 48 48 port 10/100 mb RJ45 WS-X6348-RJ-45 SAD04450L44
5 2 Supervisor Engine 720 (Active) WS-SUP720-3BXL SAD08300D5L
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 0001.0002.0003 to 0001.0002.000a 2.4 8.7(0.22)ACE A2(2.3a) Ok
2 00d0.d32e.1b42 to 00d0.d32e.1b71 1.5 5.4(2) 8.5(0.46)RFW Ok
5 000f.f7be.b17c to 000f.f7be.b17f 4.0 8.1(3) 12.2(PP_R31_ Ok
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
5 Policy Feature Card 3 WS-F6K-PFC3BXL SAD083006N2 1.3 Ok
5 MSFC3 Daughterboard WS-SUP720 SAD082905VE 2.1 Ok
Mod Online Diag Status
---- -------------------
1 Pass
2 Pass
5 Pass
Module status shows OK
15
Verifying Version and Licenses
ACE/Admin# show version
Cisco Application Control Software (ACSW)
<snip>
Software
loader: Version 12.2[121]
system: Version A2(2.3a) [build 3.0(0)A2(2.3a)
system image file: [LCP] disk0:c6ace-t1k9-mz.A2_2_3a.bin
installed license: ACE-08G-LIC ACE-VIRT-250 ACE-SSL-15K-K9
Hardware
Cisco ACE (slot: 1)
cpu info:
number of cpu(s): 2
cpu type: SiByte
cpu: 0, model: SiByte SB1 V0.2, speed: 700 MHz
cpu: 1, model: SiByte SB1 V0.2, speed: 700 MHz
Installed Licenses
16
Available System Memory and Uptime
ACE/Admin# show version Continuation of output
[...]
memory info:
total: 827128 kB, free: 335372 kB
shared: 0 kB, buffers: 3540 kB, cached 0 kB
cf info:
filesystem: /dev/cf
total: 1014624 kB, used: 529472 kB, available: 485152 kB
last boot reason: NP 2 Failed: NP ME Hung
configuration register: 0x1
ACE kernel uptime is 7 days 23 hours 42 minute(s) 25
second(s)
Displays ACE module uptime
Useful information in
case of system reload
17
ACE File System
Use the dir command to view directory listing for files
ACE/Admin# dir ?
core: Directory or filename
disk0: Directory or filename
image: Directory or filename
probe: Directory or filename
volatile: Directory or filename
The internal File system is mapped as below
/mnt/cf - Image:
Also the following compressed file systems are used
/TN-HOME = disk0:
/TN-CONFIG = Startup config
/TN-LOGFILE = Internal Storage for audit logs
/TN-CERTKEY-STORAGE : internal storage for Cert and Keys
/TN-COREFILE = core:
18
ACE File System
Load debug plug-in to access ACE file system
Startup configuration located at /mnt/cf/TN-CONFIG
ACE will generate / fix any missing or corrupted file
systems during boot
When to use the format command?
If you receive the following error
Warning!! This will erase everything in the compact flash including
startup configs for all the contexts and reboot the system!!
ACE/Admin# write memory
ERROR!config filesystem is not mounted on compact flash
19
Working with Core Files
If ACE creates a core file you can locate the files in the
core directory
All cores files are stored in dir core: (core names are self
explanatory)
ACE/Admin# dir core:
99756 Apr 5 17:57:05 2007 ixp2_crash.txt
13047 Apr 5 17:56:59 2007 loadBalance_core_log.tar.g
Ixpx_crash.txt will have some details on the core dump
If it is a kernel crash , then a file named crash info will be
available in core
Show version will show last reload reason
20 20
System Logging
21
Logging Features
Each virtual context generates logs independently and sends to
specified destinations
Syslog server, console, buffer, SNMP station, etc..
Rate limiting of syslog messages is recommended. Never log to the
console using level 7
ACE can log connection setup/teardown at the connection speed
Access-List deny entries are logged
Use the terminal monitor command to display log message when not
using console
Useful commands to troubleshoot syslog issues:
show logging statistics show logging history
show logging queue
Make sure logging queue size is set properly
22
Basic Configuration to Enable Logging
Enable logging on the ACE
logging enable
logging timestamp
logging monitor 4
logging trap 4
logging buffer 4
logging history 4
logging queue 1024
no logging message 111008
It is recommended to disable or change the severity level
of some syslog messages. Use logging message
syslog_id [level severity_level] command
To enable the logging of connection setup and teardown
messages, use the logging fastpath command. Use the
logging rate-limit to limit the rate at which the ACE
generates messages to the syslog server
23 23
Real-Time TCP Dump
24
Real-Time TCP Dump
Supportability and analysis of load balanced traffic is a
major requirement in today's load balanced environment
ACE can capture real-time packet information for the
network traffic that passes through it
The attributes of the packet capture are defined by
an ACL
The ACE buffers the captured packets, and you can copy
the buffered contents to a file in flash memory on the ACE
or to a remote server
User can also display the captured packet information on
your console or terminal; capture can also be exported
and viewed using Ethereal or Wireshark
25
Real-Time TCP Dump
To enable the packet capture on ACE use the capture
command
capture c1 interface vlan 211 access-list FILTER bufsize 64
Buffer in Kbytes
(can be circular)
Pre-defined ACL to
identify relevant traffic
Interface to apply
capture
One capture session per context
Capture triggered at flow setup
Capture configured on client interface where flow is
received
26
Real-Time TCP Dump
ACE can capture traffic based on a configured access-list
and interface
Follow the following procedure to capture traffic on ACE:
1. Specify an ACL
2. Capture on an interface or globally
access-list FILTER line 10 extended permit tcp any any eq www
capture c1 interface vlan 211 access-list FILTER
Show capture status show status and buffer size
ACE/Admin# show capture c1 status
Capture session : c1
Buffer size : 64 K
Circular : no
Buffer usage : 1.00%
Status : stopped
27
Real-Time TCP Dump
Start the capture on the ACE
ACE/Admin# capture c1 start
23:40:37.236868 0:12:43:dc:93:bb 0:0:c:7:ac:a 0800 58:
172.16.11.190.443 > 209.165.201.11.1180: S
1389739009:1389739009(0) ack 617249474 win 17408 <mss 1460>
(ttl 255, id 2401, len 44, bad cksum 0!)
23:40:37.239102 0:12:43:dc:93:bb 0:0:c:7:ac:a 0800 54:
172.16.11.190.443 > 209.165.201.11.1180: . ack 71 win 17408
(ttl 255, id 2402, len 40, bad cksum 0!)
ACE/Admin# capture c1 stop
To copy the packet capture to disk0: use the copy capture
ACE/Admin# copy capture c1 disk0: c1
Maximum buffer size is 5MB of data
28 28
Traffic Forwarding on ACE
29
ACE Load Balancer Policy
Lookup Order
There can be many features applied on a given interface,
so feature lookup ordering is important
The feature lookup order followed by
data path in ACE is as follows:
1. Access-control (permit or deny a
packet)
2. Management traffic
3. TCP normalization/connection
parameters
4. Server load balancing
5. Fix-ups/application inspection
6. Source NAT
7. Destination NAT
The policy lookup order is implicit, irrespective of the order
in which the user configures policies on the interface
30
ACE in Routed Mode
IP subnets cannot overlap within a context but can across
two contexts
Non-Load balanced traffic is routed. ACE needs to ARP
for destination before forwarding packet
Client MAC ACE MAC
Client IP VIP
Random Port VIP Port
ACE MAC Selected
Server MAC
Client IP Server IP
Random Port Server Port
31
ACE in Bridge Mode
Non-Load balanced connections are bridged from client to
server vlan
Client MAC ACE MAC
Client IP VIP
Random Port VIP Port
Client MAC Selected
Server MAC
Client IP Server IP
Random Port Server Port
32
Checking VLAN Configuration
Show interface provides you with valuable information
ACE/Admin# show interface vlan 211
vlan210 is up
Hardware type is VLAN
MAC address is 00:16:36:fc:b3:36
Virtual MAC address is 00:0b:fc:fe:1b:02
Mode : routed
IP address is 172.16.10.21 netmask is 255.255.255.0
FT status is active
Description:WAN Side
MTU: 1500 bytes
Last cleared: never
Alias IP address is 172.16.10.23 netmask is 255.255.255.0
Peer IP address is 172.16.10.22 Peer IP netmask is 255.255.255.0
Assigned on the physical port, up on the physical port
499707 unicast packets input, 155702918 bytes
1485258 multicast, 5407 broadcast
0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops
497610 unicast packets output, 46804782 bytes
6 multicast, 8201 broadcast
0 output errors, 0 ignored
33
MAC Addresses
Virtual MAC (VMAC) is used for the alias IP, VIP
address
Alias IP and Virtual IP (VIP) are associated with a
VMAC only if high availability is configured
Active context responds to ARPs for alias IP with
VMAC
One unique VMAC per FT Group 00:0b:fc:fe:1b:XX
(XX=FT group number in hex)
Packets destined to the VMAC are blocked on
standby context
34
MAC Addresses
The VMAC is a function of ft-group-id. Therefore
different cards must have different ft-group-ids
Use the show interface internal iftable to locate the
VMAC
Each ACE supports 1,024 shared VLANs, and uses
only one bank of MAC addresses randomly
selected at boot time
ACEs may select the same address bank so avoid
this conflict use the shared-vlan-hostid command
35
Key Things to Know About ARP on ACE
For unicast packets, if the destination MAC is unknown
ACE will drop the packet, instead of flooding it
So IP-address-to-MAC mapping and outgoing interface
needs to happen first
ARP entries are populated as follows:
With ARP requests
Learning through incoming ARP requests
Gratuitous ARP packets
Layer 2 mode:
ARP is the only way to learn IP to MAC and interface mapping
36 36
Admin Context
Resource Reservation
37
Admin Context Resource Reservation
If Admin context is not configured correctly, Admin could
be starved of all resources
When configuring resource allocations in ACE, it is possible to
allocate 100% of resources to non-Admin contexts, so that the
Admin context is no longer reachable via ICMP, telnet, SNMP, etc
In some cases, this could cause FT between a pair of HA
ACE modules to fail, and create an active/active situation
Highly recommended to put some safeguard in place to
ensure that the Admin context always receives at least a
small percentage of resources
38
Shows starved resources and drops for throughput
ACE/Admin# show resource usage context Admin
Allocation
Resource Current Peak Min Max Denied
-------------------------------------------------------------------------------
Context: Admin
conc-connections 9 9 0 0 0
mgmt-connections 2 12 0 0 0
proxy-connections 0 0 0 0 0
xlates 0 0 0 0 0
bandwidth 0 4715 0 0 3704068
throughput 0 4247 0 0 3704068
mgmt-traffic rate 0 468 0 125000000 0
connection rate 0 7 0 0 8
ssl-connections rate 0 0 0 0 0
mac-miss rate 0 1 0 0 0
inspect-conn rate 0 0 0 0 0
acl-memory 26816 26880 0 0 0
sticky 0 0 0 0 0
regexp 0 0 0 0 0
syslog buffer 1024 4096 0 1024 0
syslog rate 0 7 0 0 118
No resources reserved
39
Shows heartbeats missed increasing. Heartbeats are
not reaching the peer. Possibility for both ACEs to go
Active/Active
ACE/Admin# sh ft stats
HA Heartbeat Statistics
------------------------
Number of Heartbeats Sent : 1095573
Number of Heartbeats Received : 1092586
Number of Heartbeats Missed : 2987
Number of Unidirectional HB's Received : 2640
Number of HB Timeout Mismatches : 0
Num of Peer Up Events Sent : 1
Num of Peer Down Events Sent : 1
Successive HB's miss Intervals counter : 0
Successive Uni HB's recv counter : 0
40
Below shows the problem why ACE is starved of all
resources
resource-class admin
limit-resource all minimum 0.10 maximum equal-to-min
Suggest the following reserved resources for Admin
resource-class Admin
limit-resource conc-connections min 5.00 max equal-to-min
limit-resource mgmt-connections min 5.00 max equal-to-min
limit-resource rate bandwidth min 5.00 max equal-to-min
limit-resource rate ssl-connections min 5.00 max equal-to-min
limit-resource rate mgmt-traffic min 5.00 max equal-to-min
limit-resource rate conc-connections min 5.00 max equal-to-min
41 41
Access-Control Lists
and ACL Merge
42
ACL Merge Process and Enhancements
New ACL merge enhancements added to ACE
ACL merge is responsible for merging all the features and
generating a single merged list for an given interface. ACL
compiler is responsible for programming the merged list into
MTrie data structure Fast retrieval of data
ACL memory usage has been optimized to better support
incremental changes
The new implementation provides a consistent ACL memory
usage during system bootup time and during incremental
changes after the system comes up
This feature also provides an early detection of failure if the
configuration needs more ACL resources than available
Also, note ACL masks are in 255.255.x.x format (not 0.0.y.y)
43
View Total Action Nodes
Use the show np 1 access-list resource to view action
nodes
ACE/Admin# show np 1 access-list resource
ACL Tree Statistics for Context ID: 3
=======================================
ACL memory max-limit: None
ACL memory guarantee: 0.00 %
MTrie nodes(used/guaranteed/max-limit):
6 / 0 / 262143 (compressed)
2 / 0 / 21999 (uncompressed)
Leaf Head nodes (used/guaranteed/max-limit):
3 / 0 / 262143
Leaf Parameter nodes (used/guaranteed/max-limit):
7 / 0 / 524288
Policy action nodes used: 4
memory consumed: 4696 bytes resource-limited 128 bytes other
4824 bytes total.
min-guarantee: 0 bytes total.
max-limit: 78610432 bytes total, 0 % consumed
44 44
Connection Handling in ACE
45
Flow Management
Level of Flow Processing Type of Processing Feature of Function
Layer 3 and Layer 4 Balance on first packet Basic Load Balancing
Applies to TCP/UDP for layer 4 rules Source IP Sticky
Applies to all other IP protocols TCP/IP Normalization
Layer 7 TCP Splicing Terminate TCP Connection HTTP Layer 7 rules based on first
request (URL LB)
Buffer request, inspect, LB Cookie Sticky (Persistence)
Create Hardware Shortcut Generic TCP Payload Parsing
Layer 7 Re-proxy TCP Splicing + ability to parse
subsequent HTTP requests within
the same TCP
HTTP Layer 7 rules with HTTP
1.1 connections keepalive
(persistence rebalance)
Layer 7 Full-Proxy Fully terminate clients connection SSL Offload
TCP re-use
HTTP 1.1 Pipelining
Protocol Inspection (FTP,SIP)
46
Internal Mapping of TCP/UDP Flows
TCP and UDP Flows = 2 X Internal Half Flows
ACE/Admin# show conn
conn-id np dir proto vlan source destination stat
-------------+--+----+--------+-----+--------------------------+-------------------------------+---------+
9 1 In TCP 211 209.165.201.11:1867 172.16.11.190:80 ESTAB
6 1 Out TCP 411 192.168.1.11:80 209.165.201.11:1867 ESTAB
Client IP:port VIP Address
Server IP Returning half flow
automatically created for
both TCP and UDP flows
INIT, SYNACK,
ESTAB, CLOSED
SYN_SEEN, SYN_SEEN,
ESTAB, CLOSED
Non TCP shows as --
Use conn-id
to track flow
through ACE
Check the
Network
Processor
47
Troubleshooting Connections
Use the show stats connection command to show
connections statistics
Use the clear stats connection command to clear these
counters
ACE/Admin# show stats connection
+------------------------------------------+
+------- Connection statistics ------------+
+------------------------------------------+
Total Connections Created : 288232
Total Connections Current : 2
Total Connections Destroyed: 283404
Total Connections Timed-out: 892
Total Connections Failed : 3934
Note: ACE does not destroy connection. These are connections closed
correctly!!!
48
Troubleshooting Connections
Use the show stats loadbalance command to view the load
balance statistics
To clear the load balance statistical information stored in the
ACE buffer, use the clear stats loadbalance command
ACE/Admin# show stats loadbalance
+------------------------------------------------------------+
+------- Loadbalance statistics ----------------------+
+------------------------------------------------------------+
Total version mismatch : 0
Total Layer4 decisions : 0
Total Layer4 rejections : 0
Total Layer7 decisions : 24
Total Layer7 rejections : 0
Total Layer4 LB policy misses : 0
Total Layer7 LB policy misses : 0
Total times rserver was unavailable : 0
Total ACL denied : 0
49
Troubleshooting Individual Connections
Use the NP and connection ID from show conn command
to view the front-end and back-end connection statistics
using show np <#> me-stats -c <connection ID> -v
ACE/Admin# show np 1 me-stats -c 4096 v
+------------------------------------------------------------+
+------- Individual connection statistics -------------------+
+------------------------------------------------------------+
Connection ID:seq: 4096[0x1000].2
Other ConnID : 8194[0x2002].14
Proxy ConnID : 0[0x0].0
Next Q : 0[0x0]
10.1.1.22:23 -> 12.2.2.14:8739 [RX-NextHop: TX] [TX-NextHop: CP]
Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No
L3 Protocol : IPv4 L4 Protocol : 6
Inbound Flag : 0
Interface Match : Yes
Interface MatchID:24
<snip>
50
Troubleshooting Individual Connections
To further debug and check if the traffic pattern matches the
correct rule, the following command can be used:
show np 1 access-list trace vlan <inbound vlan> in protocol <IP protocol #> source
<source IP> <source port or 0> destination <destination IP> <destination port>
ACE/Admin# show np 1 access-list trace vlan 10 in protocol 6
source 10.10.10.1 0 destination 10.20.30.40 80
<snip> <look for NAT pool ID, vserver ID, etc.>
src nat 0x0 dst nat 0x0 vserver 0x66 fixup 0x0
<snip> <vserver ID here is 0x66 or 102 decimal>
Now, the internal vserver ID 102 can be looked up in the config:
ACE/Admin# show cfgmgr internal table l3-rule | inc 102
102 224 249 0 0 DATA_VALID
Internal Policy Map # is 224 and Class Map # is 249:
ACE/Admin# show cfgmgr internal table policy-map | inc 224
224 MyPolicy9 0 DATA_VALID
ACE/Admin# show cfgmgr internal table class-map | inc 249
249 MyClass4 0 DATA_VALID
51
Troubleshooting VIP
ACE/Admin# show service-policy client-vips detail
Status : ACTIVE
Description: -
-----------------------------------------
Interface: vlan 211
service-policy: client-vips
class: VIP-HTTPS
VIP Address: Protocol: Port:
172.16.11.190 tcp eq 443
loadbalance:
L7 loadbalance policy: HTTPS-POLICY
VIP Route Metric : 77
VIP Route Advertise : DISABLED
VIP ICMP Reply : ENABLED-WHEN-ACTIVE
VIP State: INSERVICE
curr conns : 22 , hit count : 22
dropped conns : 0
client pkt count : 0 , client byte count: 0
server pkt count : 0 , server byte count: 0
max-conn-limit : 0 , drop-count : 0
conn-rate-limit : 0 , drop-count : 0
bandwidth-rate-limit : 0 , drop-count : 0
L7 Loadbalance policy : HTTPS-POLICY
class/match : class-default
LB action :
primary serverfarm: backend-ssl
backup serverfarm : -
hit count : 22
dropped conns : 0
52
Troubleshooting Serverfarm
Best command for
checking server
status and load
ACE/Admin# show serverfarm HTTPS-FARM detail
serverfarm : HTTPS-FARM, type: HOST
total rservers : 4
active rservers: 4
description : -
state : ACTIVE
predictor : ROUNDROBIN
failaction : -
back-inservice : 0
partial-threshold : 0
num times failover : 0
num times back inservice : 0
total conn-dropcount : 0
----------connections-----------
real weight state current total failures
---+---------------------+--------+---------------------+-----------+------
rserver: linux-1
192.168.1.11:0 8 OPERATIONAL 0 0 0
max-conns : - , out-of-rotation count : -
min-conns : -
conn-rate-limit : - , out-of-rotation count : -
bandwidth-rate-limit : - , out-of-rotation count : -
retcode out-of-rotation count : -
53 53
Layer 7 Troubleshooting
54
Layer 7 Policy Hits
Expanding the show service-policy using the detail
option to provide hit count for layer 7 matches
ACE/Admin# show service-policy client-vips detail
Status : ACTIVE
Description: -
-----------------------------------------
Interface: vlan 211
service-policy: client-vips
<snip>
L7 Loadbalance policy : pslb
class-map : curl1
LB action :
serverfarm: s1
hit count : 3
dropped conns : 0
class-map : curl2
LB action :
serverfarm: s2
hit count : 0
dropped conns : 0
Shows hit count for layer 7
load balanced policy
55
Match URL Hit Count
Expanding the show service-policy using the url-summary
option to provide visibility on which match http url are getting
hit
ACE/Admin# show service-policy url-summary
Service-Policy: VIRTUAL-HOSTING-01 L3-Class: WEB-SSL L7-Class: VH-01
match http url /ACCOUNTING/.* hit: 42
Service-Policy: VIRTUAL-HOSTING-02 L3-Class: WEB-SSL L7-Class: VH-02
match http url /BUSINESS/.* hit: 93
match http url /SALES/.* hit: 102
match http url /SPECIAL/.* hit: 67
match http url /BUSINESSOBJECTS/.* hit: 78
match http url /CUSTOMERS/.* hit: 84
Use the show service-policy <service-policy-name> class-
map <L3-class map-name> url-summary to provide better
granularity
56
Troubleshooting HTTP
To effectively troubleshoot HTTP use the show stat http
command
ACE/Admin# show stats http
+------------------------------------------+
+-------------- HTTP statistics -----------+
+------------------------------------------+
LB parse result msgs sent : 6288 , TCP data msgs sent : 9143
Inspect parse result msgs : 0 , SSL data msgs sent : 6041
TCP fin/rst msgs sent : 135 , Bounced fin/rst msgs sent: 19
SSL fin/rst msgs sent : 13 , Unproxy msgs sent : 0
Drain msgs sent : 3107 , Particles read : 37917
Reuse msgs sent : 1539 , HTTP requests : 3145
Reproxied requests : 0 , Headers removed : 1549
Headers inserted : 1598 , HTTP redirects : 2
HTTP chunks : 0 , Pipelined requests : 0
HTTP unproxy conns : 0 , Pipeline flushes : 0
Whitespace appends : 0 , Second pass parsing : 0
Response entries recycled : 3032 , Analysis errors : 0
Header insert errors : 1509 , Max parselen errors : 0
Static parse errors : 9 , Resource errors : 0
Invalid path errors : 0 , Bad HTTP version errors : 0
57
Troubleshooting HTTP Cookies
ACE parses HTTP requests for cookies with the name
given in the configuration and can skip a certain number of
bytes and look for another specific number of bytes.
If the cookie is not found, then the ACE looks for a string
in the URL, starting with one of the characters /?&#+ and
followed by a "=", then parses that value.
If no cookie or HTTP URL cookie exists ACE defaults to the
predictor for that farm
ACE can parse HTTP headers (includes cookies) up to
64kB (default header max parse length is 2048k)
Make sure that sticky timeout (note this is more like an idle
timeout) matches the session timeout on the application
58
Troubleshooting TCP Re-Use
When using TCP connection re-use,"Connection: keep-
alive" is inserted and "Connection: close" is removed from
the clients HTTP request, to avoid closing the server
connection early
User needs to configure Source NAT in the policy map
when using TCP connection re-use
Use the show stats http | include Reuse counters to
check if see if TCP Re-use is in effect
ACE/Admin# show stats http | include Reuse
Reuse msgs sent : 1 , HTTP requests : 4
sh conn detail will also show information about server
side connection reuse
59 59
Troubleshooting HTTP
Compression
60
HTTP Compression Overview
ACE uses Cavium Octeon zip engine
Implement deflate block as defined in RFC 1951
Hardware determines fixed or dynamic Huffman encoding
History buffer is supported to achieve better compression ratio
Support two output file formats. GZIP (RFC1952) or X-
GZIP (RFC2616) and ZLIB (aka DEFLATE) RFC1950
Compression is used with HTTP connection only
Compression only supports HTTP 1.1 protocol
No decompression support
Feature Available on ACE 4710 and ACE30
61
HTTP Compression
Searching for cisco
in www.google.com
Compressed data
62
ACE Compression Traffic Flow Example
2. ACE rewrites
Clients request
GET / HTTP/1.1
Accept-Encoding: gzip,
deflate
1. Request before ACE
GET / HTTP/1.1
Accept-Encoding: identity
Request after ACE
4. ACE Inspects
response
HTTP/1.1 200 OK
Content-type: text/html
Content-Encoding: deflate
Transfer-Encoding:
chunked
6. Response after ACE
Server sends uncompressed
HTTP payload of 5963 bytes
7. Client receives compressed
HTTP payload 2577 bytes
Cisco ACE
Client
LAN
HTTP/1.1 200 OK
Content-type: text/html
Content-Length: 5963
3. Response before ACE
5. ACE
Compresses
Response
Server
WAN
63
Default Compression Controls
Parameter-map type http compression
Minimum content size (512 bytes) to compress
compress minimum-size 100 - Compress if content length is 100
bytes or more
User-Agent Exclusion (Null)
compress user-agent UnknownBrowser - Disallow compression for
Unknown Browser
Compress only http text/* type
compress mimetype image/jpeg - Compress jpeg content
64
Debugging HTTP Compression
Check the following if there no configuration error
From client side:
1.Accept-Encoding is not present or has invalid type
2.User-Agent is being excluded from the configuration
3.HTTP version is not 1.1 or higher
From server side
1.Invalid HTTP response header
2.HTTP response code not 200
3.Content type is not allowed
4.Content length is too small
5.Chunk encoding has invalid format
Get request from client:
GET HTTP/1.1
Host: www.yahoo.com
User-Agent: Mozilla/5.0 Windows; U; Windows NT 5.1;
Accept: text/html,application/xhtml+xml,
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
65
Debugging HTTP Compression (Cont.)
look at the stats from show np x me-stat -s http
Analysis errors: 0 0 General HTTP internal error
Static parse errors: 0 0 General HTTP parsing error
Compression reqs sent: 0 0
Compression rsps rcvd: 0 0
Compression bytes in : 0 0
Compression bytes out: 0 0
Compression rx data in rsp wait: 0 0
Compression no paticles: 0 0 Not enough internal buffer for
compressed output Compression no buffers fpa:0 0 Not enough internal buffer for
hardware
Compression no buffers sglist: 0 0 Not enough internal buffer for
hardware
Compression no buffers result zip: 0 0 Not enough internal buffer for
hardware
Compression session gone: 0 0 HTTP session is deleted
Compression session cleaned: 0 0
Compresssion rslt non-success: 0 0 Hardware compression error
Compression out alloc 0 0
Compression out dealloc 0 0
Compression chunk error 0 0 HTTP input chunk error
Compression error reset 0 0 HTTP compression session reset
Compression session alloc 0 0
Compression session free 0 0
Compression history set 0 0
66 66
Troubleshooting
Secure Socket Layer (SSL)
67
Troubleshooting SSL
Configuration of SSL on ACE is relatively simple. However
if you experience an issue, how to troubleshoot?
Make sure the certificate and key used in ssl-proxy are
valid. Use the crypto verify command
ACE/Admin# crypto verify RSA2048.key RSA2048.cert
Keypair in RSA2048.key matches certificate in RSA2048.cert
Check the size and location of the key. Use the show
crypto key command
ACE/Admin# show crypt key all
Filename Bit Size Type
-------- -------- ----
RSA2048.key 2048 RSA
68
Troubleshooting SSL
Review the certificate details. Use the show crypto
certificate command
ACE/Admin# show crypto certificate cisco-sample-cert
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
ad:e4:e2:f1:50:b7:ce:bd
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=IN, ST=KA, L=BLR, O=CISCO, OU=ADBU, CN=SSL-TEST
Validity
Not Before: Apr 3 09:50:55 2009 GMT
Not After : Apr 1 09:50:55 2019 GMT
Subject: C=IN, ST=KA, L=BLR, O=CISCO, OU=ADBU, CN=SSL-TEST
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public Key: (1024 bit)
Modulus (1024 bit):
00:cf:a2:60:66:5b:ce:b6:38:6f:94:df:0d:1c:61:
26:af:7a:05:49:ed:8d:93:3b
Exponent: 65537 (0x10001)
69
Troubleshooting SSL: CRL Download
Check to make sure you can download the CRL
ACE/Admin(config-ssl-proxy)# do show crypto crl test2 detail
test2:
URL: http://119.60.60.23/test.crl
Last Downloaded (Cached): Sat Aug 8 16:14:24 2009 UTC
Total Number Of Download Attempts: 1
Failed Download Attempts: 0
Successful Loads: 1 Failed Loads: 0
Hours since Last Load: 0 No IP Addr Resolutions: 0
Host Timeouts: 0 Next Update Invalid: 0
Next Update Expired: 0 Bad Signature: 0
CRL Found-Failed to load: 0 File Not Found: 0
Memory Outage failures: 0 Cache Limit failures: 0
Conn failures: 0 Internal failures: 0
Not Eligible for download: 3 HTTP Read failures: 0
HTTP Write failures: 0
Looking for all best-effort CRLs in the system and their
download status. Use the show crypto crl best-effort
command
70
Advanced SSL Debugging
This command provides the current crypto statistics
ACE/Admin# sh np 1 me-stats "-s crypto
Crypto Statistics: (Current)
------------------
ARC4 operations: 376572 0
TCP msgs received: 285260 0
APP msgs received: 235151 0
Nitrox messages forwarded to XScale: 381041 0
SSL ctx allocated: 47758 0
SSL ctx freed: 47758 0
SSL received bytes: 61070430 0
SSL transmitted bytes: 283256220 0
SSL received application bytes: 7679113 0
SSL transmitted application bytes: 275120867 0
SSL received non-application bytes: 53391317 0
SSL transmitted non-application bytes: 3292887 0
Bulk flush operations: 95037 0
ME records sent to XScale: 285808 0
ME records received from XScale: 47723 0
ME hw responses: 471516 0
First segments received: 47400 0
Handshake failure alert: 94 0
CM close: 446 0
71
Advanced SSL Debugging
The show stats crypto server command provides
statistics of the SSL handshake
ACE/Admin# show stats crypto server
+---- Crypto server termination statistics -----+
+------- Crypto server alert statistics --------+
+--- Crypto server authentication statistics ---+
+------- Crypto server cipher statistics -------+
+------ Crypto server redirect statistics ------+
+---- Crypto server header insert statistics ---+
These statistics provide details of the SSL packets for
example; which version client interacted with ACE,
which cipher is used, whether re-handshake happened,
whether session id reuse happened and which SSL
alerts are received or sent by ACE
72 72
73
Fundamentals for ACE Probing
ACE probes are fundamental to the system. It is key to not
oversubscribe the ACE health monitoring system
Use the show resource internal socket to determine how
many sockets ACE has open. This is an Admin command
ACE/Admin# show resource internal socket
Application MaxLimit Current Creates Frees
--------------------------------------------------------------
SYSTEM 4000 0 0 0
CRITICAL 50 0 0 0
AAA 256 0 0 0
MGMT 256 0 0 0
XINETD 512 1 12 11
HEALTH_MON 2500 532 193494 192962
USER_TCL 200 0 0 0
SYSLOG 256 10 14 4
VSH 256 0 0 0
OverAll - 650 194812 194162
Non Reg App Usage: 107
74
Health Monitoring Process
If you see probing issues, check the health monitoring
process. The show proc cpu command provides very
useful information
ACE/Admin# show proc cpu
CPU utilization for five seconds: 30%; one minute: 37%; five minutes: 35%
PID Runtime(ms) Invoked uSecs 1Sec 5 Sec 1 Min 5 Min Process
972 1072965 613352 1749 35.9 18.5% 21.67% 20.90% arp_mgr
HM process is only consuming 1.40%. Why is the control
plane CPU running at 30%? Check what process is
consuming CPU
ACE/Admin# show proc cpu | inc hm
CPU utilization for five seconds: 30%; one minute: 37%; five minutes: 35%
PID Runtime(ms) Invoked uSecs 1Sec 5 Sec 1 Min 5 Min Process
987 90257 57805 1561 0.0 1.40% 1.46% 1.43% hm
988 90198 58952 1530 0.0 1.49% 1.49% 1.44% hm
989 851 2947 288 0.0 0.0 % 0.1 % 0.0 % hm
990 0 2 56 0.0 0.0 % 0.0 % 0.0 % hm
75
Use the show probe detail command to determine the
status of the probe or possible last failure
ACE/Admin# show probe detail Cut output
--------------------- probe results --------------------
probe association probed-address probes failed passed health
------------------- ---------------+----------+----------+----------+-------
rserver : CAS1
10.7.53.55 24 24 0 FAILED
Socket state : CLOSED
No. Passed states : 0 No. Failed states : 1
No. Probes skipped : 0 Last status code : 403
No. Out of Sockets : 0 No. Internal error: 0
Last disconnect err : Received invalid status code
Last probe time : Wed Nov 25 18:48:16 2009
Last fail time : Wed Nov 25 18:25:16 2009
Last active time : Never
76 76
High Availability on ACE
77
High Availability Basic Building Blocks
FT PEER
Only one FT peer per ACE device
1:1 peer relationship
FT GROUP
One FT group per ACE virtual context
FT VLAN
Designated VLAN between the redundant peers
All HA related traffic sent over this VLAN
FT VLAN can be trunked between two Catalyst 6500 Chassis
Should not be used for normal traffic
Admin
Context
Context A
Context B
Context A
Context B
ACE2 (FT PEER)
FT VLAN
FT Group 2
FT Group 3
ACE1 (FT PEER)
FT Group 1
78
High Availability Control Traffic
TCP Connection between FT Peers
State Machine (Election, Preempt, Relinquish)
Configuration sync
State Sync for ARP
Heartbeats between FT peers
Heartbeats are sent over UDP
Monitors the health of the peer
Heartbeat interval and count are configurable
79
ACE High Availability State Machine
Active/Standby Election (assuming both peers are
initialized at same time)
Based on a priority scheme
Member with highest priority becomes ACTIVE
Other member enters the STANDBY_CONFIG state
If priorities are equal, member with the higher IP address
wins
STANDBY_CONFIG State
Startup Configuration Sync from Active to Standby
Running Configuration Sync from Active to Standby
Knob to turn on/off
80
STANDBY_BULK State
ARP Sync (knob to turn on/off)
Connection Table Sync
Sticky Database Sync (knob to turn on/off)
STANDBY_HOT State
Standby FT group member is ready to take over
Incremental Configuration Sync from Active to Standy
Incremental State Sync from Active to Standby
STANDBY_COLD State
Due to error during Config Sync or Incremental Config Sync
No Config or State Sync happens from Active to Standby
STANDBY_WARM State
Major version mismatch between peers (example 2.x and 4.x)
81
Mismatch in software version
FT Peer may become INCOMPATIBLE
ACTIVE ACTIVE state on both FT group members
Mismatch in Virtual Context Licenses
Configuration Sync (all types) for Admin context is disabled
State Sync for Admin context will continue to happen
For matching user contexts Configuration State Sync will
work
Mismatch in Other Licenses
Configuration and State Sync will work
After switchover, new Active will handle traffic as per its
licenses
82
ACE Redundancy Query VLAN
When no heartbeat is received, ACE can use the Query
Vlan to check the HA status
ACE tries to do a ping to the destination via the Query
VLAN
If ping fails, the Standby will transition to the ACTIVE
state
If ping succeeds, the Standby will transition to a
STANDBY_COLD state
To configure a query interface, enter the following:
ACE/Admin(config-ft-peer)# query-interface vlan 110
83 83
More Debugging Commands
84
Additional Debugging
Some more ACE debugging commands
show np <#> me-stats -cpu
show np <#> me-stats Q
show np <#> me-stats -s fp
show np <#> me-stats -s tcp
show np <#> me-stats -s icm
show np <#> me-stats -s ocm
show proc cpu
show netio stats
Show service-policy summary
85
Recommended Reading
2011 Cisco and/or its affiliates. All rights reserved. Cisco Public
86
BRKAPP-3003
Complete Your Online
Session Evaluation
Receive 25 Cisco Preferred Access points for each session
evaluation you complete.
Give us your feedback and you could win fabulous prizes.
Points are calculated on a daily basis. Winners will be notified
by email after July 22nd.
Complete your session evaluation online now (open a browser
through our wireless network to access our portal) or visit one
of the Internet stations throughout the Convention Center.
Dont forget to activate your Cisco Live and Networkers Virtual
account for access to all session materials, communities, and
on-demand and live activities throughout the year. Activate
your account at any internet station or visit
www.ciscolivevirtual.com.
87
BRKAPP-3003
87
Visit the Cisco Store for
Related Titles
http://theciscostores.com
88
BRKAPP-3003
89
BRKAPP-3003
Thank you.
90 90
Appendix and Additional
Troubleshooting Information
91
Additional Information
Layer 4 flow setup
Layer 7 flow setup
TCP Connection States
92
Layer 4 Flow Setup
SYN
SYN_ACK
Shortcut
ACK
Shortcut
Data
Shortcut
Data
Shortcut
Matches Existing
Flow
Rewrites L2/L3/L4
Matches VIP
Selects Server
Rewrites
L2/L3/L4
Basic Load Balancing
Source IP sticky
TCP/IP Normalization
93
Layer 7 Flow Setup
Client Connects to L7 VIP
SYN
Starts
Buffering
ACK
Data
ACKs Client Packets
Keeps Buffering
Matches VIP w/L7
logic
Chooses SEQ #
Replies w/SYN_ACK
HTTP L7 rules on first request
(cookie sticky, URL parsing, )
Generic TCP payload parsing
94
Layer 7 Flow Setup (Cont.)
ACE Establishes Connection to Server
Data
SYN_ACK
Empties Buffer
Sends Data to Server
Acts as Client
Does Not Forward
SYN_ACK
Parses the Data
Selects Server
Initiates TCP
95
Layer 7 Flow Setup (Cont.)
ACE Splices the Flows (UNPROXY)
ACK
Data
Shortcut
ACK
Shortcut
Data
Shortcut
Matches Existing Flow
Rewrites L2/L3/L4
and SEQ/ACK
Does Not Forward ACK
Ready to
Splice the Flows
96
Layer 7 Flow Setup
ACE Reproxies the Connection
ACK
Data
ACK
Data
Shortcut
ACK
Shortcut
Shortcut
Shortcut
Data
REPROXY
ACKs GET & Buffer
HTTP L7 rules with HTTP 1.1

connection keepalive
(persistence rebalance)
97
Layer 7 Flow Setup
ACE Acts as a Full Proxy
F
u
l
l

P
r
o
x
y
I
n
d
e
p
e
n
d
e
n
t

c
l
i
e
n
t

&

s
e
r
v
e
r

c
o
n
n
e
c
t
i
o
n
s
SYN
SYN_ACK
ACK
Data
GET/HTTP 1.1
ACK
SYN
SYN_ACK
ACK
DataGET
ACK
ACK
Data
Data
HTTP/1.1 200 OK HTTP/1.1 200 OK
Client connection Server connection

SSL offload
TCP re-use
Protocol inspections
HTTP 1.1 pipelining
98
TCP Connection States
L4 TCP Connections
SYNSEEN (Client SYN received)
INIT (Server side half flow initialized)
SYNACK (SYN ACK sent by server)
ESTAB (Client and Server; TCP Handshake completed)
L7 TCP Connections
SYNSEEN (Client SYN received)
ESTAB (Client side TCP Handshake completed; SYN
ACK sent by ACE, Client ACK received)
ESTAB (Server side TCP Handshake completed from ACE
after L7 data received from the client and parsed)
CLOSED (Client or Server FIN ACK followed by ACK)

Brkapp 3003

Загружено:

Сведения о документе

Исходное описание:

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Brkapp 3003

Загружено:

Авторское право:

Доступные форматы

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public BRKAPP-3003

HTTP L7 rules with HTTP 1.1

Вам также может понравиться