http://www.agilent.

com/comms/onenetworks

LAN Troubleshooting

Case in Point
Server Load Balancing and the MegaProxy Problem
To continue our efforts to help you with your network needs, we will be making several real-world network troubleshooting case studies available to you.
A brick and mortar retail firm that rolled-out their business online within the last year. The online retail business built customer facing web services that were hosted at a collocation facility. In order to scale the firms e-commerce capacity to the desired levels and to gain high availability of services, layer 4-7 server load balancing switches were placed in front of the web servers. Problem: Customer support was receiving calls from angry online shoppers with complaints of disconnections from the retail web site.

New case studies will be released every 2 to 3 weeks and will cover all topologies and a variety of network problems. Taken from Enterprise companies, Service Providers and Network Equipment Manufacturers, we will show you how companies, such as yours, have solved some of their network problems. You will have an opportunity to vote for the case studies of most interest to you. From these votes we will select the case studies for each posting. For additional case studies and to vote for the case studies you would like to see featured next, be sure to regularly visit our web site at:

http://www.agilent.com/comms/casestudies Feel free to pass this link on to your colleagues or friends.

1

http://www.agilent.com/comms/onenetworks

LAN Troubleshooting

Case Study: Server Load Balancing and the MegaProxy Problem

A brick and mortar retail firm rolled-out their business online within the last year. The online retail business built customer facing web services that were hosted at a collocation facility. In order to scale the firms e-commerce capacity to the desired levels and to gain high availability of services, layer 4-7 server load balancing switches were placed in front of the web servers.

Case Study: Server Load Balancing and the MegaProxy Problem

An online retailer whose customer facing web services are hosted at a collocation facility

Server load balancing switches were placed in front of the web servers
Customer support is receiving calls from angry online shoppers with complaints of disconnections from the retail web site

Customer support was receiving calls from angry online shoppers with complaints of disconnections from the retail web site. We began troubleshooting this issue by performing tests with client PCs that were placed on the Internet to make mock transactions to the retail web site. There was no connection drops observed from the external client tests performed.

http://www.agilent.com/comms/onenetworks

LAN Troubleshooting

Case Study: Server Load Balancing and the MegaProxy Problem

Tests were conducted where clients were placed externally out on the internet to make mock transactions to the web site There were no connection drops from the external client tests performed Packets were captured at the following locations: Between the http servers and the SLB switches External to the SLB switches Packet captures confirmed a relatively moderate number of hard TCP session terminations Limited debugging was turned on the SLB switches The debug information from the SLB switches provided little added help to diagnose the issue Packets were captured between the HTTP servers, SLB switches, and the public Internet with a multi-port protocol analyzer. The captures confirmed a relatively moderate number of hard TCP session terminations and resets. Limited debugging was turned on the SLB switches in a non-intrusive mode which eventually revealed little to help diagnose a cause.

Case Study: Server Load Balancing and the MegaProxy Problem

An analysis of the customers complaining was logged and analyzed A new process assigned to customer service did identify a group of customers having the problem A review of all settings on the SLB switches was done Another internet based client test was performed after the examination of the customer data The second attempt to test client connectivity from the internet confirmed the suspected ISP During the mock transactions, the source IP address would change from time to time, always rotating between the same few IP source address, suspected to be a pool of proxy servers

An analysis of the customers complaining was logged and analyzed. Incorporating a special questionnaire into the customer service operations did result in the following two discoveries: 1) Nearly 90% of all customers experiencing the connection drops belonged to the same service provider of which was very popular and perhaps one of the largest. 2) Nearly all of the customers questioned had their session terminated while in the attempt to file a purchase. The difference between a normal session and a purchase transaction session is that SSL will be invoked during the latter.

http://www.agilent.com/comms/onenetworks

LAN Troubleshooting

Case Study: Server Load Balancing and the MegaProxy Problem

SSL expects the source address not to change during a session The SLB switches were configured to balance across the servers in a round-robin method Each time a client/proxy changes its source IP address, the server load balancer treats that conversation as a new session which then round-robins it to the next server in line The new server now receives the SSL session rejects the flow as it did not originate its establishment

Another Internet based client test was performed after the examination of the customer data in item. This second attempt to test client connectivity from the Internet was more successful that time. In this instance, a number of memberships to the suspect ISP were obtained and used. Combined with switch debugging and packet captures, the test confirmed the Internet based source of the problem as well as the following: 1) Patterns of source IP addresses from the clients were logged. The sampling of clients used in the test had the same IP source address. This indicates that a proxy server might have been involved. 3) During the mock transactions, the source IP address would change from time to time, always rotating between the same few IP source addresses, suspected to be a pool of proxy servers. 4) SSL expects the source address not to change during a session and drops any connection when a source does change its IP address. It will treat the change as a completely new session.

http://www.agilent.com/comms/onenetworks

LAN Troubleshooting

Case Study: Server Load Balancing and the MegaProxy Problem

Root Cause: It was confirmed that this particular service provider did use a pool of proxy servers for securing its community SSL sessions are dropped when the source IP of a client is changed Solution: In order to provide persistence in session connection between a dynamic source and a secure web server, the SLB switch was configured to capture the SSL session ID The session ID is used to maintain the connection between client and server regardless of what IP address changes may occur The new configuration overrides its server load balancing process to maintain persistence

Review of all settings on the SLB switches. The SLB switches were configured to balance across the servers in a round-robin method. This creates a problem for SSL connections with the dynamic source IP assignments. Each time a client/proxy changes its source IP address, the server load balancer treats that conversation as a new session which round-robins it to the next server in line. The new server now receives the SSL session rejects the flow as it did not originate its establishment. It was later confirmed that this particular service provider did in fact use a rotation of proxy servers to add a level of security to its online community. But by doing so, created what is now called the MegaProxy problem. SSL sessions were dropped when the source IP of a client was changed. From the http servers perspective, the client is the proxy server. In order to provide persistence in session connection between a dynamic source and a secure web server, the SLB switch was outfitted with code that could passively capture the SSL session ID. The SSL session ID then becomes the unique identifier and when the source IP changes in the flow, the SLB switch overrides its server load balancing process to maintain persistence.