Identifying and Resolving Network Bottlenecks

Performance bottlenecks have always been the bane of networks and with the advent of virtualization, cloud computing and web-based applications, the free and easy flow of data is critical to maintain productivity and quality of service. Well have a look at factors that could clog your network, starting with the perennial suspects CPU, I/O, memory and disk, and move on to other potential bottlenecks brought on by changing technology. Processor, memory and disk space The traditional big three of bottlenecks, these are the first leads to investigating performance degrades at a node level. In spite of processor speeds getting progressively faster, CPUs still come up as a bottleneck, especially when the processor is unable to handle bursts (short periods when activity exceeds 100%). Sustained high processor utilization (say exceeding 75%-80%) could indicate potential/present bottlenecks. Similarly, memory bus can be a source of clogs discrepancies between processor and memory can slow down systems. A single server with a slow processor or memory can clog up networks but happily, this is straightforward to identify, and a good performance monitor can proactively identify the clogs, and upgrades/re-configuration/replacements can fix the problem. Additionally, server clustering, redundancy and failover techniques should be deployed for both efficiency and security. I/O Improperly configured I/O interfaces and poorly designed architecture can cause dramatic slowdowns, especially in virtual environments, where multiple virtual machines (VMs) send I/O streams to a single physical host. Likewise, I/O discrepancies are the most common source of bottlenecks in data centers, resulting in poor response time and increased latency. Proper monitoring software can help proactively identify the bottlenecks for instance, by monitoring storage metrics including queue and kernel latency, IOPS (Input/output operations per second) and throughput. Software applications and malware Poorly designed codes can also cause bottlenecks. Performance testing is a must to avoid software bugs, and can help identify potential problems before code goes into production. Logging too can provide essential pointers collecting vital data including query response times, memory usage, execution and delivery times, for example, can help in identifying bottlenecks. Likewise, spyware, viruses and Trojans are obvious suspects. A stringent protection system, including perimeter networks, patches and anti-malware, can eliminate potential malware-induced bottlenecks. Hardware Legacy systems, of course, are guaranteed chokers, and should be upgraded or replaced as necessary. This is especially vital during virtualization, or cloud deployment, since improper capacity planning and configuration i.e. a physical host unable to handle load/legacy systems incompatible with resources on the cloud can plug the network and exponentially increase response times. Network Hardware and protocols Most bottlenecks, quite unfairly, are linked to network failure. While it could be system hardware or application failure,
Identifying and Resolving Network Bottlenecks 1

switches, routers, NIC, cables and improper network configuration can cause problems. Some ports and protocols, especially TCP/UDP, of many systems and applications are known to hog resources. Improper layer 2 configuration could result in absent/filtered spanning trees and lead to broadcast storms, paralyzing switches in minutes. Proper network planning and configuration, and constantly monitoring network health, can help in avoiding these bottlenecks. Bottlenecks in virtual environments As organizations embrace virtualization, proper capacity planning is vital before the move. Shared resources are an essential feature of virtual environments, where CPU, memory, disk space, I/O are all pooled together. As the environment grows, resource usage could increase considerably, and a bottleneck in any single system would impact the physical host and the virtual systems accessing it. Again, adding virtual machines is easy, and too often we see VMs added indiscriminately, especially if there is no set policy in place. Ad hoc addition of VMs ultimately results in VM sprawl a sure starting point for degraded performance. And if resources are improperly allocated, without proper provisioning, the entire network performance can be degraded as departments add VMs that use up already stretched resources. You can avoid bottlenecks in virtual environments through: Proper capacity, application and configuration planning before deployment Resource pools - Resources should be pooled with proper allocations for VMs, depending on how business-critical they are, as reservations (guaranteed allocation), shares (over-committed resources) and limits. By properly nesting the VM resource allocations, its possible to ensure critical application/ VMs are reserved enough resources. The remaining resources can then be split up among the less critical VMs as per the shares they hold. Constant monitoring of resource allocation and usage Setting and deploying strict policies Especially with regard to VM addition, changing critical configurations, power-on/off of VMs, etc. (remember, when you have an enormous number of VMs, powering on/off can cause a huge change in resource usage) Bottlenecks can hit virtual storage environments particularly hard. VMs hogging storage bandwidth should be identified as soon as possible to avoid escalating situations, as with traditional data centers, virtual storage is also prone to potential problems caused by I/O-to-disk discrepancies. In virtual environments, the multiple I/O pipes from various VMs often can get tangled in the hypervisor, which has to juggle multi I/Os to a single physical disk. This can lead to bottlenecks and hotspots, and eventually to degraded I/O performance. This random I/O patterning is called the I/O blender effect. While virtualization players are coming up with new technologies to counter I/O blenders, you can identify storage bottlenecks by ensuring: End-to-end visibility at every level (both storage and virtual) Proper physical disk-VM mapping Constant monitoring with pertinent thresholds, alerts and filters Bottlenecks in the cloud Web-based applications are an integral part of many organizations, and Software as a Service (SaaS) is increasingly
Identifying and Resolving Network Bottlenecks 2

being used for various applications including emails, ERP, data warehousing, audio-video applications and the likes. As more business-critical applications move to the cloud, uninterrupted access and real-time rapid response times become critical. Performance downgrades due to network bottlenecks become almost unacceptable, and often result in loss of productivity and availability, poor user interaction, and can even bring on security breaches and data loss. Various measures can be adopted to deal with bottlenecks on the Web. Most commonly, you can increase the bandwidth to provide a wider highway between users and the cloud. Many organizations also deploy WAN optimization methods to increase transfer efficiencies. And, as with virtual environments, proper policies and institutional usage for cloud access and usage can also go a long way in avoiding bottlenecks. SLAs are another important area to look into most Platform as a Service (PaaS)/ Infrastructure as a Service (IaaS) vendors draw up agreements based on availability, but performance is not always clearly defined. So as consumers compete for shared resources, cloud users should ensure the SLAs reflect on the desired levels of availability and performance. WAN bottlenecks Laptops, smartphones and PDAs also clog network resources, frequently resulting in bottlenecks. While organizations employ a variety of steps to resolve/avoid them from WAN optimization techniques to changing hosting companies, an ideal first step would be to identify application speeds and throughput before deployment. WAN optimization techniques are commonly employed to improve data efficiencies across networks, and include: Deduplication - Eliminates redundant data transfer by sending references instead of actual data Traffic shaping - Controls data flow as required by critical applications/users Caching Compression Protocol optimization Improves user response times by anticipating requests and retrieving data based on specific protocol patterns Application acceleration Speeds up performances of centralized applications that are accessed over remote networks Latency optimization techniques, including window size scaling, and Layer 3 algorithms to control congestion, propagation, processing, serialization, queuing delays as well as transport layer implementations Choosing the right network monitoring solution (NMS) As mentioned, the right NMS can ensure that bottlenecks are immediately identified and swiftly resolved. With WhatsUp Gold and the network traffic monitoring solution Flow Monitor, you can be alerted when custom configurable thresholds set up by you (i.e. tracking failed connections per host) are exceeded, enabling you to proactively troubleshoot and resolve performance bottlenecks. So with WUG + Flow Monitor, you have the complete real-time visibility you need to manage bottlenecks and bandwidth utilization, and ensure optimal network performance.

Identifying and Resolving Network Bottlenecks