How to Diagnose and Fix

Network Performance Problems

John S. Graham
Global Research NOC
Synopsis
Network Performance Problems
Definitions
Network Testing
Going Loopy
Purpose
Taxonomy
Testing New Leased Lines
Dedicated Test Sets
Network Contribution to Poor
Performance
1. Incorrect MTU Configuration
1. Standardized IP MTU of 9000 Bytes
2. Check end-to-end: ping -M do -c 1 -s
8972 <far end server>
2. Causes of Packet Loss in the Network
1. Due to a Congested Line
2. Due to a Line Introducing Transmission Errors
3. Process Switching by a Router in the Path
iPerf Testing
A UDP iPerf test will probe the network for packet
loss
Always employ a UDP test:
Eliminate Layer-IV Flow Control from affecting results
Simplex test selectively probes one direction (Source
Sink or Sink Source)
The bitrate can be operator controlled
Use a modest packet rate for testing
High-bitrate tests do not reveal additional information
Excessive test traffic may cause congestion!
Whos the Villain? MY Router Interface
Or THEIR Circuit?
10G Circuit Over DWDM Ring
Direct Router-to-Router Adjacency over 10GE Interfaces
Kansas City
Houston
Miami
Atlanta
Representation of DWDM Circuit
Tributary Side Line Side
Cross-Connect Termination Points
Tributary Side Line Side
Soft Loop Classification
Facility Loops
Loop is applied to fiber side of CTP
Equipment (aka Terminal) Loops
Loop is applied to cross-connect side of CTP
Supported Only on Tributary CTP
Infinera DTN
Four Possible Configurations
Supported on Trib & Line CTPs
Ciena CoreDirector
Eight Possible Configurations
A Bestiary of Loops
Testing From a Router
1. Identify Far-End IP Address
2. Configure ARP Mapping : {IP}
FE
<-> {MAC}
NE

3. Clear Egress Interface Counters
4. Send Measured Number of Pings to {IP}
FE
5. EITHER Looped Network Clear of Errors
1. UCAST_IN = UCAST_OUT = Pings TTL
6. OR Looped Network Introduced Errors
1. UCAST_IN < UCAST_OUT
Testing a New Leased Line
10G Leased Line
iperf -c 192.168.8.1
-u -b10m
Juniper MX960
Brocade MLX-16
A/32
B/32
192.168.8.1 A/32
192.168.8.1 B/32
4
2
1
3
Static Route
Firewall Rule (Inbound)
P2P VLAN
A Typical Ethernet Tester
2 10G Ports (XFP Optics)
Management
Network Testers vs Software Probes
Network Testers
Do not process packet
headers.
Operator can design packets
with any combination of
protocols
Header fields may contain
any value
Optimized for packet
counting
Will accept, identify and
count every inbound frame;
even those returned by a
loop
Software Probes
Only accept packets with
valid headers.
Packet design severely
restricted.
Many header fields not
accessible
Accurate packet counting
difficult with standard tools
Unable to operate in
conjunction with a looped
network connection