A Framework for Testing IP QoS over ATM Networks: Implementation and Practical Experiences

Norbert KROTH, Lutz MARK, Jens TIEMANN

GMD FOKUS Kaiserin-Augusta-Allee 3 1, 10589 Berlin - Germany Phone: +49-30-3463 7000 FAX: +49-30-3463 8000 E-Mail: { kroth I mark I tiemann} @fokus.gmd.de
Abstract: Driven by increasing competition, service providers target customers to deliver value added IP services, like Voice over IP (VoIP) or Virtual Private Networks (JPN) over their networks. Thus, IP has become the de facto convergence layer which integrates all types of traffic within a heterogeneous network infrastructure On the one hand IP was originally designed for besl. effort networks. On the other hand many new applications rely on a pre-specified end-to-end Quality of Service (QoS) with statistical guarantees on delay and loss. In this context the question arises, how will the quality of IP f?ows be evaluated? It is shown th.at a modular approach is highly efficient to test QoS parameters and mechanisms. This paper describes the principles of a modular testing system implemented at GMD FOKUS and presents the results of practical experiments. The first part of the paper
introduces the architecture of a modular testing system

1. Introduction
The Internet Protocol is widely used in a converging network market. The key issue for business success is that the network can provide efficient mechanisms to handle the different categories of IP services (e.g. real-time, non real-time). The connection-oriented Asynchronous Transfer Mode (ATM) is still the most efficient way to assure quality of service in today networks, by using its native QoS mechanism. There are many different approaches to implement IP over ATM (e.g. LANE, CLIP, MPOA, MPLS) in order to support IP applications in ATM networks [ 11. The performance of these approaches need to be assessed by three main target groups: servicehetwork providers have to assure a certain quality to their customers, user of network services want to be sure to get what they have paid for, manufacturers and developers have to check their algorithms and complex systems. This paper describes an efficient approach towards IP QoS testing in ATM network environments. me modular measurement and testing system implemented at GMD FOKUS consists of both software and hardware modules. It is easily adaptable to any measurement task and supports both active and passive measurement methods. Since the early nineties GMD FOKUS has been engaged in research and development in the field of ATM, working on native ATM applications, introducing new mechanisms to ATM and ATM testing. Experience is especially gained from monitoring, measurement and analysis of ATM and IP flows in numerous research projects, also in cooperation with industry partners. To provide open interfaces for network testing and enable absolute access to all protocol layers, GMD FOKUS was forced to develop its own test tools beside using commercial test equipment.

and informs about current research activities to set up a framework for testing IP QoS over ATM networks. The second part discusses different methods to measure QoS paramei:ers in an ATM network environment. The third part presents results of current experiments concerning developments for IP QoS testing. Thus the paper gives an overview on the entire cycle of QoS measurements: from test configurations via necessary test functionalities to the presentation of test results.
Keywords: Modular Testing Approach, Testing Next Generation IP Networks, End-to-End IP QoS, Testing Hardwa.re

0-7803-5428-1/99/$10.0001999 IEEE

212

2. Architecture of a Modular Testing System for ATM Networks

2.1. Motivation
Based on experiences in ATM testing and the need to cope with an increasing complexity of test conditions in future networks a modular approach to ATM testing seemed feasible in order to create an open and scalable test and measurement system. As a result a modular testing system for ATM networks was developed. The aim was to provide an efficient but reasonable solution to ATM testing. The orientation to this problem is derived from numerous practical experiences, the principles and restrictions of offthe-shelf measurement equipment.

packages) and a GUI to control and perform measurement tasks.

2.3. Test Hardware - TANYA

The TANYA test hardware is an interface card which has been designed to access ATM networks and to implement specific functions for measurement purposes. Likewise the software, TANYA is also characterized by a modular structure, using VHDL (a hardware description language) for programming various hardware functions. The basic functionality of the TANYA hardware is: Adaptation to the physical layer (SDH/SONET) Format conversion between the cell file format to the ATM cell format Processing highly accurate timestamps to evaluate traffic profiles Interface to the host system
A
i

2.2. System Architecture

A basic principle of the system architecture is modularity. The main idea of this approach is to identify a minimal set of specific modules which can be combined to set up a powerful and open testing system. A modular system can be easily adapted and extended to special testing purposes to carry out various measurement tasks. Figure 1 shows the overall structure of the ATM testing system. The ATM testing framework is based on the combination of both software and hardware modules, which together build complex functions. The interface between the testing hardware (TANYA) and the low level software modules (cell file tools) is the cell file format (CA. Higher layer protocol analysis is done with software tools working with an additional format (packet file - pf), which is derived from cf. The software modules together represent the ATM measurement tool set (FAST). The TANYA test interface fits in this framework as a component, which realizes the format conversion between the cell file format and the ATM cell format defined by the ITU-T [ 2 ] .
r - - - - - - - - 1

L ' ,
SNNl

I Sync

'utopia

Figure 2: TANYA hardware architecture. The TANYA ATM network interface is designed on a generic architecture with three major components: SBus Interface, cell file controller and physical layer controller (see Figure 2). The flexible architecture allows one to exchange the host interface as well as the physical layer as required. To support a wide range of physical layers the well defined UTOPIA interface [3] was used. Further, the TANYA architecture additionally includes a mechanism to synchronize several TANYA cards to set up a distributed test system to perform tests in a distributed manner. The core of the design is the cell file controller (shown in Figure 3), a FPGA (Field Programmable Gate Array), which is programmable in the system (hardware configuration). The overall functionality of the cell file controller is scalable by just combining the necessary function blocks, implemented in VHDL, to carry out a specific measurement task. The basic model of the cell file controller is composed of a set of modules for format conversion and timestamp processing (with a resolution of 80 ns) and can be extended by additional function blocks. The optional modules are measurement specific and

I I
I

(G]
I

I L

- -UNlX Workstation - - - 4 ----

Figure 1: Architecture of a modular ATM tester. A complete testing system is considered as a general purpose SUN workstation running Solaris, including one or more TANYA cards and using FAST, which also offers scripts, user software (e.g. office or visualization software

21 3

realize the following hardware functions: Filters (e.g. on ATM cell header) Traffic generators OAM functions Partial AI& processing
mxn

Cellfile Controller

I I

.
UTOPIA

Transmit

StatuslCommand Register

Figure 4: FAST software architecture. The tools for ATM testing can be used separately as well as in combination for a more detailed analysis. FAST also offers functions for AAL-processing and higher layer protocol analysis (e.g. IP). These programs use the FOKUS packet file (pf) format as a standard interface. The p was defined to work on any protocol layer above ATM. f It is similar to c5 the packet are enhanced with timestamps, which are calculated from the corresponding timestamps in the cell file. Before decoding higher layer protocols AAL reassembly has to be done first. For decoding the TCP/IP protocol family tcpdump (originally written by Van Jacobsen [4]) is currently used. The IP tools are accomplished by IP-filters and programs for IP flow analysis. Decoding of ATM signaling can be done according to ATM Forum or ITU. For automated measurement processes the software tools can used within shell scripts. The FAST tool set further contains conversion modules to commercial A M - I S D N testers like HP BSTS [5] to integrate tester from other vendors into the FAST system concept. Thus, it is possible to exchange data between different testing systems and to operate widely independent of a specific tester. Vice versa, the interfaces also allow to access measurement software from other vendors to process data captured with TANYA [6]. For example data generated by the simulation tool Prolemy [7] can be used. Additionally, the FAST tool set can work with standard ATM network interfaces in cases where a low accuracy (e.g. for load generation) is acceptable.

UTOPIA

-J

Receive

Figure 3: Generic model of the cell file controller. The current available hardware configurations are classified according to the characteristic functionality of the TANYA testing interface. For this reason, three basic configuration classes are distinguished:
General Tester or Network Interjace (NIC): Functions for receive and transmit data path. Network Monitor: Mainly receive functions, an additional line-loopback is possible on the physical layer.
9

TrajFc Generator: Mainly transmit functions, continuous generation at link rate is possible.

traffic

2.4. Test Software FAST

The ATM measurement tool set (FAST) is the software required for PLTM testing. The tool set consists of different software modules implemented as programs in portable C code, currently under Solaris. The modules operating on ATM level with a common data format, which is based on the cell file format including the ATM cell with header/ payload: and ,a 32-bit timestamp. The modules (shown in Figure 4) realize certain software functions for analysis, manipulation and composition of cell files (cfl. Thus the following basic functions are available to perform tests within the ATM layer: Capture and play back of ATM traffic Analysis and presentation of captured ATM traffic Synthesis of artificial ATM traffic Filtering and manipulation of ATM traffic

3. I Quality of Service Testing P 3.1. Architectures for IP QoS

The delivery of good-quality audio or video streams typically requires QoS capabilities. Therefore, IP networks have to provide some guarantee of performance such as connectivity, speed and reliability [SI. The two different approaches to support IP QoS are:

214

Signalled QoS This approach is based on dynamic establishment of connections by signalling protocols (e.g. RSVP, ATM SVC). It is used with the Integrated Service (IntServ) model. The IntServ scheme provide per-flow classification and guaranteed delays to support also real-time traffic. Configured QoS This approach mainly represents a bandwidthmanagement scheme for IP networks. It is used with the Differentiated Services (DiffServ) model. The existing Type-of-Service (ToS) byte in every IP packet header marks the priority or service level that a packet requires. A possible way to provide QoS in IP networks is to use ATM as a transport mechanism [8] 191. Such a network is able to offer QoS mechanisms on data link layer (ATM) as well as on network layer (IP). In ATM core networks, the IP data streams are transported over virtual circuits. On the one hand this can be accomplished by using PVCs, set up by the network operator, or SVCs requested by an end system. Due to the variety of IPoA implementations we have identified an increasing need for efficient testing methods and powerful test and measurement tools. Our approach is to set up an IP QoS testing framework which is based on the TANYA testing interface and the FAST tool set. The existing testing framework was extended by additional soft- and hardware modules to evaluate quality of service on IP level.

U
Monitor

Figure 5: Generic scenario for passive flow monitoring.

Active generation of test data flows with subsequent analysis of the flows; an intrusive method. Here test traffic is generated and the monitoring tool determines the profile of these traffic streams at dedicated points of observation. The QoS parameters are detected by a comparator check of traffic streams at the measurement point(s) and a reference point. This scenario can be extended in combination with artificial load or real traffic in the background.
Generator
b

b
3.3. Test Applications

Monitor

3.2. Test Methods

Focusing on IP over ATM implementations, an overview on different methods to test QoS in ATM networks will be presented [lo] [ll]. The basic configurations for this measurements are: Passive monitoring of real data flows; a nonintrusive method. Here the monitoring tool determines the profile of the traffic streams at dedicated points of observation and checks the results against the traffic contract or specification. This can be done in combination with an observation of resource reservatiodsignalling messages (RSVP), if required. Another possibility is to request the traffic contract settings from a router. On TCP flows, packet loss can be detected by observing the number of retransmissions and by checking sequence number errors. Figure 6: Generic scenario for active generation of test traffic.

The general testing configurations have to be applied when considering IP QoS measurements, taking the three target groups into account mentioned above. For this purpose, two different types of test applications can be identified: Testing the end-to-end QoS from either the users or the servicehetwork providers perspective, e.g. when running an real-time application over a network cloud. In this scenario the user is interested in performance parameters like throughput, delay, jitter and loss on packet level, which the servicelnetwork provider has to guarantee. Testing specific QoS mechanisms (e.g. queuing or traffic shaping) in a system of one or more networking devices (e. g. network nodes). In this case e.g. the relation of low and high priority traffic to congestion or

21 5

the mechanism of Token Bucket Traffic Shaping can be exammed by also measuring the performance parameters. This scenario is dedicated to manufacturers and servicehetwork providers, where the assurance of certain quality levels has to be evaluated. In the first approach the testing of end-to-end QoS is done by monitoring a single flow of test data (network application or foreground traffic) at the edges of the IP network (system under test - SUT). Testing of QoS mechanisms is a superset of the first approach by generating multiple different flows of test data at the ingress of a sjpecific network device (SUT). The forwarded packet flows are observed at the egress. For simulating different load conditions additional background traffic is used.

Delay The time needed to send a packet from a point A to another point B over a link. In this case time counts when the first bit has left A until it arrives at B. PDU transfer delay is determined by subtracting the timestamps of identical packets captured at two measurement points. To avoid failures caused by misinsertion, the sequence number (SNR) of the frames has to be checked. For this purpose a flow containing SNRs is necessary (e.g. TCP). Jitter The variation of the delay between point A and another point B in a network under constant load. Jitter is determined by calculating transfer delay for packets captured at two measurement points. To avoid failures caused by misinsertion, the SNR of the frames has to be checked (see also Delay).

3.4. Test Entities and IP QoS Parameters

For testing IP QoS parameters, several test entities are necessary (see 3.2 Testing Methods). The sender introduces thl- test traffic which can either be caused by a network application or a special test traffic generator (TANYA). The receiver consumes the test traffic and responds if necessary. This entity is realized by corresponding network applications (e.g. video conferencing), hosts (e.g. ping, netperf) or can even be dropped in some cases. The monitor captures data flows at dedicated points of observation (PCOs) for further analysis. A g(sneratorcan be applied to adjust certain load conditions. A pre-condition for these examinations is, that the clocks of the measurement devices capturing the data flows at different PCOs must be synchronized. As stated earlier, our approach to test IP QoS is based on the measurement of the following parameters which influence the quality of data transfers: throughput, delay, loss, jirteK Their definition [I21 and the requirements to determine these values will be discussed shortly.

Loss
Percentage of frames that should have been forwarded from point A to point B in a network under determined load condition, but were not forwarded due to lack of resources or misbehavior. Loss is determined by comparing packets captured at two measurement points.

3.5. SW/HW Extensions for Testing I QoS F

The FAST ATh4 Tool Set allows easy implementations of active and passive components for measurement tasks. Different types of data flows can be constructed in real time, or off-line by using a cell file compiler. Sending the traffic with respect to external timestamps a test traffic with a defined traffic profile can be generated. For precise results (80 ns resolution) we decided to work with our TANYA ATM testing hardware rather than using a standard ATM interface card. The necessary SW/HW extensions for the modular testing architecture in order to test IP QoS will be explained in detail. For generating IP background traffic, a Burst Generator configuration for TANYA was implemented. By means of this functionality, a certain traffic pattern can be loaded onto the testing hardware which will be continuously send to the network (up to OC3 rate) without causing additional performance load for the tester. The generation of background load on IP level requires a cell file which represents IP frames. The included cell-timestamps are considered by the cell file controller on TANYA which determine the transfer rate of the packets. For generating this cell files a packet compiler - c$pgeii was developed. Apart from setting the transfer rate, this

Throughput
The maximum data transfer rate at which none of the offered frames are dropped between measurement point A and rne,asureinent point B in a network. For this purpose the transfer rate is calculated according to the timestamps of packets captured at any measurement point and throughput is determined. The given definition of throughput corresponds with lossless throughput of the ATh4 Forum [ 131, where basically three different types are distinguished: lossless throughput, peak throughput and full-load throughput.

216

tool allows to build traffic streams with specific characteristics on both frame and cell level (e.g. source/ destination IP address, VPWCI). When monitoring with TANYA the resolution of 80 ns for received ATM cells is also applied to IP packets after AAL-reassembly. A newly created packet file tool pjipperf calculates the IP QoS parameters delay, jitter and loss by comparing two IP flows captured at different measurement points. Filtering of incoming IP traffic is done at ATM level. Therefore, the test flows have to be assigned to unique V P W C I connection parameters.

background load. Every measurement began with starting the traffic generator, capturing the traffic at both monitor points (M 1, M2) for 30 seconds and an off-line analysis of the performance parameters. The background load was routed to a well known address on an output port of the router. The audio application generates a uniform stream of data, with IP packets of 200 bytes every 20 ms. The following figure shows the cell interarrival time distribution of the generated ATM flow: cells belonging to a packet appear as bursts at line rate, the gap between packets of 20 ms is visible.
FAST ATM IAT
"iai-a

4. Practical Measurements
The IP QoS measurements described in this section represent practical experiments carried out with the TANYA ATM test interface and the extended FAST ATM Tool Set. For this reason an IP over ATM test experiment has been set up at GMD FOKUS. The aim of the experiment was to demonstrate and to verify the TANYA testing hardware in combination with the extended FAST tool set. In this paper we intend to present actual test results for the following configuration.

-I 1
Load Generator Audio Sender

tirnc I s

0.001

0.01

lPoA Testbed

Figure 8: ATM cell interarrival times.

r,-

- -,i
I System under Test I I
Audio Receiver

8I
M1
A

'

??%T
L - - - A

Monitor

The IP QoS tests consist of measuring loss, delay and jitter introduced by the network (Router and Switch) under different load conditions. The variation of load conditions include: different load conditions (0, 25, 50, 75 and 100 Mbps) with traffic similar to foreground traffic (packets of 200 byte), variable packet sizes at constant bandwidth for the background traffic (at 50 Mbps where the overload condition starts) and different traffic profiles of the background load, changing the gap (interval) between bursts.

Figure 7: IP QoS Measurement scenario with TANYA. Sender and receiver are UNIX workstations with standard ATM interface cards. They are located in two different IP subnetworks which are connected over a CISCO 7000 router. An ATM switch (FORE ASX-200BX) connects the end systems with the router via PVCs. For the test traffic, we decided to use mostly an audio application (MINT) [ 141. Audio applications are very sensitive to loss and the quality can be experienced directly by listening. Additional tests were also carried out using ping and netperf for generating test traffic. The testing part of the experiment is built by two SUN workstations equipped with TANYA test interfaces: the monitor system uses two TANYA cards, the traffic generator systems uses one TANYA to generate

Throughput I Loss Throughput is determined on ATM level and therefore the bandwidth was measured before (Ml) and after (M2) the system under test (SUT). By increasing the load we detected first packet loss at 46 Mbps. As a typical example, Figure 9 displays the bandwidth of the audio application at M1 (bw-in) and at M2 (bw-out). Packet loss of 16% on the audio stream was caused by an overload condition. The router was stressed with background traffic at 50 Mbps for this test. The diagram shows the bandwidth at a graphical resolution of 0.5 s, the accuracy of the measurements here is defined by the

217

resolution of the hardware clock (80 ns).

FAST ATM BW

variations increases in overload condition

"bw.,*"
"bw-out"

Figure 9: ATM bandwidth.

Delay For the delay measurements the test traffic was monitored at two points (before and after the SUT). The packet transfer delay was calculated from the packet arrival times at the two monitor points ( M l , M2). The test should experience the impact of different background b a d s to the audio stream as described above.
IPaos - transfer dclw drrmbutmn

5. Conclusion
We have discussed IP QoS testing methods and presented a suitable test and measurement system. The modular system offers efficient soft- and hardware components necessary for evaluating IP QoS parameters in ATM networks. The IP QoS testing framework introduces the following key features: An active and a passive method for testing IP QoS. A highly flexible system architecture with open interfaces, which enables scalable functionality. A powerful but cost effective testing system by using general purpose workstations, equipped with special extension cards designed for testing purposes. The testing hardware represents a compromise between low cost and high system performance. The cell file controller, a core component of the TANYA development, runs at OC3 rate and supports hardware filters on the receiver side and traffic generators in the transmit path. A burst generator configuration allows repeated transmission of IP packets synthesized on cell level. Apart from B, capturing small bursts in high speed FIFOs (128 k ) the sustainable data rate between the TANYA card and the workstation is restricted to 45 Mbps. A PCI host interface is currently under development to overcome this limitation in order to enable data capturing at full link rate (155 Mbps). The FAST tool set complements the open structure with a variety of software modules, which allow the composition of complex measurement scenarios and the visualization of test results. According to the system performance, the evaluation of IP flows is done off-line. This is done by a comparator check of captured packets and delay calculation. The practical measurements have shown that the modular

,MC

IL u

Figure 10: Packet Transfer Delay. Figure 10 shows that the packet transfer delay as a function of occurrence. The delay varies around 220 ps if there i:; no lor only little background load (25 Mbps). When the router is overloaded (50 Mbps) the delay increases dramatically about one millisecond. By incrementing the background load up to 100 Mbps the delay increases only slightly. Jitter For the jitter measurements a constant packet stream of 5 Mbps was sent from sender to receiver. The test were performed without background load and with a background load of 50 Mbps, leading to an overload situation. Figure 11 shows that packet transfer delay

218

ATM test and measurement system is applicable to evaluate IP QoS over ATM. Based on soft and hardware extensions we were able to determine quality and performance parameters like delay, jitter, throughput and loss on packet level in a highly accurate manner. Future work on the IP QoS testing framework will consider the development of hardware components supporting filtering and flow generation on IP level as well as concepts for distributed measurements. With the availability of routers supporting DiffServ, testing experiments to evaluate QoS mechanisms will be set up in the near future.

6. References
Enrico Basso, Petr Bocek: IP Services in the Telecommunication Networks, Proceedings of ATM Technology Users Symposium-ATMTU99,Kosice, Slovak Republic, Feb. 1999, pp.260-265. ITU-T Recommendation 1.361, Version 11/1995: B-ISDN ATM Layer Specification. ATM Forum: UTOPIA, An ATM Interface Specification, Level 1, Version 2.01, 1994. McCanne, Jacobson: The BSD Packet Filter: A New Architecture for User-level Packet Capture, Lawrence Berkeley Laboratory, 1992. HP BSTS: http://www.tmo.hp.com/tmo/pia/bsts/

PIATopEnglishhsts-index.htm1
TANYA: http:Nwww.fokus.gmd.de/tip/tanya Ptolemy: http:Nptolemy.eecs.berkeley.edu/ Paul Ferguson, Geoff Huston: Quality of Service Delivering QoS on the Internet and in Corporate Networks, Wiley Computer Publishing, New York, USA, 1998. David Ginsburg: ATM Solutions for Enterprise Internetworking, Addison-Wesley,Harlow, England, 1996. 01 Robert W. Buchanan, Jr.: The At of Testing r Network Systems, Wiley Computer Publishing, New York, USA, 1996. [ 1 11 S. Bradner, J. McQuaid: Benchmarking Methodology for Network Interconnect Devices, Request for Comments 1944, May 1996. [ 121 S. Bradner: Benchmarking Terminology for Network Interconnect Devices, Request for Comments 1242, July 1991. [ 131 ATM Forum: Performance Testing Specification, Version 1.0, 1996. [ 141 Dorgham Sisalem, Henning Schulzrinne: The Multimedia Internet Terminal, Journal of Telecommunication Systems, Vol. 9, No. 3, September 1998, pp. 423-444.

219