Performance Testing Gap Analysis Case Study By V- N e s s

October 2009

V-01 October 2009

Performance Testing - Case Study

Table of Contents
1. 2. 3. 4. 5. INTRODUCTION ........................................................................................................................ 3 GATHERING INFORMATION ................................................................................................. 3 TOP-DOWN ANALYSIS ............................................................................................................. 5 FINAL RESULTS......................................................................................................................... 6 SUMMARY ................................................................................................................................... 8

V-01 October 2009

Performance Testing - Case Study

1. Introduction
One of Israel's largest banks operates a system enabling customers to trade in the stock market. The system is quite complex, interfacing with myriad applications and functioning with several infrastructure systems, including a mainframe. As one of the bank's most important systems, the trading system must not only operate continuously (24/7), but also provide a fast response time. Lack of system credibility could lead to significant customer financial loss and subsequent legal action against the bank. Underscoring the importance of the system, the bank established a special team to monitor and enhance its performance. Led by Ness Technologies load and stress experts and supported by financial and infrastructure banking personnel, the joint team defined a work methodology so as to evaluate and further improve the system.

2. Gathering Information
The team initially evaluated the system's current conditions via a two-step process:

I. System Structure Mapping

The team mapped the entire system structure including: Hardware Communications elements System-interfacing applications o Internal o External

V-01 October 2009

Performance Testing - Case Study

System Structure Diagram

Each server illustrated in the above diagram represents a server farm (i.e. many servers functioning as one large server).

II. End-to-End Load Testing

The entire system was tested as is to determine the availability, current response time and capacity (i.e. maximum number of concurrent users) for carrying out all transactions. The system's average performance for the three measured parameters was: Availability Response Time Capacity 40% 40 seconds 200 concurrent users

V-01 October 2009

Performance Testing - Case Study

3. Top-Down Analysis
Based on the initial end-to-end tests, all problematic transactions were marked. To determine the reason for the system's poor performance, correct any problems, and improve the system, the system was repeatedly tested in two ways: Lengthwise testing all transactions on an isolated architectural element Widthwise testing problematic transactions across the entire architecture The following steps were executed in each iterated test:

[1] [2]

[3] [4]

Isolating one element in the overall structure, i.e. narrowing down the tested element to a single server while leaving the remaining elements as is (i.e. at full structure) Executing capacity and response time tests on all transactions according to production distribution (i.e. number of transactions per second, number of concurrent users). The results of this step indicate whether the isolated element is creating a problem for any of the transactions Concentrating only on problematic transactions by testing their performance on the full system architecture Analyzing the results and pinpointing any required corrections

This bi-directional iterative method for testing the system (lengthwise on an isolated architectural element, widthwise across the entire architecture) ensures that all system parts will be fully corrected.
V-01 October 2009

Performance Testing - Case Study

4. Final Results
Here are samples of several test results: Response Time Developments for Transaction X on Various Test Iterations Transaction X
45 40 35 30 25 20 15 10 5 0
15 .0 9.

42 35.95

Seconds

14

15.05
ss

9.32 4.21

Balance Status

03

03

03

03

03

17 .1 1.

26 .1 1.

29 .1 0.

19 .1 1.

Date

V-01 October 2009

Performance Testing - Case Study

27 .1 1.

03

Improvement in System Response Time, Capacity, Stability and Availability

== Response time at the outset of testing iterations == Concurrent users at the outset of testing iterations == Response time at the final test iteration == Concurrent users at the final test iteration Once all tests and corrections were carried out, the system's performance improved dramatically as follows: Parameter Availability Response Time Capacity Pre-Test Performance 40% 40 seconds 200 concurrent users Post-Test Performance 99% 3-5 seconds 1,500 concurrent users

V-01 October 2009

Performance Testing - Case Study

5. Summary
In order to maintain system functionality throughout the development of multiple versions, Ness experts defined a unique procedure for executing repeated tests for each new version. Any version that failed to maintain the system's optimal response time, capacity and stability was not approved for production.

Implementation For a more in-depth look at, or assistance with implementing, Ness' methodology, please contact our performance experts at Performance-COE@Ness.com. Ness Technologies Phone: +972-3-767-5259 Fax: +972-3-769-3556 http://web.ness.com/forms/performance

V-01 October 2009

Performance Testing - Case Study