Caution Signs Along the Road

There will be jolts and delays along the way for electronic commerce: congestion is the most obvious challenge.

Performance Analysis

= Analysis + Computer System

Introduction
Performance Analyst = Mathematician + Computer Systems Person

How to describe Performance of a 'System' ?

Resource Utilization: For any system to consume an input and produce the designated output, certain resources would be required. The amount of resources consumed by the system during processing that request, defines the resource utilization. There can be different resources factored in, such as processor, disk (i/o controller), memory etc. Utilization of 80% is considered an acceptable limit. Normally utilization of 70% warrants ordering of additional hardware.

Response Time: Response time specifies the time taken by the system to respond back with the expected output. For an enterprise application Response time is defined as the time taken to complete a single business transaction. It is usually expressed in seconds

Throughput: Throughput refers to the rate which the system churns expected outputs when the designated input is fed in the system. In other words, for an Enterprise Application, throughput can be defined as the total number of business transactions completed by the application in unit time (per second or per hour).

How to DIVIDE Performance of a 'System' ?

1. 2. 3. 4. 5. 6. System Config Topology Network I /O Disk I/O Database Tuning OS

System

Application

Machine
1. 2. 3. 4. 5. App Design App Server Tuning Drivers Persistence Clustering 1. 2. 3. 4. 5. 6. JVM Selection JVM Tuning JVM Internals JIT Compilation Cache Architecture SMP Scaling

Performance Management Process Flow

These steps provide a highlevel process flow for performance management:

Develop a Management Concept of Operations

Define the Required Features: Services, Scalability and Availability Objectives Define Performance SLAs and Metrics

Measure Performance
Gather Baseline Data Measure Availability Measure Response Time Measure Accuracy Measure Utilization Capacity Planning

Perform a Proactive Fault Analysis

Use Thresholds for Proactive Fault Management Management Implementation Operation Metrics

Performance Testing : A Management Perspective

Requirement Analysis
Architecture & Design Coding

Development and Maintenance

Pref. Requirements Analysis

Modeling & Engg Design, Sizing Code Optimization Pref. Testing, Tuning Monitoring & Capacity Planning

Performance Management

Pre-production Certification of system and application performance Baseline, Load, Volume, Stress, Scalability, Endurance / Stability, Reliability Refine Analytic Models based on actual measurements Capacity Planning

Testing Production

Performance Testing : An Engineering Perspective

Requirement Analysis
Architecture & Design

Technology Stack Performance Evaluation Product Benchmark

Performance benchmark of a technology stack (combination of h/w + s/w platform) Baseline the performance levels for a product / prototype

Component Performance Test

Performance Test of a Component Capacity Planning for shared components

Construction

Testing

Performance Test for Application Release

Production

Load Test realistic loads Stress Test to determine breakpoint Volume Test over growing data volumes Sensitivity Test varying parameters Scalability Test across scaled-up h/w Endurance Test long periods

PERFORMANCE TESTING PHASES

UNIT TEST

INTEGRATION TEST

INTEGRATION LOAD TEST

Code Profiler

Memory Profiler Coverage Profiler

API UI Services UI

API

Services

Test Results for Analysis

4 Internet

PRODUCTION STAGING TEST

Tester Test Scripts

Isolated Network

PRODUCTION STAGING LOAD TEST

Performance Improvement Procedure

START

Understand System

Analyze Operations

Formulate Improvement Hypothesis Analyze Cost Effectiveness Test Specific Hypothesis

Unsatisfactory Invalid
Test Effectiveness of Modification Implement Modification

N o n e

STOP

Satisfactory STOP

Questions
Why is saturation occurring? In which parts of the system (CPU, disks, memory, queues) will a transaction or job be spending most of its execution time at the saturation points? Which are the best cost-effective alternatives for avoiding (or, at least, delaying) saturation?

11

Common Mistakes in Capacity Planning

1. No Goals 2. Biased Goals 3. Unsystematic Approach 4. Analysis Without Understanding the Problem 5. Incorrect Performance Metrics 6. Unrepresentative Workload 7. Wrong Evaluation Technique 8. Overlook Important Parameters 9. Ignore Significant Factors 10. Inappropriate Experimental Design 11. Inappropriate Level of Detail 12. No Analysis 13. Erroneous Analysis 14. No Sensitivity Analysis 15. Ignoring Errors in Input 16. Improper Treatment of Outliers 17. Assuming No Change in the Future 18. Ignoring Variability 19. Too Complex Analysis 20. Improper Presentation of Results 21. Ignoring Social Aspects 22. Omitting Assumptions and Limitations

12

Capacity Planning of Steps

1. 2. 3. 4. Instrument the system Monitor system usage Characterize workload Predict Performance under different alternatives 5. Select the lowest cost, highest performance alternative
13