Software Testing

Mohammed A. Ramadan

Computer Science Department Modern Academy in Maadi

Agenda
Testing -What Is it ?

Testing lifecycle.
Types of Testing

Test Strategy
Orthogonal Arrays Testing Strategy

(OATS)

Software Testing

22-Apr-12

Testing What Is it ???

An activity in which a system or component is executed under specified conditions, the results are observed or recorded, and an evaluation is made of some aspect of the system or component
From IEEE Std 610 Software Engineering Terminology: test

The process of executing computer software in order to

determine whether the results it produces are correct

Glass 79

The process of executing a program with the intent of finding

errors, Myers 79 Testing is the measure of software quality, Hetzel 85

3
Software Testing

22-Apr-12

Objectives of Testing
What are we looking for? Errors, faults, failures and incidents. Defect, Problem,

Incident, Anomaly, Variance, Inconsistency, Feature, Bugs !

A test is the act of exercising software with test cases with

an objective of Finding failure Demonstrate correct execution

Goal To find if the product is

Not operating as Intended

4
Software Testing

22-Apr-12

Matrix of Specification and Software

Specification Should do Shouldnt do No mention but should No mention Software Doesnt Does Doesnt Does

Non-Functional

Hard to use, Complex, slow

Software Testing

22-Apr-12

Source of Defects
10% 7%
Code
Other

Requirements Design

27%

56%

Source: Software Testing in the Real World Edward Kit, 1995

6
Software Testing

22-Apr-12

Cost of Repair
Coding Testing Maintenance

Defects removed

Repair Cost 1

Repair Cost 4

Repair Cost 10

PROJECT PHASES

The sooner you find bugs, the cheaper

Software Testing

22-Apr-12

Testing lifecycle
Development Lifecycle
Project Planning Requirements Capture Analysis & Design Implementation

Build 0 Testing in the V Model Approach

Build 1

Build 2

Test Lifecycle

Plan Test

Design Test

Setup Test Bed

Execute Test Evaluate Test

Defect Tracking/Enhancement Request

8
Software Testing

22-Apr-12

Testing lifecycle ,contd

Requirements Capture
The Scenarios Design and the Test Case Development could normally start in parallel with the Development Cycle. Test Execution Synchs up with the Development Cycle during the functional Testing Phases

Analysis
Test planning & Scenario Design

New Version

Test Case Development

Test Execution

Defect Fixing Cycle Defects

Test Cycle Closure

Test Result Analysis

Software Testing

22-Apr-12

Testing in the V Model Approach

Requirements (Business Needs) Requirements

Acceptance Testing
System Testing

Write tests

(Business Functions) Design

Run
Integration Testing Unit Testing

tests

Code

Verification
All QC activities through out the life cycle that ensure that interim deliverables meet their input specification.
10
Software Testing

Validation
The test phase of the life cycle which ensures that the end product meets the users needs.
22-Apr-12

Types of Testing

Software

Software

Black Box Testing

Static Dynamic

White Box Testing Static Dynamic

11

Software Testing

22-Apr-12

Types of Testing
Black Box
Dynamic Static Dynamic

White Box
Static Review & Analysis Peer Review Formal Review

Structure Testing Unit Testing

Test to Pass Test to Fail Equivalence Partitioning Data Testing

Specifications

Data Coverage Data Flow Code Coverage

Walkthrough Inspections Coding Standards Generic Code Review

Line Coverage
Boundary Conditions Invalid Data State Transitions Branch Coverage Condition Coverage

Data Reference Error Control Flow Errors

Regression
12

Computation Errors Input/Output Errors 22-Apr-12

Software Testing

Black Box Testing

Software

Static Black Box Testing

Testing the specification

Dynamic Black Box Testing

Behavioral Testing
Testing a running program without any

knowledge of how the software behaves.

13

Software Testing

22-Apr-12

White Box Testing

Static White Box Testing Code review and analysis

Software

(1)Formal Reviews

(2)Peer Reviews (3) Walkthroughs

Dynamic Black Box Testing

Structural testing
using information from what the code does and how it works

to determine what to test, what not to test.

Unit Testing
Data Coverage

Divide the code into data and its states Track a piece of data completely through the software

Data flow

Code coverage

Program and Line Coverage Branch Coverage Condition Coverage

14

Software Testing

22-Apr-12

Other Types of Testing

User Interface Testing Test the User Interface for operability, ease of use Performance Testing the performance of the computations Load Testing the performance of the system under varying load

conditions Stress Testing the performance and accuracy of system under resource crunch or any other kind of stress Volume Testing the accuracy of system when system has a high volume of data
15
Software Testing

22-Apr-12

Other Types of Testing ,contd

Configuration Testing the system under various hardware/software configurations Compatibility Testing the system for ability to interact with other system Installation Testing the installation of system under one or more configurations Documentation Testing the system for documentation clarity and ease of

apprehension Ad-Hoc Testing Randomly testing the system for errors with no background on the system

16

Software Testing

22-Apr-12

Why Have a Test Strategy?

Test Strategy
Describes the overall approach that the test team will

use to test, in each phase and in all.

A Test Plan
The game plan to implement what is to be tested, when, how, how long and till what happens?

Effort Estimation
Test Strategy helps the overall test effort estimation

17

Software Testing

22-Apr-12

What to Consider?
Product Requirements Nature of product Product Lifecycle Milestones Repeatability considerations Environment

Feedback
Cost
18
Software Testing

22-Apr-12

Optimal Testing
Number of Missed Bugs

Cost of Testing

Optimal Amount of Testing Over Testing Under Testing

Amount of Testing

19

Software Testing

22-Apr-12

20

Software Testing

22-Apr-12

Orthogonal Arrays Testing Strategy (OATS)

(OATS) is a systematical, statistical way of testing pair-

wise interactions by deriving suitable small set of test cases from a large number of scenarios.
can be used to reduce the number of test combinations

and provide maximum coverage with a minimum number of test cases.

(OATS) utilizes an array of values representing variable

factors that are combined pair-wise rather than representing all combinations of factors and levels.
21
Software Testing

22-Apr-12

Overview
Orthogonal arrays are two dimensional arrays of

numbers by choosing any two columns in the array you receive an even distribution of all the pair-wise combinations of values in the array.

22

Software Testing

22-Apr-12

Overview
An L9 (34) orthogonal array with 9 runs, 4 factors, 3 levels, and strength of 2.
An L9 (34 ) orthogonal array with 9 runs, 4 factors, 3 levels, and strength of 2.

Runs: the number of rows in the array. This directly translates to the number of test cases that will be generated by the OATS technique. Factors: the number of columns in an array. This directly translates to the maximum number of variables that can be handled by this array. Levels: the maximum number of values that can be taken on by any single factor. An orthogonal array will contain values from 0 to Levels-1.

Strength: the number of columns it takes to see each of the Levelstrengh possibilities equally often.
Orthogonal arrays are most often named following the pattern L runs (LevelsFactors ).
23 22-Apr-12

Software Testing

Simple OAT Example

Consider three parameters : A, B, and C.
Parameters A, B, and C each have positive values :

1, 2, and 3.
Testing all combinations of the three parameters would

involve executing a total of 27 test cases.

24

Software Testing

22-Apr-12

Simple OAT Example (contd)

With the way programming works, a fault will most likely

depend on the values of two parameters, not three.

The fault might occur for each of these three test cases:

A=1, B=1, C=1, A=1, B=1, C=2, and A=1, B=1, C=3.
Not necessary to run all 27 scenarios, 9 test cases is

suffice.
The 9 scenarios outlined in Table 1 (on the next screen)

address all possible pairs within the three parameters.

25

Software Testing

22-Apr-12

Pair-wise Combination of Parameters

All possible pair-wise combinations between parameters A and B, B and C, and C and A .

26

Software Testing

22-Apr-12

Example 2

Orthogonal Array has 3 columns, representing 3 variables. Utilize a three level-alphabet Red, Green, and Blue. Next, organize combinations into a table (next slide).

27

Software Testing

22-Apr-12

Example 2 (contd)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 All Combinations for Three Variables of Three Levels Each A B Red Red Red Red Red Red Red Green Red Green Red Green Red Blue Red Blue Red Blue Blue Red Blue Red Blue Red Blue Green Blue Green Blue Green Blue Blue Blue Blue Blue Blue Green Red Green Red Green Red Green Green Green Green Green Green Green Blue Green Blue Green Blue C Red Green Blue Red Green Blue Red Green Blue Red Green Blue Red Green Blue Red Green Blue Red Green Blue Red Green Blue Red Green Blue

28

Software Testing

22-Apr-12

Example 2 (contd)
For each pair of columns, AB, AC, and BC, each pair of colors

appears exactly three times.

To save testing effort, the appearance of each pair has been reduced

to once.
All-Pairs Array, Three Variables of Three Levels Each
A 2 4 9 12 14 16 19 24 26 Red Red Red Blue Blue Blue Green Green Green B Red Green Blue Red Green Blue Red Green Blue C Green Red Blue Blue Green Red Red Blue Green

29

Software Testing

22-Apr-12

Advantages of OATS
Utilizes variable pair combination Provides uniformly distributed coverage Useful for integration testing of software components Useful for testing configurable options on an application web page allowing font style background color page layout selections

30

Software Testing

22-Apr-12

Advantages of OATS
Conventional Test Cases: Variables:3 Orthogonal Test Cases Variables:3 Input: 4 Possible cases: 9 Variables: 3

Input: 4 Possible cases: 81=34 Variables: 3 Input: 5 Possible Cases: 243 = 35 Variables: 5 Input: 5 Possible Cases: 3125 = 55

Input: 5 Possible Cases: 11 Variables: 5 Input: 5 Possible Cases: 21

31

Software Testing

22-Apr-12

Disadvantages of OATS
Can only be applied at the initial stage of the product/process

design system.
There are some situations whereby OA techniques are not

applicable, such as a processes involving influencing factors that vary in time and cannot be quantified exactly.
Can be overwhelming to use the first time; improves with use

on each project.
Arrays can be difficult to construct.

32

Software Testing

22-Apr-12

Questions

33

Software Testing

22-Apr-12

Thank You
34
Software Testing

22-Apr-12