Overview

Structural Testing
Introduction General Concepts
Flow Graph Testing
DD-Paths
Test Coverage Metrics
Basis Path Testing
Guidelines and Observations
Data Flow Testing
Hybrid Methods
Retrospective on Structural Testing

1
CA267 Ray Walshe 2015
 An Overview of Path Testing
Coverage Criteria
Basic Measures
Statement Coverage
Decision Coverage
Condition Coverage
Multiple Condition Coverage
Condition/Decision Coverage
Modified Condition/Decision Coverage
Path Coverage

Other Measures
Function Coverage
Call Coverage
Loop Coverage

Reference for this lecture: http://www.bullseye.com/coverage.html [1]

2
CA267 Ray Walshe 2015
 Coverage Analysis
The objectives of coverage analysis are:
To determine gaps in test cases (i.e. parts of the program that are not
tested)
To establish a quantitative measure for the testing efficiency
To identify additional tests that may be needed to eliminate gaps

Coverage analysis assists on evaluating the efficiency of the test cases,

and not the quality product itself. Of course the objective is through
testing you can eliminate faults and increase the quality of the product at
the end.

As we have seen there are several coverage criteria to identify paths in

the CFG and consequently generate test cases that cover these paths.

3
CA267 Ray Walshe 2015
 Coverage Analysis and Test Plans
A good test plan utilizing coverage analysis must:

Choose the appropriate coverage methodology or criterion (i.e.

statement coverage, decision coverage, condition coverage etc.)
Establish minimum acceptable percentage level of coverage
Utilize more than one coverage criterion

Eventually, it is not a good idea to rely only on code

coverage. Other techniques are useful as well (i.e. BVA,
ECT, DTT).

4
CA267 Ray Walshe 2015
 Statement Coverage
The measure reports on the percentage of executable
statements of the code exercised by a set of test cases. Also
known as basic bloc coverage or segment coverage.

The major advantage of this measure is that it can be applied

directly to object code and does not require processing
source code. Performance profilers commonly implement
this measure.

The major disadvantage of statement coverage is that it is

insensitive to some control structures
5
CA267 Ray Walshe 2015
 Statement Coverage Examples
if(A) then int* ptr = NULL;
F1(); if (B)
F2(); ptr = &variable;
*ptr = 10;
Test Case: A=True
Statement Coverage Test Case:
achieved B=True
Statement Coverage
Achieved

Problem : if B is false the code

will fail with a null pointer
exception.
6
CA267 Ray Walshe 2015
 Statement Coverage Comments
Statement coverage does not report whether loops reach their
termination condition - only whether the loop body was executed.

Since do-while loops always execute at least once, statement coverage

considers them the same rank as non-branching statements.

Statement coverage is completely insensitive to the logical decisions.

One argument in favor of statement coverage over other measures is that

faults are evenly distributed through code; therefore the percentage of
executable statements covered reflects the percentage of faults
discovered.

Statement coverage requires in most cases very few test cases to achieve.
Not acceptable to release code based on statement coverage alone.
7
CA267 Ray Walshe 2015
 Decision Coverage
Decision coverage (also known as branch coverage, all-edges coverage,
basis path coverage, decision-decision-path testing) reports whether
boolean expressions in control structures are evaluated to both true and
false values by the test cases.

Note in decision coverage we are not interested in all combinations of

True/False values of all program predicates, just that we have exercised
all the different values in our test cases.

The entire boolean expression is considered one true-or-false predicate.

Additionally, this measure includes coverage of switch-statement cases,

exception handlers, and interrupt handlers.

8
CA267 Ray Walshe 2015
 Decision Coverage Examples
if(A) if (A && (B || F1()))
F1(); F2();
else else
F2(); F3();
if(B)
F3() Test Cases for Decision Coverage:
else A=T, B=T
F4(); A=F
Problem: F1() never gets called. This
Test Cases for Decision Coverage: problem occurs in languages with
A=T, B=T short circuiting boolean operators
A=F, B=F

9
CA267 Ray Walshe 2015
 Decision Coverage Comments
This measure has the advantage of simplicity
without the problems of statement coverage.

A disadvantage may produce gaps in tested

code in programs written in languages that
have short-circuit logic operators (C, C++,
Java, etc.)

10
CA267 Ray Walshe 2015
 Condition Coverage
Condition coverage reports the true or false outcome of each boolean
sub-expression.

Condition coverage measures the sub-expressions independently of each

other.

This measure is similar to decision coverage but has better sensitivity to

the control flow.

An extension of Condition Coverage is the Condition/Decision Coverage

which is a hybrid measure composed by the union of condition coverage
and decision coverage. It has the advantage of simplicity but without the
shortcomings of its component measures.

11
CA267 Ray Walshe 2015
 Condition Coverage Examples
if(A && B) if(A && B)
F1(); F1();
else else
F2(); F2();
if(C) if(C)
F3() F3()
else else
F4(); F4();

Test Cases for Condition Coverage: Test Cases for Condition Coverage:
A=T, B=T, C=F A=T, B=F, C=F
A=F, B=F, C=T A=F, B=T, C=T
Problem: decision coverage not achieved

12
CA267 Ray Walshe 2015
 Multiple Condition Coverage

Multiple condition coverage reports whether every

possible combination of boolean sub-expressions
occurs.

The test cases required for full multiple condition

coverage of a condition are essentially given by the
logical operator truth table for the condition.

13
CA267 Ray Walshe 2015
 Multiple Condition Coverage Examples
if(A && B) // condition 1
F1();
else
F2();
if(C || D)// condition 2
F3()
else
F4();

Test Cases for MCC:

For condition 1 For condition 2

A=T, B=T C=T, D=T
A=T, B=F C=T, D=F
A=F, B=T C=F, D=T
A=F, B=F C=F, D=F
14
CA267 Ray Walshe 2015
 Multiple Condition Coverage Comments
For languages with short circuit operators multiple condition coverage is very
similar to condition coverage (but with more test cases).

For languages without short circuit operators multiple condition coverage is

effectively path coverage.

A disadvantage of this measure is that it can be tedious to determine the

minimum set of test cases required, especially for very complex boolean
expressions.

An additional disadvantage of this measure is that the number of test cases

required could vary substantially among conditions that have similar complexity.
Consider a && b && (c || (d && e)) and the ((a || b) && (c || d)) && e [1]

As with condition coverage, multiple condition coverage does not include

decision coverage.
15
CA267 Ray Walshe 2015
 Modified MCC
Also known as MC/DC and MCDC.
This measure also known as MC/DC and MCDC requires enough test
cases to verify every condition can affect the result of its encompassing
decision.
For C, C++ and Java, this measure requires exactly the same test cases as
condition/decision coverage.

Modified condition/decision coverage was designed for languages

containing logical operators that do not short-circuit.

The short circuit logical operators in C, C++ and Java only evaluate
conditions when their result can affect the encompassing decision.

16
CA267 Ray Walshe 2015
 Modified MCC Examples
To test if (A or B)
A: T F F
B: F T F

To test if (A and B)
A: F T T
B: T F T

To test if (A xor B)
A: T T F
B: T F T
17
CA267 Ray Walshe 2015
 Path Coverage
This measure reports whether each of the possible paths in each function
have been followed. A path is a unique sequence of branches from the
function entry to the exit. This measure is also known as predicate
coverage.

Since loops introduce an unbounded number of paths, this measure

considers only a limited number of looping possibilities.

A large number of variations of this measure exist to cope with loops.

Boundary-interior path testing considers two possibilities for loops: zero
repetitions and more than zero repetitions. For do-while loops, the two
possibilities are one iteration and more than one iteration.

18
CA267 Ray Walshe 2015
 Path Coverage Examples
if (A) A
Paths:
F1(); A-F1-F2-A-F3-F4
F2(); A-F2-A-F3-F4
F1
if (A) A-F1-F2-A-F4
F3(); A-F2-A-F4
F2
F4();
Problem: only two
A are feasible
A=T
A=F
F3
19
CA267 Ray Walshe 2015 F4
 Path Coverage Comments
Path coverage has the advantage of requiring very thorough
testing.
Path coverage has two severe disadvantages.

The first is that the number of paths is exponential to the number of

branches.
The second disadvantage is that many paths are impossible to
exercise due to relationships of data.

There are many variations of path coverage to deal with the

large number of paths. Examples include, n-length sub-path
coverage reports whether we have exercised each path of
length n branches. Other examples include linear code
sequence and jump (LCSAJ) coverage and data flow 20
coverage.
CA267 Ray Walshe 2015
 Loop Coverage
This measure reports whether each loop body in the
program has been executed zero times, exactly once, and
more than once (consecutively).

For do-while loops, loop coverage reports whether the body

has been executed exactly once, and more than once.

The valuable aspect of this measure is determining whether

while-loops and for-loops execute more than once,
information not reported by others measure [1].

21
CA267 Ray Walshe 2015
 Function Coverage
This measure reports whether each function
or procedure in the program has been
invoked.

It is useful during preliminary testing to

assure at least some coverage in all areas of
the software.
22
CA267 Ray Walshe 2015
 Call Coverage
This measure reports whether each function
call in the program has been executed. The
hypothesis is that faults commonly occur in
interfaces between modules.

Also known as call pair coverage.

23
CA267 Ray Walshe 2015
 Coverage Measures Comparison
Relative strengths when a stronger measure includes
a weaker measure.
Decision coverage includes statement coverage since
exercising every branch must lead to exercising every
statement.
Condition/decision coverage includes decision coverage
and condition coverage (by definition).
Path coverage includes decision coverage.
Predicate coverage (paths as possible combinations of
logical conditions) includes path coverage and multiple
condition coverage, as well as most other measures.
24
CA267 Ray Walshe 2015
 Testing Strategy (1)
Your highest level of testing productivity occurs when we
identify the most failures with the least effort.

Effort is measured by the time

Required to create test cases,
Add the test cases to your test suite and
Run the test suites.

It follows that we should use a coverage analysis strategy that

increases coverage as fast as possible. This provides the
greatest probability of finding failures sooner rather than
later.
25
CA267 Ray Walshe 2015
 Testing Strategy (2)
One strategy that usually increases coverage quickly is to first attain some
coverage throughout the entire test program before striving for high
coverage in any particular area [1]

The sequence of coverage goals listed below illustrates a possible

implementation of this strategy [1]
Invoke at least one function in 90% of the source files (or classes).
Invoke 90% of the functions.
Attain 90% condition/decision coverage in each function.
Attain 100% condition/decision coverage.

Notice we do not require 100% coverage in any of the initial goals. This
allows us to defer testing the most difficult areas and maintain high
testing productivity; by achieving maximum results with minimum effort
[1]
26
Source:
CA267 Ray Walshehttp://www.bullseye.com/coverage.html
2015 [1]
 Testing Productivity

Source: http://www.bullseye.com/coverage.html [1] 27

CA267 Ray Walshe 2015