6.

Software testing

6.1. Testing in general

6.1.1. Motivation
Testing
Testing isis the
the process
process of
of
executing
executing aa program
program with
with the
the
intent
intent of
of finding
finding errors.
errors.
Glen
GlenMyers
Myers

CPSC 333: Foundations of Software Engineering

Limits of software testing

Good testing will find bugs
Good testing is based on requirements,
i.e. testing tries to find differences between
the expected and the observed behavior of
systems or their components
BUT: Testing can only prove the presence of
bugs - never their absence

CPSC 333: Foundations of Software Engineering

Exhaustive testing
There are 1014
possible paths!
If we execute one
test per millisecond,
it would take 3.170
years to test this
program!!
Out of question
loop < 20x

CPSC 333: Foundations of Software Engineering

Testability =
how easy can a program be tested
Features that help/influence testing:
operability - it operates cleanly
observability
the results are easy to see
distinct output is generated for each input
incorrect output is easily identified

controllability
processing can be controlled
tests can be automated & reproduced

decomposability - software modules can be tested

independently
simplicity - no complex architecture and logic
stability - few changes are requested during testing
understandability - program is easy to understand

CPSC 333: Foundations of Software Engineering

Who tests the software

developer

independent tester

Understands the system

but, will test gently
and, is driven by
delivery

Must learn about the system,

but, will attempt to break it
and, is driven by quality

CPSC 333: Foundations of Software Engineering

6.1.2. Test cases

Describe how to test a system/module/function
Description must identify
system state before executing the test
function to be tested
parameter values for the test
expected outcome of the test
Some automation should be possible after
defining them!

CPSC 333: Foundations of Software Engineering

Test case examples

system state before
executing the test

ResourcePool is not empty

function/method to be
tested

anAgent is in the
ResourcePool

expected outcome of the

test

ResourcePool is not empty

function to be tested

removeAgent(anAgent)

removeAgent(anAgent)

parameter values for the

test

system state before

executing the test

ResourcePool =
ResourcePool\anAgent

parameter values for the

test

anAgent is in NOT the

ResourcePool

expected outcome of the

test

AgentNotFoundException is
thrown

CPSC 333: Foundations of Software Engineering

Test case design

Bugs lurk in corners
and congregate at
boundaries
Boris Beizer

OBJECTIVE

to uncover errors

CRITERIA

in a complete manner

CONSTRAINT

with a minimum of effort and time

CPSC 333: Foundations of Software Engineering

Adequate set of test cases

Test the interface of the components
all public methods
Is one test case per method enough?
Usually no!
So, how many do we need?
Depends on method!

CPSC 333: Foundations of Software Engineering

6.1.3. Black box vs. white box testing

Black box testing
requirements
output
input
events
CPSC 333: Foundations of Software Engineering

White box testing

our goal is to
ensure that all
statements and
conditions have
been executed at
least once...

CPSC 333: Foundations of Software Engineering

Why bother with white box

testing?
Black box testing:
Requirements fulfilled
Interfaces available and working
But what about
the internal structure of a component,
interactions between objects?
white box testing
Also: if black box testing finds error, locating it is
easier with additional white box testing!
CPSC 333: Foundations of Software Engineering

White box testing:

Why cover all paths?
logic errors and incorrect assumptions are
inversely proportional to a paths execution
probability
we often believe that a path is not likely to be
executed; in fact, reality is often counter
intuitive
typographical errors are random; its likely that
untested paths will contain some

CPSC 333: Foundations of Software Engineering

Guaranteed cover:
Exhaustive testing
But:
There are 520=1014
(approx.) possible
paths!

If-then-else

loop < 20x

CPSC 333: Foundations of Software Engineering

Better:
Selective testing
a selected path

loop < = 20x

CPSC 333: Foundations of Software Engineering

6.1.4. Path coverage testing

1) Derive a logical complexity measure
2) Use it to define a basis set of execution paths
First, we compute the cyclomatic complexity
V(G):
number of transitions - number of nodes +2
In this case, V (G) = 4
V(G) provides an upper bound of tests that
must be executed to guarantee coverage of all
program statements

Cyclomatic complexity
A number of industry studies have indicated that
the higher V(G), the higher the probability of
errors.
modules

V(G)
modules in this range are
more error prone
CPSC 333: Foundations of Software Engineering

Path coverage set

Cyclomatic complexity defines the number of
independent paths in the basis set
Path coverage set = set of paths that will
execute all statements and all conditions in a
program at least once
Goal: Define test cases for basis set
Path coverage set is not unique!

CPSC 333: Foundations of Software Engineering

Path coverage testing

Next, we derive the independent
paths:

Since V(G) = 4, there are four paths

2
4

3
5

Path 1:

1,2,3,6,7,8

Path 2:

1,2,3,5,7,8

Path 3:

1,2,4,7,8

Path 4:

1,2,4,7,2,47,8

Finally, we derive test cases to

exercise these paths, i.e. choose
inputs that lead to traversing the
paths

Documenting test cases

Name
Number
Description of system state before running the
test case
Values for the inputs
Expected outputs
Short description (if needed)

CPSC 333: Foundations of Software Engineering

Path coverage testing - remarks

you dont need a flow chart,
but the picture will help
when you trace program
paths
each connection between
boxes counts as 1 transition
path coverage testing
should be applied to critical
modules
Connects four boxes
counts as three transitions

6.1.5. Loops
Cornerstone of every program
Loops can lead to non-terminating programs

CPSC 333: Foundations of Software Engineering

Loop testing
simple loop

nested
loops

concatenated
loops

unstructured
loops

CPSC 333: Foundations of Software Engineering

Loop testing: simple loops

Minimum conditions - simple loops
1. skip the loop entirely
2. only one pass through the loop
3. two passes through the loop
4. m passes through the loop m < n
5. (n-1), n, and (n+1) passes through the loop
where n is the maximum number of
allowable passes
CPSC 333: Foundations of Software Engineering

Nested loops
Just extending simple loop testing:
number of tests explodes
Reduce the number of tests:
start at the innermost loop; set all other
loops to minimum values
conduct simple loop test; add out of range
or excluded values
work outwards while keeping inner nested
loops to typical values
continue until all loops have been tested
CPSC 333: Foundations of Software Engineering

6.1.6. Black box testing

requirements
output
input
events
CPSC 333: Foundations of Software Engineering

Categories of errors in
black box testing
incorrect or missing functions
interface errors
errors in data structures or external database
access
performance errors
initialization and termination errors

CPSC 333: Foundations of Software Engineering

Example
abs(integer x)
Return x if x >= 0
Return -x if x<0
The concrete value of x does not matter!
Basis for Equivalence Testing

CPSC 333: Foundations of Software Engineering

6.1.7. Equivalence class testing

Equivalence partitioning
mouse picks on menu

user
queries
numerical
data
Partitioning is based
on input conditions

output
format
requests
responses
to prompts

command key input

CPSC 333: Foundations of Software Engineering

Equivalence classes (1)

Valid data

user supplied commands

responses to system prompts
file names
computational data
physical parameters
bounding values
initiation values
output data formatting commands
responses to error messages
graphical data (e.g., mouse picks)

CPSC 333: Foundations of Software Engineering

Equivalence classes (2)

Invalid data

data outside bounds of the program

physically impossible data
proper value supplied in wrong place

CPSC 333: Foundations of Software Engineering

Defining equivalence classes

Input condition is a range: one valid and two
invalid classes are defined
Input condition requires specific value: one
valid and two invalid classes are defined
Input condition is boolean: one valid and one
invalid class are defined
Then define one test case for each equivalence
class

CPSC 333: Foundations of Software Engineering

Automating software testing

Manual software testing is time consuming
Software testing has to be repeated after
every change (regression testing)
Write test drivers that can run automatically
and produce a test report

CPSC 333: Foundations of Software Engineering

Testing vs. fault tolerance design

and implementation
Testing: Find errors and fix them at appropriate
level and in appropriate components
Fault tolerant design: Assume that errors occur
during execution and try to recover from them
(also during execution)
For complex and important systems we need
both!

CPSC 333: Foundations of Software Engineering