Software Testing Methods

Lesson 16

Introduction
Definition of Testing
Software Testing Methods

Software Testing
Testing is the process of exercising a
program with the goal of finding
errors prior to delivery to the end user.
Test if the actual result is the same as expected
result

What Testing Shows

errors
requirements conformance
performance
an indication
of quality

Source: Pressman

Software Testing Limitation

Software testing shows presence of

bugs, but not their absence

Source: Pressman

Software Testing Facts

Tests should be traceable to customer requirements.
Test planning can begin after requirements model is
complete.
Definition of test cases can begin after design model has
been solidified, before coding.
Testing begins in the small & progress towards the large.

Testing Facts: Exhaustive Testing not possible!

loop < 20 X

14

There are 10 possible paths! If we execute one

test per millisecond, it would take 3,170 years to
test this program!!

Source: Pressman

Selective Testing is the Solution

Selected path

loop < 20 X

Source: Pressman

Testing Facts: Who Should Test the

Software?

developer
Understands the system
but, will test "gently"
and, is driven by "delivery"

independent tester
Must learn about the system,
but, will attempt to break it
and, is driven by quality

Source: Pressman

How to Insure Testability?

What is Testability? How easily a software can be tested and
fixed.
Some of the following characteristics should be considered
during the Design phase
Operabilityensure that software operates cleanly (e.g., exit
from errors gracefully with meaningful error messages)
*Observability
Current System states and variables can be displayed during execution
Past system states and variables can be displayed (e.g., transaction
logs)

*Controllabilitysoftware states or variables can be

controlled directly by tester, thus automated testing possible
*DecomposabilityModules can be tested independently &
problems can be isolated

Testability-contd

*Simplicity / Understandability

Functional simplicity - The software is implemented with the minimum

necessary functionality to meet requirements.
Structural simplicity - Architecture modularized to limit propagation of
faults
Code simplicity code is easy to inspect and maintain

*Stabilityfew changes are requested during testing and

are controlled (e.g., the more changes, the more re-testing)

Sample Test Document Hierarchy

TEST PLAN
Test Procedure 1

Test
Case 1

Test
Case 2

Test Procedure 2

Test Procedure Y

Test
Case n

Test Plan guides the overall testing effort

Test Procedure guides testing of a set of components
(e.g., subsystem)
Test Case guides testing of a single component

What is a Good Test Case?

1. Produces high probability of finding an error or a whole
class of errors.
2. Not redundant: every test should have different purpose.
3. Maintains good balancedoing too much in a test may
mask errors.

Software Testing Methods

black-box
methods

white-box
methods

Methods
Strategies

Source: Pressman

White-Box Testing

Our goal is to ensure that

Internal operations perform according to spec
All paths and logical conditions have
been executed at least once

Source: Pressman

OO vs. Conventional Testing Methods

Following slides describe conventional testing methods
Can be used in unit testing of individual classes / objects of
systems with an OO architecture

Basis Path Testing

(Statement Testing)
Ensure that all statements in program executed at least
once.
Ensure that every condition executed on its True & False
side.
Cyclomatic Complexity Number of independent paths in
program which make up the BASIS SET.
Independent Path = Any path that introduces at least 1 new
processing statement or a new condition.
# of independent paths is the maximum number of tests that
can be conducted.

Basis Path Testing

First, we compute the cyclomatic
complexity: V(G)
number of simple decisions + 1
or
number of enclosed areas + 1
In this case, V(G) = 4

** V(G) can be computed using PDL and flow chart

(seen above)

Source: Pressman

Cyclomatic Complexity
A number of industry studies have indicated
that the higher V(G), the higher the probability
or errors.
modules

V(G)
modules in this range are
more error prone

Source: Pressman

Basis Path Testing

Next, we derive the
independent paths:

Since V(G) = 4,
there are four paths

2
3

Path 1:
Path 2:
Path 3:
Path 4:

1,2,3,6,7,8
1,2,3,5,7,8
1,2,4,7,8
1,2,4,7,2,4,...7,8

Finally, we derive test

cases to exercise these
paths.

Source: Pressman

Basis Path Testing Notes

you don't need a flow chart,
but the picture will help when
you trace program paths
count each simple logical test

as 1; compound tests count

as 2 or more

basis path testing should be

applied to critical modules

Source: Pressman

Basis Path
Testing
Example

i = 1;
total.input = total.valid = 0;
sum = 0;

2
3

DO WHILE value[i] <> -999 AND total.input < 100

6

increment total.input by 1;
IF value[i] > = minimum AND value[i] < = maximum
THEN increment total.valid by 1;
sum = sum + value[i]
ELSE skip to increment i

1. Determine the
cyclomatic
complexity

2. List the
independent
paths

ENDIF
increment i by 1;
ENDDO
IF total.valid > 0

11

10

THEN average = sum / total.valid;

ELSE average = -999;

13

12

ENDIF
END average

Control Structure Testing

Examples:
Condition Testing
Loop Testing

Condition Testing
(Type of Control Structure Testing)
Similar to statement testing, but main criterion is that ALL branches
(True & False) in each condition executed at least once.
Also verifies if each component of the condition is correct.
Simple Condition: Boolean variable or relational expression with the
following format
E1 <relational operator> E2

Compound condition.
(E1 = E2) AND (E3 <>E4)

Types of Condition Testing

Branch testing
Test True and False branches of a simple condition
Test every simple condition in a compound condition, by testing each of their True &
False branches

Domain Testing For the expression: E1 <relational operator> E2

Test that E1, E2 are valid

Condition Testing Example

If value[i] <> -999 AND total.input < 100
In Branch Testing, develop test cases that will do the
following:
value[i] <> -999

total.input < 100

Compound
Condition

Test Case 1

Test Case 2

Test Case 3

Test Case 4

In Domain Testing, test that value[i], -999, total.input, and

100 are valid, and therefore will generate expected results.

Loop Testing
(Type of Control Structure Testing)

Focuses on validity of loops

4 classes of loops:

simple
concatenated
nested
unstructured

Loop Testing

Simple
loop

Nested
Loops

Concatenated
Loops
Unstructured
Loops

Source: Pressman

Loop Testing: Simple Loops

1.
2.
3.
4.
5.

skip the loop entirely

only one pass through the loop
two passes through the loop
m passes through the loop (m < n)
(n-1), n, and (n+1) passes through the loop

where n is the maximum number of allowable passes

Source: Pressman

Loop Testing: Nested Loops

Nested Loops
1. Start at the innermost loop. Set all outer loops to their
minimum loop counter values.
2. Test the min+1, typical, max-1 and max for the
innermost loop, while holding the outer loops at their
loop counter values.
3. Move out one loop and set it up as in step 2, holding all
outer loops at minimum values & nested loops at minimum
values
Continue this step until the outermost loop has been tested.

Source: Pressman

Loop Testing: Nested Loops

Example
While i > 0 do
While j <> 999 do
-EndWhile {inner}

EndWhile {outer}

1. When testing inner loop, assume outer loop executes 1 time

only (minimum)
2. Test inner loop (min + 1, typical, max-1, max)
3. Test outer loop (min + 1, typical, max-1, max), while holding
inner loop at typical iteration value (e.g., 2).

Source: Pressman

Loop Testing: Concatenated Loops

Concatenated Loops
for example, the final loop counter value of loop 1 is
used to initialize loop 2.
If the loops are independent of one another
then treat each as a simple loop
else* treat as nested loops
endif

Source: Pressman

Black-Box Testing
requirements

output

input

events

Source: Pressman

Black-Box (or Functional) Testing

1. To insure that all functions meet the requirements
2. Tests conducted at the user interface (like a typical user)
3. Test that input is properly accepted, output is correctly
produced and integrity of external components (e.g.,
database) maintained
4. Little regard for internal logical structure of software
5. Conducted in later stages as compared to white-box
testing techniques

Equivalence Partitioning-Inputs
Divides input of program into sets of data (equivalence
classes) from which test cases can be derived.
Use equivalence classes to define test case that uncovers
classes of errors, reducing total number of test cases
needed.

Equivalence Partitioning-Example

Software accepts data in the following form:

area code blank or 3-digit #, with valid values from 200 to 999.
prefix 3-digit number not beginning with 0 or 1
suffix 4-digit number
and so on
Equivalence Classes (EC) for the Area Code data element are:
EC 1 - Value between 200 & 999 (set of valid values)
EC 2 Value < 200 or > 999 (invalid)
EC 3 Area Code left blank (valid)
EC 4 Value not 3 digit number (e.g., 3 character string) (invalid)
Develop & execute test cases for each equivalence class.
Example Test Case # 1 uses EC 1, to test valid inputs.
In Test Case # 1, can choose inputs: 200, 250, or 999

Equivalence Partitioning Rules

If an input condition specifies a range (e.g., 0 to 100), 1 valid
(e.g., 0 to 100) and 2 invalid equivalence classes (e.g., < 0
or > 100 ; null) are defined
If an input condition requires a specific value (e.g., 5), 1 valid
(e.g., 5) and 2 invalid equivalence classes (e.g., not 5, null)
are defined
If an input condition specifies a member of a set, one valid
(e.g,. in set) and one invalid equivalence class (e.g., outside
set) are defined
If an input condition is Boolean, one valid (e.g., true) and
one invalid equivalence class (e.g., false) are defined

Boundary Value Analysis: Input

Assumption: Greater number of errors tends to occur at
boundaries of input domain (or equivalence classes) vs. the
center.
Used with equivalence partitioning. Rather than selecting any
element of an equivalence class, select input data at edges
of the class.
Guidelines below:
1. If input condition specifies range bounded by A and B, use
values A and B, just above A & B, & just below A & B.
2. If input condition specifies number of values, exercise
minimum, maximum, just above and below minimum and
maximum.

Boundary Value Analysis - Output

Apply guidelines 1 & 2 (in previous slide)
Example, if a table is required as output, output table
that has max and min allowable number of table entries

Equivalence Classes & Boundary Value Analysis

user
queries

output
input
mouse formats
prompts
picks

data

output
domain

input domain

Source: Pressman

Comparison Testing
(i.e., Back-to-back)
When reliability is absolutely critical (e.g., aircraft avionics,
automobile braking system)
Redundant hardware and software often used to minimize
possibility of error.
All versions executed in parallel with real-time comparison of
results to insure consistency.
Disadvantage: Will not uncover errors if
specification from which all versions have been
developed is in error
each independent version produces identical but incorrect
results

Other Types of Testing

Testing Documentation &
On-line Help
Testing GUIs
Testing Client/Server
Testing Real-Time Systems

Testing Documentation & On-line Help

Two (2) phases
Review and Inspection examine for editorial clarity
Live Test Use in conjunction with actual program,
preferably with independent 3rd party tester

Should be Included in Test Plan

Testing Documentation & On-line Help

Following questions should be answered during test:
o Accurately describe how to accomplish each mode of use?
o Are examples accurate?
o Are terminology, menu descriptions, and system responses consistent with
actual program?
o Is it relatively easy to locate guidance?
o Can troubleshooting be accomplished easily?
o Are table of contents and index accurate and complete?
o Is design (layout, graphics) easy to understand?
o Are all software error messages described in more detail?
o Are actions to be taken as consequence of an error message clearly
described?
o If hypertext links used, are they accurate & complete?

No Errors???

What if software testing does not detect errors in the

system?
1. The software has no errors.
2. Tests are inadequate to uncover the errors.
Similar to a strainer with large holes. Detects only the
biggest impurities.

Exercise
Do the end of lesson exercises provided by your instructor