Slide 6.

Object-Oriented and
Classical Software
Engineering
Fifth Edition, WCB/McGraw-Hill, 2002

Stephen R. Schach
srs@vuse.vanderbilt.edu

© The McGraw-Hill Companies, 2002

 CHAPTER 6 Slide 6.2

TESTING

© The McGraw-Hill Companies, 2002

 Overview Slide 6.3

● Quality issues
● Nonexecution-based testing
● Execution-based testing
● What should be tested?
● Testing versus correctness proofs
● Who should perform execution-based testing?
● When testing stops

© The McGraw-Hill Companies, 2002

 Testing Slide 6.4

● Two types of testing

– Execution-based testing
– Nonexecution-based testing

© The McGraw-Hill Companies, 2002

 Testing (contd) Slide 6.5

● “V & V”
– Verification
» Determine if the phase was completed correctly
– Validation
» Determine if the product as a whole satisfies its requirements

© The McGraw-Hill Companies, 2002

 Testing (contd) Slide 6.6

● Warning
– “Verify” also used for all nonexecution-based testing

© The McGraw-Hill Companies, 2002

 Software Quality Slide 6.7

● Not “excellence”
● Extent to which software satisfies its specifications
● Software Quality Assurance (SQA)
– Goes far beyond V & V
– Managerial independence
» development group
» SQA group

© The McGraw-Hill Companies, 2002

 Nonexecution-Based Testing Slide 6.8

● Underlying principles
– We should not review our own work
– Group synergy

© The McGraw-Hill Companies, 2002

 Walkthroughs Slide 6.9

● 4–6 members, chaired by SQA

● Preparation—lists of items
● Inspection
– Up to 2 hours
– Detect, don’t correct
– Document-driven, not participant-driven
– Verbalization leads to fault finding
– Performance appraisal

© The McGraw-Hill Companies, 2002

 Inspections Slide 6.10

● Five-stage process
– Overview
– Preparation, aided by statistics of fault types
– Inspection
– Rework
– Follow-up

© The McGraw-Hill Companies, 2002

 Fault statistics Slide 6.11

● Recorded by severity and fault type

● Compare with previous products
● What if there are a disproportionate number
of faults in a module?
● Carry forward fault statistics to the next phase

© The McGraw-Hill Companies, 2002

 Statistics on Inspections Slide 6.12

● 82% of all detected faults (IBM, 1976)

● 70% of all detected faults (IBM, 1978)
● 93% of all detected faults (IBM, 1986)
● 90% decrease in cost of detecting fault
(Switching system, 1986)
● 4 major faults, 14 minor faults per 2 hours
(JPL, 1990). Savings of $25,000 per
inspection
● Number of faults decreased exponentially
by phase (JPL, 1992)
● Warning
– Fault statistics and performance appraisal
© The McGraw-Hill Companies, 2002
 Metrics for Inspections Slide 6.13

● Fault density (e.g., faults per KLOC)

● Fault detection rate (e.g., faults detected per hour)
● By severity (major/minor), by phase
● What does a 50% increase in the fault detection
rate mean?

© The McGraw-Hill Companies, 2002

 Execution-Based Testing Slide 6.14

● Definitions
– Failure (incorrect behavior)
– Fault (NOT “bug”)
– Error (mistake made by programmer)
● Nonsensical statement
– “Testing is demonstration that faults are not present”

© The McGraw-Hill Companies, 2002

 What is execution-based tested? Slide 6.15

● “The process of inferring certain behavioral

properties of product based, in part, on results
of executing product in known environment with
selected inputs.”
– Inference
– Known environment
– Selected inputs
● But what should be tested?

© The McGraw-Hill Companies, 2002

 Utility Slide 6.16

● Does it meet user’s needs?

– Ease of use
– Useful functions
– Cost-effectiveness

© The McGraw-Hill Companies, 2002

 Reliability Slide 6.17

● Frequency and criticality of failure

– Mean time between failures
– Mean time to repair
– Mean time, cost to repair results of failure

© The McGraw-Hill Companies, 2002

 Robustness Slide 6.18

● Range of operating conditions

● Possibility of unacceptable results with valid input
● Effect of invalid input

© The McGraw-Hill Companies, 2002

 Performance Slide 6.19

● Extent to which space and time constraints are met

● Real-time software

© The McGraw-Hill Companies, 2002

 Correctness of specifications Slide 6.20

● Incorrect specification for a sort

● Function trickSort which satisfies this specification:

© The McGraw-Hill Companies, 2002
 Correctness of specifications (coptd) Slide 6.21

● Incorrect specification for a sort:

● Corrected specification for the sort:

© The McGraw-Hill Companies, 2002
 Correctness Slide 6.22

● NOT necessary
● NOT sufficient

© The McGraw-Hill Companies, 2002

 Correctness Proofs Slide 6.23

● Alternative to execution-based testing

© The McGraw-Hill Companies, 2002

 Example of Correctness Proof Slide 6.24

● Code segment to be proven correct

© The McGraw-Hill Companies, 2002

 Example of Correctness Proof (contd) Slide 6.25

● Flowchart of
code segment

© The McGraw-Hill Companies, 2002

Example of Correctness Proof Slide 6.26

© The McGraw-Hill Companies, 2002

 Correctness Proof Case Study Slide 6.27

● Never prove a program correct without testing it

as well

© The McGraw-Hill Companies, 2002

 Episode 1 Slide 6.28

● 1969 — Naur Paper

● “Naur text-processing problem”
Given a text consisting of words separated by blank or by 
(new line) characters, convert it to line-by-line form in
accordance with following rules:
(1) line breaks must be made only where given text
has blank or nl ;
(2) each line is filled as far as possible, as long as
(3) no line will contain more than maxpos characters
● Naur constructed a procedure (25 lines of Algol 60),
and informally proved its correctness

© The McGraw-Hill Companies, 2002

 Episode 2 Slide 6.29

● 1970 — Reviewer in Computing Reviews

– The first word of the first line is preceded by blank unless
the first word is exactly maxpos characters long

© The McGraw-Hill Companies, 2002

 Episode 3 Slide 6.30

● 1971 — London finds 3 more faults

● Including:
– The procedure does not terminate unless a word
longer than maxpos characters is encountered

© The McGraw-Hill Companies, 2002

 Episode 4 Slide 6.31

● 1975 — Goodenough and Gerhart

find three further faults
● Including:
– The last word will not be output unless it
is followed by blank or nl

© The McGraw-Hill Companies, 2002

 Proofs and Software Engineering Slide 6.32

● Lesson:
● Even if product is proved correct,
it must STILL be tested.

© The McGraw-Hill Companies, 2002

 Three Myths Slide 6.33

● Software engineers do not have enough

math for proofs
● Proving is too expensive to be practical
● Proving is too hard

© The McGraw-Hill Companies, 2002

Proofs and Software Engineering (contd) Slide 6.34

● Can we trust a theorem prover ?

● How to find input–output specifications,
loop invariants
● What if the specifications are wrong?
● Can never be sure that specifications
or a verification system are correct

© The McGraw-Hill Companies, 2002

Proofs and Software Engineering (contd) Slide 6.35

● Correctness proofs are a vital software engineering

tool, WHERE APPROPRIATE.
● If
– Human lives are at stake
– Indicated by cost/benefit analysis
– Risk of not proving is too great
● Also, informal proofs can improve the quality of
the product
– Assert statement

© The McGraw-Hill Companies, 2002

Who Performs Execution-Based Testing? Slide 6.36

● Testing is destructive
– A successful test finds a fault
● Solution
– 1. The programmer does informal testing
– 2. SQA does systematic testing
– 3. The programmer debugs the module
● All test cases must be
– Planned beforehand, including expected output
– Retained afterwards

© The McGraw-Hill Companies, 2002

 When Testing Can Stop Slide 6.37

● Only when the product has been irrevocably retired

© The McGraw-Hill Companies, 2002