Software Quality Engineering

Lecture 13 and 14: Verification & Validation

+ SQA Defect Removal Cost

Muhammad Awais Gondal

 Overview
• Verification, Validation and Qualification
• SQA Defect Removal Effectiveness and Cost
 Verification, Validation & Qualification
(Three aspects of quality assurance of software
products)
Verification – The process of evaluating a system or component to
determine whether the products of a given development phase
satisfy the conditions imposed at the start of that phase

Validation - The process of evaluating a system or component

during or at the end of the development process to determine
whether it satisfies specified requirements

Qualification - The process used to determine whether a system

or component is suitable for operational use
 Verification
• Are we building the product right?
• Verification is a static practice of verifying
documents, design, code and program.
• Verification uses methods like inspections,
reviews, walkthroughs etc.
 Validation
• Are we building the right product?
• Validation is a dynamic mechanism of
validating and testing the actual product.
• Examining the extent of compliance with
customer’s needs.
• Validation uses methods like black box
(functional) testing, gray box testing, and
white box (structural) testing etc.
 Qualification
• Focuses on operational aspects.
• Maintenance is the main concern.
 A model for SQA defect removal
effectiveness and cost
• The model addresses two quantitative aspects
of SQA plan for defect detection:
– Total effectiveness in removing project defects
– Total cost of removing project defects
 A model for SQA defect removal
effectiveness and cost
• The model is based on three types of data:
– Defect origin distribution (in which phase did the
defects occur)
– Defect removal effectiveness (how effective is the
removal of defects)
– Cost of defect removal (how much does it cost to
remove defects in different phases)
 Defect Origin Distribution

No. Software development phase Average percentage of

defects originating in phase

1 Requirements Specification 15%

2 Design 35%

3 Coding (coding 30%, integration 10%) 40%

4 Documentation 10%
 Defect Removal Effectiveness
• Percentage of removed defects is lower than
percentage of detected defects
WHY??

• Some corrections are ineffective

• Some defects are missed and passed on to
next phases
 Defect Removal Effectiveness
• An assumption: an activity removes at least 40% of the
incoming defects
No. Quality assurance activity Average defect filtering
effectiveness rate
1 Requirements specification review 50%
2 Design inspection 60%
3 Design review 50%
4 Code inspection 65%
5 Unit test 50%
6 Unit test after code inspection 30%
7 Integration test 50%
8 System tests / acceptance tests 50%
9 Documentation review 50%
 Defect Removal Cost
• Cost of removal of defects varies by
development phase
• Costs are much increased in later
development phases
Why??
 Defect Removal Cost
No. Software development phase Average relative
defect cost (cost units)
1 Requirements Specification 1
2 Design 2.5
3 Unit tests 6.5
4 Integration tests 16
5 System tests / acceptance tests / system 40
documentation review

6 Operation by customer (after release) 110

 Assumptions for the model
• Development process is linear, sequential following
waterfall model
• A number of new defects are introduced in each phase
• Review and test software quality assurance activities
serve as filters, removing a percentage of defects and
letting the rest pass to next development phase.
• At each phase, incoming defects are the sum of those not
removed plus new defects in current phase
• Cost of defect removal is calculated for each SQA activity
by multiplying the number of defects removed by the
relative cost of removing a defect.
• Remaining defects are passed to the customer. (heaviest
cost for defect removal)
 Parameters for the model
• POD – Phase originated defects
• PD – Passed defects (from previous phases)
• %FE - % of filtering effectiveness
• RD – Removed defects
• CDR – Cost of defect removal
• TRC – Total Removal Costs

TRC = RD x CDR
A basic representation
 The model applied on Standard
Quality Assurance Plan
• Having standard quality assurance
activities/filters
A process oriented illustration of
standard quality assurance plan
 Using a comprehensive quality
assurance plan
• A comprehensive quality assurance plan with
more activities for filtering
• Adds two quality assurance activities to be
performed in phases of design and coding
– Design Inspection
– Code Inspection
• Improves the filtering effectiveness of other
quality assurance activities
Using a comprehensive quality
assurance plan
A process oriented illustration of
standard quality assurance plan
 Conclusion
(Benefits of comprehensive plan)
• The comprehensive plan
– Removes more defects in earlier phases
– Saves resources … more economical
– Greater customer satisfaction … defects detected
in operation are less
 Conclusion
(Benefits of comprehensive plan)
• The standard plan removes 57.6% of the defects in
requirements and design compared to 90.2% in the
comprehensive plan before coding begins.
• More economical - saves 41% of the total resources
investing in defect removal, compared with the
standard plan
• Compared to the standard plan, the comprehensive
plan makes a greater contribution to customer
satisfaction by reducing the rate of defects detected
during regular operations: from 6.9% to 2.6%
 Next Lecture
• Review Objectives
• Formal Design Reviews