Defect Removal Effectiveness

Defect Removal Effectiveness
Software Quality Management 
Software Quality Management
Unit 3

G. Roy Antony Arnold
G R A A ld
Asst. Professor / CSE

GRAA
• Defect
f removall is
i one off the
h top expenses in
i
any software project and it greatly affects
schedules.
• Effective defect removal can lead to
reductions in the development cycle time and
ggood pproduct qquality.
y
• It is important for all development
organizations to measure the effectiveness of
their defect removal processes.

GRAA
• Fagan (1976) defined error detection efficiency as:
Errors found by an Inspection
X 100%
Total errors in the product before inspection

• Jones's definition (1986), stated here, is very similar to Fagan's:
– Removal Efficiency =  Defects found by removal  operation
y p
X 100%
Defects present at removal operation

Defects found
X 100%
Defects found + Defects not found (found later)

• IBM Houston received the first NASA Excellence Award for Quality 
and Productivity in 1987

GRAA
• One
One of the four metrics IBM used to manage quality is 
of the four metrics IBM used to manage quality is
the early detection percentage, which is actually 
inspection defect removal effectiveness
Early Detection Percentage =
Number of major inspection errors
Number of major inspection errors
X 100%
Total number of errors
• where total number of errors is the sum of major 
inspection errors and valid discrepancy reports 
(di
(discrepancy report is the mechanism for tracking test 
t i th h i f t ki t t
defects).

GRAA
GRAA
• The
The effectiveness measure by Dunn (1987) differs little from 
effectiveness measure by Dunn (1987) differs little from
Fagan's and from Jones's second definition. 
• Dunn‘s definition is:
Effectiveness of activity (development phase ) =
Number of defects found by activity
Number of defects found by activity
X 100%
Number of defects found by subsequent activities

• This metric can be tuned by selecting only defects present at 
the time of the activity and susceptible to detection by the 
y p y
activity.

GRAA
• Daskalantonakis (1992) describes the metrics used at Motorola for 
software development
software development.
Total Defect Containment Effectiveness (TDCE) =
Number of pre‐release defects
Number of pre‐release defects
Number of pre‐release defects + Number of post‐release defects

Phase Containment Effectiveness (PCEi) =
Number of Phase i errors
Number of Phase i errors + Number of phase i defects

• Where
Where phase i
phase i errors are problems found during that development 
errors are problems found during that development
phase in which they were introduced, and 
• Phase i defects are problems found later than the development 
phase in which they were introduced.
phase in which they were introduced.

GRAA
 Defects removed (at the step)
( )
X 100%
Defects existing on step entry + Defects injected during development of the step

GRAA
GRAA
• B
Basedd on a special
i l study
t d commissioned
i i d by
b the
th Department
D t t
of Defence, Jones estimates the defect removal
effectiveness for organizations at different levels of the
development process capability maturity model (CMM):
( )
– Level 1: 85%
– Level 2: 89%
– Level 3: 91%
– Level 4: 93%
– Levell 5: 95%
• These values can be used as comparison baselines for
organizations to evaluate their relative capability with
regard to this important parameter.

GRAA
• Based on historical and recent data from three
software engineering organizations at General
Dynamics Decision Systems, Diaz and King (2002)
report that the phase containment effectiveness by
CMM level as follows:
– Level 2: 25.5%
– Level 3: 41.5%
– Level 4: 62.3%
– Level 5: 87.3%

GRAA
 Phase Inserted
Phase Inserted Cumulative % of Defects 
Cumulative % of Defects
removed through Acceptance 
Test
Requirements 94%
Top‐Level
Top Level Design 
Design 95%
Detailed Design 96%
Code and Unit Test
Code and  Unit 94%
Integration Test  75%
System Test
System Test 70%
Acceptance Test 70%

GRAA