You are on page 1of 107

Defect Management & Metrics

Society for Software Quality April 24, 2010

Who Is This Guy?


30 years in software and hardware/software Quality 19 years in Quality & Process Management 6 years as Director of Software Best Practices and Corporate Software Metrics Leader at Intuit Past SSQ President
L

S
SSQ Presentation
021209.01

2010 Holl Consulting

Primary Relevant Experience


Designed Intuits corporate defect management process Assisted in rollout to ~1800 engineers Guided its evolution for six years As Corporate Software Metrics Leader, fostered quantitative project management and process improvement Made mistakes and learned from them
SSQ Presentation
021209.01

2010 Holl Consulting

Agenda
Summary of Previous Talk Defect Management Metrics Metrics for Project Management Metrics for Process Improvement Advanced Metric Models

SSQ Presentation

021209.01

2010 Holl Consulting

Last time
We reviewed cultural issues related to defect management We went over a defect management process:
Policies Workflow/States Fields

SSQ Presentation

021209.01

2010 Holl Consulting

Data Collection
At different states in the workflow, different data are collected Most data are collected at submit time Additional data are entered by the developer upon resolution And finally by the tester upon verification

SSQ Presentation

021209.01

2010 Holl Consulting

Agenda
Summary of Previous Talk Defect Management Metrics Metrics for Project Management Metrics for Process Improvement Advanced Metric Models

SSQ Presentation

021209.01

2010 Holl Consulting

When Does A Defect Count?


It depends on your goals
Defect Management: To track defects for resolution in order to release products
Just log functional problems found in test; deviations from the requirements that the customer will see; defects that you dont want to fall through the cracks.

Defect Analysis: To improve the process or predict


If your goal is to get better (learn) or predict the future, you need to capture much more information. Without early lifecycle data (requirements, etc.) predictive models arent that reliable
World-class companies dont just manage defects they analyze information.
SSQ Presentation
2010 Holl Consulting

021209.01

Data Collection Costs


Estimate Assumptions:
18,000 defects the defects are corrected (9,000) Data collection time: 10 minutes (validated at HP) Average defect cost: 16 hours (data from Michael Fagan) Data collection reduces defects 5% Sampling is valid, but sampling has to be managed to get good random samples and to ensure that adequate (statistically significant) samples are being collected.

Time to record (18k *10m) Time to correct (9k *16h) Time savings (450 * 16h) ROI

3,000 hrs 144,000 hrs 7,200 hrs 240%

Defect reduction (9k * 5%) 450 defects

This doesnt quantify the value of being able to predict quality and schedules
SSQ Presentation
021209.01

2010 Holl Consulting

Simple Measures
Defect cost = Fix rate = Duration = (Total Resolution Effort) Number of resolved defects Number of resolved defects Period of time (week, month, etc.) ((Closed Date) (Submit Date))* Number of resolved defects

Last two include queue time, so they depend on workload Its helpful to understand the standard deviation too Bonus: By type, root cause, injection phase
* Excel NetWorkDays is a useful function
SSQ Presentation
021209.01

2010 Holl Consulting

Relative Defect Costs


If you record injection and discovery phases and effort, you can compute relative costs Example:
Cost of a requirements defect = 1 Cost of a design defect = 10 Cost of a coding defect = 100 Cost of a production defect = 1000
SSQ Presentation
021209.01

2010 Holl Consulting

Similar to Phase Containment


Start with matrix of injection and discovery Enter the average rework (or duration) costs in each cell (resolved defects only)
Injection Phase Discovery Phase Requirements Design Code Testing Production Req's 2.4 15.3 22.7 43.8 85.3 4.1 9.5 18.5 38.5 1.2 4.3 18.3 1.2 2.2 1 Design Code Test Production

SSQ Presentation

021209.01

2010 Holl Consulting

Similar to Phase Containment


Normalize by dividing each cell by the Injection=Discovery value
Injection Phase Discovery Phase Requirements Design Code Testing Production Req's 1.00 6.38 9.46 18.25 35.54 1.00 2.32 4.51 9.39 1.00 3.58 15.25 1.00 1.83 1.00 Design Code Test Production

SSQ Presentation

021209.01

2010 Holl Consulting

Logging Defects
This is not
How many defects leak?

This is
What percent of known defects are recorded?

If you only start logging defects in test, youre missing over half of your data Try to capture data from reviews/inspections and developer unit tests
SSQ Presentation
021209.01

2010 Holl Consulting

Prioritizing Defects
This is not about how to set priority, but Priority means the most important defects get resources first To measure this, we need to know the queue time (from submit to work starts) Therefore, we need an Accepted state, or something to note when work began Why cant we use Submit to Verify duration?
SSQ Presentation
021209.01

2010 Holl Consulting

Aging Defects
Time in Resolver queue Time in Verifier queue
12 Days in Queue

Alternately, use durations minus effort as a proxy

Average Queue Time


10 8 6 4 2 0 0-Test Stopper 1 2 Priority 3 4 5

SSQ Presentation

021209.01

2010 Holl Consulting

Defect Backlog
Many groups dont count defects left over from prior releases; they only focus on defects found and not fixed in the current project This is dangerous because a product defect backlog can grow unmanaged ALL of the open defects are released, not just the new ones When producing the report of open defects, query for all open defects in the product
SSQ Presentation
021209.01

2010 Holl Consulting

The Calculation
Week 1 2 3 4

Total number of defects Number closed before the week started Number that were opened after the week ended = Number open in the week

SSQ Presentation

021209.01

2010 Holl Consulting

Calculating Backlog in Excel


Query Submit Date and Closed Date

SSQ Presentation

021209.01

2010 Holl Consulting

Calculating Backlog in Excel


Array formula. <Shift-Ctrl-Return>

This uses named ranges, where D.Num.CRs D.T.Closed = Total number of CRs opened = Range of Closed Dates

D.Submit.Date = Range of Submit Dates

SSQ Presentation

021209.01

2010 Holl Consulting

Product Defect Backlog

The Result

SSQ Presentation

021209.01

2010 Holl Consulting

Agenda
Summary of Previous Talk Defect Management Metrics Metrics for Project Management Metrics for Process Improvement Advanced Metric Models

SSQ Presentation

021209.01

2010 Holl Consulting

Some Project Management Defect Metrics

Reviews & Inspections Defect Discovery & Closure Defect Density

SSQ Presentation

021209.01

2010 Holl Consulting

A Project Axiom The quality of product in one phase is an indicator of the quality in later phases Why?
There are two answers
SSQ Presentation
021209.01

2010 Holl Consulting

Review Metrics
Reviews & inspections have proven to be the most costeffective way to detect and remove defects Inspect your key project artifacts
(requirements, designs, code, tests, etc.)

SSQ Presentation

021209.01

2010 Holl Consulting

Review/Inspection Data
There are many data collected during a review or inspection The ones that tell us about quality are:
Number of Operational defects discovered Number of Minor defects found

per changed line of inspected doc/code Number of reviews per doc/module


SSQ Presentation
021209.01

2010 Holl Consulting

Project Defect Density


Consistency in categorizing inspection defects is important
Have clear guidelines for Operational and Minor defects

Defect Density Project = Operational Defects found by inspection Delta SLOC (or LOT)
SSQ Presentation
021209.01

2010 Holl Consulting

Projecting Defect Leakage


Inspected total of 5340 SLOC Found 47 operational defects Density = 47/5340 = 8.8 defects/KSLOC Project changed 18,463 SLOC Estimated defects = 18.463 * 8.8 = 162 Remaining = 162 - 47 = 115 What should you do? Inspect more? Or Test for them?
SSQ Presentation
021209.01

2010 Holl Consulting

Verification
Use Defect Density to estimate expected yield
Similar to estimate described with reviews

Defect discovery & closure Use extrapolation to predict when defect exit criteria will be met

SSQ Presentation

021209.01

2010 Holl Consulting

Defect Discovery
Total Defects Opened
Cumulative Defects Found

Weekends

Time

SSQ Presentation

021209.01

2010 Holl Consulting

Defect Discovery Curve Assumptions Time is a proxy for exposure


Remove weekends to smooth the curve

Number of test cases run would be better, assuming


Test coverage is good Not continuing to add new test case

SSQ Presentation

021209.01

2010 Holl Consulting

Well-Behaved Curve
Total Defects Opened
Cumulative Defects Found

Time

Release Target Date

SSQ Presentation

021209.01

2010 Holl Consulting

Add: Defect Closure


Total Defects Opened & Closed
Remaining Open

Cumulative Defects

Opened Closed

Time

SSQ Presentation

021209.01

2010 Holl Consulting

Add: To QA
Total Defects Opened, Closed, & To QA
Cumulative Defects

Opened, waiting for development fix

Turned over from development to QA, awaiting verification

Opened To QA Closed

Time

SSQ Presentation

021209.01

2010 Holl Consulting

Finding The Balance


Total Defects Opened, Closed, & To QA
Cumulative Defects

Whats happening here?

Dev is a bottleneck
Opened To QA Closed

QA is a bottleneck

Time

SSQ Presentation

021209.01

2010 Holl Consulting

Add: Legacy & Target Release


Total Defects Opened & Closed
Cumulative Defects
Inherited from previous history, targeted for or found in the project/release

Total Targeted To QA Closed

Time

SSQ Presentation

021209.01

2010 Holl Consulting

Extrapolation to Release Date


Project Defects
Cumulative Defects

Should I be concerned?

Total Targeted To QA Closed

Time

Release Target Date

SSQ Presentation

021209.01

2010 Holl Consulting

Extrapolation to Release Date


Project Defects
Cumulative Defects

When will the project be done?

Total Targeted To QA Closed

Time

Release Target Date

SSQ Presentation

021209.01

2010 Holl Consulting

Computing in Excel
Query defects into Excel Convenient: Use named ranges for data

SSQ Presentation

021209.01

2010 Holl Consulting

Computing in Excel
In a second sheet, create a date column and count the defects submitted and closed for each date.
Array formula. <Shift-Ctrl-Return>

SSQ Presentation

021209.01

2010 Holl Consulting

Real Life Example


500 450 400 350 300 6/16 = Start of testing 250 50 100

Aardvark 4.7 Defect Discovery & Closure


(With extrapolations)

90 80 70 Found in 4.7 60

Total CRs

2nd order polynomial fit


200 150 100
Open

40 30 20
Closed Found in 4.7

50 0
7/1/08 7/8/08 6/17/08 6/24/08 7/15/08 7/22/08 7/29/08 8/5/08 8/12/08 8/19/08

10 0
8/26/08 Target Release Date

SSQ Presentation

021209.01

2010 Holl Consulting

Release Criteria: Defects


Goals:
Provide good-enough quality to customers
Dont release any bad defects Dont release too many defects

Minimize support costs


Customers will need support proportional to product quality

SSQ Presentation

021209.01

2010 Holl Consulting

Defect Discovery
Total Defects Opened
Cumulative Defects Found

Should I release?

Time

Release Target Date

SSQ Presentation

021209.01

2010 Holl Consulting

Bad Defects
Bad is a function of
Severity = the consequences of the failure Probability that it will occur The number of customers likely to experience it May also include important customers or functions

These are often complex to calculate, so most companies just use Severity, tempered by other factors

SSQ Presentation

021209.01

2010 Holl Consulting

Quality Threshold
No more than the following number of open defects:
Severity* Critical Important Moderate Trivial New 0 5 30 50 Old 0 10 100 200 Release team (stakeholders) need to determine values based on history. The goals are: Not to release any Critical defects Not to introduce too many new defects Manage the backlog

Where New = defects discovered in the current project, and Old = defects discovered in previous releases

SSQ Presentation

021209.01

2010 Holl Consulting

Whats Good Enough?


The two market leaders Quicken TurboTax, produced by Intuit, and TaxCut, produced by H&R Block

TurboTax offers better videos than TaxCut

It also offers better endof-return reviews

SSQ Presentation

021209.01

2010 Holl Consulting

Whats Good Enough?


TaxCut offers easier tagging of items youd like to deal with later.

It also appears to be more scrupulously proofread than TurboTax: Twice in TurboTaxs screens in my tests, incorrect information was displayed, though the tax returns themselves were unaffected.

SSQ Presentation

021209.01

2010 Holl Consulting

Whats Good Enough?


if youre choosing which program to go with for the first time this year, Id go with TaxCut solely because of the text errors in TurboTax.

I have a higher standard for accuracy in tax programs than I do for any other type of software, and while the errors we found didnt affect the bottom line, they did affect my confidence in the programs makers. Its unfortunate, because I felt TurboTax offered better features.

SSQ Presentation

021209.01

2010 Holl Consulting

Detroit Free Press


> 1.7M Readers + web site + wire service
The article came from the Monterey Peninsula Herald

SSQ Presentation

021209.01

2010 Holl Consulting

Predicting Defects
There are several models for predicting defects The predictions of interest are:
How many defects are going to be created? When will I find them? How long will it take to fix them? When will I be done? How good will the result be?
SSQ Presentation
021209.01

2010 Holl Consulting

Using Defect Density


Phase Defect Density DDPhase = Defects InjectedPhase Total Project Affected SLOC

At first this may not seem right, but its a reasonable predictor Which is more reliable?
67 specification defects per 200 pages of documentation (0.34 defects/page) 67 specification defects per 4300 SLOC (15.6 defects/KSLOC)

SSQ Presentation

021209.01

2010 Holl Consulting

Using Defect Density


Project/Process Defect Density
Number of defects per affected SLOC Affected SLOC can be difficult to get without a good tool. The best Ive found is CodeReports (aka SourceROI)
Release A Defects
Total SLOC Defect Density Project Changed SLOC Project Defect Density

Release B

452 8,600 52.6 5,400 83.7

563 43,500 12.9 4,350 129.4

http://www.polariguana.com/sourceroi.php
SSQ Presentation
021209.01

2010 Holl Consulting

Calculating DD per Phase


# Defects Injected Per Phase DDPhase = Project KSLOC
Number of Defects Injection Phase Req's Design Code Test Production 46 8 83 33 3 173 4.94 0 26 19 22 2 69 1.97 0 0 83 37 16 136 3.89 0 0 0 2 0 2 0.06 0 0 0 0 1 1 0.03

Project SLOC = 35,000


Note: if this is only test defect data, youre missing an opportunity to get better information.

Discovery Phase Requirements Design Code Testing Production Total Injected Defect Density
021209.01

SSQ Presentation

2010 Holl Consulting

Next Project
Estimated to be 20,000 SLOC
Historical Defect Density Requirements Design Code Testing Production 4.94 1.97 3.89 0.06 0.03 Total Likely to Inject 99 39 78 1 1 218 Phase Likely Effort to Containment to Leak Resolve* 26.6% 17.3% 62.1% 81.7% 100.0% 26 7 48 1 1 85.3 38.5 18.3 2.2 1 Grand Total Total Rework 2242.2 263.3 882.9 2.1 0.6 3391.0

* Worst case cost (defects discovered in test)

SSQ Presentation

021209.01

2010 Holl Consulting

SSQ Presentation

Predictions can be made in Excel by extrapolating and seeing where the discovery and close lines meet

Defect Discovery & Closure

021209.01

100000

150000

200000

250000

300000

50000

2010 Holl Consulting


10 /1 4 10 /2 0 /2 06 8 11 /2 0 /1 06 1 11 /2 0 /2 06 5/ 2 12 00 /9 6 12 / 20 /2 06 3/ 2 1/ 006 6/ 2 1/ 00 20 7 /2 0 2/ 07 3/ 20 2/ 17 0 7 /2 0 3/ 07 3/ 20 3/ 17 0 7 / 3/ 20 0 31 7 / 4/ 20 0 14 7 /2 4/ 0 0 28 7 / 5/ 20 0 12 7 /2 5/ 0 0 26 7 /2 0 6/ 07 9/ 20 6/ 23 0 7 /2 0 7/ 07 7/ 20 7/ 21 0 7 /2 0 8/ 07 4/ 20 07

Cumulative Defects Submitted

Extrapolation
This method is quick and in some cases surprisingly good But not in all cases

SSQ Presentation

021209.01

2010 Holl Consulting

Other Excel Extrapolations


Adding a trend line uses all the data You may only want to use some of the data Create a parallel column of the data you want to extrapolate and extrapolate only that curve

SSQ Presentation

021209.01

2010 Holl Consulting

Example

SSQ Presentation

021209.01

2010 Holl Consulting

Simple Linear Extrapolation


To create a linear extrapolation on the last few data points, start by extending the date column:

SSQ Presentation

021209.01

2010 Holl Consulting

Simple Linear Extrapolation


Select the cells that look linear Copy and Paste Special - Values

SSQ Presentation

021209.01

2010 Holl Consulting

Simple Linear Extrapolation


Then drag the cells down to the last extrapolated date Excel will do a linear extrapolation of the data

SSQ Presentation

021209.01

2010 Holl Consulting

SSQ Presentation
10 20 30 40 50 60 0
1/1/2008 1/15/2008 1/29/2008 2/12/2008 2/26/2008 3/11/2008 3/25/2008 4/8/2008 4/22/2008 5/6/2008 5/20/2008 6/3/2008 6/17/2008 7/1/2008 7/15/2008 7/29/2008 8/12/2008 8/26/2008 9/9/2008 9/23/2008 10/7/2008
021209.01

In order for extrapolations to be reliable, the discovery rate must have peaked

Extrapolation

Weekly Defects Submitted

2010 Holl Consulting

Done?
What does it mean to be done (from a defect perspective)?
No bad defects remain Predictable/manageable defects remain

Two categories of defects remain


Those you know Those you dont know
SSQ Presentation
021209.01

2010 Holl Consulting

Latent Defects
Latent defects (unknown at release time) can be predicted using defect density and Rayleigh PDF What needs to happen before the predictions become reliable?
The defect discovery rate must flatten

d f(t) =0 dt
SSQ Presentation
021209.01

2010 Holl Consulting

100

200

300

400

500

600

SSQ Presentation
021209.01

1/1/2008 1/15/2008 1/29/2008

2/12/2008 2/26/2008 3/11/2008 3/25/2008 4/8/2008 4/22/2008 5/6/2008 5/20/2008 6/3/2008 6/17/2008 7/1/2008 7/15/2008 7/29/2008 8/12/2008 10 20 30 40 50 60 0 1/1/2008 1/15/2008 1/29/2008 2/12/2008 2/26/2008 3/11/2008 3/25/2008 4/8/2008 4/22/2008 5/6/2008 5/20/2008 6/3/2008 6/17/2008 7/1/2008 7/15/2008 7/29/2008 8/12/2008 8/26/2008 9/9/2008 9/23/2008 10/7/2008 8/26/2008 9/9/2008 9/23/2008 10/7/2008

Discovery Rate

Cumulative Defect Submitted

2010 Holl Consulting

Weekly Defects Submitted

Fitting Data to a Model


Fitting data to a curve helps us tell the rate is decreasing

SSQ Presentation

021209.01

2010 Holl Consulting

Using Excel
As before, create a linear extrapolation of the last <criteria> days: Flattening criteria example:
The discover rate must less than 5 defects per day in the past two weeks of testing

SSQ Presentation

021209.01

2010 Holl Consulting

Using Excel
Add linear trend line Select Display equation on chart

SSQ Presentation

021209.01

2010 Holl Consulting

Using Excel
This gives the slope of the line m, where
Cumulative Defect Submitted

y=mx + b
In this case, m = 2.2 defects/day Not good enough

600 500 y = 2.163x - 85395 400 300 200 100 0

1/1/2008

4/8/2008

5/6/2008

6/3/2008

7/1/2008

1/15/2008

1/29/2008

2/12/2008

2/26/2008

3/11/2008

3/25/2008

4/22/2008

5/20/2008

6/17/2008

7/15/2008

7/29/2008

8/12/2008

8/26/2008

9/9/2008

9/23/2008

SSQ Presentation

021209.01

2010 Holl Consulting

10/7/2008

Latent or Known Defect Risk


Probability of occurrence in production
Risk = Severity * Probability

Need to have a decent measure/estimate of probability


Need a usage profile

Also, frequency does not always equal value


I may not use the Restore function very often, but it better work
SSQ Presentation
021209.01

2010 Holl Consulting

And Finally
Track latent/released defects carefully They are your best indicator of Quality

SSQ Presentation

021209.01

2010 Holl Consulting

Release Defect Density


From: Software Quality Measurement: A Framework for Counting Problems and Defects CMU/SEI-92-TR-022 http://www.sei.cmu.edu/pub/d ocuments/92.reports/pdf/tr22. 92.pdf

SSQ Presentation

021209.01

2010 Holl Consulting

Agenda
Summary of Previous Talk Defect Management Metrics Metrics for Project Management Metrics for Process Improvement Advanced Metric Models

SSQ Presentation

021209.01

2010 Holl Consulting

A True Story
A while back, I looked in the mirror and decided I needed to loose some weight. What is the first thing Im going to do?

SSQ Presentation

021209.01

2010 Holl Consulting

My First Step

SSQ Presentation

021209.01

2010 Holl Consulting

My First Step On A Scale


If I dont measure myself today, I wont know
If the changes Im planning to make are taking me the right direction Which changes have the best affect How far Ive come When Im done

Its important to establish a baseline


SSQ Presentation
021209.01

2010 Holl Consulting

What Are The Goals?


`Would you tell me, please, which way I ought to go from here?' `That depends a good deal on where you want to get to,' said the Cat. `I don't much care where--' said Alice. `Then it doesn't matter which way you go,' said the Cat.
SSQ Presentation
021209.01

2010 Holl Consulting

Closure Data Collection


This is where we record what weve learned
Total Resolution Effort Correction made Duplicate Change not justified Defect?
Yes No

Submit reason Test case failed Pass criteria unclear Tested wrong version Setup/config wrong Data-related error Other

Resolution Injection Phase Classification, Mode Corrected Component(s) Root Cause

SSQ Presentation

021209.01

2010 Holl Consulting

Defect Classification Models


Industry guidelines to standardize information collection The most prevalent are from:
IEEE IBM Hewlett Packard

SSQ Presentation

021209.01

2010 Holl Consulting

HP Defect Origins, Types, and Modes

SSQ Presentation

021209.01

2010 Holl Consulting

Defect Leakage Reasons


1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. Environmental Dependency Test Not Run Test Case Error Data Dependency Inconsistent Symptoms Test Case Success Criteria Unclear Test Execution Error Test Case Results Not Verified Code Changed After Test Passed No Test Case/Coverage Configuration Management Problem Requirements Missing/Changed
2010 Holl Consulting

SSQ Presentation

021209.01

Using Leakage Data


For each defect found in production, log a leakage reason Use the frequency (volume) to improve the related process Resist temptation to weight the frequency by something important
This assumes cause & effect, and there is none This is different from root cause, where the cost to repair is related to the type of defect
SSQ Presentation
021209.01

2010 Holl Consulting

Specific Process Improvement Metrics


Two types of process metrics:
1. Metrics that monitor a process (tell you the effect of changes)
Basic SPC-type charts

2. Metrics that tell you what to do


Phase Containment Root Cause Analysis Test Coverage Defect Leakage Reasons Static Code Analysis - Complexity
SSQ Presentation
021209.01

2010 Holl Consulting

Should all projects be measured?


Pros:
More data points Evaluate everything were doing

Cons:
Projects may use different processes Variation may be too great

Suggestions:
Dont measure projects measure processes Only measure your best projects Tailoring is okay, but separate out the data
SSQ Presentation
021209.01

2010 Holl Consulting

Hold Periodic Process Reviews


Monthly Quality Reviews focused on
Process Improvement initiatives Measures & Analysis/Interpretation Lessons Learned

Rules of metrics etiquette:


Preview data with stakeholders = no surprises

SSQ Presentation

021209.01

2010 Holl Consulting

Phase Containment
Identifies the lifecycle activities that leak defects Tells you where to improve defect detection efforts
Phase Containment
100% 90% 80% % Defects Contained 70% 60% 50% 40% 30% 20% 10% 0% Requirements Design Code Build Testing

SSQ Presentation

021209.01

2010 Holl Consulting

SSQ Presentation
021209.01

Pareto the types of errors being created, 3000 weighted by cost (effort) 2500

Root Cause Analysis

2010 Holl Consulting

1000

1500

2000

500

0
Requirements Ambiguous/vague Requirements Missing Logic flow Missing logic Requirements Changed Wrong variable used Other Case statement Typo Specification Missing Garbage collection Exception handling Parameter m ismatch Requirements Wrong Uninitialized variable Bad validation check Missing end statement Env not match test

Root Cause x Rework Hours

Root Cause T

Agenda
Summary of Previous Talk Defect Management Metrics Metrics for Project Management Metrics for Process Improvement Advanced Metric Models

SSQ Presentation

021209.01

2010 Holl Consulting

Defect Probabilities
Weibull distributions are useful engineering models, where the tail approaches zero asymptotically Weibull Probability Density Function (PDF) is:
m m

m t f(t) = t c

m - t c

m t = c c

m-1 - t c

(Wikipedia)

Where m determines the shape of the curve, and c is the scale parameter
SSQ Presentation
021209.01

2010 Holl Consulting

Different Weibull Distributions


4

Weibull Curves
Weibull curves with different shapes (values of m) can be used to model various systems (probability distributions)
3 m=0.5 m=10

m=1 1

m=2

m=4

0 0 1 2 3

SSQ Presentation

021209.01

2010 Holl Consulting

Weibull Probability Distribution


2.0

Weibull Curves
m = 0.1 to 5.0 in .1 increments
1.5

Rayleigh (m=2)
1.0

0.5

0.0 0.0 0.5 1.0 1.5 2.0

SSQ Presentation

021209.01

2010 Holl Consulting

Weibull Curves
2.0

Weibull Curves
m = 0.1 to 5.0 in .1 increments
1.5

(almost) Normal (m=3.4) Rayleigh (m=2)


1.0

0.5

0.0 0.0 0.5 1.0 1.5 2.0

SSQ Presentation

021209.01

2010 Holl Consulting

Rayleigh Probability Distribution Function


Weibull distribution with shape (m) = 2
2 2

2 t f(t) = t c
Let x = 2t

2 - t

2t = 2 c
2

t c

f(x) = c2 e
SSQ Presentation
021209.01

- x 2c

= c2

x2 - 2 4c

2010 Holl Consulting

Rayleigh Probability Distribution Function


9

f(x) =
2 e c

8 7

Rayleigh Curves
c = 0.1 to 0.5
c = 0.1

x2 4c 2

6 5 4 c = 0.2 3 c = 0.3 2 1 0 0.0 0.5 1.0 1.5

SSQ Presentation

021209.01

2010 Holl Consulting

Excel Formula

SSQ Presentation

021209.01

2010 Holl Consulting

Applying Rayleigh PDF


Let tm = time at which the curve reaches its peak To find the peak, we set

d f(t) dt

=0

c Solving gives tm = 2
To scale the area under the curve, multiply by K Substitute c = tm 2
SSQ Presentation
021209.01

2010 Holl Consulting

Fitting The Rayleigh PDF


2 2

2t f(t) = K 2 c

t c

=K

2t
(tm 2)

t
(tm 2)

e 2
2

= K 2t2 e m

2t

- t2
2t2m

f(t) = K t m

- t2

te

2t2m

If we can estimate how many defects well find (K) and when weve hit the peak (tm), we can create a PDF for our project!
SSQ Presentation
2010 Holl Consulting

021209.01

Building Your Estimate

SSQ Presentation

021209.01

2010 Holl Consulting

Predicting Latent Defects


At peak, the area under the Rayleigh curve is 39.35% This means 60.65% of the defects are yet to find Example:
At peak you found a cumulative total of 58 defects. Total = (58/39.5)*100=147.4 147 - 58 = 89 remaining.
1 year?

Latent Defects = K

1
tm

- t2

te

2t2m

dt

Release Date

SSQ Presentation

021209.01

2010 Holl Consulting

Easier that Calculus


In Excel, run a column for cumulative total and a column for remaining defects From your release date you can easily estimate latent defects

SSQ Presentation

021209.01

2010 Holl Consulting

Cumulative Rayleigh Model

CDF = 1 - e

t c

SSQ Presentation

021209.01

2010 Holl Consulting

Rayleigh Assumptions
The defect rate observed during the development process is positively correlated with the defect rate in the field. Given the same error injection rate, if more defects are discovered and removed earlier, fewer will remain in later stages.

SSQ Presentation

021209.01

2010 Holl Consulting

Exponential Models
Exponential Model (Weibull, m=1) 1 f(t) = c
t c
1.2

Exponential
1.0

Weibull,

m=1

Error detection rate Instantaneous failure rate Hazard rate

0.8

0.6

In general, this is a reliability growth model

0.4

0.2

0.0 0.0 0.5 1.0 1.5 2.0

SSQ Presentation

021209.01

2010 Holl Consulting

Fitting A Model to Data


EasyFit from www.mathwave.com

SSQ Presentation

021209.01

2010 Holl Consulting

Model Summary
Rayleigh is Weibull distribution with m=2 Other values of m produce other probability distributions There are many models; try several that fit All models are wrong. Some models are useful.
George E. Box
SSQ Presentation
021209.01

2010 Holl Consulting

Summary
A defect management program includes
A balanced solution Training & explaining Metrics that are used

SSQ Presentation

021209.01

2010 Holl Consulting

Questions?
Chris Holl
Chris_Holl@hotmail.com
Next Class: Tools for Metrics