Six Sigma in Measurement Systems:

Evaluating the Hidden Factory

Inputs

Operation

OK
Inspect

First Time
Correct

NOT
OK

Rework
Hidden Factory

Scrap

Time, cost, people

Bill Rodebaugh

Director, Six Sigma

GRACE
slide 1

Objectives

The Hidden Factory Concept

Review Key Measurement System metrics including

%GR&R and P/T ratio
Case Study at W. R. GRACE

What is a Hidden Factory?

What is a Measurement Systems Role in the Hidden
Factory?

Measurement Study Set-up and Minitab Analysis

Linkage to Process
Benefits of an Improved Measurement System

How to Improve Measurement Systems in an

Organization
slide 2

The Hidden Factory -- Process/Production

Inputs

Operation

OK
Inspect

First Time
Correct

NOT
OK

Rework
Hidden Factory

Scrap

Time, cost, people

What Comprises the Hidden Factory in a Process/Production Area?
Reprocessed and Scrap materials -- First time out of spec, not reworkable
Over-processed materials -- Run higher than target with higher
than needed utilities or reagents
Over-analyzed materials -- High Capability, but multiple in-process
samples are run, improper SPC leading to over-control

slide 3

The Hidden Factory -- Measurement Systems

Sample
Inputs

Lab Work

OK
Inspect

Production

NOT
OK

Re-test
Hidden Factory

Waste

Time, cost, people

What Comprises the Hidden Factory in a Laboratory Setting?

Incapable Measurement Systems -- purchased, but are unusable
due to high repeatability variation and poor discrimination
Repetitive Analysis -- Test that runs with repeats to improve known
variation or to unsuccessfully deal with overwhelming sampling issues
Laboratory Noise Issues -- Lab Tech to Lab Tech Variation, Shift to
Shift Variation, Machine to Machine Variation, Lab to Lab Variation
slide 4

The Hidden Factory Linkage

Production Environments generally rely upon inprocess sampling for adjustment

As Processes attain Six Sigma performance they begin
to rely less on sampling and more upon leveraging the
few influential X variables
The few influential X variables are determined largely
through multi-vari studies and Design of
Experimentation (DOE)
Good multi-vari and DOE results are based upon
acceptable measurement analysis

slide 5

Objectives

The Hidden Factory Concept

Review Key Measurement System metrics including

%GR&R and P/T ratio
Case Study at W. R. GRACE

What is a Hidden Factory?

What is a Measurement Systems Role in the Hidden
Factory?

Measurement Study Set-up and Minitab Analysis

Linkage to Process
Benefits of an Improved Measurement System

How to Improve Measurement Systems in an

Organization
slide 6

Possible Sources of Process Variation

Observed Process Variation

Actual Process Variation

Measurement Variation

Long-term

Short-term

Variation

Variation due

Variation due

Process Variation

Process Variation

w/i sample

to gage

to operators

Repeatability

Calibration

Stability

Linearity

2 Observed Pr ocess 2 Actua l Pr ocess 2 Measurement System

2 Measurement System 2 Re peatability 2 Re producibility
We will look at repeatability and reproducibility as primary
contributors to measurement error
slide 7

How Does Measurement Error Appear?

Actual process variation No measurement error

LSL

Frequency

15

USL

10

0
30

40

50

60

70

80

90

100

110

Process
15

LSL

USL

10

Frequency

Observed process
variation With measurement error

0
30

40

50

60

70

Observ ed
slide 8

80

90

100

110

Measurement System Terminology

Discrimination - Smallest detectable increment between two measured values

Accuracy related terms
True value - Theoretically correct value
Bias - Difference between the average value of all measurements of a sample and the
true value for that sample
Precision related terms
Repeatability - Variability inherent in the measurement system under constant
conditions
Reproducibility - Variability among measurements made under different conditions
(e.g. different operators, measuring devices, etc.)
Stability - distribution of measurements that remains constant and predictable over time for
both the mean and standard deviation
Linearity - A measure of any change in accuracy or precision over the range of instrument
capability

slide 9

Measurement Capability Index - P/T

Precision to Tolerance Ratio

515
. * MS
P/T
Tolerance

Addresses what percent of the tolerance is taken up by

measurement error
Includes both repeatability and reproducibility

Usually
Usually expressed
expressed
as
percent
as percent

Operator x Unit x Trial experiment

Best case: 10% Acceptable: 30%

Note: 5.15 standard deviations accounts for 99% of Measurement System (MS) variation.
The use of 5.15 is an industry standard.

slide 10

Measurement Capability Index - % GR&R

%R & R

MS

Observed Pr ocess Variation

x 100

Usually
Usually expressed
expressed
as
percent
as percent

Addresses what percent of the Observed Process Variation is

taken up by measurement error
%R&R is the best estimate of the effect of measurement
systems on the validity of process improvement studies (DOE)
Includes both repeatability and reproducibility
As a target, look for %R&R < 30%

slide 11

Objectives

The Hidden Factory Concept

Review Key Measurement System metrics including

%GR&R and P/T ratio
Case Study at W. R. GRACE

What is a Hidden Factory?

What is a Measurement Systems Role in the Hidden
Factory?

Measurement Study Set-up and Minitab Analysis

Linkage to Process
Benefits of an Improved Measurement System

How to Improve Measurement Systems in an

Organization
slide 12

Case Study Background

Internal Raw Material, A1, is necessary for Final Product production

High Impact Six Sigma project was chartered to improve an important quality variable,
CTQ1
The measurement of CTQ1 was originally not questioned, but the team decided to study
the effectiveness of this measurement

Expensive Raw Material to produce produced at 4 locations Worldwide

Cost savings can be derived directly from improved product quality, CpKs
Internal specifications indirectly linked to financial targets for production costs are used to
calculate CpKs
If CTQ1 of A1 is too low, then more A1 material is added to achieve overall quality higher
quality means less quantity is needed this is the project objective

The %GR&R, P/T ratio, and Bias were studied

Each of the Worldwide locations were involved in the study

Initial project improvements have somewhat equalized performance across sites. Small
level improvements are masked by the measurement effectiveness of CTQ1

slide 13

CTQ1 MSA Study Design (Crossed)

Site 1 Lab

Site 1 Sample 1 Site 1 Sample 2

Op 1 Op 2 Op 3
T1 T2

Site 2 Lab

Site 3 Lab

Site 4 Lab

Site 2 Sample 1..

6 analyses/site/sample
2 samples taken from each site
2*4 Samples should be representative
Each site analyzes other sites sample.
Each plant does 48 analyses
6*8*4=196 analyses
slide 14

Gage name:
Date of study:
Reported by:
Tolerance:
Misc:

Z-14 MSA
JULY 2002
All Labs
110

CTQ1 MSA Study Results (Minitab Output)

Surface Area

Components of Variation

Response By Sample

120

%Contribution
%Study Var
%Tolerance

100

Percent

890

80

840

60
790

40
20

740

0
Gage R&R

Repeat

Reprod

Sample

Part-to-Part

R Chart by Operator
Sample Range

100

CB1 CB2 CB3 LC1 LC2 LC3

V1

V2

V3 W1

Response By Operator
W2

890

W3

840
UCL=52.45

50

790
R=16.05
0

LCL=0

740

Oper

CB1 CB2 CB3 LC1 LC2 LC3 V1 V2 V3 W1 W2 W3

Xbar Chart by Operator

CB1 CB2 CB3 LC1 LC2 LC3

V1

V2

V3 W1

Operator*Sample Interaction
W2

UCL=851.5

850

Mean=821.3
800

Operator

900

W3

LCL=791.1

Average

900

Sample Mean

850

800

750

Sample

slide 15

CB1
CB2
CB3
LC1
LC2
LC3
V1
V2
V3
W1
W2

CTQ1 MSA Study Results (Minitab Session)

Source

DF

Sample

SS

MS

14221

2031.62

5.0079

0.00010

Operator

11

53474

4861.27

11.9829

0.00000

Operator*Sample

77

31238

405.68

1.4907

0.03177

Repeatability

96

26125

272.14

191

125058

Total

%Contribution
Source

VarComp

Total Gage R&R

617.39

90.11

Repeatability

272.14

39.72

Reproducibility

345.25

50.39

278.47

40.65

66.77

9.75

67.75

9.89

Operator
Operator*Sample
Part-To-Part

(of VarComp)

slide 16

Sample, Operator,
& Interaction are
Significant

CTQ1 MSA Study Results

*Conf Int not calculated with Minitab, Based upon R&R Std Dev
slide 17

DotplotsResults
of C16 by C17
CTQ1 MSA Study
(Minitab Output)
(group means are indicated by lines)

Dotplot of All Samples over All Sites

890

C16

840

790

slide 18

Site 3

WO SA

Site 2

VF SA

Site 1

LC SA

C17

CB SA

740

Site 4

CTQ1 MSA Study Results (Minitab Session)

Analysis of Variance for Site
Source

DF

SS

MS

37514

12505

26.86

0.000

Error

188

87518

466

Total

191

125032

Site

Individual 95% CIs For Mean

Based on Pooled StDev
Level

Mean

StDev

Site 1

48

824.57

15.38

Site 2

48

819.42

22.11

Site 3

48

800.98

20.75

Site 4

48

840.13

26.58

-+---------+---------+---------+----(---*---)
(---*---)
(---*---)
(---*---)
-+---------+---------+---------+-----

Pooled StDev =

21.58

795

810

825

Site and Operator are closely related

slide 19

840

Per

790

40

740
CTQ1
MSA Study Results (Minitab Output)
0
20

Gage R&R

Repeat

Reprod

Part-to-Part

X-bar
R ofbyAll
Samples for All Sites
R Chart
Operator

Sample Range

100

CB1 CB2 CB3 LC1 LC2 LC3

V1

V2

V3 W1

W2

UCL=52.45

R=16.05
0

LCL=0
0

Discrimination
Index840is 0,
however
can
790
probably see
740
differences
of 5
Oper

CB1 CB2 C

Xbar Chart by Operator

CB1 CB2 CB3 LC1 LC2 LC3

V1

V2

V3 W1

O
W2

900

W3

UCL=851.5

850

Mean=821.3
800

LCL=791.1

Most850of the
samples are
800
seen as
noise
Average

Sample Mean

900

750

Sample

slide 20

890

W3

50

Sample

Pe

50

CTQ10 MSA Study Results (Minitab Output)

Gage R&R

Repeat

Reprod

Part-to-Part

Sample Range

X-bar R ofRAll
Samples
for Site 4
Chart
by WO OP
70
60
50
40
30
20
10
0

W1

W2

W3

UCL=60.99

R=18.67
LCL=0
0

Xbar Chart by WO OP
Sample Mean

900

W1

W2

W3

UCL=875.2
850

Mean=840.1
LCL=805.0

800
0

Mean differences are seen in X-bar area

Most of the samples are seen as noise
slide 21

Sampl

CTQ1 MSA Study ResultsR=17.92

Process Linkage
0
LCL=0
760
Site
2
Example
0
LC1
LC OP
860
850
840
830
820
810
800
790
780

LC1

LC2

LC3

850

UCL=853.1

840

Mean=819.4

LCL=785.7

1000

1
1
1

900

800

4
2
22 4
6

MSA Study
820
Results with
810
800 Mean = 819.4
830

790

I and MR Chart for TSA (t)

Individual Value

LC OP*Sa

Average

Sample Mean

Xbar Chart by LC OP

Sample

11
22
22

55

6 6
662 62
2 22

UCL=899.2
Mean=832.5

2
5

LCL=765.8

700
Subgroup

100

g Range

150
1
100

200

300

400

1 1
Selected
Samples
are Representative
1
1
11
1

11
1

UCL=81.95
slide 22

2002 Historical
Process
Results with
Mean = 832.5

CTQ1 MSA
Study Results Process Linkage
I and MR Chart for TSA (t)
Site 2 Example
810

760

Gage R&R

Sample Range

Individual Value

1000

Reprod

Sample

Part-to-Part

R Chart by LC
1 1OP
1

1
1

100

900

LC1

4
50
800

222 4
6

LC3

22
22

UCL=58.54

1
0

100

860
850
100
840
830
820
50
810
800
790
0
780

LC1

1
11

LC2

11
1

22

MSA Study Results

LCL=765.8with Range = 17.92,
Calc for Subgroup

810

760

300

LC OP

400
LC1

LC2

LC3

LC OP*Sample Interaction
LC3

850

UCL=853.1

2
2

Mean=832.5
2

Mean=819.4

2
2
222

UCL=899.2

R=17.92
LCL=0

200

55860

6 6
662 62
2 22

Xbar Chart by LC
1 OP

150

By LC OP

1LC2

700

Subgroup

MovingMean
Range
Sample

Repeat

2 LCL=785.7
2

840

Average

Perc

50

830
810
800
790

Sample

UCL=81.95

820

R=25.08
LCL=0
1

2002 Historical LC O
L
Process
L
L
Results with
Range = 25.08
Calc
for
pt7 to8 pt
4
5
6

When comparing the MSA with process operation, a large

percentage of pt-to-pt variation is MS error (70%) --- a
back check of proper test sample selection
slide 23

CTQ1 MSA Study Results Process Linkage

Site 2 Example

Key issue for Process Improvement Efforts is When will we see

change?

Initial Improvements to A1 process were made

Control Plan Improvements to A1 process were initiated
Site 2 Baseline Values were higher than other sites
Small step changes in mean and reduction in variation will achieve goal

How can Site 2 see small, real change with a Measurement System with
70+% GR&R?
Use Power and Sample Size Calculator with and without impact
of MS variation. Lack of clarity in process improvement work,
results in missed opportunity for improvement and continued
use of non-optimal parameters
slide 24

CTQ1 MSA Study Results Process Linkage

Site 2 Example
2-Sample t Test

2-Sample t Test

Alpha = 0.05

Alpha = 0.05

Sigma = 22.23

Sample

Target

Actual

Difference

Size

Power

Power

2117

0.9000

530

Sigma = 6.67
Sample

Target

Actual

Difference

Size

Power

Power

0.9000

192

0.9000

0.9011

0.9000

0.9002

49

0.9000

0.9036

236

0.9000

0.9002

22

0.9000

0.9015

133

0.9000

0.9001

13

0.9000

0.9074

10

86

0.9000

0.9020

10

0.9000

0.9188

12

60

0.9000

0.9023

12

0.9000

0.9361

14

44

0.9000

0.9007

14

0.9000

0.9156

16

34

0.9000

0.9018

16

0.9000

0.9091

18

27

0.9000

0.9017

18

0.9000

0.9555

20

22

0.9000

0.9016

20

0.9000

0.9095

Simulated Reduction of Pt to Pt variation by 70% decreases

time to observe savings by over 9X.
slide 25

CTQ1 MSA Study Results Process Linkage

Site 2 Example
Benefits of An Improved MS

Realized Savings for a Process Improvement Effort

More trust in all laboratory numbers for CTQ1

Ability to make process changes earlier with R-bar at 6.67

For A1, an increase of 1 number of CTQ1 is approximately $1 per ton

Change of 10 numbers, 1000 Tons produced in 1 month (832 842)
$1 * 10 * 1000 = $10,000

Previously, it would be pointless to make any process changes within the 22 point
range. Would you really see the change?

As the Six Sigma team pushes the CTQ1 value higher, DOEs and other
tools will have greater benefit

slide 26

Objectives

The Hidden Factory Concept

Review Key Measurement System metrics including

%GR&R and P/T ratio
Case Study at W. R. GRACE

What is a Hidden Factory?

What is a Measurement Systems Role in the Hidden
Factory?

Measurement Study Set-up and Minitab Analysis

Linkage to Process
Benefits of an Improved Measurement System

How to Improve Measurement Systems in an

Organization
slide 27

Measurement Improvement in the Organization

Initial efforts for MS improvement are driven on a BB/GB project basis

Intermediate efforts have general Operations training for lab personnel,

mostly laboratory management

Six Sigma Black Belts and Green Belts Perform MSAs during Project Work
Lab Managers and Technicians are Part of Six Sigma Teams
Measurement Systems are Improved as Six Sigma Projects are Completed

Lab efficiency and machine set-up projects are started

The %GR&R concept has not reached the technician level

Current efforts enhance technician level knowledge and dramatically

increase the number of MS projects

MS Task Force initiated (3 BBs lead effort)

Develop Six Sigma Analytical GB training
All MS projects are chartered and reviewed; All students have a project
Division-wide database of all MS results is implemented

slide 28

Measurement Improvement in the Organization

Develop common methodology for Analytical GB training

slide 29

Final Thoughts

The Hidden Factory is explored throughout all Six Sigma programs

One area of the Hidden Factory in Production Environments is
Measurement Systems
Simply utilizing Operations Black Belts and Green Belts to improve
Measurement Systems on a project by project basis is not the long term
answer
The GRACE Six Sigma organization is driving Measurement System
Improvement through:

Tailored training to Analytical Resources

Similar Six Sigma review and project protocol
Communication to the entire organization regarding Measurement System
performance
As in the case study, attaching business/cost implications to poorly performing
measurement systems

slide 30