Attack The Variance - Hurst-1

Slide- Attack the Variance, Course 1 (Pittcon 2013) 1
Attack the Variance, Course 1

Tools to Understand Variance in Analytical
Methods
Olivier Guise
Ph.D. Chemist
Roger Hurst
Ph.D. Chemist
Pittcon Short Course
March 19, 2013
Section 1 Understanding your data
Evaluating data quality are there potential problems?
Concepts: Mean, standard deviation, precision, accuracy, Z-score
Tools: Hypothesis tests, p-values, Analysis Of Variance (ANOVA), t-
tests, outlier tests, regression analysis

Section 2 Howgoodistheruler?
Where is the variation sample or measurement or both?
Concepts: measured vs. observed variation, effect of interactions,
how to correct variation
Tools: Gage Repeatability and Reproducibility (GRR), fishbone
diagram (cause and effect diagram)

Section 3 Identify sources of variation . . . Then optimize!
Which factors are really important to optimize?
Concepts: Knowing which knobs to turn, which way to turn them,
and how much in order to reach the optimal method.
Tools: Screening Design of Experiment(DOE) & optimization DOE
Pittcon March 2013
Course Overview
Your Names & Backgrounds
StudentsCourseExpectations
Be involved!
Think constantly about how
you are going to use this
when you get back
Ask lots of questions!
Participate in the exercises
they are designed to make
sure you go home with more
than a notebook!
Cell phone courtesy
Bringupyourcase
studiesthatyouwouldlike
help with, from us or from
the group
If you get ahead, please
help your neighbor. We
need to stay together to
get through as much
material as possible, as
quickly as possible.
InstructorsCourseExpectations
Pittcon March 2013
Section 1
Understanding Your Data
SABIC Innovative Plastics
Analytical Technology
Evaluating data quality are there potential problems?
Concepts: Mean, standard deviation, precision, accuracy, Z-score
Tools: Hypothesis tests, p-values, Analysis of Variance (ANOVA),
t-tests, outlier tests, regression analysis
2
1
1
2
1
) (
|
|
|
|
.
|
\
|
=

=
n
x x
s
n
i
i
Mean
1o 2o 3o -1o -2o -3o
1o 68.20 % of all data
2o 95.54 % of all data
3o 99.7% of all data
Process Variability -
multiple results from
the same process
Normal
Distribution
(Gaussian)

-4o -5o -6o 4o 5o 6o

6o 99.99999980% of all data

Standard Deviation
Accurate
not precise
Address
variation
issues
Target Analogy
Corrective Action:
Precise,
Not accurate
Calibration to
correct bias
True value
Precise and
accurate
Ideal
Precision vs. Accuracy
True value
True value
Poor
Process
Capability
Measurement
Process
LSL USL
Target
LSL USL
Target
Excellent
Process
Capability
LSL USL
Target
SoHOWgoodisgoodenough??
Measurement
Process
Measurement
Process
Mean Process
Variability -
multiple results
from the same
process
When you have a
specified tolerance:
If the limits fall on the
vertical lines, the Z
score is . . .
1o 2o 3o -1o -2o -3o

-4o -5o -6o 4o 5o 6o
Z = 1
Z = 2
Z = 3
Z = 4
Higher Z is
better!
Z Score: A Measure of Variance Relative to Tolerance
Z =
USL - x
s
observed
Z =
12.0 11.0
0.4

= 2.5
Thismethod
is2.5Sigma.
Z =
12.0 7.0
2.0

= 2.5
Thismethod
is2.5Sigma.
LSL USL
x =7.0
s
obs
= 0.4
Z =
12.0 7.0
0.4

= 12.5
Thismethod
is12.5Sigma.
(Use spec limit nearer to the
mean . . . may be LSL)
Process
LSL USL
Target
x =11.0
s
obs
= 0.4
12.0
2.0
7.0
LSL USL
x =7.0
s
obs
= 2.0
Is it Good Enough? . . . Estimate Z Score
Minitab Graphical Summary:
Agoodwaytogetanoverallfeelforyourdata
Minitab Descriptive Statistics:
Agoodwaytogetanoverallfeelforyourdata
54.0 52.5 51.0 49.5 48.0 46.5 45.0
Median
Mean
50.0 49.5 49.0 48.5 48.0 47.5 47.0
1st Quartile 46.958
Median 47.930
3rd Quartile 49.852
Maximum 53.570
46.878 50.238
46.905 49.972
1.616 4.288
A-Squared 0.31
P-Value 0.496
Mean 48.558
StDev 2.349
Variance 5.517
Skewness 1.00719
Kurtosis 1.26643
N 10
Minimum 45.390
Anderson-Darling Normality Test
95% Confidence Interv al for Mean
95% Confidence Interv al for Median
95% Confidence Interv al for StDev
95% Confidence Intervals
Summary for Masses
Ho = the null hypothesis (no difference)
Ha = the alternative hypothesis (difference)
True State Test Conclusion
Not Guilty Guilty
Not Guilty Acquit Send to Jail
Guilty Acquit Send to Jail
True State Test Conclusion
In Spec Out of Spec
In Spec Pass Fail
Out of Spec Pass Fail
Which Error is worse?
considertheliability
Compare sample
mean and
population of data,
two sample means,
two sample std.
deviations, etc.
Hypothesis Testing
Hypothesis Testing (t-test used to compare means):
Ho: X
bar-A
= X
bar-B
Ho: X
bar-B
= X
bar-C
Ho: X
bar-A
= X
bar-C

Ha: X
bar-A
= X
bar-B
Ha: X
bar-B
= X
bar-C
Ha: X
bar-A
= X
bar-C

S
tech #1
= 3 S
tech#2
= 1
How do we determine if these are different?
Technique #2 Technique #1
A B C
A B C
t-distribution (Student t)
Degrees of
Freedom (df)
0.10 0.05 0.005
1 3.078 6.314 63.657
5 1.476 2.015 4.032
10 1.372 1.812 3.169
t
critical
o = 0.05
Step 1. Find t
critical

df = n-1
Where n
is # of
samples
T-critical: Determining if two means are different
t-distribution (Student t)
t
critical
n
s
x
tcalc

= o error
If t
calc
> t
critical
reject Ho, accept Ha
There is a difference between means
Ho: X
bar-A
= X
bar-B
Ha: X
bar-A
= X
bar-B
Step 2. Determine t
calc

t
calc
BUTwecannotrunt-tests every time we do
ananalysis(toomanyfactors!!!),SO.
T-Calculated: Determining if two means are different
Hypothesis Testing

Breakout #1A
Data File: T-test F-test.xls
Instructions:
1. Open the file, T-test F-test.xls
2. Copy and Paste into Minitab.
3. Stack Columns to make one column.
4. Determine if there is a difference in mean value of the groups
First, test for normality
Use Homogeneity of Variance to compare the variance
Use t-test to compare the means
Chemicoolest, Inc. has developed a new mass flow-meter for
industrial process control. Two operators have run experiments
under the same settings to test the system and operability. The mass
dataarestoredinthedatafile,T-test F-test.xls.UseMinitabto
asses the normality, variance, and mean response for this
experiment.
Breakout #1A
Stack columns
Breakout #1A Screen Shots
Check Normality for Each Operator
(or Subset) Separately

(Normal if p > 0.05)
Accept Null Hypothesis: Normal

Test for Homogeneity of Variance
Reject Null Hypothesis: Unequal Variances

F-test is for
normal data
Levenestestis
for non-normal
data
2-sample t-test compares the
means of 2 distributions
Do not assume equal variance.
F-test showed unequal variance.
2-sample t-test compares the
means of 2 distributions
Accept Null Hypothesis: Means Appear Equal

Hypothesis Testing

Breakout #1B
Data File: Hypothesis Testing Class Example.xls
Instructions:
1. Open the file, Breakout #1B Class Example.xls
2. Copy and Paste into Minitab.
3. Stack Columns to make one column.
4. Determine if there is a difference in mean value of the groups
First, test for normality
Use Homogeneity of Variance to compare the variance
Use t-test to compare the means
You just developed a new FTIR method for measuring co-polymer
composition. You and another colleague decide to gauge the method
by running some replicates and comparing the results. The mass
dataarestoredinthedatafile,Breakout#1BClassExample.xls.
Use Minitab to asses the normality, variance, and mean response for
this experiment.
Breakout #1B
Breakout #1B Screen Shots
Need to investigate more . . .

p > 0.05, normal
p < 0.05, Uh-oh
All data are normal when
analyzed together.

What can cause this?

p > 0.05, normal
All data, both
operators
Descriptive statistics shows
a possible outlier.

What would you do?
54.0 52.5 51.0 49.5 48.0 46.5 45.0
Median
Mean
50.0 49.5 49.0 48.5 48.0 47.5 47.0
1st Quartile 46.958
Median 47.930
3rd Quartile 49.852
Maximum 53.570
46.878 50.238
46.905 49.972
1.616 4.288
A-Squared 0.31
P-Value 0.496
Mean 48.558
StDev 2.349
Variance 5.517
Skewness 1.00719
Kurtosis 1.26643
N 10
Minimum 45.390
Anderson-Darling Normality Test
95% Confidence Interv al for Mean
95% Confidence Interv al for Median
95% Confidence Interv al for StDev
95% Confidence Intervals
Summary for Masses
LookingatLevenesTest(since
we had one non-normal data
set), we see that the variances
are not statistically different.

Operator B
Operator A
10 8 6 4 2
O
p
e
r
a
t
o
r
s
95% Bonferroni Confidence Intervals for StDevs
Operator B
Operator A
54.0 52.5 51.0 49.5 48.0 46.5 45.0
O
p
e
r
a
t
o
r
s
Masses
Test Statistic 0.71
P-Value 0.746
Test Statistic 0.06
P-Value 0.818
F-Test
Levene's Test
Test for Equal Variances for Masses
Given the data, we could
accept the null
hypothesis if the data
were normal. . . . How
can we be more
certain??

Multiple measurements (n>3) were taken on a given sample. One appears
to deviate wildly, hence could significantly influence the mean, stdev, and
distribution. Can I remove this point with 90% confidence?
30.38
30.23
30.34
29.98
30.29
30.31
29.7
29.8
29.9
30
30.1
30.2
30.3
30.4
30.5
1 2 3 4 5 6
Measurement No.
C
o
n
t
e
n
t

(
%
)
ApplyaGrubbsTest:
t.v. =
|29.98 30.26|
0.1438
= 1.947
Comparet.v.totestGrubbstable
t.v. =
(Grubbs
test value)
|x
i
x|
stdev
Outlier Tests WhencanIthrowoutabadresult?
In this case, n = 6 for a one-
tailed distribution.
G(6, 90) = 1.729
Since 1.947 > 1.729
the measurement is an
outlier.
P(one-tailed)
P(two-
tailed)
GrubbsTable
Words of caution with outlier tests:
Apply to multiple measurements of the same
sample, not single measurements of a set of
samples.
First, try to identify and resolve the cause of the
outlier.
If you have an outlier and need to report a
single value to represent the result, use the
median value, rather than the mean.
Q
obs
=
Suspect value nearest value
Range
If Q
obs
> Q
crit
, then you
can reject it.
Q-Test Another type of outlier test
Example:
A set of samples were made
spanning a range of different
compositions.
Each sample was measured
once.
The measured values were
plotted vs. the theoretical
values (formulation)
Is this due to measurement variation or a bad sample?
Can it be thrown out? . . .
Use a standardized (studentized) residual test.
Validation Set: Measured vs.
Theoretical
y = 0.9942x + 0.1922
R
2
= 0.9468
18.0
18.5
19.0
19.5
20.0
20.5
21.0
21.5
22.0
18.0 19.0 20.0 21.0 22.0
Theoretical Content (%)
M
e
a
s
u
r
e
d

C
o
n
t
e
n
t

(
%
)
|y
actual
y
fit
|
=residual
Standardized Residuals How to treat outliers in a
set of regression data
Strategy:
Use Minitab to plot the
standardized residuals.
Any residual > 3 is outside the
99.7% confidence interval and
can be considered an outlier.
Investigate and resolve the
reason for the outlier if
necessary.
Class Exercise:
standardized residuals.mpj
Standardized Residuals contd
26 24 22 20 18 16 14 12 10 8 6 4 2
4
3
2
1
0
-1
-2
Observation Order
S
t
a
n
d
a
r
d
i
z
e
d

R
e
s
i
d
u
a
l
Versus Order
(response is Measured)
well over 3 . . .
outlier
Tip - Itsstillrecommendedtoresolvewhythesamplewas
badthantosimplyrejectitandforgetaboutit.
Standardized Residuals contd
This tool will determine if the response means associated with the groups are
from the same population. (Used when data sets vary only in one factor).

H
o
: m
1
= m
2
= m
3
= m
4
H
a
: At least one m different from the others
If p > 0.05 then accept H
o

If p < 0.05 then
One of the means is different than the others.
One-Way ANOVA: Used for Single-Factor Studies
Grand mean
diameter
Measure
multiple
diameters for
each circle.
(Distribution
results from
measurement
error may be
instrument and/or
technique.)
The difference
WITHIN each circle
is small relative to
the difference
BETWEEN circles.
. . . So one circle is
a different
diameter.

Thisdoesntrequire
ANOVA . . . But what
if we add variation to
the groups?
Example:Asinglemeasurement(diameter)andasinglefactor(circle#).
Measure
multiple
diameters for
each shape.
(Distribution of
results comes
from
measurement
error and actual
changes in
diameter.)
Grand mean
diameter
Now it
helps to
have
ANOVA
One-way Analysis of Variance

Source DF SS MS F P
Variety 5 2.43507 0.48701 56.22 0.000
Error 18 0.15593 0.00866
Total 23 2.59100
Tabular Evaluation of 1-way ANOVA Output
Accept Ha: Factor Is Significant
TSS = SSW + SSB
Total Sum of Squared Differences
Sum of Squares Within Group
Sum of Squares Between Group
One-Way ANOVA: Interpretation of Minitab Results
Factor 2
Sample 2
SSB1 = S
abc
(for all levels of factor 1)
SSB2 = the same for all levels of factor 2
SSE is the total sum of squares error
F
a
c
t
o
r

1

Oper-
ator 1
2,1
1,2 1,1
2,2
3,1
3,2
TSS = SSB1 + SSB2 + SSE
Ybar overall mean
ANOVA: a method of
determining the differences in
means based on the variation
in measuring those means.
Used when 2 factors are varied:
Such as operator and sample
Oper-
ator 2
Sample 1
Sample 2
Sample 3
Sample 3
Avg of 3,1 & 3,2
c
Sample 1
Avg of 1,1 & 1,2
a
Avg of 2,1 & 2,2
b
Two-Way ANOVA: Graphically
ANOVA

Breakout #2
Data Files:
One-Way ANOVA.xls
Two-Way ANOVA.xls
Breakout 2A: One-Way ANOVA
Open data tab 2A in Breakouts_1.xls
Copy/paste from Excel to Minitab
Stack Columns into one column
Analyze One-Way ANOVA
What can you conclude about the means?

Breakout 2B: Two-Way ANOVA
Open data tab 2B in Breakouts_1.xls
Copy/paste from Excel to Minitab
Stack Columns into one column
Analyze Two-Way ANOVA
Are the means the same in terms of Operator? In terms of
Sample?

Breakout #2: One-Way ANOVA and Two-WAY ANOVA

1. Copy/paste from Excel
2. Analyze ANOVA
3. Choose Plots
Breakout #2A One-Way ANOVA Screen Shots
Mean Square (MS)
(Sum of Squares/df)
F value
(MS
Factor
/MS
Error
)
p is probability of an
alpha error; NOT
significant if p>0.05
Reject Null
Hypothesis:
Unequal Means

Breakout #2A: One-Way ANOVA Screen Shots
Operator 3 Operator 2 Operator 1
34.850
34.825
34.800
34.775
34.750
D
a
t
a
Boxplot of Operator 1, Operator 2, Operator 3
1. Copy/paste from Excel
2. Stack Columns into one column
3. Analyze ANOVA
Breakout #2B: Two-Way ANOVA Screen Shots
Samples are different
Operators are the same
Breakout #2B: Two-Way ANOVA Screen Shots
Operators
sample
Percent Acid Operator 2 Percent Acid Operator 1
4 3 2 1 4 3 2 1
2.00
1.75
1.50
1.25
1.00
0.75
0.50
D
a
t
a
Individual Value Plot of Data vs Operators, sample
Pittcon March 2013
Section 2
Howgoodistheruler?
Where is the variation sample or measurement or both?
Concepts: measured vs. observed variation, effect of
interactions, how to correct variation
Tools: Gage Repeatability and Reproducibility (GRR),
fishbone diagram (cause and effect diagram)
Inputs
Observations
Measurements
Data
Process
Outputs

Measure-
ment
Process

Inputs Outputs
2
t Measuremen
2
Process
2
Total
+ =
Possible Sources of Variation
Long-
term
Process
Variation
Actual Process Variation
Measurement Variation
Observed Process Variation
Short-
term
Process
Variation
Variation due
to instrument
Variation
due to
operator
Accuracy
(Bias)
Precision
(Measurement
Error)
Process
Variability
Does Sample A = B = C??
A B C
Technique #2 Technique #1
s
process
= 5
s
tech #1
= 3 s
tech#2
= 1
What about the measurement system?
A B C
A B C
How do I know where the variability is originating?
2
ility Reproducib
2
ity Repeatabil
2
t Measuremen
o o o + =
2
t Measuremen
2
Process
2
Observed
o o o + =
What about the measurement system?
Long-
term
Process
Variation
Actual Process Variation
Measurement Variation
Observed Process Variation
Short-
term
Process
Variation
Variation due
to instrument
Variation
due to
operator
Accuracy
(Bias)
Precision
(Measurement
Error)
GR&R
Measurement System
Interpretation
GR&R s 10% Very capable
10% < GR&R s 20% Capable
20% < GR&R s 30% Marginally capable
GR&R > 30% Not capable
% 100
) ( 15 . 5
% x
T
s
GRR
meas
=
% 100
) (
) ( 57 . 2
% x
SL x
s
GRR
meas
=
Two-sided spec (T=USL-LSL)
One-sided spec:
%GRRdescribeswhat%ofyourworkingrangeistakenupby
variability in your measurement system. The lower the %GRR, the
better you can distinguish actual process movement (often what your
customers are after) from measurement variability.
Meas.
StdDev
GRR indicates method precision . . .
not accuracy (see note)
Gauge Repeatability & Reproducibility (%GRR)
PolyStat, a polymer formulator, is analyzing a polymer sample for an
additive using NIR. They believe that they have optimized the method
and now want to perform a gauge R&R.
The two operators who will be running the test are asked to run the
gauge samples.
Manufacturing has said that the upper spec is 0.8% and the lower
spec. is 0.3%. They have also indicated that the standard deviation of
the process is 0.3%.
The Black Belt has picked samples she feels are representative of the
process and has told the operators how the samples are to be run.
TheresultsareintheExcelfileGRRExample.xls.
Using Minitab, calculate the GRR based on the process capability and
the tolerance. Is this a good method?
If so, why? If not, why not and what can you do to fix it?
Breakout #3: GRR
OpenthefileGRRExample.xls
Copy the .xls data into Minitab
Stack the data using the stack function in Minitab
Perform a GR&R using the Stat feature
Use the ANOVA approach, enter the process variance
(6 x 0.3 = 1.8)
Howgoodisthemethod?
Breakout #3: Instructor will demonstrate.
Breakout #4: Students will practice.
Instructions for Breakout Problem #3
Have to stack columns before GRR
analysis in Minitab
Breakout #3: GRR Screen Shots
Operator 1 Operator 2 Sample Replicate
0.49 0.48 1 1
0.49 0.45 1 2
0.45 0.49 1 3
0.55 0.55 2 1
0.57 0.59 2 2
0.62 0.63 2 3
0.23 0.21 3 1
0.23 0.21 3 2
0.25 0.24 3 3
0.5 0.55 4 1
0.49 0.53 4 2
0.48 0.48 4 3
1. Duplicate Samples
2. Set up GRR
analysis
Enter Process variation
Whatsthis?
Wellgettothatlater..
Part-to-Part Reprod Repeat Gage R&R
100
50
0
P
e
r
c
e
n
t
% Contribution
% Study Var
% Process
0.10
0.05
0.00
S
a
m
p
l
e

R
a
n
g
e
_
R=0.0462
UCL=0.1191
LCL=0
Operator 1 Operator 2
0.6
0.4
0.2
S
a
m
p
l
e

M
e
a
n
_
_
X=0.4483
UCL=0.4957
LCL=0.4010
4 3 2 1
0.6
0.4
0.2
Sample
0.6
0.4
0.2
Operator
4 3 2 1
0.6
0.4
0.2
Sample
A
v
e
r
a
g
e
Operator 1
Operator 2
Operator
Gage name:
Date of study :
Reported by :
Tolerance:
Misc:
Components of Variation
R Chart by Operator
Xbar Chart by Operator
Data by Sample
Data by Operator
Operator * Sample Interaction
Gage R&R (ANOVA) for Data
1. Good gauge
relative to
part variation
2. Repeatability
is most of
gauge error
3. No interaction
b/w operator
and part
4. Part 3 is low

Interpretation
Breakout #3: GRR Graphical Results
Now enter process tolerance (USL-LSL)
Breakout #3: Screen Shots
Part-to-Part Reprod Repeat Gage R&R
160
80
0
P
e
r
c
e
n
t
% Contribution
% Study Var
% Process
% Tolerance
0.10
0.05
0.00
S
a
m
p
l
e

R
a
n
g
e
_
R=0.0462
UCL=0.1191
LCL=0
0.6
0.4
0.2
S
a
m
p
l
e

M
e
a
n
_
_
X=0.4483
UCL=0.4957
LCL=0.4010
4 3 2 1
0.6
0.4
0.2
Sample
0.6
0.4
0.2
Operator
4 3 2 1
0.6
0.4
0.2
Sample
A
v
e
r
a
g
e
Operator 1
Operator 2
Operator
Gage name:
Date of study :
Reported by :
Tolerance:
Misc:
Components of Variation
R Chart by Operator
Xbar Chart by Operator
Data by Sample
Data by Operator
Operator * Sample Interaction
Gage R&R (ANOVA) for Data
Measurement gauge R&R
is low relative to the
tolerance (i.e. specs)
Repeatability
error takes up
more of the
tolerance than
reproducibility.
The parts chosen were
well outside the
tolerance and the
method was capable of
differentiating them.
Breakout #3: GRR Graphical Results
What does
this tell us?
Focus on the p-value:
p<0.05 means the
source is significant
GRR Breakout #3: Results in Minitab Session Window
Part-to-part variability
accounts for 158% of
the tolerance
Std. dev. parts/ Std. dev. gauge *1.41
Measurement gauge variability
accounts for 27.13% of the tolerance
GRR Breakout #3: Results in Minitab Session Window
PolyStat is trialing some manufacturing changes. The preliminary data
indicate that they are now making better product, based on weathering
and tensile data.
The upper spec is 0.8% and the lower spec. is still 0.3%. They have
indicated that the standard deviation of the process is now 0.25%.
The same two operators are asked to run the GRR experiment with a
newsetofsamples.TheirresultsareintheExcelfileGRR Example
2.xls.
Using Minitab, calculate the GRR based on the process capability and
the tolerance. Is this a good method?
If so, why? If not, why not and what can you do to fix it?
Breakout #4: GRR
Fishbone Diagram
FTIR & NIR
IR Parameters
People
Materials
Measurements
Environment Methods Machines
Mole% X
Temperature
Film Press
Sample Placement

Standard
Quality
Contamination
Spectral Regions
Humidity
Sample Prep
Sample Loading
Multitasking
Spectral Quality
Statistical Model
Homogeneity
A great way to brainstorm potential factors that may affect the result
A cross-functional effort, allow enough time
Include all input, whether known or only suspected to affect the result
Better to have too many factors than too few . . . DOE will help screen
Example Fishbone diagram
for FTIR compositional test
method
Pittcon March 2013
Section 3
Identify sources of variation . . .
Then optimize!
Introduction to the Design of Experiment (DOE)
What is a DOE? When is it appropriate?
Visualizing the Experimental Space
- Factors and Levels, Coding
Importance of Replication and Randomization
Importance of Orthogonality and Blocking

Types of DOEs
ScreeningandOptimization
- Full and Fractional Factorial DOEs
Appendix
Second Order Designs
Additional Break-out Examples

Section 3: Outline
Box, Hunter, and Hunter. Statistics for Experimenters. John Wiley and Sons, Inc., 1978.
Challenge: Optimize yield of chemical reaction
Reaction time (t)
Reaction temperature (T)
One-at-a-time approach:
maximum yield of 75 grams at 130 min and 225C
One-at-a-Time Approach
Box, Hunter, and Hunter. Statistics for Experimenters. John Wiley and Sons, Inc., 1978.
- Time consuming
- Expensive testing (time and supplies)
- Noconsiderationofvariableinteractions
When variables are changed simultaneously.
Global Maximum:
Yield of 91 grams
65 min and 255C
Local Maximum:
yield of 75 grams
130 min and 225C
One-at-a-Time Approach
Structured process, formal plan
Investigates the relationship between input and output
factors.
One-at-a-time approach of controlling independent variables
and observing dependent variables

By contrast, what is a DOE?

Experimental methodology
Multiple independent variables are controlled simultaneously
Structured experiments are conducted in a prescribed way
Efficient and accurate analysis which determines the
significance and/or the mathematical relationships of factors
to a measured output.
What is Experimental Design?
If preliminary work or expert knowledge reveals an
obvious cause and a solution exists.

If:
Root cause(s) cannot be found
Further improvement is desired, after removing
root causes
Many potential factors affect the response
You wish to quantify the relationship between the
factors and the response
Is a Designed Experiment Appropriate?
NO
YES
Challenge:
How can we maximize
reaction yield?
X + Y Z

Experimental variables from
team brainstorming:
Factor A = Temperature
Factor B = Time
Factor C = pH
125C
50C
15min 5min
Factor B
Factor A
4
125C
50C
15min
8
5min
Factor B
Factor A
Factor C
2 Factors, 2 Levels = SQUARE
3 Factors, 2 Levels = CUBE
Full Factorial:
2^ 2 = 4 runs
Full Factorial:
2^ 3 = 8 runs
# Levels^
#Factors
= # Runs
+1
-1
+1 -1
Factor B
Factor A
-1
+1
-1
+1
+1
-1
Factor B
Factor A
Factor C
Fractional Factorial
Full Factorial
What information can we obtain using factorial designs?
Main effects of factors (X
i
) and Effects of interactions (X
i
* X
j
)
Variability associated with each factor and interaction
Effects of experimental error
Predictive transfer functions, Example: y = b
1
(X
i
) + b
2
(X
j
) + b
3
(X
i
*X
j
)
2) Optimization DOE To validate screening DoE, determine
optimum input variables, characterize the input/output response surface
(check for curvature) by adding levels or using different designs.
What kinds of Experimental Designs are there?
1) Screening DOE Identify statistically significant variables
Interactive Exercise: Fishbone (Cause & Effect)
Diagram
Means of organizing brainstorming ideas on the possible causes of
some stated outcome into major categories
Put down EVERYTHING youllfilterthelistlater
Human Materials Machine
Method Nature Measurement
Outcome
Several aspects of DOE design ensure the DOE analysis results
in statistically valid and practically valid conclusions:

Replication

Randomization

Blocking

Balance & Orthogonality

Requirements for DOE Design
In order to determine statistical significance of main effects
and interaction factors, the magnitude of these effects are
compared to the experimental error present in the system.

In order to estimate the error, experimental runs are replicated.

Where to add replicates? Where you want the most
information:

Factor A
Factor B

+
+

+
Factor C
Corner-point replication
Provides error data across design
space
Keeps experiment balanced
Factor A
Factor B
+
+

+
Factor C
Center-point replication
More information about that area
of the design space
Higher number of replicates
reduces confidence interval around
the estimate of error at that point
Replication and Experimental Error
1
2
3
4
5
6
7
8
9
10
11
12
-1
+1
-1
+1
-1
+1
-1
+1
-1
+1
-1
+1
-1
-1
+1
+1
-1
-1
+1
+1
-1
-1
+1
+1
-1
-1
-1
-1
+1
+1
+1
+1
-1
+1
+1
-1
Std
Order
Factor
A
Factor
B
Factor
C
(Replicate Runs)
If the Run Order = Std Order,
then any uncontrolled factor that
varies with time (i.e. noise) will
effect response.
Example:

In an LC method, the temperature in the
laboratory can effect the chromatographic
peak resolution.

If the runs are not randomized, then the
temperature effect, which is not a
controlled factor, can effect the DOE.

Also, if all the replicates are run at the
end, then the experimental noise is not
representative of the true noise level.
The runs should be randomized.
Replicates should be spread randomly
throughout the experiment.
Randomized
Run Order
8
3
12
10
1
5
7
2
9
4
6
11
Factor A
Factor B
+
+

+
Factor C
Randomization is a MUST!
What is Blocking?
Systematic way of dividing the runs into subsets. Maximum number
of runs that can be made under homogenous conditions.
Why Block?
If there are factors which are believed to affect the response but are
of no interest to the experimenter, these factors can confound the
results and produce erroneous conclusions.
Byassigningablocktoeachlevelofthisnuisancefactor,theeffectof
this factor can be isolated from those of interest. (Time- shift, days,
etc.)
When to Block?
When there is a significant risk of a nuisance factor confounding the
experimentalresults..Withoutreplicationoftheseblocks,however,
there is no ability to test for the significance of the blocking factor.
The cost of blocking is the ability to determine higher-order effects.
Only block in screening DOEs- disregarding a source of variation will
not result in a robust design.
Blocking
Run Order A B AB
1 (1) -1 -1 1
2 a 1 -1 -1
3 b -1 1 -1
4 ab 1 1 1
Balanced
0 for each factor sum
This feature helps to simplify the analysis
X
i E
=
X
i Balance and Orthogonality
Ensure uniform information
throughout the design space
Ensure independent estimates
of factor effects
These conditions are satisfied
for full factorial designs
(Adapted from Mikel J. Harry. The Vision of Six Sigma. 1994. Page 18.9)
Unbalanced
replication
Not orthogonal
Either missed level
or operational space
was constrained.
Full Factorial DOE: Balance and Orthogonality
Orthogonal
This feature ensures the effects are independent
0 for all dot product pairs
E
=
X X
j i
Number
of Factors
Number
of Runs
1
2
3
4
5
6
.
.
.
10
2
4
8
16
32
64
.
.
.
1,024
Cost
$
$$
$$$
$$$$
$$$$$
$$$$$$
.
.
.
$$$$$$$$$
Highest order
interaction
1
2
3
4
5
6
.
.
.
10
Number of 3-
way interactions
0
0
1
4
9
16
.
.
.
119
Number of 2-way
interactions
0
1
3
6
10
15
.
.
.
45
Question to consider:
Do we have the time, money and resources
to run all these experiments, plus
replicates?
Usually-NO!!
2 ^ Factors = # Runs
Full Factorial DOE: Cost vs. Benefit
FractionalFactorialDesignsforScreeningDOEs
1
2
3
4
5
6
7
8
-1
+1
-1
+1
-1
+1
-1
+1
-1
-1
+1
+1
-1
-1
+1
+1
-1
-1
-1
-1
+1
+1
+1
+1
Std
Order
Factor
A
Factor
B
Factor
C
Full Factorial 2
3
= 8 runs
Which runs do you choose to run? (or NOT?)
Remember what information
you get from a full factorial
Grand mean: Y-bar
Main effects: A, B, C
2-Way interactions: AB, AC, BC
3-Way interaction: ABC
ABC
-1
+1
+1
-1
+1
-1
-1
+1
Run only the runs for which ABC is -1
3-Way interaction terms are probably
NOT statistically significant
Half Fraction Factorial Design:
2
k-1
= 2
3-1
= 2
2
= 4 runs
Half Fraction design utilize half the
number of runs of Full Factorial
Designs
Screening DOE: Fractional Factorial
Factor A
Factor B

+
+

+
Factor C
1) Identify number of factors

2) Identify levels of each factor

3) Calculate # runs for a full factorial
# Full Factorial Runs = Levels ^ Factors

4) Select appropriate resolution
Which interactions can you tolerate?

5) Plan DOE using Minitab
or other software

6) Finally... RUN EXPERIMENTS !!
Steps to designing a fractional factorial DOE:
Green = Res V+ Yellow = Res IV Red = Res III
Designs
2
3-1

2
4-1

2
5-1

2
5-2

2
6-1
2
6-2
2
6-3
Runs
4

8

16
8
32
16
8
Alias Structure
C = AB

D = ABC

E = ABCD
D = AB, E = AC
F = ABCDE
E = ABC, F = ACD
D = AB, E = AC, F = BC
Resolution
III

IV

V
III
VI
IV
III
Examples of fractional factorial designs
# Runs = Levels^ (Factors p)
Factorial then p = 1
Resolution of Fractional Factorial Designs - Minitab
Fractional Factorial DOE
Construction

Breakout #5
Water, extracted anions
Methylene chloride,
dissolved polymer
Current Method: Volatile Chloride in Polymers
Heat 5g sample to 300C
Collect vapor in H
2
O
Analyze by Ion Chromatography
Proposed Method: Water Extraction of Anions
Shake 2.5g sample in 20mL of CH
2
Cl
2
until dissolved
Add 15mL H
2
O & shake to extract
Allow emulsion time to settle
Analyze concentration in final
solution by Ion
Chromatography
Heated Polymer
Water,
volatile chloride
N
2

Breakout #5: Ion Chromatography Example
72% GRR!!
Instructor will demonstrate how to set up DOE.
Then students will create the DOE
Class will solve DOE together

1. Open Minitab

2. Create Factional Factorial DOE (2-Levels, 5-Factors)
with 1 center point, 1 block and 2 replicates at the corners
3. LabelXs:mass,volume,shake1,shake2,settle
4. OpenfileBreakout ppm Cl factorial doe.MPJinMinitab

5. Analyze factorial design
Graph: regular residuals with a = 0.05
6. Identify any significant interactions (check p values)
7. Investigate ANOVA Main Effects and Interaction plots
Breakout #5: Fractional Factorial DOE Construction
Breakout #5: Create Factorial DOE (Screen Shots)
Replicates let you estimate the error across design
space to help show whether factors are significant
Breakout #5: Create Factorial DOE (Screen Shots)
Add factor labels
After running experiments,
add the responses
Breakout #5: Enter Results (Screen Shots)
Breakout #5: Analyze Factorial DOE (Screen Shots)
Terms
Graphs
This problem has
been worked out
before,anditsknown
not to have any 3-way
interactions. To save
time in class, choose
order of 2.
Breakout #5: Analyze Factorial DOE (Screen Shots)
30 25 20 15 10 5 1
3
2
1
0
-1
-2
-3
Observation Order
S
t
a
n
d
a
r
d
i
z
e
d

R
e
s
i
d
u
a
l
Versus Order
(response is ppm Cl)
3 2 1 0 -1 -2 -3
99
95
90
80
70
60
50
40
30
20
10
5
1
Standardized Residual
P
e
r
c
e
n
t
Normal Probability Plot
0.9 0.8 0.7 0.6 0.5 0.4
3
2
1
0
-1
-2
-3
Fitted Value
S
t
a
n
d
a
r
d
i
z
e
d

R
e
s
i
d
u
a
l
Versus Fits
2.4 1.2 0.0 -1.2 -2.4
9
8
7
6
5
4
3
2
1
0
Standardized Residual
F
r
e
q
u
e
n
c
y
Histogram
No systematic variation
during DOE
Data are normally
distributed
Residuals are
indicative of noise
and therefore should
be normally
distributed and
should not exhibit
systematic variation
with time.

Patterns in the
residuals plots may
indicate a critical
factor that is not
being controlled!

Or that data
transformation
may be needed (i.e.
Log(x))
Points falling on the line -
normally distributed
Lookfortrumpet
heteroscedastic data
Analyze Factorial DOE - Residuals Plots
CE
BE
AC
E
BC
D
C
AD
BD
AB
DE
AE
CD
B
A
12 10 8 6 4 2 0
T
e
r
m
Standardized Effect
2.11
A mass
B v olume
C shake1
D shake2
E settle
Factor Name
Pareto Chart of the Standardized Effects
(response is ppm Cl, Alpha = 0.05)
The dotted line is
the boundary
between significant
(above the line) and
insignificant (below
the line) according
to alpha.
Quick Visual Confirmation of Significant Factors
Pareto Chart of Main Effects
Go to Stat >> DOE >> Factorial >> Factorial Plots . . .
Click Setup to select plot type and terms.
Main Effects Plots
1 -1
0.75
0.70
0.65
0.60
0.55
1 -1 1 -1
1 -1
0.75
0.70
0.65
0.60
0.55
1 -1
mass
M
e
a
n
volume shake1
shake2 settle
Main Effects Plot for ppm Cl
Data Means
Interaction Plots
1 -1 1 -1 1 -1 1 -1
0.80
0.65
0.50
0.80
0.65
0.50
0.80
0.65
0.50
0.80
0.65
0.50
mass
volume
shake1
shake2
settle
-1
1
mass
-1
1
volume
-1
1
shake1
-1
1
shake2
Interaction Plot for ppm Cl
Data Means
Go to Stat >> DOE >> Factorial >>
Factorial Plots . . .

Click Setup to select plot type
and terms.
P < 0.05 indicates a
significant factor
Hmmmdoesthismake
sense with what we know
from experience?
Shake1 * Settle is least
likely to affect the ppm Cl
DOE Session Window
Evaluation of Fit
So . . . How do we drill down
to the important factors?
Still
significant
factors
Least significant term . . .
Eliminate it next!
p-values MAY change each time a
term is eliminated.
DOE Analysis Elimination of Insignificant Terms
D
C
CD
B
A
12 10 8 6 4 2 0
T
e
r
m
Standardized Effect
2.06
A mass
B v olume
C shake1
D shake2
Factor Name
. . . After eliminating insignificant terms
significant
factors
Good! No
significant
lack of fit.
Transfer Function (in coded units):
ppm Cl = 0.6530 (0.1022 x mass) + (0.0728 x
volume) (0.0197 x shake1 x shake2) + (0.0078
x shake1) (0.0078 x shake2)
Why do we
need these?
Elimination of Insignificant Terms contd
Factor
Low
Setting
High
Setting
mass 2.5 7.5
volume 10 20
shake1 10 30
shake2 10 30
settle 5 15
If the actual low, high settings are . . .
Put into new
worksheet
Solve the Transfer Function in Uncoded Units
Factor levels now in actual units with the same run order as before
Copy responses
into last column
Analyze Factorial Design and
select the significant factors
Transfer function in actual units:

PPM Cl = 0.5602 (0.04087 x mass) +
(0.01456 x volume) (0.00020 x
shake1*shake2) + (0.00472 x shake1) +
(0.000316 x shake2)
Effect of Factors on Response Now Known . . .
As long as the effect is linear for each factor
New details in Session window
D
C
CD
B
A
12 10 8 6 4 2 0
T
e
r
m
Standardized Effect
2.05
A mass
B v olume
C shake1
D shake2
Factor Name
Helps you focus on the critical factors that influence the response
Points out important interactions between factors or directs your attention to
otherfactorsyouhadntconsidered
Transfer function should be verified experimentally by running selected
experiments at the low and high settings to check predictive capability.
The screening model assumes that each factor has a linear effect on the
response because only 2 factor levels were tested.
The screening model does not specifically address variance because each
corner of the design space is tested only once.
You may need to perform a second screening DOE to include other factors or
re-center the range.
The next sections will demonstrate how to strategically model the variance to
develop the most robust method.
An Optimization DOE on significant factors will help show any curvature in
the design space. . . . Next

Screening DOE: Summary and Important Points to
Consider
Why do an Optimization DOE?
Add center-points for curvature
Re-center the design space based on screening results
Focus only on significant factors
Improve precision of the model by adding replicates
Determineatransferfunctionbetweenyourxsandyour
response

How is the experiment different?
More than 2 levels to each factor
Usually fewer factors
Often additional replicates
Youremoreknowledgeablenow,andsetabetterdesignspace

Does it take more time than a screening DOE?
Usuallydoesnttakeanymoretime,andsometimestakesless!

Optimization DOE
Corner Points-
Assessment of
linear and 2-way
interactions
Center Points-Needed to determine if
curvature is present. Replicating center
point is a common way to get at pure error.
Star (Axial) Points-
For the assessment of
quadratic terms
Central Composite Designs Box-Behnken Designs
Exist only for
number of factors
= 3 to 7
No Corner Points-
Used in situations
when design space is
physically constrained
and corner points are
not possible.
65
75
85
95

R
i
s
i
n
g

R
i
d
g
e

-4.00
-2.00
0.00
2.00
4.00
-4.00
-2.00
0.00
2.00
4.00
A
B
Center points and star points are
needed to adequately model the
behavior of a response when the
system includes curvature.
Example of
curved
response
surface
Optimization: Second Order Experimental Designs
Method development specialists at Polystuff Products analyze the extent of
polymerization in lab-scale reactions by measuring the amount of residual
monomer following reaction in a stirred vessel. The scientists observe that, in
general, the rate of polymerization increases with increased stirrer speed.
However, when the stirrer speed gets too high, shear instability causes the
solids to coagulate out of suspension, reducing the polymerization rate. The
amount of suspended polymer can roughly be measured as turbidity by UV-Vis
absorption.
The scientists designed a central composite design DOE with two factors
(stirrer speed and mass of initial monomer) measuring two results (residual
monomer and turbidity) find the optimal settings for their process. The goal is
to achieve a residual monomer level between 410 ppm and 427 ppm, and
maximum relative turbidity between 0.7 and 0.88.
Experimental data can be found in the breakout file named:
Breakout-6 CCD Optimization-DOE.MPJ
Breakout #6 RS Design with Optimization
Optimization: Create RS (Response Surface) Design
After running
experiments,
add the responses
Factor
settings
Optimization: Create RS (Response Surface) Design
Include all terms
(main effects,
interactions,
quadratic)
Residuals plots can be
generated as shown
before
Optimization: Analyze RS Design
Analyze one
response at a time
Now eliminate least significant terms one-by-one,
leaving in the main effects
Mass * Mass is least significant
Lack of Fit is not significant.
Good
Stirrer * Mass is significant
Response = Residual Monomer
The regression is significant.
Residual Monomer = 406.774 + (9.780 x stirrer) +
(46.279 x Mass) (9.439 x stirrer x mass)
Final Transfer Function
Stirrer x Mass remains
significant. Main effects
must be kept.
Response = Residual Monomer
Response = Turbidity
Turbidity = -0.01829 + (0.12011 x stirrer) +
(0.18035 x Mass) (0.00649 x stirrer
2
)

Final Transfer Function
Stir, Mass, and Stir
2
are all significant, but what
about the constant?
After drilling down to the
critical factors . . .
RS Plots help visualize the interacting
effect of each factor on the response
Contour Plot flat graph with colored
topographical lines
Wireframe Plot 3-D projection of curve
Each plot has Setup
options (examples below)
Specify which
response and
factors to plot
Specify how to
handle other
factors if more
than 2 factors
exist.
Optimization: Plot the Response Surface
. . . But SO WHAT?
Iwanttoknowhowtosetalltheknobsonmymethodto
optimize both responses simultaneously!
2.15
2.05
1.95
1.85
1.75
2.5
0.5
1.65
3.5
Mass
0.6
4.5
0.7
1.55
5.5
0.8
0.9
6.5
1.45
7.5 8.5
Turbidity
1.35
9.510.5
1.25
11.5 Stirrer
Wireframe Plot of Turbidity vs. Stirrer and Mass
0.529268
0.564859
0.600451
0.636042
0.671633
0.707224
0.742815
0.778407
0.813998
0.849589
0.885180
0.920772
3 4 5 6 7 8 9 10 11
1.3
1.4
1.5
1.6
1.7
1.8
1.9
2.0
2.1
Stirrer
M
a
s
s
Contour Plot of Turbidity vs. Stirrer and Mass
396.719
403.990
411.260
418.531
425.802
433.072
440.343
447.614
454.885
462.155
469.426
476.697
3 4 5 6 7 8 9 10 11
1.3
1.4
1.5
1.6
1.7
1.8
1.9
2.0
2.1
Stirrer
M
a
s
s
Contour Plot of Residual Monomer vs. Stirrer and Mass
2.15
2.05
1.95
1.85
1.75
2.5
390
1.65
3.5
400
Mass
410
420
4.5
430
440
1.55
5.5
450
460
470
6.5
480
1.45
7.5 8.5
1.35
Residual Monomer
9.510.5
1.25
11.5 Stirrer
Wireframe Plot of Residual Monomer vs. Stirrer and Mass
Residual
Monomer
Turbidity
Optimization: Plot the Response Surface
Minitab 15 has snazzier plots than Minitab 12
Optimization: Optimize all Responses
Cur
High
Low
1.0000
D
Optimal
d = 1.0000
Targ: 427.0
Residual
y = 427.0000
d = 1.0000
Maximum
Turbidit
y = 0.9031
1.0000
Desirability
Composite
1.2757
2.1243
2.7574
11.2426
Mass Stirrer
[7.6026] [2.1243]
Optimal Factor
Settings
Predicted
Responses
Drag the vertical red lines
left and right to change
the factors and see the
effect on each response
Now you have it all!
Optimization: Optimize all Responses
To reset:
Right click,
Reset to
optimal
settings
Course 1 Summary
There are several tools to help understand your data
(hypothesis tests, ANOVA, GRR, etc.)
These tools can indicate sources of variation or
equivalence of data sets
Several DOE strategies are available to identify the
factors that affect the response
Screening DOE simplest design for quickly probing
linear effects for multiple factors
Optimization DOE more complex design for
understanding higher order interactions.
Proper use of these concepts can help you truly
optimize your methods.
RevisitingStudentsCourseExpectations
Did you get what you came for?
Additional references
1. Basic Statistics, Tools for Continuous Improvement. Mark J. Kiemele, Stephen
R. Schmidt, Ronald J. Berdine. Air Academy Press, 1999. ISBN 1-880156-06-7
2. Chemometrics: Data Analysis for the Laboratory & Chemical Plant. Richard G.
Brereton. Wiley, 2003. ISBN 0-471-48978-6
3. Experimental Design: A chemometric approach. Stanley N. Deming, Stephen
L. Morgan. Elsevier, 1987. ISBN 0-444-42734-1
4. Statistics: A Guide to the Use of Statistical Methods in the Physical Sciences.
J.R. Barlow. Wiley, 1989. ISBN 0-471-92295-1
5. Statistics and Chemometrics for Analytical Chemistry. Jane Miller. Prentice
Hall, 2005. ISBN 0-131-29192-0
6. Statistics for Analytical Chemistry. Jane C. Miller, James N. Miller. Prentice
Hall, 1993. ISBN 0-130-30990-7
7. Statistics for Experimenters, An Introduction to Design, Data Analysis, and
Model Building. George E. P. Box, William G. Hunter, and J. Stuart Hunter.
Wiley, 1978. ISBN 0-471-09315-7
8. Understanding Industrial Designed Experiments. Stephen R. Schmidt, Robert
G. Launsby. Air Academy Press, 1994. ISBN 978-1880156032
9. http://science.widener.edu/svb/stats/stats.html
10.www.statease.com
11.www.minitab.com/resources
12.www.itl.nist.gov

Acknowledgements
Pittcon Short Course Committee
SABIC Innovative Plastics Analytical Technology
Statistics Group at GE Global Research Center (Angie Neff,
Martha Gardner)

Pittcon March 2013
Course 1 Appendix
Appendix
Anther DOE Design: Mixture (available with Design Expert
software)
Simply another class of DoEs
Requirements:
Components dependent on
each other
Response is based on ratio
of components
If not, investigate response
surface designs

Types of mixture DoEs:
Simplex
Identical factor ranges for
each component
Non-constrained design
D-Optimal
Highly constrained design

Appendix
Example: Extraction of Pb from a soil matrix.
The primary factors considered are Time, Temp, and Acid Conc.

Create a full factorial DOE:
2
3
= 8 runs
with replicates at each point = 16 total runs

Create the DOE in Minitab

Input the experimental results from the file:
DOE ppm Pb.xls

Label columns and analyze the data.

Breakout: full factorial DOE practice
Appendix
(coded and uncoded data are available on
separate worksheets within the Excel file)
STD
Order A B C D ABCD
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
FACTORS
1. Create DOE by hand, filling in 1 & 1 levels
2. Compute the dot product for ABCD
3. Choose runs for which ABCD = 1 or 1
STD
Order A B C D=ABC ABCD
1
2
3
4
5
6
7
8
FACTORS
This is the same as using the Design
Generator D = ABC:
1.Construct a 2
3
design with A, B, and C
2.Set D=ABC and compute dot product
3.Compute dot product for ABCD What factors are estimable
from this experiment?
Consider a Full 2
4
factorial design (16 Runs)
Challenge: If you can only afford to run 8
runs, which 8 do you choose?
For a 2
4
full factorial:
Grand mean: X-bar
Main effects: A, B, C, D
2-Way interactions: AB, AC, AD, BC, BD,
CD
3-Way interactions: ABC, ABD, ACD, BCD
4-Way interaction: ABCD
16 items estimated from the 16 runs
Breakout: fractional DOE practice
Appendix

Attack The Variance - Hurst-1

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Attack The Variance - Hurst-1

Загружено:

Авторское право:

Доступные форматы

Slide- Attack the Variance, Course 1 (Pittcon 2013) 1

Attack the Variance, Course 1

Вам также может понравиться