MSAforFunctionalTests

MSAFORFUNCTIONALTESTS
RepeatabilityandReproducibilityofFunctionalFailures

Repeatabilityistheabilityofafunctiontoperformoverrepeatedeventsunder(nearly)the
sameconditions.(e.g.samesample)Typicallythetestsareperformedimmediatelysubsequent
toeachother.
Ifaproblemisnotrepeatableeachexperimentaltreatmentrequiresreplicatesofthesame
condition.Aminimum2testsmustbeperformed,butthetruenumberofreplicateswillbe
determinedbytherepeatabilityandmaybedeterminedbysamplesizecalculationsfor
categorical/proportionaldata.

Reproducibilityistheabilityofafunctiontoperformacrossdifferentconditions.(e.g.different
samplesorspecimens)
Ifaproblemisnotreproducibletheneachexperimentaltreatmentmustbereplicatedacross
differentuseconditions.Againthereplicatesizewillbedeterminedbytheamountof
reproducibility.

BeverlyDaniels

December30,2014

MSAforFunctionalTests
MSAProcess
Functionalcharacteristicsposeamorecomplexmeasurementerrormodel,sincethefunction
cannotbemeasuredexactlytwice.Measurementerrorwillnaturallybeconfoundedwiththe
causalfactors,theconditionsforuseandeventhetestconditions.
Ifthemeasurementofafunctionfailsrepeatabilityitwillbenecessarytodesignastudytoun
confoundthevariouscomponentsofvariationpriortoproceeding.
ExtraneousConditions:Somefunctionsareaffectedbyextraneousfactorsunrelatedtothe
primarycausalmechanism.Thesefactorsresultinextravariationthatwillmaskthecausal
factorsandthemeasurementerror.Inthiscase,theexperimentaldesignistoholdthe
extraneousfactorsconstantusuallyatBestCaselevelsinordertomeasurethevariation
causedbythetruecausalsystem.
ConditionsforFailure:Ifthefailurerateisverylowand/orthefailureisdependentoncertain
useorenvironmentalconditions,WorstCasetestconditionsshouldbeused.Thisallowsthe
useofsmallersamplesizes.Itiscriticaltoestablishcorrelationoftheworstcasetest
conditionstorealworldconditions.Foolishconditionswillleadtoincorrectresults.
Unknownoruncontrollableconditions:Ifthefailuresaredependentonconditionsthat
cannotbecontrolledorareunknownthentheRepeatabilityandReproducibilityofthefailures
mustbetestedunderavarietyofconditionsandthevariationmustbeunderstoodsothat
testingsamplesizesandreplicatesizecanaccommodatethemeasurementerror.The
experimentaldesignforFunctionalcharacteristicswithvaryingconditionsforuseortest
conditionsistoholdtheuseandtestconditionsconstantwithinapairofrepeated
measurementsbutensurethatthesefactorsvarybetweenpairs.

BeverlyDaniels

December30,2014

MSAforFunctionalTests
ExtraneousConditionsExample:EstablishingBestCaseConditions
VehiclePullstotheLeft
Vehiclepullingisinfluencedbythefollowing:

Slopeorcrownoftheroad

Weightdistributionwithinthevehicle

Wind

Distributionoftirepressure

Vehiclecomponentsandassembly

Preliminaryanalysisshowedthatslopeandwindwere
notfactorsinPullcomplaintsastheseconditionsare
Crownofroad
highlyvariableandthecomplaintswereofpersistentpull.
Dealerexperiencedrivingthevehiclesconfirmedthis.
Wind
Dealersreportedthatthepullonvehicleswaspersistent
evenafterbalancingthepassengerweight,cargoweight
Weight
andtirepressure.
balance
Initialtestsbythefactoryengineersconfirmedthese
findings.
TirePressure
ThevehiclepullisapersistentProblem.Correctingroad
crown,wind,weightbalanceandtirepressurebalance
doesnoteliminatePull.

VehiclePulls
totheLeft

balance
Vehicle

Environmentalandusefactorsareextraneousfactorsandarenotcontrollableinrealuse.They
mustberemovedfromthediagnostictestmethodtoreliablydetectdifferencesinthecausal
system.

BeverlyDaniels

December30,2014

MSAforFunctionalTests

Atestcoursewasconstructedonarelativelyflatsurface.Alltestvehiclesweredrivenatthe
samespeed,bythesamedriver,underthesamerelativewindconditionsandthesteering
wheelwasreleasedatthesamepointonthecourseandallowedtopull.Atapredetermined
distance,theY,pulldistancefromthecenterlinewasmeasuredforeachvehicle.Each
vehiclewastestedtwice.Carsthatwereknowncustomercomplaintcarswerealsotestedon
thecoursetoensurecorrelationtotherealworld...

Y1

Y3

Y4
Y2

Finishing Line

Nowind

Flatsurface

SameDrivereverytime

Tirepressurebalancedand
settorecommended
pressure

Vehiclecrossedthestartline
perpendicular(90)tothe
line

Vehiclecrossedat60mph

Driverletgoofsteering
wheelasthevehiclecrossed
theline

BeverlyDaniels

Set Distance

Starting Line

December30,2014

MSAforFunctionalTests
MSAresultsforVehiclePullstotheLeft:

Vehicle Pulls to the Left

Measurement Discrimination Plot
Right

20
15
10

Distnace from 0 (inches)

5
0
-5
-10
-15
-20
-25
-30
-35
-40
-40 -35 -30 -25 -20 -15 -10

-5

10

15

20

Left
Distance from 0 (inches)

e =
p =
T =

1.35

r=

10.36

Dr=
e is

10.45

Right

0.9833
10.9
1.67% ofthetotalvariation

Measurement Range Chart

6
5
4
3
2
1
0
1

10

11

12

13

14

15

16

17

18

19

20

ConditionsforFailureExample:EstablishingWorstCaseTestConditions
VehicleWaterLeaks
BeverlyDaniels

December30,2014

MSAforFunctionalTests

Whentestingforwaterleaksinavehiclethefirstdecisioniswhat
conditionstouse.
Optionsare:

Useagardenhosetopourwateroverthecarsurfaces
Useagardenhosetosprinklewateronthecar,simulatinggentlerainfall
Useacarwashlikeapproachwherewaterissprayedathighvolumeandpressureat
differentangles,simulatingadrivingrainstormthatthevehicleisdrivingthrough
Immersethevehicleinapoolofwater

Thefirst2methodsdidnotalwayspickupwaterleaksonknownleakersandweredeemedto
betooweak.
Acarwashapproachwasphysicallysimilartoaharddrivingrain(aswellasacarwash)and
consistentlypickedupknownleakers,caughtsomeminorleakersanddidnotcauseknown
goodvehiclestoleak.
ImmersionwasnotsimilartoCustomerreportedconditionsforleakingvehicles.Additionallyit
wasdeemedtobefoolishasthevehicleisnotdesignedtobecompletelywatertightunder
conditionsofvariablepressureexternalandinternaltothevehicle.

BeverlyDaniels

December30,2014

MSAforFunctionalTests
UnknownConditionsExample:BloodTestingFunctionofaHematologyInstrument
Awholebloodsamplepoolwithaknownplateletleveliscreatedtwiceaday.Thereisa
morningpoolandanafternoonpool.Eachinstrumentistestedwith3runsusingbloodfrom
thesamebloodpool.Iftheplateletcountmeanofthe3runsismorethanXXcountsfromthe
target,theinstrumentfails.ThisMSAwasperformedtodeterminetherepeatabilityand
reproducibilityofafailure.Theamountofrepeatabilityandreproducibilitywilldeterminethe
replicatesamplesizeforProblemSolvingexperiments.TheMSAused3passinginstruments
and3failinginstruments.
UnknownConditions:Theunknownconditionswere:

Sampleconditions

Sampledegradation

Samplenonhomogeneity

Integrityofsamplestreampresentedtodetectionsensors

Instrumentconditionsfromruntorunsuchashydraulicflow,pressure,temperature,
humidity,etc.

Repeatability:Theruntorunvariationwasassessedwith3sequentialrunsfromthesame
samplealiquot.Thetesttotestvariationwasassessedbyconducting3setsof3sequential
runsofthesamealiquotusing3differentaliquotsofthesamebloodpool.Theresultsindicate
goodRepeatabilityofthefailure.
Reproducibility:Therepeatabilitystructurewasreplicatedover3separatebloodpoolsto
assesstheeffectofthebloodpoolandtimeonreproducibility.

BeverlyDaniels

December30,2014

MSAforFunctionalTests

PLT Delta From Target

Results

Whiletherewasdecentseparationoftheresultsfromgoodtobad,thedatacouldonlybe
separatedintotwodistinctcategories.Thereforeeachexperimentrequired3runsofthesame
sample.Itwasdecidedthatmultiplesampleswerenotneeded.

Thestructureof3repeatedrunsunderthesameconditionscannotbeanalyzedusingaYouden
plot.Themultivarichartisadequatetoassessbasicrepeatabilityandreproducibility.

BeverlyDaniels

December30,2014

MSAforFunctionalTests
References
Donald J Wheeler, Craig Award Paper, Problems With Gauge R&R Studies, 46th Annual Quality
Congress, May 1992, Nashville TN, pp. 179-185.
Donald J Wheeler, An Honest Gauge R&R Study, Manuscript 189, January 2009.
http://www.spcpress.com/pdf/DJW189.pdf
Youden, William John, Graphical Diagnosis of Interlaboratory Test Results, Industrial Quality Control,
May 1959, Vol. 15, No. 11

BeverlyDaniels

December30,2014