2/9/2015

RepeatabilityandReproducibilityofExperiment
HPS1702Junior/SeniorSeminarforHPSMajors
HPS1703WritingWorkshopforHPSMajors
Springterm2014

RepeatabilityandReproducibilityof
Experiment
Backtocoursedocumentlist.

Theimportanceofrepeatabilityandreproducibility:
Science,December,2011
InfectionandImmunity,December,2010
JournalofIR

Definitions

http://www.pitt.edu/~jdnorton/teaching/1702_jnrsnr_sem/docs/Reproducibility/reproducibility.html

1/5

2/9/2015

RepeatabilityandReproducibilityofExperiment

http://www.pitt.edu/~jdnorton/teaching/1702_jnrsnr_sem/docs/Reproducibility/reproducibility.html

2/5

2/9/2015

RepeatabilityandReproducibilityofExperiment

FromIUPAC.CompendiumofChemicalTerminology,2nded.(the"GoldBook").
CompiledbyA.D.McNaughtandA.Wilkinson.BlackwellScientificPublications,
Oxford(1997).
(InternationalUnionofPureandAppliedChemistry)

http://www.pitt.edu/~jdnorton/teaching/1702_jnrsnr_sem/docs/Reproducibility/reproducibility.html

3/5

2/9/2015

RepeatabilityandReproducibilityofExperiment

repeatability
Theclosenessofagreementbetweenindependentresultsobtainedwiththesame
methodonidenticaltestmaterial,underthesameconditions(sameoperator,
sameapparatus,samelaboratoryandaftershortintervalsoftime).
Themeasureofrepeatabilityisthestandarddeviationqualifiedwiththeterm:
`repeatability'asrepeatabilitystandarddeviation.
Insomecontextsrepeatabilitymaybedefinedasthevaluebelowwhichthe
absolutedifferencebetweentwosingletestresultsobtainedundertheabove
conditions,maybeexpectedtoliewithaspecifiedprobability.

reproducibility
Theclosenessofagreementbetweenindependentresultsobtainedwiththesame
methodonidenticaltestmaterialbutunderdifferentconditions(different
operators,differentapparatus,differentlaboratoriesand/orafterdifferent
intervalsoftime).
Themeasureofreproducibilityisthestandarddeviationqualifiedwiththeterm
'reproducibility'asreproducibilitystandarddeviation.
Insomecontextsreproducibilitymaybedefinedasthevaluebelowwhichthe
absolutedifferencebetweentwosingletestresultsonidenticalmaterialobtained
undertheaboveconditions,maybeexpectedtoliewithaspecifiedprobability.
Notethatacompletestatementofreproducibilityrequiresspecificationofthe
experimentalconditionswhichdiffer.

Trouble?

http://www.pitt.edu/~jdnorton/teaching/1702_jnrsnr_sem/docs/Reproducibility/reproducibility.html

4/5

2/9/2015

RepeatabilityandReproducibilityofExperiment

Overthepastdecade,beforepursuingaparticularlineofresearch,scientists
(includingC.G.B.)inthehaematologyandoncologydepartmentatthe
biotechnologyfirmAmgeninThousandOaks,California,triedtoconfirm
publishedfindingsrelatedtothatwork.Fiftythreepapersweredeemed
'landmark'studies(see'Reproducibilityofresearchfindings').Itwas
acknowledgedfromtheoutsetthatsomeofthedatamightnotholdup,because
papersweredeliberatelyselectedthatdescribedsomethingcompletelynew,such
asfreshapproachestotargetingcancersoralternativeclinicalusesforexisting
therapeutics.Nevertheless,scientificfindingswereconfirmedinonly6(11%)
cases.Evenknowingthelimitationsofpreclinicalresearch,thiswasashocking
result.
...

...
Unfortunately,Amgen'sfindingsareconsistentwiththoseofothersinindustry.A
teamatBayerHealthCareinGermanylastyearreported4thatonlyabout25%
ofpublishedpreclinicalstudiescouldbevalidatedtothepointatwhichprojects
couldcontinue.Notably,publishedcancerresearchrepresented70%ofthe
studiesanalysedinthatreport,someofwhichmightoverlapwiththe53papers
examinedatAmgen.

http://www.pitt.edu/~jdnorton/teaching/1702_jnrsnr_sem/docs/Reproducibility/reproducibility.html

5/5