4/2/2015

Modellingtheeffectsofsubjectiveandobjectivedecisionmakinginscientificpeerreview:Nature:NaturePublishingGroup

nature.com
Archive

PublicationsAZindex

Volume506

Issue7486

AccessprovidedtoMichiganStateUniversitybyMSULibraries
Letters

Cart

Login

Register

Article

NATURE | LETTER

Modellingtheeffectsofsubjectiveandobjectivedecisionmakingin
scientificpeerreview
InUckPark,MikeW.Peacey&MarcusR.Munaf
Nature 506, 9396 (06February2014) doi:10.1038/nature12786
Received 01August2013 Accepted 16October2013 Publishedonline 04December2013
Print

Theobjectiveofscienceistoadvanceknowledge,primarilyintwointerlinkedways:circulatingideas,anddefendingor
criticizingtheideasofothers.Peerreviewactsasthegatekeepertothesemechanisms.Giventheincreasingconcern
surroundingthereproducibilityofmuchpublishedresearch1,itiscriticaltounderstandwhetherpeerreviewis
intrinsicallysusceptibletofailure,orwhetherotherextrinsicfactorsareresponsiblethatdistortscientistsdecisions.
Hereweshowthatevenwhenscientistsaremotivatedtopromotethetruth,theirbehaviourmaybeinfluenced,andeven
dominated,byinformationgleanedfromtheirpeersbehaviour,ratherthanbytheirpersonaldispositions.This
phenomenon,knownasherding,subjectsthescientificcommunitytoaninherentriskofconvergingonanincorrect
answerandraisesthepossibilitythat,undercertainconditions,sciencemaynotbeselfcorrecting.Wefurther
demonstratethatexercisingsomesubjectivityinreviewerdecisions,whichservestocurbtheherdingprocess,canbe
beneficialforthescientificcommunityinprocessingavailableinformationtoestimatetruthmoreaccurately.By
examiningtheimpactofdifferentmodelsofreviewerdecisionsonthedynamicprocessofpublication,andtherebyon
eventualaggregationofknowledge,weprovideanewperspectiveontheongoingdiscussionofhowthepeerreview
processmaybeimproved.
Subjectterms: Psychology

Main
Currentincentivestructuresinsciencepromoteattemptstopublishinprestigiousjournals,whichfrequentlyprioritizenew,exciting
findings.Oneconsequenceofthismaybetheemergenceoffadsandfashionsinthescientificliterature(thatis,hottopics)1,leading
toconvergenceonaparticularparadigmormethodology.Thismaynotmatterifthisconvergenceisonthetruthtopicsmaysimply
ceasetobehottopicsastheproblembecomessolved.However,thereisincreasingconcernthatmanypublishedresearchfindings
areinfactfalse1.Itiscommonforearlyfindingstoberefutedbysubsequentevidence,oftenleadingtotheformationofgroupsthat
interpretthesameevidenceinnotablydifferentways2,andthisphenomenonisobservedacrossmanyscientificdisciplines3,4.There
areanumberofrelativelyrecentexamplesofconvergenceonfalsehypotheses,suchasthetheoryofstresscausinggastriculcer
formation5.Onceestablished,thesecanbecomesurprisinglydifficulttorefute6theymaybecomemorevampiricalthan
empiricalunabletobekilledbymereevidence7.Sciencemaythereforenotbeasselfcorrectingasiscommonlybelieved8,and
theselectivereportingofresultscanproduceliteraturesthatconsistinsubstantialpartoffalseconclusions9.
Itisimportanttounderstandhowconvergenceonfalseconclusionsmaycomeabout.Anumberofpossibilitiespresentthemselves.
First,scientistsmaynotinfactberationalindividualspursuingthetruthafterallanargumentmadebysomeinfluentialsociologists
ofscience(thestrongprogramme)10ormayberationalbutstuckwithinaparticularparadigm11.Second,somescientistsmaybe
biasedorevenimmoralanumberofhighprofilecasesofdatafabricationandfraudhaveemergedinrecentyears12.Third,some
scientistsmaycaremoreaboutpublicationandcareersthandiscoveringthetruth(thatis,publishorperish),aprocesswhichmay
beconsciousorunconscious13.Incompetitivefieldscurrentincentivestructuresprioritizepositiveresults,whichmayincreasethe
http://www.nature.com/nature/journal/v506/n7486/full/nature12786.html

1/12

4/2/2015

Modellingtheeffectsofsubjectiveandobjectivedecisionmakinginscientificpeerreview:Nature:NaturePublishingGroup

likelihoodofmodificationofdataorconductingmanystatisticalteststoachievethesesimilarly,increasederrorratesmayarisefrom
multiplecompetingresearchgroupstestingthesamehypotheses14.
Ithasbeenshownthatincreasedpopularityofaparticularresearchthemereducesthereliabilityofpublishedresults14,andthat
findingspublishedinprestigiousjournalsarelessreliableandmorelikelytoberetracted15.Therefore,theconvergenceofresearch
interestonacurrenthottopicmayservetounderminethereliabilityandveracityofsubsequentlypublishedfindings.Inprinciple,peer
reviewshouldeliminateorreducetheseproblemsbut,givenempiricalevidencefortheunreliabilityofmuchpublishedresearch,it
maynotinfactbeconductedproperly,ortheprocessitselfmaybeflawed.Empiricalresearchandsimulationshaveidentifieda
numberoffactorswhichcontributetothelikelihoodthatapublishedresearchfindingisfalse1,16.However,thepeerreviewprocess
itselfhasnotbeencloselyinvestigatedasapossibleinfluence,despitethefactthatitactsastheultimategatekeeperofresearch
publication.Itisgenerallyregardedasimperfect,althoughstillthebestmodelavailabletoensureboththequalityandveracityof
publishedscientificresearch,buttherehasbeengrowingconcernthatitfails,atleastinpart,withrespecttoeachofthesetwo
goals1.
Tounderstandthepeerreviewmechanismbetter,usingaBayesianapproachinamodelofthepublicationprocess,weanalysedthe
behaviourofscientistswhohavedevelopedtheirinitialopinionsindependentlyastowhichofthetwoopposinghypotheses,AandB,
ismorelikelytobetrue.Theyknowthatonaveragetheiropinionisindeedcorrectwithprobability

,sotheyfeelconfident,

butlessthanfully,abouttheiropinion.Themorecontroversialtheissue,thelowerthevalueof.Uponreceivingamanuscriptthat
advocatesoneofthehypotheses,theeditorofahypotheticaljournalsolicitsareviewfromanotherscientist,whorecommends
acceptanceorrejection.Tofocusontheinfluenceofreviewerbehaviour,ratherthanthatofeditor,weassumethattheeditorsimply
followsthereviewersrecommendation.Subsequently,thereviewerwritesandsubmitstheirownmanuscripttothejournal,andthe
processrepeats.Thetwodecisionsforeachscientistaretherefore:(1)whetherornottorecommendacceptanceofamanuscript
thattheyarereviewing,and(2)whichhypothesistoadvocateintheirownsubmission,whichwetermthethemeoftheirmanuscript.
Asapublicationhistoryevolves(followingcyclesofsubmission,peerreviewandacceptanceorrejection)ascientistrevisestheirview
onthelikelihoodofeachhypothesisbeingtrue,inlightoftherelativeprobabilityofthisparticularhistoryoccurringwhenone
hypothesisistrueasopposedtotheother.Beingmotivatedtopromotethetruth,eachscientistwilladvocateathemethatismore
likelytobetrue,accordingtotheirrevisedviewwhentheysubmitamanuscript.
Ouraimwastounderstandhowdifferentcriteriaofreviewingdecisionsinfluencethepublicationoutcome,andhowtheresulting
publicationhistoriesandtheinformationinherentintherelevantpeerreviewcriterioninfluencethecommunityseventual
understandingofthetopic.Tothisendwemodelledandcomparedtwodifferentwaysthatscientistsapproachthereviewingdecision.
Inthefirstmodel(M1),thesubjectivecriterionofhowstronglytherevieweragreeswiththeconclusionoftheresearch(thatis,the
themeofthemanuscript)isreflectedinthedecision,inadditiontoothermoreobjectivecriteriasuchasresearchdesignand
methodology.Inthesecondmodel(M2),thedecisionreflectsobjectivecriteriaonly.Ourfindings,therefore,mayshedlighton
whethersubjectiveassessmentisdesirableinthepeerreviewprocessand,ifso,towhatextent.Asabenchmark,wealsocompared
M1andM2withadefaultmodel(M3),inwhichallmanuscriptsarepublishedwithoutanyfilteringthroughpeerreview.Asscientists
willmakeinferencesthattakeintoaccounthowreviewersarriveattheirrecommendations,theparticularpeerreviewmodelin
operationaffectshowtheyrevisetheirviewsand,thereby,theirdecisionsonwhichthemetoadvocateasanauthor,aswellastheir
decisionsasareviewer.
Theresultsofthethreemodels(Fig.1)indicatethat:(1)almostcertainly,somescientistswillsubmitmanuscriptsonthemeswhich
disagreewiththeirinitialopinion(wetermthisherding)(2)theextenttowhichthewiderscientificcommunitysperceptionofa
literatureisremovedfromthetruth(wetermthismisperception)decreaseswithnumberofpublications,butinformationtransmission
isgreatlyhamperedonceherdinghasoccurred,tosuchanextentthatnofurtherimprovementinunderstandingoccursexceptinM1
whereadegreeofsubjectivityisallowedinthereviewingdecision(thatis,reviewersaswellasauthorsactguidedbyBayesian
inference)and(3)theprobabilityofanotherpublicationonaparticularissueincreasesasthenumberofmanuscriptspublishedon
thatissueincreases,owingtoaggregationofinformationandherdingreinforcingthescientificcommunitysconsensus.
Figure1:Threemodelsofpeerreviewbehaviour.

http://www.nature.com/nature/journal/v506/n7486/full/nature12786.html

2/12

4/2/2015

Modellingtheeffectsofsubjectiveandobjectivedecisionmakinginscientificpeerreview:Nature:NaturePublishingGroup

Weshowthreemodels,M1(right),M2(middle)andM3(left),whichdifferintheextenttowhichthepeerreviewdecisiondependsonwhether
therevieweragreeswiththeconclusion.Threeoutcomesarepresented:(1)probabilityofherding(top),(2)averagemisperceptiongenerated
(middle),and(3)probabilityofacceptance(bottom).Theprobabilitythattheinitialopinioniscorrectisreflectedby,andeachoutcomeis
presentedforthreevaluesof:(1)0.55(blue,dottedline),(2),0.75(green,dashedline),and(3)0.95(red,solidline),reflectinghigh,
intermediate,andlowuncertainty,respectively.

Thephenomenonknownasherdingisinherentinthebehaviourofscientistsoperatingunderallofthemodelsweconsider.An
individualissaidtobeherdingiftheychooseathemetoadvocateintheirmanuscriptsubmissionbasedentirelyonwhattheyhave
observedfromothers,independentlyofwhattheyinitiallythoughtwastrue.Thedegreeofherdingdependsonthepeerreview
modelinoperation,thenumberofmanuscriptssubmittedsofar,andhowconfidentscientistsfeelabouttheirinitialopinion().
Herdingtakesplacerelativelyquickly(Fig.1),andweobservediscretejumpsinthemeasureofherdingearlyonintheprocess,
wheneachsignal(thatis,theinformationcarriedbyapeerreviewdecision)carriesalargeweighting.Notably,theprobabilityof
herdingandthespeedwithwhichitincreasesareeventuallylowerwhenadegreeofsubjectivityisallowedinthereviewingdecision
(M1),andonlyinthiscasecanafadbereversedfollowingasequenceofpublicationsonthesametheme.Asafadpersists,thetotal
numberofscientistsrequiredinordertoreversethisfadincreasesandatafasterrate.
Weusemisperceptiontodescribehowincorrecttheperceptionofthewiderscientificcommunityisafterahistoryofpublication
outcomes.Itisdefinedastheprobabilitythatanoutsiderassignstoahypothesisbeingcorrect,basedonBayesianinferencefrom
theobservedhistory,whenitisactuallyincorrect.Thelevelofexpectedmisperception(Fig.1)remainsrelativelystableforlowand
highvaluesof,butforintermediatevaluesofitdeclineswithincreasingnumbersofsubmittedmanuscripts.Critically,whena
degreeofsubjectivityisallowedinthepeerreviewprocess(M1),thisalwayseventuallyoutperformstheothermodels,becausein
thesemodelsinformationcompletelyfailstobetransmittedafterherdingoccurs.
Inourmodels,manuscriptsubmissiondecisionsmadebyindividualscientistsarebasedinpartoninformationinferredfromothers
actions,becauseindividualsuseinformationfromthepublicationhistorywithinaparticularfield,aswellastheirpersonalopinions,to
guidetheirdecisions.Thismayhavepositiveeffectsifthedecisionsclusteraroundacorrectoutcome,orhavenegativeeffectsifthey
clusteraroundanincorrectoutcome.Adegreeofsubjectivityinthepeerreviewprocesswill,onaverage,leadtolower
misperception,becausereviewerdecisions(andsubsequenteditorialdecisions)whichgoagainsttheherdingtrendwillcontinueto
revealnewinformation.Inaddition,theprocessisdynamic,andweshowthatselfcorrectioncaneventuallyoccurwhenadegreeof
subjectivityisallowedinthepeerreviewprocesshowever,itmaynotwhenthereviewingdecisioniscompletelyindependentofthe
reviewerssubjectiveassessmentofthethemeofthemanuscript,andisbasedonlyonother,largelyobjectivecharacteristicsofthe
manuscript,suchasthequalityoftheresearchmethodology.Inthiscasetheprobabilityofherdingreaches1withinfinitetimeforall
http://www.nature.com/nature/journal/v506/n7486/full/nature12786.html

3/12

4/2/2015

Modellingtheeffectsofsubjectiveandobjectivedecisionmakinginscientificpeerreview:Nature:NaturePublishingGroup

valuesof,andthelevelofmisperceptioncannotgobelowacertainlowerbound.Theconceptofherdinghasbeendiscussedinthe
contextofscientificresearchinthepast 17,butoursisthefirststudy,toourknowledge,tomodeltheprocessesbywhichitmay
occur.
Theseresultsraisethequestionofwhetherahigherlevelofsubjectivityinreviewerdecisionswillleadtomoreeffectiverestraintof
incorrectherding.WethereforedecidedtotestgeneralizedM1models,inwhichwevariedthedegreetowhichthereviewers
recommendationisdeterminedbytheirsubjectiveassessmentoftheconclusion.Ourresults(Fig.2)indicatethatexcessively
subjectivereviewsarenoteffectiveinrestrainingincorrectherding.Thisisbecause,inthiscase,recommendationsaresensitiveto
whethertheconclusionagreeswiththereviewersviewpointatthattime,andthisfactorispredominantlydeterminedbythe
accumulatedinformation,ratherthantheiroriginalopinion,aspublicationhistorylengthens.Inotherwords,inthiscaseeventhe
reviewersrecommendationsaresubjecttoherding.Itappearsthatamoderatedegreeofsubjectivity(asdepictedinM1)isnear
optimal.
Figure2:ExpectedmisperceptioninageneralizedversionoftheM1model.

Weshowtheexpectedmisperceptionforthreevaluesoftheprobabilitythattheinitialopinioniscorrect():(1)0.55(left),(2)0.75(middle),and
(3)0.95(right),reflectinghigh,intermediate,andlowuncertainty.Resultsareshownfordifferingdegreestowhichthereviewerssubjective
assessmentdeterminestheirrecommendation(v):(1)0.75(red,solidline),(2)1.00(green,longdashedline),(3)1.25(blue,shortdashedline),
and(4)1.50(black,dottedline).IntheoriginalM1modelv=1,whilelowervaluesreflectamoreobjectivereviewer,andhighervaluesamore
subjectivereviewer.Excessivelysubjectivereviewsarenoteffectiveinrestrainingincorrectherding(thisisnotyetvisiblefor=0.55,but
wouldbecomeapparentwithmoresubmissions).

Twoempiricalexamplesshowthatherdingoccursinthescientificliterature.First,beliefinaspecificscientificclaimcanbe(andis)
distortedthroughpreferentialcitationsofstudieswhichsupportaparticularpointofviewratherthanthosewhichdonot17.This
phenomenoncanbeattributedtoherdingcausedbypreferentialcitations,potentiallycreatingaspuriousandunfoundedsenseof
authorityforspecificclaims.Second,usingametaanalyticreviewofarecentliterature18,wecomparedclaimsmadeintheabstracts
ofthecontributingstudieswithsupportforthoseclaimsbythedatareportedtherein.Metaanalysisimposesastandardanalysisto
maximizecomparability,andtherebyminimizestheextenttowhichthepresentationofresultscanbeinfluencedbyflexibleanalytical
options19.Theseresults(Fig.3)showamismatchbetweentheclaimsmadeintheabstracts,andthestrengthofevidenceforthose
claimsbasedonaneutralanalysisofthedata,consistentwiththeoccurrenceofherding.
Figure3:Empiricalevidenceofdiscrepancybetweenclaimsandresults.

Weshowclaimsmadeintheabstractsofstudies,andtheresultsofthosestudiesderivedfromastandardizedanalysis.Abstractswerecoded
asproorcondependingonwhetheranassociationwasclaimed,basedonthejudgementofanindependentrater.Resultswerecodedaspro
orcondependingonwhethertheoveralleffectsizeforthefullsampleinthestudywasstatisticallysignificantatP<0.05.Fiveabstractscould
notbecodedaseitherproorcon.Theproportionofproversusconclassificationsdifferedforclaims(80%pro)andresults(44%pro),
suggestingherdingaroundthefirstpublishedclaim(McNemartest:P=0.016,twotailedtest).Treatingabstractsthatcouldnotbecodedas
http://www.nature.com/nature/journal/v506/n7486/full/nature12786.html

4/12

4/2/2015

Modellingtheeffectsofsubjectiveandobjectivedecisionmakinginscientificpeerreview:Nature:NaturePublishingGroup

proorcondidnotaltertheseresultssubstantially(84%versus64%pro).

Wenextconsiderwhetherscientistscandecideontheirconclusionbeforeconductinganexperiment.Wesuggestthatherdingleads
tooneoutcomebeingpreferableoveranother,andthatflexibleanalysisandselectivereportingallowsdatathatdonotconformto
eitherbetransformed19orrelegatedtothefiledrawer20.Mendelfamouslyappearstohavedroppedobservationsfromhisdataso
thathisresultsconformedtohisexpectations21,butbecausehistheorywasultimatelyprovedcorrectthisisnowgenerally
overlooked.Thereisinfactclearevidencethatthereportingandinterpretationoffindingsisofteninconsistentwiththeactual
results22,andthisappearstobeparticularlypronouncedinabstractsofresearcharticles(oftentheonlypartthatisread)23.
Scientistsmaybemotivatedbyanumberoffactors,suchasthedesiretobethefirsttoadvocateanidea,andthenaturaltendency
tosidewithothersofasimilaropinion.Herdingisthereforeexpectedwhenagentscareonlyaboutbeingpublishedandrecognize
sometopicsashot(andthereforepublishable).Ifscientistsaremotivatedinthiswayinourmodel,theninanequilibriumofthe
gametheycansimplyfollowthefirstauthorsclaimtomaximizetheprobabilityofbeingpublished(seeSupplementaryInformation).
However,ourresultsindicatethatwecanexpectherding,includingconvergenceonfalseconclusions,evenwhenscientistsbothas
authorsandreviewersarerationalandmotivatedbythepursuitoftruth.Theemergenceoffadsandfashionsinthescientific
literature(thatis,hottopics)1isthereforeunsurprising.
Thefirstherdingmodelineconomicsmodelledindividualsinvestmentchoices24.Herdingmayhavepositiveconsequences,by
drivingrapidconvergenceonacorrectdecision.Rationalindividualsprocessalltheinformationavailabletothembeforemaking
decisions,andherdingthereforearisesfromnaturalmotivesarationalindividualinpursuitoftruthcanandshouldbeinfluencedby
whatothersthink.Thathumansareinfluencedinthiswayhasbeenshownbyexperimentsinsocialpsychology25.Itisrational
becausehumansareawareoftheirownfallibility,andsotheiropinionsmaybestrengthenedorweakenedbytheviewsofothers.In
otherwords,beingawareofthewisdomofthecrowd,humansare(rationally)influencedbythecrowdinordertoupdateourbeliefs
inthelightofnewevidence,weshouldbeguidedbyBayestheorem.However,herdingmayalsohavenegativeconsequences,by
drivingconvergenceonanincorrectdecision.Thisisparticularlyproblematicifanoutsidertotheprocessisunawarethatitistaking
place,asitgivesaspurioussenseofcertaintytotheobservedconvergence.
Free,openandglobalaccesstoresearchreportshasbeenproposedasanalternativetopeerreview(http://am.ascb.org/dora/),but,
aswehaveshown,peerreviewcanrevealmoreinformationrelativetofreeandcompletesequentialpublication.Reviewer
recommendations,andresultingeditordecisions,containinformation,andthuspreventherdingfromcompletelyblockingnew
informationflow.However,thisdependsonspecificparameterssuchasthepopularityofthesubject(forexample,howmanypeople
arewritingaboutthisissue,orhowlongitisdiscussed)andhowstronglyscientistsfeelabouttheirinitialdispositions(thatis,thelevel
of).Inparticular,ifreviewers(andeditors)areexplicitlyencouragedtobeasobjectiveaspossibletheywillnotbeguidedbyBayes
theoremwhenmakingtheirrecommendationsitisonlywhenreviewersareallowedadegreeofsubjectivitythatthisisdone.Our
resultsindicatethatpeerreviewperformsbestwhenthereviewersexercisetheirsubjectivityatanintermediatelevelhigherlevels
enhancetheriskofcompleteherdinginreviewerdecisions,whereaslowerlevelscurbtheinformationflowfromreviewerdecisions.
Thepeerreviewprocessisthereforeinprincipleselfcorrectingoverasufficientlyextendedperiod(althoughdistortionsmayoccurin
theshorterterm),inthatdeherdingcanalsooccur.Inreality,deherdingwillnotalwaysoccur,becausepublicationhistorieswithina
topicmaynotpersistforsufficientlylong.Sciencemaythereforenotbeasselfcorrectingasiscommonlyassumed8,andpeerreview
modelswhichencourageobjectivityoversubjectivitymayreducetheabilityofsciencetoselfcorrect.Althoughherdingamongagents
iswellunderstoodincaseswheretheincentivesdirectlyrewardactinginaccordwiththecrowd(forexample,financialmarkets),itis
instructivetoseethatitcanoccurwhenagents(thatis,scientists)aremotivatedbythepursuitoftruth,andwhengatekeepers(that
is,reviewersandeditors)existwiththesamemotivation.Insuchcases,itisimportantthatindividualsputweightontheirprivate
signals,inordertobeabletoescapefromherding.Behaviouraleconomicexperimentsindicatethatpredictionmarkets,which
aggregateprivatesignalsacrossmarketparticipants,mightprovideinformationadvantages26.Knowledgeinscientificresearchis
oftenhighlydiffuse,acrossindividualsandgroups26,andpublishingandpeerreviewmodelsshouldattempttocapturethis.Wehave
discussedtheimportanceofallowingreviewerstoexpresssubjectiveopinionsintheirrecommendations,butotherapproaches,such
astheuseofpostpublicationpeerreview,mayachievethesameend.

http://www.nature.com/nature/journal/v506/n7486/full/nature12786.html

5/12

4/2/2015

Modellingtheeffectsofsubjectiveandobjectivedecisionmakinginscientificpeerreview:Nature:NaturePublishingGroup

Methods
Modelofthepeerreviewprocess
Weanalyseamodelinwhichn+1exanteidenticalscientistsdeliberateovertwoopposinghypotheses,labelledAandB.Itisknown
thatonlyoneofthesehypothesesiscorrect,andthatexantebothareequallylikelytobecorrect.Denotingthecorrecthypothesisby
,thisisexpressedas

.Beforethegamestarts,eachscientistireceivesaprivatesignal,

,regarding

whichisthetruehypothesis.Thesesignalsareindependentrandomvariablesthatassumeavalueequaltothecorrecthypothesis
withprobability.Thesignalsareinformativebutnotperfect,thatis,

.Lowervaluesofcanbeinterpretedasreflectinga

morecontroversialnatureoftheissueunderquestion,whenthesignalstendtobelessaccurate.
Sequentially,andmotivatedtopublishwhatistrue,differentscientistssubmitamanuscript,eachdefendingaparticularhypothesis.
Thethemeofscientistismanuscript,

,denotesthehypothesisthatisdefended.Wepostulatethat,uponreceivinga

manuscript,theeditorelicitspeerreviewfromascientistwhosestanceonthetopicisunknowntotheeditor,whicheliminatesthe
editorsinfluenceontheeditorialdecisionthroughreviewerselection.Thisisdonetofocusouranalysisonreviewerbehaviour,and
meansthatinourmodeleachmanuscriptisassignedtoascientistwhohasneithersubmittedtheirownmanuscriptnoractedasa
revieweratthatpoint(becauseotherwisetheeditorwouldhaveinferenceontheirstancefromthethemeoftheirsubmissionortheir
previousdecisionasareviewer).Theeditorfollowsthereviewersrecommendationindecidingwhethertoacceptorrejectthe
manuscript.Ifitisaccepted,itsthemebecomescommonknowledgeifitisrejected,thethemeisnotdisclosed,buttherejection
becomescommonknowledge.Then,anewsubmissionismadebyascientistwhohasnotsubmittedbefore.Inparticular,our
analysisisfocusedonthecasethatthenextscientistwhosubmitsamanuscriptistheonewhoreviewedthepreviousmanuscript.
Thus,labellingthescientistwhowritestheithsubmissionasi,eachscientist
advocatingatheme

manuscript.Scientistn+1,whoalsoreceivesasignal
theyarise.Let

sequentiallysubmitsamanuscript

,whichisreviewedbythenextscientistj=i+1,whosubsequentlywritesandsubmitstheirown
,onlyreviews.Scientistsobservethehistoryofpublicationoutcomesas

denoteahistoryofthefirstipublicationoutcomes,whereeachpublishedmanuscriptisrecordedbyits

theme,AorB,andeachunpublishedmanuscriptby.Then,therearethreeitemsofinformationavailabletoeachscientistjwhen
theymakedecisions:(1)theirownprivatesignal
history

(2)amanuscripttobereviewedwithatheme

ifj>1and(3)a

ifj>2.Thetwodecisionstomakearewhetherornottorecommendacceptanceofamanuscriptthatthey

arereviewing,andthethemeofthemanuscripttheysubsequentlysubmit.
Wemadeafewmodellingchoicesthatsimplifyrealpractices,namelythat:(1)onlyonereviewerisconsultedforeachsubmission
(2)thecurrentrevieweristhenextauthor(3)rejectionsbecomecommonknowledgeand,(4)authorsconformtotherationality
assumptionthattheyareBayesianupdaters.Choices1and2maximizethenumberofsubmissionsthatcanbereviewedbyagiven
numberofscientists,subjecttotheeditornotsolicitingareviewfromsomeonewithaknownstance.Choice3sparesscientistsfrom
havingtomakeprobabilisticinferencesastowhatothersubmissionsmighthavebeenmadebutrejected,whichwouldhavebeen
necessarytodeterminetheoptimalchoiceswhentheyact.Thesefeaturesenableustoexaminethelargestpossiblenumberof
submissionswiththeavailablecomputingpower,andthusallowustogeneratemoremeaningfuloutputswithoutchangingthe
essentialprocessesoperating.Webelievethatourmainmessagewillremainvalidwhentheseassumptionsarerelaxed(see
SupplementaryInformationforafurtherdiscussionofchoice3).However,thecomplexityofthecomputerprogramneededto
analysesuchcases,andthecorrespondingcomputingpowerrequired,willincreaseexponentially.Choice4assumesauthorsuseall
oftheinformationavailabletothem,inaccordancewithBayestheorem27,todeterminetherelativelikelihood(calledaposterior
belief)thateachofthetwoalternativehypothesesiscorrect.Then,beingmotivatedtopublishwhatistrue,eachscientistwillsubmita
manuscriptadvocatingthehypothesisthatismorelikelytobecorrectaccordingtotheirposteriorbelief,augmentedbyastandardtie
breakingruleoffollowingtheirownsignalwhenbothareequallylikely24.Thisisoneoftherationalityassumptionsthateconomists
placeonhumans.
Modelsofreviewerbehaviour
Inthefirstmodel,M1,scientistj=i+1recommendsacceptanceofscientisti'smanuscriptwiththesameprobability,denotedbyP(=
ti|,hj 2,ti,sj),thattheyinferthethemeofthemanuscripttobethecorrecthypothesis,byBayesrulebasedonalltheinformation
availabletothematthatpoint.Therefore,reviewersaswellasauthorsactguidedbyBayesianinferenceinthismodel.The
http://www.nature.com/nature/journal/v506/n7486/full/nature12786.html

6/12

4/2/2015

Modellingtheeffectsofsubjectiveandobjectivedecisionmakinginscientificpeerreview:Nature:NaturePublishingGroup

acceptanceprobabilitiesareendogenousandevolvedifferentlydependingonhowthepublicationhistoryunfolds.
Inthesecondmodel,M2,theacceptancedecisioniscompletelyindependentofthereviewerssubjectiveassessmentofthethemeof
themanuscript,andratherisbasedonother,largelyobjectivecharacteristicsofthemanuscript,suchasthequalityoftheresearch
methodology.Presumingthatthesetraitsarestatisticallyindependentofthemanuscriptsconclusion,theacceptanceprobabilitiesin
M2areindependentofboththethemeofthemanuscriptandtheassignedreviewer(insofarastheonlyfeaturethatdistinguishes
reviewersistheirassessmentofwhichhypothesisiscorrect).Thus,theacceptanceprobabilitiescanbethoughtofasthelikelihood
thatthemethodologicalqualityofthemanuscriptissufficienttowarrantpublication,andnotareflectionofwhetherornotthe
revieweragreeswiththeconclusions.However,ourmodeldoesnotspecifywhatthoseprobabilitiesshouldbe.Toaidcomparison
betweenthemodels,weconsideredtwocases.Inone,scientistj,irrespectiveoftheirownsignal,recommendsacceptanceofi's
manuscriptwithaprobabilityequaltotheexanteprobabilitythattheywouldrecommendacceptanceofismanuscriptinM1afterthe
samehistory(thisresultsinthesameexpectednumberofpublicationsinbothM1andM2).Intheother,theacceptanceprobability
remainsthesamethroughout,attheinitialexpectedacceptanceprobabilityoftheM1model,whichis.Toverifythis,notethat
scientist2wouldrecommendacceptanceofscientist1smanuscriptwithprobability
happenswithprobability

when agreeswith (which

)butwithprobability0.5otherwise.Hence,theexpectedprobabilityof

acceptanceis

.AstheresultsaresimilarinthetwocasesofM2,herewereportonlyontheformer.

Inthethird(benchmark)model,M3,allmanuscriptsarepublishedwithoutanyfilteringthroughpeerreview.Thismodelisidenticalto
M2butwiththeacceptanceprobabilityequalto1throughouttheprocess.Thisisasimplemodelofherdbehaviour24,28thathas
becomestandardineconomicswhenmodellingselfmotivated,rationalindividualswhosequentiallytakeactions.Aconsequenceof
thismodelisthateachscientistwillhaveaccesstoallprevioussubmissionswhenformingtheirdecision(becauseeverythingis
publishedinthismodel).Notethatthisdiffersfromafullinformationcase(thatis,whereeveryscientisthasaccesstoallprivate
signals,aswellaspublicactions).
InthegeneralizedM1models,scientistjrecommendsacceptancewithprobability
,andwithprobability

if
if

,wherev>0.The

casev=1correspondstotheoriginalM1model,withhighervaluesofvindicatingthattherecommendationismoreheavily
influencedbythereviewerssubjectiveassessmentontheadvocatedtheme,andlowervmeaningthatitislessso.
Definitionsandalgebraicformulae
Themisperceptionisdefinedfromtheperspectiveofoutsiderswhoobservethepublicationhistory.Usingalltheinformationavailable
tothemfromtheobservedhistory,
probability
realizesunderhypothesis

,outsideobserverswillformviaBayesruleaposteriorbeliefthatattaches
tohypothesisbeingtruefor

,where

istheprobabilitythatthehistoryhn

.Wedefinethemisperception,afterhistoryhn,astheexpectedposteriorprobabilityattachedto

thehypothesiswhichisinrealityincorrect:since

,itis:

Theexpectedmisperceptionafternsubmissionsisdefinedasaprobabilityweightedsumofmisperceptionsoverallpossiblehistories
oflengthnthatmayoccur:

Notethatthesecalculationsaredoneforanunderlyingvalueof.

http://www.nature.com/nature/journal/v506/n7486/full/nature12786.html

7/12

4/2/2015

Modellingtheeffectsofsubjectiveandobjectivedecisionmakinginscientificpeerreview:Nature:NaturePublishingGroup

Focusingonh1(forwhichweneedtwoscientists),therearethreepossiblehistories,namely

.Equation(2),above,

whichgivesustheexpectedmisperception,willhave6termswhenn=1,becauseeachofthethreehistoriescanoccurfromeither
hypothesis

.NotethatP(|h 1)issymmetricinthesensethatitsvalueremainsthesamewhenAandB(asvaluesofand

elementsofh 1)arepermuted.Aconsequenceofthissymmetryisthatweonlyneedtoconsiderthecasewhenonehypothesis(for
example,A)iscorrect,andthesumof6termswillbeequaltotwiceofthesumofthethreeitemsrelevantfor=A.Forn=2
becausethereare3 2=9possiblehistories,therewillbe9termstocalculate(aftertakingintoaccountthesymmetry).Similarly,the
expectedmisperceptionafternsubmissionscanbeobtainedbycalculating3nterms:

Herdingisdefinedforscientistswhoaresubmittingpapers.Ascientist,sayj,issaidtobeherdingiftheywouldchoosethesame
themetoadvocateregardlessoftheirprivatesignalastheirposteriorbeliefwouldattachaprobabilitymorethanonehalftoa
particularhypothesisregardlessoftheirownsignal,thatis,if:

Theprobabilityofherding,forascientistj,caneasilybecalculatedbythefollowingprobabilityweightedsum:

whereP(hj 2,tj 1)istheprobabilitythat(hj 2,tj 1)realizesfromeitherhypothesisand1Histheindicatorfunctionthatassumesa

valueof1if(4)holds,and0otherwise.
Whenherdingoccurs,somehistoriesandinformationprofileswilloccurwithprobabilityzero.Thismeansthattherewillgenerallybe
anumberoftermsin(3)and(5)thatwillneveroccur,sothecalculationsrequiredwillgenerallybeoverasmallernumberofterms
thanthetheoreticalupperbound.Nevertheless,thelargenumberoftermsthatresultfromevenamoderatenareimpossibleto
simplifytoobtainaclosedformalgebraicexpressionforeithertheexpectedmisperceptionortheprobabilityofherding.Wetherefore
wroteacomputerprogramtonumericallycalculatethealgebraicexpressionswithinavailablecomputingpower.
Computerprogram
Theprogram(codeprovidedintheSupplementaryInformation)workedbybuildingandevaluatingthealgebraicformulaetoobtain
resultsthatareaccurateuptothelevelofprecisionthecomputerusedinitscalculations(52dp),asexplainedthroughanumberof
keystepsdescribedbelowforvariousvaluesof.Theinformationareviewerjhas,

,isreferred

toastheirinformationprofile.
Step1:Foreachofthetwopossibleprivatesignalsofscientist1,
conditionaloneachofthetwohypothesis

hypothesisiscalculatedas:

if

,aprobabilityissetfortheoccurrenceofthatsignal
and

otherwise.Thus,theposterioronthetrue

Step2:Foreachsignal asubmissiondecisionofscientist1isprescribed.As
submittedpaper

willbeidenticaltotheirsignal(

,forscientist1thethemeoftheir

.Thisdeterminestheprobabilityof

Step3:Foreachpossibleinformationprofile

conditionalon

ofscientist2,theprobabilityofacceptance(ofscientist1ssubmission

withtheme )isdeterminedinaccordancewiththeadoptedmodel.ForM1(andhence,M2),thisinvolvescalculatingscientist2s
posteriorbeliefsas

where

Step4:Ifscientist1smanuscriptisrejected,ahistoryh 1=ensues.Ifaccepted,ahistoryh 1=t 1ensues.Foreachpossible

historyh 1,theconditionalprobabilityP(h 1|)isobtainedbyaggregatingtheprobabilitiesthatitarisesfromdifferentsignalprofiles
(s 1,s 2)conditionalon.Themisperceptioniscalculatedforeachhistoryaccordingtotheformula(1),andthentheexpected
http://www.nature.com/nature/journal/v506/n7486/full/nature12786.html

8/12

4/2/2015

Modellingtheeffectsofsubjectiveandobjectivedecisionmakinginscientificpeerreview:Nature:NaturePublishingGroup

misperceptionaccordingtotheformula(3).
Step5:Thesubmissiondecisionofscientist2,t 2,isequaltosuchthat
,then

ifsucha existsotherwise,thatis,if

.Thisdeterminestheconditionalprobability

.Herding(andotherresults)is

calculatedaccordingtotherelevantformulaegiven.
Step6:Steps35arerepeatedfor

foreverypossibleinformationprofile

ofscientistjwiththefollowing

modifications:scientistjsposteriorbeliefsare

where
replacesh 1and

instep3

isobtainedbycombining

andscientistjsacceptanceprobabilitygiventheirinformationprofile
replace

and

instep4and

and

,respectively,instep5.

Analyticalresultsonasymptoticproperties
Analyticcomparisonofdifferentmodelsisobtainedasymptoticallyasthenumbersofscientiststendstoinfinity.ConsiderM1.Let
denotethesetofallpossiblehistoriesoflengthn,and
for
Foreach ,let

denoteahistoryin

,constituteaninfinitesequenceof fieldson

betheexanteprobabilitythat willrealizefromeither

Bayesupdatedposteriorbeliefthat
constitutesamartingale.Let

after .Then,
.Then,with

.Let

isarandomvariabledefinedon
definedon

submartingale.BytheMartingaleConvergenceTheorem29,
withprobability1and

.Then,

.
denotethe
,and

,thesequence

constitutesa

almostsurelywhere isarandomvariablesuchthat

istakenrelativetoQ.

Considerahistory withthecorrespondingposterior

.Then,therearethreepossiblecontinuationhistoriesoflengthn+1:

followedbyA,B,or.Asthemanuscriptofscientistn+1isacceptedwithaprobabilitythatisstrictlybetween0and1,(i)atleast
twoofthethreepossiblecontinuationhistoriesrealizewithastrictlypositiveprobability.Furthermore,(ii)theposteriorsafterthese
continuationhistoriesdiffer,(iii)theydependonxbutnotonn,(iv)thedistributionovertheseposteriorsconditionalon=Afirst
orderstochasticallydominatesthatconditionalon=B.Hence,

isastrictlypositiveconstantthatdependsonx

butnotonn,andconsequently,E(X)<1isnotviable.AsE(X)1,therefore,weconcludethatE(X)=1,thatis,theposterior
convergestotruestatewithprobability1when
when

.Asasymmetricargumentappliestothecasethat

,thatis,

,themisperceptionconvergesto0asn underM1.

Next,considerthegeneralizedM1modelwithv>0.Aslongas0<v<1,itisstraightforwardtoverifythatthedeductions(i)(iv)hold
and,consequently,thesameargumentasaboveleadstothesameconclusionthatthemisperceptionconvergesto0asn .Ifv>
1,ontheotherhand,anymanuscriptonthemewillbeacceptedwithcertaintyoncetheposteriorbeliefforthethemebeingtrue
exceedsacertainthresholdlevelwhichisstrictlybelow1.Inaddition,thescientistswillsubmitonthepopularthemeregardlesstheir
ownsignaliftheposteriorforthatthemeexceedsa(different)threshold.Therefore,iftheposteriorbelieffor=Agetssufficiently
closeto1or0,boththeauthorsthemeselectionandthereviewersdecisionareuniquelydeterminedbytheprevailingposterior
independentlyofthescientistsownsignal.Oncethisstageisreached,thenthecontinuationhistoryisuniquelydetermined
(irrespectiveofwhether=AorB)unlike(i)aboveand,consequently,publicationoutcomesrevealnofurtherinformationandthe
posteriorremainsatthesamelevelforever.Therefore,theexpectedmisperceptionneverconvergesto0andremainsfixedatsome
positivelevelwithinfinitetimewithprobability1.
ForM2andM3,bythesametokentheexpectedmisperceptionneverconvergesto0andgetsstuckatsomepositiveleveloncethe
posteriorbeliefreachesalevelsuchthattheauthor'sthemeselectionisdictatedbyherdingindependentlyoftheirownsignal.

References
1. Ioannidis,J.P.A.Whymostpublishedresearchfindingsarefalse.PLoSMed.2,e124(2005)
2. Ioannidis,J.P.A.Scientificinbreedingandsameteamreplication:TypeDpersonalityasanexample.J.Psychosom.Res.73,
http://www.nature.com/nature/journal/v506/n7486/full/nature12786.html

9/12

4/2/2015

Modellingtheeffectsofsubjectiveandobjectivedecisionmakinginscientificpeerreview:Nature:NaturePublishingGroup

408410(2012)
3. Ioannidis,J.P.A.Contradictedandinitiallystrongereffectsinhighlycitedclinicalresearch.J.Am.Med.Assoc.294,218228
(2005)
4. Ioannidis,J.P.&Trikalinos,T.A.Earlyextremecontradictoryestimatesmayappearinpublishedresearch:theProteus
phenomenoninmoleculargeneticsresearchandrandomizedtrials.J.Clin.Epidemiol.58,543549(2005)
5. DaveySmith,G.inBiopsychosocialMedicine:AnIntegratedApproachtoUnderstandingIllness(ed.White,P.)77102
(OxfordUniv.Press,2005)
6. Tatsioni,A.,Bonitsis,N.G.&Ioannidis,J.P.A.Persistenceofcontradictedclaimsintheliterature.J.Am.Med.Assoc.298,
25172526(2007)
7. Freese,J.inIntergenerationalCaregiving(edsCrouterA.C.,BoothA.,BianchiS.M.&SeltzerJ.A.)145177(UrbanInstitute
Press,2008)
8. Ioannidis,J.P.A.Whyscienceisnotnecessarilyselfcorrecting.Perspect.Psychol.Sci.7,645654(2012)
9. Sterling,T.D.Publicationdecisionsandtheirpossibleeffectsoninferencesdrawnfromtestsofsignificanceorviceversa.J.
Am.Stat.Assoc.54,3034(1959)
10. Barnes,B.,Bloor,D.&Henry,J.ScientificKnowledge:ASociologicalAnalysis.(Univ.ChicagoPress,1996)
11. Kuhn,T.S.TheStructureofScientificRevolutions.(Univ.ChicagoPress,1962)
12. Yong,E.&Simonsohn,U.Thedatadetective.Nature487,1819(2012)
13. Martinson,B.C.,Anderson,M.S.&deVries,R.Scientistsbehavingbadly.Nature435,737738(2005)
14. Pfeiffer,T.&Hoffmann,R.Largescaleassessmentoftheeffectofpopularityonthereliabilityofresearch.PLoSONE4,
e5996(2009)
15. Brembs,B.,Button,K.&Munaf,M.R.Deepimpact:unintendedconsequencesofjournalrank.Front.Hum.Neurosci.7,291
(2013)
16. Button,K.S.etal.Powerfailure:whysmallsamplesizeunderminesthereliabilityofneuroscience.NatureRev.Neurosci.14,
365376(2013)
17. Greenberg,S.A.Howcitationdistortionscreateunfoundedauthority:analysisofacitationnetwork.Br.Med.J.339,b2680
(2009)
18. Murphy,S.E.etal.Theeffectoftheserotonintransporterpolymorphism(5HTTLPR)onamygdalafunction:ametaanalysis.
Mol.Psychiatry18,512520(2013)
19. Simmons,J.P.,Nelson,L.D.&Simonsohn,U.Falsepositivepsychology:undisclosedflexibilityindatacollectionandanalysis
allowspresentinganythingassignificant.Psychol.Sci.22,13591366(2011)
20. Rosenthal,R.Thefiledrawerproblemandtolerancefornullresults.Psychol.Bull.86,638641(1979)
21. Edwards,A.W.F.MoreonthetoogoodtobetrueparadoxandGregorMendel.J.Hered.77,138(1986)
22. Boutron,I.,Dutton,S.,Ravaud,P.&Altman,D.G.Reportingandinterpretationofrandomizedcontrolledtrialswithstatistically
nonsignificantresultsforprimaryoutcomes.J.Am.Med.Assoc.303,20582064(2010)
23. Gtzsche,P.C.Believabilityofrelativerisksandoddsratiosinabstracts:crosssectionalstudy.BMJ333,231234(2006)
24. Banerjee,A.V.Asimplemodelofherdbehavior.Q.J.Econ.107,797817(1992)
25. Asch,S.E.Studiesofindependenceandconformity.Psychol.Monogr.70,170(1956)
26. Almenberg,J.,Kittlitz,K.&Pfeiffer,T.Anexperimentonpredictionmarketsinscience.PLoSONE4,e8500(2009)
27. Bayes,T.&Price,R.Anessaytowardssolvingaprobleminthedoctrineofchances.PhilosophicalTransactions(16831775)
http://www.nature.com/nature/journal/v506/n7486/full/nature12786.html

10/12

4/2/2015

Modellingtheeffectsofsubjectiveandobjectivedecisionmakinginscientificpeerreview:Nature:NaturePublishingGroup

53,370418(1763)
28. Bikhchandani,S.,Hirshleifer,D.&Welch,I.Atheoryoffads,fashion,custom,andculturalchangeininformationalcascades.
J.Polit.Econ.100,9921026(1992)
29. Billingsley,P.ProbabilityandMeasure3rdedn(JohnWiley&Sons,1995)

Downloadreferences

Acknowledgements
ThisresearchwassupportedbyanEconomicsandSocialResearchCouncilUKPhDstudentshiptoM.W.P.M.R.M.isamemberof
theUKCentreforTobaccoandAlcoholStudies,aUKCRCPublicHealthResearchCentreofExcellence.FundingfromtheBritish
HeartFoundation,CancerResearchUK,EconomicandSocialResearchCouncil,MedicalResearchCouncil,andtheNational
InstituteforHealthResearch,undertheauspicesoftheUKClinicalResearchCollaboration,isgratefullyacknowledged.Theauthors
aregratefultoS.Murphyforherassistanceincodingthemetaanalysisstudyabstracts,andtoA.BirdandG.Huxleyfortheir
commentsonearlierdraftsofthismanuscript.

Authorinformation
Affiliations
DepartmentofEconomics,UniversityofBristol,BristolBS81TN,UK
InUckPark&MikeW.Peacey
DepartmentofEconomics,SungkyunkwanUniversity,Seoul110745,SouthKorea
InUckPark
DepartmentofEconomics,UniversityofBath,BathBA27AY,UK
MikeW.Peacey
MRCIntegrativeEpidemiologyUnit(IEU),UniversityofBristol,BristolBS81BN,UK
MarcusR.Munaf
UKCentreforTobaccoandAlcoholStudies,UniversityofBristol,BristolBS81TU,UK
MarcusR.Munaf
SchoolofExperimentalPsychology,UniversityofBristol,BristolBS81TU,UK
MarcusR.Munaf
Contributions
Allauthorscontributedequallytothedesignandanalysisofthemodelsandthewritingofthemanuscript.Theprojectwasconceived
byI.U.P.andM.R.M.,andthecomputerprogramwaswrittenbyM.W.P.
Competingfinancialinterests
Theauthorsdeclarenocompetingfinancialinterests.
Correspondingauthor
Correspondenceto:MarcusR.Munaf

Supplementaryinformation
PDFfiles
1. SupplementaryInformation(526KB)
http://www.nature.com/nature/journal/v506/n7486/full/nature12786.html

11/12

4/2/2015

Modellingtheeffectsofsubjectiveandobjectivedecisionmakinginscientificpeerreview:Nature:NaturePublishingGroup

ThisfilecontainsSupplementarySections14.Section1containsthecomputerprogramusedtofindthenumericalresults
presentedinthemainarticle.Section2presentsanexampleofdeherdingoccurringintheM1model.Section3discusses
theeffectofrelaxingthecommonknowledgeassumptionofrejections,andanalysesalternativemodelsforcomparisonand
Section4considersthecasethatthescientistsaremotivatedtosimplypublish(regardlessofthetruthfulnessofthe
defendedtheme),andshowsthatimmediateandcompleteherdingmayoccur.

Nature

ISSN00280836

EISSN14764687

2013MacmillanPublishersLimited.AllRightsReserved.
partnerofAGORA,HINARI,OARE,INASP,ORCID,CrossRefandCOUNTER

http://www.nature.com/nature/journal/v506/n7486/full/nature12786.html

12/12