2/28/2017 www.medscape.

com/viewarticle/875260_print

www.medscape.com

GradingtheQualityofEvidenceinComplex
Interventions
AGuideforEvidencebasedPractitioners
MHassanMuradJehadAlmasriMouazAlsawasWigdanFarah
EvidBasedMed.201722(1):2022.

AbstractandIntroduction
Abstract

Evidencebasedpractitionerswhowanttoapplyevidencefromcomplexinterventionstothecareoftheirpatientsareoften
challengedbythedifficultyofgradingthequalityofthisevidence.UsingtheGRADE(GradingofRecommendations,
Assessment,DevelopmentandEvaluation)approachandanillustrativeexample,weproposeaframeworkforevaluatingthe
qualityofevidencethatdependsonobtainingfeedbackfromtheevidenceuser(eg,guidelinepanel)toinform:(1)properframing
ofthequestion,(2)judgementsaboutdirectnessandconsistencyofevidenceand(3)theneedforadditionalcontextualand
qualitativeevidence.Usingthisframework,differentevidenceusersandbasedontheirneedswouldconsiderthesameevidence
ashigh,moderate,loworverylow.

Introduction

Complexinterventionsinmedicinearedefinedasinterventionsthatcontainseveralinteractingcomponentswithinthe
experimentalandcontrolinterventions,multiplebehavioursrequiredbythosedeliveringorreceivingtheintervention,multiple
groupsororganisationallevelstargetedbytheintervention,andpermitforvariableflexibilityandtailoringoftheintervention. [1]
Anexampleiscaremanagementprogrammesfortype2diabetes.Inonesystematicreview,programmesvariedacrossstudies
intermsofteamcomposition(physician,nurse,casemanager,diabeteseducator),deliverymethod(facetoface,telephone,
online,outpatient,inpatient),intensityandfrequencyoftheintervention,andotherfactors. [2]Complexinterventionsare
increasinglyusedinstudiesofbehaviouralchange,psychotherapy,education,publichealth,healthservices,quality
improvement,socialpolicyandmanyotherfields.

TheChallenge
Justaswithanyotherintervention,evidencebasedpractitionerswhowanttoapplyevidencefromcomplexinterventiontothe
careoftheirpatientsneedtoselecttheevidence(hopefullythroughasystematicreviewprocess)andappraiseit(ie,identifythe
extenttowhichthisevidenceistrustworthy,alsocalled,ratingthequalityofevidence).TheGRADE(Gradingof
Recommendations,Assessment,DevelopmentandEvaluation)approachisamodernmethodforratingthequalityofevidence
withgoodtransparencyandreliability. [3]AstudyofCochranesystematicreviewsshowedthattheoutcomesofcomplex
interventionsweremorelikelytoberatedas'verylow'qualityofevidencecomparedwiththoseofsimpleinterventions(37.5%vs
9.1%).Noneoftheoutcomesofcomplexinterventionreviewswereratedas'high'. [4]Webelievetheselowratingsare
inconsistentwiththedefinitionofqualityofevidenceinGRADE,whichisaconstructthatreflectsthetrustworthinessof
evidence,andweattributethisphenomenontoimproperframingoftheclinicalquestionforwhichthequalityofevidenceisbeing
rated. [5]

Rationale
Complexinterventionsareinherentlyheterogeneousthus,wemaybetemptedtolowerthequalityofevidenceforheterogeneity
(inconsistencyacrossstudies).Complexinterventionsarealsolikelytohavecomponentsthataredifferentfromtheoneswecan
orwanttoimplementinoursettingthus,wemaybetemptedtolowerthequalityofevidenceforindirectness(lackof
applicability).Therefore,byratingdownforheterogeneityandindirectness,mostcomplexinterventionstudieswillturnoutto
providelowqualityevidencewhichisfrustratingtoevidenceusers. [4]

ProposedApproach
WepresentaframeworkforevaluatingthequalityofevidenceusingGRADE.Theframeworkhingesonobtainingfeedbackfrom
theenduserofevidence(eg,aguidelinepanel)duringtheprocessofevidencesynthesis(eg,conductingasystematicreviewto
supportaguideline).Thisfeedbackseekstoinform:(1)properframingofthequestion,(2)judgementsaboutdirectnessand
consistencyofevidence(thetwodomainsthatarehighlyrelevanttocomplexinterventions)and(3)theneedforadditional
contextualandqualitativeevidencetoprovideinformationaboutthecircumstancesunderwhichtheinterventionworksbest
(figure1).Thisqualitativeevidencecanexplainbarriersandfacilitatorsofimplementation,aswellastheculturalandsocial
factorsthatcanmodifytheeffectoftheintervention.
http://www.medscape.com/viewarticle/875260_print 1/5
2/28/2017 www.medscape.com/viewarticle/875260_print

Figure1.

Thetraditionalprocessofratingthequalityofevidenceisinwhitecolour.Theadditionalproposedstepsthataddresscomplex
interventionsareingreycolour.

Althoughengagingpatientsandevidenceusersintheprocessofevidencesynthesisisnotnew[6]andisroutinelycarriedoutin
somesystematicreviews(eg,stakeholderengagementismandatedinreviewsconductedbyEvidencebasedPracticeCenters
fundedbytheAgencyonHealthcareQualityandResearch), [6]theroleofthisengagementhasbeentodevelopthescopeand
keyquestions.Here,weadvocateforusingtheirfeedbacktoinformratingthequalityofevidenceofcomplexinterventions.
Finally,evidencesynthesismethodsotherthantraditionalquantitativemetaanalysiswilllikelybeneededtoaddresslongcausal
chainsincomplexinterventions,suchasmodeldrivenandrealisticsystematicreviewsandmetanarrativeevidencesynthesis.

Example
Theframeworkisappliedthroughanillustrativeexamplethatdemonstrateshowtheperspectiveoftheevidenceuseraffects
ratingofqualityofevidenceandshowshowthesameevidencecanwarrantdifferentlevelsoftrustworthinessfordifferentusers
().Forexample,policymakersorpayers(healthinsurancecarriers)maynotbeconcernedaboutclinicalandstatistical
heterogeneitybecausetheyareinterestedinanymanagementprogrammethatimprovesdiabetescontrol.Theyrealisethat
differenthealthsystemsundertheirjurisdictionmaydevelopdifferentprogrammeswithvariablecomponentsbasedoneach
systemsettingandresources(somedeliveredfacetoface,somedeliveredremotely,someledbynurse,etc).Therefore,froma
policymakerperspective,theremaybelittleinterestinknowingmoreabouttheindividualcomponentsoftheintervention.This
differssubstantiallyfromthepatientandclinicianperspectivesbecausetheymaybeinterestedinonlychoosingthemost
effectivecomponentsofdiabetesmanagementthatcanfitinthepatientcapacity,scheduleanddailyroutine.Thethird
perspectivepresentedinthetableisthatofadiabeteseducatorresponsibleforimplementingtheprogramme,whichrequires

http://www.medscape.com/viewarticle/875260_print 2/5
2/28/2017 www.medscape.com/viewarticle/875260_print

knowingmuchmoredetailsaboutimplementationandcontextualfactors.Adoptingthisperspectiveleadstoconsideringthe
evidenceofeffectivenesstobeinsufficientforthequestionathand.Twoqualitativestudies[7,8]wereidentifiedandprovided
someoftheneededdetails.Thestudiesshedlightonthepsychologicalfactorsthatshouldbeaddressedwhendelivering
diabetesmanagementprogrammesandprovidedinsightonthefactorsthataffectadherencetosuchprogrammes().Withthese
contextualfactorsknown,thepersonsprovidingtheprogrammewillhaveincreasedconfidenceintheirabilitytodeliverthe
programmeandtheymayratethequalityoftheevidenceashigh.

Table1.Caremanagementinterventionsfortype2diabetes

Framingof Clinical Statistical Qualityofevidence

Evidenceuser Perspective
thequestion heterogeneity heterogeneity (certaintyinevidence)*

Towhat
ShouldI Noconcern,weare Noconcern,we
extentisthis
cover(pay interestedinthe areinterestedin
Policymaker/payer bundleof High
for)this effectofthewhole theeffectofthe
interventions
intervention? bundle. wholebundle.
effective?

Variabilityof
Noconcernif Moderate(ifstatistical
Whichpart Which componentsis
metaregression heterogeneitywas
ofthe componentof concerning
explains explained)
Physicianand intervention the particularlyifa
heterogeneity(ie,
patient shouldwe interventionis componentisnotwell
demonstratethe Low(ifstatistical
provideto most describedornot
effectofeach heterogeneitywasnot
thispatient? effective? feasibleinown
component). explained)
practice.

Theeffectivenessevidence
Underwhat isinadequatetojudge
HowcanI circumstances certainty.Additional
Diabetes
applythe doesthe Noconcern Noconcern contextualevidencefrom
counsellor/educator
intervention? intervention qualitativeresearchis
workbest? identifiedleadingtohigh
certainty.

Evidence:metaanalysisdemonstratedthattheinterventionssignificantlyreducedHbA1c(0.21%,95%CI0.40%to0.03%).
Acrossstudies,therewasimportantclinicalheterogeneity(varyingcomponentsoftheintervention)andstatisticalheterogeneity
(varyingeffectsizewithI 2>75%).Followingatraditionalapproach,theevidencemaybegradeddowntwiceforindirectness
(numerouscomponentsmakeapplicationchallenging)andinconsistency(heterogeneityofeffect).
*Thefocusinthisexampleisonlyonthedomainsofindirectnessandinconsistency.Weassumedthattherewerenoconcernsin
otherqualityofevidencedomainssuchasprecision,riskofbiasorconcernsaboutHbA1cbeingasurrogateoutcome.
Contextualevidenceshowsthatasignificantamountofqualitativestudiesareavailabletoprovidecontextanddetailson
implementation.Forexample,studiesdemonstratedthatbeforeprovidingeducationindiabetes,weshouldaddressnegative
feelingsthatindividualshave(angerabouttheinitialdiagnosis,andsubsequentlynegotiatelossofcontrolofaperson'sown
lifestyleandenvironment).7Qualitativestudiesalsoprovidedexplanationtolackofadherencetoaprogramme(individual,
organisationalandcontentfactors).8Addressingthesefactorsisessentialtoprogrammesuccess.
HbA1c,glycatedhaemoglobin.

Table1.Caremanagementinterventionsfortype2diabetes

Framingof Clinical Statistical Qualityofevidence

Evidenceuser Perspective
thequestion heterogeneity heterogeneity (certaintyinevidence)*

Towhat
ShouldI Noconcern,weare Noconcern,we
extentisthis
cover(pay interestedinthe areinterestedin
Policymaker/payer bundleof High
for)this effectofthewhole theeffectofthe
interventions
intervention? bundle. wholebundle.
effective?

Physicianand Whichpart Which Variabilityof Noconcernif

patient ofthe componentof componentsis metaregression Moderate(ifstatistical
intervention the concerning explains heterogeneitywas
shouldwe interventionis particularlyifa heterogeneity(ie, explained)

http://www.medscape.com/viewarticle/875260_print 3/5
2/28/2017 www.medscape.com/viewarticle/875260_print

provideto most componentisnotwell demonstratethe Low(ifstatistical

thispatient? effective? describedornot effectofeach heterogeneitywasnot
feasibleinown component). explained)
practice.
Theeffectivenessevidence
Underwhat isinadequatetojudge
HowcanI circumstances certainty.Additional
Diabetes
applythe doesthe Noconcern Noconcern contextualevidencefrom
counsellor/educator
intervention? intervention qualitativeresearchis
workbest? identifiedleadingtohigh
certainty.

Evidence:metaanalysisdemonstratedthattheinterventionssignificantlyreducedHbA1c(0.21%,95%CI0.40%to0.03%).
Acrossstudies,therewasimportantclinicalheterogeneity(varyingcomponentsoftheintervention)andstatisticalheterogeneity
(varyingeffectsizewithI 2>75%).Followingatraditionalapproach,theevidencemaybegradeddowntwiceforindirectness
(numerouscomponentsmakeapplicationchallenging)andinconsistency(heterogeneityofeffect).
*Thefocusinthisexampleisonlyonthedomainsofindirectnessandinconsistency.Weassumedthattherewerenoconcernsin
otherqualityofevidencedomainssuchasprecision,riskofbiasorconcernsaboutHbA1cbeingasurrogateoutcome.
Contextualevidenceshowsthatasignificantamountofqualitativestudiesareavailabletoprovidecontextanddetailson
implementation.Forexample,studiesdemonstratedthatbeforeprovidingeducationindiabetes,weshouldaddressnegative
feelingsthatindividualshave(angerabouttheinitialdiagnosis,andsubsequentlynegotiatelossofcontrolofaperson'sown
lifestyleandenvironment).7Qualitativestudiesalsoprovidedexplanationtolackofadherencetoaprogramme(individual,
organisationalandcontentfactors).8Addressingthesefactorsisessentialtoprogrammesuccess.
HbA1c,glycatedhaemoglobin.

Implications

Webelievethisapproachwillhopefullyleadtomoreappropriateandpossiblyconsistentjudgementsaboutstudiesofcomplex
interventionsandbetterapplicationofGRADE.Thisapproach,however,willnotsolveotherchallengesofcomplexinterventions.
Forexample,lackofblindingofoutcomeassessorsandinadequateallocationconcealment(botharefeasibleevenifthestudy
wasunblindedandopen)andpoorreportingofthedetailsoftheintervention(whichmakesthemhardtoreplicateinpractice).
Thesechallengescanbesolvedattheprimaryresearchlevel,andnotatthelevelofevidencesynthesis,appraisalorapplication.

References

1.CraigP,DieppeP,MacintyreS,etal.Developingandevaluatingcomplexinterventions:thenewMedicalResearch
Councilguidance.BMJ2008337:a1655.

2.EggintonJS,RidgewayJL,ShahND,etal.Caremanagementfortype2diabetesintheUnitedStates:asystematic
reviewandmetaanalysis.BMCHealthServRes201212:72.

3.MustafaRA,SantessoN,BrozekJ,etal.TheGRADEapproachisreproducibleinassessingthequalityofevidenceof
quantitativeevidencesyntheses.JClinEpidemiol201366:73642quiz42e15.

4.MovsisyanA,MelendezTorresGJ,MontgomeryP.Outcomesinsystematicreviewsofcomplexinterventionsnever
reached"high"GRADEratingswhencomparedwiththoseofsimpleinterventions.JClinEpidemiol201678:2233.

5.MuradMH.ApplyingGRADEtostudiesofcomplexinterventionsispossiblebutrequiresproperframingofthequestion.J
ClinEpidemiol201677:138139.

6.ShippeeND,DomecqGarcesJP,PrutskyLopezGJ,etal.Patientandserviceuserengagementinresearch:asystematic
reviewandsynthesizedframework.HealthExpect201518:115166.

7.BoyleE,SaundersR,DruryV.Aqualitativestudyofpatientexperiencesoftype2diabetescaredeliveredcomparatively
bygeneralpracticenursesandmedicalpractitioners.JClinNurs201625:197786.

8.SchwennesenN,HenriksenJE,WillaingI.Patientexplanationsfornonattendanceattype2diabetesselfmanagement
education:aqualitativestudy.ScandJCaringSci201630:18792.

Provenanceandpeerreview
Notcommissionedinternallypeerreviewed.

EvidBasedMed.201722(1):2022.2017BMJPublishingGroup
http://www.medscape.com/viewarticle/875260_print 4/5
2/28/2017 www.medscape.com/viewarticle/875260_print
ThiswebsiteusescookiestodeliveritsservicesasdescribedinourCookiePolicy.Byusingthiswebsite,youagreetotheuseof
cookies.
close

http://www.medscape.com/viewarticle/875260_print 5/5