ValidityandReliabilityin QuantitativeResearch Q

DrMohammedArif March9,2011

Validity
Theapproximatetruthofpropositions, inferences,orconclusions.

StagesofValidity
Sampling Measurement Design Analysis

SAMPLING

Sampling
Samplingistheprocessofselectinga p numberofunits(e.g., ( g people, p p representative organizations)fromapopulationofinterest, theintentbeingtogeneralizetheresultsof analyzingthesampleresultsbacktothe populationfromwhichtheywerechosen. chosen

Sampling ExternalValidity

P i lSi Proximal Similarity il it M Model d l

ImprovingExternalValidity
Userandomselection,ifpossible,ratherthan p anonrandomprocedure Trytoassurethattherespondentsparticipate inyourstudyandthatyoukeepyourdropout rateslow. Usethetheoryofproximalsimilaritymore effectivelybyapplyingtechniqueslikeconcept mapping

ConceptMapping

Sampling

NormalDistribution

ProbabilitySampling
Si Simple l Random d Sampling S li Toselect l n units i outof f N suchthateachNCn hasanequalchanceof b i selected. being l t d Use U atable t bl of frandom d numbers, b a computerrandomnumbergenerator,ora mechanicaldevicetoselectthesample sample. StratifiedRandomSampling Dividethe population l i into i nonoverlapping l i groups(i (i.e., strata)N1,N2,N3,...Ni,suchthatN1 +N2 +N3 +... +Ni =N. N Then Th do d asimple i l random d sample l of ff= n/Nineachstrata.

ProbabilitySamplingContd. Contd
SystematicRandomSampling
numbertheunitsinthepopulationfrom1toN decideonthen(samplesize)thatyouwantorneed k=N/n=theintervalsize randomlyselectanintegerbetween1tok thentakeeverykth unit

ClusterRandomSampling
dividepopulationintoclusters(usuallyalonggeographicboundaries) randomlysampleclusters measureall unitswithinsampledclusters

MultiStageSampling

NonProbabilitySampling
Accidental,HaphazardorConvenienceSampling Traditional

"persononthestreetinterviews conductedfrequentlyby televisionnewsprogramstogetaquickreadingofpublic opinion.Choiceofstudentsbecauseitisconvenientisalsoan p ofthistype yp ofsampling. p g example PurposiveSampling Wesamplewithapurpose inmind.We usuallywouldhaveoneormorespecificpredefinedgroups weareseeking.Forinstance,haveyoueverrunintopeoplein amalloronthestreetwhoarecarryingaclipboardandwho arestopping t i various i people l and dasking ki ifth theycould ldi interview t i them?Mostlikelytheyareconductingapurposivesample.

NonProbabilitySampling
PurposiveSamplingTypes
ModalInstanceSampling p g(Typical ( yp Voter) ) ExpertSampling QuotaSampling
Proportional(Proportionaltothepopulation) Non N proportional i l(Enough (E htod dothe h statistical i i ltests) )

HeterogeneitySampling(WideArrays) SnowballSampling(Choosesomeoneandaskhim torecommendmore)

MEASUREMENTS

ConstructValidity
Constructvalidityreferstothedegreeto g ybemade whichinferencescanlegitimately fromtheoperationalizations inyourstudyto thetheoreticalconstructsonwhichthose operationalizations werebased.

ConstructValidity

ExternalValidityVsConstructValidity
Externalvalidityinvolvesgeneralizingfrom yourstudy y ycontexttootherp people, p p placesor times,constructvalidityinvolvesgeneralizing fromyourprogramormeasurestothe concept ofyourprogramormeasures.

ConstructValidity
Translationvalidity
Facevalidity y Contentvalidity

Criterion C it i related l t dvalidity lidit

Predictivevalidity Concurrentvalidity Convergentvalidity Discriminant validity

TranslationalValidity
Howaccuratelyyoutranslated yourconstruct intotheoperationalization? FaceValidity seewhether"onitsface"it seemslike lik agood dtranslation l i of fthe h construct. ContentValidity y checktheoperationalization p againsttherelevantcontentdomainforthe construct. construct

Criterionrelatedvalidity
P Predictive di ti validity lidit In I predictive di ti validity lidit ,weassessthe th operationalization's abilitytopredictsomethingitshould theoreticallybeabletopredict. Concurrentvalidity Inconcurrentvalidity,weassessthe operationalization's abilitytodistinguishbetweengroups thatitshouldtheoretically ybeabletodistinguish g between. Convergentvalidity Inconvergentvalidity,weexamine thedegreetowhichtheoperationalization issimilarto (convergeson)otheroperationalizations thatit theoreticallyshouldbesimilarto. Discriminant validity Indiscriminant validity,weexamine th degree the d to t which hi hth theoperationalization ti li ti is i not tsimilar i il to t (divergesfrom)otheroperationalizations thatit theoreticallyshouldbenotbesimilarto.

ConvergentValidity
Measuresofconstructsthattheoretically should berelatedtoeachotherare,infact, observedtoberelatedtoeachother(thatis, youshouldbeabletoshowa correspondenceorconvergence between similarconstructs) CorrelationCoefficient

ConvergentValidityContd Contd.

Discriminant Validity
Measuresofconstructsthattheoretically shouldnot berelatedtoeachotherare,in fact,observedtonotberelatedtoeachother (thatis is,youshouldbeabletodiscriminate betweendissimilarconstructs)

Discriminant ValidityContd Contd.

PuttingitTogetherNow

TheNomological Network
Cronbach andMeehl, ,1955

MultitraitMultimethod Matrix (MTMM)

(CampbellandFiske,1959)

MultitraitMultimethod Matrix (MTMM)Contd.

TheReliabilityDiagonal Estimatesofthe yofeachmeasureinthematrix.You reliability canestimatereliabilitiesanumberofdifferent ways(e (e.g., g testretest, retest internalconsistency) consistency). TheValidityDiagonals Correlationsbetween measuresof fthe h sametraitmeasuredusing differentmethods.

MultitraitMultimethod Matrix (MTMM)Contd.

The h Heterotrait i Monomethod h d Triangles i l Thesearethecorrelationsamongmeasures thatsharethesamemethodofmeasurement. TheMonomethod Blocks Theseconsistofall ofthecorrelationsthatsharethesame methodofmeasurement. TheHeteromethod Blocks Theseconsistof allcorrelationsthatdonot sharethesame methods.

Example

APPLICATIONPRINCIPLES
C Coefficients ffi i t in i the th reliability li bilit diagonal di lshould h ld consistentlybethehighestinthematrix. Thatis, atraitshouldbemorehighlycorrelatedwith itselfthanwithanythingelse!Thisisuniformly trueinourexample. p Coefficientsinthevaliditydiagonalsshouldbe significantlydifferentfromzeroandhighenough towarrantfurtherinvestigation.Thisis essentiallyevidenceofconvergentvalidity.Allof thecorrelationsinourexamplemeetthis criterion.

APPLICATIONPRINCIPLESContd Contd.
Avaliditycoefficientshouldbehigherthan y ginitscolumnandrowinthesame valueslying heteromethod block. Inotherwords,(SE P&P)(SETeacher)shouldbegreaterthan(SE P&P)(SDTeacher),(SEP&P)(LCTeacher),(SE Teacher)(SDP&P)and(SETeacher)(LCP&P) P&P). Thisistrueinallcasesinourexample.

APPLICATIONPRINCIPLESContd Contd.
Avalidity lidit coefficient ffi i tshould h ldbe b higher hi h than th all ll coefficientsintheheterotraitmonomethod triangles Thisessentiallyemphasizesthattrait triangles. factorsshouldbestrongerthanmethodsfactors. Notethatthisisnot trueinallcasesinour example.Forinstance,the(LCP&P)(LCTeacher) correlationof.46islessthan(SETeacher)(SD T h ) (SET Teacher), Teacher) h )(LCTeacher), T h ) and d(SD Teacher)(LCTeacher) evidencethatthere mightmeamethodsfactor, factor especiallyonthe Teacherobservationmethod.

APPLICATIONPRINCIPLESContd Contd.
Thesamepatternoftraitinterrelationship g Theexample p shouldbeseeninalltriangles. clearlymeetsthiscriterion.Noticethatinall trianglestheSESDrelationshipis approximatelytwiceaslargeasthe relationshipsthatinvolveLC. LC

PatternMatching

Reliability
Reliabilityhastodowiththequalityof y ysense,reliability y measurement.Initseveryday isthe"consistency"or"repeatability"ofyour measures. measures

TrueScoreTheory

SoWhat?
Itis i asimple i l yetpowerful f lmodel d lfor f measurement.
Itremindsusthatmostmeasurementshaveanerror component. True T scoretheory th is i the th foundation f d ti of freliability li bilit theory.Ameasurethathasnorandomerror(i.e.,isall truescore)isperfectlyreliable;ameasurethathasno truescore(i.e.,isallrandomerror)haszeroreliability. Third, ,truescoretheory ycanbeusedincomputer p simulations asthebasisforgenerating"observed" scoreswithcertainknownproperties.

MeasurementErrors

SoWhatDoWeDo?
PilotTesting DataCollectionTraining DoubleCheckDataBeforeEntering Triangulation

TheoryofReliability
Inresearch,thetermreliabilitymeans p y or"consistency". y Ameasureis "repeatability" consideredreliableifitwouldgiveusthe sameresultoverandoveragain(assuming thatwhatwearemeasuringisn'tchanging!).

SoWhatisReliability?

CalculatingReliability
varianceofthetruescore/thevarianceof themeasure Wecan'tcomputereliabilitybecausewe can't can tcalculatethevarianceofthetruescores EstimatingReliability [covariance(X1,X2)]/sd(X1)*sd(X2)

TypesofReliability
InterRaterorInterObserver Obser erReliability Reliabilit Usedtoassessthedegreetowhichdifferentraters/observersgive consistentestimatesofthesamephenomenon. Test T tRetest R t tReliability R li bilit Usedtoassesstheconsistencyofameasurefromonetimeto another. Parallel P ll lForms F R Reliability li bilit Usedtoassesstheconsistencyoftheresultsoftwotests constructedinthesamewayfromthesamecontentdomain. Internal I lC Consistency i R Reliability li bili Usedtoassesstheconsistencyofresultsacrossitemswithinatest.

InterRaterorInterObserver Reliability

Consistency Correlation

TestRetestReliability

Correlation

ParallelFormsReliability

Correlation

InternalConsistencyReliability
Ininternal i lconsistency i reliability li bili estimation i i we useoursinglemeasurementinstrument administered d i i t dto t agroupof fpeople l onone occasiontoestimatereliability. Ineffectwejudgethereliabilityoftheinstrument byestimatinghowwelltheitemsthatreflectthe sameconstructyield i ldsimilar i il results. l Wearelookingathowconsistenttheresultsare fordifferentitemsforthesameconstructwithin themeasure.

InternalConsistencyReliability
AverageInteritemCorrelation AverageItemtotal Correlation SplitHalfReliability Cronbach's Alpha(a)

AverageInteritemCorrelation

AverageItemtotal Correlation

SplitHalfReliability

Cronbach's Cronbach s Alpha(a)

ReliabilityandValidity

QUESTIONS