CHAPTER 5

MEASUREMENT, RELIABILITY AND

VALIDITY
Nguyen Tien Dzung
Hanoi University of Science and Technology
Email: dungnt-fem@mail.hut.edu.vn
Website: http://dungnt.tk
CHAPTER OVERVIEW
1. The Measurement Process
2. Levels of Measurement
3. Reliability and Validity: Why They Are Very, Very Important
4. A Conceptual Definition of Reliability
5. Validity
6. The Relationship Between Reliability and Validity
7. A Closing (Very Important) Thought
2012 NTD 2 Research Methods - Chapter 5
1. The Measurement Process
Two definitions
Stevensassignment of numerals to objects or events
according to rules.
the assignment of values to outcomes.
2012 NTD 3 Research Methods - Chapter 5
2. Levels of Measurement
Variables are measured at one of these four levels
Qualities of one level are characteristic of the next level up
The more precise (higher) the level of measurement, the more accurate
is the measurement process
Level of
Measurement
For example Quality of Level
Ratio A is 5 10 and B is 5 5 Absolute zero
Interval A is 5 taller than B An inch is an inch is an inch
Ordinal A is taller than B Greater than
Nominal A is tall and B is short Different from
2012 NTD 4 Research Methods - Chapter 5
Nominal Scale
Qualities Example What You
Can Say
What You Cant
Say
Assignment of
labels
Gender
(male or
female)
Preference
(like or dislike)
Voting record
(for or
against)
Each
observation
belongs in its
own category
An observation
represents more
or less than
another
observation
2012 NTD 5 Research Methods - Chapter 5
Ordinal Scale
Qualities Example What You
Can Say
What You Cant
Say
Assignment of
values along
some underlying
dimension
Rank in college
Order of finishing a
race
One
observation is
ranked above
or below
another.
The amount that
one variable is
more or less than
another
2012 NTD 6 Research Methods - Chapter 5
Interval Scale
Qualities Example What You
Can Say
What You Cant
Say
Equal distances
between points
Number of words
spelled correctly
Intelligence test
scores
Temperature
One score
differs from
another on
some
measure that
has equally
appearing
intervals
The amount of
difference is an
exact
representation of
differences on
the variable
being studied
2012 NTD 7 Research Methods - Chapter 5
Ratio Scale
Qualities Example What You Can
Say
What You Cant
Say
Meaningful and
non-arbitrary
zero
Age
Weight
Time
One value is
twice as much
as another or
no quantity of
that variable
can exist
Not much!
2012 NTD 8 Research Methods - Chapter 5
Continuous vs. Discrete Variables
Continuous Variables:
can assume any value along some underlying continuum.
Examples: length, time, revenues, income
Discrete Variables:
Categorical variables
Those with values that can be placed into certain
categories that have definite boundaries.
Examples: gender, customer satisfaction (scale from 1 to
5)
2012 NTD Research Methods - Chapter 5 9
What Is All The Fuss?
Measurement should be as precise as possible
In behavioral and social science, most variables are
probably measured at the nominal or ordinal level.
2012 NTD 10 Research Methods - Chapter 5
3. Reliability and Validity
Importance of reliability and validity
The methods and tools we use to measure the interested
object influence on the research accuracy.
Reliability: ability to produce the same results for
different measurements.
Validity: ability to measure what we want to measure.
2012 NTD Research Methods - Chapter 5 11
Reliability
Observed
Score
True
Score
Error
2012 NTD Research Methods - Chapter 5 12
Trait
Error
Method
Error
Error
A Conceptual Definition Of Reliability
Method error is due to characteristics of the test or testing situation
Trait error is due to individual characteristics
Reliability of the observed score becomes higher if error is reduced!!
Method Error
Observed Score = True Score + Error Score
Trait Error
2012 NTD 13 Research Methods - Chapter 5
INCREASING RELIABILITY
Decreasing Error
Increase sample size
Eliminate unclear questions
Standardize testing conditions
Use both easy and difficult questions
Minimize the effects of external events
Standardize instructions
Maintain consistent scoring procedures
2012 NTD 14 Research Methods - Chapter 5
How Reliability Is Measured
Reliability is measured using a
Correlation coefficient
r
test1test2
Reliability coefficients
Indicate how scores on one test change relative to scores
on a second test
Can range from -1.0 to +1.0 (perfect reliability)
2012 NTD 15 Research Methods - Chapter 5
Types of Reliability
Type of
Reliability
What It Is How You Do It What the Reliability
Coefficient
Looks Like
Test-Retest A measure of stability Administer the same test/measure
at two different times to the same
group of participants
r
test1test1
Parallel Forms A measure of
equivalence
Administer two different forms of
the same test to the same group
of participants
r
form1form2
Inter-Rater A measure of
agreement
Have two raters rate behaviors
and then determine the amount
of agreement between them
Percentage of agreements
Internal
Consistency
A measure of how
consistently each
item measures the
same underlying
construct
Correlate performance on each
item with overall performance
across participants
Cronbachs alpha
2012 NTD 16 Research Methods - Chapter 5
Validity
A valid test does what it was designed to do
A valid test measures what it was designed to
measure
2012 NTD 17 Research Methods - Chapter 5
A Conceptual Definition of Validity
Validity refers to the tests results, not to the test itself
Validity ranges from low to high, it is not either/or
Validity must be interpreted within the testing context
E.g.:
Calculation: 2 + 7 = ?
Which ability does the task measure?
Addition ability or Multiplication ability or Mathematical ability?
2012 NTD 18 Research Methods - Chapter 5
Validity
Validity tool measures what-it-should
Example:
Question: 2 + 3 = 5
I check 10 times with ten addition calculations, you answer 100%
correctly your answer is very reliable.
I use the test to measure your intelligence. Is the test valid?
NO. BECAUSE THIS TEST IS VALID ONLY FOR MEASURING
ADDITION ABILITY, NOT MORE THAN THAT; INTELLIGENCE IS
VERY COMPLICATED ABILITY.
2012 NTD 19 Research Methods - Chapter 5
Example of Validity and Reliability
A wife wants to measure the loyalty of her husband
She uses a criterion: never come late
He actually never comes home late.
Is the test valid?
2012 NTD 20 Research Methods - Chapter 5
Types of Validity
Type of
Validity
What Is It? How Do You Establish It?
Content A measure of how well the
items represent the entire
universe of items
Ask an expert if the items assess
what you want them to
Criterion
Concurrent A measure of how well a
test estimates a criterion
Select a criterion and correlate
scores on the test with scores on
the criterion in the present
Predictive A measure of how well a
test predicts a criterion
Select a criterion and correlate
scores on the test with scores on
the criterion in the future
Construct A measure of how well a
test assesses some
underlying construct
Assess the underlying construct on
which the test is based and
correlate these scores with the
test scores
2012 NTD 21 Research Methods - Chapter 5
Content Validity
A measure of how well the items represent the
entire universe of items
Ask experts to check whether the items assess
what you want them to access
E.g.:
You want to measure education quality
You must ask experts. They show you some relevant
criteria to measure (teachers, libraries, facilities,
curricula, research activities and applications, reputation
)
2012 NTD 22 Research Methods - Chapter 5
Exercise for homework
Project 1:
You want to measure: personal intelligence
How to define the concept? (What are the criteria of
intelligence?)
Project 2:
You want to measure: personal happiness
How to measure?
2012 NTD 23 Research Methods - Chapter 5
Criterion Validity
Concerned with
how well a test either estimates performance (concurrent validity) or
how well a test predicts future performance (predictive vadility)
Example:
A research to measure how GPA (Grade Point in Average) of
graduate students relate to their research ability
The research ability is measured by the number of articles published
on journals
If the correlation between GPA and the number of published articles
is high, it means GPA is a good predictor of the number of published
articles (high predictive validity)
Does a high number of published articles mean a good research
ability? is the question about concurrent validity of the research
2012 NTD 24 Research Methods - Chapter 5
How To Establish Construct Validity of
A New Test
Correlate new test with an established test
Show that people with and without certain traits
score differently
Determine whether tasks required on test are
consistent with theory guiding test development
2012 NTD 25 Research Methods - Chapter 5
Multitrait-multimethod Matrix
2 research projects about children:
Project 1: To measure impulsivity (Trait 1)
Project 2: To measure activity level or movement (Trait 2)
To test a construct validity, use multitrait-
multimethod matrix
Project 1:
Method 1: paper-and-pencil instrument on childs wrists
Method 2: activity meter (e.g.: observation)
Project 2:
Method 1: paper-and-pencil instrument on childs wrists
Method 2: activity meter (e.g.: observation)
2012 NTD 26 Research Methods - Chapter 5
MULTITRAIT-MULITMETHOD MATRIX
Convergent validitydifferent methods yield similar results
Discriminant validitydifferent methods yield different results
Method 1
Paper and
Pencil
Method 2
Activity Level
Monitor
Method 1
Paper and
Pencil
Method 2
Activity Level
Monitor
Trait 1
Method 1
Paper and Pencil Moderate Low
Impulsivity
Method 2
Activity Level
Monitor
Moderate
Trait 2
Method 1
Paper and Pencil
Activity
Level
Method 2
Activity Level
Monitor
Low
Trait 1
Impulsivity
Trait 2
Activity Level
2012 NTD 27 Research Methods - Chapter 5
The Relationship Between Reliability and Validity
(The Books Point Of View)
A valid test must be reliable
But
A reliable test may not be valid
A high reliability is a necessity to have a high validity, but not
ensure
2012 NTD 28 Research Methods - Chapter 5
VALIDITY AND RELIABILITY (The teachers point of
view)
a) High validity and high reliability b) High validity but low reliability
c) Low validity but high reliability d) Low validity and low reliability
2012 NTD 29 Research Methods - Chapter 5