Educational Research

Chapter 5
Selecting Measuring Instruments
Gay and Airasian

Collecting Data

The collection of data is an extremely

important part of all research
endeavors, for the conclusions for a
study are based on what the data
reveal.
As a result, the kind (s) of data to be
collected, the method (s) of collection
to used, and the scoring of the data
need to be considered with care.

Data

In this chapter:
Define data
Present several
types of
instruments that
can be used to
collect data in a
research study

Different
properties that
scores are
assumed to
possess

Objectives:Bytheendofthischapter
youshouldbeableto:

1)

Explainwhatismeant
bythetermdata
2) Explainwhatismeant
bytheterm
instrumentation
3(Namethreewaysin
whichdatacanbe
collectedbyresearchers

Explainwhatismeantby
thetermdatacollection
instrument
Describefivetypesof
researchercompleted
instrumentsusedin
educationalresearch
Describefivetypesof
subjectcompleted
instrumentsusedin
educationalresearch

Objectives

Explainwhatismeant
bytheterm
unobtrusive
measuresandgive
twoexamplesofsuch
measures
Namefourtypesof
measurementscales
andgiveanexample
ofeach

Namethree
differenttypesof
scoresusedIn
educational
researchandgive
anexampleofeach

Objectives:
Describebrieflythedifferencebetween
normreferencedandcriterion
referencedinstrument
Describehowtoscore,tabulate,and
codedataforanalysis

Flow of Activities in
Collecting
Flow of Activities Data
Example
Identify the variable
Operationally define the variable

Locate data (measures,

observations, documents with
questions and scales)
Collect data on instruments yielding
numeric scores

Self-efficacy for learning from

others
Level of confidence that an
individual can learn something
by being taught by others
13 items on a self-efficacy
attitudinal scale from Bergin
(1989)
Scores of each item ranged from
0-10 with 10 being completely
confident.

Data Collection

Scientific and disciplined inquiry

requires the collection, analysis, and
interpretation of data

Data the pieces of information that

are collected to examine the research
topic
Issues related to the collection of this
information are the focus of this chapter

Data Collection

Terminology related to data

Constructs abstractions that cannot

be observed directly but are helpful
when trying to explain behavior

Intelligence
Teacher effectiveness
Self esteem

Identify Data Options:

Specify Variables

Independent Variables
Dependent Variables
Intervening Variables
Control
Moderating
Confounding

Identify Data Options:

Operationalize Variables

Operational Definition: The specification of

how the variable will be defined and measured
typically based on the literature
often found in reports under definition of
terms
Sometimes the researcher must construct it

Some Times When Operational Definitions

Would Be Helpful
Figure 2.2

Which of the Following Definitions Are Operational?

Page 34

1. As shown by enthusiasm in class

2. As judged by the students math teacher using a
rating scale she developed
3. As measured by the Math Interest questionnaire
4. As shown by attention to math tasks in class
5. As reflected by achievement in mathematics
6. As indicated by records showing enrollment in
mathematics electives
7. As shown by effort expended in class
8. As demonstrated by number of optional
assignments completed
9. As demonstrated by reading math books outside
class
10.As observed by teacher aides using the
Mathematics Interest observation record

Data Collection

Data terminology (continued)

Operational definition the ways by

which constructs are observed and
measured

Weschler IQ test
Virgilio Teacher Effectiveness Inventory
Tennessee Self-Concept Scale

Variable a construct that has been

operationalized and has two or more
values

WHAT ARE DATA?

The term "data" refers to the kinds of

information researchers obtain on the
subjects of their research.
The term "instrumentation" refers to the
entire process of collecting data in a
research investigation.

KEY QUESTIONS

An important consideration in the

choice of an instrument to be used in
a research investigation is validity:

the extent to which results from it

permit researchers to draw warranted
conclusions about the characteristics
of the individuals studied.

CONDITIONS

It involves not only the

selection or design of the
instruments but also the
conditions under which
the instruments will be
administered.
1. Where? -- location
2. When? - - Time
3. How often?- Frequency

4. Who?
--administration
of the instruments

How you answer

these questions
may affect the
data obtained!

Good Instruments?

The data provided by

any instrument may
be affected by nay or
all of the preceding
considerations
If administered
incorrectly, disliked
Noisy or inhospitable
conditions
Subjects are
exhausted

Every instrument if it
is of any value must
allow
researchers to draw
accurate conclusions
about the
capabilities or other
characteristics of the
people being studied

VALIDITY, RELIABILITY, AND

OBJECTIVITY

1) Validity
An important consideration in the
choice of an instrument to be used in a
research investigation is validity:
the extent to which results from it
permit researchers to draw warranted
conclusions about the characteristics of
the individuals studied.

Reliability and Objectivity

2) A reliable instrument is one that gives

consistent results.
3) Whenever possible, researchers try to
eliminate subjectivity from the judgments they
make about the achievement, performance, or
characteristics of subjects. That is, the
researchers try to be objective.

USABILITY

Is it easy to use?
How long will it take to administer?
Are directions clear?
Is it appropriate for the ethnic group or other
groups to whom it will be administered?
How easy is it to score?
To interpret the scores?

Practical Questions

How much does it cost?

Do equivalent forms exist?
Have any problems been reported?
Does Evidence of its reliability and validity
exist?

Save time, energy and

headaches!!!

Who Provides the Information

Research instruments can be classified

in many ways.

Some of the more common are in terms

of who provides the data, the method of
data collection, who collects the data,
and what kind of response they require
from the subjects.

Data Obtained

Research data are data obtained by

directly or indirectly assessing the
subjects of the study.
Self-report data are data provided by
the subjects of the study themselves.
Informant data are data provided by
other people about the subjects of a
study.

Researcher Instruments

Many types of researcher-completed

instruments exist.

Some of the more commonly used are

rating scales, interview schedules, tally
sheets, flowcharts, performance
checklists, anecdotal records, and timeand-motion logs.

Subject Instruments

The types of items or questions

used in subject-completed
instruments can take many forms,
but they all can be classified as
either selection or supply items.

Subject Instruments

The types of items or questions

used in subject-completed
instruments can take many forms,
but they all can be classified as
either selection or supply items.

Subject Instruments

There are also many types of instruments

that are completed by the subjects of a
study rather than the researcher.
Some of the more commonly used of this
type are questionnaires; self-checklists;
attitude scales; personality inventories;
achievement, aptitude, and performance
tests; projective devices; and socimetric
devices.

Subject Instruments (con.t)

Examples of selection items

include true-false items, multiplechoice items, matching items, and
interpretive exercises. Examples of
supply items include short-answer
items and essay
questions.

Where Did the Instruments come

From?

1)

Find and administer a previously existing instrument

of some sort, or
2) administer an instrument the researcher personally
developed or had developed by someone else

An excellent source for locating already

available tests is the ERIC
Clearinghouse on Assessment and
Evaluation.

Data Collection

Measurement scales

Nominal categories

Ordinal ordered categories

Rank in class, order of finish, etc.

Interval equal intervals

Gender, ethnicity, etc.

Test scores, attitude scores, etc.

Ratio absolute zero

Time, height, weight, etc.

Four Types of Measurement Scales

Figure 7.25

50

SCALE

EXAMPLE

Nominal

Gender

Ordinal

Position in race

Interval

Temperature
(in Fahrenheit)

Ratio

Money

An Ordinal Scale: The Winner of a Horse Race

Figure 7.27

51

Data Collection

Types of variables

Categorical or continuous

Categorical variables reflect nominal scales

Continuous variables reflect ordinal, interval
or ratio scales

Independent or dependent

Independent variables are the purported

causes
Dependent variables are the purported effects

Measurement Instruments

Types of instruments (continued)

Affective (continued)

Scales used for responding to items on

affective tests

Likert
Semantic differential
Thurstone
Guttman
Rating scales

Examples of Items from a Likert Scale Measuring

Attitude toward Teacher Empowerment
Figure 7.14
Instructions: Circle the choice after each statement that indicates
your opinion.
1. All professors of education should be required to spend at least
six months teaching at
the elementary or secondary level every five years.
Strongly
agree
(5)

Agree
(4)

Undecided
(3)

Disagree
(2)

Strongly
disagree
(1)

Disagree
(4)

Strongly
disagree
(5)

2. Teachers unions should be abolished.

Strongly
agree
(1)

Agree
(2)

Undecided
(3)

3. All school administrators should be required by law to teach at

least one class in a
public school classroom every year.

44

Strongly
agree
(5)

Agree
(4)

Undecided
(3)

Disagree
(2)

Strongly
disagree
(1)

Example of the Semantic Differential

Figure 7.15
Instructions: Listed below are several pairs of adjectives. Place a
checkmark () on the line between each pair to indicate how you feel.
Example Hockey:
exciting :_____:_____:_____:_____:_____:_____:_____:_____: dull
If you feel that hockey is very exciting, you would place a check in the
first space next to the word exciting. If you feel that hockey is very
dull, you would place a checkmark in the space nearest the word dull.
If you are sort of undecided, you would place a checkmark in the middle
space between the two words. Now rate each of the activities that follow
[only one is listed]:
Working with other students in small groups

45

friendly
happy
easy
fun
hot
good
laugh
beautiful

:_____:_____:_____:_____:_____:_____:_____:_____:
:_____:_____:_____:_____:_____:_____:_____:_____:
:_____:_____:_____:_____:_____:_____:_____:_____:
:_____:_____:_____:_____:_____:_____:_____:_____:
:_____:_____:_____:_____:_____:_____:_____:_____:
:_____:_____:_____:_____:_____:_____:_____:_____:
:_____:_____:_____:_____:_____:_____:_____:_____:
:_____:_____:_____:_____:_____:_____:_____:_____:

unfriendly
sad
hard
work
cold
bad
cry
ugly

Measurement Instruments

Issues for cognitive, aptitude, or

affective tests

Bias distortions of a respondents

performance or responses based on
ethnicity, race, gender, language, etc.
Responses to affective test items

Socially acceptable responses

Accuracy of responses
Response sets

Problems inherent in the use of self-report

measures and the use of projective tests

Criterion-Referenced vs. Norm-Referenced

Evaluation Instruments
Page 158

Criterion-referenced: A student . . .

spelled every word in the weekly spelling list correctly.

solved at least 75 percent of the assigned problems.
achieved a score of at least 80 out of 100 on the final exam.
did at least 25 push-ups within a five-minute period.
read a minimum of one nonfiction book a week.

Norm-referenced: A student . . .

48

scored at the 50th percentile in his group.

scored above 90 percent of all the students in the class.
received a higher grade point average in English literature
than any other student in the school.
ran faster than all but one other student on the team.
and one other in the class were the only ones to receive As
on the midterm.

Selection of a Test

Designing you own tests

Get help from others with experience

developing tests
Item writing guidelines

Avoid ambiguous and confusing wording and

sentence structure
Use appropriate vocabulary
Write items that have only one correct answer
Give information about the nature of the desired
answer
Do not provide clues to the correct answer
See Writing Multiple Choice Items

Selection of a Test

Test administration guidelines

Plan ahead
Be certain that there is consistency
across testing sessions
Be familiar with any and all
procedures necessary to administer a
test

Identify Data Options:

Select Scales of
Nominal (Categorical):
Measurement

categories that describe traits or

characteristics participants can
check
Ordinal: participants rank order a
characteristic, trait or attribute

Identify Data Options:

Select Scales of
Interval: provides continuous
Measurement

response possibilities to questions

with assumed equal distance
Ratio: a scale with a true zero
and equal distances among units

Record and Administer Data

Collection: Locate or Develop an
Instrument

Develop your own instrument

Locate an existing instrument
Modify an existing instrument

Record and Administer Data

Collection: Obtain Reliable and
Valid
Data
Validity: the ability to draw meaningful and

justifiable inferences from the scores about a

sample or a population
Types of validity

Content (representative of all possible questions that

could be asked)
Criterion-referenced (scores are a predictor of an
outcome or criterion they are expected to predict
Construct (determination of the significance,
meaning, purpose and use of the scores)

Record and Administer Data

Collection: Develop Administrate
Procedures for Data Collection

Develop standard written procedures for

administering an instrument
Train researchers to collect
observational data
Obtain permission to collect and use
public documents
Respect individuals and sites during
data gathering

Illustration of Types of Evidence of Validity

Figure 8.1

52

Reliability and Validity

Figure 8.2

53

Methods of Checking Validity and Reliability

Table 8.2, page 180

VALIDITY (TRUTHFULNESS)

Method
Content-related evidence
Criterion-related evidence
Construct-related evidence

Procedure
Expert judgment
Relate to another measure of the same
variable
Assess evidence on predictions made
from theory

RELIABILITY (CONSISTENCY)

54

Method

Content

Time
Interval Procedure

Test-retest

Identical

Varies

Equivalent
forms

Different None

Give identical instrument twice

Give two forms of instrument

Equivalent
Different Varies
forms/ retest

Give two forms of instrument, with time

interval between

Internal
Different None
consistency
Observer
Identical None
agreement

Divide instrument into halves and score

each or use KR
Compare scores obtained by two or more
observers

More About Research: Threats to Internal Validity

in Everyday Life
Box 9A, page 199

55

Consider the following commonly held beliefs:

Because failure often precedes suicide, it is therefore the
cause of suicide. (probable history and mortality threat)
Boys are genetically more talented in mathematics than are girls.
(probable subject attitude and location threats)
Girls are genetically more talented in language than are boys.
(probable location and subject attitude threats)
Minority students are less academically able than students from
the dominant culture. (probable subject characteristics, subject
attitude, location, and instrumentation threats)
People on welfare are lazy. (probable subject characteristics,
location, and history threats)
Schooling makes students rebellious. (probable maturation and
history threats)
A policy of temporarily expelling students who dont behave
improves a schools test scores. (probable mortality threat)
Indoctrination changes attitude. (probable testing threat)
So-called miracle drugs cure intellectual retardation. (probable
regression threat)
Smoking marijuana leads eventually to using cocaine and heroin.
(probable mortality threat)

Illustration of Threats to Internal Validity

Figure 9.2

56

Note: We are not implying that any of these statements are necessarily true; our guess is that some are
and some are not.
*This seems unlikely.

If these teacher characteristics are a result of the type of school, then they do not constitute a threat.

General Techniques for Controlling Threats

to Internal Validity
Table 9.1, page 202

Threat

Obtain
Obtain More
More
Choose
Standardize Information Information Appropriate
Conditions on Subjects on Details
Design

Subject characteristics
Mortality
Location
Instrumentation
Testing
History
Maturation
Subject attitude
Regression
Implementer

57

X
X
X
X

X
X
X
X

X
X

X
X
X
X
X
X
X
X
X

Technical Issues

Validity (continued)

Consequential to what extent are the

consequences that occur from the test
harmful

Estimated by empirical and expert judgment

Factors affecting validity

Unclear test directions

Confusing and ambiguous test items
Vocabulary that is too difficult for test takers

Technical Issues

Validity (continued)

Factors affecting validity

Overly difficult and complex sentence

structure
Inconsistent and subjective scoring
Untaught items
Failure to follow standardized administration
procedures
Cheating by the participants or someone
teaching to the test items

Technical Issues

Validity extent to which

interpretations made from a test
score are appropriate

Characteristics

The most important technical characteristic

Situation specific
Does not refer to the instrument but to the
interpretations of scores on the instrument
Best thought of in terms of degree

Technical Issues

Validity (continued)

Four types

Content to what extent does the test

measure what it is supposed to measure

Item validity
Sampling validity
Determined by expert judgment