This presentation was prepared by Del Siegle.

Some of the material is from an earlier

presentation by Scott Brown.

Experimental
RESEARCH

c. 2000 Del Siegle

DESIGNS
 Experimental Research
Designs have Two Purposes:

• …to provide
answers to
research
questions
• ...to control
variance
(differences)

c. 2000 Del Siegle

 The main function of the
experimental research
design is to control
variance.
• Principle: maximize
systematic variance,
control extraneous
systematic variance,
and minimize error
variance.

c. 2000 Del Siegle

In other words control variance.
Therefore the researcher
attempts to:
• maximize the variance of the variable(s) of the research
hypothesis (i.e., maximize the difference in the
dependent variable [outcome] caused by maximizing
the differences in the independent variable [treatment]).
• control the variance of extraneous or "unwanted"
variables that may have an effect on the experimental
outcomes, but which he/she is not interested (limit
factors other than the treatment (IV) that could be
causing differences in the outcome (DV) .
• minimize the error or random variance (i.e., avoid

c. 2000 Del Siegle

unreliable measurement instruments which have high
errors of measurement ).
Maximization of Experimental
Variance
• experimental variance
– the variance due to the
manipulated (i.e., treatment)
or attribute (i.e., gender)
variables (IV)
research precept:
– design, plan and conduct
research so that experimental
conditions are as different as
possible on the independent

c. 2000 Del Siegle

variable.
Control of Extraneous
Variables (EV)
• eliminate the variable (i.e., if you are worried
about gender, only include one gender in the study).
• randomization (i.e., if you randomly assign subjects to
groups, the extraneous variable should be equally
distributed among the groups)
• build it into the design – make it a moderator variable (i.e.,
if you are worried about gender, build it into the analysis [2-
way ANOVA]—we’ll learn about this later)
• match subjects (i.e., match the characteristics of subjects
and put one of each matched pair in each group)
• statistically equate groups (i.e., use ANCOVA [Analysis of
Covariance] to analyze the data with the extraneous

c. 2000 Del Siegle

variable used as a covariate—we’ll learn about this later)
Minimizing Error Variance
has Two Principle Aspects:
• reduction of errors of
measurement through
controlled conditions (i.e.,
standardize testing
procedures)
• increase in the reliability of
measures (i.e., revise test
instruments or find more
reliable ones)

c. 2000 Del Siegle

Experimental
Designs
Should be
Developed to
Ensure
Internal and
External
Validity of the

c. 2000 Del Siegle

Study
 Internal
Validity:
• Are the results of the
study (DV) caused by
the factors included in
the study (IV) or are
they caused by other
factors (EV) which
were not part of the
study?

c. 2000 Del Siegle

There are 16 common threats to internal validity.
Threats to Internal Validity

Subject
Characteristics
(Selection Bias/Differential Selection) -- The
groups may have been different from the start. If
you were testing instructional strategies to
improve reading and one group enjoyed reading
more than the other group, they may improve

c. 2000 Del Siegle

more in their reading because they enjoy it, rather
than the instructional strategy you used.
Threats to Internal Validity

Loss of Subjects
(Mortality) -- All of the high or low scoring
subject may have dropped out or were
missing from one of the groups. If we
collected posttest data on a day when the
honor society was on field trip at the
treatment school, the mean for the
treatment group would probably be much
lower than it really should have been.

c. 2000 Del Siegle

Threats to Internal Validity

Location
Perhaps one group was at a
disadvantage because of their
location. The city may have
been demolishing a building
next to one of the schools in
our study and there are
constant distractions which
interfere with our treatment.

c. 2000 Del Siegle

 Threats to Internal Validity

The testing instruments may not be

scores similarly. Perhaps the person
grading the posttest is fatigued and
pays less attention to the last set of
papers reviewed. It may be that those
papers are from one of our groups and
will received different scores than the
earlier group's papers

Instrumentation
Instrument Decay

c. 2000 Del Siegle

 Threats to Internal Validity

The subjects of one group may react differently to the

data collector than the other group. A male
interviewing males and females about their attitudes
toward a type of math instruction may not receive the
same responses from females as a female
interviewing females would.

Data Collector
Characteristics

c. 2000 Del Siegle

Threats to Internal Validity

The person collecting data my favors one group,

or some characteristic some subject possess,
over another. A principal who favors strict
classroom management may rate students'
attention under different teaching conditions with
a bias toward one of the teaching conditions.

Data
Collector
Bias

c. 2000 Del Siegle

 Threats to Internal Validity

Testing
The act of taking a pretest or posttest may influence the
results of the experiment. Suppose we were conducting
a unit to increase student sensitivity to prejudice. As a
pretest we have the control and treatment groups watch
Shindler's List and write a reaction essay. The pretest
may have actually increased both groups' sensitivity
and we find that our treatment groups didn't score any
higher on a posttest given later than the control group
did. If we hadn't given the pretest, we might have seen
differences in the groups at the end of the study.

c. 2000 Del Siegle

 Threats to Internal Validity

History
Something may happen at one site during our study that
influences the results. Perhaps a classmate dies in a car
accident at the control site for a study teaching children bike
safety. The control group may actually demonstrate more

c. 2000 Del Siegle

concern about bike safety than the treatment group.
Threats to Internal Validity

There may be natural

changes in the subjects
that can account for the
changes found in a study.
A critical thinking unit may
appear more effective if it
taught during a time when
children are developing
abstract reasoning.

Maturation

c. 2000 Del Siegle

 Threats to Internal Validity

Hawthorne Effect
The subjects may respond differently just because they are
being studied. The name comes from a classic study in
which researchers were studying the effect of lighting on
worker productivity. As the intensity of the factory lights
increased, so did the worker productivity. One researcher
suggested that they reverse the treatment and lower the
lights. The productivity of the workers continued to increase.
It appears that being observed by the researchers was
increasing productivity, not the intensity of the lights.

c. 2000 Del Siegle

 Threats to Internal Validity
One group may view that it is in competition with the other
group and may work harder than they would under normal
circumstances. This generally is applied to the control group
"taking on" the treatment group. The terms refers to the
classic story of John Henry laying railroad track.

John
Henry
Effect

c. 2000 Del Siegle

 Threats to Internal Validity

The control group may become discouraged

because it is not receiving the special attention
that is given to the treatment group. They may
perform lower than usual because of this.

Resentful
Demoralization of
the Control Group

c. 2000 Del Siegle

 Threats to Internal Validity

Regression
(Statistical Regression) -- A class that scores
particularly low can be expected to score slightly
higher just by chance. Likewise, a class that
scores particularly high, will have a tendency to
score slightly lower by chance. The change in
these scores may have nothing to do with the
treatment.

c. 2000 Del Siegle

 Threats to Internal Validity

The treatment may not be implemented as

intended. A study where teachers are
asked to use student modeling techniques
may not show positive results, not because
modeling techniques don't work, but
because the teacher didn't implement them
or didn't implement them as they were
designed.

Implementation

c. 2000 Del Siegle

 Threats to Internal Validity

Someone may feel sorry for the control

group because they are not receiving much
attention and give them special treatment.
For example, a researcher could be studying
the effect of laptop computers on students'
attitudes toward math. The teacher feels
sorry for the class that doesn't have
computers and sponsors a popcorn party
during math class. The control group begins
to develop a more positive attitude about
mathematics.
Compensatory

c. 2000 Del Siegle

Equalization of Treatment
 Threats to Internal Validity

Experimental Treatment
Diffusion
Sometimes the control group
actually implements the
treatment. If two different
techniques are being tested in
two different third grades in the
same building, the teachers
may share what they are
doing. Unconsciously, the
control may use of the

c. 2000 Del Siegle

techniques she or he learned
from the treatment teacher.
Once the researchers are confident that
the outcome (dependent variable) of the
experiment they are designing is the
result of their treatment
(independent variable)
[internal validity],
they determine for which
people or situations
the results of
their study apply
[external validity].

c. 2000 Del Siegle

 External
Validity:
• Are the results of the study generalizable to
other populations and settings?

c. 2000 Del Siegle

External validity comes in two forms: population and ecological.
 Threats to External Validity
(Population)

Population Validity is the extent to which the results of a study

can be generalized from the specific sample that was studied to a
larger group of subjects. It involves...
1. ...the extent to which one can generalize from the study
sample to a defined population--
If the sample is drawn from an accessible population, rather
than the target population, generalizing the research results
from the accessible population to the target population is
risky.
2. ...the extent to which personological variables interact with
treatment effects--

c. 2000 Del Siegle

If the study is an experiment, it may be possible that different
results might be found with students at different grades (a
personological variable).
Threats to External Validity
(Ecological)

Ecological Validity is the

extent to which the results of an experiment can be
generalized from the set of environmental conditions
created by the researcher to other environmental
conditions (settings and conditions).

There are 10 common

threats to external

c. 2000 Del Siegle

validity.
Threats to External Validity
(Ecological) Explicit
description of
the
experimental
treatment
(not sufficiently described for others to
replicate) If the researcher fails to
adequately describe how he or she
conducted a study, it is difficult to

c. 2000 Del Siegle

determine whether the results are
applicable to other settings.
Threats to External Validity
(Ecological)

Multiple-treatment
interference
(catalyst effect)
If a researcher were to apply
several treatments, it is difficult to
determine how well each of the
treatments would work individually.
It might be that only the combination
of the treatments is effective.

c. 2000 Del Siegle

Threats to External Validity
(Ecological)

Hawthorne effect
(attention causes differences)
Subjects perform differently because they
know they are being studied. "...External
validity of the experiment is jeopardized
because the findings might not generalize
to a situation in which researchers or
others who were involved in the research
are not present" (Gall, Borg, & Gall, 1996,
p. 475)

c. 2000 Del Siegle

 Threats to External Validity
(Ecological)

Novelty and
disruption effect
(anything different makes a difference)
A treatment may work because it is novel and the subjects
respond to the uniqueness, rather than the actual
treatment. The opposite may also occur, the treatment
may not work because it is unique, but given time for the
subjects to adjust to it, it might have worked.

c. 2000 Del Siegle

Threats to External Validity
(Ecological)

Experimenter effect
(it only works with this experimenter)
The treatment might have worked
because of the person implementing it.
Given a different person, the treatment
might not work at all.

c. 2000 Del Siegle

Threats to External Validity
(Ecological)

Pretest sensitization
(pretest sets the stage)
A treatment might only work if a
pretest is given. Because they
have taken a pretest, the subjects
may be more sensitive to the
treatment. Had they not taken a
pretest, the treatment would not
have worked.

c. 2000 Del Siegle

Threats to External Validity
(Ecological)

Posttest sensitization
(posttest helps treatment "fall into place")
The posttest can become a learning
experience. "For example, the posttest might
cause certain ideas presented during the
treatment to 'fall into place' " (p. 477). If the
subjects had not taken a posttest, the
treatment would not have worked.

c. 2000 Del Siegle

Threats to External Validity
(Ecological)

Interaction of
history and
treatment effect
(...to everything there is a time...)
Not only should researchers be cautious about
generalizing to other population, caution should be
taken to generalize to a different time period. As
time passes, the conditions under which treatments

c. 2000 Del Siegle

work change.
Threats to External Validity
(Ecological)

Measurement of
the dependent
variable
(maybe only works with M/C tests)
A treatment may only be evident with
certain types of measurements. A teaching
method may produce superior results
when its effectiveness is tested with an
essay test, but show no differences when

c. 2000 Del Siegle

the effectiveness is measured with a
multiple choice test.
Threats to External Validity
(Ecological)

Interaction of time
of measurement
and treatment
effect
(it takes a while for the treatment to kick in)
It may be that the treatment effect does not occur until several
weeks after the end of the treatment. In this situation, a
posttest at the end of the treatment would show no impact,

c. 2000 Del Siegle

but a posttest a month later might show an impact.
 First, and foremost, an experiment
must have internal validity. If the
researchers cannot be certain that the
results of the experiment are
dependent on the treatment, it does
not matter to which people or
situations they wish to generalize
(apply) their findings. The importance
of external validity is reliant on
having internal validity in much the same way that the validity
of a measurement instrument is reliant on the instrument
being reliable.
However, the more tightly experimenters design their study,
the more they limit the populations and settings to whom they

c. 2000 Del Siegle

can generalize their findings.
c. 2000 Del Siegle
Suppose a researcher wants to
study the effect of a reading
program on reading achievement.

She might implement the reading program with a

group of students at the beginning of the school year
and measure their achievement at the end of
the year.

XO
This simple design is known as a one-shot case
study design.

c. 2000 Del Siegle

Unfortunately, the students’ end of year reading scores
could be influenced by other instruction in school, the
students’ maturation, or the treatment.

We also do not know whether the students’ reading skills

actually changed from the start to end of the school year.

We could improve on this design by giving a pretest at the

start of the study.

OXO
This is known as a one-group pretest-posttest design.

c. 2000 Del Siegle

Unfortunately, the students’ end of year reading scores still
could be influenced by other instruction in school, the
students’ maturation, or the treatment.

Our researcher may wish to have a comparison group.

OXO
O O

c. 2000 Del Siegle

This is a static-group pretest-posttest design.
If our researcher believes that the pretest has an impact
on the results of the study, she might not include it.

OXO
O O

c. 2000 Del Siegle

This is a static-group comparison design.
Because our researcher did not pretest, she might wish
to randomly assign subjects to treatment and control
group.

Random assignment of subject to groups should spread

the variety of extraneous characteristics that subjects
possess equally across both groups.

ROXO
RO O

c. 2000 Del Siegle

This is a randomized posttest-only, control group design.
 Of course, our researcher could also include a pretest
with her random assignment.

ROXO
RO O

c. 2000 Del Siegle

This is a randomized pretest-posttest control group design.
Occasionally researchers combine the randomized
pretest-posttest control group design with the
randomized posttest-only, control group design.

ROXO
RO O
R XO
R O

c. 2000 Del Siegle

This is a randomized Solomon four-group design.
With the randomized Solomon four-group design, all
groups are randomly assigned and given the posttest.

Two of the groups are given pretests.

ROXO One of the pretest

groups is assigned to

RO O
treatment and one of
the non-pretest groups

R XO
is assigned to
treatment.

R O

c. 2000 Del Siegle

Each of the designs described in this section has
advantages and disadvantages that influence the
studies internal and external validity.

c. 2000 Del Siegle

This presentation was prepared by Del Siegle.
Some of the material is from an earlier presentation by Scott Brown.