EXPERIMENTAL RESEARCH

A very common way of research design A good design to establish cause and effect relationships among variables It is the only type of research that
directly

attempts to influence a particular variable tests hypotheses about cause-and-effect relationships

Major Characteristics of Experimental Research

The researcher looks at the effect(s) of at least one independent variable (experimental or treatment variable) on one or more dependent variables (criterion or outcome variable)

The researcher manipulates the independent variable, which is not the case in other research designs
Aim of the researcher is to see whether the treatment/manipulation has made a difference.

Some independent variables:

Methods of instruction Types of assignment Learning materials Types of questions asked by teachers

Some dependent variables:

Achievement Interest Motivation Attitudes

Now, write possible research questions and hypotheses considering these variables.

Comparison of Groups

Experimental research involves two groups of subjects

Experimental group Control / comparison group

Experimental group receives the treatment; control group receives no treatment / comparison group receives a different treatment Comparison group rather than control group in educational research

Manipulation of the Independent Variable

The researcher actively manipulates the independent variable: s/he decides the nature of the treatment, to whom and to what extent it will be applied, and when, where and how it will be done Variables that can be manipulated: teaching method, learning activities, materials, etc Variables that cannot be manipulated: gender, ethnicity, age, etc

Ways of establishing independent variables

A) one form of the variable versus another

E.g. comparing the inquiry method vs the lecture method in teaching listening

B) presence versus absence of a particular form

E.g. comparing the use of videos vs no videos in teaching speaking

C) varying degrees of the same form

Comparing the effects of different amount of homework on students motivation

Control of Variables

In experimental research, the researcher has the chance to control the variables for internal validity purposes. It is important for the researcher to be sure that NOT OTHER variables have affected the result. E.g. studying students attitudes towards English: if you dont make sure that the groups are equivalent in ability, then ability might be a reason for the cause. There are some ways to control variables and minimize threats to internal validity.

a) Randomization

Random assignment: assigning the subjects to groups Points to consider

Takes

place before the experiment starts It is a process of assigning individuals to groups It allows the researcher to assign two equivalent groups at the beginning of the study

Equivalence is not possible if there are not enough number of subjects in a group (no rules for the number, but generally not fewer than 40 in a group)

b) Holding certain variables constant

To eliminate its possible effects, the researcher can remove a variable from the study. E.g. if gender is considered to influence the study, you can restrict the subjects to females or males only.

In that case, generalization is reduced!

c) Building the variable into the design

Opposite of the previous idea E.g. The researcher includes both females and males (as distinct groups) and analyze the effects of both gender and method on outcomes

d) Matching

Pairs of subjects can be matched on certain variables of interest. E.g. if age is considered to be an effect, the researcher can assign one member of each age group to each of the comparison groups

e) Using subjects as their own control

Using the same subjects for all the treatments E.g.

teaching

the same students by an inquiry method and later by a lecture method Assessing behavior during a period of time before and after a treatment to see whether changes in behavior occur

f) Using analysis of covariance

Equating groups statistically on the basis of a pretest or other variables. The posttest scores are adjusted accordingly

Internal Validity

Possible threats to internal validity A) Selection Bias (subject characteristics:

Might be differences in groups (age, gender, intelligence)

B) Loss of Subjects (mortality)

Losing some of the subjects (e.g. some subjects may not return the questionnaires)

C) Location

Place in which data are collected or an implementation is conducted may affect the study

D) Instrumentation
instrument decay (e.g. essay scoring causes fatigue and can be score differently) data collector characteristics (age, gender, experience of data collector) data collector bias (e.g. if the collector knows the hypotheses, s/he might unconsciously distract the data)

E) Testing

Subjects can remember the test if the same is used in pre and post test at a short interval

F) History
personal

experiences during the course of the

study

G) Maturation
E.g.

young learners will change due to maturation

H) Attitude of Subjects
The

way subjects view the study (e.g. experimental group might perform better because they know they are having a special treatment)

I) Regression
(e.g.

if you include all good students in one group, they will improve no matter what you do)

J) Implementation
The

researcher can give the experimental group an advantage

The One-Shot Case Study:

Group Designs in Experimental Research: WEAK Experimental Designs

A single group is exposed to a treatment and a dependent variable is observed to assess the effect of the treatment X O treatment observation

Why weak? Absence of control: the researcher does not know whether the result is due to the treatment Not possible to compare so the researcher doesnt know whether the treatment had any effect at all To remedy this, a comparison can be made with another group who had the same course content

The One-Group Pretest-Posttest Design:

A single group is observed before and after treatment O X O Pretest treatment posttest

Better than one-shot case study (we know whether change occurred), but it is still weak. Why? Threats to internal validity: history, maturation, instrument decay, data collector characteristics, data collector bias, testing, regression, attitude of subjects and implementation (The researcher does not know if the difference is due to the treatment or any of these!) To remedy this, a comparison group which does not have the treatment can be added.

The Static-Group Comparison Design

Two

already existing groups (nonequivalent) are used X O .. O

Although it provides better results, it might still have problems. Why? Some control on testing, regression threats; a little control on history and maturation; problems of control on mortality and location

The Static-Group Pretest-Posttest Design

Similar to static group comparison. The only difference is to give a pretest to both groups O X O O O

While analyzing, the pretest score is subtracted from the posttest score, so change in each student is analyzed. Some possible threats? Subject characteristics (group scoring higher on the pretest is likely to improve more) Testing threat (pre and post test)

True Experimental Designs

The Randomized Posttest-Only Control Group Design

Involves two groups which are formed by random assignment: one receives the treatment; the other does not Treatment group R (random) X O Control group R (random) C O

Control of some threats is excellent: subject characteristics, maturation, regression No testing threat because subjects are not tested twice Possible threats: Mortality (if more subjects drop out in one, then results can be affected; the number of the subjects who dropped out should be reported) Implementation, data collector bias, location and history threats may also exist

The Randomized Pretest-Posttest Control Group

The

difference is in using both pre and post test for both groups Treatment group R O X O Control group R O X O

This design helps the researcher see if the random assignment actually succeeded (pretest). This is particularly important
if

the number in each group is small

The Randomized Solomon Four-Group Design

Used to eliminate the possible effect of a pretest Treatment group R O X O Control group R O C O -----------------------------------------------Treatment group R X O Control group R C O

The first group is like pretest-posttest control group; the second is posttest only control group Best control of threats Weakness: requires a large sample (subjects are assigned to four groups); requires a lot of energy and effort

QUASI-EXPERIMENTAL DESIGN

It is like experimental design but it doesnt use random assignment. It uses other techniques to control threats
There are many techniques used in this design but the most commonly used one in social research is nonequivalent groups design

Nonequivalent groups design It is like pretest-posttest randomized experiment but does not use random assignment. Instead similar groups are used as the treatment and the control groups E.g. you can pick up two comparable classrooms or schools.

CONCLUSION

In any type of experimental design, we create two groups that are equivalent to each other. One group gets the treatment; the other does not. In all other respects the groups are treated the same. If we observe differences in the outcome between these groups, then we believe that the difference is due to the treatment. However, in all these designs, we are setting up an artificial situation, so we have to consider all the situation. If the situation is right, then an experiment can be a very strong design to use.

EXERCISES

Exercise 1. A researcher uses morning and afternoon history classes to test a new instructional strategy. In the morning class the students are given a pretest to assess their knowledge of WWI and then they are given the opportunity to learn about this topic using a computerized program. In the afternoon class the students are also given a pretest; they learn about WWI through discussion. At the end of six weeks, a final exam is given to both classes. 1. a) How is the research designed?

Quasi-experimental (non-equivalent pre-test/post-test)

Gr. A (NR) Gr. B (NR) O O X1 X2 O O

A researcher uses morning and afternoon history classes to test a new instructional strategy. In the morning class the students are given a pretest to assess their knowledge of WWI and then they are given the opportunity to learn about this topic using a computerized program. In the afternoon class the students are also given a pretest; they learn about WWI through discussion. At the end of six weeks, a final exam is given to both classes.

1. b) What are possible threats to internal validity? Selection bias? Location? Instrumentation (data collector bias)?

A researcher uses morning and afternoon history classes to test a new instructional strategy. In the morning class the students are given a pretest to assess their knowledge of WWI and then they are given the opportunity to learn about this topic using a program. In the afternoon class the studecomputerized nts are also given a pretest; they learn about WWI through discussion. At the end of six weeks, a final exam is given to both classes. 1. c) How can you analyze the data? t-test (comparison of the means in pre- and post-tests for both groups)

Exercise 2. A social studies teacher is interested in determining the effects of a new textbook series, which has a multicultural thematic approach. The teacher will teach a unit on colonization in America. After the unit, a multicultural attitude scale will be administered to students. 2. a) How is the research designed? one-shot case study

A social studies teacher is interested in determining the effects of a new textbook series, which has a multicultural thematic approach. The teacher will teach a unit on colonization in America. After the unit, a multicultural attitude scale will be administered to students. 2. b) What are possible threats to internal validity? Almost no control

A social studies teacher is interested in determining the effects of a new textbook series, which has a multicultural thematic approach. The teacher will teach a unit on colonization in America. After the unit, a multicultural attitude scale will be administered to students. 2. c) How can you analyze the data? Descriptive statistics

Exercise 3 A school administration wants to field-test a Methodology curriculum in the ELT course that is offered in the summer. Two classes are to be offered during the summer session. Students in both classes will be given a pretest to measure their knowledge of Methodology. One class will study Methodology using traditional textbook materials with a mostly lecture-style of delivery. Another class, an experimental class, will study Methodology using films and contemporary materials along with presentations by guest speakers. The final exam for the course, a parallel form of the pretest, will be given to both groups.

3. a) How is the research designed? Quasi-experimental (pre-test/post-test nonequivalent comparison groups)

A school administration wants to field-test a Methodology curriculum in the ELT course that is offered in the summer. Two classes are to be offered during the summer session. Students in both classes will be given a pretest to measure their knowledge of Methodology. One class will study Methodology using traditional textbook materials with a mostly lecture-style of delivery. Another class, an experimental class, will study Methodology using films and contemporary materials along with presentations by guest speakers. The final exam for the course, a parallel form of the pretest, will be given to both groups.

3. b) What are possible threats to internal validity? Regression? Selection bias?

A school administration wants to field-test a Methodology curriculum in the ELT course that is offered in the summer. Two classes are to be offered during the summer session. Students in both classes will be given a pretest to measure their knowledge of Methodology. One class will study Methodology using traditional textbook materials with a mostly lecturestyle of delivery. Another class, an experimental class, will study Methodology using films and contemporary materials along with presentations by guest speakers. The final exam for the course, a parallel form of the pretest, will be given to both groups. 3. c) How can you analyze the data? t-test between and within groups

Exercise 4 With the current emphasis on parental involvement in childrens academic endeavors, a researcher is interested in assessing the confidence level of parents who will read orally to children in schools. He also wants to know if there is a difference in the confidence level of parents of different education levels. From a list of volunteers who are interested in reading orally to students, parents will be randomly selected and assigned to one of two conditionstraining or no training. Following training of parents in the experimental condition, all parent volunteers will complete a confidence scale. Describe this study using a true experimental posttest-only control group design.

Random assignment for two groups: give a number to each and draw out a number for Group A and then Group B until you finish all the numbers. Indep variable: training parents Dep variable: parents confidence level Group A will get training whereas Group B will not. Parents in both groups will read orally to children and then they will be given a confidence scale. Ttest for the mean scores obtained from this scale for each group will be found out to see the effect of training on confidence level.

Exercise 5 The dean of a college of education is interested in determining if the education majors who take an optional course in working with students with disabilities are using the strategies learned in the course. He decides to study this issue. Fifty education majors will be randomly assigned to take the course when they enter as freshman. The control group of fifty education majors will be assigned a multicultural education course. The junior field experience will occur prior to the courses, thus allowing monitoring for strategies prior to taking the course. Students will again be monitored for use of strategies during student teaching. In both the pre-course and post-course field experiences, students will each teach ten lessons in their classrooms. These lessons will be videotaped. A researcher will view the videotapes and observe the lessons to determine which strategies are used. Describe this study using a true experimental, pretestposttest control group design.

For the following exercises, work in pairs and describe the research design answering following questions: 1. How will the subjects be assigned to groups? 2. Identify relevant variables 3. Describe the treatment design 4. Identify the way the data will be analyzed.

Please read a few article which used experimental research method!!