4/18/2011

CONDUCTING EXPERIMENTS

Dr. Yan Liu

Department of Biomedical, Industrial and Human Factors Engineering
Wright State University

Selecting Research Participants

Why Sampling Participants
 In practice it is usually impossible for us to study the entire population due to the
constraints of our resources

Sampling Methods
 The method used to select participants has implications for generalizing the
research results
 Probability sampling

Each member of the population has a specifiable probability of being chosen

Must be used when it is important to accurately describe the population

Usually used in survey studies
 Nonprobability sampling

The probability that any particular member of the population is chosen is unknown

Not as sophisticated as probability sampling but quite common and useful in many
circumstances, especially in experimental methods

1
 4/18/2011

Nonprobability Sampling

Haphazard Sampling (“Convenience” Sampling)
 Can be described as “take-them-where-you-find-them” method of obtaining
participants

Select a sample of students at Wright State in any way that is convenient (e.g. asking
your classmates or friends, visiting computer labs or libraries, etc.)
 Such a procedure is likely to introduce biases into the sample so that the sample
may not be accurate representation of the population of interest

Students studying in computer labs or libraries may be more devoted to learning than
those who do not
 The experiment results may not generalize to the intended population but instead
may describe only the biased sample obtained

Nonprobability Sampling (Cont’d)

Purposive Sampling
 The purpose is to obtain a sample of people who meet some predetermined
criteria

You want to recruit participants who are under 30 years old and college students,
then you can visit some fraternity and sorority houses at Wright State
 A good way to limit your sample to a certain group of people

Quota Sampling
 Used to choose a sample that reflects the numerical composition of various
subgroups in the population

You want to have a sample of students that includes 19% freshmen, 23%
sophomores, 26% juniors, 22% seniors, and 10% graduate students, then a quota
sampling would make sure you have these percentages
 Haphazard sampling technique is used to collect data
 The problem remains that no restrictions are placed on how individuals in the
various subgroups are chosen

2
 4/18/2011

Setting the Stage

Informed Consent
 Provide the participants with the informed consent information needed for
the study and explain to them the purpose of the study

Construct an Operational Definition for the Independent Variable
 Translate the variable into a set of operations to manipulate it
 To study the effect of crowding on performance, the variable “crowding”
may be operationally defined as the number of people in a room, then,
participants are put either to a room with few people or to a room with a large
a number of people

Types of Manipulations

Straightforward Manipulations
 Manipulate a variable with relative simplicity with instructions and stimulus
presentations

Stimuli may be presented verbally, in written form, via videotape, or with a
computer
 Applied in most experiments

Staged Manipulations
 Create some psychological state in the participants

Frustration, a temporary lowering of self-esteem, etc.
 Simulate some situation that occurs in the real world

In a study on how disruptions affect cognitive performance, participants in one
condition spent 10 min. proofreading a manuscript, whereas participants in the other
condition performed the same proofreading task but were interrupted by the
experimenter from time to time and asked to go to another room to perform other
tasks
 A confederate is frequently employed

Can be used in both field and laboratory experiments 6

3
 4/18/2011

Strength of Manipulation

How to choose the levels of its independent variable?

Strong Manipulation
 Make the manipulation as strong as possible
 A strong manipulation maximizes the difference between experimental
conditions and thus increases the chance that the independent variable will have
a statistically significant effect on the dependent variable
 Particularly important at the early stage of research to examine whether the
independent and dependent variables are related

Issues with Strong Manipulation
 Strong manipulation may involve a situation that rarely occurs in the real world
 Ethics consideration

A manipulation should be as strong as possible within the bounds of ethics

Suppose you think there is a positive linear relationship between attitude similarity
and liking. So you plan to design an experiment with the attitude similarity as the
independent variable and liking as the dependent variable. A hypothesized
relationship between the two variables is shown in the following figure.

To achieve the strongest manipulation, the

participants in one group would encounter a
confederate of level 1 similarity and those in
the other group would encounter a
confederate of level 10 similarity. This
would result in the greatest difference in the
amount of liking – a 9-point difference

4
 4/18/2011

Cost of Manipulation

Issues with Cost of Manipulation
 Researchers who have limited monetary resources may not be able to afford
expensive equipment, salaries for confederates, or payments to participants in
long-term experiments
 A manipulation in which participants must be run individually requires more of
the researcher’s time than a manipulation that allows running many individuals
in a single setting
 A manipulation that uses straightforward presentation of written or verbal
material is less costly than a complex, staged, experimental manipulation

Manipulation Check of Independent Variable

What is Manipulation Check
 An attempt to directly measure whether the independent variable manipulation
has the intended effect on the participants
 Provides evidence for the construct validity of the manipulation

In a study on how anxiety affects memory, the researcher tries to produce two
levels of anxiety. A manipulation check will tell the researcher whether the
participants in the group labeled “high-anxiety” are indeed more anxious than
those in the group labeled “low-anxiety”

Advantages
 Particularly useful in a pilot study
 Advantageous if the results indicate that no relationship exists between the
independent and dependent variables

Identify whether the nonsignificant results are due to a problem in manipulating the
independent variable

10

5
 4/18/2011

Measuring the Dependent Variable

Self-Report Measures
 Measure attitudes, personal judgment, personality characteristics, emotional
states, and many other aspects of human thoughts
 Rating scales with descriptive anchors (end-points) are most commonly used

Seven-point Likert scale (1 being the least and 7 being the most)

Behavioral Measures
 Direct observations /recording of behaviors
 Record whether a possible behavior occurs

Whether or not an individual responds to a request for help, makes an error on a test,
etc.
 Decision on whether to record the rate of a behavior, response time, or duration

Depends on which is the most theoretically relevant for the study

Depends on which measure logically follows from the independent variable
manipulation
 Both self-report and behavioral measures can be appropriate for many variables
(recommended in HF experiments)
11

Measuring the Dependent Variable (Cont’d)

Physiological Measures
 Recordings of responses of the body
 Galvanic skin response (GSR)

A measure of general emotional arousal and anxiety

Measures the electrical conductance of the skin, which changes when sweating
occurs
 Electromyogram (EMG)

A measure of tension or stress

Measures muscle tension
 Electroencephalogram (EEG)

A measure of electrical activity of brain cells

Record general brain arousal as a response to different situations, activity in different
parts of the brain as learning occurs, etc.
 Magnetic resonance imaging (MRI)

Provides an image of the brain structure of an individual

Allows comparing the brain structure of individuals with a particular condition (e.g.
cognitive impairment) with the brain structure of those without the condition
12
 Body temperature, heart rate, etc.

6
 4/18/2011

Sensitivity of the Measure of

Dependent Variable

Measure of the dependent variable should be sensitive enough to
detect differences between groups
 To measure the satisfaction of a job, a simple answer with “Yes” or “No”
response alternative is less sensitive to a 7-point Likert scale which measures the
extent of satisfaction

Implication on Selection of Tasks in Experiments
 Ceiling effect

The independent variable appears to have no effect on the dependent measure only
because participants quickly reach the maximum performance level
 Floor effect

The task is so difficult that hardly anyone can perform well

13

Multiple Dependent Measures

Advantages
 A variable can have multiple operational definitions

If the independent variable has the same effect on several measures of the same
variable, our confidence in the results would be increased

It is useful to know whether the same independent variable affects some measures
but not others
 Study the effects of an independent variable on several behaviors and attitudes

Order of Multiple Measures
 If the problem of order effects appears serious, the order of presenting the
measures can be counterbalanced (the counterbalancing techniques used in
within-subject design)
 Otherwise, present the most important measures first and the less important ones
later

Single vs. Separate Experiments
 Making multiple measures in a single experiment is valuable when it is feasible
to do so 14

7
 4/18/2011

Other Considerations of Dependent Variable

Cost of Measures
 Paper-and-pencil self-report measures are inexpensive
 Measures that require trained observers or elaborate equipment (e.g. video
camera, eye tracker) become quite costly
 Physiological recording devices are expensive
 Choose appropriate measures within the monetary constraints

Ethics
 Be careful about potential invasion of privacy and address confidentiality issues

15

Participant Expectations

Demand Characteristics
 Features of study that might inform participants of the hypothesis of the study
 When participants form expectations about the hypotheses of the study, they
may behave in a way to try to support the hypotheses

Control for Demand Characteristics
 Deception

Make participants think that the experiment is doing one thing when actually it is
studying something else
 Disguise the dependent measure by using an unobtrusive measure or by placing
the measure among a set of unrelated filler items on a questionnaire
 Ask participants about their perceptions of the hypothesis of the research

If some participants do figure out the hypotheses of the study, their data should be
analyzed separately

16

8
 4/18/2011

Experimenter Expectations

Experimenter Bias (Expectancy Effects)
 Experimenter may unintentionally treat participants differently in the various
conditions of the study

Certain words may be emphasized when reading instructions to one group but not to
the other
 Experimenter may record the behaviors of participants and interpret them
differently for participants in different conditions
 Experimenter expectancies can be communicated to participants and thus
influence their behaviors

17

Experimenter Expectations (Cont’d)

Solutions to Expectancy Problem
 Experimenters should be well trained and practice behaving consistently with all
participants
 Run all conditions simultaneously so that the experimenter’s behavior is the
same for all participants

Only feasible when the experimenter’s instructions to participants are the same for
everyone or the study can be carried out with the use of printed materials
 Automate procedures by manipulating independent variables and measuring
dependent variables with computers
 Use experimenters who are unaware of the hypotheses being investigated

18

9
 4/18/2011

Pilot Study

Usages of Pilot Study

 Assess the viability of an experimental plan before embarking on the real study

Reveal whether participants understand the instructions, whether the experimental
setting seems plausible, whether there is any confusing question, etc.

Question participants in detail about their experience

Use “Think aloud” protocol in which the participants are instructed to verbalize their
thoughts about everything that is happening during the study
 Allows the experimenters to become comfortable with their roles and to
standardize their procedures
 Choosing sample size requires an estimate of variance which may come from a
pilot study

19

Debriefing

Importance of Debriefing
 Provide an opportunity for the researcher to learn more about what participants
were thinking during the experiment, which can be useful in interpreting the
results and planning future studies
 Researcher may ask the participants to refrain from discussing the study with
others if more people will be participating

20

10