Experimental Design - Thematic Glossary Baseline Value of a variable before the application of an experimental treatment.

We administer a dose of alcohol to a group of subjects and measure their performance on a reaction time test. They all have slow reaction times. Does this mean that the alcohol has slowed down the reaction time? Theres no way of telling because we didnt measure their reaction times before they took the alcohol. We could have just chosen already slow subjects. A measure of reaction time before administering the alcohol would be the baseline. Between-Subjects Design Measurement of an effect of an experimental treatment by comparing two or more treatment groups. In the alcohol example, a between-subjects design would involve the comparison of two groups, one of which was given alcohol (the Experimental Group), the other which was not (the Control Group). The idea is that the two groups are comparable in all respects apart from the alcohol consumed. Any difference in performance between the groups would therefore be due to the experimental manipulations (i.e. the alcohol) and if there were no experimental manipulation, then the two groups would presumably, remain exactly the same. (See also Within-subjects Design). Bias The effect on a measurement of any variable other than the one of interest. Weight is a crude measure of height (on average, taller people weigh more), but it is biased because weight is not only affected by height but also by girth, bone mass etc. Every bias variable should be located and controlled. Case Study Detailed investigation of a single individual. Case studies are often used in medicine, psychiatry and neurology to illustrate the factors, development and consequences of a disease or injury. Case studies can provide useful insights into normal brain functioning due to the damage sustained to a (what was assumed to be) once normal brain. There is the problem of generalisation given that its only the study of one particular individual. In general, the more a condition results from brain damage, the more useful case studies are. Ceiling Effect Measurement failure where the majority of subjects score 100% correct because the task is too easy. (see also Floor Effect). If you used a task to measure reaction time response such as blink when you hear a noise, there would be very little difference in performance between the group that received the alcohol and the group that did not because the task is just too easy. Confounding A relationship between 2 (or more) variables which prevents their effects from being evaluated separately. Confounding can affect either Independent or Dependent variables and causes problems in interpreting which variable caused the experimental effect to occur.

(The confounding variable can also be labelled the nuisance variable.) There are statistical ways of controlling the nuisance variable when you come to analyse your data, but it is always best to identify any confounding variables before you begin your experiment and try to control for them in the experimental design. The Independent Variable in the example is amount of alcohol consumed. A confounding variable could be time since last meal. It is known that the potency of alcohol is dependent upon whether a person has just eaten or has an empty stomach (the absorption rate of alcohol is altered because of the altered stomach volume). There could be no difference in performance between the experimental and the control group in your between-subjects experiment because the group that received the alcohol had just come back from lunch and their blood sugar levels were high. The Dependent Variable is the measure of performance, in this case a reaction time task. A possible confounding variable with reaction time could be time of day. If the performance of the experimental group was measured late in the evening when the individuals were tired and not concentrating (their reactions were slower) but the control group were tested early in the morning when they were alert (their reactions were faster), any differences observed between the groups could be due to time of day rather than alcohol consumption. Looking out for confounding variables is part of the art of good experimental design. Control Group Group who are not exposed to the experimental manipulation for comparison purposes with the experimental group. Counterbalancing Varying the order in which treatments are applied between subjects in a within-subjects design. Any technique used to vary systematically the order of conditions in an experiment to distribute the effects of time of testing (for example fatigue) so they are not confounded with conditions. Demand Characteristics Beliefs about expected performance created by an experimental procedure. Subjects naturally try to work out how they are expected to behave and any clues they pick up from the experimental task will probably affect what they do. Compliant subjects will try to produce what they think is the expected behaviour whereas others may seek to defy expectation. An example would be in the amount of space left in a questionnaire, if there was a large space for an answer, the subjects may expect to write a large amount. Demand characteristics are a bad thing and should be eliminated. Dependent Variable(s) The variable which is measured in an investigation. In the alcohol consumption example, performance is measured so performance is the dependent variable. Double-Blind Procedures

Method of organising drug evaluation trials so that neither the person administering the drug, nor the person testing its effectiveness is aware which subject has received the drug. Double-blind procedures try to dispel the influence of Experimenter Effects, Demand Characteristics and Placebo Effects. Under double-blind procedures the drug and placebo doses are coded (using numbers or letters), and the code is not broken until evaluation data has been analysed. Ecological Validity Extent to which behaviour produced in an investigation in natural for the organism which produces it. A laboratory behaviour would have low ecological validity if it was very unlike behaviour in the natural environment. In the alcohol example, if the behaviour you were interested in the natural environment was the ability to drive a car, then you would try to devise a task in the laboratory that was an ecologically valid measure of that ability. Experimental Conditions Treatment values (known as levels) of the Independent Variable in an experiment. Investigating how different amounts of alcohol affects performance would involve varying the amount of alcohol given to an individual. The different amounts of alcohol given to the individual correspond to the different experimental conditions. e.g. 1. No alcohol (Control group) 1. 1. 2. one optic measure of whisky two optic measures of whisky three optic measures of whisky

There are 3 experimental conditions in this experiment and one control condition although they are all usually referred to as conditions, hence this experimental design would have 4 conditions. Experimenter Effects When an experimenter inadvertently communicates expectations to the subject and this influences the outcome of the experiment (see Placebo, DoubleBlind Procedures, Demand Characteristics). Factorial Design Experimental design in which every combination of conditions occurs with equal frequency. Factorial designs are appropriate when experimental conditions might affect each other (an interaction). In pilot studies it might be noted that individuals drinking alcohol before a meal may have slower reaction times than those who take alcohol after a meal. We might look at the food variable. For example, one could look at ALCOHOL + NO FOOD, ALCOHOL + FOOD, NO ALCOHOL + NO FOOD and NO ALCOHOL + FOOD. This factorial combination of conditions allows us to see whether having food or not affects performance after alcohol as well as enabling us to be sure that being hungry on its own doesnt affect reaction times. There are usually an equal number of subjects randomly assigned to each condition.

Floor Effect Measurement failure where the majority of subjects score very poorly because the task is too difficult. (see also Ceiling Effect). If you used a task to measure reaction time response such as catch a fly with a pair of chopsticks, there would be very little difference in performance between the group that received the alcohol and the group that did not because the task is just too difficult. Hypothesis Implication of a theory derived for the purpose of testing the theory. An hypothesis is always a prediction and is therefore expressed in the future tense: The reaction times of a group of subjects that drinks one optic measure of whisky will be slower than a group of subjects that does not drink any whisky. Independent Variable(s) Variable which is manipulated (varied) in an experiment. The independent variable in the example is amount of alcohol to be consumed. Individual Differences Variations in the psychological variables between people. The use of Sampling, Randomisation techniques and statistical tests are examples of procedures which are necessitated by individual differences. The question Does the Experimental Group differ from the Control Group? must be translated into the question; Are any observed differences between the experimental and the control group greater than might be expected from random variation alone?. The null hypothesis states that any differences observed between the groups are actually due to random variation alone. Statistical tests therefore try to reject this idea - they try to reject the null hypothesis. Level of Measurement It is important to be clear on the type of Measurement Scales used for each variable, especially the Dependent Variable, since this will affect the choice of statistical test used during the analysis. There are four possible levels of measurement: nominal (also known as categorical), ordinal, interval and ratio. In statistics, ratio and interval data are treated identically. Measurement Scale A measurement scale allows the amount of a property to be given a number (quantified) by assigning it to a position on a scale. Most measurement scales have an origin (lowest possible score, often zero - see Floor Effect), a maximum (highest possible score - see Ceiling Effect), and units of measurement (e.g. inches, IQ points). It is important to chose an appropriate measurement scale so that any differences between the Experimental and the Control groups are observable. If for example, you measured reaction time in days rather than seconds or milliseconds, then any differences between the alcohol group and the no-alcohol group would not be observable. See also Level of Measurement. Order Effects Changes in the value of a dependent variable caused by the order of experimental conditions. Most order effects are produced by fatigue, practice and learning all resulting in worse performance in later trials which may not

necessarily be due to the experimental manipulations. Between-Subjects Designs, Counterbalancing and establishing a Baseline can all help to combat these effects. Pilot Study Preliminary trial of an experimental procedure intended to reveal deficiencies and provide information on the behaviour likely to occur in the main study. Pilot studies will pick up Ceiling and Floor Effects as well as things like, checking that questions and instructions are comprehensible as well as indicating the possible number of subjects to include in the full-blown study due to Individual Differences. Where pilot studies do not lead to modification of materials or procedure, data can be directly incorporated into the main study, as long as an appropriate Sampling strategy has been used to select subjects. Placebo Effect Changes in the state or performance of experimental subjects which result from the belief that they have taken a drug. Many subjects will show signs of drowsiness if they believe that they have taken a sedative, even if what they actually took was chalk. In drug-evaluation studies, the results are considerably complicated by this placebo effect which has nothing to do with the effectiveness of the drug. In order to eliminate these pseudo-effects, it is necessary to give a proportion of the subjects a placebo which looks (and tastes if necessary) exactly like the actual drug and is given in exactly the same way (injection, capsule, drink) but is inactive. In our example, it would be necessary to have 3 groups, those who receive no alcohol, those who receive the alcohol and those who receive a placebo. To eliminate any Experimenter Effects, a Double-Blind Procedure would also be implicated. Population The group to which the results of an experiment are expected to generalise. Its unrealistic to expect to test everyone who is likely to drink alcohol to see how it affects their performance, we can only select a representative Sample of people and test them with a view to generalising back to the whole population. For good comparison, the sample group should approximate to the population as much as is possible. (See Randomisation). Practice Effects Improvement in later performance as a result of earlier experience of the same (or very similar task). Practice effects are one type of Order Effects and again Between-Subjects Designs, Counterbalancing and establishing a Baseline can all help to combat these effects. Randomisation Procedure to ensure that every member of a target population has an equal chance of inclusion in a sample. Ideally, each member of the population of interest would be assigned a number and then the sample to be used in the experiment would be randomly chosen based on these numbers. This is unlikely to happen but it is good practice that once the sample is identified for testing, they are then randomly allocated to the different conditions in the

experiment. If for some reason your initial sample of the population of interest is biased (maybe you chose people who are particularly tolerant to alcohol), then the idea is that these individuals are equally distributed throughout your conditions and as such will not bias the group comparisons. Reliability Refers to the repeatability of an experimental result. If a test is reliable, then it is likely to provide the same results when given to a subject for the second time (although the length of time between test and re-test will alter the reliability because of the effects of memory, change in mental state etc.). Another way to investigate a tests reliability is to split the test in half (one possibility is to split on the basis of even and odd numbered questions), give half of the test to subjects at one point in time and then give the second half of the test to them at another point in time. If the test is reliable, then this splithalf reliability measure will show it. Establishing reliability (and Validity) of a test is time consuming so it is usual to use well-established tests that have proven reliability (and validity). Sample Portion of a population examined in an investigation. When all members of the target population are equivalent then any random sample will do. When it is known that the population is divided into sub-groups and that these are important because these sub-groups behave differently (e.g. males compared to females) then it is important to have the correct relative amounts of the subgroups in the sample. So if the normal population of interest consists of 40% females and 60% males, then it is important to have 40% females and 60% males in the sample (a structured sample). If it is suspected that there are subgroups in a target population but they are unknown then a random sample is appropriate. Subject Report Report either requested from, or provided by a subject which provides information concerning their behaviour in the experiment. It is supplemental to any experimental measures and can provide insight into why the person is responding the way they are or how they feel. It is particularly useful in the Pilot Study stage of an investigation. Theory An organised set of propositions intended to explain existing data in some domain and to predict future observations. Theories are either true or false. Good theories explain present data as well as (correctly) predicting future data. Theories generate Hypotheses in order to test the value of a theory but not all theories can be tested (e.g. Freuds Psychoanalytic Theory could explain the cases observed previous to its creation as well as making some predictions about future findings but it could not be directly tested for truthfulness because it involved un-conscious drives). Validity Refers to the ability of a test to measure what it claims to measure. Note - It does not follow that if a test is valid it is also Reliable. For example, a watch

can keep good time (i.e. it is reliable) but it could be an hour behind everyone elses watches (i.e. it is not valid). Within-Subjects Design Measurement of the effect of an experimental treatment on a single group of subjects. Subjects are compared with and without treatment and as such, an individual acts as their own Baseline. In our example, we would record the performance of the group on the reaction time task as a baseline, give them the alcohol, then measure their performance again, so the one group is both the Experimental Group (i.e. they get the alcohol) but theyre also the Control Group (i.e. their performance is measured without the alcohol). An easier example to understand is performance on say a Maths test. A withinsubjects design would mean that an individual took the Maths test at time 1 and again at time 2 and the experimenter would compare these two scores to see how performance had changed over time. Each individual contributes more than one score. (Compare with a Between-Subjects Design where instead of having an individual provide the two scores you would look at the performance of two separate groups, each providing one score.) Experimental Design - Design Checklist 1. Have you located a theory which is supposed to explain the phenomena of interest? 2. Have you developed an hypothesis which is of the form: According to the theory under test, if I measure variable Y (your Dependent Variable) under two different conditions of variable X (your Independent Variable), the measured values of Y will differ. You may be able to be more specific, saying which value of X will give the largest value of Y, or you may want to look at 3 values of X instead of 2 etc. 3. Have you identified the target population to which you intend your conclusions to apply? 4. Have you identified a satisfactory sampling strategy? 5. Have you decided whether to use a Within- or a Between- Subjects Design (whether youre going to compare two different groups of people or the same group of people under two different conditions ). Within-Subjects Designs use smaller sample sizes but are more liable to order effects when subjects must do all conditions, so there is a danger of experimental sessions getting too long. 6. Have you clearly identified your independent variable and how to manipulate it? 7. Have you clearly identified your dependent variable and how to measure it? (N.B. Level of Measurement and Measurement Scale). 8. Are there any confounding variables which might interfere with your manipulations or bias your measurements? Can you control them? 9. Does your design raise any ethical problems? (The main guideline is not to expose subjects to risk in excess of that which they normally encounter. Subjects should freely consent to participate). 10. Carry out a pilot study.

11. List any conclusions you are able to draw concerning experimentation as a source of valid and reliable data. Would you alter your pilot study when designing the full-blown study?