Types of Clinical Studies Thomas R.

Lunsford, MSE, CO ABSTRACT This article focuses on two types of clinical research in O&P, experimental and descriptive. An outline of steps is suggested and the concept of validity is reviewed as a prerequisite for understanding experimental design. Lastly, pertinent experimental designs are explained with an attempt to use orthotic and prosthetic examples. Note: Please refer to glossary of research terms . Introduction Most of what is done in O&P is based on broad generalizations that have not been subjected to the scrutiny of the scientific process. Performing clinical research helps transform these generalizations into principles that can stand up to challenge. O&P professionals need to embrace a process of critical appraisal for the many orthoses and prostheses they recommend as well as new ones they design. Once a subculture is established that constantly increases and validates O&P's body of knowledge through the scientific method, then the profession will be well on the way to self-sufficiency and distinction. Although research of all types is needed in O&P, the emphasis here is on clinical research. Does this mean practitioners should not do laboratory research? According to Slater, the difference between laboratory and clinical research is based on the extent and kind of control (1). In laboratory research most undesired aspects or variables can be controlled. In the clinical setting, controlling variables is more difficult since the researcher must consider moral, legal and ethical issues. One cannot remove feelings, human nature or prior experiences, or raise human subjects in sterile isolation like pure litters of rats. In the laboratory, efforts can be made to remove the extraneous, irrelevant and confounding aspects. The situation can be altered to suit the research. However, the clinical researcher must fit the design to the real-life situation, trying to account for or control undesired aspects. This article is divided into three sections: the pertinent categories of clinical research, the definition of internal and external validity, and specific research designs with examples. Clinical research has been written about extensively in pertinent literature and will be discussed in two major categories, experimental and descriptive. Experimental Research Experimental research involves the manipulation of one variable under controlled conditions and the observation of another. For example, the variable to be manipulated could be the type of prosthetic foot, and the variable observed could be energy expenditure or gait velocity. In orthotics, the type of articulated plastic ankle-foot orthosis (AFO) could be manipulated while observations are made on the gait deviations of a subject. In the previous examples the

independent variables are the type of prosthetic foot or articulating plastic AFO, and the dependent variables are the energy expenditure, velocity and gait deviations. The main purpose of experimental clinical research is to compare similar orthoses or prostheses and to establish cause/effect relationships between variables. One often hears of a "control group" or "controls" in relation to experimental research. This is a basic prerequisite since control provides a necessary baseline for comparison. In orthotics the magnitude of gait deviation before the application of an experimental AFO would serve as a baseline for control and later be compared to the gait deviations with the AFO. In prosthetics the baseline data may be recorded with the subject's existing prosthesis and later compared to a new component or experimental device. This pre-treatment status stands as the control against which a patient's progress is monitored in clinical practice. If a patient's condition improved after application of an orthosis or prosthesis it is logical to conclude the device is successful. Perhaps similar patients with similar problems would also benefit from the same intervention. To verify the first conclusion the patient's predevice status with and without other devices must be considered. The practitioner must attempt to identify factors (device, medications, therapy, etc.) that could explain the original functional or conditional improvement. These types of conclusions and assumptions are made daily in clinical practice. However, the practitioner is cautioned not to make generalizations until more validating data are obtained. Sometimes the accuracy and objectivity of the instrument used to measure effectiveness should be questioned. The method of measurement should be critiqued. Sometimes the patient's condition or function would have improved without the device. Results must be rigorously validated and subjected to experimentation before inferences or generalizations can be made regarding other patients. There is agreement concerning the steps involved in an experimental research project (5,7-9). They are as follows:

Determine the purpose of the experiment. Experimental research is conducted to answer a question. The key is to pose a question that is researchable.

For example: "What is the best way to orthotically manage a stroke patient?" is indeed a question, but it is not easily researchable. A researchable question must give an indication of the variables (independent and dependent) and the population on which the effect will be measured. The orthoses, prostheses or components being evaluated must be stated, the method of measurement explained and the subjects (patients) who will participate in the study described. An example of a researchable question might be: "Is an ischial containment AK socket more energy efficient than one with a quadrilateral brim in dysvascular amputees?"

Identify and review pertinent literature. A thorough review of pertinent literature is critical to planning re search. Knowing how other researchers have discovered important pieces to the puzzle will help discover the missing pieces. (Note: Many databases, such as MEDLINE and Orthobase, cover journal articles only.) Define the variables. The purpose of the clinical research must be defined in terms of variables. Determine which variable (velocity, strength, energy, etc.) best describes the performance of the orthoses or prostheses being evaluated. Also, how can this variable be measured? Are there any standard ways, procedures or instruments for measuring this variable? Consider the reliability of the instrument. Define the sample of the patient population to be considered. For research to have a meaningful outcome, researchers must use a randomly selected homogeneous sample. The sample must also be representative of the population that is to be generalized. To keep a project manageable, constrain the generalization to a specific subset. For example, consider the effect of a certain prosthetic design on adults age 20-59, or better yet, create subsets by gender and age such as males 20-29, 3039, 40-49 and 5059. Design the experiment. Experimental clinical research design is directed at achieving internal validity. Internal validity refers to the degree of assurance the results obtained and are indeed due to the device being evaluated and not due to confounding or extraneous variables (3).

For example, if the dependent variable is gait velocity and the independent variable, or intervention, is an AFO designed to restrain excessive dorsiflexion but allow free plantarfiexion, confounding variables such as multiple terrain surfaces, motivation, compliance, material thickness, trimlines, reinforcement, bilateral vs. unilateral, etc., should be controlled or eliminated for internal validity.

The reality of performing clinical research with human subjects is that control or elimination of all confounding variables is impossible. Therefore, it is advisable to use control groups that do not get the new device or component and patient (subject) selection protocols to minimize confounding variables. Moreover, the patients (subjects) can be stratified according to age, gender, history, etc., so that multiple inferences can be made about the relative effectiveness of a test device or component.

Determine the appropriate statistical analyses. One of the purposes of statistical analyses is to summarize data by reducingg long lists of multiple measures to average values. Ten knee orthoses evaluated for resistance to anterior tibial displacement with posteriorly directed forces from 50 to 400N in SON increments with five redundant sets of displacement measures will fill a laboratory manual with raw data that defies

comprehension. These data must be summarized into a one-page table of average (+/standard deviation) values to facilitate comparisons.

Statistical analyses also are used to determine if observed differences are significant according to an objective criterion. Clinicians should be wary of studies with statistically significant results that have very small treatment effects and are of questionable clinical significance.

Prepare a research proposal. Research proposals should be written to help secure funds for the project, to allow review of the methodology and experimental design by colleagues and co-investigators, and to obtain permission from the host institution to proceed with the study.

Descriptive Research Descriptive research is conducted to collect data without rigid experimental controls so characteristics, reaction or behaviors of a group can be explored. Sometimes descriptive research describes the historical events in the life of an institution or organization or the nature and extent of an illness or condition. Descriptive research includes opinion polls, case studies, surveys and normative studies (3). Data are collected through interviews, questionnaires, medical record reviews, diaries and rating scales. Surveys. Surveys are used to document conditions as they currently exist and can be national, local or even institutional in scope. Manpower needs, membership needs, incidence, prevalence and demographics are examples of the types of data collected in survey research. Questionnaires are commonly used to obtain specific information because a large number of people can be reached relatively inexpensively. It may be necessary to follow-up with a letter or phone call to remind subjects to return questionnaires. Questions must be clearly stated, unbiased and not leading. It is the author's experience that a typical response to a survey is 10 to 25 percent depending on the applicability of the questionnaire to the recipient. Case Studies. The primary purpose of a case study is to describe one person's performance with a particular intervention (device or component). Inferences and generalizations to other patients or to groups of patients cannot be made unless a formal research project, employing an adequate sample size, is conducted. Single case studies, although anecdotal, are valuable for planning further research on larger samples of a homogeneous population. For example, a clinician may have noted one of his/her patients experienced calf muscle strengthening in a solid plastic AFO, which is contrary to previous notions of atrophy. This could lead to a study of the physiological cross-section of immobilized muscle behavior under isometric control such as occurs with a cast or rigid AFO.

Normative Studies. The main purpose of normative studies is to establish norms, or baselines, for comparison. When the performance of a patient or subject falls above or below the norm, the patient deviates from the norm (3). Two types of normative studies are longitudinal and cross-sectional (3). Longitudinal studies involve repeated observation (measurement) of the same individuals at specific intervals over a period of months or even years. Cross-sectional studies involve making observations on individuals of varying chronological ages (3). Cross-sectional studies are more common because of the problems associated with following individuals over a number of years. However, longitudinal studies produce more information regarding the effects of growth, maturation, lifestyle, etc., since each person acts as his or her own point of reference. Much of the information regarding risk factors to heart disease were obtained from the famous longitudinal study called the "Framingham Project," which followed a population of men living in Framingham, Mass., over a period of 20 years (10). Internal and External Validity Experimental designs of clinical research can be described by the sources of validity or invalidity. Distinction must be made between internal and external validity. Internal validity relates to the control of the extraneous factors that confound the relationship between independent and dependent variables (1, 11-14). Ideally there is internal validity when the independent variable alone can explain the variation in the dependent variable (1, 11-13, 15). For example, consider a research project where the gait velocity of stroke patients is measured with a plastic AFO and then with a metal AFO. In this case the type of AFO (metal vs. plastic) is the independent variable, and gait velocity is the dependent variable. Obviously, the kind of ankle control (free, limited or restrained motion) could explain differences in gait velocity more so than the type of AFO. A plastic AFO with free ankle motion compared to a metal AFO with restrained motion will not permit the observer to conclude anything valid about metal vs. plastic as to the input on gait velocity of stroke patients. If the type of ankle control is identical for both the metal and plastic AFO, then the internal validity of the research design will improve, and differences in gait velocity in stroke patients can be related to intrinsic differences between metal and plastic AFOs (such as tactile contact area and weight). Internal validity also depends upon the control of confounding effects such as time, measurement and sampling (15). External validity means the extent to which the results of a study can be generalized to other subjects, such as populations, settings, treatments or instruments (15). If the results of the metal vs. plastic AFO study applied only to male left hemiplegics in the 50- to 55-year-old age group who were between one and two months post-onset, then the study would have poor external validity. In judging the external validity of a particular experimental design, the focus is on the assuredness with which the investigator can assume that the same results would occur with non-experimental subjects.

The relationship between external and internal validity is such that as investigators increase one they may weaken the other (13). Investigators should attempt to select the experimental design that maximizes internal validity, external validity, or both. Experimental Designs Campbell and Stanley discuss 16 experimental designs; however, the present discussion will be limited to a few of the more common designs (15). Internal and external validity exist in an inverse relationship. As the researcher increases the controls of the study, thereby increasing the internal validity, the ability to generalize the results to other populations-or the external validity-is decreased. The more controlled a study or experiment, the less generalizable it is to actual situations (14). For more information on experimental designs, the Campbell and Stanley reference is recommended as well as Reading Statistics and Research by Huck et al (13). One-Shot Case Study The one-shot case study is an uncomplicated, frequently used design (1, 13). An example of this design would be an orthotist who desires to study the relief of arthritic ankle pain with the use of an immobilizing AFO. The orthotist selects patients who have received such an AFO and determines the extent to which they have experienced relief. The explicit purpose is to establish the relationship between use of the immobilizing AFO and relief of pain. This one-shot case study is represented in Figure 1 .

Figure 1. The one-shot case study, where: Xl= pretreatment measurement; X2 = posttreatment measurement; TR = treatment variables; + = administered; - = not administered; EXP = experimental group

X1 is the pretest observation (none in this case). X2 is the post-test observation. The immobilizing AFO is the treatment (TR). The plus and minus signs indicate whether observations and treatments occurred. This design has poor internal and external validity, and there are no controls for temporal effects. Several events could have occurred that could explain a reduction in ankle pain. Because time was not controlled the arthritic patient could have treated the pain with medications, folk remedies, hypnotism, rest, etc. The informed researcher would explore alternative explanations. Even with this precaution, the complex and numerous events that might intervene as historical effects are not entirely identifiable and certainly not controllable. If the immobilizing AFOs have no effect on the ankle pain, but the pain spontaneously disappeared, then the investigator may erroneously attribute the relief to the AFO, or at best could not distinguish between the immobilizing effect and the effect of time. Attrition (also

known as morbidity/mortality) of the patients due to death or moving away also reduces the internal validity (1, 13). If subjects who drop out of an experiment are not similar to those who remain, the mean post-test score could differ from the mean pre test score simply because some of the subjects are not measured the second time (X2 in Figure 1 ). Perhaps the largest objection to the one-shot case study is the absence o ankle pain measurement before the application of the AFO (see the "minus" sign under Xl in Figure 1 ). It was hoped that the AFO would reduce ankle pain. . . but reduce from what level? Without pretest measurements it is not possible to calculate differences. In the sample case, differences in pain before and after intervention with the AFO cannot be calculated or compared. Statistical and clinical differences cannot be ascribed as significant or otherwise. One Group Pretest Post-test Design In the one group pretest post-test design, observations are made before and after the independent or treatment variable has been introduced to a group (1,13,15). The design is represented in Figure 2 . Again, Xl is the pretest observation and X2 is the posttest observation. With the two observations the researcher is able to make one comparison or contrast. For this hypothetical example of the effect of the immobilizing AFO on arthritic ankle pain, the researcher has the opportunity to compare statistically and clinically the differences in ankle pain before and after AFO intervention.

Figure 2. The one group pretest post-test design.

This design is better than the one-shot case study, but there are still many possible uncontrolled extraneous variables (threats to internal validity) that might explain the difference (or lack of difference) between Xl and X2. These uncontrolled variables include history, maturation, testing, instrumentation, statistical regression and mortality, and become confounded with the possible effect of the treatment variable (1,13,15). The arthritic patients might perceive pain measurements to be part of the treatment, different instruments or examiners might be involved, and the patient might purposely downplay or exaggerate the ankle pain. The patient may suffer pain at discharge, but strain himself to show improvement because he does not wish the orthotist who treated him to appear a failure or for himself to experience a personal failure. The confounding variables become rival hypotheses for the difference between Xl and X2, and they are threats to the internal validity of the study. Therefore, it is impossible for the researcher to determine whether differences or lack of differences between Xl and X2 were caused by the immobilizing AFO or by one or more of the many extraneous variables. Skill and foresight in data collection and observation and strict supervision of the circumstances under which the study is conducted may attenuate these effects, but they will not ensure the

control needed to distinguish between the real effect of the immobilizing AFO and the effect of extraneous factors. Static-Group Comparison The static-group comparison design consists of two groups, the experimental group (EXP) and a control group (CNT) (1,13,15). The experimental group receives treatment (i.e., the experimental variable) and then is compared to the control group that receives no treatment. This design is illustrated in Figure 3 .

Figure 3. Static group comparison.

The subjects in the two groups of this design are not randomly assigned to the group that receives treatment and the group that does not. Also, the dependent variable is not measured in either group prior to the treatment. Suppose, for example, an orthotist wanted to determine the post-injury differences in treated and untreated people suffering whiplash. Using automobile accident reports, the investigator compares individuals who received a cervical orthosis to those who did not. Again, the dependent variable would be pain in the cervical region. One source of internal invalidity for this study design is the selection of subjects for the two groups. The researcher cannot be certain the two groups are equivalent unless the whiplash victims are randomly assigned to the group receiving cervical orthoses and the control group that did not. Selection is a problem whenever subjects who seek treatment are compared to subjects who do not. The experimental and control groups could be so different that differences due to the use of the cervical orthosis are masked. History and maturation also degrade validity since the researcher cannot be sure the two groups are exposed to the same lifestyle. Another major weakness to the static-group comparison design is the absence of pretest measurements. Without pre-treatment data, comparisons within a group are impossible. Pretest-Post-test Control Group Design This design has two groups of subjects that are compared with respect to measurement of the dependent variable (1,13,14). Both groups are measured twice: before treatment and after. The two groups are created by randomly assigning half of the subjects to the experimental group and the other half to the control group. The pretest-post-test control group design is illustrated in Figure 4 . The (R) after the EXP and CNT group designators indicates that the members of these groups are randomly selected. An example of this type of research design

might be a prosthetist interested in comparing gait velocity of BK amputees who receive a Flex-Foot versus SACH foot. It is assumed a population of BK amputees has similar BK prostheses with SACH feet.

Figure 4. Pretest-post-test control group design.

The prosthetist desires to have 10 amputees randomly selected from the larger population and placed in the experimental group receiving the Flex-Foot and 10 in the control group who keep their SACH foot. The gait velocity of all 20 amputees is measured. Then, the experimental group [EXP(R)] receives the Flex-Foot modification (the necessary realignment and adjustments) and is given a period of time for becoming familiar with the new foot. Then gait velocity of both groups is measured again. Lastly, the velocities of each group are compared statistically to determine if significant differences exist between the two gait velocities for the control groups who did not get the Flex-Foot, the two gait velocities of the experimental group who did get the Flex-Foot, and the gait velocities of both groups initially and after the experimental group received the FlexFoot. The pretest-post-test control group design controls most threats to internal validity. There is no selection bias in the above amputee study since all 20 subjects were assigned at random. Since the groups are equal at the beginning of the study, history and maturation should affect both groups equally. If the amputees were selected on the basis of some factor, like extreme body weight, then the phenomenon of statistical regression of the mean may affect the results (13). However, this is not a threat to internal validity since statistical regression should be present in one group as much as in the other group. Testing and instrumentation will not be sources of invalidity since these elements are the same for both groups of amputees. If an instrument used to measure velocity were to develop a problem, it would affect both groups equally. Mortality also should not be a threat to internal validity since the two groups of amputees were randomly selected from the larger population and dropouts between pretest and post-test velocity measurements are equally likely for each group. Unfortunately, in this type of study, the group of amputees who do not get the experimental treatment (Flex-Foot) might feel slighted and be more likely to drop out. This may be a crucial source of invalidity. The prosthetist conducting the study must report any changes in the number of amputees in each of the two groups. Post-test-Only Control Group Design

This design is identical to the previous except the pretest is not administered to either of the two groups. The posttest-only control group design is most useful when pretests are unfeasible or unnecessary (see Figure 5 ) (1,13,15). Specifically, this design makes irrelevant any concern with the effects of measurement. In the absence of the pretest measurement, the random assignment of subjects is the basis for assuring similarity between the two groups. For example, suppose a prosthetist were concerned with the effectiveness of very early prosthetic fitting and training with congenital amputees. Birth certificates would be searched, congenital amputees identified, located and examined. By random selection a control group could be created for which no prostheses are used; parents are told to wait until the children are 4 years of age before taking action. Another group of infants is also chosen randomly, placed in the experimental group, given prostheses early and trained in their use. After this, the experimental group is periodically serviced, examined, and improvements and corrections are made to their prostheses. At 31/2 years of age, children in the experimental and control groups are compared with regard to physical, psychological, social and emotional adjustment. If the children in the experimental group demonstrate better adjustment, it can be attributed to early intervention with the prostheses, for any inherent differences in children have been canceled out between groups by random assignment. If there is no difference between the experimental and control groups, it is logical to conclude that the prostheses provide distinct advantage. Similarly, if the experimental group demonstrates poorer adjustment than the control group, one must conclude the prostheses have negative effects on adjustment. Solomon Four-Group Design One of the factors that threatens generalization of results (external validity) is the intervention between pretest and the treatment (12,13). If the pretest sensitizes the subjects to the treatment, then external validity will be limited. This is especially true in descriptive research situations where the pretest sensitizes individuals to the treatment (see Figure 6 ). Subjects are randomly assigned to four different groups. Two experimental groups receive treatment. One experimental group is measured pretreatment; one of the control groups is measured pre-treatment. All four groups are measured post-test (1,13,15). As an example of the Solomon four group design consider the following hypothetical study. Assume that the American Academy of Orthotists and Prosthetists (AAOP) has produced a film for consumer viewing extolling the advantages of ABC-certified practitioners. The AAOP wishes to determine the impact of this film on the attitude of consumers of orthoses and prostheses. A sample of 400 consumers is identified and divided by random assignment into four equal-size groups. The first group is the experimental group and asked to answer an attitudinal questionnaire, then after two weeks, subjects watch the AAOP film promoting ABC-certified practitioners. After the film they answer the questionnaire again. The second group (i.e., the first control

group) answers the questionnaire, waits two weeks, then answers the questionnaire again without viewing the promotional film. The third group (i.e., the second control group) views the AAOP film and then answers the attitudinal questionnaire. The fourth group (i.e., the third control group) answers the attitudinal questionnaire only once. The initial attitudinal questionnaire scores for the two groups who answered the questionnaire twice are compared to determine if these groups were initially the same or different as to their attitude toward ABC practitioners. Internal validity requires that they be the same. If their attitudinal scores proved not significantly different, their attitudinal scores would be compared after one of the groups had viewed the film. Similarly, post-test scores of all groups would be compared to isolate the sensitizing created by answering the questionnaire before viewing the film. The interaction effect between questionnaire and film should not be significant for internal validity (15). Few experimental designs reported in professional literature conform exactly to the experimental designs presented so far. In fact, most experimental studies use many variations of these designs. The last three designs reviewed have built-in controls for the threats to internal validity. However, these designs do not have built-in controls for all of the sources of external validity (12). External validity is divided into two main categories, population and ecological, for which there are at least 13 possible sources of threats (1,12,13,15). The reader is encouraged to review the last two references for more details on external validity. Conclusion The key to good research is careful planning, proper experimental design and an understanding of the factors affecting validity. If researchers consider these factors carefully and acknowledge weaknesses openly, they can be sure those reviewing their manuscripts for publication or those evaluating the quality of their research for possible funding will as well. Suggested Readings 1. Gehlbach SH. Interpreting the Medical Literature. Lexington, Mass.: The Cullamore Press, D.C. Heath and Co. 2. Campbell DT, Stanley JC. Experimental and Quasi-Experimental Designs for Research. Chicago: Rand McNally, 1966. 3. Hays WL. Statistics for the Social Sciences. New York: Holt, Rinehart and Winston, 1973. 4. Kaplan A. The Conduct of Inquiry: Methodology for Behavioral Sciences. San Francisco: Chandler, 1964. 5. Stanley JC. Improving Experimental Design and Statistical Analysis. Chicago: Rand McNally, 1967. 6. Winer BJ. Statistical Principles in Experimental Design. New York: McGraw-Hill, 1971.

7. Kuhn TS. The Structure of Scientific Revolutions. Chicago: The University of Chicago Press, 1970.

Thomas R. Lunsford, MSE, CO, is director of the orthotic department at The Institute for Rehabilitation Research in Houston. He is also president of the Academy. References: 1. Slater SB. The design of clinical research. J of the APTA 1966;46:265-8. 2. Stuckey 5, Beekman C. Clinical research in physical therapy: a resource manual. 1st ed. Sacramento, CA.: California Chapter, APTA, 1985: 3. Makrides L, Richman I. Research methodology and applied statistics - a seven-part series. Part 1: general principles and basic concepts, Physiotherapy Canada, 1980 1981. 4. Shepard KE. Qualitative and quantitative research in clinical practice. Phys Ther, 1987;67: 12:1891-4. 5. Payton OD. Research: the validation of clinical practice, Philadelphia: F.A. Davis Co. :43-8. 6. Michels E. Design of research and analysis of data in the clinic: an introductory manual for clinical research. APTA, Alexandria, Va. 1-4. 7. Madison M, Chapman MW. Organizing a scientific clinical research project. Orthopedics 1989;12:1O:1297-1303. 8. Hulley SB, Cummings SR. Designing clinical research. Philadelphia: Williams and Wilkins, 1992. 9. Currier DP. Elements of research in physical therapy. 2nd edition. Baltimore: Williams & Wilkins, 1992. 10. Kannel WB, Castelli WP, Gordon T, McNamara PM. Serum cholesterol, loopoproteins and the risk of coronary heart disease: the Framingham study. Ann. Intern. Med 1971;74:1. 11. Campbell DT. Factors relevant to the validity of experiments in social settings. Psychol Bull 1957;54:297-312. 12. Bracht GH, Glass GV. The external validity of experiments. American Educational Research J 1968;5:437-74. 13. Huck SW, Cormier WH, Bounds Jr. WG. Reading statistics and researchHarper Collins, 1974;258-67.

14. Domholdt E. Physican therapy research: Principles and applications, Philadelphia: W.B. Saunders Co., 1993;86-104. 15. Gage NL, Campbell DT, Stanley JC. Experimental and quasi-experimental designs for research on teaching. Handbook of research on teaching. Chicago: Rand McNally, 1964:275-99.