Академический Документы
Профессиональный Документы
Культура Документы
3
4
1 C-O-N-T-E-N-T-S
2
3
4 Welcome and Introduction.................................................................................3
5 Susan Neuman
6
7 What is Scientifically Based evidence?
8 What is Its Logic?
9 Valerie Reyna.......................................................................................5
10
11 The Logic and the Basic Principles of
12 Scientific Based Research
13 Michael Feuer....................................................................................25
14 Lisa Towne........................................................................................34
15
16 Research..........................................................................................................46
17 Stephen Raudenbush
18
19 Math Education and Achievement....................................................................65
20 Russell Gersten
21
22 Implications for Scientific Based Evidence
23 Approach in Reading
24 Eunice Greer.....................................................................................78
25
26 Safe and Drug-Free Schools.............................................................................92
27 Judy Thorne
28
29 Comprehensive School Reform......................................................................103
30 Becki Herman
31
32
33
34
6
7
1 P-R-O-C-E-E-D-I-N-G-S
2 9:05 a.m.
4 Neuman. I'm Assistant Secretary for Elementary and Secondary Education. It's
9 critical. As many of you know, we have counted one hundred and eleven times
11 What our goal today is, is a very practical one. What we want
15 office, the Office of Elementary and Secondary Education, and that is, how do we
17 What you are going to hear today is not only some wonderful
19 characteristics, what it is and what it isn't. But, then, after a break, what we hope
20 to do is really focus on what does this mean for safe and drug-free schools,
23 all of our programs so that we begin to really look at the scientific basis underlying
24 what we say and what we do for schools in districts across the country.
26 pace. You'll see that there is opportunity to ask lots of questions. We ask you that
9
10
1 the questions you raise, please focus on the implications of this issue, not whether
4 that I was already late. What we are going to do is we are going to keep people
5 moving in a very fast pace and then give time for your questions. Then have a little
6 break, move it on to implications and then, finally, have a panel where you really
7 are able to address even more questions. We are delighted to have you all today.
9 the Deputy OERI, Office of Educational Research and Improvement. Her topic is
14 and I gather that our well-organized organizer is going to keep the question and
20 burning that's informational, if there's something that doesn't make sense at all, it
21 wouldn't be a good idea not to communicate. So, please do raise you hand for
23 kind of give and take session is what I am really looking forward to, so that I can
11
12
13
1 although I don't think in the very short time that I have available that I could really
2 give you a coherent argument that supports and defends the notion of scientific
5 that it's useful to think about what is the alternative to scientific research? If you
8 course), it includes such things as tradition -- this is the way we've always done it,
9 for example, superstition, there are -- you know, you throw the salt over your left
10 shoulder and the reading scores go up! No, actually, there are things that are not
11 based in fact that in fact become lore that if we really knew the scientific basis of it
12 we would discover that those things in fact are just superstition. They are
13 unfounded beliefs.
16 really a good question. Anecdote is a series of stories that you tell about things
17 that have happened to you in your life. They can be very entertaining anecdotes.
19 however, and this is, of course, well known in medicine, is that individual cases
24 out to be false and misleading. Sometimes they are very representative, sometimes
14
15
16
1 drawn on already.
3 used to bleed people. People would get sick. You know, I think it was when
6 think they probably were well-intentioned, I don't think they were trying to hurt the
7 president, why is it that they didn't notice that it wasn't working? It wasn't just
8 with this one patient, it was with many patients. Yet, somehow, personal
11 only in the 1940s that the randomized experiment where you know you had 2
12 groups, and you randomly assigned and all of that became routine and a standard,
13 the gold standard in medicine. That is very recent in historical terms. Prior to that,
14 we relied on those things I talked about in the first slide, like tradition and bleeding
15 people.
16 One of the reasons why clinical trials are not sufficient has to do
17 with the psychology of human thinking. I won't go into it in any depth, but I'm
18 actually a cognitive psychologist and there's been research done about when you
19 ask people to report about things they have directly observed and directly
20 witnessed and the biases that can creep into that type of reporting. These are
21 normal human biases that are generally adaptive, but they have predictable pitfalls.
22 So, if you rely on your memory for past events, we know that that memory will be
23 biased, and so on. Drawing simply on your personal experience alone is not a solid
25 Clinical trials in fact are the only way to really be sure about
26 what works in medicine. The logic of it -- and the other speakers are going to go
17
18
19
1 into far more depth than I really have the time to do, the logic of it is basically the
2 following: You have a group of people that you want to make a conclusion about.
3 You want to say this intervention -- whatever it is, if it's a new reading technique,
6 you flip a coin essentially as to whether they are going to be in the group that
7 actually gets the intervention or gets some kind of comparison, like what you
8 would have done had you not done this new thing. Standard treatment, that's a
9 common control.
10 The idea is that if you do this enough times and you get big
11 enough groups, you've got two groups, the fact that you're flipping a coin ensures
12 that these two groups, if you have enough people in them, are going to be
14 Why is that? Because there was nothing that put one person in
15 one group as opposed to the other. It was all by chance alone that you ended up in
16 the reading intervention group as opposed to the control group. And, so, all the
17 ways in which people do in fact differ, and people do differ, should be represented
18 in both groups. They should be comparable in every way, except the one thing that
19 you made different in their lives, therefore, we can isolate the effect of the outcome
23 interpretations.
24 Now if you have too small a sample, obviously the logic doesn
25 t follow. Because you can have all the smart people in one group, the not so smart
20
21
22
1 if you only have a few. If you do this enough times, you get a big enough group,
2 they will be representative. That has been proven mathematically by things like --
4 The bottom line here is these same rules about what works and
5 how to make inferences about what works, they are exactly the same for
6 educational practice as they would be for medical practice. Same rules, exactly the
7 same logic, whether you are talking about a treatment for cancer or whether you're
8 talking about an intervention to help children learn. The same logic applies. In
9 fact that's something I've said in talks for a period of time and the National
10 Academy of Sciences report, which I know Mike and Lisa are going to talk about,
11 in fact makes a similar claim. The rules of the game are the same.
12 I have the word "brain surgery" up there. The reason I have the
13 word "brain surgery" up there is that I think, you know, when we talk about
14 medicine and things like brain surgery and cancer, it is very, very important to get it
15 right. We all recognize that and most of us buy into that. You know, that you've
16 got to have randomized clinical trials because we want to be able to benefit for
19 brain surgery. We are effecting them one way or the other. Sometimes what we
22 benefiting that student or not. We really don't know. It may be well intentioned,
23 but that's not sufficient as we can see from the example from bleeding. So, it is
24 brain surgery essentially and it deserves the same kind of respect for the nature of
26 Next slide. So, I just told you that the randomized clinical trial,
23
24
25
1 this randomized experiment where you can assign people to two groups and
2 chance alone determines which one they end up in so that they are comparable in
3 every way except for that key thing you want to look at in terms of cause and
4 effect, I said that is the best form of evidence, and it is. It is the best form of
5 evidence.
7 that you can draw on? Now, we've exhorted you through legislation and a number
8 of other things, you must use this, but is there a lot of gold standard level evidence
9 out there about all the things we do on a daily basis in the classroom?
11 there. A lot of the evidence, however, is lower on the hierarchy of the strength of
12 evidence. I am going to just touch on this briefly. Again, the other speakers are
16 we can describe as quasi experimental or large data bases that essentially have lots
17 of characteristics of students in them that you can correlate with one another and
20 the real world randomness is a very rare thing. It's a very artificial thing. In the
23 with everything, you know, your neighborhood, your number of books in the
26 through statistical magic, that's basically it, and you can artificially create a sort of
26
27
28
3 statistically attempt to control, to capture basically the logic of that gold standard,
4 the randomized experimental trial. That's always the logic, that's always the goal.
7 something you are not controlling for that in fact is causing your outcome. That's
8 always possible.
10 you at least know that something is maybe probably true, that there's a large
11 number of what's called in public health epidemiological studies, and there would
13 things controlled for. You know, you could at least say, well, it's probably true.
14 That's certainly better than we have no idea, much better than no evidence, well,
15 what do you think? It's not the top level of evidence, but at least it is evidence.
17 is evidence based theory, and the evidence based theory is the crucial part.
18 Theories whose predictions have been confirmed and disconfirmed -- you know,
19 there's been an opportunity to disconfirm them as well, they've been tested -- that
20 are explanatory, that go into the mechanisms of how people learn, how they learn,
23 intervention was effected, than you have some clue as to whether you can
24 generalize it to your classroom, because you know the mechanism. You know
25 what's relevant and what's irrelevant to the causal course of that intervention.
29
30
31
1 that? Because we have an inclusive theory of how learning happens and it doesn't
2 have to include peoples shoe size. Right? So, if we have a tested theory, we can
3 sometimes extrapolate beyond just the limited group that was originally studied.
4 You know, sort of the boundary conditions for when an intervention is likely to be
5 effective.
7 sometimes it can turn out to be that it doesn't follow for that group for other
12 principals, so on and so forth and the student? Doesn't science really take the heart
13 out of things?
17 values, for example, or science and emotion. That is a false dichotomy. When we
18 think about values, I think it is important to recognize that evidence does not
19 determine our decision solely. It is not just the facts. It's the facts plus values. But
20 without the facts, we might make the wrong decision, even based on our values.
22 The facts, the evidence is necessary to make decisions that effect students' lives,
23 but it's not sufficient. But it is necessary. That is what we're promulgating, that, at
24 least, it be part of the discussion so that we can base practice on it. So, we're
25 talking about science with a human face, and that's a person --.
32
33
34
1 practice is very complicated. There is even research on how to do that. It's called
4 OERI, the Office of Educational Research and Improvement. We are thinking very
5 hard about how to do this, how to most effectively be useful to you and to support
8 might have. I am going to stay for the whole day and practical suggestions about
9 education and
10 training, that sort of thing, would be enormously helpful for us. I think this
11 symposium we have here today is a wonderful first step in that. But, it's the kind
14 go through the next slides much more rapidly. I'm just going to sort of allude to
15 points, and then if people want to talk to me more in depth, I'd be happy to do
17 We can't get the slides up over there? Can you see and can you
18 hear?
21 again, we don't have even the second level evidence about all the practices that
22 currently occur in the classroom. Nor do we have even second or third tier level
25 comes into play, to fill those gaps in evidence. That is inevitable. You have to
26 apply your judgment. There are whole books written and research done just on
35
36
37
1 the nature of human judgment. As you make decisions, you might want to dip into
2 that literature. It's actually quite helpful. Leaders of industry and business often
3 get consultants to advise them about the nature of decision making and decision
4 analysis.
8 It's an inevitable thing that has to be an ingredient today and probably for many,
10 We are just not going to know everything right now. That is the
11 nature of science, and we are going to discover new things that make the old
12 knowledge obsolete.
14 past, if it's truly science. It doesnt throw away things people have learned that in
15 fact have been effective. That is not the nature of science. Science is by its
16 essence cumulative.
18 what's up on that slide, is that it's objective evidence. It's the kind of evidence that
19 if two people watched something, they'd say yes, that's what happened there. The
21 with causal theory. That's a whole other level, but at least what happened at a
22 surface level is agreed on. Then you make hypotheses about why it happened and
23 you test those and you can be wrong in science. That's the nature of empirical
24 evidence.
26 dimensions. One of them is the quality, and that is primarily in terms of scientific
38
39
40
1 merit, and that has to do with the method. When I was talking before about
3 methods of analyses. That has a lot to do with the quality of the evidence. So, if
4 it's high on the hierarchy, if it's the gold standard, it's top quality. If it's one notch
5 down, it's second level quality and so on until you get to things that are really at
6 the level of anecdote which are maybe slightly suggestive, but they're not the
9 Scientific merit and good methods alone don't make the best scientific research. It
10 has to be relevant to your practice and it has to be significant. The more significant
11 it is, the more people are effected by something, the more severe the issue is that's
14 example, and you look at the way they evaluate grants that they receive in the
15 sciences, it turns out to be exactly those two criteria: scientific merit, relevance and
16 significance.
17 Next slide. So, here's a little bit more detail on what I talked
18 about before about levels of evidence. What are the levels of evidence? Again, for
19 those people who can't see, we'll make this available in some form or other.
21 about his. But, we have our randomized trial at the top, then our quasi
22 experiment, then our simple correlational study, and so on down the case studies.
23 Go ahead. This is the logic once again in more detail about why
24 randomized control studies are the gold standard, why they're the highest level of
25 evidence, why it's what you should rely on with the greatest weight by far.
41
42
43
1 What that means is people are assigning themselves to groups in the real world and
2 it's not random. People of a certain type tend to belong to certain groups to do
3 certain things.
5 coffee or is it the smoking? Well, you have to control for the drinking of coffee.
8 on this ways in which people are -- differ that are correlated with one another.
13 thing. Scientific merit you should use the hierarchy of evidence as your guide and
16 much more difficult. But, one of the key things you can look for is does the study
18 how many times people say there's evidence for something, then you go look it up
19 and some very obvious things are wrong like they studied something else.
20 They say one thing and it's really something else. So, they say,
21 okay, the effect of the graphing calculator on the ability to, you know, do certain
23 some arguments about transfer. And they didn't look at graphing calculators, they
25 So, you'd be amazed at how many things you can screen out by
26 asking some simple, common sense questions about relevance. You'll screen out a
44
45
46
2 One of the things you can do is you can search the literature,
3 obviously. Some of that requires, however, you know, folks that have advanced
4 training. And how to do that and how to bridge that is something we should talk
5 about.
8 touchstones. You can search for evidence that has been interpreted. For example,
11 they are summarizing the research in a field, the quality of those summaries varies
12 a lot. Some of them are essentially an opinion piece. This is what I think.
13 People's opinions are interesting, but it is not something you want to necessarily
16 scientific and another person looking at the same literature would make a similar
17 conclusion, those are the ones you want. So, meta-analysis is totally superior to a
18 narrative one.
21 These are our goals and they are in our strategic plan and we
22 really mean them. We're trying very hard to achieve these goals.
24 ultimately looking for here though is that, as it is in medicine today, that at some
25 point and I think this point is inevitable in the future, at some point the use of
26 scientific research as a basis for educational practice will become routine. It will
47
48
49
1 become customary and people won't be able to imagine a time when that wasn't
5 I'd like now to introduce Michael Feuer and Lisa Towne. They
9 many copies.
11 for Education at the National Research Council. Michael is the director of the
12 Center for Education at the National Research Council of the National Academy of
13 Sciences.
15 logic and the basic principles of scientific based research, as well as help us focus
18 invitation, Susan, and thank you to all of you for coming out to listen to lectures
20 We're here to tell you a little bit about a report that was released
21 at the end of November in this handsomely bound pre-publication form. It's called
23 some of the highlights of both why we were asked to do this and what you would
24 find if and when you opened the book and read it which I hope you do.
50
51
52
1 are not part of the government, although we work closely with the government
2 and on behalf of the American people. This is an idea that goes back actually to
3 the 19th century when President Lincoln looked around and discovered that there
4 were some serious problems that perhaps science and technology could help him
5 with. I'll just tell you one quick story which my poor staff hears this so much that
6 they tend to nod off when I get into this, but if they'll indulge me.
7 One of the very first problems that this new Academy was
8 confronted with had to do with a problem in the Civil War which was the ironclad
9 ship. This, as you recall from history class, was an invention that actually
11 There was a problem with the ironclad ship, however, and that
12 is that they couldn't get the compasses to work because of the magnetic fields.
13 Now, if you are ever interested in a sort of classic case of the collision between
14 science and public policy just think about a ship that you can't get to -- you know,
15 knowing the difference between north and south with the Civil War at hand is not a
16 trivial matter.
17 This was one of the first problems that the Academy was asked
18 to solve and, indeed, a small committee of physicists and engineers was brought
19 together and they actually solved the problem, and I am actually happy to tell you
21 (Laughter.)
23 not the first time the Academies have been asked to weigh in on this. There were
24 reports going back even to the late 1950s, and then later through the '70s and '80s
25 and '90s. And that, in itself, I would submit, is an interesting little bit of evidence
26 (perhaps anecdotal, but maybe not) of the very perception that education research
53
54
55
1 has, at least in part, an important scientific component. Because, after all, we are
2 not the National Academy of Poetry, we are the National Academy of Sciences,
3 and when we were asked to take on a question of the scientific quality of education
4 research, I don't think that was coincidental. I think that is part of a very important
7 I began speaking with some of the distinguished scholars around the country. And,
9 education research, I have to tell you that one of these very distinguished scholars
10 said, "Well, that's great. Finally, we'll have a short report from the academy."
12 and that is that the general perception of a low level of quality in education
15 support or refute the claim of the overall quality of education research being poor.
16 But we did take as a datum that the perception that it is poor is important and that
17 it is, therefore, worthy of the attention of some very distinguished scientists and
20 requests for study of the scientific nature of education research should come at a
21 time when we probably have more information, more data and a more relentless
22 flow of ideas about how to fix the schools than perhaps at any time in history.
23 Again, I haven't done the empirical research on this, but I would bet that education
24 policy gets more headline attention than almost any other item on the domestic
25 agenda. To some extent, I think the Administration, and Congress have conveyed
56
57
58
1 particular just after this horrible season of terrorism that we have just come
4 But, that said, there are lots and lots of folks who have gone to
5 school and who therefore have very firm opinions about how to fix the schools.
7 mean, we're responsible for some of the standards documents. And, I sympathize
8 with people in the real world such as yourselves and with teachers and educators
9 all around who have to sift through this morass and make something significant
10 and effective. That's where the appeal of science becomes very strong. It is after
11 all an enterprise that attempts to distill from the cacophony of ideas and anecdotes
12 and impressions, the nuggets of really enduring value, and that kind of knowledge
13 upon which you would want to base important decisions about kids, about schools
15 Having said all that, let me just offer a little bit of a foundation
16 here for what Lisa is going to tell you more specifically and that is some of what's
19 to take on a set of questions having to do, really, with first principles: What is
20 science? That in itself took a few weeks to sort through. What are the principles
21 of science and how do they apply to the science of education? These are very
22 tough questions. What you will hear about is some of the key findings of an
24 education scientists, statisticians. This is the way we do our work. We bring these
25 types of people together. And, after all, the National Academy of Sciences
26 obviously exists in some measure to promote the values and the ethos of science
59
60
61
2 So, much of what Valerie has said resonates with the underlying
3 purposes and -- are we trying to follow along with the slide show? Because
4 nothing I've said so far is on any of these slides. We have a unit at the Academy
6 (Laughter.)
9 education research.
10 On the one hand, I think what you would see in the report and
11 what you'll hear about is a great deal of enthusiasm and encouragement for the
17 the underlying proposition here that has been articulated in the law and that most
18 of you now are going to have to turn into the real practical day to day.
24 to this I think are quite significant because of his work on what human rational
26 The story that he liked to tell was about the traveling salesman
62
63
64
1 who had the following problem: to visit 15 cities and to work with customers in
2 15 different cities and wanted to minimize the costs of visiting those customers,
3 fuel costs, time and so forth. What's the rational way to approach that problem?
5 routes you could take and then calculate how much it would cost because of the
8 necessarily. And that's because by the time you lay out all of the different routes
9 and you mathematicians out there will figure this out pretty quickly that 15
10 factorial routes is a pretty large number. And, so by the time you have gotten to
11 the end of the list, 20 years have passed. Your competitor who is using a less
12 rigorous, less optimal approach has gone to Cleveland and then figured out that
13 the next stop ought to be Buffalo because that's closer than Houston. And, you're
17 depends on the time you have, and, frankly, as Valerie has I think so eloquently
18 reminded us all, a lot of the decisions that have to be made are going to be made
21 challenges is to encourage the field of research to provide you with better and
22 better useful evidence. And, don't think for a minute that we researchers have
23 figured all this out and the only problem is you people in the real world aren't using
24 it. We know that's not true. The research community has a lot to do to shape up
65
66
67
2 I don't want to take time away from Lisa because the real
3 messages of this report are what I think are going to count at the end of the day.
4 So, I thank you for letting me give you a little sermon about
5 rational decision-making. And, now I will try to sit down rationally and let you
7 (Applause.)
9 want to, with time considerations, just sort of pick up where Michael left off and
10 like he said just give you a brief sort of tour through what's in a somewhat longer
11 volume.
13 people. I wasn't able to bring them here today but I will work with her to make
14 sure that we can do that and that you will have the pleasure and the privilege of
17 give you a brief idea of what's in here with respect to the question we were asked
18 to talk briefly about today which is what are the basic principles of science. As you
19 might expect and as I'm very grateful to report, they do reflect in many ways what
24 be great.
25 What I'm going to do, just to give you a sense of what I'm going
26 to talk about today is talk briefly about the principles of science that actually are
68
69
70
1 common across all disciplines and fields. This is, again what Valerie said, that at a
2 fundamental level, medicine (that was the example that she used), ecology,
3 economics, all of the applied fields like medicine and agriculture, that there is a lot
5 The principles of science that I'll talk about today is what the
6 committee who wrote this report believes are those common elements.
7 Then, I'll spend a few minutes at the end talking about what is it
8 about education that makes the application of these principles look a little different.
9 Because you might be sitting there thinking, "Wow, looking at something that a
10 physicist does sure does seem a heck of a lot different than what an education
11 researcher does."
12 So, I'll talk a little bit about how these principles play out in
13 studying education and why it is that they tend to look very different.
15 very straight forward, perhaps self evident, but actually the process of posing a
16 new and different question is often times itself what is the basis of a scientific
17 breakthrough, someone thinking about a problem in a new way and asking a new
18 question.
19 There's a couple of words here that I'll just touch on briefly that
23 its relevance to the core problems of teaching and learning and schooling.
25 question can also derive from what has come before it. In other words, does this
26 question help to advance the field and consensus, and the cumulative nature of
71
72
73
1 science which is a theme that Valerie touched on and that this report also tries to
4 simply in very straightforward terms means can be observed. The only reason this
5 is relevant here is because there are some questions that are relevant to what
6 teachers do every day that can't be answered by science. Should students be asked
7 to say the Pledge of Allegiance every day, for example, has to do with our values
10 I will go on to the next slide, and talk about the principle that
11 has to do with theory, and again Valerie alluded to this as well. The importance of
12 theory is really very important in education research and the other sciences as well.
14 testing of theories that helps you explain some aspect of the world.
16 like evolution. Grand theories like that don't typically pop up in education but
17 certainly they are relevant and they certainly are kind of an organizing conception
18 for scientific work. Valerie mentioned a theory of how children learn, that's a great
22 investigation. The important point here is that data in an of themselves aren't really
24 conceptual idea that you have going in like about how children learn or about how
25 educational resources translate into, hopefully, better outcomes for schools and for
26 students.
74
75
76
2 with the implementation of this law, what works, there is some implicit theory
3 about how the program is supposed to actually translate into better outcomes for
4 kids. Should that point to the basis of a program evaluation? That's what Carol
5 Weiss calls "a program theory." So, sometimes it's explicit and sometimes it's
7 I will go onto the next principle on the next slide. This has to
8 do with methodology, which Valerie has already, thankfully, covered very well for
9 me.
10 I will just make three main points about the role of methodology
11 in scientific research.
15 anthropologists, they're sort of studying a different part of the animal and they all
16 bring their tools of the trade to bear on that. So, by definition, there are a range of
19 education research, that multiple methods used together tends to strengthen the
20 inferences or the conclusions that one can draw when studying these things
21 scientifically.
22 The last point that I will make about methodology and this gets
24 legitimate methods that can be used in studying education, some methods are
25 better than others for particular purposes. Valerie, actually, kind of very nicely laid
26 out kind of a hierarchy of evidence within the class of questions that are causal.
77
78
79
2 descriptive research. There's research that looks at mechanism. And, within those
3 classes of questions, there's also different kinds of methods that can be used. So
4 that the method itself, taken out of the context of a particular study, can't really
10 the logic behind science which, again, Valerie, has talked about and handled quite
11 well.
12 So, I'll go on to the next slide which is principle five, and this
15 the fact that since in any particular study you're only relying on a limited set of
16 observations, to what extent does what you're looking at here and now generalize
17 to other times, places and contexts. In education, as you know, this is a critical
18 question. Teachers and researchers alike have been knowing for years that
19 something that works in a particular classroom may not work in the classroom
20 next door and may not work in the same classroom a year later. So attention to
21 sort of what's going on in the classroom at that time can help you understand the
22 conditions under which things tend to work and therefore how to think about how
25 transparency of the scientific enterprise. Valerie alluded to this as well. This just
26 has to do with the role of the scientific community actually working together to try
80
81
82
1 and make sense of all of the findings and all of the conclusions that come from
3 among the research community and we'll grant you that there is some bickering.
4 But there is actually something important to say about that and that is that
5 researchers are actually trained and employed and paid money to be skeptical
6 observers and to ask critical questions. That's their job. So, this critical kind of
7 work, critiquing other peoples findings and trying to make sense of them is actually
12 education that help understand how these principles are actually translated in the
13 study of education. How much time do I have for that? One minute? I am
16 the so-called hard and soft sciences. And, that has to do with differences that
17 emanate from studying inanimate objects and studying people, which are complex
18 and do crazy things that we often can't understand or predict very well.
19 So, there are some things that are different. Broadly, research
20 or control is one of them. Think of it this way, a petri dish of heart cells is a heck
21 of a lot better behaved than a classroom of third graders. Anyone whose tried to
22 study education research and has done cell biology, as one of my committee
24 There's other things that are different. I'll just touch on this last
25 one on the slide which has to do with certainty. Valerie said, and the committee
83
84
85
2 general terms, in the physical sciences we because of this ability to control the
3 environment tend to have more certainty associated with them than sciences that
6 things in education, specifically, that actually explain and help understand the
7 nature of education research. Values and politics, Valerie talked about this as well,
8 the role of schooling in our democracy is one that is appropriately and historically
11 research is one part of that decision process and it should be, but interacts in a very
14 the fact that people don't always have the same agenda as a researcher might and
15 they might move around and mess up samples and do things like that. So, there's
23 the districts, in the state and even at the federal level to really have a good sense of
26 that I'll just touch on and then wrap up, about what characterizes education
86
87
88
2 One is something I've alluded to already and that is the fact that
4 agriculture, like medicine. So there are a lot of disciplines that legitimately bear on
6 understanding it.
8 be concerned with the ethical implications of what they're doing. Studying kids
9 who are a vulnerable population sometimes entails things that you have to do with
10 methodology and plan for research in order to make sure that they're protected.
11 Most of the time education research doesn't pose any risk and is exempt from the
12 federal regulations that govern them, but, none the less, it is something that factors
15 can't do their job without the cooperation of schools and students and all the
16 different actors who are in the education system. At the very least, they need the
17 cooperation for them to go in and collect data, to test them occasionally and
19 researchers and educators who are on the ground doing education day to day so to
20 speak, actually work collaboratively in a way that tries to both improve practice
21 through research, but also inform and improve the research process by better
24 (Applause.)
89
90
91
2 He's a colleague of mine at the University of Michigan, and he's one of those
6 Steve?
8 made me promise not to show any equations! I will have no slides, so if anything's
10 (Laughter.)
12 the American Academy of Sciences, not to be confused with the National Academy
13 of Science. The topic of the meeting was how to improve the scientific quality of
14 educational research.
16 Howard Hyatt and Fred Mosteller. For Mosteller and Hyatt it was a kind of a déjà
17 vu because they had been among the most influential people half a century earlier
19 They felt that the time was appropriate to make the same argument now in
20 education.
22 they were met with considerable skepticism. There was a famous (at that time, at
23 least) well publicized debate between Hyatt and a heart surgeon. Hyatt was
24 arguing that we should do experiments to see whether new surgical procedures are
25 really effective as compared to let's say medication. The heart surgeon asked him
26 in a very poignant moment, "Sir, have you ever held the beating heart of a human
92
93
94
1 being in your hand?" The surgeon argued that the cold logic of science did not
4 of cases the medical profession really doesn't know what the best thing is to do and
5 that in that situation it is unethical not to find out, and in fact if we can find out
6 what works best than over the many years many millions of people perhaps will
7 benefit and that would reveal the true ethical character of basing decisions more on
8 science.
10 Mosteller and Hyatt, has in many ways I'd say largely won out, that we now in fact
11 accept and admire the commitment of medical professionals to base, not all
12 certainly, but some of their key decisions on research from clinical trials.
14 caused the sea change in medicine and is it likely that anything like that might
15 happen in education. That's way too big of a question for me to try to answer, but
16 there is an interesting vignette, I guess, a part of the story that has to do with the
18 In the early studies in the '40s and early '50s on the Salk
19 vaccine, the studies seemed to show basically that the vaccine wasn't effective.
20 People who had the vaccine were almost as likely or it may have been in fact
21 equally likely to get polio as those who did not. By the way, at that time the
22 vaccine had not been perfected. It was certainly far from perfect.
24 were more likely to get the vaccine and higher income families in this case were
25 more like to in fact get polio. It transmitted in places like swimming pools, places
26 where high SES people, social class people actually had a higher risk.
95
96
97
2 randomized clinical trial on the vaccine. This was a double blind trial in which
3 physicians didn't know what vaccine, what treatment they were giving to people,
4 whether it was actually the vaccine or just the placebo of sugar water. And, the
5 people who were getting it didn't know what they were getting. Having grown up
6 in that era, you have to realize when you got sick in those days and the doctor
7 came to your house, remember when the doctor used to come to your house?
8 (Laughter.) Your parents would stand by in mortal fear as the doctor exercised
9 your legs and did various things to see whether it was polio.
11 clinical trial and the people didn't know what they were getting and the doctors
12 didn't know what they were giving. It's quite remarkable that this happened.
13 But the results showed definitively that the vaccine was far
14 more effective than not having the vaccine which led to further perfection, further
18 actually a very durable and interesting one, so maybe not being educated can cause
19 a loss of lives.
21 of the Head Start program showed roughly equal cognitive skills at the end of the
22 study if you compare the Head Start and the non-Head Start kids. But subsequent
23 research showed that the Head Start kids had higher levels of poverty than the
24 non-Head Start kids. Some then argued that the results actually showed that the
25 Head Start program must be effective because the kids were doing better than you
98
99
100
1 So, here's a result of two groups basically being the same and
2 one group of people saying this shows Head Start is no good and the other group
3 of people saying this shows Head Start is really good. The same evidence, but the
4 evidence is so weak that it can't really decide the question. Unfortunately, there
6 This leads to a crucial point that Valerie made. In both there are
7 striking parallels, as I said, between medicine and educational research. In both the
8 early vaccine non-experimental trial and in the Head Start evaluation, there is
11 more likely to get the vaccine, but also more likely to get the disease and therefore
12 the evaluation that didn't use random assignment was biased against finding the
15 biased also against finding an effective Head Start because in this case the Head
16 Start group kids were higher in levels of poverty which was associated with lower
17 achievement.
20 You see, we could match the kids -- we could have done a better study than the
21 first one. We've done many better evaluations since the original Head Start
23 sure that the people we're comparing are the same with respect to income or other
24 social indicators. But we can never be sure that we have matched on some of the
25 relevant confounders. Variables that predict getting the treatment that are also
101
102
103
1 eliminate confounders, all confounders even the ones we haven't thought of, and
4 form the basis of this paper and I will go through them rather straightforwardly,
5 through them rather quickly here. I've got actually ten of them.
6 The first one is: Am I then saying that only studies that use
7 random assignment are scientific? The answer is no, I'm not saying that.
9 question on the table. There are many terrifically important questions for
12 graduation rates changed over the past ten years? Which kinds of kids in which
13 kinds of cities and states and in which kinds of schools are at highest risk of
14 dropping out? Tremendously important for policy to know the answer to that
17 So, not all questions are causal. But, secondly, even when a
19 analogy with medicine: researchers have come to a strong consensus that smoking
20 causes lung cancer, but we never had a clinical trial where we randomly assigned
21 people to smoke two packs a day. Yet, we had a variety of scientific inquiry that
22 led to a strong conclusion. We need to know how family conflict effects school
23 achievement, but can you imagine the experiment that would test that causal
24 hypothesis? (Laughter.)
26 circumstances that limit the generalized ability of their findings. I won't go into
104
105
106
1 detail, but sometimes you need corroborating evidence from studies in a natural
2 setting that aren't randomized and across -- the randomized evidence might be
3 crucial, but you need to supplement it to see whether a new program works in a
6 questions, how do I then judge the scientific quality of the study that doesn't use
7 random assignment? I guess, what I would say here is that in all of science at the
10 So, if I see a study over here where these kids had a new
11 writing program and these kids didn't, and these kids, the kids in the writing
12 program are doing better than the ones who didnt, I don't just say, "That shows
13 the writing program is good." I think about other explanations for why that might
14 have happened and I evaluate them. It's harder to do when you don't have a
19 writing study, we randomly assigned kids to do the writing program or not, we'd
20 still need to develop alternative explanations for why the program worked. The
21 experiment might tell us that the program works. But we want to go further to
22 know what are the crucial ingredients because that may be very helpful to
107
108
109
1 implemented. So, people who drop out of the study, you'll have missing data in
2 the two groups. We still have to worry about subtle or not so subtle biases.
4 whether there was random assignment, but whether the investigators have
7 to expect this scientist, this investigator to police me, let's say, to police myself and
8 I'm a human being with biases and I'm supposed to evaluate all these things. Well,
9 the key point here is the burden of objectivity does not fall entirely or even
12 scientific community who can -- and this relates to democracy, being able to freely
13 evaluate alternative points of view, not feel that there's going to be some
14 censorship.
16 mentioned who evaluate this, the process of objectivity really involves this group
18 trained to be skeptical and that process can really work. What's really in the final
25 causal questions.
110
111
112
5 classes. I'm sure you're going to hear about some more of them today actually a
6 little bit later in the next session, if I don't talk too long.
8 evaluations of the James Comer whole school reform program. There have been
11 So certainly they can be done. The fifth question then is: How
12 can we do them ethically? In the paper, I sketch some scenarios where we can
16 "Success for All" simply because as an early literacy program, it's a program that
17 has -- there are over a thousand schools already in it. Many schools want to get
18 into it, but it's expensive. So, a lot of people want to get it, but they don't get it.
19 And also the people who run that program can only implement it in so many
22 who want to do it, perhaps give it to them free or at a reduced cost and just say
23 there's only one condition, we can't give it to you all at the same time. We're going
24 to have a lottery that's going to determine who gets it first which is a very fair way
26 So, during that interim period where one group of schools has
113
114
115
1 started to do the program and the others are still waiting, you have a randomized
2 experiment, and a very ethically organized one. That's just one example. There
5 do it in medicine. Like I said, the Salk vaccine trial was incredible, the double
6 blind experiment. We need to be able to make the argument and we need to learn
9 education are causal, and can I give you a few examples? I'm not going to give
10 you too many. But I do want to mention that we may not have been doing such a
11 good job in education of doing impact studies, causal comparative studies, what
15 have come out of those studies. And, I'm on the -- I'm going to toot the horn of --
16 the AERA National Science Foundation grants committee which has given out
18 report that shows hundreds of terrific scientific contributions coming out of that,
19 but generally not of the strong causal character because it's really based in fact on
20 survey research.
24 There is this class. We can't just forget about the fact that a lot of the scientific
25 research is not causal. So, we have a bunch of questions: How did we select the
116
117
118
3 assessed?
4 Those are some principles. But, once again, the key point is in
5 the final analysis it's scientific peer review that applies those principles in a case by
8 Does qualitative research play a role? I would say, yes, without doubt. Because
9 we need to not just test the impact of things out in the field, we need to do a lot
10 more of that. We haven't done enough. But we have to have good things to take
11 into the field. We have to have good ideas about how to teach math, how to teach
12 reading. Those ideas come from up close, careful study of expert practitioners in
13 real settings and how kids learn. So, we need that up close kind of research but
14 see we've got to do a better job of connecting that research with field trials of what
17 you have to have experiments and you have to have surveys and you have to have
18 quality of research, how do you combine the insights from the different kinds of
19 inquiry? I hate to go back to a medical example, but it's a very telling one. It's the
22 two packs a day, but you could do a randomized experiment on animals. Strong
25 nonsmokers using the best possible survey methods and qualitative research.
26 Here the analogy is looking at lung tissue and finding out that
119
120
121
1 the lung tissue of smokers is damaged in ways that we might think would be linked
2 to cancer. You put them all together and the weight of evidence, the experimental
3 evidence on the animals, the survey evidence on people and the lung tissue --
4 qualitative, put them together and you get a very compelling case.
7 My tenth and final question is: Is there any danger here that we
8 are going to be overselling the role of science in education? I think there is.
9 I've got a quote here from E. L. Thorndike who wrote the lead
11 won't read the entire quote except to say that Thorndike felt that a scientific
14 Unfortunately, by overselling what science can do, it led to a crisis of, you might
15 say, rising expectations that couldn't be met. For a long time thereafter science in
18 solving, the idea that we would have kind of a social engineering model. We'd try
19 programs, we'd evaluate them, we'd get feedback, the programs would get better
20 and the great society was going to be born out of this sort of scientific and
21 engineering model. That was an overselling. We couldn't really pull that off.
22 So, let's make sure that we have a balanced view this time. I am
24 this time. I am delighted to have had the chance to be here because I think we're at
25 a point in history where there seems to be for some reason a confluence of factors
26 and the determination of people who have some power here who organized this, to
122
123
124
1 really improve the quality of research in education and the link between science
4 (Applause.)
6 I know that we went a little bit longer. What we'll do is, I think,
7 take a break and then come back at quarter to, and then what we'll do is we'll
8 combine the two discussion sessions, since I really do want time for questions.
10 programs.
16 Let me tell you that all the talks are going to be on the web, as
17 well as in print. I know I forced people to rush through their presentations. The
21 introduce Linda Wilson. She's the deputy in OIIA, the Office of Intergovernmental
125
126
127
2 priorities for education over the next five years. It has very strong accountability,
3 much like the No Child Left Behind Act, and it will guide our work here at the
4 department.
6 nation's educational system. It's built on six strategic goals, which are create a
8 strong character, transform education into an evidence based field, enhance the
9 quality of and access to post secondary and adult education and establish
10 management excellence.
12 wall. It's a living document that will guide the course of our work here through
15 his intention to hold each department of education program, office and employee
18 your input to this process. As I said, it's going to be available on the web
20 21st.
23 scientific based research approach to our programs, so many of our programs that
26 really want to give you opportunity to ask questions and make comments.
128
129
130
2 of his work over the years. He's at the University of Oregon. He's done a lot of
3 work on reading comprehension, teacher knowledge, and today what he's going to
4 be talking about is the scientific based evidence and what that means for math
7 be brief on because there isn't a lot of scientific research in math. There's some.
10 there is no question that most teachers, even most parents, -- the reading is the big
11 emphasis there compared to math. But it's not that simple. For other reasons, the
12 math community of math educators at least for forty-plus years has looked at their
15 few exceptions of really systematically using the methods that Valerie and others
16 talked about earlier to build a knowledge base, but rather to study using the more
18 So, this is something that can change. There have always been
19 little glimmerings of change. There's a slight increase in the amount, but overall
20 the math education community has been quite resistant to that, where let's say in
21 the reading field there have always been at least two schools of thought, one in the
22 experimental group.
23 But rather than just dealing with how little we know and getting
24 us all depressed, I am going to give some highlights of some work we recently did
25 actually for the state of Texas who was beginning a big initiative in the area of
26 math, getting kids ready for algebra. So, it was basically, these kind of low
131
132
133
1 achieving kids who got to middle school and just were weak in all areas of math.
2 We tried to put together the scientific research, using the procedures we've heard
3 about in terms of meta-analysis and all, in the area of math for low achieving kids.
4 I did this with my colleagues Scott Baker and Dae Sik Lei.
6 with what we've been hearing about during the first session. We looked for studies
7 that used random assignment. We did include the quasi-experiments, the ones that
8 are kind of close, but they only were included if they had measures showed that the
9 groups were comparable at the beginning. So, if they just used the school down
10 the road, they were thrown out. They had to have at least one math performance
11 measure, which sounds weird. But there were articles published in journals that
12 either had teachers grades or students attitudes or certain interviews that we had
16 were students whose documentation was well below grade level, at least below the
18 But some of the things that worked, and again we don't have a
19 lot of replications, but they were pretty decent studies, is that when kids and/or
20 their teachers get ongoing information, every two weeks, every four weeks, of
21 where they are in math in terms of either the state standards or some framework, it
24 much of romantic work done in math. But the idea of having a system to know
25 where kids are and what they really know, rather than saying this kid is struggling,
26 this kid is struggling with fractions, manipulating fractions, more than one, with
134
135
136
1 dividing fractions, with a sense of place value once you get into the hundreds.
2 That information can be critical for low achieving kids, can be a life or death issue.
3 The second group we found, there was only six studies, is peer
4 assisted learning. It's usually tutoring. This is something that could revolutionize
5 practice. Invariably, when kids are partnered up, and it seems to be better if they're
6 heterogeneous pairs, there's one stronger student and one weaker student and they
8 So, peers can be excellent tutors. I'm not talking here about
9 cooperative groups of four, five, six kids. It's two. And if you see the difference in
10 classrooms when there are two, it's very easy for the teacher to quickly monitor
11 and get a sense of what's going one. Because kids are either working on stuff
12 together, giving each other feedback, taking turns, or they're not. When it's a
13 group of four or five, you're never quite sure what's this group discussing, these
15 So, the advantage of this, again we're not dealing with these
16 profound things but with these kind of building blocks of improving practice and
17 especially if this is based on the kind of data we were talking about can lead to
19 The one thing about the studies, and then we'll go on with the
20 finding, is that 60 percent of them used random assignment so they met the gold
21 standard. Another third were this quasi-experimental group, so overall the small
22 set we had were of good quality. And seven percent were partial -- they randomly
23 assigned teachers and gave us some evidence that the groups were equal at the
26 critical is did somebody come in and see were people doing what they're supposed
137
138
139
1 to be doing? Because one of the key findings from the 1960s is sometimes these
2 evaluations were done of people who were supposed to be doing science this way,
3 or math this way, reading this way, but there was no evidence that they were really
4 doing it. And, in fact, when people did drop-in site visits, they found they were not
5 doing it.
6 So, two out of three studies did have an observer come in once
7 or twice a week and make sure the thing was happening which sounds mundane
8 and all but was a critical thing. So the quality indicators of the studies were good.
10 up. With the peer-assisted learning, the six studies consistently showed moderate
11 effects -- and I'm not giving the exact numbers, but there's statistical ways to cut
13 When kids saw the data, and it was almost always on the
14 computer, how they were doing, which skills they needed work on, whether they
15 were making progress, these were moderately large, these were pretty large. This
16 was especially true not so much for special education students but for that other
17 that kind of at-risk group who are sometimes in Title I programs who sometimes
18 need tutoring, that giving kids this kind of feedback seems invariably to help.
20 two ways: explicit instruction, that includes both the very, very heavily tightly
21 sequenced work that Carnan and some of his colleagues did in math which has
22 everything sequenced exactly for kids and a beautiful array of examples, and some
25 looking kindergarten through eighth grade, but there is some evidence that
26 providing this degree of explicitness to kids, showing them strategies, letting them
140
141
142
3 there are many in the schools who do not advocate for such practice. This is
4 invariably useful and when that's removed from children, especially the children
7 very, very exciting ideas about the discussion teaching fractions and getting kids
8 immersed in real world problems that involve measuring and fractions and
9 equivalents. And the results? I put a question mark there. When we averaged
10 them together -- and again we're only dealing with four studies -- it came out about
11 zero.
15 that were done in inner city Philadelphia schools in terms of giving concrete
16 feedback to parents on how kids are doing. These are low achieving kids and
18 What the researchers found and they did two things. They set
19 up the tutoring, was one thing they did, and then using this randomized idea for
20 about half those kids and about half of the control group kids they also gave the
22 This was their reasoning -- and this isn't the only approach in
23 terms of communicating with parents -- that often by middle school when kids are
24 D students and basic math, whatever it may be called, the lower track courses, in
25 tending to get feedback it tends to be very negative. So, the teachers, if the kid
26 was having problems, they gave that information for the peer tutoring session. But
143
144
145
1 when the kid did well, they sent notes home, they called, now they could e-mail --
2 these studies were done a while ago -- and said you're kid is doing well you folks
3 should celebrate this. Go walk up the mountain, a pizza party, whatever it is. So
4 that the parents started to know the weeks, their daughter or son was doing well in
5 math.
7 findings. We don't. I just have a couple thoughts towards the future. Susan, if I
11 those studies. Because as you see from this very small group of approximately 15
12 studies, we found some things that could be immediately useful for helping the
15 a couple of just my thoughts on what I envision is. As in the area of early reading
16 about twenty years or so ago there was this insight and some beginning work on
17 the phonological or phoneme awareness idea and how critical that was. Initially, it
18 was very vague and no one quite knew what to do with it. There were some
19 programs that seemed to have parts of it. It took a long time for that to solidify.
21 late Robbie Case and Bob Siegler and others, in the beginnings of math. And, at
22 least in math, unlike years ago, we do have some measures that can predict. In
23 kindergarten, we're doing some work in Eugene Research Institute in both Oregon
24 and Texas at looking at predicting things by the end of kindergarten that will tell
25 you which kids are likely to be at-risk. So you can start to screen and get a sense
26 of stuff.
146
147
148
2 validly predict and I know David Gehry at NIH is doing some work along this
3 lines. So, we're maybe twenty years behind reading in this early intervention mode
7 called "number sense." You'll see it around a lot. Nobody knows exactly what it
8 is. It's sort of a sense of numbers, the way some kids just sort of take to it. You
9 ask them, well, you know, here are six things, we want nine, how many more do
10 you need? They'll just go "three." And, others will just go, "Well, you need some
11 more."
13 understanding and doing and strategizing. We have his general notion. It seems a
15 do the kind of array of scientific methods. So, that's one huge area.
16 And I'm only going to do one other one. But this is something
17 we've thought a lot about. One reason there's so little intervention research in
18 education is people who've done it you leave totally exhausted. You're developing
19 a new curriculum, you're training teachers, you're going in to see are they
20 implementing it the right way. You're problem solving. You're going, oh, my god,
21 why did we sequence the fourth week this way. You know, these things happen.
23 know, you do one or two of those. Then you say, well, maybe I'll do more, you
26 (Laughter.)
149
150
151
1 And you look at any discipline, and it's amazingly few people
3 But one system that the late Ann Brown developed is a very
4 good one. What it calls for it says let's be honest. You can't just run in there and
5 say this is a good way to teach math problem solving, where kids learn the stuff
6 and then they practice in context. You need a while to do what she called "design
7 experiments." To really go in and see what happens and collect data and not do
8 the control groups and the randomization. You need one or two of those to get
10 And they are not really just pilot studies. They are serious
12 developmental psychology, but trying to put them into useable packages that there
14 Math is a long way from this. But this combination of doing the
15 design experiments, but then not stopping there, to then test with the kind of
18 In terms of the last one, towards the future, I think because we're seeing such
19 consistence sense that when the teachers or kids get ongoing data where kids are
20 and what they need to learn once a month as opposed to once a year. It's a great
21 way in October to say, you know, this kid doesn't know how to multiply fractions.
22 So, he's in the 7th grade, but let's get that under her belt, his belt, so we can move
23 forward and this kid isn't going to get lost in pre-algebra. So, we need strategies
26 could do as twenty years ago Thomas Goode and Douglas Grouse did, which is
152
153
154
1 look at what's happening in schools and try to link them to outcomes. Because
2 we've got a huge array of measures in math, but we don't have a sense of which
4 So, those are my four thoughts towards the future and my sense
6 (Applause.)
8 today. She has done much work in the state of Illinois and been a director of
14 thing, and I'm not exaggerating, to be the seven or eight year old sitting in the
17 pick the 5 percent to 15 percent of the children in that room who will not learn to
18 read and figure out how you're going to tell them that it's okay. How are you
19 going to tell their parents? It's not okay. Leave no child behind.
21 build and see a converging body of evidence that tells us that we know something
23 early reading classrooms that will help ensure that all children learn to read.
155
156
157
1 Now, twenty years ago, when someone would say to us, well,
2 how do you teach kids to read, we were left standing there with our hands in our
3 pockets saying, well, a lot of different things work for different kids. We've come a
4 long way since then. It's much more comfortable. I'm much more comfortable
5 standing up here this morning, then I would have been fifteen years ago, saying,
6 well, there's a lot of stuff that might work, and if one thing doesn't work, try
7 something else.
9 Reading Panel Report that was delivered late in the year 2000. The panel sifted
10 through over 100,000 studies and the sieve that they used to sift these studies
11 through to identify the studies that met their criteria for inclusion in their analyses
12 were the studies had to come from a refereed journal, be published in English.
13 They had to focus on reading instruction for children pre-K through grade 12 and
17 straight experimental design, there was not a lot there, and we still have a whole
19 But as the panel looked at the studies that emerged from their
20 sorting and as they read the results, findings began to converge around these five
23 elements and talk briefly about some of the truths and some of the misconceptions.
25 awareness? Well, it's the ability to notice and think about and work with the
158
159
160
1 Before children learn to read, they need to know that words are
2 made up of one or more sounds, and that you can take those apart and change
3 them and that they make different words. They need to be able to work with
4 speech sounds.
6 we? What do we know about phonemic awareness? Well, we know that we can
7 teach it. There are systematic instructional practices that we can use to teach kids
8 to become more phonemically aware. Children who are more phonemically aware
9 are better at learning to read and to spell, and it also influences young children's
10 comprehension.
12 worksheets because it's working with sounds. So, if you go into a classroom and
13 all these little five year olds have their heads down and those big logs in their hands
14 that we call primary pencils, they're not working on phonemic awareness. They
15 need to be making noise. It's most effective when teachers work with small groups
16 of kids.
17 Now, let's look at the flip side. What are some of the
19 No, this is not an endpoint. There are a lot of other things that have to go on
22 minute. It is not the same thing as phonics. It's about spoken sounds.
23 It is just for at-risk readers? No, the research tells us that all
26 on every day for four years? No, 18-20 hours for most kids. Now let me tell you,
161
162
163
1 if you haven't been in a building in a while, kindergartners spend more time in the
2 bathroom in a year than 18-20 hours. It's a finite thing that needs to go for kids.
4 language and sounds so that they can use it to read and to write words.
10 we teach children letter sounds and relationships and then we let them practice
11 those on things that they're reading. We don't ask them to spend a lot of time
13 So, if we're working on "B"s and "A"s and "T"s, we don't ask
14 kids to read the word: can. We work on words like "bat" and "at." And, we give
15 them practice using the tools that they are learning, so that they see the efficacy of
16 those tools and they begin to see and discover the routineness and some of the
17 patterns in our language. Phonics instruction is most effective when it's begun in
20 There isn't. When the panel looked at the research on various programs of phonics
23 Phonics is just for kids who come from low SES backgrounds.
164
165
166
1 activity. This is an activity that involves repeated practice in applying phonic skills
2 to reading and to writing, so that kids have an opportunity to write and read and
7 reading instruction. When we say fluency, what we mean is rapid accurate reading
8 with expression.
9 Now, when kids can read rapidly and accurately what this does
10 is this frees up their little brains so that they can attend to what the text is about,
13 some very nice research. They explained the notion of cognitive capacity. If
14 you're spending all of your sort of brain energy sounding out words and trying to
15 identify words, you have nothing left to attend to what the text is about.
17 ounce of capacity that they have can be put toward the outcome that we're looking
21 Now, the best strategy for developing fluency that we've seen
22 coming out of the research is to give students many opportunity to read the same
23 passage orally, and these need to be reasonably easy for the kids. They need to be
24 at what we call their independent reading level, so they can read them with about
25 95 percent accuracy.
167
168
169
1 model of what this text sounds like, and then give them opportunities to practice
2 reading it orally.
4 thing as authenticity. No, authenticity is just saying words right and fast. That's
7 not. No. You're fluency varies with the text and with the topic and with the
8 conditions and the expectations for what you read. The same thing applies for
9 young children.
11 surprised by this finding, but there's no evidence that sustained silent reading
14 There is a lot of research that needs to be done, but sending kids off to read for
15 thirty minutes by themselves and not holding them accountable and not asking
22 vocabulary. Their oral vocabulary is typically much larger than their reading
23 vocabulary. The larger a student's reading vocabulary, the easier it is for them to
24 comprehend. The larger their oral vocabulary, the easier it is for them to
25 comprehend and to read. Because when the come to a word they don't know, they
26 have a whole bank of words to try to match that up with and to associate it with.
170
171
172
1 So, the more words they know, the more likely it is that they're going to
4 instruction in vocabulary is where the teacher introduces the word, discusses it,
5 talks about it, lets kids write in sentences, work with it. Teachers can typically
6 cover about 8-10 words a week in that method. That's not very many words when
7 you think about how many new words a child is confronted with every week.
9 through listening to adults read and talk and through reading on their own.
11 out unknown words. No. Beany Babies are ubiquitous. Could mean beautiful,
13 (Laughter.)
15 They need to know about dictionary skills and reference aids, and they need to
18 prefixes, suffixes, roots. All of those strategies help them deal with unknown
19 words.
21 really about three levels of word knowing that we talk about. There are unknown
22 words. There are words that you're acquainted with. You sort of know what they
23 mean. "He went down to the cay to watch the boats." Well, I sort of know that's
25 And, then, there are established words that we really know well.
26 They are our old friends. We know their multiple meanings. We know how they
173
174
175
1 are used. We know the affect that they convey. Those are words that are
4 Obviously not, if a teacher can only cover 8-10 words a week, well, direct
5 instruction of vocabulary words is not going to be the best and only way to go.
7 Text comprehension, that's where we want to get kids. The other things are means
8 to an end. They are contributing factors. But we always need to remember, our
9 final goal is to get kids who are purposeful and active readers, and all five of the
11 getting there.
13 active when they read. They read for a purpose and they're always thinking and
14 working through the text. Their brains are very active while they are reading.
15 There are six strategies that research has shown us that improve
17 comprehension; teaching them to use graphic and semantic organizers which are
18 maps; sort of organizational pictures of the text content; being able to answer
19 questions about what you've read; being able to generate questions about what
20 you've read; being able to recognize the story structure --Is it narrative, Is it
22 All of those things are aids to being able to understand the text
25 strategy, modeling it for the child, giving the child guides to practice with the
26 strategy, giving kids repeated opportunities to apply and use the strategy. These
176
177
178
1 are all effective techniques for teaching kids strategies to use when they are
4 the basics to teach comprehension. No, comprehension begins at the get go. We
5 begin with listening and story comprehension, and as soon as they begin to read,
7 fluent.
13 We need to encourage research that focuses on finding out more about the reading
20 to teach reading, but those principals in those early elementary buildings need to
21 know about early reading instruction. They really need to be effective leaders. If
22 they are going to be effective leaders of reading instruction, they have to know it.
23 They need their own professional development. They're not the same as teachers.
25 reflect what we know about early reading instruction. Teachers and principals
26 need professional development around how to collect and use this data to inform
179
180
181
1 instruction.
3 about putting out there, we need to support and encourage teachers' use of
6 reading instruction, and we need to remember that the goal is fluent readers.
8 If any of you are standing up in front of a roomful of teachers, they are only going
9 to see you once in their lives. Why should they trust you? They don't trust you.
11 But, if we hit them again and again with the same message, it's a
12 consistent message, it comes from all of our organizations, it comes from the Hill,
13 the consistency proxies for trust, and they begin to listen to us and change and
15 (Applause.)
18 (Laughter.)
22 and violence prevention research is somewhere between the depressed scale of the
25 some things.
182
183
184
1 field. The first is what's been going on in the Department of Education under the
2 Safe and Drug Free Schools, or as it started out, The Drug Free Schools and
3 Communities Act.
5 primarily at least I find has been in the way of helping us to understand and know
8 working on back in 1998 through '91 that looked at the initial implementation of
9 the Act.
10 Then there was a longitudinal study that followed from that and
11 used some of the information form the descriptive study to select a group of school
12 districts that we then looked at longitudinally and drew relationships between the
13 kinds of programs that they were implementing and the outcomes for students.
17 extensive and well implemented programs and the less extensive and less well
18 implemented programs were small. They were significant but they were small.
20 is that very few of the school districts and schools were implementing models that
21 we were then coming to understand that there was a research base growing to
22 support specific models of prevention education, and very few of those were being
25 coordinator rather than someone who shared that role with five or six other roles
26 in their district, those districts with the full-time prevention coordinator had better
185
186
187
4 department, one that focused on school violence, another that looked again at
7 safe and drug-free schools funded programs and the Middle Schools Coordinator
8 Initiative where funding has been provided to actually have full-time coordinators
10 At the same time and sort of outside this realm of studies that
11 focussed just on safe and drug-free schools, is a growing body of literature and
12 findings to support specific ways of going about more often drug prevention
13 education, but also violence prevention education, and I must say that they overlap
14 a great deal because a lot of the risk factors in youth and in their communities are
15 very similar.
18 through those studies to make recommendations about which appear to be the best
21 of Education has had a panel to look at these and come up with exemplary and
22 promising programs. The Center for Substance Abuse Prevention has done so. An
24 rankings of prevention strategies. So, we have some specific curricula that can be
25 recommended.
188
189
190
1 these research studies and have isolated certain content that they believe is the
2 most effective parts of these curricula and also instructional strategies that seem to
3 be common to the most effective strategies and absent in the least effective
4 strategies.
5 So, unlike the discussion that we just had in math and reading,
6 I'm not going to go through the research and tell you what those strategies are, my
7 main point here is that there are some established pieces of research and some
10 doing away with drugs and violence among our youth? No, so we haven't reached
13 about 1998, and now are re-emphasized and expanded on in the No Child Left
14 Behind legislation telling school districts and schools to implement research based
15 programs, there are some places that they can turn to find out what those are and
16 figure out what would be best used in their own schools and school districts.
18 from the studies we've done of what's going on, is that these research based
19 programs are not widely implemented. We find very few districts and schools
22 Department, and another one done by Chris Ringwald, Susan Annid and myself
23 and others in North Carolina but looking at a national sample of schools and
24 districts found that few are really looking at -- about 25 percent, I think, were
191
192
193
1 But when you look at the content and delivery, things that have
3 reported that they were delivering the content that meta-analysis said was
4 important, but not very many of them are using the teaching strategies that the
6 Now, why is this happening? Well, one is that there is not a big
7 transfer of knowledge from the research community to the schools. Another is, I
8 think, a lack of money to do this. I don't know how well the research supported
9 curricula can be implemented on the amount of funding that they get from Safe
10 and Drug Free Schools, which is about seven dollars per child, or could be reduced
11 to around $3.50 if they decide to divert those funds for other purposes. So, they
14 time in class. The schools are under tremendous pressure to meet standards in
15 academic areas. Unless they see and strongly believe in a link between the
16 behavior and health of their students and those academic achievement areas, then
17 it's really tough to make the pitch for a lot of time being spent in the classroom or
20 Principles of Effectiveness have been out in the field for a few years and are
21 strongly reinforced by the new legislation, and I think that that as it keeps being
26 district who has time to really focus on these issues and figure out what strategies
194
195
196
3 research, we face a number of challenges. We've heard about the kinds of designs
4 and methods that ought to be used in school based research. I believe that
5 experimental and quasi-experimental designs can be used. But they require very
6 careful planning. They require large numbers of schools. They require enough
7 time up front to really get your ducks in a row, get your entities selected. If it's
8 going to be schools that you randomize, that can't be done sort of after the fact,
9 after some schools have gotten funding to do something and go hunting around for
10 maybe some comparable schools to compare them to. It takes a very concerted,
15 I think this is what one of the earlier speakers was talking about
16 in terms of field studies. Take the approaches that are research based or found in
19 Most of the research that we're basing all of our actions on was
22 national evaluation of the entire Safe and Drug Free Schools program. I could do
23 a very long presentation on why I think that, but I think I will move on.
197
198
199
1 hard to respond to, to try to pull all these issues in a field together and get them
2 delivered.
5 all of the research that we've been talking about? There is not an infinite number of
6 schools out there. Many of them are already engaged in specific research
7 activities.
8 And, if they are not involved in a study of a particular intervention, they've been
9 survey twelve times in the last year. It is tough to talk about this kind of research
10 and then think about -- if you're in a school district or a school, you know how
11 many times you've been asked lately to participate in studies. And you often have
12 to turn them down because you just don't have the time available to do it.
14 have reached the point where we have some evidence to go on and some models
15 to try out in a field based setting. And I think we can use experimental designs for
18 have the time in advance to plan it and if we have strong support from the
19 administrations of those schools. One of the challenges that I sort of skipped over
20 is sort of the whole logic model of what is the intervention, how can you tell when
21 it's well implemented, and how do you measure the outcomes? Measuring the
22 outcomes in this area is tough. I mean, I hesitate to say this when we heard how
23 depressing things were in math, but I don't think the challenges are quite equal
24 across all of the fields in terms of research. You know, not being a math researcher
26 (Laughter.)
200
201
202
2 in math. Driving violence prevention, we're looking at stuff we can't even see.
3 We're not supposed to see in the classroom. We want to know what those kids are
4 doing when they're not in the classroom. How do we find that out? Well,
5 probably the best way we've come up with so far besides urine tests is surveys.
6 And surveys, well, all the schools are over-surveyed to start with, but secondly,
7 we're facing the Grassley Amendment which tells us not to survey students on
8 sensitive behaviors, which illegal behaviors like drug use and violence are, unless
9 we have explicit criminal signed consent. That just adds a further difficulty to the
10 research there.
12 are the proven approaches affordable and effective in the real world, what new
13 approaches are effective, and don't forget the non-classroom activities. Most of
14 the research that I'm aware of at any rate deals with curriculum. And as I've said
15 before in that longitudinal study, we've got a pretty good sense that the non-
16 classroom activities were important as well. And by that I mean things that happen
18 programs, other kinds of things that happen in schools or around school time that
21 look at those targeted studies of approaches, but also to continue to monitor the
22 implementation of research based programs in the school setting. So, I see a really
24 schools and school districts about the specific models they are implementing to
25 find out if in fact that transfer is happening and to somehow help that happen.
26 Thank you.
203
204
205
1 (Applause.)
3 but I think you probably heard a startling statistic in that last presentation, which is
4 25 percent of all of the programs in Safe and Drug Free are research based. So, it
8 Research Analyst from AEIR talking about comprehensive school reform. Becki?
9 MS. BECKI HERMAN: Well, thank you very much for the
10 opportunity to come here and talk with you about scientifically based research and
13 overview of the research on CSR, and I won't delve too much into the actual
14 findings that really focus on the quality of the methods in the research. And talk
15 about what it means to apply the definition of scientifically based research to the
17 suggest some possible effects of using this view of research standards on the
18 CSRD program.
22 unifying theme. It should be touching all grades and key subjects, English and
23 math for starters, and it should touch all aspects of the school, and this is a key
206
207
208
2 universities and private organizations have developed models that can be selected
3 by schools and adopted by schools. But CSR is not just models. CSR can involve
4 schools developing their own approach where they're thinking of how they're
5 going to revise and revamp their instruction and curriculum and their management
6 around this unifying theme, or if they chose to adopt a model, it might be adopting
7 a model and working with other separate practices that they want implement in
8 conjunction with this model, they all fall under this unifying theme.
11 Demonstration program. It's not the only support, but it's one of the biggest.
13 outcomes research focuses on models and so that's really what I'm going to focus
14 on when I talk about the research but I want to remind you not to lose sight of the
15 fact that models are only part of the story. There's a missing part of the story that's
16 not necessarily being told because the research is a little weak there.
18 the Educators Guide to School Reform which profiled and reviewed the research
21 found 130 outcome studies, and we set some limits for what we called an outcome
25 methodology. We used criteria such as what I have listed there under study
209
210
211
1 experimental? Did they use controls? What kind of construct, internal, external,
2 validity evidence was there? What's the duration of the study? Was it
3 longitudinal? What about the sample? The size of the sample, attrition, those sorts
4 of issues. And measures? Independent and are they well-respected, high quality
8 met the gold standard which is true random assignment and also strong in all these
14 methods.
15 As Lisa and Valerie have pointed out before, the quality of the
16 research base overall matters. It's not just the methodology used in the
17 independent studies, but it's a replication of findings. It's that all of the research
18 converges in a certain direction and points a way to a finding that can be useful to
19 schools.
20 We found that there were very few models that had more than
21 ten strong outcome studies and no models had absolutely consistent findings.
22 There was always a school or a grade or a set of students that didn't do well with a
23 certain approach. We were unable to come up with conclusive findings that said
25 But we were able to find that the bulk of the research, limited
26 though it was, pushed in certain directions and that there were some models that
212
213
214
3 when you don't have a lot of gold standard studies, when you don't have a lot of
4 random assignment studies because if you have hundreds of studies that are quasi-
5 experimental study and no random assignment study, you might want to put some
8 there is very little CSR outcomes research that's not focused on models. OERI is
9 currently sponsoring a set of studies that look at some of the issues that transcend
10 models. They look at models and the study says well, but there are some issues
11 that are greater. For example, some of these studies are looking at what is the
12 impact of comprehensiveness? Is the whole greater than the sum of the parts?
13 Does a comprehensive reform work better than a set of discrete reforms within a
14 school? Or some of the studies together are looking at the relative effectiveness of
15 different approaches to CSR and some of the factors that help explain the
16 variation.
17 In the last few years, there has been a marked increase in the
19 The two Cook studies that studies that Steve Raudenbush mentioned earlier, a
20 Success for All study that Steve Raudenbush described is actually one of the OERI
21 funded studies where they're using random assignment and the issues that they're
22 running into in conducting the study are too numerous to mention. But suffice it
23 to say that they're committed to doing it and they've worked out a strategy for
24 doing it, but there are real world issues with trying to do this.
26 the research on CSR, I'd like to turn to the circumstances under which the
215
216
217
2 from Baruch's chapter in an in-press book, Evidence Matters, for these five criteria
3 for when you would apply the standard of -- for him he was saying random
6 solution is unproven, other study designs will not provide satisfactory results, the
7 results will inform policy decisions and the rights of participants can be protected.
8 Three of these criteria are easily met for CSRD: the problem is
9 serious and the solution has not been unequivocally proven, although there's some
10 evidence moving in some directions. And the results will probably inform such
11 policy decisions.
12 However, the third criteria (that other studies will not provide
13 satisfactory results) well, that depends on the question you're asking as almost
14 every speaker today has said. If the research question is outcomes, does CSR
15 improve student achievement, a causal question, yes, you'll get more defensible
16 results using scientifically based research than using say case studies or some
18 implementation, well, scientifically based research is not necessarily the only or the
19 best strategy but certainly is part of the strategy for answering that question. But
20 case studies can provide some very good information on what are issues with
23 based research to CSRD, is that the rights of participants can be protected. In this
24 high stakes, outcome oriented environment for reforming schools that's a difficult
26 approach that does not seem to be working when they are under incredible
218
219
220
1 pressure to produce results quickly for the duration of the study that you need to
3 (Laughter.)
4 It's also difficult -- and this is a problem with some of the CSR
5 studies that are trying to use random assignment, there's the problem of getting and
7 guarantee that there's no slippage that they don't go ahead and adopt either exactly
8 the condition you were testing or a competing condition, but, in other words,
11 Comprehensive School Reform approach for the duration of a study, but there are
14 looking for causal effects and where you're able to protect the rights of
19 based research in the legislation. The first component which is "proven methods
20 and strategies are based on scientifically based research" means the strategy for
24 comprehensive, which are less testable within experimental design. They are more
25 about the development and the implementation and they are different sorts of
26 issues.
221
222
223
2 significant improvements in academic achievement," the idea that the practices that
3 you're using in your CSR program work and they work as a set collectively. That
4 idea is also held to the standard of requiring evidence from scientifically based
6 I was talking to a few people before starting and some said that
7 they were curious about what I was going to say and I said one of the first things I
8 want to say is I'm not a soothsayer. I can't tell you how this new definition of
9 scientifically based research will effect the program. But, I can make some
10 suggestions of possible effects and I'd be interested to see what actually pans out.
12 the CSRD, the expectation that CSR programs use proven practices, one of the
15 might have a practice like parent involvement. You might have a set of practices
16 around curriculum. You might have a set of practices around instruction and a set
19 out there and do the research on them. There's no single source that says this is
20 the best way to go about instruction or this is the only effective curriculum. So, a
21 school that's thinking about adopting CSR needs to be able to investigate all these
24 know that there's been mention already of the What Works Clearinghouse which
25 will hopefully be able to provide some support for schools in this area. There are
26 organizations, the Department of Education is not the least, that provide a lot of
224
225
226
1 information to help schools look at the research. But, it's still very modest. That
2 might deter some schools that are considering applying for CSRD grants if they're
5 positive effect of this new definition of scientifically based research that focuses on
6 the practices. Schools will be looking at the practices within the model, not just
7 the model. They might be able to see whether there is evidence for all of the
9 practices, to see whether they think that this is the right approach for them and that
12 comprehensive, whether there might not be some practices that are not part of the
13 model, say parent involvement, that they might want to investigate themselves.
16 series of practices.
18 them to rethink how they are using models and what practices they would like to
22 of research for them to look at whether something works. That is, provided, that,
23 as I said, they have the resources to help them collect the research and have the
26 detrimental effect on externally developed CSR models which at this point is one
227
228
229
2 There are a lot of different models, some that are more mature.
3 They are in a lot of schools and have a strong research base. And then there are
4 some that are smaller. They're newer. They aren't in a lot of schools. There's not a
6 With this kind of selection, schools can find a good fit for their
7 own situation. They can find models developed around a theme that works for
8 them. They can find a model that has a series of, a set of effective practices that
9 they believe are right for their own strengths and weaknesses.
11 scientifically based research. If they are in few schools and they have not had time
12 to develop a strong research base, this might prune the field. If you hold new
14 that you only have one or two big approaches that are mature for schools to be
17 where you hold a different standard to the more mature models than to the newer
19 demonstration approaches differently from the more mature models in some way.
22 schools. I think I've mentioned this several times, researchers are trying to make
23 decisions and they're held to the requirement that these decisions need to have
24 some scientific evidence. So, one of the biggest movements I could see is
25 providing more support for helping schools access the research and for helping
26 them understand and discern between the various levels of research and quality of
230
231
232
1 research.
2 (Applause.)
6 based evidence is truly a fascinating one. I was fascinated to see how many of you
7 all stayed throughout the discussions, as well as the wonderful papers. I had read
8 every one of these papers prior to today, and yet, I found the delivery of those
9 papers still fascinating. The issues you raise are just really important.
12 I'm sure people are willing to stick around a little bit after.
14 today. They will be up on the web and please feel free to contact me or these
15 wonderful speakers.
17 (Applause.)
19 a.m.)
20
21
22
23
24
25
26
233
234
235
236
237