Вы находитесь на странице: 1из 23


A Dialogical Turn to Evaluation and Learning

Theo J. H. Niessen, Tineke A. Abma, Guy A. M. Widdershoven,

and Cees van der Vleuten

Introduction understands learning as an a priori planned and

individual cognitive act.
In this chapter, we explore a case in which a problem- In this chapter, we argue that learning is far
based learning (PBL) course at Maastricht more; it is a complex, contextually sensitive, and
University is responsively evaluated. Within dialogical process that situates itself within the
responsive evaluation, the value of an educational interstices between people being attentive and
practice or educational regime is dependent on mindful about what is said on the spot during
the multiple, sometimes conflicting, perspectives the evaluative process. Learning is not a planned,
of stakeholders in the evaluation setting. linear, and individual cognitive process; rather, it
A common presumption about evaluation is is a dialogical accomplishment attuning oneself
that it contributes to the knowledge of program, to what the other is bringing up. In this case, rec-
participants, and other stakeholders and that ognizing, acknowledging, and playing with this
they will learn from an evaluation. Yet what it plurality was a rewarding process for all partici-
means to learn from an evaluation and what we pants. We illustrate these narrative and collective
assume about learning, knowing, and resistance processes of learning drawing on our evaluation
are underdeveloped concerns in evaluation of the course on PBL for newly appointed staff
practice (Schwandt, 2004). A traditional model members at Maastricht University. PBL as used
of learning from evaluation assumes that learn- within Maastricht University is a didactical
ing occurs when evaluators transmit findings approach. Within PBL, students collaboratively
and conclusions to program participants and within small groups engage in learning tasks to
stakeholders and that they will then process and obtain the necessary knowledge and communi-
absorb that information. This transmissional cation skills. For teachers, this also assumes
and linear view on information “processing” other, more coaching-related skills. They should

378——PART IV Educational Evaluation as Learning and Discovery

refrain from frontal instructions. New staff premises. Then we present the case example. This
members coming to Maastricht University are is followed by a section on the issue of learning
often not or insufficiently acquainted with these within and from evaluation. Finally, we deal with
principles of PBL. Within “The introductory the issue of how local case knowledge provided
course on PBL for new staff members,” newly within this chapter can be of global importance.
appointed teachers are trained in the basics of
PBL. It is this course—as part of a PhD study on
teacher learning that was finished in 2007 Responsive Evaluation
(Niessen, 2007)—that we have evaluated
through means of responsive evaluation. Guba and Lincoln (1989) have depicted the
Responsive evaluation is an approach in development of evaluative research into four
which the value of a practice or program is historical generations or orientations, denoting
established by focusing on the perspectives of them as the measurement, description, judgment,
and deliberative dialogues between stakeholders and negotiation generation. The measurement
in the evaluation setting. Evaluation is thus orientation is about the collection of data as, for
defined as a dialogue or conversation with all example, the measurement of student outcomes
stakeholders about the value and meaning of a as a denominator for educational success.
particular program or practice as a vehicle for Description entails pointing out the characteris-
learning, understanding, and improvement tics to an educational policy or program. In the
(Abma, 2005; Abma & Widdershoven, 2005). case of an educational course, this means
Worldwide, governments, schools, and other describing the activities taken by teachers, stu-
organizations are under increasing pressure to dents, and the interactions within the classroom.
deliver tangible results and to compete for the This information may be supported by descrip-
best performance (Greene, 1999; Stronach, tive statistics. The generation phrased as judg-
1999). These, however, are measured against ment will pass a final ruling about the quality,
standardized performance indicators often for example, by comparing the factual effects of
defined by one stakeholder group. Against the an educational program with the formed goals
background of results-oriented management— and standards. A fixed set of standards and cri-
assessing efficiency and effectiveness—one teria are used, determined by one stakeholder
becomes aware of the paradoxical nature of per- group (policymakers). The assessment is con-
formance indicators: On the one hand, they centrated on the realized outcomes (and less or
are “frozen ambitions,” yet on the other hand, not on the process of implementation). This
they must facilitate dialogue and learning type of evaluation—whereas it may be called
(Noordegraaf & Abma, 2003; Van der Knaap, performance measurement or standard-based
2006). We would even state that these ambitions evaluation—is prevalent within contemporary
and indicators obtain their meaning within dia- society. It is policy-centered and stands in con-
logue. Building on a responsive approach as a trast to the pluralistic, interactive, and dialogical
dynamic perspective on evaluation, we are able character of fourth-generation evaluation (nego-
to question the validity of the policy objectives tiation generation). Within fourth-generation
and performance indicators without compro- evaluation, the design emerges on the basis of a
mising their value and significance altogether. conversation with and among stakeholders and
The “dialogical turn” in evaluation helps us to their issues of concern. The deliberative dia-
raise the quality of dialogue, learning, and deci- logues between stakeholders evoke a collective
sion making among all stakeholders. process of social learning and enhance the per-
Within this chapter, the reader first gets sonal and mutual understandings. This stands in
acquainted with responsive evaluation and its contrast to the idea of learning within a standards-
development, core concepts, and worldview based evaluation approach, where learning

starts after the evaluation on the basis of the kinds of data: goals and outcomes. Stake wanted
application of evaluation findings. Table 21.1 to make responsive evaluation meaningful to all
summarizes the main differences between the different stakeholders experiences (Abma &
responsive and standard-based evaluation. Stake, 2001). According to Stake, there was a
need for contextualized data, which acknowl-
edged the situated and cultural nature of
Strands Within instructional programs. To Stake, this entailed
Responsive Evaluation an approach that was responsive to the multiple
perspectives and values that any program or
Before their seminal work in which Guba and practice incorporates. Being responsive meant
Lincoln (1989) proclaimed the need for a more for him that a researcher should take into
democratic and participative form of evalua- account the multiple perspectives in a manner
tion, Robert Stake articulated his dismay about that was as truthful as possible to the intentions
the narrow selection of data being used for for- and values of each stakeholder. Providing emic
mal evaluation within his “countenance paper.” tales and using thick descriptions, these cases
Responsive evaluation, a vision and rationale for were portrayed as holistic.
evaluation within education, can be traced back In line with Lincoln (2003), Abma and
to Stake (1975; see also Stake & Abma, 2005), Widdershoven (2005; Abma 2000) elaborated the
who developed it in the mid-1970s as an alter- responsive approach extending the ideas of Stake
native to “preordinate evaluation.” This he saw and Lincoln within a social-critical frame. Within
as the dominant approach emphasizing strong such a framework, evaluation should actively
(preferably experimental and quantitative) mea- steer toward the inclusion of marginalized voices.
surement procedures and only legitimizing two A measurement orientation is biased because

Table 21.1 Differences Between Responsive and Standard-Based Evaluation

Standards-Based Approach Responsive Evaluation

Evaluation criteria • A priori set by one stakeholder • Emergent to deliberative

and standards group dialogue between stakeholders
• Policy-centered • Pluralistic: the values and
interests of all stakeholders
• Effects and outcomes • Effects and process of

Evaluation process • Fixed hypothetical-deductive design • Emergent design based on

• Stakeholders are information-givers stakeholder issues
• Monological • Stakeholders are partners in
• Evaluator as expert the process
• Interactive and dialogical
• Evaluator as facilitator

Learning • Individual/cognitive phenomenon • Collective/social phenomenon

• Starts after the evaluation with the • Begins during the evaluation
application of data process
• Enhanced knowledge “about” an • Personal and mutual
evaluand understanding
380——PART IV Educational Evaluation as Learning and Discovery

only the intentions from the management are Widdershoven, a responsive evaluator should
used to judge the worth of a program or practice. be a sound anthropologist, as well as a caring
Participants are solely taken serious for the infor- researcher trying to actively attend marginalized
mation they are able to provide. They are voices. This is what Heron and Reason (1997)
approached instrumentally. This does not lead to call the axiological dimension within research
acceptance and implementation of evaluation providing human flourishing through means of
data and results. Within responsive evaluation, action research (action turn). This axiological
deliberative dialogue striving toward consensus is dimension is missing from the constructivist
pertinent. In case of an education program, perspective of Stake as well as Guba and Lincoln.
teachers, students, but also parents and directly Within Stake’s view, but also that of Guba and
linked other partners may enter a mutual dia- Lincoln, the researcher responsible for respon-
logue about the value of an education practice. sive evaluation is framed as a distant interpreter,
whereas within the account of Abma and
Widdershoven, the responsive evaluator is more
Worldview Premises of a “passionate participant as facilitator of a
and Core Concepts multivoice reconstruction” (Guba & Lincoln
2005, p. 196), and the purpose is to attend to
We have argued elsewhere (Niessen,Vermunt, those who are in danger of being ignored.
Abma, Widdershoven, & Vleuten, 2004) that core A seemingly small difference between Guba
theoretical concepts are grounded within more and Lincoln’s approach and that of Abma and
encompassing worldviews. A worldview might Widdershoven resides in the term used to
be denoted as a more or less coherent outlook on denote the exchange between the participants
how we view reality (ontology), what can be within responsive evaluation. Guba and
known about it (epistemology), how to acquaint Lincoln—in line with the fourth depicted devel-
ourselves with it (methodology), and how we opmental stage within evaluation—speak about
should deal with it all (axiology; Heron & negotiation. Abma and Widdershoven (2005)
Reason, 1997; Niessen et al., 2004). Placing respon- refer to it as deliberative dialogue evaluating the
sive evaluation within this framework, we can worth of a program:
elaborate the differences among Stake’s, Guba
and Lincoln’s, and Abma and Widdershoven’s Deliberation refers to the interaction and
interpretation of responsive evaluation more dialogue between participants. They do not
clearly. Although the differences regard various just accept each other’s beliefs and persua-
issues, we confine ourselves to the researcher’s sions, but will explore these. Listening,
role and the implied worldview. probing and dialogue characterise this
Stake (1986) characterizes the researcher’s process, rather than confronting, attacking
role within responsive evaluation as that of an and defending. Central feature of dialogue
anthropologist describing or portraying stake- are openness, respect, inclusion and
holders’ interests as accurately as possible. This engagement. (p. 280)
could be done by means of using thick descrip-
tions (Abma & Stake, 2001). Although Abma For us, the term negotiation is placed within a
and Widdershoven (2005) agree on the assump- more competitive interpretational framework in
tion that the value of a program is not being which formality, not inclusion, is paramount.
established a priori, but rather accomplished in To provide more texture to these terms, we
conversation by the different stakeholders, the also could look at the basic philosophies that are
role of the evaluator as well as the ontological informative to the research paradigm and schol-
postures differ. According to Abma and ars’ worldviews. Abma and Widdershoven’s

(2005) account of responsive evaluation is by Heron and Reason (1997) as: “This form of
grounded within narrative psychology and dia- radical empiricism is not to be confused with
logical ethics (Abma, 1999; Abma, Molewijk, & behaviourism, which has never been empirical
Widdershoven, 2007; Widdershoven & Sohl, enough, since it preconceives and delimits expe-
1999). Narrative psychology helps to understand rience in terms of its positivist paradigm. On the
how people make sense of and give meaning to contrary our empiricism is the radical sort long
their own identity and life context. Stories help since commended by phenomenologists: a pris-
to endow meaning to situations, weave events tine acquaintance with phenomena unadulter-
into a meaningful whole, and relate varied ele- ated by preconceptions (Spiegelberg, 1960)”
ments into a plotline (Josselson & Lieblich, (p. 2). Schwandt (2003) has called it an engage-
1999). A narrative is always context-bound; it ment with the “rough ground.” He explains:
positions a character in a specific time and place. “Rather than first thinking ‘scientifically’and
A story describes a specific situation and enables ‘theoretically’ about ourselves, others and our
the narrator to find guidelines for action and to world, we begin with our being in the world and
influence others to adjust their actions. Stories the ways we interweave our talk and action to
have a performative character. Stories are appro- develop and sustain ways of connecting with and
priate to make sense of situations because they relating to one another” (p. 355). What we infer
acknowledge particulars (Josselson & Lieblich, from these segments and what is central within
1999; Lyons & LaBoskey, 2002). enactivism is that being is grounded, first and
In recent work, Niessen, Abma, Widder- foremost, in a nonpropositional resonating “feel-
shoven, Akkerman, and van der Vleuten (2008) ing” of interconnectedness.
place responsive evaluation within an enactivist Moreover, from this inherent relational
framework, thus changing and/or acknowledg- ontology enactivism as well as the participatory
ing stakeholder participation as an ontological research paradigm explicitly and naturally fore-
necessity rather than an epistemological issue. ground the axiological dimension to research.
Responsive evaluation is then placed in what The axiological question deals with the issue of
Heron and Reason (1997) would refer to as the what is intrinsically worthwhile. Within evalua-
participatory research paradigm: “Co-researchers tive research, this “ethical” turn can be found in
are initiated into the inquiry process by facilita- the work of Whitmore (1994) and Greene
tor/researcher and learn through active engage- (1997). Whitmore (1994) is grounded in the
ment in the process; facilitator/researcher empowerment tradition and focusing on the
requires emotional competence, democratic per- participation of unheard voices providing them
sonality and skills” (p. 290). a voice and control to shape their own destiny,
Enactivism and the participatory paradigm as whereas Greene (1997)—also wanting to pro-
exemplified by Heron and Reason (1997) share vide stakeholders with a voice—departs more
many characteristics. Within enactivism, the cen- from a democratic tradition in which delibera-
tral point is that life in all its forms (e.g., educa- tion and dialogue are the means to accomplish
tion, nursing, etc.) is actively enacted as we this task. Cousins and Earl (1995) are most
engage in it. To enact means to bodily work on pragmatic (i.e., instrumental in their axiological
and experientially engage in a preexisting world quest), stating that participation within evalua-
that already has meaning. The world is conceived tive research will stimulate stakeholders to use
as dialogical (e.g., “who we are” emerges in our the information that is gathered. There is no ref-
moment-to-moment coping with the contingen- erence to an ethical commitment to involve
cies of our existence). This continuous coping stakeholders a priori. The participatory frame-
with our existence (bodily and experientially) work as we have appropriated it within
requires a radical empirical turn, described our research has most commonalities with the
382——PART IV Educational Evaluation as Learning and Discovery

work of Greene (1997). While denoting this University had changed considerably over the
resemblance, we acknowledge that at the final last 30 years. In some instances, students study
stretch what is intrinsically valuable in our eyes less. Their preparation time to the PBL sessions
(our axiological aim) is human flourishment, has dropped increasingly. PBL is structured
meaning balancing autonomy, cooperation, along a stepwise frame. Students tend to skip
and hierarchy in our culture in general and certain steps. Staff members also altered the PBL
within research. This ideal of human flourishing model. Instead of facilitating the PBL process,
and research initiatives to practice this ideal are they tend to revert in old frontal modes of teaching.
visible in the work of Titchen and McCormack Research demonstrated that these changes crept
(2008). in coincidently or were sometimes implemented
Although not explicitly based on enactivism by faculty officials explicitly (Moust, Berkel, &
or the participatory paradigm, the work of Schmidt, 2005). Whether this was desirable was
Abma has evolved over time into this direction. subject of debates among curriculum developers
The shared responsibility within her interpreta- and theoreticians. We hoped that the evaluation
tion of responsive evaluation searching for a of the PBL course would create a social learning
minimal shared experiential (practical) base platform to also include the voices of the teach-
using analytical and nonanalytical procedures is ers and teacher-trainers, and their voices and
a reflection of this turn (Abma & Broerse, 2007). experiences with PBL.
In our context of the PBL case, we included
new staff members (junior teachers), the course
Case Example: Responsive developers, and teacher-trainers in dialogues
Evaluation of a PBL Course about their experiences with PBL. We did not
include the students, although we understand
Within Maastricht University, PBL has been the them to be the real end users. The reason to not
prime instructional method to introduce uni- include the students could be contested certainly
versity students for the last 30 years. PBL as used given the shared and distributed nature and
within Maastricht University is a didactical meaning of PBL. We chose not to do so because
approach. Within PBL, students collaboratively the nature of the project focused on teacher
engage within small groups in learning tasks to learning. We did not evaluate the PBL cur-
obtain the necessary knowledge and communi- riculum, but a training course for (new) staff
cation skills. Teachers should refrain from members (teachers) at Maastricht University.
frontal instructions and are expected to acquire Students could well have made remarks about
other, more coaching-related skills. New staff the PBL in general, but not about the PBL course
members coming to Maastricht University are because they did not attend the training. The
often not or insufficiently acquainted to these new staff members, often junior teachers follow-
principles of PBL. Within “the introductory ing the course, can be considered students in a
course on PBL for new staff members,” newly way. They were trained by a group of teacher-
appointed teachers are trained in the basics of trainers in the educational department of the
PBL. It is this course we have evaluated through university. The relation between the trainers and
means of responsive evaluation. junior teachers was asymmetrical; the trainers
At the time of the evaluation of the PBL were educational experts, whereas the teachers
course, a discussion took place at the university lacked formal educational expertise. Most of the
about the nature of PBL and its future. Several new teachers were quite young and inexperi-
curriculum developers and academics noticed enced. Although the teachers did not have to do
that the PBL curriculum at Maastricht an exam, they were assessed by the trainers.

Given this asymmetry, we paid deliberate atten- evaluation study, we used the term thick
tion to the stories of the teachers and amplified stories—in line with Geertz’s (1973) phrase thick
their voices. descriptions—to denote the dense texture of
these experiences. Roth, Lawless, and Tobin
(2000) refer to these experiences and stories as
Enhancing Personal Understandings the lived or enacted curriculum, as opposed to
Through Learning Histories the theoretical curriculum.
During the first phase of our study, we com-
We have characterized responsive evaluation pleted 19 individual conversational interviews
as a dialogical approach for evaluating programs with participants from the relevant stakeholder
or practices. A full-fledged responsive evaluation groups: 10 teacher-trainers and 9 former partici-
involves a cyclical sequence of individual and pants to the introductory course (new staff
group interviews with and among different members/teachers). The maximum variation sam-
stakeholders. First, we started to identify stake- pling technique (Patton, 1990) was used to attain
holder issues and concerns. To the introductory the participants. These differed with regards to fac-
course on PBL, the relevant stakeholder groups ulty affiliation: Medicine (4), Law (4), Economics
were the former participants of the introductory (3), Health Sciences (4), Arts & Culture (3), and
course (new staff members at Maastricht University College (1). In all interviews, we had
University), the teacher-trainers (providing the we started off with a general question (“What
course), and the course developers (which were stood out for you having participated in the
in most cases also the teacher-trainers). Introductory PBL course?”); through subsequent
Stakeholder issues and concerns are not probing, the contextual, embodied, and personal
given, but are to be developed in conversation nature became heightened.
with the stakeholder groups. The design is emer- Box 21.1 provides an example of the type of
gent (as opposed to a preordinate design) to issues identified in the individual interviews.
attend to the issues evolving in the process. To The example is drawn from an interview with
delineate stakeholder issues and underlying Josie. She is a junior teacher-trainer at the
meanings, several methods can be used; in this Faculty of Economics and Business Admini-
case, we held open, conversational interviews. stration. As a former student at Maastricht
These are appropriate to gain an insight into the University, she has had experience with the PBL
experiences of respondents from an insider’s system. Working as a teacher-trainer helping
perspective and understand the meanings the new staff members to get acquainted with the
respondents endow to their teaching experi- PBL system is another topic altogether. As a
ences. Within conversational interviews, partici- trainer, Josie tries to “get the group exited about
pants are invited to talk about the topic being PBL.” However, sometimes this appears to be
evaluated—in this case, the PBL course, sharing difficult. Josie relates about their experiences
experiences and stories. The interviews and with PBL in the context of a group of experi-
focus groups were entirely transcribed by the enced students. It did not go very well in her
researcher (Niessen). The individual interviews opinion; with all the information coming from
were held at a place most appropriate and com- the group members, she had not been able to
fortable to the respondent. In many cases, the provide the students with a closure. This experi-
conversations took place at the university, ence triggered her to rethink and reflect on her
although we also organized one interview at a ideas about PBL and whether she had been a
person’s home and one in a quiet café. The focus good teacher in that situation. Box 21.1 depicts
groups were held at the university. Within our Josie’s story.
384——PART IV Educational Evaluation as Learning and Discovery

Box 21.1 Josie’s Case

Josie: In a training session last week, there was a group of student tutors, and
this group was really very critical because they had attended other PBL
courses. That was when I found myself trying to create more structure—
that’s where I felt inadequate because there were so many people with so
much experience. In these instances, it’s important to offer students guide-
lines and structure. You should be able to deviate from this structure—but
only in those cases—when it’s possible. Some teachers see this very clearly.
Personally, I tend to create structure together with the group—on the spot.
With some groups this works out just fine, and with other groups it would
have been better if I had provided a clear structure from the start. We
would have come further.

Researcher: Students get restless?

Josie: No, yes, well, there’s too much input and too few conclusions. I think that’s
a major thing in PBL—it’s a major issue that too often, maybe, no actual
conclusion is reached. That’s really what I think is probably my own short-
coming, something that as a student I thought was missing in the system.
That structure—the framework in which you work.

Researcher: What does this framework look like—what is it made of?

Josie: A connecting thread.

Researcher: You say that, on the one hand, you’re trying to find this thread—and you
want to connect it with the experiences of the participants—but that’s dif-
ficult because their experiences are so diverse and a common theme is hard
to discern.

Josie: Well, maybe that’s because there just isn’t one single thread and because
PBL is based on the assumption that the available knowledge is relative. So
you cannot say there’s one single solution to a particular problem. The
important thing is that you are working toward a solution.

Josie’s account illuminates that experiences her own experiences as a student. Josie notices
with teaching and PBL are influenced by the that she often missed a structure in the PBL
context of the situation. PBL might work well in courses, and she is afraid that her students will
a small group of students with similar experi- experience the same. Josie wants to prevent that
ences, whereas it is much harder in a context of pitfall, but has not yet found a manner to pro-
a diverse group, according to Josie. It also vide students with a “connecting threat” in situ-
becomes clear in the interview with Josie that ations where there is a diverse input from the
her evaluation of the situation is influenced by group. Like Josie, we found that other teachers

and teacher-trainers were also influenced by characterized by conventional frontal, teacher-

their own educational background and personal centered teaching. Robert’s traditional educa-
history. We therefore decided to analyze the tional background was reinforced by his father
interviews of the respondents in terms of “learn- being a history teacher of the traditional school
ing histories.” The term learning history is used and proud to be so. This background fed
here to refer to our reconstruction of someone’s Robert’s initial misgivings about PBL. These mis-
educational learning path during childhood and givings were not based on deliberate reflection.
adult life. It is composed of the experiences the Rather, they stemmed from Robert’s upbringing
participants told us about (Basten, 2000). The in a family of traditional school teachers where
guiding framework to reconstruct these experi- good teaching was considered synonymous with
ences into a learning history was developed imparting to students what you knew (i.e., being
partly parallel and partly after the responsive a proper teacher). His initial reservations about
evaluation (Niessen, Abma, Widdershoven, PBL were changed into a more neutral/positive
Akkerman, & van der Vleuten, 2008). attitude by his educational experiences at the
Box 21.2 illustrates a learning history. The his- Central College Study Abroad Program in
tory is based on an interview with Robert. Robert Leiden. As a teacher, he learned firsthand that
is new at Maastricht University. He teaches many hours of frontal classroom teaching can be
contemporary history at University College exhausting. As a result, he gradually came to see
Maastricht. In the interview, Robert talks exten- teaching as a process in which students might
sively about his education in primary and sec- take an increasingly active role in their own
ondary school and at university, which was learning. Box 21.2 depicts Robert’s story.

Box 21.2 Robert’s Story

I’ve come to realize that my primary and secondary school were very traditional schools.
Then I went to Leiden. That was in the early 1980s, where I read history, and there teach-
ing was again mostly extremely traditional. The sad thing of course is that I really didn’t
have all that many ideas about PBL but just thought the term seemed outrageous, just
one of those modernisms. Perhaps this had something to do with my father. He is also a
teacher or was, until he took early retirement a year ago. He taught history too. He was
a real story teller, who told stories and I think he would tell you that pupils enjoyed lis-
tening to his stories.
I don’t remember when I first heard about the PBL system. It must have been years
ago. In the meantime, I had been teaching and I had also thought things like. . . . Well,
stimulated by those Americans, I noticed that I didn’t like it myself, because at the col-
lege in Leiden we had to teach in long blocks of many hours, say 3 hours on the same
subject every week. Then you discover that it is really boring to talk nonstop for 3 hours,
and from the start I had thought well, they should do something with that, so you start
with a sort of paper which you want to be really about what has been dealt with in class
and not just some sort of add-on and that gradually expands into projects which students
report in class and discussions.
386——PART IV Educational Evaluation as Learning and Discovery

Robert learned about PBL in the introduc- threatened at any point by the teacher-trainers,
tory course at Maastricht. After his early experi- and although there is no hierarchal relationship
ences at school and university and his gradual between both groups, there is nevertheless a dif-
change of view induced by his teaching experi- ference with regards to the knowledge both have
ences, he immediately felt attracted to the inter- about PBL. This might influence their relation-
active aspect of PBL. His concerns related to the ship. This is the reason that we chose to have
introductory course, which he (paradoxically) separate focus group interviews. Two focus
experienced as too traditional and consisting of groups were organized: The first group was
talks and training learning skills. composed of six former participants of the PBL
Our intention with these learning histories course, and the second group consisted of five
was to provide the participants with valuable teacher trainers. Case descriptions based on the
learning experiences. Some participants did concerns and issues generated in Phase 1 were
indicate that our learning histories or reenact- used to start a dialogue within the groups. Box
ments provided them with a sense of closure 21.3 presents an example of the emerging con-
and a grasp of the past and helped them attain versation in a focus group among teacher-train-
a sense of self or identity (Giddens, 1991). The ers to illustrate the type of issues raised and the
enhancement of personal understanding was development of these issues in interpersonal
considered an indicator of the quality of our interactions.
evaluation. Guba and Lincoln (1989) call The citation starts with Marc, a teacher-
this enhancement of personal understanding trainer at the faculty of Health Sciences of
ontological authenticity1: the extent to which Maastricht University. He enters the conversa-
individual respondents’ own constructions tion at a point when the other teacher trainers
are improved, matured, expanded, and/or are reaching consensus about a central issue in
elaborated. PBL (i.e., whether teachers should restrict their
role to facilitating the group process or whether
it is also acceptable for them to make content-
Communal Conversations and related contributions). Among new staff mem-
Developing Shared bers, there is a widespread unwritten persistent
Understandings of PBL belief that in PBL teachers must rigorously
refrain from formally instructing students in
The next phase of the responsive evaluation any way. We discovered that this makes new
process aimed to engage stakeholder groups in staff members feel guilty—think of Josie—
deliberative dialogues about the issues generated when they talk too much during tutorials
in the first phase. To attain this goal, we pro- instead of adhering to their role of facilitator of
vided the different stakeholder groups the the learning process, encouraging students to
chance to discuss their issues about PBL in gen- find the necessary information and insights for
eral and about the PBL course specifically. We themselves. However, to Marc, this is not the
set up homogeneous groups so the stakeholders core issue. His concern is how to raise the
would not feel threatened; this provided the awareness of teachers about the learning
stakeholders the opportunity to voice and crys- processes among students. The participants in
tallize their thoughts among like-minded the discussion recognize his concern and relate
people. Although the teachers (former par- about their own experiences of how to alert
ticipants of the PBL course) did not seem teachers to students’ development.

Box 21.3 Group Discussion Among Teacher-Trainers

Marc: Can I make a suggestion about the group process? We should distinguish it
from the learning process. Then there are not two, but three components
you have to work with as a tutor.* Self-directed learning, let’s call that the
learning process. I have great difficulty explaining to my tutors how they can
intervene in students’ learning processes to help them become self-directed
learners. This I find the hardest part of the role of the tutor. The content-
related aspect is easy to deal with: take it with a bigger or smaller pinch of
salt. It depends on the situation, what you feel comfortable with, et cetera.
For the group process, we have quite a few instruments as well. Usually you
can explain all sorts of things to people—use texts, use videotapes. But as
for the learning process, that’s very hard to get across in groups, at least in
our tutor training sessions in the Faculty of Health Sciences. In our faculty,
the tutors who attend the tutor training sessions are mostly graduate stu-
dents or prospective graduate students. Well, this is a very special popula-
tion. All these people say, well I’ve done PBL. Nearly all of them have been
a tutor at some time in their student career. And then you find yourself fac-
ing a group in which it is really hard to come to grips with self-directed
learning. That’s my real problem.

Lucy: I think that you go too far for a group in an introductory course. In my expe-
rience, you should mention it, but it goes a bit too far to discuss this in detail
in the tutor training. In my experience, this is a topic for advanced tutor
training. Novice tutors are all very much focused on content. They’re very
pleased when they manage to master that. What I do find important, and
that’s something I’m very clear about in training the tutors, is that tutors
should come to an understanding with the students about the rules for the
group in the first meeting. About what the tutor expects of the students,
how they see the role of the discussion leader, about the quality of the cred-
its, about the quality of the presentation.

Dorothea: It’s funny to see that we have three aspects that are very important in
teacher training: that is the content, the group process, and the learning
process. I agree with Marc; you cannot put the learning process at the very
start of the course, but it does remain the hardest part; how to alert tutors
and show them how learning processes work. How do you get an idea of
what students are going through in a learning process?

Elisabeth: One of the things you can observe as a tutor is that certainties are being
called into question. When that happens, you see symptoms which at first

388——PART IV Educational Evaluation as Learning and Discovery


may mistakenly be seen as negative. Students start to display a violent

hatred toward you or are troubled or think they’re headed for chaos. I read
an official term for it—and I thought that’s a really good way of putting it—
and that’s disjunction. When you’re learning something really new, it’s never
enjoyable from a to z. No way. Then you’re going through a crisis. Often
they say to you afterward. That means that as a tutor you have to prepare
the students and that’s something you don’t learn. . . . Students are so keen
to enjoy themselves. . . . They want to have a good time all of the time, and
we give them that impression by talking about education in terms of con-
sumers and products. Learning is not enjoyable all of the time.

Marc: One of the biggest problems is that the teachers don’t know anything about
didactics. A teacher in primary education has had 4 or 5 years of teacher
training, and university teachers have had hardly any training in our situa-
tion. You are lucky when they show up on all 4 days. I know that the tim-
ing is not right, for these teachers have only just started their professional
development, but well—you have only 4 days, so you try to put some pres-
sure on them. What I try to do is to give them some didactic assignments
to make them think about the question: What is teaching really all about?
* A tutor is a synonym for teacher within a PBL-related teaching environment. Tutor is derived from
the Latin word tueri, which means, according to the Merriam-Webster online dictionary, “a person
charged with the instruction and guidance of another.” According to the Oxford pocket dictionary,
tutor signifies teaching of a single student or a small group of students. It may be noted that in this
chapter where teacher is used, we refer to the explanation mentioned here.

Notice that the ideas about how to raise should be trained in didactics, yet at the end of
teachers’ awareness about learning processes the conversation, he also acknowledges Lucy’s
develop in close interactions that are not with- comment that the timing might not be right for
out strife. Participants exchange experiences those teachers who have just started their pro-
and, as a result of the conversation, come to a fessional development. Moreover, he formulates
fuller, contextually embedded understanding of his claim more modestly, saying that new staff
what it means to train teachers in PBL. members should become somewhat more
Participants explore three components in which aware. This suggests that Lucy has made him
PBL trainers have to be trained: handling the aware of an important issue. The enhanced
content, the group dynamics, and the learning mutual understanding is exactly what we would
processes. Participants develop these ideas in a consider as the additional value of communal
dialogue where each of them brings in various, dialogues between and among stakeholders in a
sometimes conflicting ideas and experiences. As responsive evaluation.
the conversation evolves, they start to acknowl- Referring to Josie’s fragment, but also the
edge the things brought in by each of them. subsequent fragments taken from the learning
Slight changes in views can be observed. Marc, history and dialogue during one of the group
for example, remains convinced that teachers interviews, one might wonder whether the

enactivist perspective is not overly naive or ide- the way the world is a web within which
alistic, assuming a too egalitarian and democra- actions are performed and motivated and
tic portrayal of teaching. Within teaching, power understandings are directed. (p. 28)
inevitably plays a role, potentially disturbing the
processes of mutual tuning and balancing. In This calls for sensitivity and a mindful attitude
case of the example of Josie, she might have been at the side of the researcher.
a person holding on rigidly to her epistemologi- In the third phase of the responsive evalua-
cal notions. This would have been an obstacle to tion process, the conversations were extended to
reaching mutual understanding. We think that dialogues among stakeholders in two heteroge-
learning can only take place when curiously and neous focus groups. Within a heterogeneous
courageously opening up to the other within focus group, the participants of all identified
dialogue. Holding on mindlessly to one’s own stakeholder groups are mixed to engage in delib-
authority is a major obstacle, potentially harm- erative dialogue (in our case, the former partici-
ing any learning taking place within student- pants, the teacher-trainers, and the course
teacher encounters (Langer, 1989, 1997). We developers). This last phase aimed to formulate
acknowledge that this happens all too often in an agenda for resolved and unresolved issues
our schools and universities. Yet we also notice among the stakeholders. It is in this meeting that
that in many instances people are willing to lis- we engaged the group first in developing ground
ten to others, such as Josie. She does not just give rules to which the participants would agree.
up her beliefs (which would have made her Guba and Lincoln (1989) acknowledge this pro-
incredible), but is willing to adjust them to the cedure as a way of creating an environment that
situation ad hand. Of course, consensus is not can facilitate deliberative dialogue.2 Within a
always possible. Yet in many instances, people do deliberative dialogue, the participants are willing
reach a temporary, practical agreement that to participate, are open and willing to change,
enables them to cooperate and collaborate. Such and are willing to distribute power. To share or
a consensus is fragmentary; differences and con- distribute power in a setting such as this might
flicts may occur over time and will yield a new mean that teacher-trainers would not be force-
process of searching and learning. The role of fully and strategically proclaiming their opinions
the evaluator/researcher within enactivist on the staff members due to their information
research in general, and the way it has been advantage. However, because information differ-
played out within Josie’s case, is that of an open ences between both groups are realistic with
and careful listener. A bicolor reflects and mir- regards to PBL, we used this procedure. Both
rors what he is provided with through the stories parties agreed on the previously mentioned
that are told, not explicitly with the intention to claims. Within responsive evaluation, there is not
change. Change or learning often occurs, as was absolute need to reach consensus about the
also the case in this situation, when people speak raised issues. Often the end outcome is an
out loud and are probed by the researcher to agenda for further dialogues. This was also the
restory their experience. As researchers, we tried case within this third phase. Mutual understand-
to act with fidelity, which means, according to ing was heightened. Both parties found the con-
Blumenfeld-Jones (1995), versations enriching. They made arrangements
to continue the dialogues also after the research
by being true to the situation of the teller by had finished. Besides this, both the teacher-trainers
recognizing, constructing, and establishing and the teachers having talked to each other
linkages between events, small and large, identified concrete “nuggets of knowledge” (i.e.,
immediate and distant, immediate and his- immediately applicable knowledge bits for their
torical. An attempt at fidelity illuminates schooling; Scribner, 1999).
390——PART IV Educational Evaluation as Learning and Discovery

Reflections on the Responsive in the expanding spaces. To do so, facilitators

Process in the Case should ask for context, thickening the stories told
and adding detail to experiences. We have done so
We already mentioned that the responsive in the interviews and in the representation of
evaluation process of the PBL course enhanced their stories in the form of personal learning his-
the participants’ personal understanding. The tories. This enhances the participants’ indexical
learning histories, such as that of Robert, appeared sensitivity (Roth et al., 2000), meaning the capac-
to be a fruitful tool for helping participants in ity to feel and accurately denote small, but signif-
restorying their experiences, finding new mean- icant differences between contexts. Langer (1997)
ings in their experiences, and gaining a fuller sense has called this the capacity of mindfulness (i.e.,
of self. The process also helped to attain a richer, the ability to think beyond habitually formed
contextually embedded understanding of what it demarcations, boundaries, and dichotomies).
means to train PBL teachers and what it means to When all participants are being stimulated to
be a PBL teacher. Another value of the responsive think this way—helped by a facilitator—all
process is found in the reestablishment of connec- parties are invited to think beyond the given into
tions among stakeholder groups and participants new creative directions. In the next section, we
within stakeholder groups. The teacher-trainers provide a more in-depth discussion of the social
mentioned that they often miss contacts with fel- learning processes taking place during the
low teacher-trainers. This supports the statement responsive evaluation.
that teaching and training are still individualized
professions in which it is not common to talk with
each about the dos and don’ts of a course in gen- Learning Within
eral and their functioning within it in particular. Responsive Evaluation
Still another value to the teacher-trainers was their
need of concrete information on which to change Traditional cognitivist models of learning from
their course. Listening to and probing each others’ evaluation assume that learning occurs when
experiences was a rewarding learning experience evaluators transmit findings and conclusions to
because they each could take out the issues rele- program participants and stakeholders and that
vant to their practice. One of the first actions they will then process and absorb that informa-
within responsive evaluation is the identification tion. This transmissional view on information
of the key stakeholders to the program or practice processing understands learning as a cognitive
being evaluated. When talking about educational act, something that occurs in the mind of an
practices, the inclusion of students is of eminent individual and separated from other activities.
importance when evaluating pedagogical training Conducting an evaluation, acquiring knowledge,
programs that are central to students’ educational and applying it are thought of as distinct steps.
experiences. Schwandt (2004) specified the underlying
Within responsive evaluation, the evaluator assumptions, which also can be found in both
takes on the role of a facilitator creating condi- behaviourist and cognitive theories of learning:
tions for genuine dialogues. First, he or she (a) basic psychological processes of learning and
should be a conversationalist: assisting partici- cognition are considered the starting point for
pants in making explicit what is unfolding what it means to know and learn; (b) knowledge
around them and inside them, continually is thought of in terms of generalized proposi-
renaming these changing nuances, and unlocking tions (statements) and symbolic representations
the persistent grasp of old categories and that one “possesses” or “has” as a kind of knowl-
dichotomies. Second, the researcher becomes a edge capital; and (c) learning is an “internal”
story developer, helping to trace and meaningfully operation—it takes place in the mind of indi-
record the interactions of the actors and objects vidual knowers.

This conceptualization of learning has been cracks surfacing as a result of the collision of ele-
criticized by a number of scholars who argue ments that make up each teaching situation.
that this model is based on questionable Learning also can be regarded as an inner dia-
assumptions (Brown & Duguid, 1996; Nicolini, logical process among multiple voices or collid-
Gherardi, & Yanow, 2003; (Niessen et al., 2008; ing I-positions (Hermans, 2002; Hermans,
Schwandt, 2004). First of all, practitioners do Kempen, & Van Loon, 1992). Robert, for exam-
not have a static relation to knowledge. Ideas ple, talked enthusiastically about his childhood
and knowledge change over time and in relation and his father being a classical teacher. From this
to context. Second, this model does not position, it made sense to talk about teaching as
acknowledge the fact that people will have to telling students what they need to know. Talking
interpret knowledge and that interpretation and about other experiences and subsequent school-
application of knowledge is normative and thus ings, he felt more at ease being a teacher that was
always influenced by interests and values. These more facilitating and less telling students what to
criticisms suggest that a new way of studying do. This was another more contemporary voice
what it means to learn from an evaluation is to acquired during later schoolings. The struggles
attend carefully to the actual and unfolding and resistances he met related to the question of
learning process amid the people entering the which of these voices or I-positions should take
dialogue. To provide an understanding to the precedence within what teaching situation. The
concept of learning within evaluation, we look difference between this story and that of Josie
more closely to the information gathered during was that the latter was set at the interface level
the responsive evaluation process. We outline being in dialogue with students and other ele-
and understand the learning taking place during ments. The experiences from Robert showed that
evaluation using an enactivist account and simultaneously on an inner level a dialogue took
drawing on complexity theories. place between different I-positions.

Individual Dialogical Learning Communal Dialogical Learning

From an enactivist perspective, learning is In our responsive evaluation of the introduc-

not a “thing” or a characteristic found “over tory course on PBL, we did not only engage in
there” “in the individual” objectively. Josie, for individual talks with participants. We also held
example, talked about an occasion in which her homogeneous and heterogeneous focus group
usual teaching approach to create a common interviews trying to capture some of the issues
thread together with the participants did not and dynamics mentioned earlier in situ. We have
work. Reflecting on it, she realized that it could shown that changes and learning within the dia-
not work in this particular situation because the logues as part of responsive evaluation came
group was too large, the participants had too about by evolution, not revolution. People did
much experience and her trainings skills at that change and alter their views within the dialogues
point had reached the level of an advanced setting place. However, we also acknowledge that
beginner. The learning in this example consisted these changes might be overlooked by the partic-
of the realization that the confrontation between ipants or be ignored as insignificant because they
her ideals and the specifics (other elements) of are minor and taking place within short and
the teaching situation led to the situation as it quick timeframes.
did (i.e., her being unable to create a common Within one of our interviews, for example, a
thread). The learning that took place within this teacher-trainer, talking about training new staff
situation is not to be located solely within the members, specifically referred to the culture in his
individual, but within the interstices or the faculty and the specific population of participants
392——PART IV Educational Evaluation as Learning and Discovery

within his introductory course. Highlighting Conclusion

these specifics and talking about them in con-
versation with others resulted in adding even Our responsive evaluation of the introductory
more detail. The process of adding context and course on PBL presupposes that teacher involve-
texture to general PBL-related issues and con- ment within curriculum revision initiatives is of
cerns within an open atmosphere resulted in utmost importance. In line with enactivist theo-
more openness and susceptibility to detail. It ries, we propose a way of thinking that is dialog-
also allowed participants to alter their views in ical in nature. This dialogical approach regards
small steps. This is important because people do the learning process of the participants, as well
not change their views radically overnight. as our own work as evaluators.
Given that the changes in perspective are often Responsive evaluation states that the need for
small, they might not be acknowledged as revisions and the kind of alternations needed in
changes. Often the change in a participant’s view the revision of an educational program are not
rests in the acknowledgment of the other person’s primarily determined by the scientist-researcher-
view, such as Marc acknowledging Lucy’s remarks evaluator. Responsive evaluation stresses that
about the appropriateness of talking to novice realities are constructed within social-relational
teachers about didactics (see Box 21.3), thus processes. This means that the changes one
changing the tone of the conversation minutely, would like to make concerning a curriculum will
yet significantly. Such changes, although minor, depend on the position one has vis-à-vis the cur-
enable adaptation to local and situational contin- riculum being evaluated. The teachers may want
gencies. Potentially, such acknowledgments do to change other features of the course or the cur-
provide the space for participants to fuse hori- riculum than the teacher-trainers or course
zons. The verb occasioning, as used by Davis developers. Within the responsive evaluation
(2004), aptly represents the way in which minor process, the different stakeholders should engage
perturbations (the contributions by different par- in deliberative dialogues about the meaning of
ticipants in a conversation) may influence dia- their practice and the modifications that are
logues in small but significant ways. In contrast to needed. Within such a conversation, one stake-
a linear approach, the term occasioning signifies holder group should not claim precedence over
unawareness of the direction in which perturba- the other. As pointed out by Roth et al. (2000),
tions and their effects are heading. educationalists and curriculum revisionists
Conversations are meaningful because they should differentiate between the theoretical and
contain various implicit values. References are enacted curriculum. Both are necessary to make
made to values grounded in personal experiences the program or curriculum work.
and related to issues that are important to partic- We think a responsive approach could be
ipants at a personal level. Perhaps the phrase applied more systematically throughout the
meaningful knowing is a more appropriate whole design or revision process of a curriculum,
description of the dialogic process. The aim is not as is done within participatory design (Button &
necessarily to reach consensus about certain val- Dourich, 1996). Within this approach, the end
ues. Rather, it may be to converse and learn about users of the program or curriculum are legiti-
these values so as to achieve some clarification. As mate partners throughout the design or revision
participants in a conversation reflect on and activities. The common or general goal within
explicate their values to each other, a small dis- participatory design is to democratize technolo-
tance or breach is created between the situation at gies and invite end users of a technology as legit-
hand and the values that are being enacted. This imate partners in the design process because the
breach is a precondition for becoming attentive to daily praxis cannot be accounted for by the
one’s own and other people’s values. designers accurately (Button & Dourich, 1996).

The role of the end user and end product should Interdependences between actors increase,
and cannot be determined only by the goal the and in such a multistakeholder environment,
designer has in mind. The dialogue among par- “evaluators have to become consensus builders,
ticipants in a gradually developing design process conciliators and negotiators. . . . When the pur-
is constitutive of working programs (Anderson poses of evaluation include empowerment then
& Crocca, 1993). the need for these skills becomes even more
Stressing egalitarian dialogue, and thus blur- vital” (p. 299). Responsive evaluation provides a
ring the lines between educational developer and model for engagement and dialogue and fosters
teacher, does not mean that the contribution by interactive learning circles building on culturally
the educationalist should be minimized or dis- sensitive, locally inspired, and contextually
missed. Rather, the value of a program is deter- forms of knowing.
mined dialogically and anchored within the
positions of the relevant stakeholders to the pro-
gram being evaluated. This results in local contex- Notes
tually sensitive knowledge. Thick descriptions,
such as the ones provided in this chapter, assist 1. Guba and Lincoln (1989), moreover, identify
readers to translate locally developed knowledge to catalytic authenticity, educative authenticity, tactical
authenticity, and fairness.
other contexts. Thick descriptions illuminate the
2. “Deliberation refers to the interaction and dia-
complexity of a studied setting and provide read-
logue between participants. They do not just accept
ers with a vicarious experience. Local knowledge each other’s beliefs and persuasions, but will explore
naturally fits with people’s need for stories. these. Listening, probing and dialogue characterise this
According to Stake (1986), people change their process, rather than confronting, attacking and defend-
views and ideas evolutionarily, relating vicarious ing. Central feature of dialogue are openness, respect,
experiences to their own experiences. In this inclusion and engagement” (Abma, 2005, p. 280).
respect, Stake calls for“naturalistic generalizations”
(Abma & Stake, 2001). Over time, researchers can
deduce petite generalizations from several case his- References
tories to build small (vs. grand) theories.
Abma, T. A. (1999). Powerful stories about the role of
Within a society that values evidence, facts, stories in sustaining and transforming profes-
general knowledge, and grand theories, contex- sional practice within a mental hospital. In
tually sensitive and empirically embedded R. Josselson (Eds.), The narrative study of lives
knowledge sometimes meets hostility. Expert, (Vol. 6, pp. 169–196). Thousand Oaks, CA: Sage.
scientific knowledge is claimed to be objective, Abma, T. A. (2000). Dynamic inquiry relationships:
general, and abstract, as well as to be superior to Ways of creating, sustaining, and improving the
what is then called subjective, emotional knowl- inquiry process through the recognition and
edge. Local knowledge is underprivileged: It is management of conflicts. Qualitative Inquiry,
associated with antimodern traditionalism with 6(1), 133–151.
backward parochialism (Yanow, 2004). Yet at the Abma, T. A. (2005). Responsive evaluation: Its meaning
and special contribution to health promotion.
same time, we notice that there remains a fun-
Evaluation and Program Planning, 28, 279–289.
damental need among people as storied beings
Abma, T. A., & Broerse (2007). Agency in science:
for the particular, tacit, everyday understand- Patient participation in theory and practice.
ings. In an age of globalization, defined as a Lemma: Den Haag.
process of transformation of local phenomena Abma, T. A., Molewijk, B., & Widdershoven, G. A. M.
into global ones, the capability of organizations (2007). Good care in ongoing dialogues.
and governments to deal with “local” problems Improving the quality of care through moral
has been further reduced (Stern, 2006). deliberation and responsive evaluation. Health
394——PART IV Educational Evaluation as Learning and Discovery

Care Analysis. Retrieved January 13, 2009, from Hermans, H. J. (2002). Special issue on dialogical self.
Springerlink.com. Theory & Psychology, 12(2), 147–280.
Abma, T. A., & Stake, R., E. (2001). Stake’s responsive Hermans, H. J. M., Kempen, H., & Van Loon, R. J. P.
evaluation: Core ideas and evolution. In J. Greene (1992). The dialogical self beyond individualism
& T. Abma (Eds.), New Directions for Evaluation and rationalist. American Psychologist, 47(1), 23–33.
(pp. 7–21). San Francisco, CA: Jossey-Bass. Heron, J., & Reason, P. (1997). A participatory inquiry
Abma, T. A., & Widdershoven, G. A. M. (2005). paradigm. Qualitative Inquiry, 3(3), 274–294.
Sharing stories: Narrative and dialogue in Josselson, R., & Lieblich, A. (1999). Making meaning
responsive nursing evaluation. Evaluation & the of narratives: The narrative study of lives (Vol. 6).
Health Professions, 28(1), 90–109. Thousand Oaks, CA: Sage.
Anderson, W., & Crocca, W. (1993). Engineering prac- Lave, J., & Wenger, E. (1991). Situated learning:
tice and co-development of product and proto- Legitimate peripheral participation. Cambridge:
types. Communication of the ACM, 36(4), 49–56. Cambridge University Press.
Basten, F. (2000). Metaforen en verhalen over organ- Langer, E., J. (1989). Mindfulness. New York: Addison
isatiewetenschap en onderwijsinnovatie. Een Wesley.
leergeschiedenis. Venlo, Netherlands: Repro Press. Langer,E.,J.(1997).The power of mindful learning.Reading,
Blumenfeld-Jones, D. (1995). Fidelity as a criterion for MA: Merloyd Lawrence Book, Perseus Books.
participating and evaluating in narrative inquiry. Lincoln, Y. (2003, January 27–28).Tracks toward a
Qualitative Studies in Education, 8(1), 25–35. postmodern politics of evaluation. Earlier version
Brown, J., & Duguid, P. (1996). Organizational learn- of paper prepared for delivery at the fifth annual
ing and communities-of-practice. In M. Cohen conference of the Southeast Evaluation Association,
& L. Sproull (Eds.), Organisational learning Tallahassee, FL.
(pp. 40–57). London: Sage. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic
Button, G., & Dourich, P. (1996). Technomethodology: inquiry. Beverly Hills: CA: Sage.
Paradoxes and possibilities. Cambridge: Cambridge Lyons, N., & LaBoskey, V. K. (2002). Narrative inquiry
University Press. in practice: Advancing inquiry in practice. New
Cousins, J. B., & Earl, L. (1995). Participatory evalua- York: Teacher’s College Press.
tion in education: Studies in evaluation use and Moust, J. H. C., Berkel, H. J. M. V., & Schmidt, H. G.
organizational learning. New York: Falmer Press. (2005). Signs of erosion: Reflections on three
Davis, B. (2004). Inventions of teaching: A genealogy. decades of problem-based learning at Maastricht
Mahwah, NJ: Lawrence Erlbaum Associates. University. Higher Education, 50, 665–683.
Geertz, C. (1973). Thick description: Toward an inter- Nicolini, D., Gherardi, S., & Yanow, D. (2003).
pretive theory of culture. In C. Geertz (Ed.), The Introduction: Toward a practice-based view
interpretation of cultures: Selected essays (pp. 3–30). of knowing and learning in organizations. In
New York: Basic Books. D. Nicolini, S. Gherardi, & D. Yanow (Eds.),
Giddens, A. (1991). Modernity and self-identity: Self Knowing in organizations: A practice-based
and society in the late modern age. Stanford, CA: approach. Armonk, NY: M. E. Sharpe.
Stanford University Press. Niessen, T. J. H. (2007). Emerging epistemologies: Making
Greene, J. C. (1997). Participatory evaluation. In sense of teaching practice. Unpublished doctoral
L. Mabry (Eds.), Evaluation and the postmodern dissertation, University of Maastricht, Maastricht.
dilemma, Advances in program evaluation (Vol. 3, Niessen, T. J. H., Vermunt, J. D. H. M., Abma, T. A.,
pp. 171–190). Greenwich, CT: JAI Press. Widdershoven, G. A. M., & Vleuten, C. P. M.
Greene, J. C. (1999). The inequality of performance (2004). On the nature and form of epistemolo-
measurements. Evaluation, 5(2), 160–172. gies: Revealing hidden assumptions through an
Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation analysis of instrument design. European Journal
evaluation. Beverly Hills, CA: Sage. of School Psychology, 2(1), 39–64.
Guba, E. G., & Lincoln, Y. S. (2005). Paradigmatic Niessen, T. J. H., Abma, T. A., Widdershoven, G. A. M.,
controversies, contradictions, and emerging Akkerman, S., & van der Vleuten, C. P. M. (2008).
confluences. In N. Denzin & Y. Lincoln (Eds.), Contemporary epistemological research in edu-
Handbook of qualitative research (pp. 191–215). cation: Reconciliation and reconceptualisation
Thousand Oaks, CA: Sage. of the Field. Theory & Psychology, 18(1), 27–45.

Noordegraaf, M., & Abma, T. A. (2003). Management Stern, E. (2006). Contextual challenges for evaluation
by measurement. Journal of Public Administration, practice. In I. F. Shaw, J. C. Greene, & M. M.
81(4), 853–872. Mark (Eds.), The Sage handbook of evaluation
Patton, M. (1990). Qualitative evaluation and research (pp. 292–314). London: Sage.
methods. Newbury Park, CA: Sage. Stronach, I. (1999). Shouting theatre in a crowded
Roth, G.-W., Lawless, D., & Tobin, K. (2000). Time to fire: “Educational effectiveness” as cultural per-
teach: Towards a praxeology of teaching. formance. Evaluation, 5(2), 173–193.
Canadian Journal of Education, 25, 1–15. Titchen, A., & McCormack, B. (2008). Chapter 4: A
Schwandt, T. A. (2004, October). Some thoughts on methodological walk in the forest: Critical cre-
learning in and from evaluation. Stimulus paper ativity and human flourishing. In K. Manley, B.
presented at the panel “Learning Evaluation: The McCormack, & V. Wilson (Eds.), International
Relevance of Communities of Practice,” practice development in nursing and healthcare
European Evaluation Society sixth biennial con- (pp. 59–83). New York: Blackwell Publishing.
ference, Berlin, Germany. Van der Knaap, P. (2006). Responsive evaluation and
Schwandt, T. S. (2003). Back to the rough ground! performance measurement: Overcoming the
Beyond theory to practice in evaluation, downsides of policy objectives and performance
Evaluation, 9(3), 353–364. indicators. Evaluation, 12(3), 278–293.
Scribner, J. P. (1999). Professional development: Whitmore, E. (1994). To tell the truth: Working with
Untangling the influence of work context on oppressed groups in participatory approaches to
teacher learning. Educational Administration inquiry. In P. Reason (Ed.), Participation in human
Quarterly, 35(2), 238–266. inquiry (pp. 82–98). Newbury Park, CA: Sage.
Stake, R. E. (1975). Evaluating the arts in education: A Widdershoven, G. A. M., & Sohl, C. (1999). Inter-
responsive approach. Columbus, OH: Merrill. pretation, action and communication: Four sto-
Stake, R. E. (1986). An evolutionary view of program ries about a supported employment program. In
improvement. In E. House (Ed.), New directions T. A. Abma (Ed.), Telling tales. On narrative and
in educational evaluation (pp. 89–102). evaluation, advances in program evaluation (Vol.
Philadelphia: The Falmer Press. 6, pp. 109–130). Westport, CT: JAI Press.
Stake, R. E., & Abma, T. A. (2005). Responsive evalua- Yanow, D. (2004). Translating local knowledge at
tion. In S. Mathison (Ed.), Encyclopedia of evalu- organizational peripheries. British Journal of
ation (pp. 376–379). Thousand Oaks: Sage. Management, 15, 9–25.
About the Contributors

Tineke A. Abma is Associate Professor and Program Leader of “Autonomy and

Participation in Chronic Care” at VU Medical Centre, EMGO Institute,
Department of Medical Humanities, Amsterdam. Her scholarly work concentrates
on participatory and responsive evaluation approaches, dialogue and moral delib-
eration, narrative and storytelling and patient participation in health research. She
has conducted many evaluation projects in the fields of healthcare (psychiatry,
elderly care, intellectual disabilities, rehabilitative medicine, palliative care), social
welfare, and higher education.
Raymond J. Adams, BSc (Hons), DipEd, MEd(Melb), PhD (Chicago), FACE
Professor Adams is Professorial Fellow of the University of Melbourne and an
independent consultant specializing in psychometrics, educational statistics, large-
scale testing, and international comparative studies. He has led the OECD
PISA Programme since its inception. Ray has published widely on the technical
aspects of educational measurement, and his item response modeling software
packages are among the most widely used in educational and psychological mea-
surement. He has served as chair of the technical advisory committee for the
International Association for the Evaluation of Educational Achievement and as
Head of Measurement at the Australian Council for Educational Research.
Debra D. Bragg is Professor of the Department of Educational Organization and
Leadership in the College of Education at the University of Illinois. She is respon-
sible for coordinating the College of Education’s Higher Education and
Community College Executive Leadership programs, and she is the principal inves-
tigator for research and evaluation studies funded by the U.S. Department of
Education, state agencies, and the Lumina Foundation for Education. Her research
focuses on P-16 policy issues, with special interest in high school-to-college transi-
tion and various policies and practices focused on addressing the educational needs
of underserved students.
Madhabi Chatterji, PhD, is Associate Professor of Measurement and Evaluation
and Codirector of the Assessment and Evaluation Research Initiative, Teachers
College, Columbia University. Her research, currently focusing on diagnostic class-
room assessment, evidence standards, and educational equity, has been recognized
by the Fulbright Commission (2007–2008), the American Educational Research

About the Contributors——565

program evaluation, school-based evaluation, and student assessment. His current

research is focused on dialogue evaluation, combining internal and external evalu-
ation and working with schools and teachers to improve their evaluation capabili-
ties and their ability to cope with external evaluation requirements and
accountability. Dr. Nevo is the author of Evaluation in Decision Making (with
Glasman, Kluwer, 1988) and School-Based Evaluation: A Dialogue for School
Improvement (Pergamon, 1995), the editor of School-Based Evaluation: An
International Perspective (Elsevier, 2002) and the editor-in-chief of Studies in
Educational Evaluation. He served as Head of School of Education, Tel Aviv
University, and Chief Scientist of the Israeli Ministry of Education.

Theo Niessen is a researcher and lecturer at Fontys University of Applied Sciences-

Faculty of Nursing. He is also appointed as the head of an ethics committee at a
home for elderly people. Within his PhD, he developed an enactivist epistemologi-
cal framework to understand teachers´ learning processes during responsive evalu-
ation. Currently his research is concentrating on ethics and moral deliberation and
practice improvement.

Joe O’Hara, PhD, is Senior Lecturer at the School of Education Studies, Dublin City
University, with responsibility for Initial Teacher Education. His research interests
include educational evaluation and initial teacher education. He is an active mem-
ber of the Irish Evaluation Network and European Evaluation Society. He is cur-
rently a member of the General Council of the European Educational Research
Association and is Vice President of the Educational Studies Association of Ireland.
Their (see McNamara) most recent publications include Trusting Schools and
Teachers: Developing Educational Professionalism Through Self-Evaluation (Peter
Lang, 2008) and “Importance of the Concept of Self-Evaluation in the Changing
Landscape of Educational Policy,” Studies in Educational Evaluation, 34, 173–179.

Sharon F. Rallis is Dwight W. Allen Distinguished Professor of Education Policy

and Reform at the University of Massachusetts, Amherst. A past president of the
American Evaluation Association, Rallis has worked with evaluation for over three
decades and has published extensively in evaluation journals. Her research focuses
on local implementation of policy-driven programs. She has taught on education
leadership and policy faculties at University of Connecticut, Harvard, and
Vanderbilt. Rallis’ doctorate is from Harvard University. Her books include
Learning in the Field: An Introduction to Qualitative Research (with Gretchen
Rossman) and Leading with Inquiry and Action: How Principals Improve Teaching
and Learning (with Matthew Militello).

Dana K. Rickman was recently named the Director for Research and Policy at the
Annie E. Casey Foundation, Atlanta Civic Site. Before then, she was a Senior
Research Associate at Georgia State University, Andrew Young School for Policy
Studies. Rickman has participated in a variety of public policy evaluations includ-
ing North Carolina’s Disadvantaged Student Supplemental Fund, Georgia’s
Universal Pre-K, and Georgia’s TANF system. Rickman has published in the fields
of evaluation, and education policy.
About the Contributors——567

Christina Segerholm is Senior Lecturer at MidSweden University, Sweden. Her

research interest is mainly directed toward critical studies of evaluation impact in
education. A more recent interest is evaluation as global policy and governance. Some
of her studies include National Evaluations as Governing Instruments: How Do They
Govern?, Evaluation, 7(4); Governance Through Institutionalized Evaluation:
Recentralization and Influences at Local Levels in Higher Education in Sweden (co-
author Eva Åström), Evaluation, 13(1); and New Public Management and Evaluation
under Decentralizing Regimes in Education Dilemmas of Engagement: Evaluation and
the New Public Management (Saville Kushner & Nigel Norris, Eds.).

Nick L. Smith (PhD, University of Illinois, 1975) is Professor and Chairperson in

the Instructional Design, Development, and Evaluation Department, School of
Education, at Syracuse University. He has served on numerous editorial boards,
including as Editor of New Directions for Evaluation. Professor Smith has received
distinguished awards from the Association of Teacher Educators, the American
Psychological Association, and the American Evaluation Association. He is a Fellow
in the American Educational Research Association and the American Psychological
Association, and is a 2004 President of the American Evaluation Association. His
primary research interests concern the theory and methods of evaluation and
applied research.

Peter M. Steiner is Assistant Professor in the Department of Sociology at the Institute

for Advanced Studies in Vienna and Visiting Assistant Professor at Northwestern
University. He holds a master’s degree and a doctorate’s degree in statistics from the
University of Vienna, as well as a master’s degree in economics from the Vienna
University of Economics and Business Administration. His research interests are in
the methodology of causal inference, particularly quasi-experimental designs in edu-
cation and experimental vignette designs in survey research.

Claudia V. Tamassia, MEd (Columbia, MO), PhD (Champaign, IL), works at the
Educational Testing Service coordinating the OECD Programme for the
International Assessment of Adult Competencies (PIAAC). Her primary interests
are international education and comparative and international assessment. She has
worked at the Ministry of Education in Brazil, at the OECD in Paris in managing
the Programme for International Student Assessment, and at the Chicago Public
Schools. As a consultant, she has worked with UNESCO and taught at the
University of Illinois. She completed her undergraduate studies in Brazil and her
graduate work at the University of Missouri-Columbia and at the University of
Illinois at Urbana-Champaign.

Cees van der Vleuten is Professor in Education at the department of Education,

Maastricht University. He is appointed as Professor of Education at the Faculty of
Health, Medicine, and Life Sciences; Chair of the Department of Educational
Development and Research; and Scientific Director of the School of Health
Professions Education (www.she.unimaas.nl). His area of expertise lies in evalua-
tion and assessment. He has published widely on these topics and holds several aca-
demic awards for this work. He has frequently served as an educational consultant
568——The SAGE International Handbook of Educational Evaluation

Ove Karlsson Vestman is Professor of Education at Mälardalen University, Sweden,

where he directs the Mälardalen Evaluation Academy. He was one of the founders
and the first vice president of the Swedish Evaluation Society. His work concentrates
on building evaluation capacity. In this work, he typically is using participatory and
mixed-method approaches. He has published on dialogue methods and critical the-
ory, as well as the role of values and politics in evaluation: Between Democratic and
New Public Management Evaluation, International Handbook of Educational
Evaluation (Kellaghan & Stufflebeam, Eds.); Democratic Evaluation: An Arena for
Dialogue and Learning? Evaluating Educational Reforms: Scandinavian Perspectives
(Schwandt & Haug, Eds.); and The Relationship Between Evaluation and Politics,
Handbook of Evaluation (Shaw, Greene, & Mark, Eds.).
Guy A. M. Widdershoven is Professor in Ethics of Health Care at the Department
of Health, Ethics, and Society and Scientific Director of the VU Medical Centre,
EMGO Institute, Department of Medical Humanities, Amsterdam. His work con-
centrates on the development of contextual approaches to ethics (narrative ethics,
ethics of care, communicative ethics, hermeneutic ethics), moral deliberation,
patient participation, and responsive evaluation.
Angela Wroblewski is Assistant Professor in the Department of Sociology at the
Institute for Advanced Studies in Vienna and Lecturer at the Vienna University of
Economics and Business Administration and the University of Vienna. She teaches
research methods to students at the BA, MA, and PhD levels. Her research interests
are in evaluation research (especially of education and labor market programs with
a gender focus) and equal opportunities in education and labor markets.

Похожие интересы