Using Realtime Writing Technology to Enable

Continuous Formative Evaluation of

Collaborative Knowledge Work

Brian McNely1 and Paul Gestwicki2

1
Department of English, Ball State University, Muncie IN 47306
bjmcnely@bsu.edu
2
Department of Computer Science, Ball State University, Muncie IN 47306
pvgestwicki@bsu.edu

Abstract. Contemporary higher education should prepare students for

collaborative knowledge work, empowering them to work with infor-
mation that continually circulates across disciplinary and professional
domains. Much of that work is instantiated in and through writing
activities, and these activities have long been seen as explicitly epis-
temic, fostering new knowledge through the ongoing negotiation of pro-
visional knowledge. However, formative evaluation—through both peer
and expert review—can be challenging in educational environments that
employ a knowledge work approach because collaborations have tradi-
tionally occurred across multiple communication channels and have of-
ten been ephemeral. This paper describes ongoing research with Google
Wave as an enabling technology for surfacing, tracing, and visualizing
knowledge work and peer review in collaborative learning environments,
with the goal of fostering more meaningful student metacognition and
modeling practices.

Keywords:
knowledge work, Google Wave, formative evaluation, peer review, collabora-
tion, visualization

1 Knowledge Work, Computer-Supported Peer-Review,

and Formative Evaluation

While peer-review has traditionally been practiced in a variety of academic con-

texts, the growth of applications enabling computer-supported collaboration has
provided significant new affordances for tracing and visualizing not only peer re-
view, but other forms of collaborative activity. Collaboration occurring through
previously ephemeral or distributed practices such as backchannel communica-
tion, sketching and prototyping activities, and conversations around the sharing
of files is now more easily enacted within a single interface—initially by way
of Learning Management Systems, and more recently through systems such as
Google Wave. Among the most relevant features of these systems are the abil-
ity to maintain and visualize an ongoing record of project development and peer
feedback for all participants within a given wave, thus providing strong opportu-
nities for modeling appropriate activities and encouraging metacognition. These
applications hold promise for enacting more meaningful and ongoing formative
evaluation of student collaboration—for peer review, for instructor evaluation,
and even for the participation of relevant stakeholders in the community.
Much of the collaborative work that takes place in such environments oc-
curs in and through writing activities which have long been seen as explicitly
epistemic [1, 2, 8], fostering the development of new knowledge through the on-
going negotiation of provisional knowledge. We argue that by visualizing these
epistemic writing activities, not only are opportunities for continuous formative
evaluation and peer review more readily enabled, but so are explicit opportunities
for fostering more meaningful student metacognition and modeling practices.

1.1 Collaborative Knowledge Work

The concept of knowledge work has become especially trenchant in professional
and technical communication research and practice. Johnson-Eilola [7] explicitly
links concepts in distributed knowledge work to Reich’s [10] figurations of the
symbolic-analytic (a job category Reich positions as distinct from routine pro-
duction and in-person services), where professionals work “within information,
filtering, rearranging, transforming, and making connections to address specific,
specialized problems” [7, p. 19]. The notion of knowledge work has received in-
creased attention since 2005, most notably in response to the development and
realization of complex distributed work activities in online spaces [13, 14]. Spin-
uzzi defines knowledge work as “work in which the primary product is knowledge,
information that is continually interpreted and circulated across organizational
boundaries” [13, p. 1]. He also argues that knowledge work “tends to be organized
in distributed, heterogeneous networks rather than in modular hierarchies” [13,
p. 1]. Increasingly our students’ work mirrors that of professional knowledge
workers as they grapple with information sources that are varied across geogra-
phies, cultures, and disciplinary or professional domains. Spinuzzi notes that this
kind of work is “deeply interpenetrated, with multiple, multidirectional informa-
tion flows” [14, p. 268]. Grabill and Hart-Davidson contend that knowledge work
often “looks like writing (indeed, often is writing),” and that “writing is how
knowledge work carries value in organizations” [3, p. 1]. Attention to networked
writing practices—what writing does—is paramount to our understanding of the
collaborative knowledge work of students.

1.2 Participatory Pedagogies

A knowledge work approach is an integration of participatory pedagogies exem-
plified by Contributing Student Pedagogy [4], Studio-Based Learning [5], and
participatory education [6]. These approaches share a philosophical foundation
in constructivist, community-based learning. Hamer et al. define a Contributing
Student Pedagogy (CSP) as one that “encourages students to contribute to the
learning of others and to value the contributions of others” [4, p. 194]. Like the
figurations of knowledge work described above, CSP is sensitive to the expecta-
tions of industry, where constructivist learning and communities of practice are
organizational realities in distributed work environments. Studio-Based Learn-
ing (SBL) applies techniques from art and architecture studios to create collab-
orative learning environments [5]. SBL is characterized by formal and informal
formative evaluation by peers and experts, as well as students’ constructions of
their own representations. While pedagogically significant in themselves, these
approaches have meaningful implications for the design of curricula in fields
beyond computing.
Participatory education is a recent constructivist pedagogy based on theories
of participatory culture, and it has seen increasing implementation in humanities
curricula. Jenkins et al. provide the foundation for an understanding of participa-
tory culture, a phenomenon that includes characteristics such as informal men-
torship, strong social connections, and mutual validation of peer work [6]. This
view of participatory education may be productively articulated with knowledge
work, CSP, and SBL approaches, where participation is always already grounded
in literate activity, particularly in writing work. CSP, SBL, and participatory
learning emphasize the role of collaborative communication activities—and the
enabling technologies that support them—for making student contributions and
peer feedback more visible and traceable. The ability to better trace and visu-
alize these activities, through a robust platform such as Google Wave, affords
significant opportunities for meaningful formative evaluation.

2 Google Wave as Enabling Technology for Formative

Evaluation

Participatory approaches to education must be enacted, at least in part, in dig-

ital environments that allow greater participation from relevant stakeholders.
In these kinds of environments, opportunities for collaboration extend beyond
the course, foregrounding the need for students to be continually and explic-
itly metacognitive—aware of how they model knowledge-sharing for others. The
enabling technologies for these environments also establish multiple opportuni-
ties for formative evaluation, where peer feedback, instructor feedback, and even
relevant community feedback can all be traced in realtime, through epistemic
writing work. For Hamer et al., “technology supports collaboration and commu-
nication and the development of attitudes and skills” [4, p. 198]. Perhaps more
importantly, the enabling technologies of knowledge work afford more robust
surfacing and tracing of collaborative activities, and this increases opportunities
for formative evaluation. In this way, the enabling technologies of knowledge
work are also central to more effective formative evaluation, both from peers
and from instructors.
Google Wave affords specific advantages to the practice of student peer re-
view, namely: a pliable development environment, synchronous collaboration,
and the persistent visualization of writing work. Backed by a robust developer
community, an already large and growing ecosystem of Wave extensions (soft-
ware plug-ins that interoperate with Wave), and the recent release of the Wave
Data application programming interface (API), the development environment for
enhancing educational uses of wave is already rich. More importantly, the latter
two affordances have bearing on student metacognition and modeling practices.
As a synchronous writing environment, Wave shapes metacognition differently
than asynchronous writing; this is akin to the differences in interaction that oc-
cur in face to face exchanges as opposed to an email exchange. When students
know that anyone can see them working in real time, they may think about their
writing very differently. Moreover, writing within wave is tangible and persistent.
Students can return to examples provided by instructors or other students again
and again, using the “playback” function to visualize the evolving character of
a particular instance of peer feedback.
Google Wave provides the technological framework for such formative eval-
uation and peer feedback. Wave can best be described as a re-imagining of
Internet-based communication, designed for the needs and on the infrastruc-
ture of the 21st century. An individual wave is a collaboratively-created digital
artifact that can consist of rich text, hyperlinks, and embedded multimedia,
as well as Wave extensions. The extension mechanism imbues Wave with es-
sentially limitless upward scalability (provided adequate Internet infrastructure
and computing capability). The Wave Federated Protocol is an open protocol,
which is critical to its success: this means that anyone can host a Wave service
that will interoperate with others, similar to how email servers interoperate to-
day. Hence, institutions have complete control over who has access to data and
services within their local Wave servers, allowing compliance with established
privacy regulations.
As noted above, Wave supports real-time collaboration, meaning that users
can see contributions as they are happening. Our approach involves building
Wave extensions that leverage this rich data for realtime analysis as well as offline
data mining. Wave robots are automated agents that can be added as collabo-
rators to a wave, and we are developing custom robots to gather wave activity.
These extensions will allow for the design of both lightweight dashboard-style
visualizations and comprehensive interactive visualizations. Dashboard visual-
izations provide contextual insight into collaborative activity within Wave. Such
visualizations can be embedded into wave for real-time monitoring, while comple-
mentary visualizations can be situated externally, such as on the Web or mobile
devices. Comprehensive visualizations can be built upon the same database of
interactions, leveraging techniques from visualization of software repositories [9],
threaded persistent discussions [12], and collaboratively edited documents [15].
This software will be incorporated into courses at Ball State University during
the 2010–2011 academic year. We are planning a thorough qualitative study
through which we will measure the impact of our approach on students’ learning,
attitudes, behaviors, peer interaction, and metacognition. The extensions we
develop will be evaluated according to the multi-dimensional, in-depth, long-
term case study approach described by Shneiderman and Plaisant [11].

References
1. Emig, J.: Writing as a mode of learning. College Composition & Communication
28(2), 122 – 128 (1977)
2. Emig, J.: Inquiry paradigms and writing. College Composition & Communication
33(1), 64 – 75 (1982)
3. Grabill, J., Hart-Davidson, W.: Understanding and supporting knowledge work in
everyday life. Language at Work (2010)
4. Hamer, J., Cutts, Q., Jackova, J., Luxton-Reilly, A., McCartney, R., Purchase, H.,
Riedesel, C., Saeli, M., Sanders, K., Sheard, J.: Contributing student pedagogy.
SIGCSE Bull. 40(4), 194–212 (2008)
5. Hundhausen, C.D., Narayanan, N.H., Crosby, M.E.: Exploring studio-based in-
structional models for computing education. In: SIGCSE ’08: Proceedings of the
39th SIGCSE technical symposium on Computer science education. pp. 392–396.
ACM, New York, NY, USA (2008)
6. Jenkins, H., Purushotma, R., Weigel, M., Clinton, K., Robison, A.: Confronting
the challenges of participatory culture: Media education for the 21st century. MIT
Press, Boston, MA, USA (2009)
7. Johnson-Eilola, J.: Datacloud: Toward a new theory of online work. Hampton
Press, Cresskill, NJ, USA (2005)
8. Lauer, J.: Writing as inquiry: Some questions for teachers. College Composition &
Communication 33(1), 89 – 93 (1982)
9. Ogawa, M., Ma, K.L.: Code swarm: A design study in organic software visualiza-
tion. IEEE Transactions on Visualization and Computer Graphics 15, 1097–1104
(2009)
10. Reich, R.: The work of nations: Preparing ourselves for 21st century capitalism.
Vintage Press, New York, NY, USA (1992)
11. Shneiderman, B., Plaisant, C.: Strategies for evaluating information visualization
tools: multi-dimensional in-depth long-term case studies. In: BELIV ’06: Proceed-
ings of the 2006 AVI workshop on BEyond time and errors. pp. 1–7. ACM, New
York, NY, USA (2006)
12. Smith, M.A., Fiore, A.T.: Visualization components for persistent conversations.
In: CHI ’01: Proceedings of the SIGCHI conference on Human factors in computing
systems. pp. 136–143. ACM, New York, NY, USA (2001)
13. Spinuzzi, C.: What do we need to teach about knowledge work? Digital Writing &
Research Lab, University of Texas at Austin, Whitepaper 060925-1 (Sep 2006)
14. Spinuzzi, C.: Guest editor’s introduction: Technical communication in the age of
distributed work. Technical Communication Quarterly 16(3), 265 – 277 (2007)
15. Viégas, F.B., Wattenberg, M., Dave, K.: Studying cooperation and conflict between
authors with history flow visualizations. In: CHI ’04: Proceedings of the SIGCHI
conference on Human factors in computing systems. pp. 575–582. ACM, New York,
NY, USA (2004)