Body, Language and Mind Vol.1 - Embodiment

Body, Language and Mind
Volume 1: Embodiment
Cognitive Linguistics Research

35.1
Editors
Dirk Geeraerts
Rene Dirven
John R. Taylor
Honorary editor
Ronald W. Langacker
Mouton de Gruyter
Berlin New York
Body, Language and Mind
Volume 1: Embodiment
Edited by
Tom Ziemke
Jordan Zlatev
Roslyn M. Frank
Mouton de Gruyter
Berlin New York
Mouton de Gruyter (formerly Mouton, The Hague)
is a Division of Walter de Gruyter GmbH & Co. KG, Berlin
Printed on acid-free paper
which falls within
the guidelines of the ANSI
to ensure permanence and durability.
Library of Congress Cataloging-in-Publication Data
Body, language, and mind. Volume 1, Embodiment / edited by Tom
Ziemke, Jordan Zlatev, Roslyn M. Frank.
p. cm. (Cognitive linguistics research ; 35.1)
Includes bibliographical references and index.
ISBN 978-3-11-019327-5 (hardcover : alk. paper)
1. Language and languages Philosophy. 2. Mind and body.
3. Semiotics. I. Ziemke, T. (Tom), 1969 II. Zlatev, Jordan.
III. Frank, Roslyn M.
P107.B63 2007
401dc22
2007028708
Bibliographic information published by the Deutsche Nationalbibliothek
The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie;
detailed bibliographic data are available in the Internet at http://dnb.d-nb.de.
ISBN 978-3-11-019327-5
ISSN 1861-4132
Copyright 2007 by Walter de Gruyter GmbH & Co. KG, D-10785 Berlin
All rights reserved, including those of translation into foreign languages. No part of this book
may be reproduced or transmitted in any form or by any means, electronic or mechanical,
including photocopy, recording, or any information storage and retrieval system, without
permission in writing from the publisher.
Printed in Germany
Table of contents
List of contributors
Introduction: The body eclectic
Tom Ziemke and Roslyn M. Frank
Section A: Historical roots
We are live creatures: Embodiment, American Pragmatism
and the cognitive organism
Mark Johnson and Tim Rohrer
Bringing the body back to life: James Gibson's
ecology of embodied agency
Alan Costall
From the meaning of embodiment to the embodiment
of meaning: A study in phenomenological semiotics
Goran Sonesson
Embodiment and social interaction: A cognitive
science perspective
Jessica Lindblom and Tom Ziemke
Section B: Body and mind
Representing actions and functional properties
in conceptual spaces
Peter Giirdenfors
From pre-representational cognition to language
Takashi Ikegami and Jordan Zlatev
VII
1
17
55
85
129
167
197
vi Table ofcontents
Making sense of embodied cognition: Simulation theories
of shared neural mechanisms for sensorimotor
and cognitive processes
Henrik Svensson, Jessica Lindblom and Tom Ziemke
Phenomenological and experimental contributions
to understanding embodied experience
Shaun Gallagher
Section C: Body, language and culture
Embodiment, language, and mimesis
Jordan Zlatev
The body in space: Dimensions of embodiment
TimRohrer
On the biosemiotics of embodiment and
our human cyborg nature
Claus Emmeche
Embodiment and self-organization of human categories:
A case study of speech
Luc Steels and Bart de Boer
Communication as situated, embodied practice
Wolff-Michael Roth
Index
241
271
297
339
379
411
431
457
List of contributors
Bart de Boer did a Master's degree in computer science at the Rijksuni-
versiteit Leiden (1994) and a PhD in artificial intelligence at the AI lab of
the Vrije Universiteit Brusse1 (1999) under professor Luc Steels. He has
worked as a postdoc in Brussels and at the University of Washington under
professor Patricia Kuhl. He has also performed linguistic fieldwork in Ne-
pal. His main research interest is in computer mode1ing the evolution of
speech. He is currently working as an assistant professor in cognitive ro-
botics at the Rijksuniversiteit Leiden.
e-mail: b.de.boer@ai.rug.n1
Alan Costall is Professor of Theoretical Psychology at the University of
Portsmouth, England. His research interests are wide, but are held together
by a commitment to interdiscip1inarity, and a broadly ecological or mutu-
alist perspective. He has been increasingly involved in work on the history
of modem psychology and its relations (or lack of them) to other disci-
plines. Recent publications include: A. Costa11 and o. Dreier, (eds.), Doing
Things with Things. (London: Ashgate, 2006); A. Costa11, I. Leudar, and V.
Reddy. (2006) "Failing to see the irony in 'mind-reading. '" Theory & Psy-
chology 16(2): 163-167; Bard, K.A., M. Myowa-Yamakoshi, M. To-
monaga, M. Tanaka, A. Costa11, A., and T. Matsuzawa. (2005) "Group
differences in the mutual gaze of chimpanzees (Pan Troglodytes)." Devel-
opmental Psychology 41: 616-624; Rogers, S. D., E. E. Kadar and A.
Costa11, A. (2005) "Gaze patterns in the visual control of straight road
driving and braking as a function of speed and expertise." Ecological Psy-
chology 17: 19-38; Costa11, A., and I. Leudar. (2004). "Where is the 'the-
ory' in theory of mind?" Theory and Psychology 14: 625-648; Costa11, A.,
M. Sinico and G. Parovel. (2003) "The concept of 'invariants' and the
problem of perceptual constancy." Rivista di Estetica, n.s. 24(3), 49-53;
Ost, J., and A. Costa1l. (2002) "Misremembering Bart1ett: A study in serial
reproduction." British Journal ofPsychology 93: 243-255.
e-mail: a1an.costa11@port.ac.uk
Claus Emmeche is a theoretical biologist, Ph.D., Associate Professor and
Head of the Center for the Philosophy of Nature and Science Studies, 10-
viii List ofcontributors
cated at the Niels Bohr Institute, University of Copenhagen. The Faculty of
Science founded the center in 1994 to explore a new and more science-
related way to do philosophy of nature, yet keeping a notion of science as
more than natural science. Emmeche has taught courses in philosophy of
biology and philosophy of science and his current research interests in-
clude biosemiotics, artificial life, ontology, organism/body/cyborg rela-
tions, and philosophy of nature. He is active in the Copenhagen biosemi-
otics school (cf. Reading Hoffmeyer: Rethinking Biology, with Kalevi Kull
and Frederik Stjemfelt) and in developing a cluster of mandatory science
studies courses for the bachelor programmes in Denmark.
e-mail: emmeche@nbi.dk
Roslyn M. Frank is Professor Emeritus in the Department of Spanish and
Portuguese at the University of Iowa. She is co-editor of Cognitive Models
in Language and Thought: Ideology, Metaphors and Meaning (2003);
Language and Ideology, Vol. 2. Cognitive Description Approaches (2001)
and has published extensively in the field of cognitive linguistics as well as
in ethnoscience, most particularly in ethnomathematics and ethnoastro-
nomy. Her research on the Basque language has taken her to Euskal Herria,
the Basque Country, where she has done extensive fieldwork and given
numerous seminars. In addition she has given presentations on these re-
search topics throughout Europe.
e-mail: roz-frank@uiowa.edu
Shaun Gallagher is Professor and Chair of Philosophy and Cognitive Sci-
ences at the University of Central Florida; he has been occasional Visiting
Professor at the University of Copenhagen (2004-2006) and Visiting Sci-
entist at the Medical Research Council's Cognition and Brain Sciences Unit
at Cambridge University (1994). He is co-editor of the interdisciplinary
journal Phenomenology and the Cognitive Sciences. His research interests
include phenomenology and philosophy of mind, cognitive sciences, her-
meneutics, theories of the self and personal identity. His most recent book,
How the Body Shapes the Mind, is published by Oxford University Press
(2005). He is co-editor of the forthcoming Does Consciousness Cause Be-
havior? An Investigation ofthe Nature of Volition (MIT Press, 2006). He is
currently working on several projects, including a co-authored book, The
Phenomenological Mind: Contemporary Issues in Philosophy ofMind and
the Cognitive Sciences (Routledge, 2007). His previous books include:
Hermeneutics and Education (1992) and The Inordinance of Time (1998).
List ofcontributors ix
He has edited or co-edited volumes including: Ipseity and Alterity: Inter-
disciplinary Approaches to Intersubjectivity (2004), Models of the Self
(1999); Hegel, History, and Interpretation (1997). Home page: http://
pegasus.cc.ucf.edu/"-Jgallaghr
e-mail: gallaghr@mail.ucf.edu
Peter Gardenfors is professor of cognitive science at Lund University
(Sweden). He leads the Ph.D. program in Cognitive Science there (LUCS).
He has published numerous books and articles on decision theory, episte-
mology, belief revision, concept formation and the evolution of cognition
(see http://www.lucs.lu.se/People/Peter.Gardenfors/bibli02000.html). The
most important books are Knowledge in Flux: Modeling the Dynamics of
Epistemic States (Bradford Books, MIT Press, 1988); Conceptual Spaces
(Bradford Books, MIT Press, 2000); How Homo Became Sapiens: On the
Evolution of Thinking (Oxford University Press, 2003); and The Dynamics
ofThought (Springer, 2005).
e-mail: peter.gardenfors@lucs.lu.se
Takashi Ikegami earned his Ph.D in Physics (1989). He works on Artifi-
cial Life and Complex Systems by simulating computational models. His
publications range from self-reproduction, ecological systems, embodied
cognition to cognitive linguistics. Some of his recent articles are: Ikegami,
T. (2005). "Neutral phenotypes as network keystone species." Population
Ecology 47: 21-29; Iizuka, H. and T. Ikegami. (2004) "Adaptability and
diversity in simulated turn-taking behavior." Artificial Life 10: 361-378;
Ikegami, T., and G. Morimoto. (2003) "Chaotic itinerancy in coupled dy-
namical recognizers." CHAOS 13: 1133-1147; Ikegami, T. (1999). "Evolv-
ability of machines and tapes." J. Artificial Life and Robotics 3( 4): 242-
245; Ikegami, T. and M. Taiji. (1998). "Structures of possible worlds in a
game of players with internal models." Acta Polytechnica Scandinavica
91: 283-292.
e-mail: ikeg@sacral.c.u-tokyo.ac.jp
Mark Johnson is Professor of Philosophy and Knight Professor of Liberal
Arts and Sciences at the University of Oregon. His research has focused on
the philosophical implications of the role of human embodiment in mean-
ing, conceptualization, and reasoning. He is co-author, with George Lakoff,
of Metaphors We Live By (1980) and Philosophy in the Flesh (1999) and
author of The Body in the Mind (1987) and Moral Imagination (1993). He
x List ofcontributors
is currently completing a book on the aesthetic dimensions of meaning,
drawing on evidence from cognitive science, phenomenology, neurosci-
ence, and the arts that reveals the origins of meaning in felt qualities, sen-
sorimotor patterns, and emotions.
e-mail: markj@oregon.uoregon.edu
Jessica Lindblom is a cognitive science Ph.D. candidate working at the
School of Humanities and Informatics, University of Skovde, Sweden. She
previously received a master's degree in computer science (2001) and
bachelor's degree in cognitive science (2000). Her main research interests
are social aspects of embodied and situated cognition, and their implica-
tions to interactive technology. Some of her publications, together with her
supervisor professor Tom Ziemke, are Lindblom and Ziemke (2003) "So-
cial situatedness of natural and artificial intelligence. Vygotsky and be-
yond." Adaptive Behavior 11(2): 79-96, and Lindblom and Ziemke (2005)
"Body-in-motion: broadening the social mind." In: Bruno G. Bara, Law-
rence Barsalou and Monica Bucciarelli (eds), Proceedings of the XXVII
Annual Conference ofthe Cognitive Science Society, 1284-1289. Mahwah,
NJ: Lawrence Erlbaum.
e-mail: jessica.1indb10m@his.se
Tim Rohrer has published extensively on metaphor and embodiment in
diverse disciplines for over fifteen years. His work has ranged from ex-
perimental cognitive neuroscience to information technology policy and
from the politics of conflict resolution to the philosophy of language. In
addition to the lines of investigation reflected in this volume, he is also
researching how metaphors shape attitudes toward wildfire mitigation in
the western United States. He is perhaps best known as the founder and
maintainer of the Center for the Cognitive Science of Metaphor Online, a
collection of formative articles in metaphor theory and cognitive semantics
(http://zakros.ucsd.edu;'--.Jtrohrer/metaphor/metaphor.htm). He holds a PhD
in philosophy from the University of Oregon and has been a Thomas J.
Watson scholar, a Fulbright researcher at the Center for Semiotic Research
in Aarhus, Denmark, and a NllI fellow at the Institute for Neural Compu-
tation at the University of California at San Diego. At present he directs the
Colorado Advanced Research Institute in Boulder, Colorado.
e-mail: rohrer@cogsci.ucsd.edu
List ofcontributors xi
Wolff-Michael Roth is Lansdowne Professor of Applied Cognitive Sci-
ence at the University of Victoria, Canada. His research focuses on cul-
tural-historical, linguistic, and embodied aspects of scientific and mathe-
matical cognition and communication from elementary school to
professional practice, including, among others, studies of scientists, techni-
cians, and environmentalists at their work sites. The work is published in
leading journals of linguistics, social studies of science, sociology, learning
sciences, and education and various subfields of education (curriculum,
mathematics education, science education). His recent books include To-
ward an Anthropology of Science (Kluwer, 2003), Rethinking Scientific
Literacy (Routledge, 2004, with A. C. Barton), Talking Science (Rowman
and Littlefield, 2005), and Doing Qualitative Research: Praxis of Method
(SensePublishers, 2005).
e-mail: mroth@uvic.ca
Goran Sonesson is Professor of semiotics and Director of the Department
of Semiotics at Lund University. He holds a doctorate in general linguistics
from Lund University, as well as a doctorate in semiotics from the Ecole
des Hautes Etudes en Sciences Sociales, Paris. Between 1978 and 1983, he
was involved in Paris with the semiotics of gesture, and then worked on
Mayan language and culture in Mexico, after which he has occupied dif-
ferent research positions in semiotics in Lund. Sonesson's main work is the
monograph Pictorial Concepts (Lund: Lund University Press 1989), which
is a critical survey of different contributions to pictorial semiotics, re-
viewed in the context of findings in perceptual psychology and cognitive
science. An important part of the book is devoted to a critical assessment
of the theories of iconicity presented by, among others, Goodman and Eco.
This work has subsequently been extended in numerous articles, published
in Semiotica, RSSL Zeitschrift fur Semiotik, VISID, Degres, Sign System
Studies, Current Anthropology, etc. His most recent publications are con-
cerned with bringing a semiotic perspective to the study of evolution.
e-mail: goran.sonesson@semiotik.lu.se
Luc Steels is a professor of Computer Science at the Vrije Universiteit
Brussel (VUB). He graduated in linguistics at the University of Antwerp
and in computer science at the Massachusetts Institute of Technology,
working in the MIT AI Laboratory. After that he worked in the domain of
geophysical measurement interpretation as a project leader for geological
expert systems at Schlumberger. In 1983 he founded the VUB Artificial
xii List ofcontributors
Intelligence Laboratory, which he still directs this day. He was cofounder
and chairman (from 1990 until 1995) of the VUB Computer Science De-
partment (Faculty of Sciences) and also founder and director of the Sony
Computer Science Laboratories in Paris. His scientific research interests
cover the whole field of artificial intelligence, including natural language,
vision, robot behavior, learning, cognitive architecture, and knowledge
representation. His publications can be found in major AI/cogsci journals
such as Behavioral and Brain Sciences, Trends in Cognitive Science, Arti-
ficial Intelligence Journal, etc. He also has edited a dozen books. At the
moment his research focus in on fundamental research into the origins of
language and meaning.
e-mail: steels@arti.vub.ac.be
Henrik Svensson is currently pursuing a Ph.D. in cognitive science at the
School of Humanities and Informatics, University of Skovde, supervised
by professor Tom Ziemke. He received his B.S. degree (2001) and M.Sc.
degree (2002) from the University of Skovde. His main research interest
concerns the relation between agent-environment interaction and higher-
level cognition. Some of his recent publications are: Svensson, H. and T.
Ziemke (2005) "Embodied representation: What are the issues", in: B
Bara, L. Barsalou, and M. Buccarelli (eds.), Proceedings ofthe 27
th
Annual
Meeting of the Cognitive Science Society, 2116-2121. Mahwah, NI: Law-
rence Erlbaum; and Svensson and Ziemke (2004) "Making sense of em-
bodiment", In: K. Forbus, D. Gentner and T. Regier (eds.), Proceedings of
the 26
th
Annual Conference of the Cognitive Science Society, 1309-1314.
Mahwah, NI: Lawrence Erlbaum.
e-mail: henrik.svensson@his.se
Tom Ziemke is Professor of Cognitive Science in the School of Humani-
ties and Informatics at the University of Skovde, Sweden. His research is
mainly concerned with embodied and distributed cognition, i.e. theories
and models of how cognition is shaped by the living body and its interac-
tion with the material and social environment. He is coordinator of a large-
scale European project on robotic models of embodied cognition, called
"Integrating Cognition, Emotion and Autonomy" (www.his.se/icea). and
member of the executive committee of euCognition - The European Net-
workfor the Advancement ofArtificial Cognitive Systems. He is also asso-
List ofcontributors xiii
ciate editor of the journals New Ideas in Psychology and Connection Sci-
ence.
e-mail: tom.ziemke@his.se
Jordan Zlatev is Associate Professor at the Centre for Languages and
Literature, Lund University, Sweden. His PhD thesis (Stockholm Univer-
sity, 1997) is the monograph Situated Embodiment: Studies in the Emer-
gence of Spatial Meaning, in which he formulates a synthetic social-
cognitive framework for the study of language, and applies this to the ty-
pology and acquisition of spatial semantics. He is the co-founder of the
annual international workshop series Epigenetic Robotics: Modelling Cog-
nitive Development on Robotic Systems, first held in Lund, 2001 and the
bi-annual conference Language, Culture and Mind, first held in Ports-
mouth 2004. Zlatev collaborates extensively with semioticians, cognitive
scientists and philosophers within the project Language, Gestures and
Pictures in Semiotic Development and more recently within the highly
interdisciplinary EU-project Stages in the Evolution and Development of
Sign Use (SEDSU) (www.sol.lu.se/sedsu). His own work concentrates on
differences in primary intersubjectivity between great apes and humans,
and on a cross-cultural study of the ontogeny of gestural communication.
The key theoretical concept for his current work is that of bodily mimesis,
understood (following Donald 1991) as the conscious use of the body for
representational means. Zlatev has published on these topics extensively
over the past years in refereed journals and books, and is currently working
on the monograph Bodily Mimesis and the Grounding ofLanguage and co-
editing the book The Shared Mind: Perspectives on Intersubjectivity.
e-mail: jordan.zlatev@ling.1u.se
Introduction: The body eclectic
Tom Ziemke and Roslyn M Frank
1. Background
This is the first volume of a two-volume set with the title Body, Language
and Mind. While this volume focuses on the concept of embodiment, i.e.
the bodily and sensorimotor basis of phenomena such as meaning, mind,
cognition and language, the second volume addresses social situatedness,
i.e. the ways in which individual minds and cognitive processes are shaped
by their interaction with sociocultural structures and practices. Naturally,
the volumes overlap significantly, and in fact they have both to some de-
gree emerged out of a one-day theme session on embodiment held at the 8
th
International Cognitive Linguistics Conference in Logrofio, Spain, in the
summer of 2003. Some of the contributors to these volumes also partici-
pated in the original theme session, whereas others have been invited later
to complement the range of perspectives represented.
The concept of embodiment has received a great deal of attention in the
cognitive sciences during the last twenty years, and as a result terms like
embodied mind, embodied action, embodied cognition are now commonly
used, often in juxtaposition to concepts like situated action (Suchman
1987), situated cognition (e.g. Clancey 1997), distributed cognition
(Hutchins 1995) or the extended mind hypothesis (Clark and Chalmers
1998). In fact, by the 1990s several authors had already declared embodied
cognitive science, which is often taken to more or less include all of these
concepts, to be a new paradigm in cognitive science (e.g. Varela, Thomp-
son and Rosch 1991; Clark 1997, 1999; Pfeifer and Scheier 1999). All of
this might give you the impression: (a) that there is a clear notion of what
embodiment is, and subsequently, a consensus concerning in what sense
cognitive processes (or perhaps certain types of cognition) are embodied,
and (b) that embodied cognitive science in fact is a theoretical framework
that is more or less established and agreed upon by researchers working in
the field. Somewhat surprisingly, however, neither (a) nor (b) are actually
true.
2 Tom Ziemke and Roslyn M. Frank
Although it is by now widely agreed that cognition is embodied, in the
sense that it is shaped by the body and sensorimotor interaction with the
environment, it is less clear exactly what this means (cf. e.g., Chrisley and
Ziemke 2003; Clark 1999; Wilson 2002; Ziemke 2001a, 2003, 2007). Is it
the physical, the biological, the animate, the phenomenal (experienced), or
the social body that shapes cognition, or perhaps all of these? And, exactly
how does the body shape cognition; is it, for example, only involved in
actual sensorimotor interaction with the environment, i.e. in the grounding
of mental representations in the traditional sense (cf. e.g. Harnad 1990;
Ziemke 1999), or does its influence go further, i.e. is the body also cru-
cially involved in thought, language, and other supposedly abstract activi-
ties, such as mathematics (cf. e.g. Lakoff and Nufiez 1999)?
The following brief overview provides several useful distinctions in
conceptions of embodiment that might help to clarify differences in theo-
retical frameworks and commitments in the field that sometimes remain
hidden under a superficial agreement on 'embodiment'. First, we have
Nufiez (1999) who distinguished between trivial, material, and full em-
bodiment. Trivial embodiment simply is the view that "cognition and the
mind are directly related to the biological structures and processes that
sustain them". Obviously, this is not a particularly radical claim, and con-
sequently few cognitive scientists would reject it (dualist philosophers of
consciousness, on the other hand, might). According to Nufiez, this view
further "holds not only that in order to think, speak, perceive, and feel, we
need a brain - a properly functioning brain in a body - but also that in or-
der to genuinely understand cognition and the mind, one can't ignore how
the nervous system works". Material embodiment makes a stronger claim,
but it only concerns the interaction of internal cognitive processes with the
environment, i.e. the issue of grounding, and thus considers reference to
the body to be required solely for accounts of low-level sensorimotor proc-
esses. In Nufiez's terms: "First, it sees cognition as a decentralized phe-
nomenon, and second it takes into account the constraints imposed by the
complexity of real-time bodily interactions performed by an agent in a real
environment". Full embodiment, finally, is the view that the body is in-
volved in all forms of human cognition, including seemingly abstract ac-
tivities, such as language or mathematical cognition (e.g. Lakoff and Nufiez
1999). In Nufiez's own words:
Full embodiment explicitly develops a paradigm to explain the objects cre-
ated by the human mind themselves (i.e., concepts, ideas, explanations,
forms of logic, theories) in terms of the non-arbitrary bodily-experiences
Introduction: The body eclectic 3
sustained by the peculiarities of brains and bodies. An important feature of
this view is that the very objects created by human conceptual structures and
understanding (including scientific understanding) are not seen as existing in
an transcendental realm, but as being brought forth through specific human
bodily grounded processes.
In a similar vein, Clark (1999) distinguished between the positions of weak
embodiment and radical embodiment. According to the fonner, traditional
cognitive science can roughly remain the same; i.e. theories are merely
constrained, but not essentially changed by embodiment. This is similar to
Nufiez's view of material embodiment. The position of radical embodi-
ment, on the other hand, largely compatible with Nufiez's notion of full
embodiment, is, as Clark fonnulated it, "radically altering the subject mat-
ter and theoretical framework of cognitive science".
More recently, Wilson (2002) distinguished between six views of em-
bodied cognition, of which only the last one requires full or radical em-
bodiment whereas the first five might be considered variations or aspects
of material embodiment: (1) cognition is situated, i.e. it occurs "in the
context of task-relevant inputs and outputs", (2) cognition is time-
pressured, (3) cognition is for the control of action, (4) we off-load cogni-
tive work onto the environment, e.g. through epistemic actions (Kirsh and
Maglio, 1994), i.e. manipulation of the environment 'in the world', rather
than 'in the head', (5) the environment is actually part of the cognitive
system, e.g. according to Clark and Chalmers' (1998) notion of the 'ex-
tended mind', and (6) 'off-line' cognition is body-based, which according
to Wilson is the "most powerful claim" (cf. Svensson, Lindblom and
Ziemke this volume).
Complementary distinctions and classifications are proposed in several
of the contributions to this volume. Emmeche (this volume), for example,
points out that different disciplines have different perspectives on the body.
He therefore distinguishes between physical embodiment (physics), or-
ganismic embodiment (biology), animate embodiment (zoology), and an-
thropic embodiment (anthropology, sociology). However, Emmeche also
points out: "The point is not the exact number of levels (these are contin-
gent upon a historically relative state of science) but the fact that irreduci-
ble levels do exist". Rohrer (this volume) takes a different approach and
catalogues what he considers twelve important senses or dimensions of
embodiment in current research on the topic: philosophy, socio-cultural
situation, phenomenology, perspective, development, evolution, the cogni-
4 Tom Ziemke and Roslyn M Frank
tive unconscious, neurophysiology, neurocomputational modelling, mor-
phology, directionality ofmetaphor, and grounding.
It should be noted that, naturally, not all of these views, types or dimen-
sions of embodiment are equally relevant for all disciplines and perspec-
tives. For example, the question of whether or not an embodied mind actu-
ally needs to be physical or biological might not be a meaningful question
to a neuroscientist, given that real brains and bodies obviously are both
physical and biological. However, for the philosopher wrestling with dual-
ist or functionalist conceptions of mind, or for the artificial intelligence
researcher trying to build, or at least model, minds, these are still burning
questions (cf. e.g. Ziemke 2001a, 2007; Zlatev 2001, 2003; Johnson and
Rohrer this volume; Lindblom and Ziemke this volume). Similarly, the
phenomenal experience of the lived body is perhaps not the primary inter-
est for the cognitive linguist interested in syntax or metaphor, but it cer-
tainly is relevant to many other aspects of language and mind (cf. e.g. Son-
esson this volume; Gallagher this volume; Zlatev this volume; Roth this
volume).
To some degree the diversity of perspectives on embodied cognition can
of course be attributed to the inter- or multidisciplinary nature of cognitive
science as such. After all, it is not surprising that, for example, linguists,
neuroscientists and artificial intelligence (AI) researchers would be inter-
ested in different aspects of embodiment, for the same reasons that they
offer different, but hopefully complementary perspectives on the study of
mind and language. True complementarity, however, rather than mere co-
existence of different disciplines and perspectives, would seem to require
some kind of common framework and terminology. For traditional cogni-
tive science, the common ground was provided by theories of functional-
ism, computationalism and representationalism, the basic assumptions of
which were summarized by Gardner (1987: 6) as follows:
First of all, there is the belief that, in talking about human cognitive activi-
ties, it is necessary to speak about mental representations and to posit a level
of analysis wholly separate from the biological or neurological, on the one
hand, and the sociological or cultural, on the other.
Consequently, the complementarity and division of labor between different
disciplines was, perhaps, relatively clear in the old, pre-embodiment days
when everybody agreed that thought was the operation of computational
processes on mental, presumably symbolic representations. Linguists were
mostly concerned with language as a symbol system, without much interest
in its biological implementation, while neuroscientists tended to be inter-
ested in how and where in the brain the relevant computations and repre-
sentations were implemented, and AI researchers were striving to synthe-
size the very same computations and representations in artificial imple-
mentations.
Apart from the fact that the above picture of traditional, computational-
ist cognitive science is of course a bit too rosy (the exact nature of those
mental representations, for example, caused, and still causes, considerable
debate), we have to admit that there is no equivalent consensus yet in em-
bodied cognitive science. We could say that is because, as a discipline or
paradigm, embodied cognitive science is still relatively young, but then
again it is 20-30 years old by now, depending on exactly how you situate
its origins in time. Moreover, one could rightly point out that this diversity
of perspectives is due to the expanding nature of the community of re-
searchers concerned with this approach. In short, since the embodied cog-
nition perspective is still growing and gaining ground, it might be too early
to hope for a convergence of perspectives, indeed, at this stage such a con-
vergence of opinions might be premature. However, the current situation
might also provoke comparisons with soap bubbles that are destined to
burst sooner or later if there is nothing under the surface to hold them to-
gether.
One could also say that things were simpler for traditional cognitive
science, because that approach was able to define itself from the start both
with respect to what it rejected, behaviorism, and with respect to a common
vision, the computer metaphor for mind. But this, in turn, raises the ques-
tion of exactly what the common vision might be in the case of embodied
cognitive science (as well as its common unifying metaphor). Some people
would say that simulation theories (cf. Svensson, Lindblom and Ziemke
this volume) are what constitute the common vision or framework in em-
bodied cognitive science - very roughly speaking, the idea that the same
neural mechanisms are used for both sensorimotor interaction and abstract
thought. Nonetheless these ideas certainly still need to be worked out in
much more detail. ~
Whichever way you look at it, although embodied cognitive science un-
deniably is an exciting and vibrant area of research, it is hard not to agree
with Wilson's (2002: 625) assessment of what might be perceived as a
conceptual muddle:
While this general approach is enj oying increasingly broad support, there is
in fact a great deal of diversity in the claims involved and the degree of
controversy they attract. If the term "embodied cognition" is to retain
meaningful use, we need to disentangle and evaluate these diverse claims.
In fact, that is exactly, what we hope to offer readers with the collection of
chapters making up this volume, namely, a means to disentangle and evalu-
ate the different perspectives, and not least to understand which of them
are actually competing or rather complementary. The papers selected for
inclusion in this first volume have been written by some of the leading
cognitive scientists, cognitive linguists, psychologists, philosophers, AI
researchers, semioticians, and phenomenologists working on embodied
cognition today. That means, the perspective of cognitive linguistics is here
put into the context of a number of related disciplines. The abovemen-
tioned second volume, on the other hand, focuses on social situatedness
and contains more directly linguistically oriented contributions.
Admittedly, we cannot promise that after reading all of the contribu-
tions to this volume you will feel much wiser in terms of exactly which
notion of embodiment is the "correct" one. But you will have learned a
great deal about the most important current perspectives on embodiment
and their historical backgrounds as well as the overlaps and controversies
existing between them. That, we believe, is the best deal anybody can offer
at the moment when it comes to understanding the multidisciplinarity of
modem embodiment research. In that sense, this book can be read as an
update to some of the "classical", highly influential books on embodiment
that appeared in the 1990s, such as Varela, Thompson and Rosch's (1991)
The Embodied Mind, Clark's (1997) Being There, or Lakoff and Johnson's
(1999) Philosophy in the Flesh, as well as other multi-author collections on
the topic (e.g. Nunez and Freeman 1999; Ziemke 2002; Robbins and
Aydede in press). Other books provide more detailed accounts on specific
topics, such as Damasio's (1994, 1999,2003) books on the role of the body
in emotion and consciousness, Goldin-Meadow's (2003) book on gesture
as embodied thought, Pfeifer's books on embodied AI (Pfeifer and Scheier
1999; Pfeifer and Bongard 2006), or Maturana and Varela's (1980, 1987)
classical books on the biology of cognition. There are also several recent
books that do a very good job at integrating some of the perspectives, such
as Gallagher's (2005) How the Body Shapes the Mind, Gibbs' (2006) Em-
bodiment and Cognitive Science, and Thompson's (2007) Mind in Life.
However, we believe that these works need to be complemented with a
broad spectrum of perspectives, from different disciplinary and historical
backgrounds, as collected in this multi-author volume.
2. Overview of the contributions to this volume
This volume is divided into three parts or sections, each of which consists
of four or five chapters. The first section consists of four chapters that ex-
plore in detail the historical roots underlying current discussions of em-
bodiment in cognitive science and linguistics.
First, Johnson and Rohrer contrast the dualistic view of mind and body
as two ontologically different entities, connected through internal repre-
sentations of external reality, with what they call embodied realism, ac-
cording to which cognition and language must be understood as arising
from organic processes. They trace the rejection of mind-body dualism
from early American Pragmatists, such as James and Dewey, forward to
recent embodied cognitive science, which they argue needs to be pragmati-
cally-centered.
Costall's chapter points out that the move from mechanistic behavior-
ism to cognitive psychology and the computer metaphor for mind, through
its focus on internal information processing, in fact simply maintained the
view of the mechanical body as a mere shell or container for cognitive
processes. As an alternative, he elaborates on the contributions that Gib-
son's "ecology of embodied agency" and his concept of affordances can
make to our understanding of embodiment and "being in the world".
Sonesson's chapter addresses the contributions that phenomenology and
semiotics, with their focus on consciousness and meaning, can provide to
current discussions of embodiment in cognitive linguistics, biosemiotics,
and other disciplines, discussions which, in his view, tend to ignore rele-
vant distinctions between different types and forms of meaning. Further-
more, building on Piaget's concept of differentiation, he suggests a devel-
opmental sequence going from schemas to signs and external
representations.
Lindblom and Ziemke, in the last chapter of the "historical roots" sec-
tion, point out that, despite much recent emphasis on the bodily and social
basis of cognition, the role of embodiment in social interaction is still
relatively poorly understood. They therefore trace the role of biological
and sociocultural factors in explanations of cognition from Darwin to
modem cognitive science, and discuss further steps and conceptual clarifi-
cations that will be required in the development of a science of embodied
cognition and social interaction.
The second section of this volume consists of four chapters that address
from different perspectives the relation between body and mind as well as
the actual mechanisms underlying embodied cognition.
Gardenfors elaborates his theory of conceptual spaces, i.e. representa-
tional spaces built up from "quality dimensions". While earlier work on
conceptual spaces focused on perceptual concepts, here it is shown how
this analysis can be extended to actions as spatiotemporal patterns of
forces andfunctional concepts as regions of affordances in an action space.
The discussion extends work in cognitive linguistics/semantics by Johnson
and Talmy on "force dynamics" and discusses the transition from an em-
bodied, first-person notion of force to a third-person perspective.
Ikegami and Zlatev argue that there is a tendency in embodiment theo-
ries to "resolve" oppositions between, for example, interaction and repre-
sentation, or procedural and declarative lmowledge, by simply ignoring the
differences, and thus, in effect, typically eliminating the more "disembod-
ied" side of such oppositions. They therefore explore how mechanisms of
pre-representational cognition such as dynamical categories, internal
meaning spaces, and synaesthesia can play a role in the grounding of con-
cepts, representations, and language.
Drawing on experimental evidence from neuroscience, psychology and
other disciplines, Svensson, Lindblom and Ziemke propose that a key to
understanding the embodiment of cognition is the "sharing" of neural
mechanisms between sensorimotor processes and higher-level cognitive
processes. The latter are argued to be embodied in the sense that they make
use of partial simulations or emulations of sensorimotor processes through
the re-activation of neural circuitry also active in bodily perception and
action.
Gallagher, in the last chapter of the "body and mind" section, clarifies
the distinction between body image and body schema from a phenomenol-
ogical perspective, and then discusses how this distinction can contribute
to our understanding of embodied experience and intersubjectivity. Relat-
ing these concepts to empirical research concerning issues such as inten-
tional action, imitation and mirror neurons, he connects the phenomenol-
ogical insights of Husserl and Merleau-Ponty to the most recent
neuroscience research on social cognition.
The third and final section contains five chapters that address the rela-
tion between body, language and culture from a range of different per-
spectives.
Zlatev argues that much recent work on embodiment tends to under-
value concepts such as representation, consciousness and conven-
tion/normativity, which he claims makes it difficult for current "embodi-
ment theories" to account for human language and cognition. To illustrate
these difficulties, he critically examines the approach presented in Lakoff
and Johnson's (1999) highly influential book Philosophy in the Flesh, and
develops his own alternative notion of mimetic schema as a mediator be-
tween the individual human body and collective language.
Rohrer reviews the variety of standard usages that the term "embodi-
ment" currently has in cognitive science and contrasts notions of embodi-
ment and experientialism at different levels of investigation. With the goal
of developing a broad-based theoretical framework for embodiment he
examines examples of these usages and in the process brings into view
related research issues such as mental imagery, mental rotation, spatial
language and conceptual metaphor across several levels of investigation,
with a focus on whether and how different conceptualizations can form a
cohesive research program.
Emmeche presents biosemiotics as a new perspective on living systems
inspired by von Uexklill and Peirce. He suggests necessary distinctions
between physical, biological, animate, phenomenal and social body, and
develops a Neo-Aristotelian evolutionary emergentist perspective which he
argues to be necessary in order to coherently account for these dimensions
of embodiment. Furthermore, he characterizes humans as techno-culturally
embedded beings within a space of meanings that are not only symbolic,
but also socially empowered by different kinds of sociocultural systems.
From the perspective of artificial intelligence, Steels and de Boer con-
sider the kinds of categories that have been found to be involved in human
behavior, and how they are shaped by embodiment as well as the collective
dynamics generated by social interactions. Using a computational case
study developed for speech sounds, they show how a group of autonomous
agents equipped with a sufficiently realistic perceptual and auditory appa-
ratus can arrive at a shared repertoire of vowels that exhibits the same uni-
versal trends as found in human vowel systems, although not biased by
innate a priori categories.
Roth, finally, in the last chapter of the volume, argues that many re-
searchers are pre-occupied with written language as the main paradigm of
communication. A large part of human communication, however, involves
participants in face-to-face conversation, attending to semiotic resources
such as gestures, body positions, and material structures in the environ-
ment. He provides a dialectical account of human activity and analyses an
episode from a science classroom to exemplify the central role of the body
in communication and to illustrate his view of communication as materi-
ally and socially situated and embodiedpractice.
3. Conclusion
Embodied cognitive science is still a relatively young approach to the study
of mind and language. Critics of the approach might argue that it has so far
failed to produce a coherent theoretical framework that integrates and uni-
fies different approaches and disciplines. However, one could also argue
that it was in fact the premature convergence of traditional cognitive sci-
ence on functionalist/computationalist and representationalist/symbolic
theories of mind that for a long time disconnected cognitive science from
many of its historical roots which are now being rediscovered (cf. e.g.
Lindblom and Ziemke this volume; Johnson and Rohrer this volume).
Hence, we are convinced that a fuller understanding of embodied cognition
will emerge from the type of broad multidisciplinary interaction that has
led to the production of this multi-author volume, rather than from the
more narrow insights that any single discipline or approach could produce
on its own. At the same time we are also convinced that the road towards a
unified theoretical framework in embodied cognitive science is neither
short nor straightforward. Indeed, there still is much work ahead for theo-
rists and modelers of embodied cognition. But, as they say, even the long-
est journey starts with the first steps. And one of the most important steps
in the development of a truly interdisciplinary science of embodied cogni-
tion certainly is to acquire a clear comprehensive understanding of the
different perspectives that characterize this field today.
References
Chrisley, Ron and Tom Ziemke
2003 Embodiment. In: Encyclopedia of Cognitive Science, 1102-1108.
London: Macmillan.
Clancey, William J.
1997 Situated Cognition: On Human Knowledge and Computer Repre-
sentations. New York: Cambridge University Press.
2003
1987
1999
1999
Embodiment and Cognitive Science. Cambridge: Cambridge Univer-
sity Press.
Goldin-Meadow, Susan
2003 Hearing Gesture: How Our Hands Help Us Think. Cambridge, MA:
Harvard University Press.
Hamad, Stevan
1990 The Symbol Grounding Problem. Physica D, 42: 335-346.
Hutchins, Edwin
1995 Cognition in the Wild. Cambridge, MA: MIT Press.
Kirsh, David and Paul Maglio
1994 On Distinguishing Epistemic from Pragmatic Action. Cognitive Sci-
ence, 18 (4): 513-549.
Lakoff, George and Mark Johnson
1999 Philosophy in the Flesh: The Embodied Mind and its Challenges to
Western Thought. New York: Basic Books.
Lakoff, George and Rafael Nunez
1999 Where Mathematics Comes From. New York: Basic Books.
Maturana, Humberto and Francisco J. Varela
1980 Autopoesis and Cognition: The Realization of the Living. Dordrecht,
The Netherlands: D. Reidel Publishing.
The Tree of Knowledge - The Biological Roots of Human Under-
standing. Boston: Shambalaya.
Being There - Putting Brain, Body and World Together Again.
Cambridge, MA: MIT Press.
An embodied cognitive science? Trends in Cognitive Sciences 3 (9):
345-351.
Clark, Andy and David Chalmers
1998 The extended mind. Analysis 58 (1): 7-19.
Damasio, Antonio R.
1994 Descartes' Error: Emotion, Reason, and the Human Brain. New
York: Avon Books.
The Feeling of What Happens: Body and Emotion in the Making of
Consciousness. New York: Harcourt Inc.
Lookingfor Spinoza: Joy, Sorrow and the Feeling Brain. New York:
Harcourt Inc.
Gallagher, Shaun
2005 How the Body Shapes the Mind. Oxford: Oxford University Press.
Gardner, Howard
1987 The Mind's New Science. New York: Basic Books.
Gibbs, Ray
2006
Clark, Andy
1997
Nuiiez, Rafael
1999 Could the Future Taste Purple? Reclaiming Mind, Body and Cogni-
tion. Journal ofConsciousness Studies, 6 (11-12): 41-60.
Nuiiez, Rafael and WaIter J. Freeman (eds.)
1999 Reclaiming Cognition - the Primacy of Action, Intention and Emo-
tion. Exeter: Imprint Academic.
Pfeifer, Rolf and Christian Scheier
1999 Understanding Intelligence. Cambridge, MA: MIT Press.
Pfeifer, Rolf and Josh Bongard
2006 How the Body Shapes the Way We Think: A New View of Intelli-
gence. Cambridge, MA: MIT Press.
Robbins, Philip and Murat Aydede
in press Cambridge Handbook on Situated Cognition. Cambridge, U.K.:
University of Cambridge Press.
Suchman, Lucy
1987 Situated Action. Cambridge, MA: MIT Press.
Svensson, Henrik, Jessica Lindblom and Tom Ziemke
this vo!. Making sense of embodied cognition: simulation theories of shared
neural mechanisms for sensorimotor and cognitive processes.
Thompson, Evan
2007 Mind in Life: Biology, Phenomenology, and the Sciences of Mind.
Cambridge, MA: Harvard University Press.
Varela, Francisco J., Evan Thompson and Eleanor Rosch
1991 The Embodied Mind: Cognitive Science and Human Experience.
Wilson, Margaret
2002 Six views of embodied cognition. Psychonomic Bulletin and Review
9 (4): 625-636.
Ziemke, Tom
1999 Rethinking Grounding. In: Alexander Riegler, Markus Peschl and
Astridvon Stein (eds.), Understanding Representation in the Cogni-
tive Sciences: 177-190. Plenum Press: New York.
2001 a Are robots embodied? In: Christian Balkenius et a!. (eds), Proceed-
ings ofthe First International Workshop on Epigenetic Robotics, 75-
83. Lund, Sweden: Lund University Cognitive Studies, vo!. 85.
2001 b The construction of 'reality' in the robot. Foundations of Science 6
(1): 163-233.
2003 What's that thing called embodiment? In: Richard AIterman and
David Kirsh (eds.), Proceedings of the 25
th
Annual Meeting of the
Cognitive Science Society, 1305-1310. Mahwah, NJ: Lawrence Erl-
baum.
2007 What's life got to do with it? In: Antonio Chella and Riccardo
Manzotti (eds.), Artificial Consciousness: 48-66. Exeter: Imprint
Academic.
Ziemke, Tom (ed.)
2002 Special issue on situated and embodied cognition. Journal of Cogni-
tive Systems Research 3 (3).
Zlatev, Jordan
1997 Situated Embodiment. Studies in the Emergence ofSpatial Meaning.
Stockholm: Gotab Press.
2001 The epigenesis of meaning in human beings, and possibly in robots.
Minds and Machines, 11 (2): 155-195.
2003 Meaning = Life (+ Culture). An outline of a unified biocultural the-
ory of meaning. Evolution ofCommunication 4 (2): 253-296.
Section A
Historical roots
We are live creatures: Embodiment, American
Pragmatism and the cognitive organism
Mark Johnson and Tim Rohrer
Abstract
The philosophical tradition mistakenly asks how the inside (i.e., thoughts, ideas,
concepts) can represent the outside (i.e., the world). This trap is a consequence of
the view that mind and body must be two ontologically different entities. On this
view the problem of meaning is to explain how disembodied "internal" ideas can
represent "external" physical objects and events. Several centuries have shown that
given a radical mind-body dichotomy, there is no way to bridge the gap between
the inner and the outer. When "mind" and "body" are regarded as two fundamen-
tally different kinds, no third mediating thing can exist that possesses both the
metaphysical character of inner, mental things and simultaneously possesses the
character of the outer, physical things.
Embodied Realism, in contrast to Representationalist theories, rej ects the notion
that mind and body are two ontologically distinct kinds, and it therefore rejects the
attendant view that cognition and language are based on symbolic representations
inside the mind of an organism that refer to some physical thing in an outside
world. Instead, the terms "body" and "mind" are simply convenient shorthand ways
of identifying aspects of ongoing organism-environment interactions - and so cog-
nition and language must be understood as arising from organic processes. We
trace the rejection of this mind-body dualism from the philosopher-psychologists
known as the early American Pragmatists (James and Dewey) forward through
recent cognitive science (such as Varela, Maturana, Edelman, Hutchins, Lakoff,
Johnson, Brooks). We argue that embodied realism requires a radical reevaluation
of the classical dualistic metaphysics and epistemology - especially the classical
Representationalist theory of mind - and we conclude by investigating the implica-
tions for future investigations for a new, pragmatically-centered cognitive science.
Keywords: cognitive linguistics, cognitive science, embodiment, image schema,
metaphor, neurobiology, pragmatism, Representationalism, semantics.
18 Mark Johnson and Tim Rohrer
1. Introduction: What difference does embodied realism make?
When a young child crawls toward the fire in the hearth and a mother
snatches up the child before the child gets burned, is that cognition? When
a team of British mathematicians decodes enemy ciphers during wartime, is
that cognition? When ants carrying food back to their nest lay down
chemical signals and thereby mark trails to a food source, is that cognition?
Note the commonalities among these situations. In each case the body
(both individual and social) is in peril. First, the well-being and continued
successful functioning of the organism is at risk. Action must be under-
taken to ensure the continued flourishing of the living, physical, individual
body of the organism. To survive and flourish, the organism must make
adjustments in its way of acting, both within its current environment and in
its relations with other creatures. The child must be snatched from the im-
minent danger of the flames, the mathematicians desperately work to pre-
vent their country from being overrun by the enemy, and the ants must find
food and bring it back to the queen in order for the colony to survive. Sec-
ond, note that in each case the cognition is social, composed of multiple
organisms co-operating in response to current or anticipated problems
posed by the environment. That environment is not merely physical but
also includes the social "body" - whether the family, the nation or the ant
colony - whose survival and flourishing is at risk. And finally, note that
each of these situations have been taken by theorists as emblematic of cog-
nition par excellance (Dewey 1925; Hodges 1983: 160-241; Deneubourg
et al. 1983; Brooks and Flynn 1989).
The importance of embodiment in cognition is now widely appreciated
in the cognitive sciences, yet there remains considerable debate as to what
the term "embodiment" actually means (Rohrer 2001a; in press; Ziemke
2003; Anderson 2003). Is "the body" merely a physical, causally deter-
mined entity? Is it a set of organic processes? Is it a felt experience of sen-
sations and movement? Is it the individual physical body, or does it include
the social networks such as families without which it would cease to exist?
Or is the body a socially and culturally constructed artifact? In this chapter,
we argue that each of these views contributes something important to an
adequate theory of embodied cognition, and that a proper understanding of
embodiment can be found within the philosophical context first elaborated
in early American Pragmatism in the works of thinkers such as William
James and John Dewey. As we see it, embodiment theory inherits several
key tenets of how these Pragmatist philosophers viewed cognition:
We are live creatures 19
(1) Embodied cognition is the result of the evolutionary processes of
variation, change and selection.
(2) Embodied cognition is situated within a dynamic ongoing organism-
environment relationship.
(3) Embodied cognition is problem-centered, and it operates relative to
the needs, interests, and values of organisms.
(4) Embodied cognition is not concerned with finding some allegedly
perfect solution to a problem, but one that works well enough rela-
tive to the current situation.
(5) Embodied cognition is often social and carried out cooperatively by
more than one individual organism.
Note that the Pragmatists advance a radically different view of cognition
than the one we are most familiar with from "classical" cognitive science,
where it is assumed that cognition consists of the application of universal
logical rules that govern the manipulation of "internal" mental symbols,
symbols that are supposedly capable of representing states of affairs in the
"external" world. Fodor summarizes this theory as follows:
What I am selling is the Representational Theory of Mind ... At the heart of
the theory is the postulation of a language of thought: an infmite set of
'mental representations' which function both as the immediate objects of
propositional attitudes and as the domains of mental processes. (Fodor 1987:
16-17)
These internal representations in the "language of thought" acquire their
meaning by being "about" - or referring to - states of affairs in the external
world. Fodor aclmowledges that his Representationalist theory of meaning
requires "a theory that articulates, in nonsemantic and nonintentional
terms, sufficient conditions for one bit of the world to be about (to express,
represent, or be true of) another bit" (Fodor 1987: 98). Typically the first
"bit" would be a symbol in the internal language of thought while the sec-
ond "bit" that it represents might be either some thing or event in the ex-
ternal world or else a brain state underlying a conception of some fictive
entity or scene.
The internal/external split that underlies this view presupposes that
cognition could be detached from the nature and functioning of specific
bodily organisms, from the environments they inhabit and from the prob-
lems that provoke cognition. Given this view, it would follow that cogni-
tion could take place in any number of suitable media, such as a human
brain or a machine. This theoretical viewpoint, functionalism, was instru-
mental in the developing the first electronic calculating machines and gen-
eral-purpose computers. In fact, these machines were originally developed
by the British military to reduce the tedious workload of military mathe-
maticians (or human "computers" - in the sense of humans who compute).
But this thought experiment did not end merely with offloading the tedium
of calculation onto electronic machines. From its original conception in the
work of Alan Turing (1937), the idea of a universal computing machine
became the metaphor of choice for future models of the brain. For example
in Newell and Simon's (1976) conception of the brain as a physical symbol
system, they consider the human brain to be just a specific instance of a
Turing-style universal machine. In short, for classical cognitive science
cognition is defined narrowly as mathematical and logical computation
with intrinsically meaningless internal symbols that can supposedly be
placed in relation to aspects of the external world.
The Pragmatist challenge to classical cognitive science should come as
no surprise, since one of the Pragmatists' chief targets was the tendency
within the philosophical tradition to assume that what demarcates "ra-
tional" humans from "lower" animals is the supposedly unique ability of
humans to engage in symbolic representation between internal thoughts/
language and the external world. The remedy offered by the Pragmatists is
based on their view that cognition is action, rather than mental mirroring of
an external reality. Moreover, cognition is a particular kind of action - a
response strategy that applies some measure of forethought in order to
solve some practical real-world problem. During World War II the practi-
cal problem of breaking the German codes was of utmost importance to the
British war effort, and this led to the development of a series of machines
(the Bombes) which could try a vast number of possible cipher keys
against intercepted German communications. These decoding machines
were among the predecessors of the modem computer. Early computers
were designed to model human action - computing possible cipher keys -
so that machines would replace human labour (Hodges 1983: 160-241).
However, this success in the modelling of a very specific intellectual
operation was soon mistakenly regarded as the key to understanding cog-
nition in general. If one thinks that mathematical and logical reasoning is
what distinguishes human beings from other animals, one might errone-
ously assume that any computational machine that could model aspects of
this peculiarly human trait could also be used to model cognition in gen-
eral. Hence the MIND As COMPUTER metaphor swept early (first-
generation) cognitive science. This is a disembodied view of rationality. By
contrast, on the Pragmatist view, our rationality emerges from, and is
shaped by, our embodied nature. Thus, Dewey famously asserted that "to
see the organism in nature, the nervous system in the organism, the brain in
the nervous system, the cortex in the brain is the answer to the problems
which haunt philosophy" (Dewey 1925: 198).
In the following sections we show how the Pragmatist view of cognition
as action provides an appropriate philosophical framework for the cogni-
tive science of the embodied mind. We begin by describing the non-
dualistic, non-Representationalist view of mind developed by James and
Dewey. Their understanding of situated cognition is reinforced by recent
empirical research and developments within the cognitive sciences. We cite
evidence from comparative neurobiology of organism-environment cou-
pling ranging from the amoeba all the way up to humans, and we argue that
in humans this coupling process becomes the basis of meaning and
thought. We describe the patterns of these ongoing interactions as image
schemas that ground meaning in our embodiment and yet are not internal
representations of an external reality. This leads to an account of an emer-
gent rationality that is embodied, social and creative.
2. James and Dewey: The continuity of embodied experience and
thought
In many ways the American Pragmatist philosophers James and Dewey
provide us today with exemplary non-reductionist and non-Representat-
ionalist models of embodied mind. Their models combined the best biol-
ogy, psychology and neuroscience of their day with nuanced phenomenol-
ogical description and a commitment that philosophy should address the
pressing human problems of our lives. James and Dewey understood
something taken for granted in contemporary biological science: cognition
emerges from the embodied processes of an organism that is constantly
adapting to better utilize relatively stable patterns within a changing envi-
ronment. One problem for such a naturalistic account of mind is to explain
how meaning, abstract thinking, and formal reasoning could emerge from
the basic sensorimotor capacities of organisms as they interact with the
environment and each other.
The fundamental assumption of the Pragmatists' naturalistic approach is
that everything we attribute to "mind" - perceiving, conceptualizing,
imagining, reasoning, desiring, willing, dreaming - has emerged (and con-
tinues to develop) as part of a process in which an organism seeks to sur-
vive, grow and flourish within different kinds of situations. As James puts
it:
Mental facts cannot be properly studied apart from the physical environment
of which they take cognizance. The great fault of the older rational psychol-
ogy was to set up the soul as an absolute spiritual being with certain faculties
of its own by which the several activities of remembering, imagining, rea-
soning and willing, etc. were explained, almost without reference to the pe-
culiarities of the world with which these activities deal. But the richer in-
sight of modem days perceives that our inner faculties are adapted in
advance to the features of the world in which we dwell, adapted, I mean, so
as to secure our safety and prosperity in its midst. (James 1900: 3)
This evolutionary embeddedness of the organism within its changing envi-
ronments, and the development of thought in response to such changes, ties
mind inextricably to body and environment. The changes entailed by such
a view are revolutionary. From the very beginning of life, the problem of
knowledge is not how so-called internal ideas can re-present external reali-
ties. Instead, the problem of knowledge is to explain how structures and
patterns of organism-environment interaction can be adapted and trans-
formed to help deal constructively with changing circumstances that pose
new problems, challenges and opportunities for the organism. On this view,
mind is never separate from body, for it is always a series of bodily activi-
ties immersed in the ongoing flow of organism-environment interactions
that constitutes experience. In Dewey's words:
Since both the inanimate and the human environment are involved in the
functions of life, it is inevitable, if these functions evolve to the point of
thinking and if thinking is naturally serial with biological functions, that it
will have as the material of thought, even of its erratic imaginings, the events
and connections of this environment. (Dewey 1925: 212-213)
Another way of expressing this rootedness of thinking in bodily experience
and its connection with the environment is to say that there is no rupture in
experience between perceiving, feeling and thinking. In explaining ever
more complex "higher" functions, such as consciousness, self-reflection
and language use, we do not postulate new ontological kinds of entities,
events, or processes that are non-natural or super-natural. More complex
levels of organic functioning are just that - levels - and nothing more,
although there are emergent properties of "higher" levels of functioning.
Dewey names this connectedness of all cognition the principle of continu-
ity, which states that "there is no breach of continuity between operations
of inquiry and biological operations and physical operations. 'Continuity'
[...] means that rational operations grow out of organic activities, without
being identical with that from which they emerge" (Dewey 1938: 26).
What the continuity thesis entails is that any explanation of the nature
and workings of mind, even the most abstract conceptualization and rea-
soning, must have its roots in our organismic capacities for perception,
feeling, object manipulation and bodily movement. Furthermore, social and
cultural forces are required to develop these capacities to their full poten-
tial, including language and symbolic reasoning. Infants do not speak or
discover mathematical proofs at birth; Dewey's continuity thesis requires
both evolutionary and developmental explanations. For James and Dewey,
this meant that a full-fledged theory of human cognition must have at least
three major components:
(1) There must be an account of the emergence and development of
meaningful patterns of organism-environment interactions - patterns
of sensorimotor experience shared by all organisms of a certain kind
and meaningful for those organisms. Such patterns must be tied to
the organism's attempts to function within its environment.
(2) There must be an account of how we can perform abstract thinking
using our capacities for perception and motor response. There would
need to be bodily processes for extending sensorimotor concepts and
logic for use in abstract reasoning, as well as an account of how the
processes embodying such abstract reasoning capacities are learned
during organismic development. This story has at least two parts: (a)
an evolutionary and physiological account explaining how an adult
human being's abstract reasoning utilizes the brain's perceptual and
motor systems, and (b) a developmental and anthropological account
of how social and cultural behaviors educate the sensorimotor sys-
tems of successive generations of children so that they may speak
and perform abstract reasoning.
(3) There must be an account of how values and behavioural motiva-
tions emerge from the organism's ongoing functioning. This expla-
nation will include (a) the physical and social makeup of organisms,
(b) the nature of their emotional responses, and (c) the kinds of envi-
ronments (e.g., material, social, cultural) they inhabit. In the present
space we are able to offer only a very compressed and partial treat-
ment of such an account.
3. Organism-environment coupling
3.1. Maturana and Varela: From chemotaxis to the nervous system
Dewey's principle of continuity states that there are no ontological gaps
between the different levels of an organism's functioning. One way to see
what this entails is to survey a few representative types of organism-
environment couplings, starting with single-cellular organisms and moving
up by degrees to more complex animals. In every case we can observe the
same adaptive process of interactive co-ordination between a specific or-
ganism and recurring characteristics of its environment. But does that mean
that we can trace human cognition all the way back to the sensorimotor
behavior of single-cellular organisms? On the face of it, this seems pre-
posterous - viewed from an evolutionary biologist's perspective, there are
clear differences in the size, complexity and structural differentiation of
human beings as compared with single-cellular organisms like bacteria.
Single-cellular organism behaviour is not ordinarily relevant to the behav-
iour of multi-cellular organisms - except insofar as there might be struc-
tural morphological analogies between the sensorimotor activity of single-
cellular organisms and particular sensorimotoric cells within the multi-
cellular body.
Just this sort of morphological analogy plays a key role in Maturana and
Varela's argument that central nervous systems evolved in multi-cellular
organisms to co-ordinate sensorimotor activity (Maturana and Varela 1998:
142-163). In a single-cellular organism locomotion is achieved by dynami-
cally coupling the sensory and motoric surfaces of the cell membrane.
When an amoeba engulfs a protozoan, its cell membranes are responding to
the presence of the chemical substances that make up the protozoan, caus-
ing changes in the consistency of the amoeba's protoplasm. These changes
manifest as pseudopods - digitations that the amoeba extends around the
protozoan as it prepares to feed upon it. Similarly, certain bacteria have a
tail-like membrane structure called a flagellum that is rotated like a pro-
peller to move the bacterium. When the flagellum is rotated in one direc-
tion the bacterium simply tumbles, while reversing the direction of rotation
causes the bacterium to move. If a grain of sugar is placed into the solution
containing this bacterium, chemical receptors on the cell membrane sense
the sugar molecules. This causes a membrane change in which the bacte-
rium changes the direction of rotation of its flagellar propeller and gradu-
ally moves toward the greatest concentration of the sugar molecules (che-
motaxis). In both cases, changes in the chemical environment cause sen-
sory perturbations in the cellular membrane, which invariably produces
movement. The key point here is that, without anything like an internal
representation, single-cellular organisms engage in sensorimotor co-
ordination in response to environmental changes. Even at this apparently
primitive level, there is a finely tuned ongoing coupling of organism and
environment.
Multi-cellular organisms also accomplish their sensorimotor co-
ordination by means of changes in their cell membranes. However, the
cellular specialization afforded by a multi-cellular organism means that not
every cell needs to perform the same functions. Maturana and Varela
(1998) discuss the example of an evolutionarily ancient metazoic organism
called the Hydra (a coelenterate). The Hydra, which lives in ponds, is
shaped like a two-layered tube with four or six tentacles emanating from its
mouth. On the inside layer of the tube, most cells secrete digestive fluids,
while the outside layer is partly composed of radial and longitudinal mus-
cle cells. Locomotion is accomplished by contracting muscle cells along
the body of the organism: some of these contractions cause changes in the
hydrostatic pressure within the organism, changing its shape and direction
of locomotion.
Between the two layers of cells, however, are specialized cells - neu-
rons - with elongated membranes that can extend over the length of the
entire organism before terminating in the muscle cells. These tail-like cel-
lular projections are the axons, and evolutionarily speaking they are the
flagella of the multi-cellular organism.! Changes in the electrochemical
state in other, smaller cellular projections of the cells (the dendrites) cause
larger changes in the electrochemical state of the axonal membrane, which
in turn induces the muscle cells to contract. These neural signals typically
originate in either the tentacles or the "stomach" of the Hydra, such that
their electrochemical state responds to the molecules indicating the pres-
ence or absence of food and/or excessive digestive secretions. These neu-
rons consistently terminate in the longitudinal and radial muscles that con-
tract the Hydra body for locomotion or for swallowing. The topology of
how the nerve cells interconnect is crucially important: when touched, a
chain of neurons fire sequentially down a Hydra tentacle toward its mouth
1. Recent research shows that this may be more than a surface morphological
analogy: all microtubular cellular projections stem from a common ancestor
(Erickson et al. 1996; Goldberg 2003).
and cause the muscle cells to curl the tentacle about its prey even as its
mouth begins to open. The Hydra does not "represent" an external world;
instead, the structural coupling between organism and environment allows
the Hydra to contract the correct muscles to swallow, or to move up and
left, or right and down. Like the Hydra opening its mouth as a reflexive
part of bringing food to it with its tentacles, we humans think in order to
act and we act as part of our thinking - cognition is action. But how is it
that we humans can learn new behaviors, while the Hydra generally can-
not?
3.2. From neural maps to neural plasticity
Although still surprisingly continuous with the Hydra, human cognition is
a little more similar to what happens in frogs, owls and monkeys in that all
of these organisms have nervous systems that include neural maps and
adaptive neural plasticity. Frogs have a certain regularly occurring prag-
matic problem - they need to extend their tongues to eat a fly - which was
the subject of a classic experiment in the early history of neurobiology
(Sperry 1943). When a frog is still a tadpole, it is possible to rotate the
frog's eye 180 degrees, making sure to keep the optic nerve intact. The
tadpole is then allowed to develop normally into a frog. The frog's tongue
extends to exactly the opposite point of the frog's visual field from where
the fly is located. No amount of failure at catching the fly will teach the
frog to move its tongue differently; the frog acts entirely on the basis of the
rewired neural connections between the retinal image and the tongue mus-
cles. Maturana and Varela conclude thatfor thefrog "there is no such thing
as up or down, front and back, in reference to an outside world, as it exists
for the observer doing the study" (1998: 125-126). The frog has no access
to our notion of the external world and our 180-degree rotation of its eye; it
has only its experience of the world found in the neurons comprising its
(experimentally inverted) retinal map.
One of the most profound findings in neuroscience is that nervous sys-
tems exploit topological and topographic organization. In other words,
organisms build neural "maps". In neural maps, adjacent neural cells (or
small groups of neural cells) fire sequentially when a stimulus in adjacent
positions within a sensory field moves. For example, scientists have
stimulated the frog's visual field and measured the electrical activity of a
region of its brain to show that as one stimulates the frog's visual field, the
neurons of its optic tectum will fire in co-ordination with the visual stimu-
lus. Fraser (1985) covered the frog's optic tectum with a 24-electrode grid,
with each electrode recording electrical activity that was the sum of the
signals from a receptive field containing many optic nerve fiber terminals.
When a point of light was moved in a straight line from right to left and
then from bottom to top in the frog's right visual field, the electrode grid
recorded neuronal activity in straight lines, firing sequentially, first from
the rostral (front) to the caudal (back) and then from the lateral to the me-
dial. We call this the frog's retinal (or retinotectal) map because it encodes
environmental visual stimuli in a topographically consistent manner. The
spatial orientation of this topography is rotated in various ways. Thus vis-
ual right-to-Ieft has become front-to-back and so on, but the topographic
mapping between movement in the vertical visual plane and the plane of
the retinotectal neural map remains consistent. Even though there is con-
siderable spatial distortion in the neural map, the key relational structures
are preserved. In some other cases, such as some auditory maps and colour
maps, where the correspondences can be less about shape and position, the
organization is more properly called topologic than topographic, but the
organizing principle of the neural mapping of sensation still holds.
The degree to which such neural maps might be plastic has been the
subject of much recent study. In the case of rotating the eye of the frog,
Sperry performed a radical and destructive intervention that is outside the
realm of "normal" Darwinian deviation - in other words, if this were to
occur by natural selection such a frog would die quickly without passing on
its genes. However, interventions which are less radical and perhaps more
likely to occur in nature, such as cutting the optic nerve and destroying part
of the optic tectum of a goldfish, result in a recovery of function in which
the optic nerve axons regenerate to make a complete retinal map in the
remaining part of the tectum (Gaze and Sharma 1970). Although radical
interventions can "break" the neural maps, even the more evolutionarily
determined neural networks exhibit some range of adaptive neural plastic-
ity to environmental factors.
Plasticity is particularly profound in cross-modal neural maps. Consider
another subtle environmental intervention: suppose we were to have an owl
wear glasses that changed its perception of the visual field. Similar to the
frog, owls have developed an extremely accurate method of attacking prey.
The owl hears a mouse rustling on the ground and locates the mouse using
the tiny difference in time it takes for a sound to reach one ear versus the
time it takes the sound to reach the owl's other ear. This establishes the
mouse's approximate position in the owl's retinotectal map, and the diving
owl then visually confirms the exact location of its prey before it strikes.
Knudsen and colleagues (Knudsen 1998, 2002) put prismatic glasses on
adult and juvenile owls which distorted the owls' vision by 23 degrees.
After 8 weeks with glasses, adults raised normally never learned to com-
pensate, but juveniles were able to learn to hunt accurately. Moreover,
when the glasses were reintroduced to the adult owls who had worn them
as juveniles, they were then able to readjust to the glasses in short order; in
other words, the prism-reared owls could successfully hunt with or without
glasses.
These behavioural adaptations have anatomical underpinnings in the
plasticity of the neural maps. When injected with an anatomical tracing
dye, comparison of the neural arbors from normally-reared and prism-
reared owls revealed a different pattern of axonal projections between
auditory and spatial neural maps, "showing that alternative learned and
normal circuits can coexist in this network" (Knudsen 2002: 325). In other
words, in order to deal with wearing glasses, the owl brain had grown per-
manent alternative axonal connections in a cross-modal neural map of
space located in the external nucleus of the inferior colliculus (ICX). The
ICX neural arbor of prism-reared owls was significantly denser than in
normally developing owls, with neurons typically having at least two dis-
tinct branches ofaxons (DeBello, Feldman and Knudsen 2001). By con-
trast, the retinotectal maps of the visual modality alone do not exhibit the
same plasticity, either in owls (whose retinotectum did not change) or in
frogs. Analogous anatomical research on frogs reared and kept alive with
surgically rotated eyes has shown that after five weeks, the retinotectal
neural arbors initially exhibited a similar pattern of "two-headed axons" -
that is, they had two major axonal branches. However, after ten weeks the
older axonal connections are starting to decay and disappear, while after
sixteen weeks no two-headed axons could be traced (Guo and Udin 2000).
Apparently, the frog's single-modal retinotectal maps do not receive
enough reentrant neural connections from other sensory modalities to sus-
tain the multiple branching neural arbors found in the cross-modal map of
the prism-reared owls.
Working on neural plasticity in adult squirrel and owl monkeys, Mer-
zenich and colleagues (Merzenich et al. 1987; reviewed in Buonomano and
Merzenich 1998) have shown that it is possible to dynamically reorganize
the sensorimotor cortical maps subject to certain bodily constraints. Similar
to the owls and frogs that grew dual arborizations, these monkeys exhibited
a plasticity based on their brains' ability to select which parts of their neu-
ral arbors to use for various kinds of input. In a series of studies, Mer-
zenich and colleagues altered the monkey's hand sensory activity by such
interventions as (1) cutting a peripheral nerve such as the medial or radial
nerve and (la) allowing it to regenerate naturally or (lb) tying it off to
prevent regeneration; (2) amputating a single digit; and (3) taping together
two digits so that they could not be moved independently. The results show
that cortical areas now lacking their previous sensory connections (or inde-
pendent sensory input in the third condition) were "colonized" in a couple
of weeks by adjacent neural maps with active sensory connections. In other
words, the degree of existing but somewhat dormant neural arbor overlap
was large enough to permit reorganization. And in the case of (la), where
the nerve was allowed to regenerate, the somatosensory map gradually
returned to occupy a similar-sized stretch of cortex, albeit with slightly
different boundaries. Learning in adults is accomplished in part by neural
gating between redundant and overlapping neural arbors.
All of these examples of ontogenetic neural change suggest that there is
a process of neural arbor selection akin to natural selection taking place in
concert with specific patterns of organism-environment interactions. On
precisely these grounds the neurobiologist Gerald Edelman (1987) has
proposed a theory of "Neural Darwinism", or "neuronal group selection",
to explain how such neural maps are formed in the organism's embryonic
development. Different groups of neurons compete to become topological
neural maps as they migrate and grow during neural development. Success-
ful cortical groups, driven primarily by regularities in the environment
passed on from those neurons that are closer to a sensory apparatus, will
fire together and wire together in a process of axonal sprouting and synap-
togenesis. Some neuronal groups will fail to find useful topological con-
nections, and they eventually die and are crowded out by the successful
neuronal groups, while others will hang on in something of an intermediate
state of success (Edelman 1987: 127-140). In the adult organism, the latent
axonal arbors from only partly successful attempts to wire together lay
dormant, ready to reorganize the map as needed by means of further syn-
aptogenesis. Edelman (1987: 43-47) calls these latent reorganizations of
the neuronal groups secondary repertoires, as distinguished from their
normal primary repertoires.
Like frogs, owls and monkeys, we humans have sets of visual, auditory
and somatosensory neural maps. The more obvious of these map perceptual
space in fairly direct analogs - preserving topologies of pitch, the retinal
field, colour, the parts of the body and so on - but subsequent maps pre-
serve increasingly abstract topological structure (or even combinations of
structure) such as object shape, edges, orientation, direction of motion and
even the particular degree of the vertical or horizontal. Like the frog, we
live in the world of our maps. Topologically speaking, our bodies are in
our minds, in the sense that our sensorimotor maps provide the basis for
conceptualization and reasoning. We perceive the patterns of our daily
organism-environment interactions in image-like fashion, constantly seek-
ing out various topological invariances in those patterns that prove useful
to us. In the following section we will show how our imagination and our
reason are constituted by patterns of activation within these neural maps.
But before proceeding to human cognition, we must first address why neu-
ral "maps" are not classical Representations.
3.3. Neural maps are not internal representations
Some people might suppose that talk of neural "maps" would necessarily
engender Representationalist theories of cognition. On this view, the map
would be construed as an internal representation of some external reality.
But the account we have been giving does not entail any of the traditional
metaphysical dualisms that underlie Representationalist views - dichoto-
mies such as inner/outer, subject/object, mind/body, self/world. Such di-
chotomies might describe aspects of organism-environment interactions
from an observer's perspective, but they do not indicate different ontologi-
cal entities or structures. According to our interactionist view, maps and
other structures of organism-environment co-ordination are prime exam-
ples of non-representational structures of meaning, understanding and
thought.
2
2. We are certainly not suggesting that neuroscientists should purge the term "rep-
resentation" from their vocabulary. Nor are we suggesting that there is no sense
in which it would be appropriate to say that some neuronal structure is a repre-
sentation from the perspective of the scientist who is studying cognitive proc-
esses. For example, we do not object to neuroscientists saying that a particular
neural map in the auditory cortex can "represent" various pitch relations among
musical tones, though we prefer to employ more enactive terms such as "map"
and "activation contours". However, such casual usage doesn't necessarily en-
tail the Representational Theory of Mind that we are challenging here. Instead,
we argue that Representationalism is based on a mistaken philosophical analogy
Maturana and Varela (1998: 125-126) make this important philosophi-
cal point quite clear. We must not read our scientific or philosophical per-
spectives (i.e., our theoretical stance) on cognition back into the experience
itself that we are theorizing about. We must not uncritically assume that
distinctions we make in explaining a certain cognitive experience are
thereby part of the person's experience. To do so is to fall prey to what
James termed the "Psychologist's Fallacy". In observing something scien-
tifically, one must always consider the standpoint of the scientist in relation
to the object of study. When we use terms such as "retinal map", "pitch
maps", "sensorimotor maps", "color maps" and so forth to describe the
operations of various neural arrays in a frog's nervous system, or in human
nervous systems, we are doing so from our standpoint as observers and
theorists who can see mappings between those neural structures and our
own experience of the "external world". But for the frog, and for the hu-
man in the act of perceiving, that map is the basis for its experience of the
world. The map constitutes the sensorimotor experience of a certain part of
the frog's world. The frog's neural map itself has its origin not in the im-
mediate mappings that we observers see in the moment, but in a longitudi-
nal evolutionary and developmental process during which those neural
connections were "selected for" by Darwinian or neo-Darwinian mecha-
nIsms.
In short, what we (as scientists) theoretically recognize and describe as
an organism's "maps" are not for that organism internal representations.
Rather, what we' call sensorimotor and somatosensory maps (whether in
multi-cellular organisms, monkeys, or humans) are for that organism pre-
cisely the structures of its experienced world! Consequently, we must be
careful not to be misled by philosophers of mind and language who would
treat these maps as internal representations of external realities, thereby
surreptitiously introducing an "inner/outer" split that does not exist in real-
ity for the organism.
(namely "the language of thought" framework in which a mental state refers to
the world much as a word supposedly simply refers to an object or a state-of-
affairs in the world). In order to undermine such Representationalist theories,
we argue that actual neural representations are perpetually situated in dynamic
organism-environment interactions that are continually changing along experi-
ential, developmental and phylogenetic timelines. Hence, it is a mistake to think
that neural maps are representations in virtue of an immediate word-world ref-
erential mapping, whether that word is a linguistic entity or a mental entity in a
"language of thought".
4. Ontological continuity and human thought: Image schemas and
amodal perception
Since the earliest episodes of ancient Greek philosophy, humans have been
distinguished from "brute" animals and all lower organisms by their sup-
posedly unique capacity for abstract conceptualization and reasoning. Ac-
cording to this view, human reason is what makes it possible for us to form
abstract mental representations that stand for and point to states of affairs
that are either external to us or are not currently present in our experience
(i.e., are past or future). But the Pragmatists' Continuity Thesis denies the
inner/outer dichotomy upon which Representationalist theories are based.
Consequently, the problem for an embodied view of cognition is how to
explain our marvellous human feats of abstraction, reasoning and symbolic
interaction, yet without positing an ontological rupture between "lower"
animals and humans.
The key, once again, is the coupling (the interactive co-ordination) of an
organism (here, a human one) and its environment. Recurring adaptive
patterns of organism-environment interaction are the basis for our ability to
survive and flourish. In humans, these patterns are no more "internal" rep-
resentations than they are in other creatures. Let us consider briefly some
of the most basic kinds of structural couplings that make up a human be-
ing's experience of its world.
4.1. Image schemas and cross-modal perception
The character of our experience is delineated in large part by the nature of
our bodies and brains, the kinds of environments we inhabit, and the values
and purposes we have. The patterns of our ongoing interactions (or "enac-
tions" as Varela, Rosch and Thompson (1991) have called them, to stress
their active, dynamic character) define the contours of our world and make
it possible for us to make sense of, reason about, and act reliably within
this world. Thousands of times each day we see, manipulate and move into
and out of containers, so containment is one of the most fundamental pat-
terns of our experience. Because we have two legs and stand up within a
gravitational field, we experience verticality and up-down orientation. Be-
cause the qualities (e.g., redness, softness, coolness, agitation, sharpness)
of our experience vary continuously in intensity, there is a scalar vector in
our world. For example, lights can grow brighter or dimmer, stoves get
hotter or cooler, iced tea gets sweeter as we add sugar. We are subject to
forces that move us, change our bodily states and constrain our actions, and
all of these forces have characteristic patterns and qualities. We are bound
inextricably to our world interactively (enactively) by means of these re-
curring patterns that are the very conditions for us to survive, grow and
find meaning. Without such patterns, and without neural maps of such
characteristic patterns, each moment of our experience would be utterly
chaotic, as though we had to make sense of our world from scratch, over
and over again as each new moment arose.
What Johnson (1987) and Lakoff (1987) called "image schemas" are
precisely these stable recurring patterns of sensorimotor experience by
which we engage a world that we can understand and act within to further
our purposes. There are numerous sources of evidence for the existence of
image schemas, ranging from experimental psychology to linguistics to
developmental psychology. We hypothesize that these image schemas are
neurally embodied as patterns of activation in and between our topological
neural maps. Image schemas are thus part of our non-representational cou-
pling with our world, just as barn owls and squirrel monkeys have image
schemas that define their types of sensorimotor experience.
Image schematic structure is the basis for our understanding of all as-
pects of our perception and motor activities. An example from Lakoff and
Nufiez (2000) illustrates this image-schematic basis of spatial concepts in
humans. What we call our concept IN is defined for us by a CONTAINER
image schema that consists generically of (1) a boundary that demarcates
(2) an interior from (3) an exterior.
When we say The car is in the garage, we understand the garage as a
bounded space, we profile (Langacker 1986) the interior of that space, and
we regard the car as what cognitive linguists call a trajector within that
space, with the garage (as container) serving as a landmark in relation to
which the trajector is located. Similarly, when we hear the sentence
Grandpa walked from the outhouse to the garage, we understand that
situation via a SOURCE-PATH-GOAL schema that consists of (a) a starting
point, (b) a destination (endpoint), and (3) a path from the starting location
to the destination. In other words, the from-to construction is image-
schematic. The English word into is understood via a superimposition of
the SOURCE-PATH-GOAL schema on the CONTAINER schema, as follows:
in activates a CONTAINER schema with the interior profiled.
to activates a SOURCE-PATH-GOAL schema with the GOAL (endpoint)
profiled.
The GOAL (endpoint) is mapped onto the interior of the CONTAINER
schema.
We thus understand Grandpa's (as trajector) movement as beginning
outside the garage (CONTAINER) and terminating inside the garage
(as landmark), as a result of motion along a path from the exterior to
the interior.
into in English is thus an elementary composition of two image
schemas.
Image schemas are realized as activation patterns (or "contours") in human
topological neural maps. As with much interdisciplinary research in the
neurosciences, the evidence for this first emerged from intracranial neu-
ronal recordings on monkeys and was later extended to humans via analo-
gous neuroimaging studies. When Rizzolatti and colleagues (Fogassi et al
2001; see review in Rizzolatti, Fogassi and Gallese 2002) showed macaque
monkeys visual imagery of another monkey grasping a banana with their
hands, they were able to record activity from "mirror" neurons in the same
areas of secondary somatomotor cortex that would be implicated if the
monkey himself were performing the particular grasping action. Analogous
human neuroimaging experiments (Buccino et al. 2001) in which partici-
pants watched a video clip of another person performing an action showed
increased activation in the human secondary somatomotor cortices that are
known to map human hand and arm grasping motions. Along with Rizzo-
latti's colleague Gallese, we interpret these and related results as having
shown that these neural maps contain image schematic sensorimotor acti-
vation patterns for grasping (Gallese and Lakoff 2005).
An explicit attempt to model image schemas using known facts about
our neural maps can be found within the neurocomputational modelling
literature. Regier (1996) has developed what he calls "structured" or "con-
strained" connectionist neural models for a number of image schemas.
"Constrained" neurocomputational connectionism builds into its neural
models a small number of structures that have been identified in research
on human visual and spatial processing. These include center-surround cell
arrays, spreading activation, orientation-sensitive cells and neural gating.
Regier has shown how these constrained connectionist models of image
schemas can learn spatial relations terms.
3
There is also a growing body of research from developmental psychol-
ogy suggesting that infants come into the world with capacities for experi-
encing image-schematic structures. Stem (1985) described certain types of
experiential structures that infants are able to detect, and he argues, first,
that these capacities form the basis for meaning and the infant's sense of
self; and, second, that these capacities continue to play a central role in
meaning, understanding and thinking even in adults who are capable of
propositional thinking. Let us briefly consider two of these basic struc-
tures: (1) cross-modal perception, and (2) vitality affect contours.
Stem begins with a well-known experiment (Meltzoff and Borton 1979)
in which blind-folded infants were given one of two pacifiers to suck. One
was the typical smooth pacifier, while the other had protruding nubs. When
the blindfolds were removed and smooth and nubbed pacifiers were placed
on either side of the infant's head, most of the time (roughly 75%) the in-
fant would attend to the nipple of the pacifier just sucked. Based on this
and other studies (e.g., Lewkowicz and Turkewitz 1981), Stem suggests
that
Infants thus appear to have an innate general capacity, which can be called
amodal perception, to take information received in one sensory modality
and somehow translate it into another sensory modality. [...]
These abstract representations that the infant experiences are not sights and
sounds and touches and nameable objects, but rather shapes, intensities and
temporal patterns - the more "global" qualities of experience. (Stem 1985:
51)
Although he speaks of these structures of cross-modal perception as amo-
dal, abstract "representations", Stem also makes it clear that these percep-
tual structures are not inner mirrorings of external things but rather are the
contours of the infant's experience: the cross-modal shapes, intensities and
temporal patterns that we call image schemas.
Like infants, we adults have a ROUGH/SMOOTH image schema, which
we can use as we anticipate the change in surface texture as we walk. For
3. This does not, of course, prove that human cognition necessarily works this
way, but Regier's use of computational neural models built on known human
neural architectures offers a number of advantages over traditional PDP
connectionist models. Moreover, Regier's models can be appropriated into pro-
grams that allow robots to perform certain bodily movements.
example, we can see where we will step from the rough carpet of the hall-
way onto the slippery tile of the bathroom, and we transfer this information
from the visual to the somatomotor system so that our feet will not slip.
Such patterns of cross-modal perception are especially clear examples of
how image schemas differ from being just a topographically mapped image
in a neural map; they are sensorimotoric patterns of experience which are
instantiated in and coordinated between unimodal neural maps. Our image
schematic experience may, as in the case of the owl, become instantiated in
its own cross-modal neural map; or, as in the case of monkeys, it might
consist of coordinated activation patterns between a network of more mo-
dal neural maps, including possibly calling on the secondary rather than
primary repertoires of those maps. We predict that cases analogous to each
will be observed in human neuroanatomical studies.
A second type of pattern that makes up the infant's (and adult's) image-
schematic experience is what Stem (1985) calls "vitality affect contours".
Stem illustrates this with the notion of a "rush", or the swelling qualitative
contour of a felt experience. We can experience an adrenaline rush, a rush
of joy or anger, a drug-induced rush, or the rush of a hot-flash. Even
though these rushes are felt in different sensory modalities, they are all
characterizable as a rapid, forceful building up or swelling contour of the
experience across time. Stem notes that understanding how such affect
contours are meaningful to creatures like us gives us profound insight into
meaning generally, whether that meaning comes via language, vision, mu-
sic, dance, touch, or smell. We crave the emotional satisfaction that comes
from pattern completion, and witnessing even just a portion of the pattern
is enough to set our affect contours in motion. The infant just needs to see
us begin to reach for the bottle, and she already begins to quiet down - the
grasping image schema does not even need to be completely realized in
time before the infant recognizes the action. When as adults we hear a mu-
sical composition building up to a crescendo, this causes increasing emo-
tional tension that is released at the musical climax. The emotional salience
of the vitality affect contours in image schemas shows that image schemas
are not mere static "representations" (or "snapshots") of one moment in a
topographic neural map (or maps). Instead, image schemas proceed dy-
namically in and through time.
To summarize, image schemas can be characterized more formally as:
(1) recurrent patterns of bodily experience,
(2) "image"-like in that they preserve the topological structure of the
perceptual whole, as evidenced by pattern-completion,
(3) operating dynamically in and across time,
(4) realized as activation patterns (or "contours") in and between topo-
logic neural maps,
(5) structures which link sensorimotor experience to conceptualization
andlanguage,and
(6) structures which afford 'normal' pattern completions that can serve
as a basis for inference.
Image schemas constitute a preverbal and pre-reflective emergent level of
meaning. They are patterns found in the topologic neural maps we share
with other animals, though we as humans have particular image schemas
that are more or less peculiar to our types of bodies. However, even though
image schemas typically operate without our conscious awareness of how
they structure our experience, it is sometimes possible to become reflec-
tively aware of the image-schematic structure of a certain experience, such
as when I am consciously aware of my cupped hands as forming a con-
tainer, or when I feel my body as being off balance.
4.2. Abstract conceptualization and reasoning
Pragmatism's Continuity Thesis claims that we must be able to move,
without any ontological rupture, from the body-based meaning of spatial
and perceptual experience that is characterizable by image schemas and
affect contours, all the way up to abstract conceptualization, reasoning and
language use. Although there is not yet any fully worked out theory of how
all abstract thought works, some of the central mechanisms are becoming
better understood. One particularly important structure is conceptual meta-
phor (Lakoff and Johnson 1980; 1999). The most sweeping claim of Con-
ceptual Metaphor Theory (CMT) is that what we call "abstract" concepts
are defined by systematic mappings from bodily-based sensorimotor source
domains onto abstract target domains. These metaphor mappings are found
in patterns motivated by image schematic constraints - for example, if we
map an interior from the source domain, we can expect to map the exterior
as well; if we have source and destination mappings, we can expect a path
mappIng.
Consider the sentence We have a long way to go before our theory is
finished. Why can we use the phrase a long way to go, which is literally
about distance in motion through space, to talk about the completion of a
mental project (i.e., developing a theory)? The answer is that there is a
conceptual metaphor PURPOSEFUL ACTIVITIES ARE JOURNEYS, via which
some cultures understand progress toward some nonphysical goal as prog-
ress in moving toward a destination. The metaphor consists of the follow-
ing conceptual mapping:
The PURPOSEFUL ACTIVITIES ARE JOURNEYS METAPHOR
Source (motion in space) Target (mental activity)
Starting point A -7 Initial state
Ending location B -7 Final State
Destination -7 Purpose to be achieved
Motion from A to B -7 Process of achieving purpose
Obstacles to motion -7 Difficulties in achieving goals
This conceptual mapping also makes use of one of our culture's most basic
metaphors for understanding the passage of time, in which temporal change
is understood metaphorically as motion along a path to some location. In
this metaphor, the observer moves along a time line, with the future ar-
rayed as the space in front of her and the past as the space behind. Conse-
quently, when we hear We have a long way to go until our campaign fund
drive is finished, we understand ourselves metaphorically as moving along
a path toward the destination (completion of the fund drive), and we under-
stand that there can be obstacles along the way that would slow our prog-
ress.
Conceptual Metaphor Theory proposes that all abstract conceptualiza-
tion works via conceptual metaphor, conceptual metonymy and a few other
principles of imaginative extension. To date there is a rapidly growing
body of metaphor analyses of key concepts in nearly every conceivable
intellectual field and discipline, including the physical and biological sci-
ences, economics, morality, politics, ethics, philosophy, anthropology,
psychology, religion and more. For example, Lakoff and Nlinez (2000)
have carried out extensive analyses of the fundamental metaphorical con-
cepts that underlie mathematics, from simple models of addition all the
way up to concepts of the Cartesian plane, infinity and differential equa-
tions. Winter (2001) analyzes several key metaphors that define central
legal concepts and are the basis for legal reasoning. Grady (1997) examines
"primary metaphors" (such as PURPOSES ARE DESTINATIONS) that are
combined systematically into more complex metaphors (such as PUR-
POSEFUL ACTIVITIES ARE JOURNEYS).
The reason that conceptual metaphor is so important is that it is our
primary means for abstract conceptualization and reasoning. Pragmatism's
principle of continuity claims that abstract thought is not disembodied;
rather, it must arise from our sensorimotor capacities and is constrained by
the nature of our bodies, brains and environments. From an evolutionary
perspective this means that we have not developed two separate logical and
inferential systems, one for our bodily experiences and one for our abstract
reasoning (as a pure logic). Instead, the logic of our bodily experience pro-
vides all the logic we need in order to perform every rational inference that
we do. In our metaphor-based reasoning, the inferences are carried out via
the corporeal logic of our sensorimotor capacities, and then, via the source-
to-target domain mapping, the corresponding logical inferences are drawn
in the target domain.
For example, there is definite spatial or bodily image-schematic logic of
containment that arises in our experience with containers:
(a) An entity is either inside the container or outside it, but not both at
once.
(b) If I place an object 0 within a physical container C and then put
container C inside of another container D, then 0 is in D.
In other words, our bodily encounters with containers and objects that we
observe and manipulate teach us the spatial logic of containers.
Next, consider the common conceptual metaphor CATEGORIES ARE
CONTAINERS, in which a conceptual category is understood metaphorically
as an abstract container for physical and abstract entities. For example, we
may say: The category 'human' is contained in the category 'animals,'
which is contained in the category 'living things. ' Similarly, we may ask
Which category is this tree in? Based on the inferential image-schematic
structure of the source domain, and via the source-to-target mapping, we
then have corresponding inferences about abstract concepts:
(a') An entity either falls within a given category, or falls outside it, but
not both at once [e.g., Charles cannot be a man and not a man at the
same time, in the same place, and in the same manner]. (The Law of
the Excluded Middle).
(b') If an entity E is in one category C', and C' is in another category D',
then that entity E is in category D' [For example, All men are mortal
(C' is in D ~ and Socrates is a man (E is in C ~ , therefore Socrates is
mortal (E is in D ~ ] .
Thus, according to CMT we would then predict that the abstract inferences
are "computed" using sensorimotor neural maps, and those inferences are
activated as target-domain inferences because there are neural connections
from sensorimotor areas of the brain to other areas that are responsible for
so-called "higher" cognitive functions. The hypothesis is that human be-
ings don't run an inferential process at the sensorimotor level and then
perform an entirely different inferential process for abstract concepts;
rather, human beings utilize the inference patterns found in the sensorimo-
tor brain regions to perform "abstract" reasoning. Just as the Pragmatist
Principle of Continuity requires, there is no need to introduce a new kind
of reasoning (with a different ontological basis) to explain logical reason-
ing with abstract concepts.
4.3. Evidence for conceptual metaphor and abstract reasoning using
conceptual metaphors
Recently several new sources of evidence have become available to explain
the possible neural bases for the image-schematic mappings that operate in
conceptual metaphors. The new evidence comes from both the patient-
based neurological literature and neuroimaging studies of normal adults.
While we have long known that patients can develop anomias reflecting
selective category deficits for animals, tools and plants (Warrington and
Shallice 1984), several recent studies have reported a selective category
deficit for body part terms (Suziki, Yamadori and Fujii 1997; Shelton,
Fouch and Caramazza 1998; Coslett, Saffran and Schwoebel 2002). The
deficit work suggests that lesions in the secondary motor cortices, in re-
gions which likely contain both somatotopic and egocentric spatial maps,
can cause difficulties in tasks such as body part naming, naming contigu-
ous sections of the body, and so on. This finding suggests that the compre-
hension of body part terms requires the active participation of these neural
maps.
Two other neuroimaging studies also show that we can drive the human
somatomotor maps with both metaphoric and literal linguistic stimuli re-
lating to the body. In an fMRI study, Hauk, Johnsrude and Pulvermiiller
(2004) have shown that single word terms such as "smile", "punch" and
"kick" differentially activate face, armlhand and leg regions within the
somatomotor maps, suggesting that literal language can differentially acti-
vate body-part related somatomotor neural maps. Similarly, an fMRI neu-
roimaging study by Rohrer (2001b, 2005) shows that both literal and meta-
phoric sentences using hand terms (e.g., She grasped the apple and He
grasped the theory) activate primary and secondary hand regions within the
primary and secondary sensorimotor maps. After the presentation of the
linguistic stimuli, Rohrer also mapped the hand somatic cortex of each
study participant using a tactile hand-stroking task. A comparison between
the tactile and the sentential conditions shows a high degree of overlap in
the primary and secondary somatomotor cortex for both language tasks, cf.
Figure 1.
Metaphoric Hand Sentences
Figure 1. fMRI activation courses in response to literal and metaphoric action
sentences. Areas active and overlapping from a hand somatosensory
task were outlined in white (Rohrer 2001b).
There is also evidence from neurocomputationally inspired models of con-
ceptual metaphor and abstract reasoning. Building on Regier's work on
modelling the image-schematic character of spatial relation terms, Naraya-
nan (1997; Feldman and Narayanan 2004) developed a constrained
connectionist network to model how the bodily logic of our sensorimotor
systems enables us to perform abstract reasoning about international eco-
nomics using conceptual metaphors. For example, the system was able to
successfully interpret both In 1991, the Indian government deregulated the
business sector and In 1991, the Indian government loosened its strangle-
hold on business. Narayanan's model can perform inferences either entirely
within the sensorimotoric domain or in the linguistic domain using com-
mon conceptual metaphor mappings. Taken together with the neurophysi-
ological and neuroimaging evidence for image schemas and conceptual
metaphors, these neurocomputational models support the image-schematic
and metaphoric basis of our language and abstract reasoning.
5. The continuity of embodied social and cultural cognition
In this chapter, we have been presenting evidence for the embodied char-
acter of cognition, and we have suggested an appropriate Pragmatist philo-
sophical framework for interpreting that evidence. Contra Representation-
alism, we have argued that cognition is not some inner process performed
by the "mind", but rather is a form of embodied action. We argued this by
giving examples of how cognition is located in organism-environment in-
teractions, instead of being locked up in some allegedly private mental
sphere of thought. However, an exclusive focus on the organism's engage-
ment and coupling with its environment can lead to the mistaken impres-
sion that thought is individual, not social. Therefore, we must at least
briefly address the crucial fact that language and abstract reasoning are
socially and culturally situated activities.
Thus far, we have discussed only one socio-cultural dimension, albeit a
crucially important one, namely, development. Our brief discussion of
development was framed more within the context of nervous systems than
within socio-cultural interactions. We stressed the point that epigenetic
bodily interactions with the world are what shape our neural maps and the
image schemas in them. For humans, a very large and distinctive part of
that involves interacting with other humans. In other words, human under-
standing and thinking is social. This raises the question: How do socially
and culturally determined factors come to play a role in human cognition?
Perhaps a sceptic might say that the locus of the distinctively human lies
in a socially and culturally learned capacity for classical Representational-
ism. Once again, however, the Representationalist proposal rests on two
mistakes. First, there is not a radical ontological break from the rest of the
animal kingdom with respect to socially and culturally transmitted behav-
iors, both in general and specifically in the cases of linguistic and symbolic
communication. Second, having challenged the "inner mind" versus "outer
body" split, we must not then proceed to replace it with another equally
problematic dichotomy - that between the "individual" and the "social".
We must recognize that cognition does not take place only within the brain
and body of a single individual, but instead is partly constituted by social
interactions and relations. The evidence to which we now turn comes from
cognitive ethology and distributed cognition. Of course there are ways in
which our socio-cultural behaviors are peculiarly human, but the story is
once again much more complex and multi-dimensional than classical Rep-
resentationalists suppose.
Following Maturana and Varela (1998: 180-184) we would define so-
cial phenomena as those phenomena arising out of recurrent structural
couplings that require the co-ordinated participation of multiple organisms.
They argue that just as the cell-to-cell interactions in the transition from
single to multi-cellular organisms afford a new level of intercellular
structural coupling, so also recurrent interactions between organisms afford
a new level of inter-organism structural coupling.
The social insects are perhaps the most basic example of this kind of re-
current inter-organism behaviors. For example, ants must feed their queen
for their colony to remain alive. Individual workers navigate their way to
and from the nest and food sources by leaving trails of chemical markers,
but these markers are not distinctive to the individual ant. When seeking
food, an individual ant moves away from markers dropped by other ants.
Naturally the density of such markers decreases in proportion to the dis-
tance from the nest. But when one finds food they begin to actively seek
denser clusters of markers, thus leading them back to the nest. Further-
more, whenever a worker ant eats, their chemical markers change slightly.
These chemical markers attract, rather than repel, other ants. Thus the ants
gradually begin to form a column leading from a food source to a nest.
Note that the ants' cognition is both social, in that it takes place between
organisms, and distributed, in the sense that it offloads much of the cogni-
tive work onto the environment. No single ant carries around an "internal
representation" or neural map of where the ant colony is. Ant cognition is
thus nonrepresentational in that it is both intrinsically social and situated in
organism-environment interactions.
The evolutionarily programmed social cognition of insects, however,
does not include the capacity for spontaneous imitation which is so central
to human cognition. For a social behavior to become a learned behavior
and then continue across generations, a capacity for spontaneous imitation
is crucial. However, zoological ethologists have long known that this imi-
tative capacity is not unique to humans. Researchers studying macaques
left sweet potatoes on the beach for a colony of wild monkeys who nor-
mally inhabit the jungle near the beach. After gradually becoming habitu-
ated to the beach and becoming more familiar with the sea, one monkey
discovered that dipping the potatoes in a tidepool would cleanse them of
the sand that made them unpalatable. This behavior was imitated through-
out the colony in a matter of days, but the researchers observed that older
macaques were slower to acquire the behavior than the younger ones (Ka-
wamura 1959; McGrew 1998). Maturana and Varela (1998: 203) define
cultural behavior precisely as this kind of relatively stable pattern of such
transgenerational social behavior.
The culturally acquired behavior most often held up by classical Repre-
sentationalists as the hallmark of the distinctively human is language.
However, even here there is not a clear break from the animal kingdom in
terms of basic cognitive capabilities, as we see when considering the re-
sults of researchers who have been trying to teach symbolic communica-
tion to other primates. Instead, their observations are consonant with our
theory of how language and image schemas emerge from bodily processes
involving cross-modal perception. In experiments done by Savage-
Rumbaugh and colleagues (1988), three chimpanzees who had been trained
in symbolic communication were able to make not only cross-modal asso-
ciations (i.e., visual to tactile), but were able to make symbolic to sensory-
modal associations. For example, Kanzi was able to hear a spoken English
word and accurately (100% of the time) choose either the corresponding
visual lexigram or a visual picture of the word. Sherman and Austin were
able to choose the appropriate object by touch when presented with a vis-
uallexigram (100% correct), and conversely they were also able to choose
the appropriate visuallexigram when presented with a tactile-only stimulus
(Sherman: 96% correct, Austin: 100%) or olfactory-only stimulus
(Sherman: 95% correct, Austin 70%: correct). Their ability to perform such
symbol to sensory-modality coordination enhanced their performance on
tasks measuring solely cross-modal coordination; as Savage-Rumbaugh et
al. observe: "these symbol-sophisticated apes were able to perform a vari-
ety of cross-modal tasks and to switch easily from one type of task to an-
other. Other apes have been limited to a single cross-modal task" (1988:
623). Although these chimpanzees will never approach the linguistic capa-
bilities of humans, these results show that the continuity of our human
capacity for abstract cross-modal thought is shared by at least some mem-
bers of the animal kingdom.
4
In fact, related recent research on primates suggests that it is the dis-
tinctively human socio-cultural environment (and not some great zoologi-
cal discontinuity in comparative cognitive capacity) that facilitates the
cross-modal cognitive capabilities underlying language and abstract reason.
We have already noted the neural development of the cross-modal maps of
juvenile owls can be modified by epigenetic stimulation, but it is equally
important to realize that the cross-modal basis for many of our image
schemas require epigenetic stimulation of the kind presented by human
parents. Tomasello, Savage-Rumbaugh and Kruger (1993) compared the
abilities of chimpanzees and human children to imitatively learn how to
perform novel actions with novel objects. They tested 3 conspecific
(mother-reared) chimpanzees and 3 enculturated chimpanzees, along with
18 and 30 month-old human children. They introduced a new object into
the participant's environment, and after observing the participant's natural
interactions with the object, the experimenter demonstrated a novel action
with the object with the instruction "Do what I do". Their results showed
that the mother-reared chimpanzees were much poorer imitators than the
enculturated chimpanzees and the human children, who did not differ from
one another. A human-like sociocultural environment is an essential com-
ponent not only for the development of our capacity for imitation, but also
for the development of our capacities for the cross-modal image schemas
that underlie language and abstract reasoning (see also Fouts, Jensvold and
Fouts 2002).
Finally, there is also considerable evidence from cognitive anthropology
that adult humans do not think in a manner consistent with the dichotomies
posed by classical Representationalism. Like the social insects, we tend to
offload much of our cognition onto the environments we create. We tend to
accomplish this in two ways - first, we make cognitive artifacts to help us
engage in complex cognitive actions, and, second, we distribute cognition
among members of a social organization. As an example of the first,
4. This conclusion is further supported by results showing that human children
with specific language impairments show deficiencies in their ability to perform
cross-modal tasks (Montgomery 1993).
Hutchins (1995: 99-102) discusses how medieval mariners used the 32-
point compass rose to predict tides. By superimposing onto the compass
rose the 24-hour day (in 45-minute intervals), the mariners could map the
lunar "time" of the high tide (the bearing of the full moon when its pull
causes a high tide) to a solar time of day. As long as we know two facts -
the number of days since the last full moon and the lunar high tide for a
particular port - we simply count off a number of points on the compass
rose equal to the days past the full moon to compute the time of next high
tide. Without the schema provided by the cognitive artifact, computing the
next high tide is a much more laborious cognitive task. As an example of
the second, Hutchins (1995: 263-285) discusses how the partially overlap-
ping knowledge distributions of a group of three navy navigation personnel
function cognitively within the team considered as a team. Although no
single team member is expected to constantly maintain a complete internal
representation of all the navigational data, Hutchins shows how the social
distribution of the cognitive tasks functions as a brake on serious naviga-
tional errors that could imperil the ship, because the participants each know
some of the spatial relations and procedures immanent to another team
member's job. In short, the offloading of some of the cognitive load onto
the environment, as found both in cognitive artifacts and the social distri-
bution of cognitive tasks, is crucial to many of our daily cognitive activi-
ties.
A fully adequate treatment of the social dimension of thought would re-
quire substantially more evidence and analysis than we can provide here.
We have only attempted to suggest that sociocultural cognition in general
is not unique to humankind, that the common bases for cross-modal cogni-
tion and symbolic/linguistic communication are not unique to humans, and
that human cognition cannot be locked up within the private workings of
an individual mind. Since thought is a form of co-ordinated action, it is
spread out in the world, co-ordinated with both the physical environment
and the social, cultural, moral, political and religious environments, insti-
tutions and shared practices. Language - and all forms of symbolic expres-
sion - are quintessentially social behaviors. Dewey nicely summarizes the
intrinsically social character of all thought in his argument that the very
idea of thinking as a kind of inner mental dialogue is only possible because
of socially established and preserved meanings, values and practices:
When this introspectionist thinks he has withdrawn into a wholly private
realm of events disparate in kind from other events, made out of mental
stuff, he is only turning his attention to his own soliloquy. And soliloquy is
the product and reflex of converse with others; social communication not an
effect of soliloquy. If we had not talked with others and they with us, we
should never talk to and with ourselves. Because of converse, social give
and take, various organic attitudes become an assemblage of persons en-
gaged in converse, conferring with one another, exchanging distinctive ex-
periences [...]. Through speech a person dramatically identifies himself with
potential acts and deeds; he plays many roles, not in successive stages of life
but in a contemporaneously enacted drama. Thus mind emerges. (Dewey
1925: 135)
HThus mind emerges!" It emerges as, and is enacted through, social cogni-
tion. There is no radical rupture with our bodily experience of meaning;
instead, that meaning is carried forward and given voice through language
and other forms of social symbolic interaction and expression.
6. Embodied meaning, thought and language
We have been arguing against disembodied views of mind, concepts and
reasoning, especially as they underlie Representationalist theories of mind
and language. Our alternative view - that cognition is embodied - has roots
in American Pragmatist philosophy and is being supported and extended by
recent work in second-generation cognitive science. Pragmatists like James
and Dewey understood that philosophy and empirical science must develop
in mutual cooperation and criticism, if we are ever to have an empirically
responsible understanding of the human mind and all of its marvelous ca-
pacities and acts. Pragmatism is characterized by (1) a profound respect for
the richness, depth and complexity of human experience and cognition, (2)
an evolutionary perspective that appreciates the role of dynamic change in
all development (as opposed to fixity and finality), and (3) recognition that
human cognition and creativity arise in response to problematic situations
that involve values, interests and social interaction. The principle of conti-
nuity encompasses the fact that apparently novel aspects of thought and
social interaction arise naturally via increased complexity of the organism-
environment interactions that constitute experience. Pragmatists thus argue
that all of our traditional metaphysical and epistemological dualisms (e.g.,
mind/body, inner/outer, subject/object, concept/percept, reason/emotion,
knowledge/imagination and theory/practice) are merely abstractions from
the interactive (enactive) process that is experience. Such distinctions are
not absolute ontological dichotomies. Sometimes they serve us well, but
oftentimes they serve us quite poorly, depending on what problems we are
investigating, what values we have, and what the socio-cultural context is.
In recent years the number of researchers engaged in some variation of
"embodied cognition" has swelled prodigiously. Once upon a time, cogni-
tive science seemed defined by the Representationalist view that the body
is inconsequential to the study of the mind. But that has changed dramati-
cally. Some Representationalists have recently argued for a very limited
sense of embodiment that would keep intact much of the first generation of
cognitive science's representational baggage (Clark 1998). Today we are
witnessing a new generation of cognitive science emerging which defines
"embodied cognition" as a fundamentally non-representational project.
Contributions to a radical theory of embodied cognition are being made by
dynamic systems theorists who argue that cognition, though amenable to
mathematical description, is not computational (van Gelder 1995), by neu-
robiologists whose experiments show us how metaphors of information
transfer mislead us in understanding the population dynamics behind neu-
ral organization (Edelman 1992), and by cognitive roboticists who under-
stand that having a body is perhaps not such a bad thing after all (Brooks
1991; Brooks and Stein 1994). Even Alan Turing, a leader among that lost
first generation who so errantly steered cognitive science toward disem-
bodiment, was willing to admit he might be wrong when it came to how we
might teach a robot language:
It can also be maintained that it is best to provide the machine with the best
sense organs that money can buy, and then teach it to understand and speak
English. That process could follow the normal teaching of a child. Things
would be pointed out and named, etc. Again, I do not know what the right
answer is, but I think both approaches should be tried. (Turing 1950: 460)
We have already tried the disembodied Representationalist approach, and
its failures have breathed new life into the Pragmatist approach to embod-
ied cognition.
The themes we have been tracing throughout this chapter - our animal
engagement and cognition, our ongoing coupling and our falling in and out
of harmony with our surroundings, our active value-laden inquiry to re-
establish harmony and growth, and our community of social interactions -
are beautifully encapsulated by Dewey in his attempt to recover the value
of the aesthetic dimensions of meaning in human life:
At every moment, the living creature is exposed to dangers from its sur-
roundings, and at every moment, it must draw upon something in its sur-
roundings to satisfy its needs. The career and destiny of a living thing are
bound up with its interchanges with environment, not externally but in the
most intimate needs.
The growl of a dog crouching over his food, his howl in time of loss and
loneliness, the wagging of his tail at the return of his human friend are ex-
pressions of the implication of a living in a natural medium which includes
man along with the animal he has domesticated. Every need, say for hunger
for fresh air or food, is a lack that denoted at least a temporary absence of
adequate adjustment with surroundings. But it is also a demand, a reaching
out into the environment by building at least a temporary equilibrium. Life
itself consists of phases in which the organism falls out of step with the
march of surrounding things and then recovers unison with it - either
through effort or some happy chance [... ].
These biological commonplaces are something more than that [mere bio-
logical consequences]; they reach to the roots of the esthetic in experience.
(Dewey 1934: 13-14)
We humans are live creatures. We are acting when we think, perhaps fal-
ling in and out of step with the environment, but never are our thoughts
outside of it. Via our bodily senses the environment enters into the very
shape of our thought, sculpting our most abstract reasoning from our em-
bodied interactions with the world.
Acknowledgements
The authors would like to acknowledge the many insightful comments by
the reviewers and the editors of this volume. Tim Rohrer would like to
acknowledge the stimulating intellectual environment of the Sereno and
Kutas cognitive neuroscience laboratories at UCSD as he conducted the
fMRI research outlined here, as well as the generous research support of a
Nlli fellowship.
References
Anderson, Michael
2003 Embodied cognition: A field guide. Artificial Intelligence 149: 91-
130.
1934
1938
Clark, Andy
1998
Brooks, Rodney A.
1991 Intelligence without representation. Artificial Intelligence Journal
47: 139-159.
Brooks, Rodney A. and Anita M. Flynn
1989 Fast, cheap and out of control: A robot invasion of the solar system.
Journal ofthe British Interplanetary Society 42: 478-485.
Brooks, Rodney. A. and Lynn A. Stein
1994 Building brains for bodies. Autonomous Robots 1 (1): 7-25.
Buccino, Giorgio, F. Binkofski, Gereon R. Fink, Luciano Fadiga, Leonardo Fo-
gassi, Vittorio Gallese, Riidiger J. Seitz, Karl Zilles, Giacomo Riz-
zolatti and Hans-Joachim Freund
2001 Action observation activates premotor and parietal areas in a soma-
totopic manner: An fMRI study. European Journal of Neuroscience
(2): 400-4004.
Buonomano, Dean V. and Michael M. Merzenich
1998 Cortical plasticity: From synapses to maps. Annual Review ofNeuro-
science 21: 149-186.
Being There: Putting Brain, Body and World Together Again. Cam-
bridge, MA: MIT Press.
Coslett, H. Branch, Eleanor M. Saffran and John Schwoebel
2002 Knowledge of the human body: a distinct semantic domain. Neurol-
ogy 59: 357-363.
DeBello, William M., Daniel E. Feldman and Eric I. Knudsen
2001 Adaptive axonal remodeling in the midbrain auditory space map.
Journal ofNeuroscience 21 (9): 3161-3174.
Deneubourg, Jean-Louis, Jacque Marie Pasteels and Jean-Claude Verhaeghe
1983 Probabilistic behavior in ants: A strategy of errors. Journal of Theo-
retical Biology 105: 259-271.
Dewey, John
1925 Experience and Nature. In: John Dewey, The Later Works, 1925-
1953, vol. 1, Jo Ann Boydston (ed.). Carbondale: Southern Illinois
University Press, 1981.
Art as Experience. New York: Penguin-Putnam Inc., reprinted 1980.
Logic: The Theory of Inquiry, In: John Dewey, The Later Works,
1925-1953, vol. 12, Jo Ann Boydston (ed.). Carbondale: Southern
Illinois University Press, 1981.
Edelman, Gerald
1987 Neural Darwinism. New York: Basic Books.
1992 Bright Air, Brilliant Fire: On the Matter of Mind. New York: Basic
Books.
2001
Fodor, Jerry
1987
Erickson, Harold P., Dianne W. Taylor, Kenneth A. Taylor and David Bramhill
1996 Bacterial cell division protein FtsZ assembles into protofilament
sheets and minirings, structural homologs of tubulin polymers. Pro-
ceedings of the National Academy of Sciences USA. 93 (1): 519-
523.
Feldman, Jerome and Srini Narayanan
2004 Embodied meaning in a neural theory of language. Brain and Lan-
guage 89 (2): 385-392.
Psychosemantics: The Problem of Meaning in the Philosophy of
Mind. Cambridge, MA: MIT Press.
Fogassi, Leonardo, Vittorio Gallese, Giorgio Buccino, Laila Craighero, Luciano
Fadiga and Giacomo Rizzolatti
Cortical mechanism for the visual guidance of hand grasping move-
ments in the monkey: A reversible inactivation study. Brain 124 (3):
571-586.
Fouts, Roger S., Mary Lee A. Jensvold and Deborah H. Fouts
2002 Chimpanzee signing: Darwinian realities and Cartesian delusions. In:
Marc Bekoff, Colin AlIen and Gordon M. Burghardt (eds.), The
Cognitive Animal: Empirical and Theoretical Perspectives in Animal
Cognition. Cambridge, MA: MIT Press. 285-291.
Fraser, Scott E.
1985 Cell interaction involved in neural patterning: An experimental and
theoretical approach. In: Gerald Edelman, W. E. Gall and W. M.
Cowan (eds.), Molecular Bases of Neural Development, 581-507.
New York: Wiley.
Gallese, Vittorio and George Lakoff
2005 The brain's concepts: The role of the sensory-motor system in reason
and language. Cognitive Neuropsychology 22: 455-479.
Gaze, R. M. and Sansar C. Sharma
1970 Axial differences in the reenervation of the goldfish optic tectum by
regenerating optic nerve fibers. Experimental Brain Research 10:
171-181.
Goldberg, Jeffrey L.
2003 How does an axon grow? Genes and Development 17 (8): 941-958.
Grady, Joseph
1997 Foundations of meaning: primary metaphors and primary scenes.
Ph.D. Dissertation, University of California.
Guo, Yajin and Susan B. Udin
2000 The development of abnormal axon trajectories after rotation of one
eye in Xenopus. Journal ofNeuroscience 20 (11): 4189-4197.
Hauk, Olaf, Ingrid Johnsrude and Friedemann Pulvermfiller
2004 Somatotopic representation of action words in human motor and
premotor cortex. Neuron 41 (2): 301-307.
Hodges, Andrew
1983 Alan Turing: The Enigma. New York: Walker and Co.
Hutchins, Edwin
James, William
1900 Psychology (American Science Series, Briefer Course). New York:
Henry Holt and Company.
Johnson, Mark
1987 The Body in the Mind: The Bodily Basis of Meaning, Imagination
and Reason. Chicago: University of Chicago Press.
Kawamura, Syunzo
1959 The process of subculture propagation among Japanese macaques.
Primates 2: 43-60.
Knudsen, Eric 1.
1998 Capacity for plasticity in the adult owl auditory system expanded by
juvenile experience. Science 279 (5356): 1531-1533.
2002 Instructed learning in the auditory localization pathway of the barn
owl. Nature. 417 (6886): 322-328.
Lakoff, George
1987 Women, Fire and Dangerous Things. Chicago: University of Chi-
cago Press.
1980 Metaphors We Live By. Chicago: University of Chicago Press.
1999 Philosophy in the Flesh: The Embodied Mind and Its Challenge to
Lakoff, George and Rafael Nunez
2000 Where Mathematics Comes From: How the Embodied Mind Brings
Mathematics Into Being. New York: Basic Books.
Langacker, Ronald
1986 Foundations of Cognitive Grammar. 2 vols. Stanford: Stanford Uni-
versity Press.
Lewkowicz, David J. and Gerald Turkewitz
1981 Intersensory interaction in newboms: modification of visual prefer-
ences following exposure to sound. Child Development 52 (3): 827-
832.
Maturana, Humberto R. and Francisco J. Varela
1998 The Tree of Knowledge: The Biological Roots of Human Under-
standing, revised edition. (Robert Paolucci, trans. 1987 Arbol del
conocimiento.) Boston: Shambhala Press.
McGrew, William C.
1998 Culture in nonhuman primates? Annual Review ofAnthropology 27:
301-328.
Meltzoff, Andrew N. and Richard W. Borton
1979 Intermodal matching by human neonates. Nature 282 (5737): 403-
404.
Merzenich, Michael M., Randall J. Nelson, Jon H. Kaas, Michael P. Stryker, Max
S. Cynader, A. Schoppmann and John M. Zook
1987 Variability in hand surface representations in areas 3b and 1 in adult
owl and squirrel monkeys. Journal Comparative Neurology 258 (2):
281-296.
Montgomery, James W.
1993 Haptic recognition of children with specific language impairment:
effects of response modality. Journal Speech Hearing Research 36
(1): 98-104.
Narayanan, Srini
1997 KARMA: Knowledge-based active representations for metaphor and
aspect. Ph.D. Dissertation, University of California at Berkeley.
Newell, Alan and Herbert Simon
1976 Computer science as empirical inquiry: Symbols and search. Com-
munications ofthe ACM 19: 113-126.
Regier, Terry
1996 The Human Semantic Potential Spatial Language and Constrained
Connectionism. Cambridge, Mass.: MIT Press.
Rizzolatti, Giacomo, Leonardo Fogassi and Vittorio Gallese
2002 Motor and cognitive functions of the ventral premotor cortex. Cur-
rent Opinion in Neurobiology 12 (2): 149-154.
Rohrer, Tim
2001 a
2001 b
2005
in press
Pragmatism, Ideology and Embodiment: William James and the
philosophical foundations of cognitive linguistics. In: Rene Dirven,
Bruce Hawkins and Esra Sandikcioglu (eds.), Language and Ideol-
ogy: Cognitive Theoretic Approaches: Volume 1, 49-81. Amster-
dam: John Benjamins.
Understanding through the Body: fMRI and of ERP studies of meta-
phoric and literal language. Paper presented at the 7th International
Cognitive Linguistics Association Conference, July 2001.
Image schemata in the brain. In: Beate Hampe (ed) From Perception
to Meaning: Image Schemas in Cognitive Linguistics, 165-196. Ber-
lin: Mouton de Gruyter.
Embodiment and experientialism. In: Dirk Geeraerts and Hubert
Cuyckens (eds), Handbook of Cognitive Linguistics. New York: Ox-
ford University Press.
Savage-Rumbaugh, Sue, Rose A. Sevcik and William D. Hopkins
1988 Symbolic cross-modal transfer in two species of chimpanzees. Child
Development 59 (3): 617-625.
Shelton, Jennifer R., Erin Fouch and Alfonso Caramazza
1998 The selective sparing of body part knowledge. A case study. Neuro-
case 4: 339-351.
Sperry, Roger W.
1943 Effect of a 180-degree rotation of the retinal field on visuomotor
coordination. Journal ofExperimental Zoology 92: 263-279.
Stem, Daniel
1985 The Interpersonal World ofthe Infant. New York: Basic Books.
Suziki, K., A. Yamadori and T. Fujii
1997 Category specific comprehension deficit restricted to body parts.
Neurocase 3: 193-200.
Tomasello, Michael, Sue Savage-Rumbaugh and A. C. Kruger
1993 Imitative learning of actions on objects by children, chimpanzees and
enculturated chimpanzees. Child Development 64 (6): 1688-1705.
Turing, Alan M.
1937 On computable numbers, with an application to the Entscheidung-
sproblem. Proceedings of London Mathematical Society, Series 2,
42:230-265
1950 Computing machinery and intelligence. Mind 59 (236): 433-460.
van Gelder, Tim
1995 What might cognition be, if not computation? Journal ofPhilosophy
92 (7): 345-381.
Varela, Francisco, Evan Thompson and Eleanor Rosch
1991 The Embodied Mind: Cognitive S ~ i e n c e and Human Experience.
Cambridge: MIT Press.
Warrington, Elizabeth K. and Tim Shallice
1984 Category specific semantic impairments. Brain 107: 859-854.
Winter, Steven
2001 A Clearing in the Forest: Law, Life and Mind. Chicago: University
of Chicago Press.
Ziemke, Tom
2003 What's that thing called embodiment? Proceedings of the 25th An-
nual Meeting of the Cognitive Science Society, 1305-1310. Lawer-
ence Erlbaum.
Bringing the body back to life: James Gibson's
ecology of agency
Alan Costall
" [...] the self, or rather the soul, by which I am what I
am, is entirely distinct from the body, is indeed easier
to know than the body, and would not cease to be what
it is, even if there were no body." (Descartes 1637
[1960]: 61).
Abstract
In its supposed revolutionary move beyond "mechanistic behaviourism", modem
psychology replaced the image of the passive subject with that of a highly active
information processor modelled upon the general purpose computer. But such ac-
tivity has been envisaged as essentially subcutaneous. For, in addition to the ex-
plicit and modem metaphor of the mind as computer, modem psychology simply
retained the traditional mechanistic metaphor of the body as a stimulus-response
machine. Thus, even though the ghost inside the machine has been replaced by a
highly active "processor of information", the passive, mechanical body that har-
boured the ghost continues to structure psychological theory.
Gibson's early work was an explicit attempt to develop a pure stimulus-
response psychology, but he eventually came to realize that this schema should not
be merely repaired by invoking mediating cognitive processes but had, instead, to
be rejected in its entirety. He replaced the stimulus-response formula still so fun-
damental to mainstream psychology with an "ecology of embodied agency": an
exploration of the material conditions - affordances and information - that support
our "being in the world".
Keywords: embodiment, James Gibson, affordances, proprioception, information,
nature-culture dualism, mutualism.
56 A/an Costall
1. Introduction
Psychophysical dualism, as formulated by Rene Descartes (1596-1650)
excluded the mind from the realm of natural science, and at the same time
promoted the ambitious claim of the new mechanistic science to include
everything else. Indeed, it was precisely by excluding mind from the physi-
cal order of things that the universal claim of the new science could be
protected. For, as Margaret D. Wilson (1980: 42) has put it, an "immaterial
principle is by definition not a part of 'nature''', and so does not count as a
proper object for science. Thus the apparent intractability of mind to
mechanistic treatment could hardly be deemed a failure on the part of sci-
ence, but a reflection, instead, upon the "non-scientific" nature of mind
itself.
Descartes did, however, leave our bodies to science. And, since within
Cartesian theory the mind was supposed to be the sole source of activity -
the sole mover - the body had to be regarded as not only separate and alien
from ourselves, but also entirely inert. Falling squarely on the physical side
of the Cartesian psychophysical divide, the body - along with the rest of
extended matter - was relegated to the realm of the essentially passive. The
body thus ends up in Cartesian theory much as it had first made its appear-
ance within modem science, inert and alien - as a corpse.
Modem psychologists would insist that psychological theory has come
a very long way since Descartes. His dualisms of mind and body, and of
mind and world, are, they would patiently explain, very much part of the
prescientific history of our discipline, and nothing to do with present-day
psychology.
In this chapter, I will argue that a dualism between body and mind per-
sists in modem cognitive psychology in a blindingly obvious way, and
which amounts to a dualism more extreme and more systematic than any-
thing to be found in the writings of Descartes. In a crucial sense, the body
has largely disappeared from mainstream psychological theory, figuring as
nothing more than a passive channel of "stimuli" and "responses". I will
then turn to the work of one of psychology's most centrally placed and
interesting misfits, James Gibson (1904-1979), who over a long career not
only subjected mainstream psychology to sustained criticism, but sought to
construct a non-dualistic alternative, which I have come to describe as an
ecology of agency. Gibson is important to the current debates about em-
bodiment because, by challenging the "stimulus-response" thinking that
still constrains much of modem psychological theory, he has helped to
Bringing the body back to life: James Gibson's ecology ofagency 57
bring psychology's moribund body back to life - a revived body that is
active, social and lived.
2. Was Descartes a "Cartesian"?
So far, I have presented Descartes as a good, straightforward Cartesian.
But, in fact, he was much more complicated and sensible than he appears
in most of the standard textbook accounts. For example, it turns out that he
was never very much bothered by the "problem of other minds" that would
seem to ensue from a stark division of body and mind (Avramides 2001).
Maybe he was not even a dualist (Baker and Morris 1996). Sometimes he
even entertained the possibility that the mind might ultimately be under-
stood in mechanistic terms (Cottingham 1992). Perhaps he was even a
"proto-phenomenologist" (Tibbets 1973). My purpose here in this section
is not to provide a review of the revisionist accounts of Cartesian philoso-
phy that have appeared over the last fifty years or so. But I do want to draw
attention to some important and surprising differences between "Cartesian
dualism" as presented in the official potted histories, and the more subtle,
possibly inconsistent, things Descartes himself had to say about the relation
of mind and body.
Descartes has most often been criticized for failing to provide an expla-
nation of the relation of body and mind, where, by explanation, is meant a
mechanical account of how the two relate. But, of course, that criticism
completely misses his fundamental point, that the mind is not part of the
physical order of things (Richardson 1982). Indeed, he makes very clear
that he does not see that there is any general problem that needs explana-
tion. The relation of body to mind is - and perhaps has to be - taken for
granted. Mind and body have to be intimately related if we are to respond
to injury and also to hunger and thirst with urgency and immediacy. As he
strikingly put it in his Meditations:
For nature also teaches me by these feelings of pain, anger, thirst, and so on,
that I am not just lodged in my body like a pilot in his ship, but that I am in-
timately united with it, and so confused and intermingled with it that I and
my body compose, as it were, a single whole. If this were not so, I should
feel no pain when my body was injured; I should simply note the injury with
my understanding, as a pilot sees with his eyes any damage done to his ship.
(Descartes 1641 [1960]: 161)
58 Alan Costall
Evidently, Descartes was, at the very least, in two minds about dualism.
Indeed, he eventually discloses that he did not regard the union of body and
mind as a real problem at all, but one created by reflective thought. It can-
not be understood in an external way, but can only be lived:
[...] what belongs to the union of soul and body can be understood only in an
obscure way either by pure intellect or even when the intellect is aided by
imagination, but is understood very clearly by means of the senses. [...] it is
just by means of ordinary life and conversation, by abstaining from medi-
tating and from studying things that exercise the imagination, that one learns
to conceive the union of soul and body. (Descartes 1643 [1971]: 279-280)
Clearly, there is a need to be wary about our supposed Cartesian past, but
not simply to put it aside. It is precisely the distorted, mythical, "official"
history of psychology, even about distant figures such as Descartes, that
really counts when trying to understand historically how the current disci-
pline of psychology has been structured. After all, agenda setting, includ-
ing self-congratulation and the induction of new students, is what "official"
disciplinary history is primarily about. The dodgy history is powerful his-
torical stuff. So, here is my attempt to distil the already potted essence of
the Cartesian dualism of mind and body as it exists within the collective
memory of psychologists:
1. There is a stark ontological dualism between mind and body: they
are entirely different kinds of "stuff'.
2. The body is a passive stimulus-response mechanism.}
3. The mind is the only active principle, and mediates between stimulus
and response (in the case of voluntary action).
3. Psychology's missing body
Modem psychologists would certainly not identify themselves as Carte-
sians in the above sense. They would argue that they have emphatically
rejected ontological dualism by developing naturalistic accounts of mind,
primarily based upon analogies with reassuringly substantial devices, such
as computers. Furthermore, given their rejection of stimulus-response be-
1. Descartes' place in the history of the reflex and of "stimulus-response think-
ing" within psychology is, again, a good deal more complex than the stan-
dard accounts imply (cf. Danziger 1983).
haviourism, they would insist they have also rejected the old mechanistic
psychologies based on the concept of the reflex. And if this were all true,
you would have to conclude that modem psychologists have undermined
the old mechanistic psychology by a thoroughly new mechanistic psychol-
ogy, based on analogies with new kinds of machines never envisaged by
the early proponents of a mechanistic psychology. As Stephen Toulmin
(1993: 146) has put it:
If Descartes, Newton or Leibniz had been shown a late 20th century com-
puter, [...] they could only have reacted by declaring: "That's not [what I
would have called] a 'machine' at all."
Yet, it is not the case that mainstream cognitive psychology entirely re-
placed traditional mechanism. It retains the old mechanistic image of the
body. The new mechanism of mind has been merely assimilated to the old
dualism of mind and body, along with the existing conception of the body
as a passive machine. This dualism is now, however, reformulated in terms
of two radically different kinds of machines - a machine within a machine,
a new mechanical mind implanted with the old mechanical body. However,
we have all been so fixated upon how to theorize the mind in terms of the
new mechanism that this retention of the old mechanistic schema of the
body has been systematically overlooked.
Ever since the rise of cognitive psychology, with its self-conscious
rhetoric of Kuhnian revolution, its proponents have claimed to be rejecting
stimulus-response psychology, whilst in the same breath reinstating its
basic schema:
The past few years have witnessed a noticeable increase in interest in an in-
vestigation of the cognitive processes. [... ] It has resulted from a recognition
of the complex processes that mediate between the classical "stimuli" and
"responses" out of which stimulus-response learning theories hoped to
fashion a psychology that would by-pass anything smacking of the "mental".
(Bruner, Goodnow and Austin 1956: vii [emphasis added])
It seems obvious to us that a great deal more goes on between the stimulus
and response than can be accounted by a simple statement about associative
strengths. (Miller, Galanter and Pribram 1960: 9 [emphasis added])
[...] the dramatic shift away from behaviorism, which dominated the field for
over thirty years, to cognitivism, .. [has] allowed one to study not just
learning but memory, not just speech but language, and not just stimulus and
response but the processes that mediate them. (Hirst 1988: vii [emphasis
added])
60 Alan Costall
[...] in contrast to early behaviorists, but like their intellectual descendants
such as cognitive psychologists and ethologists [... ], I believe that scientists
can and should construct models for elucidating the knowledge and cogni-
tive processes that connect stimulus and response. (Schiffer 1999: 8 [em-
phasis added])
In an early assessment of the cognitive revolution, Donald Hebb, in his
presidential address to the American Psychological Association, claimed
that the stimulus-response formula was indeed essential to the new cogni-
tive approach because it served to define what psychologists could possibly
mean by "cognitive":
[...] the whole meaning of the term "cognitive" depends on [the stimulus-
response idea], though cognitive psychologists seem unaware of the fact.
The term is not a good one, but it does have meaning as a reference to fea-
tures of behavior that do not fit the S-R formula; and no other meaning at all
as far as one can discover. The formula, then, has two values: fIrst, it pro-
vides a reasonable explanation of much reflexive human behavior, not to
mention the behavior of lower animals; and secondly, it provides a funda-
mental analytical tool, by which to distinguish between lower (noncognitive)
and higher (cognitive) forms of behavior. (Hebb 1960: 737)
Here is a very recent formulation of the modem "alternative" to stimulus-
response psychology by Rom Harre (2002: 104), an influential critic of
mainstream psychology. He is using the following example of word recog-
nition to make a much more general point about how we should theorize in
psychology:
Instead of the behaviorist pattern:
Stimulus (retinal sensation) ~ Response (perception of word)
we must have
Observable stimulus (retinal sensation) together with unobservable Cogni-
tive process ('knowledge utilization') ~ Observable response (recogni-
tion of word)
Cognitive psychologists keep presenting their own position as an alterna-
tive to stimulus-response behaviouristic psychology, when what they are
actually offering is no more than an elaboration of that traditional frame-
work. As Edward Reed has put it, cognitivism has largely ended up as the
"flip-side" of stimulus-response behaviourism, where "mental processes"
are defined as whatever "is left over after one tries to stuff all psychologi-
cal phenomena into the S-R box" (Reed 1997: 267). And because of this
legacy, psychologists keep failing to recognize real alternatives when they
come along, because they cannot understand alternatives to their own posi-
tion as anything other than reversions to the traditional "pure" version of
the stimulus-response framework. Thus, in their otherwise insightful over-
view of theory and research on visual perception, Bruce and Green (1990:
389) claim that the newer developments (including Gibson's ecological
psychology) are giving rise to an important new debate "about the neces-
sity for a cognitive or psychological level of theory to stand between the
"stimulus" and the "response"".
But why have cognitive psychologists been so slow to notice their re-
tention of the stimulus-response schema? First of all, the very rhetoric
about "cognitive revolution" and liberation from behaviourism has given
rise to a remarkably long-lasting bout of intellectual complacency. In fact,
modem cognitive psychology has retained a number of the central features
of the behaviourism it claims to have replaced: the attempt to repair
stimulus-response psychology by appeal to mediating processes (a central
feature of neo-behaviourism), a commitment to the "hypothetico-deductive
method", and also "methodological behaviourism", the assumption that the
data of psychology are confined to meaningless "behaviour" (cf. Still and
Costa111991; Leahey 1992; Neisser 1997: 248).
Secondly, the stimulus-response schema also conforms perfectly to the
procedures by which psychologists still conduct most of their research, and
it is enshrined within those procedures. Thus the participants in experi-
ments are "presented" with a stimulus or an "independent variable" and are
required to respond to that stimulus. They are not expected to transform the
"conditions" imposed upon them, nor take themselves off to quite a differ-
ent one. For the duration of the experiment, participants agree, in effect, to
"lend" their agency to the investigator. Admittedly, such a state of passiv-
ity is not purely an artifact of the laboratory - it is commonplace in our
lives at work and school. But, fortunately, it is not our sole mode of being
in the world. As George Kelly (who, like Gibson, was another member of
psychology's "awkward squad") once warned:
Behaviour is man's way of changing his circumstances, not proof that he has
submitted to them. What on earth, then, can present-day psychology be
thinking about when it says it intends only to predict and control behaviour
scientifically? Does it intend to halt the human enterprise in its tracks? (cited
by Westland 1978: 69).
It is, however, the standard metaphor of the computer that has given the
stimulus-response formula (now reformulated in the language of input and
62 Alan Costall
output) its new lease of life, and, hence, made the body largely disappear
from psychological theory (see also Costall 1991). First of all, the per-
son/computer is purely an information processor, passively receiving an
"input" (in other words, a stimulus), operating upon it, and producing out-
put. Such a person/computer does not get about, and certainly does not
have much of a life, and so has no need of a body, other than to support
"cognitive processing" and to provide an "interface" to the outer world - a
passive "channel" for both input and output.
2
There is also a surprisingly neglected theoretical aspect of the standard
computer metaphor that has led to the disappearance of the body. This
metaphor gives rise to a two-part division between the software (the pro-
gram) and hardware (the actual computing machine). According to this
reassuringly substantial metaphor of the computer, with its tidy division of
program and machine, the business of the psychologist is neatly and con-
veniently circumscribed: to infer the software of the mind without refer-
ence to the "machinery" of the body. The software, after all, exists in quite
a different order from the computer, and cannot be reduced to it, and so
psychologists need no longer be anxious that they are engaged in a spuri-
ous and unscientific undertaking, nor fear that they might eventually be
taken over by the biochemists or the neurologists. As one of its many en-
thusiasts proudly proclaimed, cognitivism provides us with "a science of
structure and function divorced/rom material substance" (Pylyshyn 1986:
68 [emphasis added]). On the face of it, the current obsession within cog-
nitive psychology about the neural localization of psychological functions
would seem to represent a radical move against disembodied theorizing,
yet this new kind of phrenology has done nothing to bring the body seri-
ously back into psychological theory, since the "body" in question is noth-
ing more than bits of the brain.
2. This standard computer metaphor has been succeeded by other computer mod-
els, but not, within cognitive psychology, superseded by them. Connectionism,
which has been widely taken up within cognitive psychology, is generally pre-
sented as complementary to the classical, symbol-based, approach to informa-
tion processing, and, in any case, also treats the body as passive in relation to
the world. In contrast, the more recent research upon robotics and "autonomous
agents" has made remarkably little impact within mainstream cognitive psy-
chology, and is seldom mentioned in the textbooks. However, we cannot safely
assume that even "the new AI" has escaped from the schema of the-machine-
within-the-machine (cf. Ziemke 2001).
But let us now look more closely at the other side of the standard com-
puter metaphor, the hardware or computer. What precisely is the hardware
supposed to represent? Presumably not the mind. But is it the entire body,
or the brain, or, indeed, just those structures that specifically support cog-
nitive processing? Psychologists have been remarkably unconcerned about
this side of the analogy, and about the kind of theoretical work this side of
the computer analogy is supposed to be doing. For, according to the ideal
of the computer as a general purpose machine, the computer itself (as-
suming it is switched on and operating properly) imposes absolutely no
constraints on how it actually functions. It is the program that entirely
specifies how the machine will operate and this is why it offers the obvious
and sufficient explanation of what is going on when the program is "run-
ning" a computer.
3
According to the standard computer metaphor, therefore, the body as
computer ends up doing no explanatory work at all, any more than does the
paper on which physicists write down their equations. And so, to the extent
that theorists wish (within the terms of the traditional computer metaphor)
to theorize the body as anything other than as a transparent, completely
unconstraining entity, they would have to represent it as part of the soft-
ware. To repeat, it is the program, according to the ideal of the computer as
a general purpose machine, that is the only source of constraint!
Finally, the theoretical focus of the cognitivist research project itself
turns our attention resolutely away from the body and bodily activity. Even
if what people are actually doing necessarily provides the data for psychol-
ogy, their behaviour is not supposed to constitute the real object of inquiry:
To take behavior as the focus of attention for psychology is as big an error
as to take tracks in cloud chambers as the main object of study in particle
physics. Such tracks are interesting only as clues to the existence of certain
particles and to their properties. (Macnamara 1999: 241.)
What people can be observed to be doing can thus only be regarded as
merely an indirect "clue" or index of an underlying mental apparatus, and
3. The virtue of programs as a language in which to formulate psychological
theories lies precisely in the fact that they have to do all the theoretical work,
just as, in astronomy, differential equations do all the explanatory work in
accounts of planetary motion - though it does not follow that, to the extent
that programs provide an effective language for theorizing in psychology, the
objects of our theorizing are themselves necessarily "running programs", any
more than the planets are staying on course by solving differential equations.
64 Alan Costall
is not, in itself, of any focal psychological interest. And, when you think
about it, what people are required to do in most psychological experiments
could be of no other interest, given that what they are usually doing is
nothing more than just pushing buttons or else keys on computer key-
boards. Such "responses" do not have any significance at all, except as
indices with a meaning tied to the particular experiment in question. And,
thus, once again, the structure of the psychology experiment not only rep-
resents but in turn reinforces the dualism of modem cognitivism: that be-
haviour, as bodily movement, has no intrinsic or manifest meaning, its
meaning deriving, instead, from underlying and unobservable mental
structures which are deemed to be "the main object of study" in the par-
ticular experiment in question.
The disappearance of the body is also reflected in the illustrations that
accompany the textbooks. For example, when Descartes describes how the
body responds to injury, he provides us with a figure of a rather chubby
naked boy with his toe dangerously close to a fire (Descartes 1664 [1966]:
271). When Warren (1922: 4), in his textbook, Elements of Human Psy-
chology, discusses the stimulus-response arc, he presents the well-defined
figure of a man dressed in "plus-fours", the fashionable leisurewear of the
time. When Thurstone (1923: 355) discussed the "stimulus-response fal-
lacy" he included an illustration of "our minds", but this mind still takes a
definite bodily form, even if merely that of some amoeba-like creature.
So, how does the body (dis)appear in the modem textbooks? As an out-
line box, of course: a schematic interface to the outside world, and a con-
tainer of many other, much more interesting, boxes representing various
cognitive modules and the connections between them. In modem psychol-
ogy, the body, construed as a receptive stimulus-response machine, has
atrophied through many years of intellectual neglect to a shapeless and
abstracted container. Psychologists have been so busy mechanizing the
mind, they forgot about the other side of their mechanistic theorizing, the
stimulus-response body. James Gibson was an important exception.
4. James Gibson and the ecology of agency
As a student at Princeton, Gibson was greatly influenced by Edwin B. Holt,
who had, in turn, been taught and inspired by William James. Gibson was
thus familiar with the Darwinian adaptationist orientation of American
psychology. But, remarkably, during his early career at Smith College, he
was also in close contact with the Gestalt psychologists, Kurt Koffka, Fritz
Heider and Kurt Lewin, who had emigrated from Europe. As a conse-
quence, Gibson's approach brought together the functionalist emphasis
upon the coordination of animal and environment with the Gestaltist reac-
tion against atomistic analysis in favour of a relational, holistic approach.
Gibson also drew upon many other more "exotic" intellectual sources,
including Merleau-Ponty and Marx, though he could have been a good
deal more forthcoming about such influences (Heft 2001; Reed 1988). His
career was a continual project towards undermining the many dualisms
within the human sciences, and his work a remarkably productive fusion of
American functionalism, behaviourism, Gestaltism, phenomenology and
his own remarkable obstinacy. As his wife, Eleanor Gibson approvingly
put it, "he adored arguments". (cited in Szokolosky 2004: 277). He was not
prepared to take "taken-for-granted assumptions" for granted, not even
those once central to his own work.
It may seem odd to introduce Gibson in this chapter, and at this point,
since according to his most influential critics he was surely a throw-back to
pure stimulus-response behaviourism (e.g. Gregory 1997). If they were
referring to the Gibson of the 1940s and 1950s, their claim would have
some justification. But, as I will explain, like Wittgenstein, there were at
least two Gibsons: an "early" and a "later" one.
Before the "cognitive revolution" (variously dated from the late 1950s
to the early 1970s), Gibson (e.g. 1950, 1958, 1959) had already taken ex-
ception to the self-contradictory nature of psychological theorizing, with,
on the one hand, its fundamental commitment to a mechanistic stimulus-
response psychology, and then, on the other hand, its attempt to dream up
possible processes intervening between the stimulus and response that
might possibly explain the lack of any lawful causal relation between the
so-called stimuli and responses. He thought this whole business of invok-
ing efficient causality between stimulus and response and then invoking a
deus ex machina to "explain" why such efficient causality seldom held true
was scientifically disreputable. As he put it, he had "no patience with the
attempts to patch up the S-R formula with hypotheses of mediation. In
behavior theory as well as psychophysics you either find causal relations or
you do not" (Gibson 1967: 132).
The "early Gibson" attempted to reinstate stimulus-response theory, or a
"perceptual psychophysics", both by redefining the stimulus in a more
holistic, Gestalt way, and also by explaining how the higher-order structure
of the stimulus was, in turn, determined by the very structure of the envi-
66 Alan Costall
ronment itself. For example, in the case of a terrestrial environment, there
is a ground surface extending around the animal which gives rise to a gra-
dient of texture at the retina, and, in an uncluttered environment, also a
"visible horizon".
Even in his early writings, however, you can already find Gibson begin-
ning to notice something fundamentally wrong with the stimulus-response
formula. Having been concerned in the Second World War with aviation,
he had come to place great emphasis upon what he later called "optic
flow", the streaming of optical texture primarily as a result of our own
activity in the world:
The normal human being [.. ] is active. His head never remains in a fixed po-
sition for any length of time except in artificial situations. If he is not walk-
ing or driving a car, or looking from a train or airplane, his ordinary adjust-
ments of posture will produce some change in the position of this eyes in
space. Such changes will modify the retinal images in a quite specific way.
(Gibson 1950: 117 [emphasis added])
Once we allow for the active nature of human beings and other animals, the
"stimulus" can no longer figure as an efficient cause, nor be considered as
the "starting point" of perceiving. After all, if we are to retain the old static
language of stimulus and response, it is typically the "response" that pre-
cedes the "stimulus", and which gives rise to it. In other words, "one could
say that the behavior is the first cause of all the stimulations" (Merleau-
Ponty 1942 [1965]: 13).
By the late 1950s, Gibson had rejected the stimulus-response schema
entirely. People, he insisted, are not passive recipients of stimuli, except
under conditions such as those of the psychological laboratory, where im-
mobility is imposed upon us, either through the instructions or else more
intrusively through the use of clamps:
The headrest of the laboratory prevents the observer from turning his head
and looking around [...]. It also, of course, prevents him from getting up and
walking around. (Gibson 1979: 1)
However, when we are doing things, and even when just "observing" our
surroundings, we are active not just in our heads (as much of modem the-
ory still insists), but bodily. We are acting upon and exploring our sur-
roundings. Thus, according to Gibson, the visual system, for example, does
not just involve the eyes and a brain (cf. Gregory 1997), but must be de-
fined functionally rather than anatomically. The eyes, which themselves
are under muscular control, are part of a moving head, which, in turn, is set
on top of a body that gets around in the world. Thus, as Gibson liked to put
it, the visual system also has legs. Indeed, when we bring an object to our
eyes to inspect it more closely, our hands, from this functionalist perspec-
tive, should also be regarded as part of the visual system (Cowie 1993).
Perversely, the textbooks routinely force Gibson into the category of a
"bottom-up" theorist. In such theories, the "processing" of the stimulation
is supposed to be completely "data-driven", whereas in "top-down" theo-
ries, the input is assumed to be subject to active interpretation or hypothe-
sis construction (Cavanagh 1999). This modem-sounding distinction can,
in fact, be found in Kepler's proposal, published in 1604, that the retinal
image is the startingpoint of vision:
In what manner this image or picture is brought together by the visual spirits
which reside in the retina or in the nerves, and whether it is made to appear
before the soul or tribunal of the faculty of vision by a spirit within the
cerebral chambers, or whether the faculty ofvision, as a magistrate sent by
the soul, goes out from the council chamber ofthe brain to meet this image
in the optic nerves and retina descending to a lower court, these things I
leave to the natural philosophers [.. ] for disputing. (cited by Straker 1976:
20 [emphasis added])
The underlying continuity between Kepler's account and most modem
theories of vision concerns precisely the assumption that the body is a pas-
sive recipient of external stimulation.
Now, if we really did spend all our lives just waiting for things to hap-
pen to us - as the participants in psychology experiments are typically re-
quired to do - then whatever "activity" is involved in perceiving would
necessarily be confined to "internal processing" (cf. Ben Zeev 1984). Ac-
tivity would necessarily be "subcutaneous". But this was the very assump-
tion Gibson was rejecting. And this is why the very distinction between
"bottom-up" and "top-down" is irrelevant to Gibson's later theory given he
rejected the very concept of the stimulus.
If we really must talk of Gibson in terms of "ups" or "downs", then Gib-
son is best regarded (according to a very nice slip of a student's pen) as a
"bottom-down theorist". After all, Gibson regarded perceiving as a way of
making "contact" with our surroundings, a "reaching out" into the world.
4
4. Arnheim (1956 [1969]: 33) held a rather similar view: "[...] vision is anything
but a mechanical recording device. [...] Rather, in looking at an object, we reach
out for it. With an invisible finger we move through the space around us, go out
to the distant places where things are found, touch them, catch them, scan their
68 Alan Costall
This tactile conception of perceiving was captured in, and probably en-
couraged, by an important study conducted upon active touch (Gibson
1962; cf. Ikegami and Zlatev this volume). Participants were required to
recognize various objects either when these objects were simply placed
into their motionless hands, or else when they were given permission to
explore them actively in their hands. In passive touch, there is the dull
sense of "something" sitting on the surface of the hand: in active touch,
there is the vivid sense of a coherent object passing between the palm and
fingers. Gibson's contrast between active versus passive touch nicely en-
capsulates one of the radical shifts in his theoretical position.
Having started from an explicit commitment to stimulus-response the-
ory, Gibson was, in fact, one of the few experimental psychologists to re-
ject entirely the mechanistic framework of traditional perceptual theory.
After all, the very ideal of experimental investigation in psychology has
surely become that of imposing conditions upon our "subjects" and then
determining how they react. Seldom are they allowed to explore, let alone
change, the situations in which they are placed. Yet, as Gibson came to
realize, perceiving is an embodied activity, one involving skill and intelli-
gence.
5. Resources for agency
The two key concepts developed by Gibson in relation to his ecological
approach are information and affordance. Consistent with Gibson's rejec-
tion of stimulus-response framework, these should not to be regarded as
"efficient causes" but as resources for an embodied and active subject.
5.1. Affordances
Let us start with Gibson's concept of affordances, a concept that was
mainly set out, and then rather sketchily, in his final writings (e.g. Gibson
1979). Its purpose was to undermine the dualisms of the mental and physi-
cal, of meaning and materiality, of the world and us, that continue to
structure that last outpost of scientism and individualism, modem cogni-
surfaces, trace their boundaries, explore their texture. It is an eminently active
occupation."
tivist theory. Yet it is not as though its implications are restricted to that
target, for, even when the human sciences, as in social constructivism and
postmodernism, try to go their own way, they often manage to retain these
traditional dualisms through a failure to engage seriously in a radical ex-
amination of all this "modernist" metaphysics. Thus, we find the anthro-
pologist, Roy Ellen (1996: 31), having argued for a discursive view of na-
ture to replace a scientistic one - and in a book devoted precisely to
"redefining nature" - coming to the remarkably traditional conclusion that
culture "emerges from nature as the symbolic representation of the latter."
And we have the social constructivist, Stuart Hall, insisting that meaning is
confined to a representational realm of symbols, in opposition to the mate-
rial world:
[...] it is not the material world which conveys meaning: it is the language
system or whatever system we are using to represent our concepts. (Hall
1997: 25)
Gibson's concept of affordances attempted to undermine the dualisms of
subject and object, matter and meaning by treating animal and environment
as mutually defining:
The affordances of the environment are what it offers the animal, what it
provides or furnishes, either for good or ill. [...] I mean by it something that
refers to both the environment and the animal in a way that no existing term
does. It implies the complementarity of the animal and the environment.
(Gibson 1979: 127)
Affordances are, in essence, relationa1.
5
They concern the meanings of
"things" in terms of what could be done with them, and hence implicate an
agent. But these are not meanings that are "projected" onto things; they
very much have to do with the nature of the objects involved. Take the
example of stairs. Stairs are not stimuli. They do not force us to climb
them. And there are a limitless number of things we can do with them: sit
on them, break them up for firewood, and make a grand entrance down
them. Woodworm can even live in and eat them. But the fact that we can
do some many different things with things does not imply that we can do
anything with anything. Thus to use stairs in their canonical way - for go-
5. Gibson was blatantly inconsistent on this point, sometimes insisting that affor-
dances (and information) were independent of animals and thus undermining his
attempt to go beyond subject-object dualism (cf. Costall 1995). I will return to
this point.
70 Alan Costall
ing up and down them - requires (among many other things) that their ris-
ers and treads are of the appropriate dimensions. Beyond a certain critical
point the stair no longer affords climbing. To ascend a staircase, we need to
be able to reach the next step with our foot, and, furthermore, then be able
to lift our body so that its weight is centered on that step. Going down
stairs is more precarious, since we also need to check that we do not
overdo things and end up in a painful fall.
Normally, young children do have serious problems going up and down
stairs. In fact, one can find age "norms" in the textbooks on motor devel-
opment and developmental psychology, where the failure or success to
achieve that norm is taken to reflect the intrinsic developmental condition
of the child (Gesell, Ilg and Ames 1977). There are striking differences in
the age norms for going down as opposed to going up stairs, and also for
alternating the feet between steps as opposed to moving one foot forward
and then gingerly following through by putting the other foot onto that
same step. With the normal staircases that children normally encounter, it
is not until they are around the late age of four and a half years that they
begin to risk alternating their feet when descending.
However, although the "climbability" of a staircase is a function of its
dimensions, it also depends upon the size of the person in question. People
do not come in standard sizes, and, in particular, young children are gener-
ally much smaller than adults. Yet these age norms have been based on
"normal" staircases - stairs, in other words, designed for adults, not chil-
dren.
It is very curious that in our schools there are child-sized chairs and ta-
bles, but not child-sized stairs. In an inspired study, Josep Roca and his
colleagues simply checked to see how children would cope with a scaled-
down staircase where the steps were just 10 cm high and 20 cm. deep
(Roca et al. 1986). The children coped remarkably well. The mean age at
which they could climb either up or down the staircase was about twelve
months, and alternating the feet between steps was achieved only slightly
later at around eighteen months.
Even this study did not make allowances for the fact that some of the
participants were smaller than others, and not least because of their differ-
ent ages. Yet it is the precise relation between the dimensions of the stair
and of the individual user that is crucial. For example, according to Warren
(1995), when the ratio of height of a step to leg length is greater than .88, it
is simply no longer possible to step up onto it, and one must then resort to
climbing with ones hands and knees. There is also a definite optimum ratio
where energy expenditure in climbing is least, and this, again, is a conjoint
function of both the stair and the user. Under the specific conditions used
in Warren's research, where the diagonal distance between successive
stairs was held constant at 14 inches, the optimum ratio of riser height to
leg length was .26.
Let us take just one more example. The ways in which young children
(and indeed other primates) are able to grasp objects has been of great in-
terest to developmental psychologists, paediatricians and primatologists,
especially the precision grip where the object is held between thumb and
forefinger. Presumably in the sake of the serious pursuit of "scientific con-
trol", many of the main studies on grasping have used a target object with a
single, standard size. Primates, however, including human children and
indeed even adults, do not come in a standard size, and the graspability of
the object surely depends on the relation between the object and the agent.
Karl Newell and his colleagues presented both young children and
adults with a range of cubes of different sizes and recorded the different
kinds of ways they picked them up, for example between the flat of the
palms, within the palm of one hand with all of the fingers grasping the
object, and the prestigious precision grip (NewelI et al. 1989). When these
investigators related the frequency of these various grip patterns, not to the
absolute size of the cubes, but body-scaled to the individual participants,
the transitions between different kinds of grip corresponded to definite
ratios common to both the children and adults. Indeed, it was the bodily-
scaled dimensions of the cubes, rather than age that accounted for most of
the variability between the different grip patterns. Although the children in
this study were older than the age at which children have been recorded as
first being able to use the precision grip, the results suggest we need to be
wary about age norms that make no reference to the child's relation to the
test situation.
There are critical points, therefore, concerning "body-scale" that define
the limits within which we can act upon something in a particular way -
and beyond which we simply cannot. But scale is not the only issue, and
nor does it function in isolation. For example, whether - and how - we
might grasp an object depends upon a host of its other characteristics: its
fragility, its slipperiness, its mass and also the distribution of its mass, its
value, and so on. And what "holds" scale together with all of these other
characteristics together and gives them meaning is the animal or person in
question, though not so much as a "perceiver" (cf. Gibson 1979: 137; Heft
2001: 132; Ingold 2000: 168) but, much more fundamentally, as an agent:
72 Alan Costal!
[Affordances] have unity relative to the posture and behavior of the animal
being considered. So an affordance cannot be measured as we measure in
physics. (Gibson 1979: 127-128 [emphasis added])
This, then, is how the concept of affordances helps to undermine, rather
than merely "bridge", the old psychophysical dualism. Affordances con-
stitute the material resources for action but they do not fall on the far side
of the material-mental divide. They are, as Gibson put it, both physical and
mental, because they already implicate the needs and purposes of an agent,
who, in turn, is envisaged as existing within - rather than beyond - the
natural order of things (Heft 1989).
There can be no question that Gibson intended affordances to include
the culturally specific, as in his much discussed example of a postbox,
which "affords letter-mailing to a letter-writing human in a community
with a postal system" (Gibson 1979: 139). Yet for many in the human sci-
ences, the culturally specific - the "conventional" - is somehow supposed
to crowd out materiality. Yet, even though a postal system constitutes a
highly specific human practice, materiality still matters. Postboxes, for
example, as part of this system, have to perform the function of accepting
and temporarily storing letters, but they also need to be distinguishable
visually from litter bins and other kinds of things. And, of course, we do
not discover what postboxes mean just by peering at them! It is through
being a member of a community in which postboxes are actually used that
we come to understand what they are supposed to afford, including being
instructed by other people about how to use them, seeing other people us-
ing them, and, if all else fails, consulting a manual on how to use them.
Indeed, if, as in the case of autism, we are somehow excluded from con-
necting with other people, the normal use of objects can be seriously dis-
rupted (Williams, Costall and Reddy 1999).
5.2. Information
Gibson's concept of information points to a second essential resource for
effective agency: experiencing the world in relation to ourselves, including
our bodily capacities. According to Gibson, the information available to
perceivers is limitless and sufficient to specify the important features of
our environments and our relations to it. Traditional perceptual theory, in
contrast, not only stresses the poverty and unreliability of the available
information but also regards the perception of the world and of ourselves
as quite separate.
Gibson often quoted, approvingly, Bishop Berkeley's claim that vision
enables animals "to foresee the damage or benefit which is like to ensue
upon the application of their own bodies to this or that body which is at a
distance" (Berkeley 1707 [1975]: 24; cf., for example, Gibson 1950: xiv,
1966: 156, 1979: 232).) There is now a substantial body of research with
adults and children showing that we can be very effective in "foreseeing"
affordances, for example, whether a staircase is climbable and even the
optimum riser height for one"s own height, whether one can walk through a
gap either with or without turning one's shoulders, or ducking one's head,
and so on (e.g. Mark 1987; Warren 1984, 1995; Warren and Whang 1987).
And we typically do this not by first visually "measuring" the object and
only then comparing it to our own body. Rather, we experience the object
in relation to ourselves. This is obviously the case, for example, when we
are about to pick up an object. We do not just see the object nor just our
hands, but the object and our hands in relation to one another. The relation
of the object to our bodies is also specified in more subtle ways. Thus our
eye-height is specified by the "visible horizon" (Gibson 1979: 162-164).
Objects extending above the horizon are higher than our own eye-level,
and the proportion in which it intersects the horizon corresponds to the
height of that object relative to one's eye-height. Like the critical ratios for
stepping on stairs, and picking up cubes, these horizon relations relative to
the height of the point of observation, lack an objective metric, they are
dimensionless and body-scaled. They directly relate to us, and are all the
better for that:
[...] the "knowledge" of his height that comes to the observer simply from
living in his body is both more fundamental and more meaningful to him
than the knowledge communicated to him by a statement such as "X is Y
feet long". (Sedgwick 1973: 47)6
Complementarily, we develop, and sustain, a relatively stable sense of the
limits and capacities of own bodies-in-relation-to-the-world in the very
course of our activities (Stins, Kadar and Costall 2001). Interestingly (if
6. The variety of "anthropocentric" measurements of length is remarkable, be-
ing based on the size of the hand, the length or width of the fingers, the
thumbnail, the fist and outstretched thumb, the foot, the pace of the legs, and
so on (Klein 1975; Connor 1987).
74 Alan Costall
not surprisingly), this sense of our own bodies can be disrupted during
periods of rapid growth (Heffernan and Thomson 1999).
5.3. Proprioception.
In traditional theory, perceiving the world and perceiving oneself have
been regarded as quite separate issues. Perceiving the world was supposed
to be mediated by "exteroceptors" in the eyes, ears, nose and skin, and
perceiving the self was supposed to be achieved through specialized "inter-
oceptors" within our muscles, joints and the semicircular canals within the
inner ear. Gibson regarded this division - based on a dualism of the objec-
tive and subjective - as simply mistaken: "The awareness of the world and
one's complementary relations to the world are not separable" (Gibson
1979: 141). As we have seen, this complementarity is fundamental (if often
tacit) in Gibson's discussion of the perception of affordances: perceiving
what an object affords implicates a "subject" with certain motives and
capacities. This matter of "reciprocity" of object and subject is focal, how-
ever, in Gibson's treatment of the proprioceptive function of vision: "our
awareness of being in the world" (Gibson 1979: 239). The "external sense"
of vision is richly informative about ourselves, and our relation to our sur-
roundings (cf. also Butterworth 1995; Neisser 1994).7
The point is central to Gibson's account of "optic flow", the deforma-
tions of optical structure that derive from our own movements within the
world. The flow structure is, Gibson argued, the basis for our "awareness
of movement or stasis, of starting and stopping, of approaching or retreat-
ing, of going in one direction or another, and the imminence of an encoun-
ter" (Gibson 1979: 236). Similarly, the "visible horizon" expresses the
reciprocity of self and environment: it "is neither subjective nor objective"
(Gibson 1979: 164).
Then, of course, we can literally see ourselves in the world. As I am
typing this chapter, I can see my hands and arms, trunk and legs extending
in front of me, and also, though less distinctly, the frames of my glasses,
eye-brows and nose. Gibson included the famous illustration from Ernst
7. Compare Merleau-Ponty (1993: 37): "Every localization of objects in the
world presupposes my locality. In a sense, an object of perception continu-
ously speaks to us of ourselves. As incarnate subjects, we are expressed by
the object."
Mach's Analysis ofSensations (1885 [1959]) in all of his three main books
(Gibson 1950, 1966, 1979). It is a "view" of a man's office that includes
the "visible ego" of the observer himself, including his nose and the end of
his impressively long moustache. Now, as Hans Lubbe has noted, the im-
plications of this simple image were, indeed, radical:
This drawing is an illustrated criticism of the divorce of subject from object
which renders theory of perception quite incapable of relating one to the
other. The drawing demonstrates how, even in the simple process of seeing,
there is no phenomenal world in which the subject itself is not already pres-
ent, and that there is no subject which is not already present in the world.
(Liibbe 1960 [1978]: 115.)
In a neat switch, Gibson also drew our attention to what we cannot see -
namely, ourselves restricting our own view:
Ask yourself what it is that you see hiding the surroundings as you look out
on the world - not darkness, surely, not air, not nothing, but the ego! (Gib-
son 1979: 112)
6. Conclusion
In this chapter, I have tried to explain the importance of Gibson's critique
of the passive and atrophied body behind much of psychological theory,
and his attempt to provide a radically different kind of psychology based,
not on "stimuli" and "responses", but on an ecology of agency - in other
words, on the material resources for our effective and collective being in
the world. Many of those who have become discontented with the restric-
tive cognitivist or representationalist thinking (cf. Johnson and Rorher this
volume) that pervades the human sciences have been attracted to Gibson's
challenge to the dualisms of subject and object, and of matter and meaning,
that constrain much of modem - and post-modem - thought. But, as I have
been arguing for some time now, we need to be wary of a fundamental
inconsistency in Gibson's work: a vacillation between his insistence upon
the mutuality of organism and environment and his retreat into a standard
kind of realism that would exclude us from the world to be known (e.g.
Costa111981, 1989, 1995,2003,2004; Costall and Still 1989).
It is the realist version of Gibson that dominates not just the introduc-
tory textbooks (Costall and Morris 2004), but also the advanced literature.
On this view, Gibson is indeed committed to a universalistic account of the
76 Alan Costal!
environment, including affordances and information. Thus, even Lakoff
(1987: 216), someone highly appreciative of Gibson's emphasis upon em-
bodiment, has claimed that "the Gibsonian environment is monolithic and
self-consistent and the same for all people", and the ecological approach
cannot make sense of "experiential or cultural categories". But these sup-
posed limitations only follow if we lose sight of the principle of mutuality,
and suppose, as many of Gibson's closest followers do, that "animals or
humans do not enter the picture, except as a scale/actor" (Mace 1977: 50
[emphases added]).
The mutualist alternative is to bring the animated body, the embodied
agent, squarely into the picture. To repeat the crucial passage from Gibson
that I used earlier:
[Affordances] have unity relative to the posture and behavior of the animal
being considered. So an affordance cannot be measured as we measure in
physics. (Gibson 1979: 127-128 [emphases added])
This principle of relativity holds no matter how historically-specific or
indeed idiosyncratic the activity happens to be. Indeed, it is likely that a
good deal of what Gibson himself had to say about the visual control of
locomotion could be most relevant to forms of high speed movement never
experienced by humans before the relatively recent introduction of trains,
cars and planes. Nevertheless, materiality still matters, and what the "later
Gibson" in his mutualist mode was opening up was a view of materiality
and culture as interpenetrating, rather than in opposition (see also Ingold
1996, 2000). Gibson was certainly important for his critique of the dual-
isms fundamental to so much of psychology and social theory. But as I
have been trying to explain, he has also provided us with promising con-
ceptual resources for going beyond those dualisms. There would be no
point however in taking our existing unworldly conceptions of the social,
the cultural and the symbolic and trying to tag them onto the body as pre-
sented to us by the realist Gibson. Although that body is no longer the
passive stimulus-response machine of standard psychological theory, it is
abstracted from any specific circumstances or history or identity. We
would be left with more of the same: the opposition of the "human world"
and the "natural world" that keeps leading the human sciences to suppose
that the social and the cultural exist solely within our individual or collec-
tive heads, and that symbolism could be nothing more than the re-
presentation of a separate and inherently meaningless world (e.g. Ross
2004: 65).
The idea that we do not belong to the natural order of things has a very
long past, extending well beyond Descartes, and it has been central to the
metaphysics of modem science. Gibson is important because he has pre-
sented us with an alternative mutualist scheme in which we can at last see
ourselves as part of nature - part of what nature has, and will, become.
Affordances and information are, as Gibson claimed, "in" the world, but
only in the sense that we are too. It is not just a question of spatial location.
As John Dewey (1958: 295) insisted, we do not exist in the world "as mar-
bles are in a box" but rather "as events are in history, in a moving, growing
never finished process".
References
Arnheim, Rudolph
1969 Art and Visual Perception. (Rev. ed.) London: Faber. [First pub-
lished in 1956.]
Avramides, Anita
2001 Other Minds. London: Routledge.
Baker, Gordon and Morris, Katherine J.
1996 Descartes' Dualism. London: Routledge.
Ben Zeev, Aaron
1993 The Perceptual System: A Philosophical and Psychological Per-
spective. New York: Peter Lang.
Berkeley, George
1709 [1975] Philosophical Works including the Works on Vision. [Introduction
and notes by M.R. Ayers.] London: Dent.
Bruce, Vicki and Patrick Green
1990 Visual Perception: Physiology, Psychology and Ecology. (2nd
edition) Hove, UK: Erlbaum.
Bruner, Jerome, Jacqueline Goodnow and George Austin
1956 A study ofthinking. New York: Wiley.
Butterworth, George
1995 An ecological perspective on the origins of self. In: Jose L. Ber-
mudez, Naomi Eilan and Anthony Marcel (eds.), The Body and the
Self, 87-105. Cambridge, MA: MIT Press.
Cavanagh, Patrick
1999 Top-down processing in vision. In: Robert A. Wilson and Frank C.
Keil (eds.), The MIT Encyclopedia of the Cognitive Sciences, 844-
845. Cambridge, MA: MIT Press.
78 Alan Costall
1989
1991
2004
1995
2003
Connor, Robert D.
1987 The Weights and Measures of England. London: Her Majesty's Sta-
tionery Office (Science Museum).
Costall, Alan
1981 On how so much information controls so much behaviour. In: George
Butterworth (ed.), Infancy and Epistemology, 30-51. New York: St.
Martin's Press/Brighton: Harvester Press.
A closer look at 'direct perception'. In: Angus Gellatly, Don Rogers
and John A. Sloboda (eds.), Cognition and Social Worlds, 10-21.
Oxford: Clarendon Press.
Graceful degradation: Cognitivism and the metaphors of the com-
puter. In: Arthur Still and Alan Costall (eds.), Against Cognitivism:
Alternative Foundations for Cognitive Psychology, 151-170. New
York: Harvester-Wheatsheaf.
Socializing affordances. Theory and Psychology 5: 467-481.
From direct perception to the primacy of action: A closer look at
James Gibson's ecological approach to psychology. In: Gavin J.
Bremner and Alan M. Slater (eds.), Theories ofInfant Development,
70-89. Oxford: Blackwell.
From Darwin to Watson (and Cognitivism) and back again: The
principle of animal-environment mutuality. Behavior and Philosophy
32: 179-195.
Costall, Alan and Paul Morris
2004 The Textbook Gibson: Misrepresentations of an anti-representation-
alist. European Workshop on Ecological Psychology, Torri del
Benaco, Lake Garda, Verona, Italy, 26-29 June, 2004.
Costall, Alan and Arthur Still
1989 Gibson's theory of direct perception and the problem of cultural
relativism. Journalfor the Theory ofSocial Behavior 19: 433-441.
Cottingham, John
1992 Cartesian dualism: Theology, metaphysics and science. In: John
Cottingham (ed.), Cambridge Companion to Descartes, 236-257.
Cambridge: Cambridge University Press.
Cowie, Roddy
1993 On acting in order to see things better. Paper presented at the Work-
shop on "The Primacy of Action", International Society for Ecologi-
cal Psychology, University of Manchester, 13-14 September, 1993.
Danziger, Kurt
1983 The schema of stimulated motion: Towards a pre-history of modem
psychology. History ofScience 21: 183-210.
Descartes, Rene
1960 Meditations. In: Discourse on Method. [A. Wollaston, trans.] Lon-
don: Penguin Books. Original 1637.
1960 Discourse on Method. [A. Wollaston, trans.] London: Penguin
Books. Original 1641.
1971 Correspondence with Princess Elizabeth. In: Elizabeth Anscombe
and Peter Geach (eds.), Descartes: Philosophical Writings, 274-282.
Indianapolis: Bobbs-Merrill. Original 1643.
1966 On mechanism in human action [translated by Tamar March from
L'Homme]. In: Richard Herrnstein and Edwin Boring (eds.), A
Source Book in the History of Psychology, 266-272. Cambridge,
MA: Harvard University Press. Original 1664.
Dewey, John
1958 Experience and Nature. New York: Dover. [Based on the Paul Cams
lectures of 1925.]
1959
1967
1979
1962
1966
Introduction. In: Roy Ellen and Katsuyoshi Fukui (eds.), Redefining
Nature: Ecology, Culture and Domestication, 1-36. Oxford: Berg.
Gesell, Amold, Frances L. Ilg and Louise Bates in collaboration with Glenna E.
Bullis
1977 The Childfrom Five to Ten. (Rev. ed.) New York: Harper and Row.
Gibson, James
1950 The Perception ofthe Visual World. Boston: Houghton Mifflin.
1958 An interpretation of Woodworth's theory of perceiving. In: S. Geor-
gene and John P. Seward (eds.), Current Psychological Issues: Es-
says in Honour ofRobert s. Woodworth, 39-52. New York: Holt.
Perception as a function of stimulation. In: Sigmund Koch (ed.),
Psychology: A Study of a Science, Vol. I, 456-501. New York:
McGraw-Hill.
Observations on active touch. Psychological Review 69: 477-491.
The Senses Considered as Perceptual Systems. Boston: Houghton
Mifflin.
Autobiography. In Edwin Boring and Gardner Linzey (eds.), A His-
tory of Psychology in Autobiography, Vo!. 5, 127-143. New York:
Appleton-Century-Crofts.
The Ecological Approach to Visual Perception. Boston: Houghton
Mifflin.
Gregory, Richard
1997 Eye and Brain: The Psychology of Seeing, 5th ed. Princeton, NJ:
Princeton University Press.
Ellen, Roy
1996
80 Alan Costall
Hall, Stuart
1997 Representation: Cultural Representations and Signifying Practices.
London: Sage.
Harre, Rom
2002 Cognitive Science: A Philosophical Introduction. London: Sage.
Hebb, Donald
1960 The American revolution, American Psychologist 15: 735-745.
Heffeman, Dorothy and Thomson, James
1999 Gone fishin': perceiving what is reachable with rods during a period
of rapid growth. In: Dorothy Heffeman and James A. Thomson
(eds.), Studies in Perception and Action V, 223-228. Mahwah, NJ:
Erlbaum.
2000
2001
Heft, Harry
1989 Affordances and the body: An intentional analysis of Gibson's eco-
logical approach to visual perception. Journal for the Theory of So-
cial Behaviour 19: 1-30.
Ecological Psychology in Context: James Gibson, Roger Barker and
the Legacy of William James's Radical Empiricism. Mahwah, NJ:
Lawrence ErlbaumAssociates.
Hirst, William
1988 Preface. In: William Hirst (ed.) The Making of Cognitive Science:
Essays in Honor of George A. Miller, i-xi. Cambridge: Cambridge
University Press.
Ikegami, Takashi and Jordan Zlatev
this vo!. Frompre-representational cognition to language.
Ingold, Tim
1996 Situating action V: The history and evolution of bodily skills. Eco-
logical Psychology 8: 171-182.
The Perception of the Environment: Essays in Livelihood, Dwelling
and Skill. London: Routledge.
Johnson, Mark and Tim Rorher
this vo!. We are live creatures: Embodiment, American Pragmatism and the
cognitive organism
Klein, H. Arthur
1975 The World ofMeasurements. London: George AlIen and Unwin.
Lakoff, George
1987 Women, Fire and Dangerous Things: What Categories Reveal about
the Mind. Chicago: University of Chicago Press.
Leahey, Thomas
1992 The mythical revolutions of American psychology. American Psy-
chologist 47: 308-318.
1993
Mach, Ernst
1885
Liibbe, Hans
1978 Positivism and phenomenology: Mach and Husserl. [Trans. A. L.
Hammond.] In: Thomas Luckman (ed.), Phenomenology and Sociol-
ogy, 90-118. London Penguin Books. [Article fust published in
1960.]
Mace, William
1977 James J. Gibson's strategy for perceiving: Ask not what's inside your
head, but what your head's inside of. In: Robert Shaw and John
Bransford, (eds.), Perceiving, Acting and Knowing: Toward an
Ecological Psychology, 43-65. Hillsdale, NJ: Erlbaum.
The Analysis of Sensations and the Relation of the Physical to the
Psychical. New York: Dover. Reprint 1959.
Macnamara, John
1999 Through the Rearview Mirror: Historical Reflections on Psychology.
Cambridge, MA: MIT/Bradford Books.
Mark, Leonard
1987 Eyeheight-scaled information about affordances: A study of sitting
and stair climbing. Journal of Experimental Psychology: Human Per-
ception and Performance 13: 361-370.
Merleau-Ponty, Maurice
1965 The Structure of Behaviour. [trans. Alden L. Fisher.] London:
Methuen. [Original French edition published in 1942.]
The experience of others (1915-1952). (F. Evans and H. J. Silver-
man, trans.) In: Keith Hoeller (ed.), Merleau-Ponty and Psychology,
33-63. New Jersey: Humanities Press. [First English translation of
Merleau-Ponty's 1951-1952 lecture course L'Experience d'autrui,
given at the University of Paris (Sorbonne).]
Miller, George, Eugene Galanter and Karl Pribram
1960 Plans and the Structure of Behavior. New York: Holt, Rinehart and
Winston.
Neisser, Ulrich
1994 Self-perception and self-knowledge. Psyke and Logos 15: 392-407.
1997 The future of cognitive science: An ecological analysis. In: David M.
Johnson and Christina E. Emeling (eds.), The Future ofthe Cognitive
Revolution, 247-260. New York: Oxford University Press.
Newell, Karl, Diedre Scully, F. Tenenbaum and S. Hardiman
1989 Body scale and the development of prehension. Developmental Psy-
chobiology 22: 1-13.
Pylyshyn, Zenon
1986 Computation and Cognition. Cambridge, MA: MIT Press.
82 Alan Costal!
Reed, Edward
1988 James Gibson and the Psychology of Perception. New Haven: Yale
University Press.
1997 The cognitive revolution from an ecological view. In: David M.
Johnson and Christina E. Emeling (eds.), The Future ofthe Cognitive
Revolution, 261-273. New York: Oxford University Press.
Richardson, Robert C.
1982 The "scandal" of Cartesian interactionism. Mind 91: 20-37.
Roca, Josep, Martinez, Lizandra Mireia, Anna Fabregas Mireia and Anna Cordoner
1986 Registres evolutius motors: Una observaci6 critica. Apunts 6: 61-64.
Ross, Norbert
2004 Culture and Cognition: Implications for Theory and Method. Thou-
sand Oaks, CA: Sage.
Schiffer, Michael Brian (with Andrea R. Miller)
1999 The Material Life of Human Beings: Artifacts, Behavior and Com-
munication. London: Routledge.
Sedgwick, Howard
1973 The visible horizon: A potential source of visual information for the
perception of size and distance. Unpublished doctoral dissertation,
Comell University, January 1973.
Still, Arthur and Alan Costall (Eds.)
1991 Against Cognitivism. New York: Harvester Wheatsheaf.
Stins, John, Endre Kadar and Alan Costall
2001 A kinematic analysis of hand selection in a reaching task. Laterality
6: 347-367.
Straker, Stephen
1976 The eye made "other"; Durer, Kepler and the mechanisation of sight
and vision. In: Louis A. Knafla, Martin S. Staum and T. H. E. Tra-
vers (eds.), Science, Technology and Culture in Historical Perspec-
tive, 7-25. University ofCalgary Studies in History, No. 1.
Szokolosky, Agnes
2004 An interview with Eleanor Gibson. Ecological Psychology 15: 271-
281.
Thurstone, Louis
1923 The stimulus-response fallacy in psychology. Psychological Review
30: 354-369.
Tibbetts, Paul
1973 Historical note on Descartes' psychophysical dualism. Journal of the
History ofthe Behavioral Sciences 9: 162-165.
Toulmin, Stephen
1993 From clocks to chaos: Humanizing the mechanistic world view. In:
Herman Haken, Anders Karlqvist and Uno Svedin (eds.), The Ma-
chine as Metaphor and Tool, 139-153. Berlin: Springer-Verlag
Warren, Howard
1922 Elements ofHuman Psychology. Boston: Houghton Mifflin.
Warren, William and S. Whang
1987 Visual guidance of walking through apertures: Body-scaled informa-
tion for affordance. Journal of Experimental Psychology: Human
Perception and Performance 13: 371-383.
Warren, William, Jr.
1984 Perceiving affordances: Visual guidance of stair climbing. Journal of
Experimental Psychology: Human Perception and Performance 10:
683-703
1995 Constructing an econiche. In: John Flach, Peter Hancock, Jeff Caird
and Kim Vicente (eds.), Global Perspectives on the Ecology of Hu-
man-Machine Systems, Vo!' 1,210-237). Hillsdale, NJ: Erlbaum.
Westland, Gordon
1978 Current Crises in Psychology. London: Heinemann.
Williams, Emma, Alan Costall and Vasu Reddy
1999 Children with autism experience problems with both objects and
people. Journal of Autism and Developmental Disorders 29: 367-
378.
Wilson, Margaret
1980 Body and mind from the Cartesian point of view. In: Robert W. Rie-
ber (ed.), Body and Mind: Past, Present and Future, 35-55. New
York: Academic Press.
Ziemke, Tom
2001 The construction of 'reality' in the robot. Foundations of Science 6
(1): 163-233.
From the meaning of embodiment to the
embodiment of meaning: A study in
phenomenological selDiotics
Goran Sonesson
A Qualisign /---I cannot actually act as a sign until it is
embodied; but its embodiment has nothing to do with
its character as a sign. A Sinsign /---I involves a qual-
isign, or rather, several qualisigns. But these qualisigns
are of a peculiar kind and only form a sign through
being actually embodied. Charles S. Peirce, Nomen-
clature and Divisions ofTriadic Relations
Abstract
Unlike much of the contemporary discussion of embodiment, phenomenology is
really involved with the body as a kind of meaning appearing to consciousness; and
it does not only attend to the body of the biological organism, but also to the kind
of organism-independent artefacts which are required by some sign systems. Be-
cause it is concerned with meaning, phenomenology is akin to semiotics. From the
point of view of the latter discipline, however, signs must be distinguished from
other meanings, and clear criteria are needed for doing so. At least one such crite-
rion can by found in the work of Piaget: differentiation. Meaning in the more gen-
eral sense of organisation and selection is at the basis of the common sense world,
and thus accounts for what is known in Cognitive Linguistics as "image schemas".
Cognitive Linguistics, just as biosemiotics, ignores this important distinction.
Moreover, some cognitive linguists seem to deny the distinction between organism
and environment, which must prevail if "image schemas" are to be acquired, along
the lines of earlier conceptions of schematisation. On the basis of these considera-
tions, a developmental sequence can be suggested going from schemas to signs and
organism-independent artefacts.
Keywords: body, ecology, embodiment, evolution, Lifeworld, memory, phenome-
nology, picture, semiotic function, semiotics, sign.
86 Goran Sonesson
1. Introduction
In our time, in which the term "embodiment" is put to quite new (and, to
my mind, either fuzzy or redundant) uses, authors such as Johnson (1987)
and Varela, Thompson and Rosch (1991) have not failed to suggest a con-
tinuity with an earlier discussion of embodiment, taking place about a
century ago, notably within phenomenological philosophy (e.g., Husserl
1973b; 1976). Yet these references to phenomenology seem to me to be
fairly superficial, and the grasp of the phenomenological notion of em-
bodiment shown often appears to be incomplete, if not inadequate. This is
why I will start out by explaining the emergence of the problem of em-
bodiment within phenomenological philosophy. Taking a cue from the
phenomenologists themselves, I will also suggest that phenomenology may
be interpreted as a branch of psychology, and thus serve as an ingredient of
cognitive science as well as a basis for semiotic theory. From there on, my
search for the multiple "bodies of the mind" will follow a somewhat spi-
ralling movement: first, I will argue that the concept of sign or representa-
tion, which I take to be indispensable for our understanding of human con-
sciousness, supposes something of a body of its own. Then we will see
how meaning, which is not specifically embodied in signs, is a requisite, in
both a systematic and an evolutionary sense, for the attainment of the sign
function (Piaget 1945; Sonesson 1992b). I will go on to suggest that what is
elsewhere known as "image schemas" (e.g. Johnson 1987; Lakoff and
Johnson 1999, Johnson and Rohrer this volume) do indeed constitute a
level of meaning prior to the sign but, for that very reason, are not directly
involved in metaphors, which, in my view, and that of the tradition of clas-
sical rhetoric, must be construed as signs, and indeed signs standing for
other signs (cf. Sonesson 1989, 1998b). Finally, we will look at embodi-
ments of meaning in a rather different sense, of the kind which develops,
phylogenetically and perhaps also ontogenetically, after the attainment of
the (linguistic) sign, such as pictures, writing, and theories, that is, organ-
ism-independent sign-vehicles spanning time and/or space. My aim is not
to exhaust the repertory of embodiments of meaning, but merely to ex-
pound some of their varieties, and to pinpoint their different evolutionary
import.
From the meaning ofembodiment to the embodiment ofmeaning 87
2. The Cartesian divide: Where angels fear to tread
In the philosophical tradition, embodiment emerges as a problem within the
philosophy of consciousness, which aims to reconstruct the world as given
to a (generic) subject. In this sense, embodiment gives rise to two separate
strands in the particular version of the philosophy of consciousness inaugu-
rated by Russerl, known as phenomenology:
in relation to the physical body of the subject itself and/or his or her
counterpart in perceptual space, the generic other;
in relation to signs and other overarching structures, which, like the
physical body, appear in the mind, without being of the mind, and
seem to require some kind of physical substratum in order to exist.
2.1. Phenomenology from the phenomenological point of view
The justification for a philosophy of consciousness is of course that in the
common sense world, which Russerl later was to baptise the Lifeworld,
everything there is is accessible to us through consciousness. The paradox
is that, at the same time, the body, our own, as well as that of the other,
cannot be a mere figment of consciousness. To paraphrase the classical
dictum of 19
th
century psychology reemerging in the modem discussion of
consciousness (cf. Dennett 1991), the body is not a mere epiphenomenon of
consciousness. Indeed, this transcendence of our physical being to con-
sciousness is itself part of the Lifeworld. As Max Scheler (quoted by Gur-
witsch 1985) nicely put it, "we know that we are no angels", that is, no
free-floating sprits without bodies.
The second strand is quite different: genuine semiotic structures such as
mathematical concepts, logic, and even language appear to transcend con-
sciousness much in the mode of a Regelian "absolute spirit". They are, in
Russerlean terms, "idealised" in order to be detached from their depend-
ence on individual subjects - which is why they may harbour what Deacon
(2003) has recently called "semiotic constraints", whose origin is inde-
pendent of both nature and nurture. And yet, as Russerl (1962a: 365-386)
recognised in his study of the origin of geometry, for the idealisation to be
complete, its products have to be "embodied" in some kind of notational
system, because only in that way can they gain a stable, public existence in
a domain completely separate from their instantiations in the practical
88 Goran Sonesson
situations of the Lifeworld. More recently thinkers from separate traditions
such as Ivins (1953), Innis (1950), and Donald (1991), have regained this
insight in some form or other.
The task of phenomenology, as Russerl saw it, was to explain the possi-
bility of human beings having knowledge of the world; as a philosophical
endeavour, phenomenology is about the way the world of our experience is
"constituted". As a contrast, psychology is not about the world, but about
the subject experiencing the world. Rowever, every finding in phenome-
nological philosophy, Russerl claims, has a parallel in phenomenological
psychology, which thus could be considered a tradition within psychologi-
cal science (cf. Russerl 1962b; Gurwitsch 1974). If consciousness is a re-
lation connecting the subject and the world, then phenomenology is con-
cerned with the objective pole and psychology is about the subjective one.
It is often forgotten that Russerl not only inspired but himself was inspired
by the Gestalt psychologists. Close followers of Russerl such as, most no-
tably, Gurwitsch (1957, 1966), were as much involved with phenomenol-
ogical psychology as with philosophy and discussed the findings not only
of the psychology of perception but of contemporary contributors to neuro-
biology such as Gelb and Goldstein. Also the early Merleau-Ponty (1942,
1945),1 was, in this respect, an exponent ofphenomenological psychology.
Many of those who are concerned with embodiment today appear to
come from the diametrically opposite camp. Edelman (1992), for instance,
clearly does not discover the body from the horizon of consciousness, but
quite the opposite, he implies that the mind cannot be divorced from the
body. In a sense, this is hardly controversial: unlike those hypothetical
angels, human beings can only boast a mind as long as they have a body.
But, if this is true in the order of existence, it is not necessarily so from the
point of view of investigation. After all, Brentano (1885) did not use a
scalpel, much less fMRI, to discover the property of intentionality (in the
sense of directedness), which Edelman recognises as an irreducible char-
acteristic of consciousness; nor did James (1890) find any of those "Jame-
sian properties" of consciousness repeatedly mentioned by Edelman in
such a way.
Indeed, far from being "a deliberately non-scientific set of reflections
on consciousness and existence" (Edelman 1992: 159), phenomenology
started out from the fact of intentionality and attempted to probe ever
1. Who may not quite deserve the hero status given to him by Varela, Thompson
and Rosch (1991); See also Gallagher, this volume.
deeper into its ramifications, in order to rediscover and amplify those very
Jamesian properties of consciousness mentioned by Edelman. Russerl and
Gurwitsch may have been wrong to think of phenomenology as a discipline
completely separate from biology and psychology, but the relative discon-
nection of phenomenological reflections, like those of Brentano and James,
from biologicallmowledge has no doubt borne rich intellectual fruit. If "a
biologically based theory of mind" can in some respects "invigorate" phe-
nomenology, the opposite is certainly just as true.
Interestingly, Edelman (1992; Edelman and Tonini 2000) claims that
consciousness as such cannot be a spurious occurrence, because that would
not have made evolutionary sense. That is, consciousness is not an epiphe-
nomenon. But we have seen that, to classical embodiment philosophy, the
problem is to show that the body is not an epiphenomenon.
2.2. The science of common sense and its operations
The apparent paradox arises because, in the two cases, the point of view is
entirely different. Phenomenology, like the science of semiotics, takes as
its point of departure the way things make sense to us, that is, how they
mean. In this very broad sense phenomenology accomplishes a semiotical
reduction: things are considered only from the point of view of their having
meaning to us (where "we" might be people of a particular culture or sub-
group, or humankind in general)? From a phenomenological point of view,
there is, in a sense, no way of overcoming the divide formulated by Des-
cartes, for Descartes did not invent it: it is intrinsic to that phenomenon
which, in Descartes' own words, is the most widely distributed one in the
world, common sense. Common sense is not notorious for being right, but
if we ask ourselves how the body (and the rest of the world) makes sense to
us, then common sense is our very subject matter. Even so, common sense
gives rise to an apparent contradiction: my body is necessarily experienced
through my consciousness, but in my consciousness it is experienced as
2. Elsewhere (Sonesson 1989 26ft), I have opposed, in this sense, the qualitative
reduction to the more familiar quantitative one, characteristic of the traditional
natural sciences. There are similarities, but also differences, to the series of "re-
ductions" distinguished by Husserl: the phenomenological and eidetic reduc-
tions, notably.
90 Goran Sonesson
being outside of it.
3
All post-Cartesian meditations, including those of
Russerl (1973a) and those of Merleau-Ponty (1945), have been concerned
to account for this paradox. To do so, it is necessary to accomplish a
painstaking analysis (of which there can be no better example than the
posthumous papers of Russerl himself, together with the - also largely
posthumous - works of Peirce) of all those structures of the mind that are
normally at the margin of consciousness (cf. below 5.2).
In this sense, all human and social sciences which aspire to discover
regularities, such as linguistics and other semiotic sciences, necessarily
start out from phenomenology - and we should be happy if those phe-
nomenological investigations sometime manage to be as meticulous as
those of Russerl and Gurwitsch.
Saussure famously observed that "linguistics and the other semiological
sciences" are so difficult, because they are not concerned with anything
material: indeed, he continued, their subject matter is the point of view we
take on material things. Starting from this principle, Prieto (1975a: 140ff,
1975b: 215ft) has claimed, that, contrary to what is ordinarily taken for
granted, it is natural science which is subjective, since it has to take a stand
on physical reality, which as such is indifferent, whereas semiotics is capa-
ble of objectivity, in so far as it describes the subjective point of view of
individuals and communities. According to another formulation, the object
of linguistics is the knowledge common to the speaker and hearer (1975a:
110), i.e. it produces knowledge about knowledge, not, as the natural sci-
ences, about the material world (1975a: 140ft). Prieto thus postulates a
simple coincidence between the object and the discourse of semiotics. It is,
however, less the phoneme, than the features defining it, which are relevant
to linguistics, and these are not ordinarily identified by the speaker. In
more recent linguistics, it is the "deep structure" or the "image schemas"
which are claimed to be relevant for linguistic knowledge, not the particu-
lar syntactic form or stylistic turn, of which the speaker is usually aware.
We therefore conclude that the linguist, and the semiotician generally, may
have to descend at least one level of analysis below the ultimate level of
which the user is aware.
Put into traditional epistemological terms, we may say that after coin-
ciding with the user in his understanding of the phoneme, the semiotician
3. Strictly speaking, this is not the problem of our own body, nor of the other, but
the more general one of the external world, as pointed out by Gurwitsch (1979:
26f). Still, it is quite sufficient for us to note that it also applies to the body.
goes on to explain the conditions of possibility of this understanding on the
level of distinctive features. In this case, semiotics contains the knowledge
of the user and something more, and, quite apart from the problem of ob-
taining the correct understanding, this explicative part introduces an ele-
ment of subjectivity. We shall say that what is of primary importance to
semiotics is operative knowledge, i.e. knowledge that must exist at some,
probably low, level of awareness, in order to render behaviour understand-
able (and thus explainable); thus, it is not discursive knowledge, the spon-
taneous theories of the user, which might be what we first tend to identify
with common sense. The operation of ideation, familiar to the phenome-
nologist, the commutation text of structuralist linguistics, the grammatical-
ity or acceptability judgement of the grammarian, and some varieties of
psychological experimentation are all techniques for attaining these layers,
bringing that which is at the margin of consciousness into its centre (cf.
Sonesson 1989: 27ff; and see Zlatev, this volume, for a similar argument).
In phenomenological semiotics, then, we are concerned, in the first
place, with the figure of the body as it appears on the horizon of con-
sciousness. Once we have described this figure - better than James,
Russerl, and so on - we may try to explain it, delving ever deeper into the
margins of consciousness. We can of course try to search for explanations
outside of consciousness, but we must be aware that this is a complete
change of direction.
Most contemporary theories of embodiment do not appear to pose the
question of meaning. Varela, Thompson and Rosch (1991) start out from
the phenomenology of Merleau-Ponty, but, after the first few pages, it is
not really clear how the issues they discuss relate to the phenomenological
problem of the body, i.e. the body as it appears to consciousness. Lakoff
and Johnson (1999: 102) distinguish three different levels of embodiment,
which they refer to as "the neural level, phenomenological conscious expe-
rience and the cognitive unconscious", none of which, in the end, seems to
have anything to do with meaning, as opposed to neurobiology.4
Both senses of embodiment characterised from a phenomenological
perspective at the beginning of this section involve a process by which
something not recognized as a body presents itself as a being one: in the
first case, a mind is being situated in the world; in the second case an idea
4. See Zlatev this volume for a discussion of whether these levels can reasonably
be separated, and, in particular, of the problematic character of the "cognitive
unconscious".
92 Goran Sonesson
is being reified into an object publicly accessible to all. By denying the
distinctions both between body and mind and expression and content,
scholars such as Lakoff and Johnson deprive themselves of the very foun-
dations needed by their own notion of "image schemas". To see this, how-
ever, we have to start by specifying the concepts of sign and schema.
3. Meaning embodied in signs
It is true of both main traditions of semiotics, the Saussurean and the Peir-
cean, that they have never really offered any specific definition of the sign
- by which I mean a set of criteria permitting us to separate meanings
which are signs from other meanings. The same thing appears to apply to
the notion of representation in cognitive science (cf. Sonesson 1992b,
2003a, 2003b, 2006, forthcoming). This goes a long way to explaining why
many semioticians (such as Greimas, Eco, etc.) have rejected the sign,
without much of an argument, and why the second generation of adepts to
cognitive science (e.g. Lakoff and Johnson 1999; Johnson and Rohrer this
volume) now seem to be doing the same thing with reference to the notion
of representation. So before we can pose any questions about the psycho-
logical and evolutionary role of the sign concept, we have to be clear about
what it is. This involves not only deciding the criteria for analysing a phe-
nomenon of meaning into two separate parts, but also those allowing us to
posit an asymmetrical relation between these parts: not only does the ex-
pression have to be separate from the content, but the former should stand
for the latter, not the reverse.
3.1. From pebbles to feathers: The notion of differentiation
When Peirceans and Saussureans quarrel over the presence of two or three
entities in the sign, they seldom pause to ask themselves what kind of ob-
jects, defined by what type of features, are involved. The whole question
becomes moot if there is no reason to analyse meaning into two parts, as
suggested by both contemporary cognitive scientists and old-time existen-
tialists and Lebensphilosophen. What, then, is it that permits us to deter-
mine that an object endowed with meaning is made up of an expression, or
"representamen", and a content, or "object" (where further instances of the
Peircean version are not relevant)? Peirceans and Saussureans alike would
no doubt agree that signs have something to do with the classical formula,
often quoted by Jakobson (1975), aliquid stat pro aliquo ("something in
the place of something else"), or, as, Jakobson also puts it, more simply,
with renvoi, or reference. But this formula itself is vague or ambiguous.
Before we can separate signs from other meanings, we have to spell out
those criteria for something being a sign that are simply taken for granted,
both in the Peircean and in the Saussurean tradition. This can be done by
combining what Russerl says about appresentation (something which is
directly present but not thematic refers to something which is indirectly
present but thematic) and what Piaget says about the semiotic function
(there is a differentiation between expression and content in the double
sense, I take it, that they do not go over into each other in time and/or
space, and that they are perceived to be ofdifferent nature).
Phenomenology, which is not afraid of spelling out the self-evident,
may offer some help here. Saint Augustine, who has often (as so many
others) been hailed as the first semiotician, defined the sign as "a thing
which, over and above the impression it makes on the senses, causes
something else to come into thought as a consequence" (as translated by
Deely 1982: 17ff). Russerl's (1913, 1939) own definition of the sign,
which describes the expression as something which is directly perceived
but not in focus, and the content as being indirectly perceived while at the
same time being the focus of the relation, could be taken as a way of speci-
fying the Augustinean suggestion.
5
Piaget certainly abides by Saussure opposing the sign to the symbol
(where the latter is the motivated sign). What Piaget added to Saussure was
most obviously a developmental perspective, in particular on the level of
ontogeny. But, just as importantly, though it has seldom been observed (cf.
Sonesson 1992b, etc.), he realised that not all meanings are signs or sym-
bols, and he even began groping for a definition of that which accounts for
the specificity of the sign. According to Piaget the sign function (which
Piaget himself called first the symbolic, and then the semiotic function) is a
capacity acquired by the child at an age of around 18 to 24 months, which
enables him or her to imitate something or somebody outside the direct
presence of the model, to use language, make drawings, play "symboli-
5. These observations could be taken to imply that the content is "embodied" in
the expression. Expression would stand to content as body to soul. This was ex-
plicitly suggested by Cassirer (1957: 100), but it is also hinted at in some pas-
sages by Peirce. The parallel is nonetheless, in my view, seriously flawed (as
will be discussed in Section 3).
94 Goran Sonesson
cally", and have access to mental imagery and memory. The common fac-
tor underlying all these phenomena, according to Piaget, is the ability to
represent reality by means of a signifier, which is distinct from the signi-
fied. Indeed, Piaget argues that the child's experience of meaning predates
the sign function, but that such meaning does not suppose a differentiation
of signifier and signified (see Piaget 1945, 1967, 1970). In several of the
passages in which he refers to the sign function, Piaget goes on to point out
that "indices" and "signals" are possible long before the age of 18 months,
but only because they do not suppose any differentiation between expres-
sion and content. The signifier of the index, Piaget (1967: 134ft) says, is
"an objective aspect of the signified"; thus, for instance, the visible ex-
tremity of an object which is almost entirely hidden from view is the signi-
fier of the entire object for the baby, just as the tracks in the snow stand for
the prey to the hunter. But when the child uses a pebble to signify candy,
he is well aware of the difference between them, which implies, as Piaget
tells us, "a differentiation, from the subject's own point of view, between
the signifier and the signified" (ibid.)
Piaget is quite right in distinguishing the manifestation of the sign func-
tion from other ways of "connecting significations", to employ his own
terms. Nevertheless, it is important to note that, while the signifier of the
index is said to be an objective aspect of the signified, we are told that in
the sign and the "symbol" (i.e. in Piaget's terminology, the conventional
and the motivated variant of the sign function, respectively) expression and
content are differentiated from the point of view of the subject. Curiously,
this distinction between the subjective and objective points of view is
something Piaget seems to lose track of in his further discussion.
We can, however, imagine this same child that in Piaget's example uses
a pebble to stand for a piece of candy having recourse instead to a feather
in order to represent a bird, or employ a pebble to stand for a rock, without
therefore confusing the part and the whole: then the child would be em-
ploying a feature, which is objectively a part of the bird, or the rock, while
differentiating the former form the latter from his point of view. Only then
would he be using an index, in the sense in which this term is employed in
semiotics, that is, in (what this semiotician takes to be) the Peircean sense
of the term. Contrary to what Piaget implies, the hunter, who identifies the
animal by means of the tracks, and then employs them to find out which
direction the animal has taken, and who does this in order to catch the ani-
mal, does not, in his construal of the sign, confuse the tracks with the ani-
mal itself, in which case he would be satisfied with the former. Both the
child in our example and the hunter are using indices, or indexical signs,
where the "real" connection is transformed into a differentiation.
On the other hand, the child and the adult will fail to differentiate the
perceptual adumbration in which they have access to the object from the
object itself; indeed, they will identify them, at least until they change their
perspective on the object by approaching it from another vantage point.
And at least the adult will consider a branch jutting out behind a wall as
something that is non-differentiated from the tree, to use Piaget's example,
in the rather different sense of being a proper part of it.
6
In the Peircean
sense an index is a sign, the relata of which are connected, independently
of the sign function, by contiguity or by that kind of relation that obtains
between a part and the whole (henceforth termedfactorality). But of course
contiguity and factorality are present everywhere in the perceptual world
without as yet forming signs: we will say, in that case, that they are mere
indexicalities. Perception is perfused with indexicality.7
An index, then, must be understood as indexicality (an indexical rela-
tion or ground, to use an old Peircean term) plus the sign function. Analo-
gously, the perception of similarities (which is an iconic ground) will only
give rise to an icon when it is combined with the sign function. Deacon
(1997: 76ft) must therefore be wrong when he claims that camouflage in
the animal world such as the moth's wings being seen by the bird as "just
more tree" are essentially of the same kind as those "typical cases" of ico-
nicity we are accustomed to call pictures. As always, there are passages in
Peirce's work which may be taken in different ways, but it makes more
systematic and evolutionary sense to look upon iconicity and indexicality
as being only potentials for something being a sign which still have to be
"embodied" (cf. Section 4).
While the introduction of the notion of differentiation is a substantial
accomplishment on the part of Piaget, he unfortunately never spells out its
import. He defines differentiation in terms of the subject's point of view,
but then uses examples in which the disconnection already exists objec-
tively, as pointed out above. Objectivity can here, I take it, be identified
6. About proper parts, perceptual perspectives, and attributes as different ways of
dividing an object and thus different indexicalities, cf. Sonesson 1989: 1.2.).
7. I am using "indexicality" here (just as "iconicity") in the sense of something
which is necessary for a sign being an index (or an icon), but which cannot
function "as a sign until it is embodied". See, in particular, Sonesson (1993a,
1998a, forthcoming)
96 G6ran Sonesson
with the common sense world, which the child, in Piagetean terms, is in the
process of "constructing".
Differentiation should not be identified with displacement as defined by
Hockett (1977), which (rightly, no doubt) appears as one of the "design
features" of language in most introductory textbooks. As in the case of the
tracks left by the hunted animal, displacement may be a consequence of
differentiation. But differentiation only comes on its own when the sign is
in presence of its referent, for then it allows us to construe reality in differ-
ent ways ("subjectively", as Piaget would have said), picking out that
which is relevant, and ignoring, or downplaying other features.
We must be careful not to confuse different relationships involving the
sign. Differentiation, in Piaget's sense, must pertain to the signifier and the
signified, which are always equally present in the here and now of the sign
usef, since they are mental (or, in some cases, intersubjective) entities. To
the hunter, both the signifier and the signified of the tracks are present here
on the ground (Of, to be precise, on the ground as he perceives it). But the
signified contains the information that is itself only part of a larger whole
(or rather something once contiguous to a larger whole) which was present
here at an earlier time, but which is now elsewhere, more precisely in the
direction indicated by the tracks. And the displacement, in Hockett's sense,
has taken place between that signified whole and the real animal, which is
now present somewhere else.
When the sign, whether it is a stretch of discourse, a picture, or an ani-
mal track, is present along with the referent, however, the signified allows
us to refocus the referent, in other words, to present it in a particular per-
spective. For this the sign requires independence: that is so say, a "body"
of its own.
3.2. Some other ways of "connecting significations"
As presented here, the concept of sign (or representation) does not include
ordinary perception: our way of being in the world is not to be likened to
the presence at some kind of private theatre. Second generation cognitive
scientists (cf. Johnson and Rohrer this volume) are therefore quite right in
rejecting the notion of representation of their forbears. They are wrong,
however, to reject all kinds of representation (to the extent that it corre-
sponds to the sign function). More fundamentally, they commit a serious
error by not attending to the definition of representation before rejecting it
altogether. A few notions of history may help us to disengage ourselves
from the present-day conceptual muddle.
As was noted above, Augustine seems to have been responsible for
making explicit the common sense notion of sign on which later thinkers
such as Saussure and Russerl (and, at least in his definitions, Peirce) are
tacitly building: it is, he tells us (in the convenient paraphrase of Deely
1994: 58) "something which, on being perceived, brings into awareness
another besides itself'. Thomas Aquinas already had some misgivings
about this definition, without ever daring to reject it outright. The followers
of Aquinas in Paris may have been somewhat bolder. In a written form
which has come down to us, however, we first know this criticism from the
works of Pedro da Fonseca, who was active in Coimbra, Portugal, in the
16
th
century. To Fonseca and his followers, the definition of the sign must
be considerably broader: a sign is anything which serves to bring into
awareness something different from itself, whether the sign (in the sense of
the signifier) itself becomes subject to awareness in the process or not. If
the sign itself does not have to be perceived in order for us to come to an
awareness of that which is signified, Fonseca described it as beingformal;
but if the sign cannot lead to the awareness of anything at all unless it is
itself perceived, he called it instrumental (cf. Deely 1982: 52ff, 1994:
58ff). Thus, Fonseca pointed to a distinction, which seems to have been
lost by latter-day semioticians and cognitive scientists.
What is here called an instrumental sign clearly is that which we, fol-
lowing Russerl and Brentano, but also Edelman, have described as the
fundamental trait of consciousness, intentionality, that is, the property of
being directed to that which is outside of consciousness. In fact, when
closely considered, Fonseca's observations really go against the grain of
the identification of our awareness of the world with the sign. It echoes
Russerl's as well as Gibson's description of the perceptual act as some-
thing which points beyond itself without itself being present to conscious-
ness (cf. Sonesson 1989: ill.3.2).
Indeed, when Gibson (1978: 228) observes that, when we are con-
fronted with the cat from different points of view, etc., what we really see
is all the time the same invariant cat, he actually recovers the central theme
of Russerlian phenomenology, according to which the object is entirely,
and directly, given in each of its perspectives or noemata (see Russerl
1939, 1962a, 1962b, 1973b; Sonesson 1989: 1.2.2). In a similar fashion,
Russerl's favourite example is a cube which can be observed from differ-
ent sides. In Gibsonean terms, these are "the surfaces of the world that can
98 Goran Sonesson
be seen now from here" (Gibson 1978: 233). Husserl's cube and Gibson's
cat instantiate the same phenomenal fact.
Just as Husserl called into question the conception of his contemporary
Helmholtz, according to which consciousness is like a box within which
the world is represented by signs and images, from whose fragmentary
pieces we must construct our perceptions (cf. Kiing 1973), so Gibson's
strawmen are the followers of Helmholtz, the so-called "constructionists"
(who have recently re-emerged within cognitive science; cf. Hoffman
1998), who claim that hypotheses are needed to build up perceptions from
the scattered pieces offered us by sensation (cf. Sonesson 1989: 111.3.3).8
Husserl rejected the picture metaphor of consciousness, showing Brentano
and Helmholtz to be in error in their very conception of pictures and other
signs, because they ignored the transparency of the expression to the con-
tent. Gibson (1978) instead emphasises the dissimilarity of the picture from
a real-world scene, thus showing numerous experiments using pictorial
stimuli to study normal perception to be seriously misguided. To both
Husserl and Gibson, normal perception gives direct access to reality; pic-
tures, however, constitutes a kind of indirect perception to Gibson, while
to Husserl (1980) they are "perceptually imagined" (cf. Sonesson 1989:
ill.3.6, forthcoming).
To perceive surfaces is a very different thing from perceiving marks on
surfaces, Gibson (1980) maintains. Depth is not added to shape, but is im-
mediately experienced. In fact, the perception of surfaces, of their layout,
and of the transformations to which the latter are subjected, is essential to
the life of all animal species, but the markings on these surfaces have only
gained importance to man, notably in the form of pictures (Gibson 1980:
xii, 1978: 229). Surfaces have the kind of meaning which Gibson else-
where calls "affordances"; the markings on surfaces, however, have "refer-
ential meaning". Without discussing the exact import that should be given
to the term "affordance" (cf. Costall this volume), we may safely conclude
that "referential meaning" is a property of what we have called the sign
function. That is, surfaces do not standfor other surfaces, but the markings
on surfaces may possibly do so. The pattern of a surface and the pattern on
a surface are different, and can usually be distinguished by an adult. The
8. Reed (1996) notes some parallels between Gibson and the American pragma-
tists (without, however, referring to Peirce). On Gibson's sources, also see
Costall this volume.
surface on which a "graph" has been executed can be seen underneath the
"graph".
To Gibson, then, the picture is a surface among other surfaces before
becoming a sign. Gibson (1978: 231) observes that, besides conveying the
invariants for the layout of the pictured surfaces, the picture must also
contain the invariants of the surface that is doing the picturing: those of the
sheet of paper, the canvas, etc., as well as those of the frame, the glass, and
so on. Although Gibson does not use the term, he clearly describes the
picture as a sign, in the strict, Augustinean sense of the word: as a surface
which, on being perceived, brings into awareness something besides itself.
Gibson never specifies what he means when he claims that surfaces are
only seen to stand for something else by (adult) human beings, in contra-
distinction to animals and infants. If he meant to suggest that surfaces can
never be taken to be something else than surfaces by animals and children
he was clearly wrong: we know that even doves may react the same way to
a picture as to that which is depicted (cf. Sonesson 1989: ill.3.1). The dif-
ficulty, clearly, consists in seeing, at the same time, both the surface and
the thing depicted.
We should grant Fonseca the insight that there is some kind of analogy
between signs and intentional acts. However, to use the term sign in both
cases dangerously suggests that there is no important distinction to be
made. In his late life, Peirce realised that all his notions were too narrow:
instead of "sign", he reflected, he really ought to talk about "medium" or
"mediation" (manuscript quotations given in Parmentier 1985). In the fol-
lowing, we will use the term mediation for this general sense of meaning
which Fonseca called sign and to which Peirce sometimes also may be
hinting. In some respects, at least, it seems to correspond to Gibson's "af-
fordances", and to Piaget's notion of "connecting significations".
4. On the way to the human Lifeworld
If there is meaning before signs, then even the immediate experience of
perception is in some very general sense "mediated". The semiotician A. J.
Greimas (1970: 49) once suggested that there could be a cultural science of
nature, a semiotics of the natural world - which was concerned, then, with
the world that is natural to us, just as a particular language is our "natural
language". But Greimas was not the first to conceive of a cultural science
of nature. His semiotics of the natural world, together with Husserl's sci-
100 Goran Sonesson
ence of the Lifeworld, and "ecological physics" as invented by Gibson are
all sciences of normality, of that which is so much taken for granted that it
is ordinarily not considered worthy of study (cf. Sonesson 1989, 1994,
1996,).
It may seem strange to put together ideas and observations made by a
philosopher, a psychologist, and a semiotician; yet these proposals are
largely the same; indeed, there are indications that both Greimas and Gib-
son took their cue from Russer!' Greimas, Gibson, and Russerl all felt the
need for such a science because they realised that the "natural world", as
we experience it, is not identical to the one known to physics but is con-
ceived from the standpoint of human consciousness. Russerl's Lifeworld as
well as Gibson's ecological physics, but not Greimas' natural world, take
this level to be a privileged version of the world, "the world taken for
granted", in Schiitz's (1967) phrase, from the standpoint of which other
worlds, such as those of the natural sciences, may be invented and ob-
served (cf. Sonesson 1989: 26-29, 30-34, and passim).
4.1. The ecology taken for granted: the Lifeworld
Every particular thing encountered in the Lifeworld is referred to a general
type. According to Schiitz ([1974] 1932, 1967), other people, apart from
family members and close friends, are almost exclusively defined by the
type to which they are ascribed, and we expect them to behave accordingly.
Closely related to the typifications are the regularities that obtain in the
Lifeworld, or, as Russerl's says, "the typical way in which things tend to
behave". This is the kind of principles tentatively set up which are at the
foundation of Peircean abductions. Many of the "laws of ecological phys-
ics", formulated by Gibson (1982: 217ft), and which are defied by magic,
are also such "regularities [that] are implicitly known": that substantial
objects tend to persist, that major surfaces are nearly permanent with re-
spect to layout, but that animate objects change as they grow or move; that
some objects, like the bud and the pupa transform, but that no object is
converted into an object that we would call entirely different, as a frog into
a prince; etc. Some of the presuppositions of these "laws", such as the dis-
tinction between "objects that we would call entirely different", are also at
the basis of the definition of the sign function (cf. Sonesson 1992a, 2000,
2001).
It has been suggested (notably by Smith and Varzi 1999) that the Life-
world, in this sense, is simply the niche, in the sense of (non-Gibsonean)
ecology, in which the animal known as the human being stakes out his life
(cf. Sonesson 2001: 99). The niche, then, in this sense, is the environment
as defined by and for the specific animal inhabiting it. In Husserlean lan-
guage, the niche is subjective-relative - relative to the particular species.
The precursor of the niche, understood in this way, is the notion of Umwelt
introduced by von Uexkiill (1956, 1973), which is one of the key concepts
of the field known as biosemiotics (see Emmeche this volume).
Uexkiill's notion of meaning centres on the environment, the Umwelt,
which is differently determined for each organism. As opposed to an ob-
jectively described ambient world, the Umwelt is characterised for a given
subject, it terms of the features of the world which the subject perceives
(Merkwelt) and the features which it impresses on the world (Wirkwelt),
which together form a functional circle (Funktionskreis). According to a by
now classical example, the tick hangs motionless on a branch until it per-
ceives the smell of butyric acid emitted by the skin glands of a mammal
(Merkzeichen), the effect of which is to send a message to its legs to let go
of the support (Wirkzeichen). When the tick drops onto the body of the
mammal, a new cycle is started, because the tactile cue of hitting the
mammal's hair incites the tick to move around in order to find the skin of
its host. Finally, a third circle is initiated when the heat of the mammal's
skin triggers the boring response, which permits the tick to drink the blood
of its host. Together, these different circles consisting of perceptual and
operational cue bearers make up the interdependent wholes of the subject,
corresponding to the organism, and the Umwelt, which is the world as it is
determined for the subject in question.
Scholars involved with biosemiotics tend to take this model, immensely
enlightening as it is in itself, and simply project onto it the sign conception
suggested by Peirce. The first difficulty with this approach, of course, re-
sides in finding out the real import of the Peircean sign conception. Since
this is in itself an infinite task, any scrutiny of the parallel risks getting
bogged down very early on. If we confront the sign conception defined in
this chapter with the world of the tick, however, it will be easy to see that
the two are entirely distinct. Not only is there no distinction between ex-
pression and content to the tick; there is no separation of sign and reality.
At least in part, this is also an opposition between the Umwelt and the Peir-
cean SIgn.
102 Goran Sonesson
4.2. From Umwelt to Lebenswelt: the thematic field
Pending the invention of biosemiotics, Cassirer (1942: 29ff, 1945: 23ft)
was no doubt the first thinker outside of biology to take von Uexkiill's
ideas seriously. After pointing out that, to human beings, all experience is
mediated (a case of Vermittlung), he observed that this is also true of ani-
mals, as described by von Uexkiil1. But he makes no mention of the fact
that, to von Uexkiill (1956, 1973), the Funktionskreis is a "theory of
meaning" (Bedeutungslehre). In fact, he opposes "animal reactions" to
"human responses". Cassirer may be wrong in not seeing the similarity
between signs and other meanings (though he suggests it in passing using
the term 'Vermittlung'), but he is quite right, I submit, in insisting on the
difference.
Very tentatively, let us suppose that, in the biosemiotic conception, the
features of the world observed by the animal correspond to the sign-vehicle
or expression (Peirce's "representamen"); the object or referent would then
be that which causes theses features to be present to the animal; and the
Peircean interpretant or content would in turn correspond to the pieces of
behaviour which tend to make up the reaction of the animal to the features
in question. There is no point getting lost here in Peircean exegesis: if
anything, we are faced with a "formal sign", as conceived in the Fonseca
tradition. As we are using the terms, we would have some kind of media-
tion (Cassirer's Vermittlung), but not a sign.
As Ziemke and Sharkey (2001: 709) point out, it is hard to find the ob-
ject of the sign, in the ordinary sense of its referent in the "outside world".
Indeed, that which is for us, as observers, three cues to the presence of a
mammal, the smell of butyric acid, the feel of skin, and the warmth of the
blood, do not have to be conceived, in the case of the tick, as one single
entity having an existence of its own (a "substance", in Gibson's terms),
but may more probably constitute three separate episodes producing each
its own sequence of behaviour. In fact, Ziemke and Sharkey go on to quote
an early text by von Uexkiill, in which he says that "in the nervous system
the stimulus itself does not really appear but its place is taken by an en-
tirely different process" (von Uexkiill 1909, quoted here from Ziemke and
Sharkey 2001, my italics). Uexkiill calls this a "sign", but it should be clear
that it does not in any way fulfil the requirements of the sign function. In-
deed, expression and content are not differentiated, already because they
do not appear to the same consciousness. The butyric acid is there to the
tick; the mammal is present only to us.
What is lacking here - to the tick - is real Thirdness: the reaction to the
primary reaction, that is, the reaction which does not respond to a simple
fact (Firstness), but to something which is already a reaction, and thus a
relation (Secondness; see Table 1). Without having to enter into the earlier
discussion of differentiation, we see that, even from a strictly Peircean
point of view, there is no Thirdness for the tick: it does not respond to any
relationship, since it is not aware (even in the most liberal sense of the
term) of any second term (the mammal) to which the first term (the butyric
acid) stands in a relation.
Table 1. The relationship between principles, grounds, and signs, from the point
of view ofPeirce.
Firstness Secondness Thirdness
Principle Iconicity
Ground Iconic ground Indexicality = in-
dexical ground
Sign Iconic sign Indexical sign (in- Symbolicity =
(icon) dex) symbolic ground =
symbolic sign
(symbol)
In fact, things are even more complicated. In a true sign relation, the
mammal is not really the object, in the Peircean sense, for which the bu-
tyric acid is the representamen (the expression). Or, to be more precise, it
is not the dynamical object. At the very most, it is the immediate object. In
Peirce's conception, while the immediate object is that which directly in-
duces the sign process, the dynamical object is something much more com-
prehensive, which includes all those things which may be known about the
same object, although they are not present in the act of inducing. Indeed,
the dynamical object is that which corresponds to the potentially infinite
series of different interpretants resulting from the same original immediate
object. It should be clear that, for the tick and similar beings, there could be
no distinction between direct and dynamical object, because there is no
room for any further development of the chain of interpretants. In this
104 Goran Sonesson
sense, Deacon's (1997: 63) idiosyncratic reading of Peirce, according to
which only signs such as those found in human language (his "symbols")
give rise to chains of interpretants seem to have some justification - in
reality, ifnot in Peircean theory (cf. Sonesson 2003a, 2006).9
To account for the distinction between the "immediate object" and the
"dynamical object", we need the concept of ground.
lo
In one of his well-
known definitions of the sign, a term which he here, as so often, uses to
mean the sign-vehicle, Peirce (1931-58, 2: 228) describes it as something
which "stands for that object not in all respects, but in reference to a sort of
idea, which I have sometimes called the ground ofthe representation" (my
italics; see Table 1). Some commentators have claimed that Peirce is here
talking about some properties of the expression, whereas others favour the
content. In fact, however, the ground must concern the relation between
them. Such an interpretation seems to be born out by Peirce's claim that the
concept of ground is indispensable, "because we cannot comprehend an
agreement of two things, except as an agreement in some respect". (1.551).
In another passage, Peirce himself identifies ground with an abstraction
exemplifying it with the blackness of two black things (1.293). It therefore
seems that the term "ground" must stand for those properties of the two
things entering into the sign function by means of which they get con-
nected, i.e. both some properties of the thing serving as expression and
some properties of the thing serving as content. In case of the weathercock,
for instance, which serves to indicate the direction of the wind, the content-
ground merely consists in this direction, to the exclusion of all other prop-
erties of the wind, and its expression-ground is only those properties which
makes it turn in the direction of the wind, not, for instance, the fact of its
being made of iron and resembling a cock (the latter is a property by means
of which it enters an iconic ground, different from the indexical ground
making it signify the wind). If so, the ground is really a principle of rele-
vance, or, as a Saussurean would say, the "form" connecting expression
and content: that which must necessarily be present in the expression for it
to be related to a particular content rather than another, and vice-versa (cf.
Sonesson 1989: ill. 1, 1995, forthcoming).
The butyric acid, the hairiness, and the warmth form the immediate ob-
jects of the tick, while the mammal as such is the dynamical object. The
9. The problem, however, is that true indices and icons, as experienced as least by
human beings, have as many interpretants as symbols.
10. This was independently noted by S0ren Brier (2001).
difference, however, is that there is no way that the tick, unlike human
beings, may learn more about the "dynamical object" than that which is
given in the immediate one. Meaning here appears as a kind of "filter": it
lets through certain aspects of the "real world" which, in is entirety, is un-
knowable, though less so for human beings than for the tick. The Kantian
inspiration of von Uexkiill is of course unmistakable. Indeed, in the terms
of another thinker with a Kantian inspiration, Biihler (1934), the filter
model is based on "abstractive relevance", the neglect of such physical
properties which are not endowed with meaning, similar to those properties
of the physical sound which vary a lot without the units of meaning (the
phoneme, the word, etc.) being changed, which Saussure and Hjelmslev
characterised as "substance" in opposition to "form".
Returning to modem day biosemiotics, it can be easily shown that what
these authors are involved with has little to do with meaning as sign func-
tion, but very much concerns meaning as relevance, organisation, configu-
ration andlor filtering. In their early joint paper, Emmeche and Hoffmeyer
(1991: 4), criticising the concept of information in information theory,
point out (paraphrasing Bateson), that they are interested in "a difference
that makes a difference to somebody". They go on to say that living beings
"respond to selected differences in their surroundings" (their italics in both
cases). The formulation clearly invokes relevance, and even some kind of
filtering device. Later on in the paper, however, when the Peircean sign
concept is introduced, the DNA-sequence of the gene is said to be the rep-
resentamen, the protein its object, and the interpretant the cellular-
biochemical network. It is, however, difficult to detect any sign function
here. According to Emmeche and Hoffmeyer, the contribution of Peircean
semiotics is to show us that "the field of genetic structures, or a single
gene, cannot be seen in isolation from the larger system interpreted" (1991:
34). This certainly suggests meaning in the sense of a whole or a configu-
ration.
In a later paper, Emmeche (2002) sets out to show that in the living be-
ing function and meaning are the same. This can also be demonstrated,
because Emmeche understands meaning in the sense of function: the rela-
tion of the part to the whole. But even in this article, there are traces of the
filtering concept of meaning: we learn that "the whole operates as a con-
straint".
Saying that cytochorme c means something to the cell is the same as saying
that is has a function. It is not just any molecule. We could well synthesise
small proteins and artificially introduce them into the cell. They would be
106 Goran Sonesson
without importance or they would be dysfunctional or, with certain fortui-
tous strokes of luck, they would actually fulfil some function in the cell.
(Emmeche 2002: 19)
This implies that the meaning of the enzyme "is structural" in the sense
that "the cell's molecules form a system of dissimilarities (like the ele-
ments of language in Saussure)" (Emmeche 2002: 20). This parallel is cor-
rect to the extent that there are relevancies in cells, in particular if these
relevancies result from a system of oppositions, like those of Saussurean
language. From this point of view, everything that is in the cells is also in
language. But the opposite cannot be true. There is, of course, no sign
function as we have defined it.
It is useful to distinguish relevance from filtering, although they have
something in common: picking up a limited set of features from the totality
of the environment. However, relevance, strictly speaking, does not ex-
clude anything: it merely places some portions of the environment in the
background, ready to serve for other purposes. Thus, in the case of lan-
guage, properties that are not relevant for determining the meaning of
words and sentences, still may serve to inform about the dialect, or even
identify the person speaking (Hjelmslev's "connotational language"; cf.
Sonesson 1989). Indeed, relevance lets the difference between "immediate
object" and "dynamical object" subsist, in the vague sense which they re-
tain in the "scholastic" interpretation of Peirce (see above): that which is
directly given, in contrast with that which is potentially given for further
exploration. Thus, Biihler (1934), added to the principle of "abstractive
relevance" that of "apperceptive supplementation", which explains the
projection of properties not physically present in perception to the mean-
ingful experience. In contrast,jiltering simply crosses out that which is not
let through the filtering device.}}
The difference between relevance and filtering no doubt has something
to do with the capacity to be aware of the borders of one's Umwelt. It re-
quires some kind of "metacognition": to the tick, to paraphrase Wittgen-
stein, the limits of its language are the limits of its world, but not so (in
spite of Wittgenstein) to human beings. Or rather, the limits of any par-
ticular Umwelt are not the limits of our Lebenswelt. Schiitz (1967) sug-
11. It can now be seen that Btihler's principles of abstractive relevance and apper-
ceptive supplementation go much further than the sign. They have been found in
the studies of the systems of cooking and clothing realised by Levi-Strauss,
Barthes, and others (as demonstrated by Sonesson 1989.
gested there are really "multiple provinces of meaning", such as dreaming,
religious experience, the art world, the play world of the child, and that
esoteric practise we lmow as science. The peculiarity of the Lifeworld, in
this context, is that is offers access to the other worlds, and is accessible to
all of them. In this sense, the human Lebenswelt is different from the Um-
welt of other animals. Or at least is has the capacity for being different. In
Peircean terms, human beings may reach for the dynamical objects beyond
the immediate ones. They may try to transform Nature into Culture. How-
ever, as Wittgenstein (1971) observed, even if we had a common language
game, we would perhaps not have so much to discuss with a lion. The lion,
presumably, does not try to go beyond his own Umwelt to grasp the prop-
erties of the objects that lie behind it. There is, so to speak, no "dynamical
object" beyond the immediate one to him.
If the Umwelt is an organised network offilters and/or relevancies, as I
suggested above, it seems that maturing in the child consists in breaking
out of one Umwelt and going on to another, broader one, until reaching the
human Lifeworld. Between each Umwelt and the next, which encompasses
it, there is, to borrow a famous expression from Vygotsky (1978) a "zone
of proximal development". In this sense, ontogenesis itself forces us to go
through a series of "finite provinces of meaning", in the sense of Schiitz. A
temporal dimension is thus added.
It might therefore be said that what most perspicuously differentiates
the tick from the human being (without prejudging for the moment on the
question where the exact border is to be placed) is the structure of the field
of consciousness: in Gurwitsch's (1957, 1964, 1985) terms, human con-
sciousness is made up of a theme which is the centre of attention, a the-
maticfield around it consisting of items which are connected to the present
theme by means of intrinsic links permitting it to be transformed into a
theme in its own right, as well as other items present "at the margin" at the
same time, without having any other than temporal relations to the theme
and its field.
12
The tick of course has access neither to the thematic field
nor to the margin. In a way, this is simply another way of saying that the
tick cannot reach beyond the immediate object. But Gurwitsch's analysis
breaks up that of Peirce: it implies that, not only is there no way for the tick
12. Gurwitsch is right, I believe, in suggesting that this thematic structure translates
to language (and no doubt also to other semiotic resources), as most clearly il-
lustrated in the transposition of the functioning of pronouns from the perceptual
world to discourse (cf. Gurwitsch 1985); it is unfortunate, however, that he fails
to attend to the difference in structuring occasioned by the sign function.
108 Goran Sonesson
to "go on from here" (the Husserlean "etcetera principle"), its experience
of the here and now is also very limited. In other words, there is no real
"immediate object" to the tick, not only because it is not opposed to a fu-
ture more extensive dynamical object, but also because even in the here
and lmow, what is immediately experienced does not appear as a thematic
structuring, or perspective, on such a dynamical object.
I have suggested, then, that an important difference between human
beings and (some) other animals consists in the thematic structure of con-
sciousness, or, in other words, the function of attention. Some similar dif-
ference in the structure of attention have been discussed in very different
quarters lately, separating human beings and apes, as well as children of
different ages (cf. Tomasello 1999; Tomasello et al. 2005; Zlatev 2002,
2003, this volume). A discussion of such a progression in the development
of attention presupposes an analysis of our awareness of the other's body
and mind, which would take us out of the limits of this chapter. Something
will be said, nevertheless, on the attention to one's own body in the next
section.
Before that, however, it will be necessary to take stock. I suggested
above that there were really two differences between the way in which
ticks and other lower animals have access to meaning and the human way.
The first of these is the thematic structure: there is no immediate object,
because there is no dynamical object in relation to which it may be seen as
an adumbration. But there is more to it: there is no representamen (expres-
sion), either, because no distinction can be made between such a represen-
tamen and the object, either immediate or dynamic.
Taking into account the Fonseca tradition, we earlier noted that one
kind of mediation (for which I reserve the term sign) consists of a signifier
(expression) which has to be perceived as such in order to usher into the
perception of the corresponding signified (content); and another one
(which following the Brentano-Husserl tradition, can be called intentional-
ity) which may consist in a signifier which is not ordinarily perceived as
such but still somehow serves to mediate the perception of a signified. It
will be remembered that, according to von Uexkiill, "in the nervous system
the stimulus itself does not really appear but its place is taken by an en-
tirely different process" (my italics). If so, this is not even mediation in the
broad sense of the term. As Husserl and Gibson have insisted, we are alter-
natively confronted with different view of the cube or the cat, etc., but
what we really see is all the time the same invariant cube or cat. The tick
smells the same invariant butyric acid, period. In the world of the tick,
there are no signs, as distinct from the world itself. Differentiation has not
even started. In other words, signification has not acquired a "body" of its
own.
5. The body in the Lifeworld
In the previous section I suggested that in identifying the functional cycle
with the Peircean concept of sign, biosemiotics conflates meaning in the
most general sense of organisation and relevance with the more specific
sign function. Inversely, contemporary embodiment theorists such as La-
koff and Johnson reduce the sign to the more general model of the pick-up
of features from the environment. If Lakoff and Johnson engage in one
form of reductionism, biosemiotics seems to accomplish its inversion. The
result, however, is the same: distinctions, which are important, both theo-
retically, and from the point of view of phylogeny and ontogeny, can no
longer be maintained.
The hybrid term "image schema" has many antecedents, at least as to its
latter part. Before the advent of Lakoffs and Johnsons's work, the most
familiar usage was no doubt that of Piaget (1970: 41): as a kind of "ab-
straction from action" taking place at different stages through child devel-
opment. Schiitz ([1974] 1932), however, used the term to refer to all kinds
of fossilized (or, in his words, "sedimented") sequences of action, which
could be used to make sense of new actions within the common sense
world. The idea of a spatial, if not specifically bodily, projection, is im-
portant to the notions of schema in the psychology and sociology of Janet
(1928), Halbwachs (1925, 1950) and Bartlett (1932; cf. Sonesson 1988). In
all these conceptions, schemas are the result of earlier actions. This seems
to accord with the definition by Johnson (1987,2005) of image schemas as
being abstractions from the interaction of organism and environment. If so,
as I will suggest below, image schemas should require some kind of sepa-
ration of the human Umwelt into body and world.
5.1. The body as the axis of the world
It is not surprising that the figure of the body looms large at the horizon of
consciousness. After all, the body is our condition of access to all possible
experience of the world. It is at the origin of one fundamental characteristic
110 Goran Sonesson
of the Hussserlean Lifeworld: that everything in it is given in a subjective-
relative manner. This means that the access in question is not a merely
physical fact: it amounts to the insertion of the mind in the meaningful
whole, which is the common sense world. That is, the body appears (also)
as meaning.
The same observations apply to language. The body (though often pre-
sented as a faceless Ego) is at the centre of language, in the I-here-now.
Many pronouns and adverbs serve as marks of what Benveniste (1966) has
called the "taking into charge" of the language system by the subj ect: these
marks can only be understood with reference to the position in space and
time of the person doing the speaking. Just as the perceptual world, lan-
guage is adumbrated from the position of the subject, whose insertion in
the world can only be accomplished by the body.
Proxemics is concerned with the subject as a body occupying the central
position of space. According to Hall (1966) all cultures define their public,
social, personal and intimate, spheres, but the distances that characterise
each one of these spaces are different in different cultures. When subjects
coming from different cultures meet, their respective spaces tend to clash.
According to one of Hall's classical examples, a person from an Arab cul-
ture, who posits himself within what is from his point of view the personal
sphere, the distance from which it is comfortable to have a chat, inadver-
tently enters the intimate sphere of a Westerner, the sphere in which it is
proper to "fight or make love".
From a proxemic point of view, the subject could thus be seen as a
topological construction: a series of concentric circles demarcating the
public, social, personal and intimate, spaces (in relation to another subject),
within which is found the bodily envelope, all of which are defined by the
fact that they may be penetrated and thus produce an effect of meaning (see
Figure 1 and Sonesson 1993b; 2001). This is to say that these "protective
shells",' as Hall calls them, are more or less permeable. In topological
terms, they possess the property of being open or closed. More exactly, in
merotopological terms, some parts of them have the property of being open
and others that of being closed. They produce a meaning when their bor-
ders are overstepped. This is of course the case with the Arab conversa-
tionalist stepping into the sphere of fighting and love of the Westerner. The
case of the bodily envelope is however more easily illustrated: it possesses
a series of openings (mouth, nostrils, etc.), but it may also be penetrated
elsewhere, with more serious consequences, such as injury.
Public
sphere
Figure 1. The body envelope and its surrounding proxemic spheres (cf. Hall
1966, Spiegel and Machotka 1974). The arrows illustrate entries
through designated openings and through the closed borders, respec-
tively.
The final protective shell of the body, the skin, did not form part of Hall's
original model. It was added later, by Spiegel and Machotka (1974), who
also pointed out the difference between orifices permitting penetration, and
other places where entry can only be forced. In this respect, their contribu-
tion connected to another tradition, the Freudian one, whose model of the
body is reminiscent of some of the image schemas suggested by Lakoff and
Johnson, if we generalise the sexual interpretation to a more general bodily
practice (cf. Sonesson 1989).
5.2. The bucket theory of the body
The function of the image schemas, as conceived by some cognitive lin-
guists (cf. Hampe 2005), seems to be to project our experience of the body
(even if experience per se sometimes seems to be dispensed with) to the
interpretation of the world, thus accounting for pervasive linguistic phe-
nomena such as metaphor, metonymy and polysemy. In the same vein,
Gardner (1970: 360ft), elaborating on an idea of the psychoanalyst Erik
112 Goran Sonesson
Erikson, claims that certain holistic properties are given a particular import
from being first experienced in the relationship between one's own body
and the field of objects outside the body, sometimes in relation to the
keeping of portions of the environment inside the body, and sometimes in
relation to the release of what was once part of the body. Since each bodily
zone has a characteristic mode, and since each mode possesses several
vectorial properties, the modal/vectorial properties can be seen to form a
system: to the oral-sensory zone (mouth and tongue), there corresponds
passive and active incorporation; in relation to the anal-secretory zone,
retention is experienced; in respect to the sphincter, there is expulsion; and,
finally, to the genital zone (penis/vagina), there corresponds intrusion and
inclusion. The result of this Freudian parti pris is not only an insistence on
the primacy of sexual interpretations, but the neglect of some essential
bodily relationships.
Just as, according to Piaget, conceptual schemas are abstracted from
actions through the many stages of intellectual development, the mo-
dal/vectorial properties, as Gardner presents them, may also be conceived
to take their origin in the actions of one's own body, but rather than being
abstracted, they are immediately seen as global characteristics, and while
they may be transposed to other objects than the body, and other relation-
ships than that of the body to the world, as is the case in "symbols", they
somehow remain bound up with the body in all their further applications as
being the deeper source of their sense (cf. Sonesson 1989). However, Arn-
heim (1966: 215ff) is right in arguing against Freudian pansexualism that a
piece of pottery and a womb have the common class meaning of being
containers, rather than the first signifying the second, and that the pre-
dominance of the sexual interpretation is due to cultural factors. It seems
more probable that bodily experience of a more general kind, including that
of enclosing an apple in one's hand and sticking the hand into a hollow in
the ground, is the primary basis of modal/vectorial properties. When Arn-
heim suggests that, going up the tree of Porphyry, both the womb and the
piece of pottery will be found to be containers, he is certainly not making
the kind of analysis that Porphyry or his followers (as for instance Eco
1984: 46ff) would accept, since the womb does not meet the necessary and
sufficient conditions for being a container ordinarily conceived; nor is it
referable to the container prototype in a strict sense. Rather it is a member
of the extended class of containers acceptable in "symbolism". On this
interpretation, of course, the womb would be a deviant piece of pottery,
rather than the reverse.
In making this kind of argument (cf. Sonesson 1989: 1.4.5), I found my-
self in a terrain very close to Cognitive Linguistics without knowing it. The
first step was to identify the modal-vectorial properties as being topologi-
cal. In the Piagetean conception, the geometry of the child's first space is
topological, that is, it contains the kind of relations that would be preserved
in a figure drawn on a piece of rubber (cf. Vuipillot 1967: 104ff). Proper-
ties of this kind are neighbourhoodness or proximity, separation, succes-
sion, inclusion or interiority/exteriority, and continuity.
If we now merely introduce a distinction between two instances, the ego
and the world, or the other, it will be possible to derive all of Gardner's
"modes" from the topological property of inclusion, to which another
topological property, that of succession, is applied. Clearly, intrusion and
inclusion are opposites, as are incorporation and expulsion, but rentention
seems to call for some corresponding term: this must be resistance (postu-
lated in Sonesson 1989, in complete ignorance of force-dynamical theories,
for which cf. Talmy 1988; Gardenfors this volume), of which there are two
variants, the resistance of the world to us, and of us to the world. Actually,
there is nothing very new about resistance as a fundamental concept: it has
been the basis of the definition of reality in philosophical epistemology,
from Berkeley over the ideologues Destutt de Tracy and Maine de Biran to
Sartre. Indeed, "this sense of being acted upon, which is our sense of the
reality of things" is the definition of Secondness in the conception of
Peirce (1998: 4):
A door is slightly ajar. You try to open it. Something prevents. You put your
shoulder against it, and experience a sense of effort and a sense of resis-
tance. These are not two forms of consciousness; they are two aspects of one
two-sided consciousness. It is inconceivable that there should be any effort
without resistance, or any without a contrary effort. This double-sided con-
sciousness is Secondness. (Peirce 1998: 268)
Since Peirce goes on immediately to note that "all consciousness, all being
awake, consists in a sense of reactions between ego and non-ego", it is
curious that he should not recognise the difference between the case in
which the ego takes the active part, and the case in which the non-ego has
the initiative and the ego is reduced to resistance. However, it is clear that
from a Peircean point of view, the Freudian interpretations of incorpora-
tion, retention, expulsion, and so on, are only special cases of more general
bodily processes. For they no doubt continue to be bodily based: it is the
body of the ego which first enters into a clash with the non-ego.
114 Goran Sonesson
These operations obviously serve more humble purposes than proving
the existence of the outside world. Just as such image schemas as PATH and
CONTAINER, they are generalizations of "a recurrent pattern, shape, and
regularity in, or of, [ ...} ongoing ordering activities" as actions, percep-
tions, and conceptions (Johnson 1987: 29, original italics). In spite of what
is suggested by this definition, Lakoff and Johnson often do not seem to
have any use for the body as an experienced meaning, as opposed to the
way it appears to the biological sciences.
What is at issue is the exact role played by the body. Whether the ac-
tions which sediment to form "images schemas" are accomplished in rela-
tion to the inside of the body, or to something outside of it, a minimum
requirement for their schematisation must be the existence of the bodily
envelope as a relevant level of analysis. It is difficult to understand how
such schemas may even come into existence if human beings are as tightly
embedded in their Umwelt as the tick, entirely merged with their environ-
ment. Yet, in introducing the theory of image schemas, this is precisely the
view propounded by Johnson (2005).
In defining image schemas, Johnson (1987, 2005, Johnson and Rohrer
this volume) uses expressions that are clearly reminiscent of Piaget, al-
though the latter is never quoted. However, in order to acquire sensory-
motor schemas, let alone "symbols" (Piaget 1945) or "mimetic schemas"
(Zlatev 2005), some sense of a distinction between the acting subject and
the world resisting him or her is clearly required. In order to arrive at
schemas of a higher order (including the sign function), the subject must
somehow take cognizance of the more basic schemas themselves. Johnson
and Rohrer (this volume) who do not believe in "the supposedly unique
ability of humans to engage in symbolic representation", consider it only
an illusion resulting from our seeing the interaction of organism and envi-
ronment "from our standpoint as observers and theorists". But this leaves
unexplained the fact that we are ever able to reach such a standpoint. No
doubt there must be a number of progressive stages leading from sensory-
motor experience to our capacity for engaging in theory.
6. The body as portable memory
Is has been suggested by Donald (1991, 2001) that there are several dis-
continuities in hominid evolution, all involving the acquirement of a dis-
tinct kind of memory, considered as a strategy for representing knowledge.
Although Donald's model concerns phylogeny, parallels in ontogeny are
readily suggested (cf. Zlatev 2002, 2003, 2005, this volume; Ikegami and
Zlatev this volume). Without necessarily taking Donald's model at face
value, I am going to make use of it here since it permits a productive inte-
gration with semiotic theory. Indeed, the Tartu school characterizes signs
as memory devices and defines culture as collective memory. According to
Lotman et al. (1975), material objects and information are similar to each
other, and at the same time differ from other phenomena, in two ways: they
can be accumulated, whereas for example, sleep and breathing cannot be
accumulated, and they are not absorbed completely into the organism, un-
like food, but remain separate objects after the reception. The interesting
thing, however, not discussed by the Tartu school, is how material artefacts
and signs come to work together.
According to Donald's conception, many mammals are already capable
of episodic memory, which amounts to the representation of events in terms
of their time and place of occurrence. The first transition, which antedates
language and remains intact at its loss (and which Donald identifies with
Homo erectus) brings about mimetic memory, which is required for such
abilities as the construction of tools, miming, imitation, coordinated hunt-
ing, a more complex social structure and simple rituals. This stage thus in
part seems to correspond to what we have called the attainment of the sign
function (though Donald only notes this obliquely, in talking about the use
of intentional systems of communication and the distinction of the refer-
ent). Yet, it should be noted already at this point that while all abilities
subsumed in this stage seem to depend on iconic relations (perceptions of
similarity), only some of them are signs because they do not involve any
asymmetric relation between an expression and the content for which it
stands.
Only the second transition, occurring with Homo sapiens, brings about
language with its semantic memory, that is, a repertory of units that can be
combined. This kind of memory permits the creation of narratives, that is,
mythologies, and thus a completely new way of representing reality. Inter-
estingly, however, Donald does not think semiotic development stops
there, although further stages are no longer based on any biological
changes. However, the third transition obviously would not have been pos-
sible without the attainment of the three earlier stages. What Donald calls
theoretical culture presupposes the existence of external memory, that is,
devices permitting the conservation and communication of knowledge in-
dependently of face-to-face interaction between human beings. The first
116 Goran Sonesson
apparition of theoretical culture coincides with the invention of drawing.
For the first time, knowledge may be stored externally to the organism. The
bias having been shifted to the visual modality, language is next transferred
to writing. It is this possibility of conserving information externally to the
organism that later gives rise to science (cf. Figure 2).
- - - - \ . . . - - - - - - - - - - " " ' . ~
'V'"
o Sign function J5
+:i Toolmakjng .2 ~ Pictures
~ -Iolitation ~ Language 0 Writing
~ - Gesture:E ~ Theory
I-
iconicity
indexicality
symbolicity
iconicity
indexicality
Figure 2. Donald's model of evolution related to the notion of sign function.
There are two remarkable features in Donald's analysis. The stage preced-
ing the attainment of the language capacity requires memory to be located
in the subject's own body. But, clearly, it can only function as memory to
the extent that it is somehow separable from the body as such. The move-
ment of the other must be seen as distinct from the body of the other in its
specificity, so that it can be repeated by the self. This supposes a distinc-
tion between token and type (that is, relevance) preceding that of the sign
function.
The stage following upon language supposes the sign to acquire a
"body" of its own, that is, the ability to persist independently of human
beings. Language only seems to require the presence of at least two human
beings to exist: they somehow maintain it between themselves. But it is not
enough for two persons to know about a picture for it to exist: there must
be some kind of organism-independent artefact on which it is inscribed.
The picture must be divorced from the bodies (and minds) of those making
use of it. 13 Writing is of course, by definition, the transposition of language
to independent artefacts. The case of "theory" may be less obvious: why
should not two persons be able to entertain a theory between them? As
Husserl (1962a) noted well before Donald, complex sign systems, such as
mathematics and logic, only seem to function as such when given an exis-
tence independent of human organisms. In the case of pictures, Ivins
(1953) has observed that it is their reproducibility (as in Floras, for in-
stance) that makes them into scientific instruments. In their capacity of
being permanent records, pictures are not, as art historians are wont to say,
unavoidably unique, but, on the contrary, are destined for reproduction.
Indeed, they permit repeated acts of perception, as do no earlier memory
records.
Students of prehistoric pictures such as White (2000) often suggest that
creators of such works must have been capable of language. In fact, not
much can be concluded on the basis of the depictions having come down to
us: even though pictures, by their nature, must have been made on material
which conserves the markings on the surface, they might at first have been
created on surfaces (such as sand) which only preserve them for a short
time. And it is not easy to establish any clear-cut relation between linguis-
tic capacity and the sophistication of the depictions (whatever that is).
There are, however, more fundamental reasons for supposing pictures to be
later in phylogeny than language: they suppose a record which is independ-
ent of the human body; and they require us to see a similarity within an
over-arching dissimilarity.
Posner (1989) distinguishes two types of artefacts: the transitory ones
(as the sound of a woman's high heeled shoes against the pavement) and
enduring ones (as the prints that the woman's shoes may leave in clay, in
particular if the latter is later dried). The transitory a r t e f a ~ t s , in this sense,
also have a material aspect, just as the lasting ones; they only have the
particularity of developing in time, which is why they cannot be accumu-
lated without first being converted. Strictly speaking, the sound sequence
produced by high heels against the pavement, and other transitory artefacts,
can of course be accumulated (as opposed to being converted into an en-
during artefact, which is the case of the sound tape), in the form of the
13. This is of course what is known, mainly in Marxist literature, as the process of
reification. As shown by Cassirer (1942: 113ft), this process, far from being
only a "tragedy of culture", is the prerequisite for (huma)n culture.
118 Goran Sonesson
(typical) leg movements producing this sound, that is, as a mimetic record,
accumulated in the body, but still distinct from it, since the movements can
be learnt and imitated, and even intentionally produced as signs of (tradi-
tional) femininity. Posner's example of an enduring artefact is interesting
in another way: the cast of prints left by the woman's high heels is of
course an organism-independent record, just as the marks of a Roman sol-
dier's sandals found in prehistoric caves, and the hand-prints on cave walls.
Another case in point may very well be the so-called Berekhat Ram figure
(250000-280000 BP), which, if it is not the likeness of a woman, as has
been claimed with very little justification, could be the result of abrasion
produced by regular movements indicating the intervention of a human
agent (that is, "anthropogenic" movements). This suggests that the first
organism-independent records may be indexical, rather than iconic, in
character. However, even if objects like these were independent objects
already in prehistory, there is nothing to prove they were perceived as
signs, that is, as expressions differentiated form contents, before pictures
were so perceived.
Episodic memory, in Donald's sense (which should not be confused
with earlier uses of the term) is most clearly "disembodied" memory: it
only goes as far as the attention span does. It may refer to a bodily act, such
as going in or out of a container-type object, but it is unable to generalise
this movement beyond a particular moment and place, and thus it does not
give rise to any kind of independent embodiment (cf. Table 2). Mimetic
memory still accumulates in the subject's own body, but it only becomes
such, to the extent that what is recorded in the body also exists elsewhere,
in at least one other body (or perhaps, in same cases, in other moving arte-
facts), which supposes generalisation or, more exactly, typification: the
creation of a type referring to different tokens instantiated in different
bodies. As tokens, then, they are in the body; as types they are shared by
different uses. Typification, in this sense, does not require the sign func-
tion, but is no doubt a prerequisite for it: indeed, it is during this stage,
most likely, that the sign function emerges.
Mythic memory (which I would prefer to call linguistic memory or per-
haps, as Donald sometimes does: semantic memory) is different again: it
has a separate existence, but, like some kind of real-world ectoplasm, is
requires the collaborative effort of a least two consciousnesses (which no
doubt have to be embodied) for this existence to be sustained. Transitory
artefacts, as spoken language or (as Posner would have it) the sound of
high-heeled shoes on the pavement, acquire a body only to the extent that a
sender and a receiver agree roughly on what they are. Only theoretic mem-
ory has a distinct "body" of its own: it subsists independently of the pres-
ence of any embodied consciousness, because it is itself embodied. Of
course, without anybody able to perceive it, organism-independent records
are not of any use. Without any human beings present, they are really
worse off than the famous acorn falling from a tree without anybody
around to hear its sound.
Table 2. Donald's memory types analysed in relation to the nature of
accumulation (in the sense of Lotman et al. 1975)
1.1. Type of
memory
Episodic
Mimetic
Mythic
Theoretic
7. Conclusions
1.2. Type of
accumulation
Attention span
(event in ti-
me/space)
Action sequence
co-owned by Ego
and Alter
Transient arte-
fact co-produced
by Ego and Alter
Enduring artefact
co-externalised
by Ego and Alter
1.3. Type of
embodiment
Own body
In the interac-
tion between
Ego and Alter
External in re-
lation to Ego
and Alter
In this chapter I have tried to relate different notions of embodiment stem-
ming from the phenomenological tradition, and contemporary conceptions
of embodiment, taking their origin in Cognitive Linguistics, biology, and
cognitive science. My aim has been to show that the various forms of em-
bodiment in both traditions are very different, but that once they are prop-
erly analysed, they may be connected with each other, and placed on
something like an evolutionary scale similar to the one proposed by Donald
120 Goran Sonesson
(1991). Indeed, the whole point of making these distinctions has been to
show the complexity of the "ladder" from (non-human) animals to human
beings, a ladder that requires a series of very different steps, only one of
which is the capacity for language.
Another goal of the essay has been to suggest the way in which the sign
function, the general faculty for conceiving signs, emerges out of one kind
of embodiment and constitutes a requirement for attaining another one. In
the process, I have suggested that we must distinguish meaning in a very
general sense, akin to organisation and/or selection, from the sign function,
which requires the peculiar property of differentiation. I have claimed that
this distinction is not observed in either biosemiotics nor in some parts of
Cognitive Linguistics. Moreover, I have argued that what is called "image
schemas" by cognitive linguists is basically a kind of bodily meaning, re-
sulting from the position of the human body at the centre of the common
sense world, known in phenomenology as the Lifeworld.
Acknowledgements
The author wishes to thank Jordan Zlatev and Tom Ziemke for their de-
tailed comments on an earlier draft of this chapter.
References
Arnheim, Rudolf
1966 Towards a Psychology ofArt. London: Faber and Faber.
Bartlett, Fredrick C.
1932 Remembering. A Study in Experimental and Social Psychology.
Cambridge: Cambridge University Press, Reprint 1967.
Benveniste, Emile
1966 Problemes de linguistique generale. Paris: Gallimard.
Brentano, Franz
1885 Psychologie von empirischen Standpunkt. Leipzig: Mener. Reprint
1924.
Brier, Seren
2001 Cybersemiotics and Umweltslehre. On the cybersemiotic integration
of Umweltlehre, enthology, autopoiesis theory, second order cyber-
netics and Peircean biosemiotics. Semiotica. Special issue on Jakob
von Uexkall, 134(1/4): 779-814.
2001
1994
Deely, John
1982
Biihler, Karl
1934 Sprachtheorie. Fischer, Frankfurt/M: Ullstein. Reprint 1978.
Cassirer, Emst
1942 Zur Logik der Kulturwissenschaften. Goteborg: Elanders.
1945 An Essay on Man. New Haven: Yale University Press.
1957 The Philosophy of Symbolic Forms. Ill. The Phenomenology of
Knowledge. Translated by Ralph Mannheim. New Haven: Yale Uni-
versity Press.
Costall, Alan
this vol. Bringing the body back to life: James Gibson's ecology of embodied
agency.
Deacon, Terry
1997 The Symbolic Species: The Co-Evolution ofLanguage and the Brain.
New York: Norton.
2003 Universal grammar and semiotic constraints. In: Christiansen, Mor-
ton H. and Simon Kirby (eds.), Language evolution, 111-139. Ox-
ford: Oxford University Press.
Introducing Semiotic. Its History and Doctrine. Bloomington: Indi-
ana University Press.
New Beginnings. Early Modern Philosophy and Postmodern
Thought. Toronto: University of Toronto Press.
Dennett, Daniel
1991 Consciousness Explained. Boston: Little, Brown, and Co.
Donald, Merlin
1991 Origins of the Modern Mind. Three Stages in the Evolution of Cul-
ture and Cognition, Cambridge, Mass.: Harvard University Press.
A Mind So Rare. The Evolution of Human Consciousness. New
York: Norton.
Eco, Umberto
1982 Semiotics and the Philosophy of Language. Bloomington: Indiana
University Press.
Edelmann, Gerald
1992 Bright Air, Brilliant Fire. On the Matter of the Mind. London: Pen-
guinBooks.
Edelmann, Gerald and Guilio Tonini
2000 A Universe ofConsciousness. New York: Basic Books.
Emmeche, Claus
2002 The chicken and the Orphean egg. On the function of meaning and
the meaning of function. Sign System Studies 30 (1): 15-32.
this vol. On the biosemiotics of embodiment and our human cyborg nature.
122 Goran Sonesson
Emmeche, Claus and Jesper Hoffmeyer
1991 From language to nature - the semiotic metaphor in biology. Semi-
otica 84 (1/2) 1--42.
Gallagher, Shaun
this vol. Phenomenological and experimental contributions to understanding
embodied experience.
Gardenfors, Peter
this vol. Representing actions and functional properties in embodied concep-
tual spaces.
Gardner, Howard
1970 Frommode to symbol. The British Journal ofAesthetics 10 (3): 357-
375.
Gibson, James
1978 The ecological approach to visual perception of pictures. Leonardo,
4 (2): 227-235.
1980 A prefatory essay on the perception of surfaces versus the perception
of markings on a surface. In: Margaret Hagen (ed.), The Perception
of Pictures. Volume I: Alberti's Window, xi-xvii. New York: Aca-
demic Press.
1982 Reasons for Realism. Selected Essays of James J Gibson. Edward
Reed and Rebecca Jones (eds.). Hillsdale, New Jersey: Lawrence
Erlbaum.
Greimas, Algirdas J.
1970 Du sens. Paris: Seuil.
Gurwitsch, Aron
1957 Theorie du champ de la conscience. Bruges: Desclee de Brouver.
1964 The Field ofConsciousness. Pittsburgh: Duquesne University Press.
1966 Studies in Phenomenology and Psychology. Evanston: Northwestern
University Press.
1974 Phenomenology and the Theory of Science. Evanston: Northwestern
University Press.
1979 Human Encounters in the Social World. Pittsburgh: Duquesne Uni-
versity Press.
1985 Marginal Consciousness. Athens, Ohio: Ohio University Press.
Halbwachs, Maurice
1925 Les cadres sociaux de la memoire. Paris: PUF. Reprint 1952.
1950 La memoire collective. Paris: PUF. Reprint 1968.
Hall, Edward
1966 The Hidden Dimension. London: Bodley Head Ltd.
Hampe, Beate (ed.)
2005 Perception to Meaning: Image Schemas in Cognitive Linguistics,
Berlin: Mouton de Gruyter.
Hoffman, Donald
1998 Visual Intelligence. How We Create What We See. New York: Nor-
ton.
Hockett, Charles
1977 The Viewfrom Language. Athens: University of Georgia Press.
Husserl, Edmund
1913 Logische Untersuchungen. Tiibingen: Niemeyer. Reprint 1968.
1939 Erfahrung und Urteil. Prag: Academia Verlagsbuchhandlung.
1962 a Die Krisis der europiiischen Wissenschaften und die transzendentale
Phiinomenologie. Husserliana VI. The Hague: Nijhoff.
1962 b Phiinomenologische Psychologie. Husserliana IX. The Hague: Ni-
jhoff.
1973 a Cartesianische Meditationen. Husserliana I. The Hague: Nijhoff.
1973 b Ding und Raum. Husserliana XVI. The Hague: Nijhoff.
1976 Ideen zu einer reinen Phiinomenologie und phiinomenologische
Psych0 logie. Husserliana Ill. The Hague: Nijhoff.
1980 Phantasie, Bildbewusstsein, Erinnerung. Husserliana XXIII. The
Hague: Nijhoff.
Ikegami, Takashi and Jordan Zlatev
this vo!. Frompre-representional cognition to language.
Innis, Harold
1950 Empire and Communication. Toronto: University of Toronto Press.
Reprint 1972.
Ivins, William M.
1953 Prints and Visual Communication. Cambridge, Mass.: Harvard Uni-
versity Press.
Jakobson, Roman
1975 Coup d'adl sur le developpment de la semiotique. Bloomington:
Indiana University Press.
James, William
1890 The Principles ofPsychology. New York: Dover Publications.
Janet, Pierre
1928 L 'evolution de la memoire et la notion du temps. Paris: Chahine.
Johnson, Mark
1987 The Body in the Mind. Chicago and London: University of Chicago
Press.
2005 The philosophical significance of image schemas. In: Beate Hampe
(ed.), From Perception to Meaning: Image Schemas in Cognitive
Linguistics., 15-33. Berlin: Mouton de Gruyter.
Johnson, Mark and Tim Rohrer
this vo!. We are life creatures: Embodiment, American Pragmatism, and the
cognitive organism.
1967
1998
Piaget, Jean
1945
124 Goran Sonesson
Kiing, Guido
1973 Husserl on pictures and intentional objects. Review of metaphysics.
(June) XXXVI (4): 670-680.
1999 Philosophy in the Flesh. New York: Basic Books.
Lotman, Jurij M., Boris A. Uspenskij, Vjaceslav. V. Ivanov, V. N. Toporov and A.
M. Pjatigorski
1975 Thesis on the Semiotic Study of Culture. Lisse: The Peter de Ridder
Press.
1942 La structure du comportement. Paris: PUF.
1945 La phenomenologie de la perception. Paris: PUF.
Parmentier, Richard J.
1985 Signs's place in medias res: Peirce's concept of semiotic mediation.
In: Elizabeth Mertz and Richard J. Parmentier (eds.) Semiotic Me-
diation. Sociocultural and Psychological Perspectives, 23-48. Or-
lando: Academic Press.
La formation du symbole chez l'enfant. Neuchatel: Delachaux and
Niestle. Reprint 1967.
1967 La psychologie de I 'intelligence. Paris: Armand Colin.
1970 Epistemologie des sciences de l'homme. Paris: Gallimard.
Peirce, Charles Sanders
1931-58 Collected Papers I-VIII. C. Hartshome, P. Weiss and A. Burks
(eds.). Cambridge, MA: Harvard University Press.
The Essential Peirce, Volume II. Ed. by the Peirce Edition Project.
Bloomington and Indianapolis: Indiana University Press.
Posner, Roland
1989 What is culture? Toward a semiotic explication of anthropological
concepts. In: Waiter Koch (ed.), The Nature of Culture, 240-295.
Bochum. Brochmeyer.
Prieto, Luis J.
1975 a Pertinence et pratique. Paris: Minuit.
1975 b Essai de linguistique et semiologie generales. Geneve: Droz.
Reed, Edward S.
1996 The Necessity ofExperience. New Haven: Yale University Press.
Schiitz, Alfred
1932 Der sinnhafte Aufbau der sozialen Welt. Vienna: Springer. Reprint
1974.
Collected Papers I: The Problem of Social Reality. Nijhoff: The
Hague.
Smith, Barry and Achille Varzi
1999 The niche. Nous 33 (2): 198-222.
Sonesson, Goran
1988 Methods and models in pictorial semiotics. Semiotics Project: Lund
University.
1989 PictorialConcepts. Inquiries into the Semiotic Heritage and its Rele-
vance for the Analysis of the Visual World. Lund: Aris/Lund
University Press.
1992 a Bildbetydelser. Inledning till bildsemiotiken som vetenskap. Lund:
Studentlitteratur.
1992 b The semiotic function and the genesis of pictorial meaning. In: Eero
Tarasti (ed.), Center/Periphery in representations and institutions.
Proceedings from the Conference of The International Semiotics In-
stitute, Imatra, Finland, July 16-21, 1990, 211-156. Imatra: Acta
Semiotica Fennica.
1993 a Pictorial semiotics, Gestalt psychology, and the ecology of percep-
tion. Semiotica 99 (3-4): 319-399.
1993 b The multiple bodies of man. Project for a semiotics of the body.
Degres 74: d-d42.
1994 Prolegomena to the semiotic analysis of prehistoric visual displays.
Semiotica 100 (3): 267-332.
1995 On pictorality. The impact of the perceptual model in the develop-
ment of visual semiotics. In: Thomas Sebeok and Jean Umiker-Se-
beok (eds), .The Semiotic Web 1992/93: Advances in Visual Semiot-
ics, 67-108. Berlin and New York: Mouton de Gruyter.
1996 An essay concerning images. From rhetoric to semiotics by way of
ecological physics. Semiotica 109 (1/2): 41-140.
1998 a /entries/. In: Paul Bouissac (ed.) Encyclopaedia of Semiotics. New
York and London: Oxford University Press.
1998 b That there are many kinds of pictorial signs. VISIO 3 (1): 33-54.
2000 Iconicity in the ecology of semiosis. In: Troels Deng Johansson,
Martin Skov and Berit Brogaard (eds.), Iconicity - A Fundamental
Problem in Semiotic, 59-80. Aarhus: NSU Press.
2001 From Semiosis to Ecology. On the theory of iconicity and its conse-
quences for the ontology of the Lifeworld. In: Andrew W. Quinn
(ed.), Cultural Cognition and Space Cognition/Cognition culturelle
et cognition spatiale. VISIO 6 (2/3): 85-110.
2003 a The Symbolic Species revisited. Considerations on the semiotic turn
in cognitive science and biology, SGBWP3. Lund: Lund University.
2003 b Why the mirror is a sign - and why the television picture is no mir-
ror. Two episodes in the critique of the iconicity critique. In S. Euro-
pean Journalfor Semiotic Studies 15(2-4): 217-232.
126 Goran Sonesson
2006 The meaning of meaning in biology and cognitive science. A semi-
otic reconstruction. In Trudy po znakonym sistemam - Sign Systems
Studies 34.1, 135-214.
forthc. From iconicity to pictorality. A view from ecological semiotics. To
appear in VISIO: thematic issue: Iconicity revisitedlIconicite revisite,
Sonesson, Goran (ed.).
Spiegel, John and Pavel Machotka
1974 Messages ofthe Body. New York: The Free Press.
Talmy, Leonard
1988 Force dynamics in language and cognition. Cognitive Science 12:
49-100.
Tomasello, Michael
1999 The Cultural Origins ofHuman Cognition. Cambridge, MA: Harvard
University Press.
Tomasello, Michael, Malinda Carpenter Joseph Call, Tanya Behne, and Henrike
Moll
2005 Understanding and sharing intentions: The origins of cultural cogni-
tion. Behavioral and Brain Sciences 28: 675-735.
Varela, Francisco, Evan Thompson and Eleanor Rosch
1991 The Embodied Mind. Cognitive Science and Human Experience.
Cambridge, Mass.: MIT Press.
Von Uexkiill, Jakob
1909 Umwelt und Innenwelt der Tiere. Berlin: Springer.
1956 Streifzuge durch die Umwelten von Tieren und Menschen
Bedeutungslehre. Hamburg: RowoWt.
1928 Theoretische Biologie. Frankfurt/M.: Suhrkamp. Reprint 1973.
Vuipillot, Eliane
1967 La perception de l'espace. In: Paul Fraisse and Jean Piaget (eds.),
Traite de psychologie experimentale VI: La perception, 101-186.
Paris: PUF.
Vygotsky, Lev
1978 Mind in Society. Cambridge. Mass.: Harvard University Press.
Wittgenstein, Ludwig
1971 Philosophische Untersuchungen. Suhrkamp: Frankfrut/M.
White, Randall
2000 Prehistoric Art. New York: Harry N. Abrahams, Inc.
Ziemke, Tom and Noel E. Sharkey
2001 A stroll through the worlds of robots and animals: Applying Jakob
von Uexkiill's theory of meaning to adaptive robots and artificial life.
Semiotica 134 (1-4): 701-746.
Zlatev, Jordan
2002 Mimesis: The "missing link" between signals and symbols in phylog-
eny and ontogeny. In: Anneli Pajunen (ed.), Mimesis, Sign and the
Evolution ofLanguage, 93-122. Turku: University of Turku Press.
2005 What's in a schema? Bodily mimesis and the grounding of language.
In: Beate Hampe (ed.), From Perception to Meaning: Image Sche-
mas in Cognitive Linguistics, 313-342. Berlin: Mouton de Gruyter.
this vol. Embodiment, language, and mimesis.
Embodiment and social interaction: A cognitive
science perspective
Jessica Lindblom and Tom Ziemke
We respond to gestures with an extreme alertness and,
one might almost say, in accordance with an elaborate
and secret code that is written nowhere, known to no
one and understood by all. Sapir 1928.
Abstract
Much recent work in cognitive science has focused on the embodied, situated and
distributed nature of mind. This is a radical shift away from the computer metaphor
for mind that characterized traditional cognitive science. However, although much
attention is paid nowadays to the biological and bodily basis of cognitive processes
as well as their sociocultural embedding, the role of embodiment in social interac-
tion is still relatively little understood. In this chapter we trace the role of biological
and sociocultural factors in explanations of cognitive processes from Darwin to
modem cognitive science. We discuss different degrees of commitment to em-
bodiment and point out further steps and conceptual clarifications that will be re-
quired in the further development of a science of embodied cognition and social
interaction.
Keywords: embodied cognition, history of cognitive science, interactive technol-
ogy, radical embodiment, social interaction.
1. Introduction
Embodiment has become a much discussed concept in cognitive science in
recent years, and many take it, together with situatedness, to be the defin-
ing feature of a new approach to the study of mind commonly referred to as
embodied cognitive science or embodied cognition (e.g., Clark 1997, 1999;
Varela, Thompson and Rosch 1991). Embodied cognition offers a radical
shift in explanations of the human mind - a Copernican revolution in cog-
130 Jessica Lindblom and Tom Ziemke
nitive science, you might say - which emphasizes the way cognition is
shaped by the body and its sensorimotor interaction with the world. This is
a reaction against the computer metaphor of traditional cognitive science,
which views cognition as symbol manipulation, centralized and taking
place inside the skull while the body only serves as some kind of input and
output device, i.e., a physical interface between internal program (cognitive
processes) and external world.! That means, embodiment only played a
marginal, if any, role in traditional cognitive science which instead focused
on mental representations and computational processes. Gardner (1987: 6),
for example, stressed the major assumption for the then "new science of
mind" as follows:
First of all, there is the belief that, in talking about human cognitive activi-
ties, it is necessary to speak about mental representations and to posit a level
of analysis wholly separate from the biological or neurological, on the one
hand, and the sociological or cultural, on the other.
Consequently, the computer and its syntax-driven way of manipulating
symbols became the general model of how the human mind functions. Ac-
cordingly, the traditional view of social interaction has been that agents
relate to each other much in the same way they relate to other parts of the
external world, i.e., by having more or less explicit internal representations
of each other. That means, the "secret code" that Sapir referred to in the
above introductory quote was quite literally taken to be an actual sym-
bolic/representational code. For example, one agent might encode her
mental states into some form suitable for transmission via some communi-
cation channel, such as language or gestures, and another would receive
and decode the transmitted message, and thus come to an understanding of
the first agent's mental states and actions. That means, traditional cogni-
tive-scientific explanations of both individual and social cognition very
much focused on internal, individual processes and representations,
whereas the interaction with the environment - physical, social or cultural
- was treated as fairly peripheral. Gardner (1987), for example, argued that
such "murky concepts" as context, history and culture only could cause
problems for efforts to find the "essence" of individual cognition, and
therefore should not be addressed and integrated until later on.
Today's proponents of embodied, situated and/or distributed cognition,
on the other hand, paint a much more complex picture of the mind, its reIa-
1. In cognitive science the traditional view is referred to as "cognitivism" which,
however, should not be confused with the cognitivist approach in linguistics.
Embodiment and social interaction: A cognitive science perspective 131
tion to the body, and its interaction with the physical and sociocultural
environment. Hutchins (1995), for example, argued that there are unnoticed
costs involved when we disregard culture, context and history, which he
considered important factors in the development of individual intelligence.
The body's role in cognitive processes has also received much attention in
recent discussions in cognitive science (cf. e.g., Clark 1997, 1999; Varela,
Thompson and Rosch 1991; Ziemke 2002) and a large variety of notions of
embodiment and embodied cognition have been developed (cf. e.g., Chris-
ley and Ziemke 2003; Clark 1999; Rohrer this volume; Svensson, Lind-
blom and Ziemke this volume; Wilson 2002; Ziemke 2001a, 2003; Ziemke
and Frank this volume).
From a psychological perspective the interaction between individual
agent and environment has been addressed, while from the perspectives of
philosophy and linguistics the role of body and world in the constitution of
meaning has received much attention. Moreover, from the AI perspective it
has been discussed what kind of body an artificial system might need to
count as an embodied cognizer. However, despite the fact that embodied
cognitive science pays much attention to both the sociocultural embedding
of cognitive processes and their bodily basis, there still is surprisingly little
work on the role of embodiment in social interaction, as recently recog-
nized by several researchers, who have argued that current theories of em-
bodiment need to be developed further to include the social dimension
(e.g., Lindblom and Ziemke 2005; Riegler 2002; Semin and Smith 2002;
Sinha and Jensen 2000; Ziemke 2002, 2003).
Traditionally, the study of the body in social interaction in cognitive
science has been limited to non-verbal communication, commonly viewed
as a trivial form of "body language" or an "appendage" to the real intel-
lectuallanguage and mind. However, it has been estimated that nearly two
thirds of the meaning in a social situation is "received" from these non-
verbal signs (Burgoon, Buller and Woodall 1996). Famell (1999) pointed
out that the widespread neglect of the body in the social sciences is a con-
sequence of the Platonic-Cartesian heritage that has resulted in the view of
the mind as an internal locus of rationality, thought, language and knowl-
edge (cf., Damasio 1995, 1999; Johnson and Rohrer, this volume). The
body, on the other hand, is still widely regarded as the sensate locus of
irrationality and feeling, a view that has been supported by the Christian
disregard
2
of the flesh as the locus of sinful desire and irrationality. Theo-
ries of cultural and social cognition therefore mainly tend to overlook the
bodily aspects of social interaction (cf. Tomasello 1999). Varela (1992),
for example, argued that "[s]ocial scientists are body-dead because they are
conceptually brain-dead to signifying acts within the semiotics of body
movements". Trevarthen (1977; cf., Hendriks-Jansen 1996) pointed out
that one reason for the neglect of the body in psychological research was
that the bodily movement patterns of humans were difficult to observe with
current technology, and cognitive science therefore became more of a static
science of perception, cognition and action than a science of embodied
dynamic interactions. On the other hand, when researchers actually paid
attention to embodied movement, it often appeared that, as Famell (1995)
formulated it, the moving body has lost its mind. Another reason why re-
searchers commonly overlook the role of the body is perhaps the wide-
spread fear of slipping into biological reductionism which is also why most
social scientists tend to, or prefer to, view mind as superior to and inde-
pendent of the body (Famell 1999).
Hence, it should be noted that investigating the role of embodiment in
social interaction is not the same as relapsing into (socio-) biological re-
ductionism. In fact, the underlying supposed dichotomy between nature
and culture, i.e., biological and sociocultural factors, is highly misleading
in the first place, despite the fact that it has a long tradition in scientific
discussions. Ingold (2000) used the example of learning to walk as an il-
lustration of the traditional conception of the relation between "nature" and
"culture". It is commonly argued that walking is an innate human capabil-
ity, but Ingold does not categorize human walking as either innate or an
acquired. A child learns to walk according to the standard manner of its
social and cultural environment, and there is no one "natural" or "pure"
biological way of walking, as one might assume. That means, the human
skill of walking can be viewed as "biological" in the sense of being a part
of the functions of the human organism, but it is also a result of the child's
involvement in a social and physical world during normal development.
Ingold (2000) therefore pointed out that instead of speaking of embodi-
ment, the term "enmindment" could be used as well, since body and mind
are not separate, but in some sense two sides of one coin, i.e., two ways of
2. See also Barbour (1999) who points out that the dichotomous concept of man in
Christianity is a result of the Greek dualism of body and soul and actually not
supported by the biblical view.
describing the same process, namely the activity of the (human) organism
in its social and physical environment.
Currently, the increased interest in both bodilylbiological and so-
ciocultural processes in cognitive science has led to a partial rediscovery of
psychological, philosophical and biological theories that predated cognitive
science. Yet these theories received very little attention during the first
decades of cognitive science which at the time was dominated by the com-
puter metaphor. This applies, for example, to the works of the Russian
scholar Vygotsky, who already in the 1930s suggested that social interac-
tion is not an add-on to, but in fact a necessary requirement for individual
cognitive development, an idea revived in recent humanoid robotics re-
search (cf., Lindblom and Ziemke 2003), and the French philosopher Mer-
leau-Ponty, who already in the 1940s argued that what he called intercor-
poreality constituted the basis of intersubjectivity and social interaction, an
idea that has been rediscovered or confirmed in recent neuroscientific
findings involving so-called mirror neurons (cf., Gallagher this volume).
These examples illustrate that modem cognitive science still might have
much to gain from re-evaluating and possibly incorporating some of these
"old" ideas. The next section therefore provides some historical perspec-
tives, ending with the "cognitive revolution" and the criticisms thereof.
Section 3 then further discusses the role of embodiment and social interac-
tion in cognitive science today. Next, Section 4 discusses the positions of
"simple" vs. "radical" embodiment in current theories of embodiment, and
finally, Section 5 discusses implications for embodied cognitive science
and its possible contributions to the development of interactive technology.
2. Historical perspectives
Cognitive science, which as a discipline only has been around for a few
decades, can be said to have a short history but a long past, given that its
philosophical roots can be traced back to ancient Greece (Gardner 1987).
Even the view of the human being as a reasoning device, calculating ac-
cording to rules, has its origin in the work of Plato, who argued that phe-
nomena that could not be formalized explicitly, such as bodily skills and
feelings, should not count as knowledge. Consequently, he distinguished
between the rational mind and the body with its emotions and skills. This
was the starting point of the Western philosophical tradition, assuming that
reckoning is the "language of the mind" (Dreyfus 1979; cf., Johnson and
Rohrer this volume; Lakoff and Johnson 1999). This conceptual separation
of mind and body was further developed by Aristotle who divorced the
practical from the theoretical and defined the human being as the "rational
animal". Descartes then confirmed this view, rather than adding anything
radically new, and formulated the dualist viewpoint of mind and body as
two different substances (Cisek 1999; Dreyfus 1979; see also Costall this
volume). That means, while other animals, according to Descartes, are
mere mechanisms, or bodies without minds, humans alone possess a soul
which he also described as the "pilot of the corporeal boat" (Dreyfus 1979;
Freeman and Nufiez 1999). Vitalists, on the other hand, disagreed with this
mechanistic view of life and argued that all living creatures had some sort
of "vital energy" comparable to a spirit or soul. The issue never really got
resolved, but new life was brought into the debate through the work of
Darwin (cf., Ziemke 2001b).
After Charles Darwin's 1859 book, The Origin of Species, many re-
searchers began to search for unified theories that would explain the be-
havior of humans as well as other organisms. Interestingly, this resulted in
two almost opposite lines of research, which nevertheless can be traced
back, at least partly, to Darwin himself. Darwin tried to explain how differ-
ent species had evolved by assuming that patterns of behavior were the
product of evolution and that there was a mental linkage between animals
and humans (Cosmides et al. 1997; McFarland 1993). In modem terms,
Darwin certainly viewed the mind as embodied and did not believe in a
mind separate from the body. For example, he wrote in his personal note-
book (1836-1844) that "experience shows the problem of the mind cannot
be solved by attacking the citadel itself - the mind is function of [the] body
- we must bring some stable foundation to argue from" (Sheet-Johnstone
1999: 435). However, instead of studying mind as "function of the body",
many later researchers reduced the body and its interactions with the envi-
ronment to only include the brain itself (cf., Sheets-Johnstone 1999).
Darwin's 1872 book, The Expression of the Emotions in Man and Ani-
mal, can be considered the first modem book on behavior, although he
focused primarily on facial expressions and did not pay much attention to
gestures and body language. Darwin himself combined psychology and
biology, but he could also be said to have caused the split between the two.
Although his work was a serious attack on dualism in the sense that he
tried to bridge the presumed gulf between body and soul, his fear of public
opinion resulted in two parallel lines of explanations for the development
of the human mind, namely a phylogenetic line that stressed descent, and
an adaptationist line that emphasized selection (Cosmides et al. 1997).
The adaptationist perspective has generally been ignored within psy-
chology, while the phylogenetic branch has been further explored, partly
following Darwin, who mainly tried to explain the human mind from a
phylogenetic perspective (Cosmides et al. 1997). Given that Darwin be-
lieved in a single universal line of development, in order to make this claim
work, he had to run counter to his earlier ideas which stressed the role of
environmental conditions and adaptations, resulting in several lines of
evolutionary development. Consequently, he had to argue for some kind of
supremacy of reason in reference to some general inherited endowment.
The adaptationist perspective, however, would not fully support this view,
since it focused on the differences between separate species. Darwin was
worried that this perspective could be interpreted as supporting the dualist
view, in the sense that different qualities in separate species actually could
lead to a uniquely human soul (Cosmides et al. 1997).
Darwin's work had great general impact and brought about increased
interest in the study of behavior (AlIen and Bekoff 1997; Cosmides et al.
1997; McFarland 1993). Subsequently, two contrary lines of behavioral
research emerged. Firstly, there were those who attributed human-like
subjective mental qualities to other vertebrates and even invertebrates.
Darwin (1872) himself, for example, described dogs as feeling pleasure
when doing what they considered their duty, and ants as feeling despair at
having lost their home to a spade (cf. Sparks 1982). He was, nevertheless,
well aware that many of these behaviors were purely instinctive. His pro-
tege Romanes (1882) was less careful and provided anthropomorphic ex-
planations for a large range of animal behaviors, attributing to animals
human-like degrees of intelligence, consciousness and emotions, based
solely on his own observations (cf. Ziemke 2001b).
Secondly, there were the mechanists who also believed in common be-
havioral mechanisms in humans and other organisms, but considered the
attribution of subjective mental qualities as unscientific. Instead they
viewed animals - unlike Descartes, often including humans - as Cartesian
puppets, i.e., mere objects guided by their environment. Darwin himself in
his 1880 book, The Power ofMovement in Plants, compared the downward
digging behavior of earthworms and moles to the downward growth of
roots in response to gravity. Von Sachs (1882) further elaborated the view
of plants as guided by "forced movements", which he termed tropisms, and
demonstrated the orientation of various plant organs towards or away from
stimuli such as gravity, light or moisture. Loeb (1890, 1918) extended von
Sachs' theory of tropisms to animal behavior to stress what he saw as the
fundamental identity of the curvature movements of plants and the loco-
motion of animals. Loeb's own experiments revealed many different kinds
of tropisms in caterpillars, moths, shrimps, etc., directed towards or away
from light, gravity, chemicals, etc. This led him to apply his theory to hu-
man intelligence and to claim even humans to be mere mechanisms "com-
pelled to react to physical conditions like corn waving in the wind" (Sparks
1982). Thus, his theory became the opposite of Romanes' view of feeling,
conscious animals.
At about the same time, Sherrington identified the reflex as the ele-
mentary unit of behavior and described the reflex arc as consisting of three
elements: an effector organ, a conducting nervous pathway leading to that
organ, and a receptor to initiate the reaction. As a consequence, experi- .
mental psychology made the supposedly distinct elements of stimulus and
response to its units of analysis and treated them as largely separate (cf.
Sherrington 1906; Ziemke 2001b). Sherrington's own initial description of
the reflex arc, nonetheless, implied an integrated neural circuitry of "coor-
dinated action". Dewey (1896) was also very critical of the more restricted
interpretation of the reflex arc and argued that this psychological model
was a legacy of dualism. He rejected the linear model of stimulus and re-
sponse, proposing instead the concept of circular sensorimotor coordina-
tion (Dewey 1896; cf. Clancey 1997). Hence, he put forward an alternative
explanation of how the mind works, without an intervening "conscious-
ness" controller as a kind of soul, while at the same time stressing the em-
bodied nature of mind, with sensorimotor coordination as the key underly-
ing mechanism (cf. Johnson and Rohrer this volume).
Cisek (1999) has pointed out two important issues in the development
of behavioral research at the beginning of the 20
th
century. Firstly, there
was a split in the study of living organisms, resulting in one line that con-
centrated on studying behavior, whereas the other focused on physiology.
Secondly, the behavioral sciences as a whole suffered from a certain envy
of formal approaches. Many researchers therefore tried to explain behav-
iors and mental phenomena in a mathematical fashion in order to validate
the study of behavior "scientifically", worried that otherwise their research
might not be considered serious science.
Behaviorism, which became the dominant approach in psychology in
the beginning of the 20
th
century, in particular in the US, argued that only
observations of overt behavior should be the object of study (AlIen and
Bekoff 1997). The approach had its roots in mechanistic theories and the
animal learning work of Pavlov, who viewed the biological body as a ma-
chine (McFarland 1993). Behaviorists aimed to explain behavior as the
result of learning, in terms of classical and operant conditioning. Antimen-
ta/ism and equipotentia/ity, the assumption that there was no inherent bias
in organisms towards some stimulus-response associations over others,
became the two pillars of behaviorism (Cosmides et al. 1997). This was the
first rigorous attack against the dualistic assumption in the sense that it
totally ignored mental content. Accordingly, little, if any, interest was di-
rected towards embodiment other than in a mechanistic sense.
All along, while psychology was dominated by mechanistic and behav-
iorist ideas, there were researchers working on alternative, constructivist
and phenomenological conceptions of behavior and mind. Some of these
can be viewed as the forerunners or roots of modem embodied and situated
theories of cognition. Researchers like von Uexldill, Heidegger and Mer-
leau-Ponty, for example, like behaviorists, emphasized, in different ways,
the situated and embodied nature of mind and the crucial role of agent-
environment interaction. However, completely unlike behaviorists, they
focused on the subjective, phenomenal dimensions of "being-in-the-
world", as Heidegger (1962) called it. At about the same time, Vygotsky
and Piaget developed their highly influential theories of cognitive devel-
opment which, in different ways, emphasized the embodied, (socially)
situated and constructive nature of cognition. Due to their relevance to
modem theories of embodiment and/or its role in social interaction, the
contributions of these researchers will be discussed in more detail in the
following.
Jakob von Uexldill (1864-1944), a German biologist strongly inspired
by the Kantian insight that all knowledge is determined by the knower's
subjective ways of perceiving and conceiving, considered it the task of
biology to expand Kant's research by investigating the role of the body and
the relationship between subjects and their objects (von Uexldill 1928; cf.
Ziemke 2001b). According to von Uexldill, the main problem with mech-
anist and behaviourist ideas was that they overlooked the organism's sub-
jective nature. In his own words:
The mechanists have pieced together the sensory and motor organs of ani-
mals, like so many parts of a machine, ignoring their real functions of per-
ceiving and acting, and have gone on to mechanize man himself. According
to the behaviorists, man's own sensations and will are mere appearance, to
be considered, if at all, only as disturbing static. But we who still hold that
our sense organs serve our perceptions, and our motor organs our actions,
see in animals as well not only the mechanical structure, but also the opera-
tor, who is built into their organs as we are into our bodies. We no longer
regard animals as mere machines, but as subjects whose essential activity
consists of perceiving and acting. (von Uexkii1l1957: 6)
It should be noted that von Uexkiill's view is not vitalistic (cf. Emmeche
2001; Ziemke and Sharkey 2001). He described animal behavior as guided
by "successive reflexes" each of which is "elicited by objectively demon-
strable physical or chemical stimuli"; at the same time he also emphasized
that the organism's components are forged together to form a coherent
whole, i.e., a subject that acts as a behavioral entity which forms a "sys-
tematic whole" with its Umwelt, namely, its subjective world of perception
and action.
The French philosopher Maurice Merleau-Ponty (1908-1961), strongly
inspired by Gestalt psychology and also partly by Heidegger and von
Uexkiill, argued that the mind was essentially embodied and interacting
with the surrounding world (e.g., Merleau-Ponty 1963, 1962; cf. Dreyfus
1979). More specifically, he argued that it is actually the body which pro-
vides meaning or intentionality for the mind. His core concept was the idea
that "I am my body" as a kind of "embodied cogito", which means that it is
not only the brain that does the thinking, but the whole body (Priest 1998;
cf. Roth this volume). He argued that it is the body that has the necessary
"knowledge" to perform tasks present at hand, since it "knows how to act"
and "how to perceive" through the history of its phylogenetic and ontoge-
netic interactions with the environment.
Merleau-Ponty (1969) tried to find the fundamental primitives of the
mutual interdependence between the organism and its world, and he con-
cluded that both are different aspects of the same primary whole of "brute
being", which he referred to as the flesh (cf. Mingers 2001). That means,
the lived body, according to Merleau-Ponty, has a twofold character, being
able to see or touch on the one hand, and being seen or touched on the
other. In his own words, the flesh is "not matter, is not mind, is not sub-
stance. To designate it, we should need the old term "element" [... ] in the
sense of a general thing, midway between the spatio-temporal individual
and the idea" (Merleau-Ponty 1969, quoted in Mingers 2001: 116).
In addition to questions concerning low-level functions and bodily in-
tentions, Merleau-Ponty tried to situate his theory in a cultural context,
arguing that bodily behavior constitutes the most basic form of communi-
cation and cooperation among organisms (cf. Gallagher this volume). Even
human language is an extension of these bodily acts, and when humans
have acquired a language, it will control human cognition and communica-
tion (Loren and Dietrich 1997; Neuman 2001). Merleau-Ponty (1969)
treated the relation between language and thought as an intertwined proc-
ess, arguing that the use of language actually is the process of thinking;
either we speak to ourselves or to other persons (Mingers 2001). To sum-
marize, the main characteristics of Merleau-Ponty's work are his non-
dualistic and anti-behavioristic view of embodied experience, his argument
that bodies in fact are intentional in themselves, and the view of cognition
as a biological phenomena, rather than a mental one.
The Swiss biologist/psychologist Jean Piaget (1896-1980) similarly
stressed the importance of sensorimotor activity for the emergence of in-
telligent behavior (e.g., Piaget 1952). He first and foremost viewed "cogni-
tion as an instrument of adaptation, as a tool for fitting ourselves into the
world of our experiences" (von Glasersfeld 1995: 14). That means, cogni-
tion in Piaget's sense is about the organization of an agent's sensorimotor
experiences and interactions with the environment - a view very similar to
that of von Uexkiill (cf. Ziemke 2001b). Hence, the relationship between
action and lmowledge is central in Piaget's genetic epistemology, which
posited lmowledge as constructed by the child in interaction with its envi-
ronment. The basic organizational force of intellectual development, ac-
cording to Piaget, is logic, and this is highlighted in his theory which char-
acterizes different forms of logical thinking. Consequently he claimed that
his theory of cognitive development was universal (cf. Sinha and Jensen
2000; Wadsworth 1996).
It has been argued that Piaget neglected the social dimension, but Sinha
and Jensen (2000) have pointed out that this is a common misunderstand-
ing (cf. Cole and Wertsch 1996). For example, Piaget stated that "[h]uman
lmowledge is essentially collective and social life constitutes an essential
factor in the creation and growth of lmowledge, both pre-scientific and
scientific" (Piaget 1995: 30). He actually stressed the social dimension as
an essential factor for the development of cognition, supposing that so-
ciocultural factors could either accelerate or retard the developmental pro-
cess, but perhaps not change its direction towards the terminal point of
(logical) cognitive development (Sinha and Jensen 2000).
Piaget's most significant contribution probably is the claim that cogni-
tive changes are the outcome of a constructive developmental process that
occurs in sensorimotor interaction with the environment. However, it
should also be mentioned that Piaget's theory has been criticized heavily
(cf. e.g., Miller 1983; Wadsworth 1996). Boden (1994), for example, noted
that Piaget actually underestimated children's innate capabilities, both
initially and later on during development. Furthermore, Piaget's theory was
based on intellectual ideals of the Western tradition, and did not pay much
attention to cultural differences in cognitive development, as illustrated by
his claim that his theory was universal (cf. Cole and Wertsch 1996).
The role of culture was strongly emphasized, on the other hand, in the
development theory of Russian scholar Lev Vygotsky (1896-1934). He
rejected mechanistic and behaviorist theories because they disregarded
what he considered essential differences between human and animal intel-
ligence (cf. Lindblom and Ziemke 2003). He wanted a psychological the-
ory that would describe the development of what he considered exclusively
human abilities, and, as a result, proposed that individual cognitive devel-
opment requires a sociocultural embedding through certain transformation
processes. These transformations, from elementary to higher mental func-
tions, take place via signification, i.e., the shift of control from the envi-
ronment in elementary mental functions, to the individual's voluntary
regulation of her behavior in higher mental functions. This voluntary
regulation of individual behavior, according to Vygotsky (1978), is accom-
plished through the use of mediating artificial or self-generated stimuli,
functioning as a link between external stimulus and response. He argued
that the incorporation of such psychological tools was the key difference
between animal and human behavior, and summarized this position in the
statement that "the central fact about our psychology is the fact of media-
tion" (Vygotsky 1933, quoted from Wertsch 1985:15). Psychological tools,
according to Vygotsky, also function as regulators of human social behav-
ior, and, among these tools, language is an especially important "organ-
izer", both in the form of speech and written text. The other transformation
process of cultural cognitive development is internalization, as stated in
Vygotsky's (1978) general law of cultural development, which states that
every function in the child's cognitive development appears twice, first
between humans (at a social interaction level) and then in the child's mind
(becomes internalized at the individual level). Hence, the cognitive abilities
of an "enculturated" adult are the product of developmental processes, in
which "primitive" and "immature" humans are transformed into cultural
ones through social interactions, leaving room for different forms of intel-
ligence.
A common criticism (and misunderstanding) ofVygotsky's work is that
he focused almost exclusively on the sociocultural aspects, but neglected
the biological line of development (e.g., Davydov and Radzikhovskii
1985). However, as Wertsch (1985) pointed out, Vygotsky considered
biological factors as necessary, but not sufficient conditions, stating for
example that "culture creates nothing; it only alters natural data in confor-
mity with human goals" (Vygotsky 1960, cited in Wertsch 1985:47). Else-
where he stated:
Both planes of development - the natural and the cultural - coincide and
mingle with each other. The two lines of change interpenetrate each other
and essentially form a single line of sociobiological formation of the child's
personality.3 (Vygotsky 1960, quoted from Wertsch 1985:41)
In summary, despite the fact that Vygotsky's and Piaget's theories are
commonly contrasted as focused on the "child in society" and the "child as
a solitary thinker" (cf. Cole and Wertsch 1996), respectively, they are in
fact largely compatible and agree in viewing knowledge as constructed
through the interaction of biological and sociocultural factors in the course
of cognitive development.
Despite serious criticisms and the development of the above competing
theories, behaviorism remained the dominant approach in psychology,
particularly in the US, throughout the first half of the 20
th
century. Its im-
pact started to decrease in the mid-1950s, mainly for two reasons: firstly,
ethologists challenged the equipotentiality assumption (Breland and Bre-
land 1961; Garcia and Koelling 1966); secondly, the advent of the com-
puter seemed to challenge anti-mentalistic assumptions and paved the way
for the so-called "cognitive revolution" (Gardner 1987). Cognitive science
in general, from its beginnings in the mid-1950s, and its focus on internal
representations in particular, were largely conceived as reactions against
the anti-mentalistic behaviorist stance. Little attention, if any, was there-
fore directed to studies in biology and animal behavior. This was partly
because cognitive science aimed for supposedly higher mental processes
which animals were assumed to lack, and partly because studies of animal
behavior were associated with the "horror" of behaviorism (Gardner 1987).
As a result, other theories that stressed agent-environment interaction, such
as the work of von Uexki.ill, Heidegger, Vygotsky, Merleau-Ponty and
Piaget, were largely ignored during the era of computationalist cognitive
science. In a nutshell, while behaviorists had treated mind as an opaque
box in a transparent world, computationalist cognitive science treated it as
3. It should be noted that the translation "sociobiological" should not be confused
with the later use of the term "sociobiology" by Wilson (1975).
a transparent box in an opaque world (Lloyd 1989; cf. Sharkey and Ziemke
1998).
According to the computer metaphor in cognitive science, cognition
takes place inside the skull in the form of abstract symbol manipulation,
while the body only serves as an input and output device, i.e., a physical
interface between internal program (cognitive processes) and external
world (cf. e.g., Pfeifer and Scheier 1999). According to this view, the es-
sence of cognition is that minds or brains, just like computers, "accept
information, manipulate symbols, store items in "memory" and retrieve
them again, classify inputs, recognize patterns and so on" (Neisser 1976:
5). Representations, viewed as based on a correspondence or mapping be-
tween elements of the external world and their internal correspondents,
played a crucial role in this conception of mind, since in some sense they
constituted the only link between agent and environment. Agents, and their
environments, were of course also still considered to be physical, but the
relation between body and mind was considered as similar to the relation
between hardware and software in a computer. That means, the body was
viewed a mere physical implementation of the mind, which obviously
needs some physical instantiation (cf. Chrisley and Ziemke 2003), but apart
from that is largely implementation-independent. This functionalist view
(e.g., Putnam 1975), often summarized by the slogan that "[m]inds are
simply what brains do" (Minsky 1975), might sound similar to Darwin's
statement that "mind is function of the body". Nonetheless, it left out the
body and was interpreted entirely differently. Pfeifer and Scheier summa-
rized the functionalist view as follows:
Functionalism [... ] means that thinking and other intelligent functions need
not be carried out by means of the same machinery in order to reflect the
same kinds of processes; in fact, the machinery could be made of Emmental
cheese, so long as it can perform the functions required. In other words, in-
telligence or cognition can be studied at the level of algorithms or computa-
tional processes without having to consider the underlying structure of the
device on which the algorithm is performed. From the functionalist position
it follows that there is a distinction between hardware and software: What
we are interested in is the software or the program. (Pfeifer and Scheier
1999:43)
The reason for the success of the computer metaphor for mind was that it
offered convincing answers to major questions that confronted cognitive
science during its initial decades (Cisek 1999). Computationalism offered
non-dualistic explanations of internal states, such as "memory". Further-
more it offered the exciting metaphor of mental states and processes acting
as the software running on the brain's hardware. This seemed to be an ele-
gant solution to the mind-body problem, bridging the gulf between
body/biology (hardware) and mind/psychology (software). Finally, com-
putationalist psychology also provided a longed-for mathematical formal-
ism for the mechanisms underlying behavior. It is therefore hardly surpris-
ing that the computer metaphor became the dominant model of how the
mind works (Cisek 1999).
However, in the late 1970s several criticisms of computationalism, in
particular the resulting research in artificial intelligence (AI), emerged
(Dreyfus 1979; Searle 1980; cf. Ziemke 2001b). The overall concern in
these criticisms was in fact the lack of embodiment and situatedness. Drey-
fus (1979), strongly inspired by the Heideggerian notion of being-in-the-
world, questioned traditional AI's use of explicit, symbolic representations,
arguing that the resulting computer programs lacked situatedness since they
represented descriptions of isolated bits of human knowledge "from the
outside". Dreyfus' (1979) explanation of human situatedness and a com-
puter program's lack thereof is still worth quoting at length:
Humans [... ] are, as Heidegger puts it, always already in a situation, which
they constantly revise. Ifwe look at it genetically, this is no mystery. We can
see that human beings are gradually trained into their cultural situation on
the basis of their embodied pre-cultural situation [... ] But for this very rea-
son a program [... ] is not always-already-in-a-situation. Even if it represents
all human knowledge in its stereotypes, including all possible types of hu-
man situations, it represents them from the outside [... ]. It isn't situated in
anyone of them, and it may be impossible to program it to behave as if it
were.
[... ] It seems that our sense of our situation is determined by our changing
moods, by our current concerns and projects, by our long-range self-
interpretation and probably also by our sensory-motor skills for coping with
objects and people - skills we develop by practice without ever having to
represent to ourselves our body as an object, our culture as a set of beliefs,
or our propensities as situation-action rules. All these uniquely human ca-
pacities provide a "richness" or a "thickness" to our way of being-in-the-
world and thus seem to play an essential role in situatedness, which in turn
underlies all intelligent behavior. (Dreyfus 1979: 52-53)
From this very brief overview of the historical roots, predecessors and
critiques of traditional cognitive science, it should be clear that the idea of
an embodied mind is by no means new. Although theorists have sought to
understand the mind from different perspectives, the rational and formal-
ized view was the dominating approach in cognitive science for a long
time, and consequently the role of the body and the environment - physical
as well as social- was largely disregarded. As Dreyfus (1979) pointed out,
the computer, as well as the computer metaphor for mind, is the product of
over 2,500 years of traditional thinking in Plato's footsteps. In that sense,
as Cisek (1999) noted, the cognitive revolution was just "old wine in new
bottles". Since the late 1980s, however, cognitive science has re-discovered
or re-invented many, although not all, of the "pre-revolutionary" ideas and
revived them in theories that acknowledge the embodied, situated, distrib-
uted and sociocultural nature of the human mind, a topic which will be
discussed in more detail in the following section.
3. Embodiment and social interaction: Current perspectives
Today, there is a growing interest in embodiment in many areas of cogni-
tive science. Indeed, many researchers consider embodiment a necessary
requirement for intelligence and mind. However, among proponents of
embodied cognition there are nowadays a large number of notions and
conceptions of embodiment and embodied cognition (e.g., Chrisley and
Ziemke 2003; Ziemke 2003; cf. Ziemke and Frank this volume), a fact also
amply illustrated by the diversity of perspectives collected in this volume.
This diversity is also somewhat characteristic of the current state of
cognitive science, which during the last twenty years has developed from a
fairly unanimous agreement on the centrality of computation and (sym-
bolic) representation to a pluralistic inter-discipline that constantly chal-
lenges and re-evaluates its own foundations and conceptions. As already
mentioned in the introduction, in recent years both the neuroscientific and
the anthropological perspective have steadily gained ground in cognitive
science. This has lead to new sub-disciplines such as social cognitive neu-
roscience (e.g., Blakemore, Winston and Frith 2004) which, also for tech-
nical reasons would have been inconceivable only ten years ago. Thus, it is
becoming increasingly accepted that cognition does not, as Gardner (1987)
described it, constitute or require "a level of analysis wholly separate from
the biological or neurological, on the one hand, and the sociological or
cultural, on the other" (cf. above). In fact, the opposite is true: many cog-
nitive scientists today are actively researching the dependence of cognitive
processes on their neurobiological realization, their sensorimotor embodi-
ment as well as their sociocultural situatedness.
Somewhat surprisingly, however, despite the rich discussion of differ-
ent aspects and forms of embodied cognition, the social dimension of em-
bodiment has not yet received much attention. However, recent work in
cognitive science and related disciplines indicates that the body has several
important roles in social interactions and cognition. Barsalou et aI. (2003),
for example, have identified a number of different empirical findings,
mostly from social psychology experiments, which illustrate the role of
embodiment and social cognition. More specifically, they identified four
types of social embodiment effects which are discussed in more detail in
the following.
Firstly, perceived social stimuli do not only produce cognitive states,
but also bodily states. For example, it has been reported that high school
students receiving good grades in an exam adopted a more erect posture
than students that received poor grades (cf. Barsalou et aI. 2003). As Bar-
salou et aI. pointed out, it is unclear whether the bodily reaction is trig-
gered by the social event/stimulus directly, or by mediating mechanisms. In
this example it could be the case that an emotional state was triggered first
which in turn resulted in a bodily state. Another example is that subjects
primed with concepts related to elderly people (e.g., "gray", "bingo",
"wrinkles") exhibited embodiment effects such as slower movement when
leaving the experimental lab, as compared to a control group primed with
neutral words. Several other studies also show similar effects (cf. Barsalou
et aI. 2003).
Secondly, the observation of bodily states in others often results in bod-
ily mimicry in the observer. Studies show, for instance, that experimental
subjects often mimic an experimenter's actual behavior, e.g., rubbing the
nose or shaking a foot, and that subjects also tend to mimic observed facial
expressions, a fact documented widely in the literature (cf. Barsalou et aI.
2003).
Thirdly, bodily states produce affective states, which mean that em-
bodiment not only facilitates a response to social stimuli but also produces
tentative stimuli. For example, evidence offacial elicitation or facial feed-
back is well documented in the literature. According to Barsalou et aI.,
many studies demonstrate that the shaping of the face to an emotional ex-
pression tends to produce the corresponding affective state. For example, it
has been demonstrated that subjects rated cartoons differently when hold-
ing a pen between their lips than when holding it between their teeth. The
latter triggered the same musculature as smiling, which made the subjects
rate the cartoons as funnier, whereas holding the pen between the lips acti-
vates the same muscles as frowning and consequently has the opposite
effect. That means, the expressions associated with the musculature af-
fected the evaluation of the cartoon although the subjects were not even
aware of the fact that their musculature had been manipulated into a facial
expression. Similarly, evidence of bodily elicitation has been provided in
studies showing that bodily positions or postures actually influenced the
subjects' affective state, for example, that subjects in an upright position
experienced more pride than subjects in a slump position (cf. Barsalou et
al. 2003).
Finally, compatibility between bodily and cognitive states enhances per-
formance. For instance, several motor performance compability effects
have been reported, and it has been demonstrated that subjects actually
responded faster to "positive" words (e.g., "love") than "negative" words
(e.g., "hate") when asked to pull a lever towards them. Other studies have
shown that embodiment-cognition compatibility reduces the required proc-
essing resources in secondary task performance. Taken together, these
examples and other studies demonstrate that there is a strong relation be-
tween embodied and cognitive states. Barsalou et al. concluded that em-
bodiment is fundamental in human cognitive processing, and that a com-
mon mechanism seems to produce such effects across various domains.
Barsalou et al. (2003) offered a theoretical framework of social em-
bodiment to explain these phenomena based on internal simulations (cf.
Svensson, Lindblom and Ziemke this volume), which can be considered as
a mix composed of the traditional view of symbolic information process-
ing, mental imagery and embodiment. Even though their framework is an
important step towards highlighting the importance of sensorimotor aspects
in cognitive processing, it still constitutes an example of the position that
Clark (1999) referred to as "simple embodiment". According to this view,
traditional cognitive science can roughly remain the same; or, in Clark's
words, "facts about embodiment and environmental embedding" can be
treated as mere "constraints upon a theory of inner organization and proc-
essing". In Barsalou et al.' s case, for example, it could be said that their
theory adds an embodied icing to the traditional information-processing
cake in the sense that it acknowledges embodiment by taking as its starting
point perceptual/modal representations. Yet it continues to explain cogni-
tion largely in terms of internal representations and the computational pro-
cesses manipulating them.
The position of radical embodiment, on the other hand, is, as Clark
(1999:348) formulated it, "radically altering the subject matter and theo-
retical framework of cognitive science" by challenging traditional concep-
tions of representation and computation and questioning the conceptual
separations between perception and action as well as between brain, body
and world. Work on so-called mirror-neurons as the neurobiological
mechanism of "intercorporeality" as the basis of social interaction (cf.
Gallagher this volume) as well as simulation theories of mind-reading (cf.
Gallese and Goldman 1998) are good examples of more radically embodied
views of social cognition.
Dautenhahn (1997), for example, hypothesized that a phenomenological
dimension of social understanding might be founded in embodied mecha-
nisms that allow humans to read social signs and other agent's minds. She
suggested that an agent's own body can be used as the point of reference
for "simulating" another agent's emotional stance. Such a mechanism
might be found in a particular type of visuo-motor neurons in premotor
cortex, so-called mirror neurons, which are active in both the execution of
goal-related motor actions (e.g. grasping movements) and their observation
in conspecifics (e.g. Gallese and Goldman 1998). Although the existence
of mirror neurons so far has been experimentally confirmed only in mon-
keys4, there is strong empirical evidence that such a system actually is pre-
sent in human beings as well (e.g., Arbib 2006; Gallese et al. 2002). This
suggests that the function of this matching system might be a part of, or a
precursor to, a general mind-reading capability that allows you to adopt the
point of view of other conspecifics by matching or simulating their mental
states with a resonant state of your own, i.e., putting yourself in another's
"shoes" in order to understand or predict mental states and behavior. Em-
pirical findings support the general idea (cf. Svensson, Lindblom and
Ziemke this volume). For example, it has been found that observers under-
take motor facilitation in the same muscles as used by the observed indi-
vidual. That means, even while only observing actions of another individ-
ual, a neural "triggering" event in fact takes place in the observer. Gallese
et al. (2002: 459) therefore suggested "that the capacity to empathize with
others [... ] may rely on a series of matching mechanisms that we just have
started to uncover". Although Gallese and Goldman (1998) have stressed
4. Single neuron recordings of the type used in these experiments eventually de-
stroy the neurons recorded from, and thus for ethical reasons cannot be used in
humans.
that imitation behavior has not been observed in connection to mirror neu-
ron activity, Rizzolatti et al. (2002) have hypothesized that various low-
and high-level resonance mechanisms may be used under the banner of
"imitation", ranging from response facilitation to emulation and "true"
imitation.
Currently the role of mirror neurons in embodied simulation processes
as the basis of social interaction in general (e.g., Gallese, Keysers and Riz-
zolatti 2004), and language in particular (cf. Arbib 2005; Rizzolatti and
Arbib 1998), is receiving a great deal of attention. Nevertheless, these
mechanisms still are far from being well understood, and there is a need to
further integrate them into embodied/situated cognitive theories to develop
a more thorough understanding of the mechanisms that facilitate the human
sensitivity to social cues from the perspective of radical (social) embodi-
ment.
4. Discussion: Simple vs. radical (social) embodiment
As discussed in the introduction, as well as in several other contributions to
this volume, the role of the body in cognitive processes has received an
increasing amount of attention in recent years from different perspectives
and disciplines. While this certainly is a step, or in fact several steps, in the
right direction, i.e., towards an interdisciplinary understanding of the em-
bodied and socioculturally situated mind, it might be worth keeping in
mind that there also is a substantial risk of premature superficial agree-
ment. As several authors have pointed out, there currently is a wealth of
diverse notions, definitions and conceptions of embodiment (cf. e.g.,
Chrisley and Ziemke 2003; Clark 1999; Rohrer this volume; Svensson,
Lindblom and Ziemke this volume; Wilson 2002; Ziemke 2001a, 2003;
Ziemke and Frank this volume). On the one hand, this can be interpreted
positively since it shows that the embodiment of mind is studied from a
number of perspectives which eventually might converge to form a fuller
understanding than any individual discipline or approach could have pro-
duced on its own. On the other hand, after two decades of research on em-
bodied cognition, the apparent lack of coherence, could be interpreted as
confirmation of the recurrent criticism that in fact all that embodied cogni-
tive theories have in common is their rejection of traditional cognitive sci-
ence (cf. Chrisley and Ziemke 2003).
That not everybody means the same thing by "embodiment" should be
clear by now. As discussed in the previous section, one distinction par-
ticularly relevant to cognitive science is that made by Clark (1999) be-
tween the positions of "simple embodiment" and "radical embodiment". A
typical example of the former is the position of robotic functionalism (Har-
nad 1989), according to which cognition is computational after all, but
mental representations need to be "grounded" in sensorimotor interaction
with the environment (Hamad 1990). The body's role, according to this
view, is still very much that of the (software) mind's physical (hardware)
interface to the world it represents. That means, although the view of
grounded mental representations rejects behaviorism, the mechanistic view
of the body as such is largely maintained (cf. Ziemke 2001b). While this
view of embodiment and representation grounding is often not formulated
explicitly, it nevertheless implicitly underlies many current conceptions of
embodied cognition, in particular in artificial intelligence (cf. Ziemke
2004). This fact is also reflected in the way that some of the theories dis-
cussed in Section 2 have been incorporated into today's research. Von
Uexkiill's notion of Umwelt, for example, i.e., the idea that each creature
necessarily has its own subjective, perception- and action-dependent view
of the world, has received much attention in' embodied artificial intelli-
gence and robotics (e.g., Brooks 1986; Clark 1997; Emmeche 2001; Prem
1997; Sharkey and Ziemke 1998; Ziemke and Sharkey 2001). However, the
organismic roots (cf. Damasio 1995, 1999) and the phenomenal nature of
subjective experience, which were strongly emphasized by von Uexkiill
and Merleau-Ponty, are still largely ignored. Similarly, the developmental
psychology of Vygotsky and Piaget has received much attention in human-
oid robotics. Nonetheless in these cases it is also the functional perspective
on agent-environment interaction that has been adopted, while the biologi-
cal dimension has been largely ignored or abstracted in the form of com-
putationalleaming mechanisms (cf. Lindblom and Ziemke 2003). It should
be noted that, while the examples from artificial intelligence are perhaps
the most obvious cases (cf. Ziemke 2004), instantiations of the simple em-
bodiment position are of course equally common in other areas of cognitive
SCIence.
A prime example of "radical embodiment", on the other hand, is the
work of Maturana and Varela (1980, 1987) on autopoiesis and the biology
of cognition (cf. Johnson and Rohrer this volume). As we have discussed in
detail elsewhere (Ziemke and Sharkey 2001; Ziemke 2001b), the notion of
organisms as autopoietic, i.e., self-creating and maintaining, is very similar
in spirit to von Uexkiill's distinction between organisms as autonomous
subjects and mechanisms as heteronomous physical objects. Particularly
noteworthy in the context of radical embodiment is also Varela, Thompson
and Rosch's (1991) work towards an enactive cognitive science (cf. also
Thompson and Varela 2001), that has drawn much inspiration from the
works of Merleau-Ponty and Piaget, among others, and in its turn inspired
much current work seeking to further integrate phenomenology with em-
bodied cognitive science and neuroscience (e.g., Dreyfus 2002; Gallagher
this volume; Thompson and Varela 2001). Thus, many aspects of pre-
cognitive-science theoretical biology, psychology and philosophy have
found their way into current theories of embodied cognition, although their
interpretation can vary significantly depending on the interpreter's own
commitment to simple or radical embodiment.
It should also be noted that the distinction between simple and radical
embodiment should neither be overrated nor underestimated. On the one
hand, it really describes a continuum of positions rather than a binary dis-
tinction. Here, for example, we chose to categorize the position of Barsalou
and colleagues as simple social embodiment because it maintains elements
of computationalism and representationalism in the traditional sense,
whereas we consider much work on mirror neurons to fall into the radical
embodiment category because it pays more attention to the phenomenal
nature of subjective and intersubjective experience (cf. Gallagher, this
volume). But these distinctions are not as clear-cut as they might seem; for
example, there are also purely computational interpretations of mirror neu-
rons.
On the other hand, it is useful, or in fact essential for developing an em-
bodied cognitive science, to keep in mind that there are distinctions to be
made and that there might be much theoretical tension hidden under a pos-
sibly premature, superficial agreement on "embodiment" and the rejection
of traditional, computationalist ideas. It should be noted that traditional
cognitive science "missed out" on much by simply rejecting behaviorism
while ignoring the competing theories discussed here that indicated the
importance of embodiment long before cognitive science picked up on it.
Likewise, current embodied cognitive science would be ill advised to sim-
ply frame itself in opposition to the computationalism of traditional, sup-
posedly disembodied cognitive science.
5. Implications for cognitive science and interactive technology
We would like to close this chapter by pointing out some issues that need
to be addressed in order for current theories of embodied cognition, radical
embodiment in particular, to develop into a "mature science of the mind"
(Clark 1999: 349).
Firstly, although embodiment has been much discussed, the role of the
body in social interaction and cognition has been oversimplified. Much of
the discussion has focused on what you might call the "static" body. For
example, in AI it has been discussed what kind of physical realization, or
implementation, of the "body" is necessary for cognitive processes, ranging
from very simple agents with a minimum of sensors and motors to very
complex organism-like, e.g., humanoid, robot bodies (cf. Ziemke 2001 a,
2003). However, the crucial aspect of the body in motion has received
relatively little attention, although research in child development and an-
thropology has shown the relevance of locomotion experience for human
cognition (cf. Campos et al. 2000; Famell 1995, 1999; Sheets-Johnstone
1999). We have elsewhere discussed in more detail (Lindblom and Ziemke
2005, 2006) the example of the so-called "nine month revolution" (Toma-
sello 1999), which is the time when Euro-American infants begin to de-
velop triadic joint attention abilities and a basic understanding of a "self'.
How and why this transition occurs is still little understood, but we have
argued that it is no coincidence that this "revolution" occurs at around the
same point in time as the onset of self-produced locomotion behavior, i.e.,
when infants start to creep or crawl (Lindblom and Ziemke 2005, 2006). In
other words, when children begin to locomote by themselves, they acquire
an individual experience of the surrounding world and how it is affected
through their own actions and perceptions. It should be stressed that loco-
motion itself might not be the crucial factor. Instead, the child's cognitive
and emotional development emerges from the dynamics of the bodily expe-
rience that result from its own locomotion behavior. As a result, the child
begins to distinguish between itself and its surrounding world, i.e., the
child quite literally experiences different perspectives on the world, and
how these perspectives depend on its own actions, and thus a primitive
"self' emerges. This emerging understanding is bootstrapped through so-
cially scaffolded bodily experience, which gives the child access to the
actual meaning of the social-communicative situation. Consequently, that
understanding of perspective-taking might be used during embodied simu-
lations of the type discussed in previous sections, allowing the child to
simulate "off-line" what it would be like to be in the other person's situa-
tion, i.e., seeing the world from the other's physical and social perspective.
This example illustrates once more how firmly "culture" and "nature", as
well as "body" and "mind", are seam1ess1y intertwined and grounded in the
dynamics of socially situated embodied experience.
Secondly, side-stepping the dualist traps of "mind" vs. "body" as well
as "nature" vs. "culture" (cf. Famell 1995; Rogoff 2003; Varela 1995)
requires certain methodological consequences. A promising starting-point
is to take a dialectical stance, following Vygotsky, and stress that the hu-
man mind is shaped through a course of "natural cultural development"
(Rogoff 2003). Moreover, alternative analytic tools for studying the body
and its movements are needed, which go beyond dista1 observers' "objec-
tive" descriptions of behavior. The latter are flawed from the perspective of
embodied social interaction, since they do not consider the non-observable
social situation at hand, which in many cases actually provides the meaning
to the visible embodied actions. For example, it is necessary to investigate
so-called "body language" or "non-verbal behavior" as one dynamically
embodied system that unites mind and body in action (Famell 1995). By
using the term "action" instead of "behavior", it is emphasized that socially
embodied actions are a set of movements that have agency, meaning or
intentions for the actual person or agent, since "bodies do not move and
minds do not think - people just do" (Famel1 1995). That means, the "en-
actment" of the body is a social act, and in order to direct yourself, you
have to consider how others will act and react in response to your own
actions. In that sense, our biological embodiment constrains while cultural
customs affect, but do not determine, the organization of socially embodied
interactions (Fame11 1999). Consequently, there are some obvious meth-
odological problems when observing, ana1yzing and illustrating socially
embodied actions. For example, if you suppose that there is no split be-
tween verbal and non-verbal communication, that implies that there are two
ways of expressing the same thought, verbal and gestura1 (non-verbal), that
might be appropriate in different contexts (cf. Go1din-Meadow 2003).
Hence, the analysis will have to consider both the spoken words and ges-
tural movements. The dynamic nature of socially embodied actions causes
problems when it comes to representation and illustration. Common media
such as verbal descriptions, pictures or photographs, are quite "static" and
therefore more dynamic representational forms might need to be devel-
oped.
Thirdly, and closely related to the previous point, there is a need for an
integrated framework of radical social embodiment that addresses the role
and relevance of the body and its sensorimotor processes in social interac-
tion. In doing so, one should not intend to bridge the 'gap' between, e.g.,
verbal vs. nonverbal interaction. Many crucial questions have been ad-
dressed only partly and/or only from one of the above perspectives. For
example, what role(s) do bodily movement and gesture play in cognitive
processes? How do the body and its sensorimotor processes affect social
interactions and, more specifically, which social cognitive processes are
affected, and what functional roles does the body serve? Such a framework
of the fundamental roles of embodiment in social interaction is developed
by Lindblom (2006, 2007). The developed framework explains and illus-
trates how embodiment is the part and parcel of social interaction and cog-
nition in the most general and specific ways, in which dynamically em-
bodies actions themselves provide meaning and agency. Furthermore, as
the multitude of perspectives briefly described here indicates, there is also
a need to investigate and analyze the different notions, perspectives and
aspects of social embodiment addressed in the area, and disentangle their
different roles in social interaction. For the moment, different perspectives
on social interaction range from environment-mediated cooperation in so-
cial insects over the "expressiveness" of different bodily communication
abilities in animals, humans and robots to theory of mind in humans (and to
some degree in other primates and robots). But, from a comparative per-
spective, it still remains unclear to what degree different kinds and levels
of embodied social interactions are grounded in bodily and/or cognitive
differences but see Lindblom 2007.
Last, but not least, it should be noted that investigating the role of em-
bodiment in social interaction and cognition is not just yet another varia-
tion of the old philosophical mind-body problem. It also is highly relevant
to and has strong implications for the development of interactive technol-
ogy for human-computer interaction (HCI) and computer-supported coop-
erative work (CSCW) (cf. also Emmeche this volume). This is due to the
fact that with the increasing use of technology in practically all areas of
human life, social aspects have become more and more relevant. This is
equally true for technology-supported social interaction between humans,
i.e., the increasing use of advanced communication technology in social
interaction or collaborative work. Moreover, human users interact with
technology that increasingly tends to adopt (partly) social modes of inter-
action, e.g., between humans and humanoid robots (cf. Fong, Nourbakhsh
and Dautenhahn 2003) or so-called "embodied" conversational computer
interfaces (e.g., Cassell et al. 2000). Much of this technology is developed
to better meet the strong human sensitivity to social stimuli and interac-
tions. However, it has been noted, for example in the area of computer-
supported cooperative work, that there is a gap, the so-called social-
technical gap, between what technology ideally should support socially
and what currently it actually can support (cf. e.g., Ackerman 2002; Erick-
son and Kellogg 2002). That means, there still is a stark contrast between
our embodied social interactions in the lived human world and the level of
support that is offered by contemporary technology. At least partly, this is
due to the fact that there is no sufficient understanding of the mechanisms
underlying human social interaction which would allow a more seamless
use or integration of technology. In particular the role of embodiment in
human social interaction is still far too little understood, and current de-
signs of information and communication technology are still dominated by
outdated information processing models of human communication.
In conclusion, we suggest that further re-evaluation and integration of
some of the pre-cognitive-science theories discussed in this chapter, as well
as other contributions to this volume, into the framework of modem theo-
ries of embodied cognition, along the lines pointed out above, can signifi-
cantly contribute to both the further theoretical development of embodied
cognitive science as such and its capacity to aid the development of future
interactive technology.
References
Ackermann, Mark S.
2002 The intellectual challenge of CSCW: The gap between social re-
quirements and technical feasibility. In: John M. Carroll (ed.), Hu-
man-Computer Interaction in the New Millennium, 303-324. New
York: Addison Wesley.
AlIen, Chris and Marc Bekoff
1997 Species ofMind: The Philosophy and Biology ofCognitive Ethology.
Arbib, Michael A.
2005 From monkey-like action recognition to human language: An evolu-
tionary framework for neurolinguistics. Behavioral and Brain Sci-
ences 28: 105-167.
Barbour, Ian G.
1999 Neuroscience, artificial intelligence and human nature: theological
and philosophical reflections. Zygon 34 (3): 361-398.
Barsalou, Lawrence W., Paula M. Niedenthal, Aron K. Barbey and Jennifer A.
Ruppert
2003 Social embodiment. In: Brian H. Ross (ed.), The Psychology of
Learning and Motivation 43: 43-92. San Diego, CA: Academic
Press.
Blakemore, Sarah-Jayne, Joel Winston and Uta Frith
2004 Social cognitive neuroscience: Where are we heading. Trends in
Cognitive Sciences 8 (5): 216-222.
Boden, Margaret A.
1994 Piaget. London: Fontana Press.
Breland, Keller and Marian Breland
1961 The misbehavior of organisms. American Psychologists 16: 661-
664.
Brooks, Rodney A.
1986 Achieving artificial intelligence through building robots. MIT AI Lab
Memo 899.
Burgoon, Judee K., David B, Buller and William G Woodall
1996 Nonverbal Communication: The Unspoken Dialogue (2
nd
ed). New
York: McGraw-Hill.
Campos, Joseph J., David I. Anderson, Marianne A. Barbu-Roth, Edward M. Hub-
bard, Matthew J. Hertenstein and David Witherington
2000 Travel broadens the mind. Infancy 1 (2): 149-219.
Cassell, Justine, Joseph Sullivan, Scott Prevost and Elizabeth Churchill (eds.)
2000 Embodied Conversational Agents. Cambridge, MA: MIT Press.
London: Macmillan.
Cisek, Paul
1999 Beyond the computer metaphor. In: Rafael Nunez and WaIter J.
Freeman (eds.), Reclaiming Cognition - the Primacy of Action, In-
tention and Emotion, 125-142. Exeter: Imprint Academic.
Clancey, William J.
1997 Situated Cognition: On Human Knowledge and Computer Repre-
sentations. New York: Cambridge University Press.
Clark, Andy
1997
1999
Being There - Putting Brain, Body and World Together Again.
An embodied cognitive science? Trends in Cognitive Sciences 3(9):
345-351.
Cole, Michael and James V. Wertsch
1996 Beyond the individual-social antimony in discussions of Piaget and
Vygotsky. Human Development 39 (5): 250-256.
Cosmides, Leda, John Tooby, Jonathan H. Turner and Boris M. Velichkovsky
1997 Looking back: Historical context of present practice - biology and
psychology. In: Peter Weingart, Sandra D. Mitchell, Peter J. Richer-
son and Sabine Maasen (eds.), Human by Nature - between Biology
and the Social Sciences, 52-64. Mahwah, NJ: Lawrence Erlbaum.
Costall, Alan
agency.
Damasio, Antonio R.
1995 Descartes' Error: Emotion, Reason and the Human Brain. New
York: Avon Books.
1999 The Feeling of What Happens: Body and Emotion in the Making of
Consciousness. New York: Harcourt Inc.
Darwin, Charles
1859 The Origin ofSpecies. London: Murray.
1872 The Expression ofEmotions in Man and Animals. London: Murray.
1880 The Power ofMovements in Plants. London: Murray.
Dautenhahn, Kerstin
1997 I could be you: The phenomenological dimension of social under-
standing. Cybernetics and Systems 25 (8): 417-453.
Davydov, V. V. and L. A. Radzikhovskii
1985 Vygotsky's theory and the activity-oriented approach to psychology.
In: James V. Wertsch (ed.), Culture, communication and cognition,
35-65. New York: Cambridge University Press.
Dewey, John
1896 The reflex arc concept in psychology. Psychological Review 3: 357-
370.
Dreyfus, Hubert L.
1979 What Computers Can't Do - a Critique of Artificial Reason. New
York: Harper and Row.
2002 Intelligence without representation - Merleau-Ponty's critique of
mental representation: The relevance of phenomenology to scientific
explanation. Phenomenology and the Cognitive Sciences 1 (4): 367-
383.
Emmeche, Claus
2001 Does a robot have an Umwelt? Reflections of the qualitative bio-
semiotics of Jakob von Uexkiill. Semiotica 134 (1/4): 653-693.
this vol. On the biosemiotics of embodiment and our human cyborg nature.
Erickson, Thomas and Wendy A. Kellogg
2002 Social translucence: Designing systems that support social processes.
In: John M. Carroll (eds.), Human-Computer Interaction in the New
Millennium, 325-330. New York: Addison Wesley.
Famell, Brenda
1995 Do You See What I Mean? Plains Indian Sign Talk and the Embodi-
ment ofAction. Austin: University of Texas Press.
Famell, Brenda
1999 Moving bodies, acting selves. Annual Review of Anthropology 28:
341-373.
Freeman, WaIter J. and Rafael NUflez
1999 Restoring to cognition the forgotten primacy of action, intention and
emotion. In: Rafael Nunez and Waiter J. Freeman (eds.), Reclaiming
Cognition - the Primacy of Action, Intention and Emotion, ix-xix.
Exeter: Imprint Academic.
Fong, Terrence, Illah Nourbakhsh and Kerstin Dautenhahn
2003 A survey of socially interactive robots. Robotics and Autonomous
Systems 42: 143-166.
Gallagher, Shaun
Gallese, Vittorio and Alvin Goldman
1998 Mirror neurons and the simulation theory of mind-reading. Trends in
Gallese, Vittorio, Christian Keysers and Giacomo Rizzolatti
Cognitive Sciences 8: 396-403.
Gallese, Vittorio, Pier F. Ferrari, Evelyn Kohler and Leonardo Fogassi
2002 The eyes, the hand and the mind: behavioral and neurophysical as-
pects of social cognition. In: Marc Bekoff, Colin AlIen and Gordon
M. Burghardt (eds.), The Cognitive Animal - Empirical and Theo-
retical Perspectives on Animal Cognition, 451-461. Cambridge,
MA: MIT Press.
Gardner, Howard
1987 The Mind's New Science. New York: Basic Books.
Garcia, J. and R. A. Koelling
1966 Relation to cue to consequence in avoidance learning. Psychonomic
Science 4: 123-124.
2003 Hearing Gesture: How Our Hands Help Us Think. Cambridge, MA:
The Belknap Press of Harvard University Press.
2005
2007
2006
Evolving skills. In: Hilary Rose and Steven Rose (eds.), Alas, poor
Darwin: Arguments against Evolutionary Psychology, 273-297.
New York: Harmony Books.
Johnson, Mark and Tim Rohrer
this vo!. We are live creatures: Embodiment, American pragmatism and the
cognitive organism.
Lindblom, Jessica
2006 Embodied action as a 'helping' hand in social interaction. In: Ron
Sun and Noami Miyake (eds.), Proceedings of the 28
th
Annual Con-
ference of the Cognitive Science Society, 477-482. Mahwah, NJ:
Lawrence Erlbaum.
Minding the Body: Interacting Socially through Embodied Actions.
Doctoral dissertation, University of Linkoping, University of Skovde,
Sweden.
Lindblom, Jessica and Tom Ziemke
2003 Social situatedness of natural and artificial intelligence: Vygotsky
and beyond. Adaptive Behavior 11 (2): 79-96.
Embodiment-in-motion: Broadening the social mind. In: Brono G.
Bara, Lawrence Barsalou and Monica Bucciarelli (eds), Proceedings
of the XXVII Annual Conference of the Cognitive Science Society,
1284-1289. Mahwah, NJ: Lawrence Erlbaum.
The social body in motion: Cognitive development in infants and
androids. Connection Science 18 (4): 333-346.
Lloyd, Dan E.
1989 Simple Minds. Cambridge, MA: MIT Press.
Hamad, Stevan
1990 The symbol grounding problem. Physica D 42:335-346.
1989 Minds, Machines and Searle. Journal ofExperimental and Theoreti-
cal Artificial Intelligence 1: 5-25.
Heidegger, Martin
1962 Being and Time. New York: Harper and Row. Originally published
in German in 1927.
Hendriks-Jansen, Horst
1996 Catching Ourselves in the Act: Situated Activity, Interactive Emer-
gence, Evolution and Human Thought. Cambridge, MA: MIT Press.
Hutchins, Edwin
Ingold, Tim
2000
Loeb, Jacques
1918 Forced Movements, Tropisms and Animal Conduct. Philadelphia:
Lippincott.
1890 Der Heliotropismus der Thiere und seine Ubereinstimmung mit dem
Heliotropismus der Pjlanzen. Wiirzburg, Germany: Georg Hertz.
Loren, Lewis A. and Eric Dietrich
1997 Merleau-Ponty, embodied cognition and the problem of intentional-
ity. Cybernetics and Systems 28 (5): 345-358.
1987 The Tree of Knowledge - the Biological Roots of Human Under-
McFarland, David J.
1993 Animal Behavior: Psychology, Ethology and Evolution. Harlow:
Longman.
1962 Phenomenology of Perception. London: Routledge and Kegan Paul.
Originally published in French in 1945.
1963 The Structure of Behavior. Boston, MA: Beacon Press. Originally
published in French in 1942.
1969 The Visible and the Invisible. Evanston: Northwestern University
Press. Originally published in French in 1964.
Miller, Patricia H.
1983 Theories of Developmental Psychology. New York: W. H. Freeman
and Company.
Mingers, John
2001 Embodying information systems: the contribution of phenomenology.
Information and Organization 11: 103-128.
Minsky, Marvin
1975 A framework for representing knowledge. In: Patrick Winston (ed.),
The Psychology ofComputer Vision, 211-277. McGraw Hill.
Neisser, Ulric
1976 Cognition and Reality: Principles and Implications of Cognitive
Psychology. San Francisco: C. A. Freeman.
Neuman, Yair
2001 On Turing's carnal error: some guidelines for a contextual inquiry
into the embodied mind. Systems Research and Behavioural Sciences
18: 557-564.
The Origin of Intelligence in the Child. New York: Basic Books.
Originally published in French in 1936.
Sociological Studies. London: Routledge. 1995
Prem, Erich
1997 Epistemic autonomy in models of living systems. In: Phil Husbands
and Inman Harvey (eds.), Fourth European Conference on Artificial
Life, 48-73. Cambridge, MA: MIT Press.
Priest, Stephen
1998 Merleau-Ponty. London: Routledge.
Putnam, Hilary
1975 Philosophy and our mental life. In: Hilary Putnam (ed.), Mind, Lan-
guage and Rationality, 48-73. Cambridge: Cambridge University
Press.
Riegler, Alexander
2002 When is a cognitive systems embodied? Cognitive Systems Research
3(3): 339-348.
Rizzolatti, Giacomo and Arbib, Michael A.
1998 Language within our grasp. Trends in Neurosciences 21: 188-194.
Rizzolatti, Giacomo, Vittorio L. Fadiga, Leonardo Fogassi and Vittorio Gallese
2002 From Mirror Neurons to Imitation: Facts and Speculations. In: An-
drew Meltzoff and Wolfgang Prince (eds.), The Imitative Mind: De-
velopment, Evolution and Brain Bases, 247-266. Cambridge, MA:
Cambridge University Press.
Rogoff, Barbara
2003 The Cultural Nature of Human Development. New York: Oxford
University Press.
Piaget, Jean
1952
Rohrer, Tim
this vol. The body in space: Dimensions of embodiment.
Romanes, George
1882 Animal Intelligence. London: Kegan Paul.
Roth, Wolff-Michael
this vol. Communication as situated, embodied practice.
Sapir, Edward
1928 The unconscious patterning of behavior in society. In: E. S. Dummer
(ed.), The Unconscious, 114-142. New York: Knopf.
Searle, John
1980 Minds, brains and programs. Behavioral and Brain Sciences 3: 417-
457.
Semin, Giin R. and Eliot R. Smith
2002 Interfaces of social psychology and situated and embodied cognition.
Cognitive Systems Research 3 (3): 385-396.
Sharkey, Noel E. and Tom Ziemke
1998 A consideration of the biological and psychological foundations of
autonomous robotics. Connection Science 10 (3-4): 361-391.
Sheets-Johnstone, Maxine
1999 The Primacy ofMovements. Amsterdam: John Benjamins.
Sherrington, Charles S.
1906 The Integrative Action of the Nervous System. New York: C. Scrib-
ner's Sons.
Sinha, Chris and Kristine Jensen de Lopez
2000 Language, culture and the embodiment of spatial cognition. Cogni-
tive Linguistics 11: 17-41.
Sparks, John
1982 The Discovery ofAnimal Behaviour. London: Collins.
Svensson, Hemik, Jessica Lindblom and Tom Ziemke
this vol. Making sense of embodied cognition: simulation theories of shared
Thompson, Evan and Francisco J. Varela
2001 Radical embodiment: Neural dynamics and consciousness. Trends in
Tomasello, Michael
1999 The Cultural Origins ofHuman Cognition. Cambridge, MA: Harvard
University Press.
Trevarthen, Colwyn
1977 Descriptive analysis of infant communicative behaviour. In: H. R.
Schaffer (ed.), Studies in Mother-Infant Interaction Proceedings of
Loch Lomond Symposium, 227-270. New York: Academic Press.
Varela, Charles R.
1992 Harre and Merleau-Ponty: beyond the absent moving body in em-
bodied social theory. Journal for the Theory ofSocial Behaviour 24
(2): 167-185.
1995 Cartesianism revisited: The ghost in the moving machine or the lived
body. In: Brenda Famell (ed.), Human Action Sign in Cultural Con-
texts: the Visible and Invisible in Movement and Dance, 216-293.
New Jersey: Scarecrow Press.
von Glasersfeld, Emst
1995 Radical Constructivism - a Way ofLearning and Knowing. London:
Falmer Press.
von Sachs, Julius
1882 Vorlesungen aber Pflanzenphysiologie. Leipzig, Germany: W. En-
gelmann.
von Uexkiill, Jakob
1928 Theoretische Biologie. Berlin: Springer.
1957 A stroll through the worlds of animal and men a picture book of
invisible worlds. In: Claire H. Schiller (ed.), Instinctive Behaviour-
The Development of a Modern Concept, 5-80. New York: Interna-
tional Universities Press. Also appeared in 1992 in Semiotica 89(4):
319-391. Originally published in German in 1934.
Vygotsky, Lev S.
1978 The Mind in Society: The Development ofHigher Mental Processes.
Cambridge, MA: MIT Press. Original work published in Russian in
1934.
Wadsworth, Barry J.
1996 Piaget's Theory of Cognitive and Affective Development (5
th
ed.).
New York: Longman Publishers.
Wertsch, James V. (ed.)
1985 Vygotsky and the Social Formation of Mind. Cambridge, MA: Har-
vard University Press.
Wilson, Edward O.
1975 Sociobiology - the New Synthesis. Cambridge, MA: Harvard Univer-
sity Press.
Wilson, Margaret
9 (4): 625-636.
Ziemke, Tom
2001 a Are robots embodied? In: Christian Balkenius et al. (eds), Proceed-
ings of the First International Workshop on Epigenetic Robotics:
Modelling Cognitive Development in Robotic Systems, 75-83. Lund,
Sweden: Lund University Cognitive Series, vol, 85.
2001 b The construction of 'reality' in the robot. Foundations of Science 6
(1): 163-233.
2002 Introduction to the special issue on situated and embodied cognition.
Cognitive Systems Research 3 (3): 271-274.
2003 What's that thing called embodiment? In: Richard Alterman and
David Kirsh (eds), Proceedings of the 25
th
baum.
2004 Embodied AI as science: Models of embodied cognition, embodied
models of cognition, or both? In: Fumiya Iida et al. (eds), Embodied
AI, 27-36. Heidelberg: Springer Verlag.
Semiotica 134 (1-4): 701-746.
Ziemke, Tom and Roslyn M. Frank
this vol. Introduction: The body eclectic.
Section B
Body and mind
Representing actions and functional properties in
conceptual spaces
Peter Giirdenfors
Abstract
The book Conceptual Spaces (Gardenfors 2000) presents a theory of concepts
based on geometrical and topological structures in spaces that are built up from
"quality dimensions". Most of the examples in the book deal with perceptual con-
cepts based on dimensions such as colour, size, shape and sound. However, many
of our everyday concepts are based on actions and functional properties. For in-
stance most artefacts, such as chairs, clocks and telephones, are categorized on the
basis of their functional properties.
After giving a general presentation of conceptual spaces, I suggest how the
analysis in terms of conceptual spaces can be extended to actions and functional
concepts. Firstly, I will argue that "action space" can, in principle, be analysed in
the same way as e.g. colour space or shape space. One hypothesis is that our cate-
gorization of actions to a large extent depends on our embodied "perception" of
forces. In line with this, an action will be described as a spatio-temporal pattern of
forces. When it comes to functional properties, the key idea is that the function of
an object can be analysed with the aid of the actions it affords. Functional concepts
can then be described as convex regions in an appropriate action space.
Within Cognitive Semantics, image schemas are mainly based on perceptual
and spatial dimensions (e.g. Langacker 1987; Lakoff 1987). Two exceptions are
JoOOson's (1987) and Talmy's (1988) work on "force dynamics" that shows the
importance of forces, and metaphorical uses of forces, for the semantics of many
kinds of linguistic expression. I shall argue that a more developed understanding on
"action space" would allow us to extend the semantic analyses pioneered by JoOO-
son and Talmy. In particular, I shall make a distinction between fIrst-person and
third-person perspectives on "forces". My hypothesis is that we start out from an
embodied notion of force or "power" that is then extended to forces that are exerted
by other individuals and to forces that act on objects outside our control.
Keywords: conceptual spaces, force dynamics, image schemas.
168 Peter Giirdenfors
1. The problem of modelling concepts
A central problem for cognitive science is how representations should be
modelled. There are currently two dominating approaches to this problem.
The symbolic approach starts from the assumption that cognitive systems
can be described as Turing machines. On this view, cognition is seen as
essentially being computation involving symbol manipulation (e.g. Fodor
1975; Pylyshyn 1984; Pinker 1997). The second approach is association-
ism, where associations between different kinds of information elements
carry the main burden of representation. Connectionism is a special case of
associationism that models associations using artificial neuron networks
(e.g. Rumelhart and McClelland 1986; Quinlan 1991). Both the symbolic
and the associationist approaches have their advantages and disadvantages.
They are often presented as competing paradigms, but since they are used
to analyse cognitive problems on different levels of granularity, they
should rather be seen as complementary methodologies.
However, there are several aspects of concept formation for which nei-
ther symbolic representation nor connectionism seem to offer appropriate
modelling tools. In this chapter, I will advocate a third way to represent
information that is based on using geometrical structures rather than sym-
bols or connections between neurons. Using these structures, similarity
relations can be modelled in a way that accords well with human (and ani-
mal) judgments. The notion of similarity is crucial for the understanding of
many cognitive phenomena. I shall call this way of representing informa-
tion the conceptual form since I believe that such representations can ac-
count for more of the essential aspects of human concept formation than
symbolic or connectionist theories.
Based on my recent book (Gardenfors 2000), I shall first present a the-
ory of conceptual spaces as a particular framework for representing infor-
mation on the conceptual level. A conceptual space is built up from geo-
metrical representations based on a number of quality dimensions. Most of
the examples I discussed in my book deal with perceptual concepts based
on dimensions such as colour, size, shape and sound. However, there is
strong evidence that many of our everyday concepts are based on actions
and functional properties. For instance, most artefacts, such as chairs,
clocks and telephones, are categorized on the basis of their functional
properties (Nelson 1986; Mandler 2004).
In this chapter, I shall outline how the analysis in terms of conceptual
spaces can be extended to functional concepts. Firstly, I will argue that
Representing actions andfunctional properties in conceptual spaces 169
"action space" can, in principle, be analysed in the same way as e.g. colour
space or shape space. One hypothesis is that our categorization of actions
to a large extent depends on our "perception" of forces. In line with this, an
action will be described asa spatio-temporal pattern of forces. I shall also
argue that the most cognitively fundamental forces are those that act upon
or emanate from one's own body. In this sense my analysis will be based
on an embodied perspective.
When it comes to functional properties, the key idea is that the function
of an object can be analysed with the aid of the actions it affords. Func-
tional concepts can then be described as convex regions in an appropriate
action space. I shall outline a research programme indicating that action
space should be seen as a special case of a conceptual space.
2. Quality dimensions
As introductory examples of quality dimensions one can mention tem-
perature, weight, brightness, pitch and the three ordinary spatial dimen-
sions height, width and depth. I have chosen these examples because they
are closely connected to what is produced by our sensory receptors
(Schiffman 1982). The spatial dimensions of height, width and depth as
well as brightness are perceived by the visual sensory system, pitch by the
auditory system, temperature by thermal sensors and weight, finally, by the
kinaesthetic sensors. However, since there are also quality dimensions that
are of an abstract non-sensory character, one aim of this chapter is to argue
that force dimensions are important for the analysis of action concepts and
functional categories.
Quality dimensions correspond to the different ways stimuli are judged
to be similar or different. In most cases, judgments of similarity and differ-
ence generate an ordering relation of stimuli (Clark 1993: 114). For exam-
ple, one can judge tones by their pitch that will generate an ordering of the
perceptions. The general assumption is that the smaller the distance is be-
tween the representations of two objects, the more similar they are. In this
way, the similarity of two objects can be defined via the distance between
their representing points in the space. The dimensions form the "frame-
work" used to assign properties to objects and to specify relations between
them. The coordinates of a point within a conceptual space represent par-
ticular instances of each dimension, for example, a particular temperature,
a particular weight, etc.
The notion of a dimension should be understood literally. It is assumed
that each of the quality dimensions is endowed with certain geometrical
structures (in some cases they are topological or orderings). As a first ex-
ample, Figure 1 illustrates such a structure, the dimension of "weight"
which is one-dimensional with a zero point, and thus isomorphic to the
half-line of non-negative numbers. A basic constraint on this dimension,
commonly made in science, is that there are no negative weights.
o
I
Figure 1. The weight dimension.
A psychologically interesting example of a domain involves colour per-
ception. In brief, our cognitive representation of colour can be described by
three dimensions. The first dimension is hue, which is represented by the
familiar colour circle going from red via yellow to green and to blue and
then back to red again. The topological structure of this dimension is thus
different from the quality dimensions representing time or weight which
are isomorphic to the real line.
The second psychological dimension of colour is saturation, which
ranges from grey (zero colour intensity) to increasingly greater intensities.
This dimension is isomorphic to an interval of the real line. The third di-
mension is brightness that varies from white to black and is thus a linear
dimension with end points. Together, these three dimensions, one with
circular structure and two with linear, constitute the colour domain which
is a subspace of our perceptual conceptual space. This domain is often
illustrated by the so-called colour spindle (see figure 2). Brightness is
shown on the vertical axis. Saturation is represented as the distance from
the centre of the spindle. Hue, finally, is represented by the positions along
the perimeter of the central circle. The circle at the centre of the spindle is
tilted so that the distance between yellow and white is smaller than the
distance between blue and white.
White
Green
Black
Figure 2. The colour spindle.
A conceptual space can now be defined as a collection of quality dimen-
sions. However, the dimensions of a conceptual space should not be seen
as totally independent entities, rather they are correlated in various ways
since the properties of the objects modelled in the space co-vary. For ex-
ample, in the domain of fruits the ripeness and the colour dimensions co-
vary.
It is impossible to provide a complete list of the quality dimensions in-
volved in the conceptual spaces of humans. Some of the dimensions seem
to be innate and to some extent hardwired in our nervous system, as, for
example, colour, pitch, force and probably also ordinary space. Other di-
mensions are presumably learned. Learning new concepts often involves
expanding one's conceptual space with new quality dimensions (Smith
1989). Two-year-olds can represent whole objects, but they cannot reason
about the dimensions of the object. Goldstone and Barsalou (1998: 252)
note:
Evidence suggests that dimensions that are easily separated by adults, such
as the brightness and size of a square, are treated as fused together for chil-
dren [...]. For example, children have difficulty identifying whether two ob-
j ects differ on their brightness or size even though they can easily see that
they differ in some way. Both differentiation and dimensionalization occur
throughout one's lifetime.
Still other dimensions may be culturally dependent. Finally, some quality
dimensions are introduced by science. Witness, for example, Newton's
distinction between weight and mass, which is of pivotal importance for
the development of his celestial mechanics, but which has hardly any cor-
respondence in human perception. To the extent we have mental represen-
tations of the masses of objects in distinction to their weights, these are not
given by the senses but have to be learned by adopting the conceptual
space of Newtonian mechanics in our representations. In order to separate
different uses of quality dimensions it is important to introduce a distinc-
tion between a psychological and a scientific interpretation. The psycho-
logical interpretation concerns the cognitive structures (perceptions,
memories, etc) of human beings and other organisms. The scientific inter-
pretation, on the other hand, treats dimensions as a part of a scientific the-
ory. The distinction is relevant when the dimensions are seen as cognitive
(psychological) entities, in which case their structure should not be deter-
mined by scientific theories which attempt to give a "realistic" description
of the world, but by psychophysical measurements that determine how our
concepts are represented.
The conceptual space of Newtonian particle mechanics is, of course,
based on scientific (theoretical) quality dimensions and not on psychologi-
cal dimensions. The quality dimensions of this theory are ordinary space
(3-D Euclidean), time (isomorphic to the real numbers), mass (isomorphic
to the non-negative real numbers), and force (3-D Euclidean space). In this
theory, an object is thus represented as a point in an 8-dimensional space.
Once a particle has been assigned a value for these eight dimensions, it is
fully described as far as Newtonian mechanics is concerned.
I want to make it clear that the dimensions I consider in my analysis of
concepts should be given the psychological interpretation. This applies in
particular to the dimension of "force" that will be analysed in the latter
sections of this chapter (5-8). A problem for my distinction may be that in
Western cultures, the psychological concept of "force" has been tainted by
the Newtonian world-view. I will return to this topic in Section 7.
3. Concept formation described with the aid of conceptual spaces
The purpose of this section is to show how conceptual spaces can be used
to model concepts. I will focus on concepts that are "natural" in the sense
that they can, in principle, be learned without relying on linguistic descrip-
tions and, when described, have simple expressions in most languages. A
first rough idea is to describe a natural concept as a region of a conceptual
space S, where "region" should be understood as a spatial notion deter-
mined by the topology and metric of S. For example, the point in the time
dimension representing "now" divides this dimension, and thus the space
of vectors, into two regions corresponding to the concepts "past" and "fu-
ture". But the proposal suffers from a lack of precision as regards the no-
tion of a "region". A more precise and powerful idea is the following crite-
rion where the geometric characteristics of the quality dimensions are
utilized to introduce a spatial structure on concepts:
Criterion C: A "natural concept" is a convex region ofa conceptual space
A convex region is characterized by the criterion that for very pair of points
v1 and v2 in the region all points in between v1 and v2 are also in the re-
gion. The motivation for the criterion is that if some objects which are
located at v1 and v2 in relation to some quality dimension (or several di-
mensions) both are examples of the concept C, then any object that is lo-
cated between v1 and v2 on the quality dimension(s) will also be an exam-
ple of C. Criterion C presumes that the notion of betweenness is
meaningful for the relevant quality dimensions. This is, however, a rather
weak assumption which demands very little of the underlying dimensional
structure.
Most concepts expressed by basic words in natural languages are natu-
ral concepts in the sense specified here. For instance, I conjecture that all
colour terms in natural languages express natural concepts with respect to
the psychological representation of the three colour dimensions. In other
words, the conjecture predicts that if some object 01 is described by the
colour term C in a given language and another object 02 is also said to have
colour C, then any object 03 with a colour that lies between the colour of
01 and that of 02 will also be described by the colour term C. It is well-
known that different languages carve up the colour circle in different ways,
but all carvings seem to be done in terms of convex sets. Strong support for
this conjecture has been presented by Sivik and Taft (1994). Their study
can be seen as a follow-up of the investigations of basic color terms by
Berlin and Kay (1969) who compared and systematized color terms from a
wide variety of languages. Sivik and Taft (1994) focused on Swedish color
terms, while Taft and Sivik (1997) compared color terms from Swedish,
Polish, Spanish and American English. On the other hand, the reference of
an artificial colour term like "grue" (Goodman 1955) will not be a convex
region in the ordinary conceptual space and thus it is not a natural concept
according to Criterion C.
l
Another illustration of how the convexity of regions determines con-
cepts and categorizations is the phonetic identification of vowels in various
languages. According to phonetic theory, what determines the quality of a
vowel are the relations between the basic frequency of the sound and its
formants (higher frequencies that are present at the same time). In general,
the first two formants F] and F2 are sufficient to identify a vowel. This
means that the coordinates of two-dimensional space spanned by F] and F2
(in relation to a fixed fundamental frequency F0) can be used as a fairly
accurate description of a vowel. Fairbanks and Grubb (1961) investigated
how people produce and recognize vowels in "General American" speech.
Figure 3 summarizes some of their findings.
As can be seen from the diagram, the preferred, identified and self-
approved examples of different vowels form convex sub-regions of the
space determined by F] and F2 with the given scales? As in the case of
colour terms, different languages carve up the phonetic space in different
ways (the number of vowels identified in different languages varies con-
siderably), but I conjecture again that each vowel in a language will corre-
spond to a convex region of the formant space.
Criterion C provides an account of concepts that satisfies the desidera-
tum, formulated by Stalnaker (1981: 347), that a concept "[...] must be not
just a rule for grouping individuals, but a feature of individuals in virtue of
which they may be grouped". However, it should be emphasized that I only
view the criterion as a necessary but perhaps not sufficient condition on a
natural concept. The criterion delimits the class of concepts that are useful
for cognitive purposes, although it may not be sufficiently restrictive.
1. For an extended analysis of this example, see Gardenfors (1990).
2. A selfapproved vowel is one that was produced by the speaker and later ap-
proved of as an example of the intended kind. An identified sample of a vowel
is one that was correctly identified by 75% of the observers. The preferred
samples of a vowel are those which are "the most representative samples from
among the most readily identified samples" (Fairbanks and Grubb 1961: 210).
3K
2KI---
2
IK
500
200 250 500 IK
Figure 3. The vowel space of American English (from Fairbanks and Grubb
1961). The scale of the abscissa and ordinate are the logarithm of the
frequencies of F] and F2 (the basic frequency of the vowels was 130
cps).
4. Relations to prototype theory
Describing concepts as convex regions of conceptual spaces fits very well
with the so called prototype theory of categorization developed by Rosch
and her collaborators (Rosch 1975, 1978; Mervis and Rosch 1981; see also
Lakoff 1987). The main idea of prototype theory is that within a category
of objects, like those instantiating a concept, certain members are judged to
be more representative of the category than others. For example, robins are
judged to be more representative of the category "bird" than are ravens,
penguins and emus; and desk chairs are more typical instances of the cate-
gory "chair" than rocking chairs, deck-chairs, and beanbag chairs. The
most representative members of a category are called prototypical mem-
bers. It is well-known that some concepts, like "red" and "bald" have no
sharp boundaries and for these it is perhaps not surprising that one finds
prototypical effects. However, these effects have been found for most con-
cepts including those with comparatively clear boundaries like "bird" and
"chair".
In traditional philosophical analyses of concepts based on truth-
conditions or functions from possible worlds to extensions (Montague
1974), it is very difficult to explain such prototype effects (see Gardenfors
2000, section 3.3).3 Either an object is a member of the class assigned to a
concept (relative to a given possible world) or it is not and all members of
the class have equal status as category members. Rosch's research has been
aimed at showing asymmetries among category members and asymmetric
structures within categories. Since the traditional definition of a concept
neither predicts nor explains such asymmetries, something else must be
gOIng on.
In contrast, if concepts are described as convex regions of a conceptual
space, prototype effects are indeed to be expected. In a convex region, one
can describe positions as being more or less central. For example, if colour
concepts are identified with convex subsets of the colour space, the central
points of these regions would be the most prototypical examples of the
colour. In a series of experiments, Rosch has been able to demonstrate the
psychological reality of such "focal" colours. For another illustration, we
can return to the categorization of vowels presented in the previous section.
Here the subjects' different kinds of responses show clear prototype ef-
fects.
For more complex categories like "bird" it is perhaps more difficult to
describe the underlying conceptual space. However, if something like Marr
and Nishihara's (1978) analysis of shapes is adopted, we can begin to see
how such a space would appear.
4
Their scheme for describing biological
forms uses hierarchies of cylinder-like modelling primitives. Each cylinder
is described by two coordinates (length and width). Cylinders are com-
bined by determining the angle between the dominating cylinder and the
3. Indeed, the approach to semantics in truth-functional semantics is anti-
psychological in the sense that the goal is to provide an analysis of the meaning
of words and sentences that is independent of human cognition.
4. This analysis is expanded in Marr (1982, Ch. 5). A related model, together with
some psychological grounding, is presented by Biederman (1987).
added one (two polar coordinates) and the position of the added cylinder in
relation to the dominating one (two coordinates). The details of the repre-
sentation are not important in the present context, but it is worth noting that
on each level of the hierarchy an object is described by a comparatively
small number of coordinates based on lengths and angles. Hence, the ob-
ject can be identified as a hierarchically structured vector in a (higher or-
der) conceptual space. Figure 4 provides an illustration of the hierarchical
structure of their representations. It should be noticed that this representa-
tion of animal concepts is purely shape-based. Animal concepts depend on
many other domains, some of which may be of the functional character that
will be analysed in Section 6.
..------_._-_.__ _ _--_.
cylioder
....--..-------------_ __-_ .
cr.{1_""".
'{ i 1\,
......1]"J
ill U
limb
thick
limb
n
f

jj
U
thIn
limb
biped
human
ape
bird

<"l'Strich
dove
..................................................._-....__.......-.._----------'
Figure 4. A flIst-order approximation of shape space (from Marr and Nishihara
1978).
Even if different members of a category are judged to be more or less pro-
totypical, it does not follow that some of the existing objects must repre-
sent "the prototype". If a concept is viewed as a convex region of a con-
ceptual space, this is easily explained, since the central member of the
region (if unique) is a possible individual in the sense discussed above (if
all its dimensions are specified) although it need not be among the existing
members of the category. Such a prototype point in the region need not be
completely described, but is normally represented as a partial vector, where
only the values of the dimensions that are relevant to the concept have been
determined. For example, the general shape of the prototypical bird would
be included in the vector, while its colour or age would presumably not.
It is possible to argue in the converse direction too and show that if
prototype theory is adopted, then the representation of concepts as convex
regions is to be expected. Assume that some quality dimensions of a con-
ceptual space are given, for example, the dimensions of colour space de-
scribed above, and that we want to partition it into a number of categories,
for example, colour categories. If we start from a set of prototypes P1, ... ,
Pn of the categories, for example, the focal colours, then these should be
the central points in the categories they represent. One way of using this
information is to assume that for every point P in the space one can meas-
ure the distance from P to each of the Pi's, that is, that the space is metric.
If we now stipulate that p belongs to the same category as the closest pro-
totype Pb it can be shown that this rule will generate a partitioning of the
space that consists of convex areas (convexity is here defined in terms of
an assumed distance measure). This is the so-called Voronoi tessellation, a
two-dimensional example of which is illustrated in Figure 5.
Thus, assuming that a metric is defined on the subspace that is subject
to categorization, by this method a set of prototypes will generate a unique
partitioning of the subspace into convex regions. Hence there is an intimate
link between prototype theory and the proposed analysis where concepts
are described as convex regions in a conceptual space.
Figure 5. Voronoi tessellation based on six prototypes.
5. Representing actions by forces
So far, the examples have all been of a static nature where the properties
modelled are not dependent on the time dimension. However, it is obvious
that a considerable part of our cognitive representations concern dynamic
properties (see, for example, van Gelder 1995; Port and van Gelder 1995).5
If we, for the moment, consider what is represented in natural languages,
verbs normally express dynamic properties of objects, in particular actions.
Such dynamic properties can also be judged with respect to similarities:
"walking" is more similar to "running" than to "throwing".
An important question is how the meaning of such verbs can be ex-
pressed with the aid of conceptual spaces. One idea comes from Marr and
Vaina (1982), who extend Marr and Nishihara's (1978) cylinder models to
an analysis of actions. In Marr and Vaina's model an action is described
via differential equations for movements of the body parts of, say, a walk-
ing human (see Figure 6).6
5. To be accurate, van Gelder and his affiliates would avoid using the notion of
representation since they associate this with the symbolic approach to cognition.
See also the discussion in Johnson and Rohrer (this volume).
6. More precisely, Marr and Vaina (1982) only use differential inequalities, for
example, expressing that the derivative of the position of the upper part of the
right leg is positive in the forward direction during a particular phase of the
walking cycle.
180 Peter Gardenfors
HUMAN
A1
l-lJ
RIGHT LEG LEFT LEG
......
t<O
Figure 6. "Walking" represented by cylinder figures and differential equations
(fromMarr and Vaina 1982).
Applying Newtonian mechanics, it is clear that these equations can be de-
rived from the forces that are applied to the legs, arms, and other moving
parts of the body. Even though our cognition may not be built precisely for
Newtonian mechanics, it appears that our brains have evolved the capacity
for extracting the forces that lie behind different kinds of movements and
action (see below). In accordance with this, I submit that the fundamental
cognitive representation of an action consists of the pattern offorces that
generates it. However, it should be emphasized that the "forces" repre-
sented by the brain are psychological constructs and not the scientific di-
mension introduced by Newton. The patterns of forces are thus embodied
and they can be seen as a form of "mimetic schemas" as discussed by
Zlatev (this volume). Such patterns can be represented in principally the
same way as the patterns of shapes are described above. For example, the
force pattern involved in movements when somebody runs is different from
the pattern of a person walking; and the force pattern for saluting is differ-
ent from that of throwing (Vaina and Bennour 1985).
There is, so far, not very much direct empirical evidence for this repre-
sentational hypothesis. However, one interesting example comes from
phonetics.
7
Fujisaki (1992) has developed a theory of how the fundamental
frequency F0 in speech is generated. He treats the F0 contour as generated
from a linear superposition of two force dimensions that are called phrase
and accent commands. The phrase command acts over the intonation
phrase, shaped as an initial rise followed by a long fall to an asymptote
line. This is generated by a phrase control mechanism, activated by a pulse
command with varying magnitude (see Figure 7). The accent command is a
local peak on an accented syllable, generated by the accent control mecha-
nism. The two force dimensions are implemented as muscular control of
the larynx. On this approach, speech is analysed as a special form of ac-
tion. In the left part of Figure 7, the two force dimensions are represented
on a time scale, where the spurts on the phrase command and accent com-
mand dimensions result in the F0 curve represented in the right part of the
figure.
J __
:InF.(t)
..t
FUNOAt",1ENTAt.
FREOUENCY
CONTOUR
Figure 7. Functional model based on two force dimensions for generating the
F0 contour (from Fujisaki and 0000 1996).
Another indirect source of empirical support for the representational hy-
pothesis comes from psychophysics. During the 1950's, the Uppsala psy-
chologist Gunnar Johansson developed a patch-light technique for analys-
ing biological motion without any direct shape information.
8
He attached
light bulbs to the joints of actors that were dressed in black and moved in a
black room. The actors were filmed while performing various actions, such
7. I wish to thank Lauri Carlson for directing me to this theory.
8. For a survey of the research, see Johansson (1973).
as walking, running or dancing. From the films, where only the light dots
could be seen, subjects could within tenths of a second recognize the ac-
tion. Furthermore, the movements of the dots were immediately interpreted
as coming from a human being. Later experiments by Runesson and Fryk-
holm (1981, 1983) have shown that subjects can extract subtle details of
the action, such as the gender of walkers or the weight of lifted objects
(where the objects were not seen on the movies).
One lesson that can be learned from the experiments by Johansson and
his followers is that the kinematics of a movement contains sufficient in-
formation for identifying the underlying dynamic force patterns. Runesson
(1994: 386-387) claims that we can directly perceive the forces that con-
trol different kinds of motion. He argues that one need not make any dis-
tinction between visible and hidden properties:
The fact is that we can see the weight of an object handled by a person. The
fundamental reason we are able to do so is exactly the same as for seeing the
size and shape of the person's nose or the colour of his shirt in normal illu-
mination, namely that information about all these properties is available in
the optic array.
According to his perspective, the information that our senses, primarily
vision, receive about the movements of an object or an individual is suffi-
cient for our brains to be able to extract, with great precision, the underly-
ing forces. Furthermore, the process is automatic - we cannot help but see
the forces. Of course, the perception of forces is not perfect - we are prone
to illusions, just as we are in all types of perception. He formulates this as a
principle of kinematic specification of dynamics (the KSD-principle) that
says that the kinematics of a movement contains sufficient information to
identify the underlying dynamic force patterns.
9
It goes without saying that this principle accords well with the repre-
sentation of actions that I have proposed here. One difference is that Run-
esson has a Gibsonian perspective on the perceptual information available,
which means that he would find it methodologically unnecessary to con-
sider mental constructions such as conceptual spaces. The Gibsonian per-
spective means that the world itself contains sufficient information about
9. In contrast to humans, recent results of causal reasoning in apes and monkeys
indicate that non-human primates often fail to understand the hidden causes, in
particular forces, behind certain effects (Povinelli 2000). There seems to be a
paucity of research on force perception and how forces affect how we catego-
rize actions.
objects and events so that the brain can just "pick up" that information in
order to categorise the entity. According to this perspective, mental repre-
sentations are thus not needed. However, I will here not develop the con-
trasts between the representational and Gibsonian positions.
Another area where actions and objects show similarities in structure is
in the graded structure of the action concepts. There are good reasons to
believe that actions exhibit many of the prototype effects that Rosch (1975,
1978) has presented for object categories. For example, Hemeren (1997)
demonstrated that there is a strong inverse correlation (r = -.81) between
judgments of most typical actions and reaction time in a WORD-ACTION
verification task. He has also shown that subjects in a free listing task of
words or phrases for actions show clear effects concerning base level vs.
subordinate level concepts (Hemeren 1996). For example, "running" was
more frequent and occurred earlier in the lists than "jogging" and "sprint-
ing" and the same applies to "talking" in relation to subordinates such as
"whispering" and "arguing".
To identify the structure of the action space, similarities between ac-
tions should be investigated. However, this can be done with basically the
same methods as for similarities between objects.
Even though the empirical evidence is still very incomplete, my pro-
posal is that by adding force dimensions to a conceptual space, we obtain
the basic tools for analysing dynamic properties of actions and other
movements. As we shall see below, the forces involved need not only be
physical forces, but they can also be emotional or social forces.
6. The cognitive neuroscience of action space
The distinction between perception and action spaces can to some extent
be correlated with the findings from neuroscience on how visual informa-
tion is handled in the brain. Giese and Poggio (2003) note that there is a
ventral pathway from the visual cortex that handles form recognition and a
corresponding dorsal pathway for motion recognition. These two pathways
operate in parallel. Of special interest in relation to my hypothesis, Giese
and Poggio speculate that in the dorsal motion pathway there exist neurons
(located in the superior temporal sulcus) specialized for motion patterns:
The representation of motion is based on a set of learned patterns. These
patterns are encoded as sequences of "snapshots" of body shapes by neurons
in the form pathway, and by sequences of complex optic flow patterns in the
motion pathway. (Giese and Poggio 2003: 181)
On the surface, Giese and Poggio' s model does not concern dynamics, but
kinematics since they describe a sequence of "snapshots" of a movement.
Better evidence for dynamic representation of motion comes, for example,
from the literature on representational momentum (Freyd and Finke 1984).
In one of the first experiments on this phenomenon, Freyd and Finke
showed subjects a rectangle at three positions in a possible path of orienta-
tion. Subjects were told to remember the third orientation and were then
presented with a rectangle at a fourth position that was either rotated
slightly less, or exactly the same, or slightly more than the remembered
triangle (see Figure 8 A). Subjects found it more difficult to detect differ-
ences in the direction of the implicit motion of the sequence of rectangles.
This suggests that their mental representations of the rectangles induced
I
VIV
Figure 8.
2
Two experiments on representational momentum (from Freyd and
Finke 1984: 128)
a certain "momentum" that influenced their memory of the third triangle.
This effect disappeared when the ordering of the two first rectangles was
reversed so that the subjects could no longer perceive a path of motion (see
Figure 8 B).
Along the same lines, Kourtzi and Kanwisher (2000) showed subjects
photos of situations that contained dynamic information. In an fMRI study,
they found greater activity in the medial temporal/medial superior temporal
region of cortex compared to when subjects were viewing photos with no
implied motion. The medial temporal region is one of major brain areas
engaged in analysis of visual motion. These glimpses from the cognitive
neuroscience of action representations indicate how the brain projects
forces, even when the stimuli do not contain any motion. This is a side of
"embodiment" that merits further investigation. By combining experiments
from cognitive psychology with different kinds of brain imaging, we may
hope to acquire the empirical results needed for testing a more elaborate
theory of the structure of action space.
7. Representing functional properties in action space
Another large class of properties that cannot be analysed in terms of per-
ceptual dimensions in a conceptual space are the functional properties that
are often used for characterizing artefacts. A nice description of the role of
functional properties comes from Paul Auster's novel City of Glass (1992:
77):
Not only is an umbrella a thing, it is a thing that performs a function - in
other words, expresses the will of man. When you stop to think of it, every
object is similar to the umbrella, in that it serves a function. A pencil is for
writing, a shoe is for wearing, a car for driving.
Now my question is this. What happens when a thing no longer performs its
functions? Is it still the thing, or has it become something else? When you
rip the cloth off the umbrella, is the umbrella still an umbrella? You open
the spokes, put them over your head walk out into the rain, and you get
drenched. Is it possible to go on calling this object an umbrella? In general,
people do. At the very limit, they will say the umbrella is broken. To me this
is a serious error, the source of all our troubles.
In agreement with Auster's intuition, Vaina (1983) notes that when decid-
ing whether an object is a "chair", the perceptual dimensions of the object,
like those of shape, colour, texture and weight, are largely irrelevant, or at
least extremely variable. Since I have focused on such dimensions in my
description of conceptual spaces, the analysis of functional properties
seems to be an enigma for my theory.
I propose to analyse these properties by reducing them to the actions
that the objects "afford". To continue with the example, a chair is
prototypically an object that affords back-supported sitting for one person,
that is, an object that contains a flat surface at a reasonable height from the
ground and another flat surface that supports the back. In support of this
analysis, Vaina (1983: 28) writes: "[T]he requirement for efficient use of
objects in actions induces strong constraints on the form of representation.
Each object must first be categorized in several ways, governed ultimately
by the range of actions in which it can be become involved."
The notion of "affordance" is borrowed from Gibson's (1979) theory of
perception. 10 However, he interprets the notion realistically, i.e. as inde-
pendent of the viewer, while for me the affordances are always identified
in relation to a conceptual space, which means that I interpret "affordance"
from a cognitivist representational perspective.
In more general terms, I propose that function concepts be interpreted in
terms of an action space. This is in contrast to the perceptual dimensions
that I have presented in my earlier examples in this chapter. To be more
precise, I put forward the following special case of Criterion C:
Functional properties are convex regions in action space.
The actions involved in the analysis of a functional property may then, in
turn, be reduced to force dynamic patterns as was explained above. This is
accomplished by representing a functional property as a vector in a high-
dimensional space where most dimensions are constituted of the force di-
mensions of the action space. In this sense, the functional space is super-
venient on the action space. Functional properties are thus "higher order
properties" in the sense of Gardenfors (2000, Section 3.10). The main
problem with this proposal is that we know even less about the geometry
and topology of how humans (and animals) structure action space than we
know about how they structure shape space. This is an area where further
research is badly needed.
11
10. However, as Costall (this volume) notes, Gibson's characterization of "affor-
dance" changed over the years.
11. Within robotics, Chella, Gaglio and Pirrone (2001) use Fourier transforms of
motions to represent the movements of objects and of a robot. This solution
makes sense from an implementational point of view, but it is uncertain whether
the brain uses anything like this to represent actions.
The upshot of the proposal is that, even if this road of analysis is long
and to a large extent unexplored, in principle, functional properties can be
explained in terms of more basic dimensions such as forces.
8. The embodiment of forces
In the tradition of Cognitive Semantics, the meanings of expressions have
been analysed in semi-geometrical constructs called image schemas. In
earlier writings, I have shown how these image schemas can be given a
more precise description in terms of conceptual spaces (Gardenfors 1996).
For Cognitive Semantics too, the focus has been on the spatial structure of
the image schema (the very term "image" schema indicates this). Lakoff
(1987: 283) goes as far as putting forward what he calls the "spatialization
of form hypothesis" which says that the meanings of linguistic expressions
should be analyzed in terms of spatial image schemas plus metaphorical
mappings.
However, there are exceptions to this emphasis on spatial structure. One
researcher who at a very early stage brought forward the role of forces in
cognitive semantics is Johnson (1987). He argues that forces form percep-
tual Gestalts that serve as image schemas (even though the word "image"
may be misleading here). He writes:
Because force is everywhere, we tend to take it for granted and to overlook
the nature of its operation. We easily forget that our bodies are clusters of
forces and that every event of which we are a part consists, minimally, of
forces in interaction. [... ] We do notice such forces when they are extraordi-
narily strong, or when they are not balanced off by other forces. (Johnson
1987: 42)
Johnson presents a number of "preconceptual Gestalts" for force. These
Gestalts function as the correspondences to image schemas but with forces
as basic organizing features rather than spatial relations. The force Gestalts
he presents are "compulsion", "blockage", "counterforce", "diversion",
"removal of restraint", "enablement" and "attraction" (Johnson 1987: 45-
48).
Another early exception is Talmy (1988), who emphasizes the role of
forces and dynamic pattern in image schemas in what he calls "force dy-
namics". He develops a schematic formalism that, for example, allows him
to represent the difference in force patterns in expressions like "The ball
kept rolling because of the wind blowing on it" and "The ball kept rolling
despite the stiff grass".
Talmy's dynamic ontology consists of two directed forces of unequal
strength, the focal called "Agonist" and the opposing element called "An-
tagonist", each force having an intrinsic tendency towards either action or
rest, and a resultant of the force interaction, which is either action or rest.
All of the interrelated factors in any force-dynamic pattern are necessarily
copresent wherever that pattern is involved. But a sentence expressing that
pattern can pick out different subsets of the factors for explicit reference -
leaving the remainder unmentioned - and to these factors it can assign dif-
ferent syntactic roles within alternative constructions. (Talmy1988: 61)
Despite these exceptions, it appears that the role of forces has been under-
rated within Cognitive Semantics. In Piaget's theory of sensory-motor
schemas, developed for modelling cognitive development and not seman-
tics, motor patterns are central. These can be seen as a special case of the
dynamic patterns that form our fundamental understanding of the world. I
would suggest that many ideas from the schemas of developmental psy-
chologists can fruitfully be incorporated in the construction used by cogni-
tive semanticists.
Analysing the use of forces in Cognitive Semantics has led me to an
ambiguity in the very notion of "force". In academic circles, Newtonian
physics has become a role model for science; and when we speak of
"force" it is natural to think of and represent them as Newtonian forces -
as force vectors in a conceptual space. But when it comes to everyday hu-
man thinking, it is important to distinguish between a first-person (phe-
nomenological) and a third-person perspective of forces.
From the first-person perspective, it is the forces that act directly on you
that are considered. These "forces" are not just the physical Newtonian
forces, but more importantly also the social or emotional forces that affect
you. It is perhaps more appropriate to call forces seen from a first-person
perspective "powers". First-person powers are experienced either as physi-
cal forces or as emotional or social pressures that make you move in a par-
ticular direction.
From the third-person perspective, one sees forces acting upon an object
from the outside, so in this case you don't experience the forces directly,
but your perceptual mechanisms derive them. Therefore such forces are not
embodied in the same way as in the first-person perspective. From the first-
person perspective, powers act directly on you, while from the third-person
perspective forces act at a distance (pace Newton). 12
One reason for why this distinction is seldom made is that we are ex-
tremely good at perceiving forces acting upon other objects.
I3
As we have
seen in Section 5, the Uppsala school of psychology claims that we can
directly perceive the forces that control different kinds of motion. Accord-
ing to their Gibsonian perspective, information about the movements of an
object is sufficient for our brains to extract the underlying forces.
The importance of this distinction is that our understanding of the third-
person perspective presumably derives from the first-person person per-
spective. (This is one reason why Newton had such problems in convincing
his contemporaries about forces acting at a distance). If this is the case,
then the meanings of words such as push and pull that are based on first-
person powers should be seen as cognitively more fundamental than
meanings based on third-person forces. In other words, my hypothesis is
that the meanings of the force elements of image schemas are grounded in
the actual experience of forces on one's own body.
There is much in Johnson's (1987) book that implicitly points to the
centrality of the first-person "power" perspective. For one thing, he focuses
on the role of interaction: "[F]orce is always experienced through interac-
tion. We become aware of a force as it affects us or some object in our
perceptual field" (Johnson 1987: 43). Interaction is primarily seen from a
first-person perspective, while forces are abstractions that are seen from a
third-person view. Then, in his description of the "enablement" Gestalt or
schema, he explicitly focuses on first-person "powers":
If you choose to focus on your acts of manipulation and movement, you can
become aware of a felt sense of power (or lack of power) to perform some
action. You can sense that you have the power to pick up the baby, the gro-
ceries, and the broom but not to lift the front end of your car. While there is
no actualized force vector here, it is legitimate to include this structure of
possibility in our common gestalts for force, since there are potential force
12. There is also a second-person perspective where the subject can "put himself in
the shoes of the other". This perspective is what is involved in empathy, joint
attention and other aspects of a "theory of mind" (see Gardenfors 2003, ch. 4).
Some researchers put forward "mirror neurons" as a possible mechanism behind
this perspective (e.g. Rizolatti and Arbib 1998; Gallese 2000)
13. However, it seems that other animal species may not have this capacity to the
same extent (Povinelli 2000).
vectors present and there is a definite "directedness" (or potential part of
motion) present. (Johnson 1987: 47)
In contrast to Johnson and Talmy, I view social power relations as semanti-
cally fundamental, and physical forces that act at a distance from the sub-
ject as derived. For example, Winter and Gardenfors (1995) and Garden-
fors (1998) argue that the meanings of modal verbs are based on social
power rather than physical force. Even Talmy (1988: 79) concedes that "[a]
notable semantic characteristic of the modals in their basic usage is that
they mostly refer to an Agonist that is sentient and to an interaction that is
psychosocial, rather than physical, as a quick review can show". I com-
pletely agree, but see this as an argument for the primary meaning of the
modals being determined by social power relations, while the (few) uses of
modals in the context of physical forces are derived meanings.
In a sense, the focus on social power relations makes the conceptual
analysis more intricate, because Newtonian force vectors, viewed as natu-
ral representations of the third person forces, may not be entirely appropri-
ate to represent the emotional and social aspects of power. Again, more
empirical investigations of how human subjects mentally conceive of these
powers will be needed.
9. Conclusion
The main purpose of this chapter has been to outline an extension of the
theory of conceptual spaces to actions and functional properties. In the first
part, I have provided an analysis of concepts with the aid of the notion of
conceptual spaces. A key notion is that of a natural concept which is de-
fined in terms of convex regions of conceptual spaces - a definition that
crucially involves the geometrical structure of the various domains.
As a complement to the perceptual dimensions treated in Gardenfors
(2000), I have in the latter part of the chapter focused on "action space". I
submit that action space can, in principle, be analysed in the same way as
e.g. colour space or shape space. Admittedly, this will take extensive psy-
chological experimentation to establish. The core hypothesis is that our
categorization of actions to build on our perception of forces (which, in-
deed, seems to be perceptions). The hypothesis is that the cognitive repre-
sentation of an action can be described as a spatio-temporal pattern of
forces. I have argued that functional properties "live on" action space.
When it comes to functional properties, the key idea is that the function of
an object can be analysed with the aid of the actions it affords. An empiri-
cally testable prediction is that functional concepts can be described as
convex regions in an appropriate action space. However, there is, so far,
not much empirical support for the prediction. Consequently, it must be left
as a research programme for the time being.
I also believe that conceptual spaces in general and their application to
force dimension in particular can be a useful tool to sharpen Cognitive
Semantics. With the aid of the topological and geometric structure of the
various quality dimensions, one can obtain a more precise foundation for
the concept of image schemas that form the core of the theories of e.g.
Lakoff (1987), Johnson (1987) and Langacker (1987). I have emphasized
the role of forces in image schemas and argued that the first-person per-
spective on forces is more fundamental than the third-person perspective. I
believe that this distinction could also be fruitfully applied within other
areas of cognitive semantics.
Acknowledgements
An early version of this chapter was written while the author was a fellow
at the Swedish Collegium for Advanced Study in the Social Sciences
(SCASSS) in Uppsala. I want to thank the Collegium for providing me with
excellent working conditions. I also want to thank Paul Hemeren, Martin
Raubal, Tom Ziemke, Jordan Zlatev and an anonymous referee for very
helpful comments.
References
Auster, Paul
1992 The New York Trilogy: City of Glass, Ghosts, The Locked Room.
London / Boston: Faber and Faber.
Berlin, Brent and Kay Paul
1969 Basic Color Terms: Their Universality and Evolution. Berkeley, CA:
University of California Press.
Biederman,lrving
1987 Recognition-by-components: A theory of human image understand-
ing. Psychological Review 94: 115-147.
Chella, Antonio, Salvatore Gaglio and Roberto Pirrone
2001 Conceptual representations of actions for autonomous robots. Ro-
botics and Autonomous Systems 34: 251-263.
Clark, Austen
1993 Sensory Qualities. Oxford: Clarendon Press.
Costall, Alan
this vol. Bringing the body back to life: James Gibson's ecology of language.
Fairbanks, Grant and Patti Grubb
1961 A psychophysical investigation of vowel formants. Journal ofSpeech
and Hearing Research 4: 203-219.
Fodor, Jerry A.
1975 The Language of Thought. Cambridge, MA: Harvard University
Press.
Freyd, Jennifer J. and Ronald Finke
1984 Representational momentum. Journal of Experimental Psychology:
Learning, Memory and Cognition 10: 126-132.
Fujisaki, Hiroya
1992 Modeling the process of fundamental frequency contour generation.
In: Y. Tohkura, E. Vatikiotis-Bateson and Y. Sagisaka (eds.), Speech
Perception, Production and Linguistic Structure, 313-326. Amster-
dam: IOS Press.
Fujisaki, Hiroya and Sumio 0000
1996 Prosodic parameterization of spoken Japanese based on a model of
the generation process of FO contours. Proceedings of 1996 Interna-
tional Conference on Spoken Language Processing, 4: 2439-2442.
Philadelphia, PA.
Gallese Vittorio
2000 The inner sense of action: Agency and motor representations. Jour-
nal ofConsciousness Studies 7 (10): 23-40.
Gardenfors, Peter
1990 Induction, conceptual spaces and AI. Philosophy of Science 57: 78-
95.
1996 Conceptual spaces as a basis for cognitive semantics. In: Andy Clark,
Jesus Ezquerro and Jesus M. Larrazabal (eds.), Philosophy and Cog-
nitive Science, 159-180. Dordrecht: Kluwer.
1998 The pragmatic role of modality in natural language. In: Paul Wein-
gartner, Georg Schurz and Georg Dom (eds.), The Role ofPragmat-
ics in Contemporary Philosophy, 78-91. Vienna: Holder-Pichler-
Tempsky.
2000 Conceptual Spaces: The Geometry of Thought. Cambridge, MA:
MIT Press.
Gibson, James J.
1979 The Ecological Approach to Visual Perception Hillsdale, NJ: Law-
rence ErlbaumAssociates.
Giese, Martin A. and Tomaso Poggio
2003 Neural mechanisms for the recognition of biological movements.
Nature Reviews Neuroscience 4: 179-192.
Goldstone, Robert L. and Lawrence W. Barsalou
1998 Reuniting perception and conception, Cognition 65: 231-262.
Goodman, Nelson
1955 Fact, Fiction and Forecast. Cambridge, MA: Harvard University
Press.
Hemeren, Paul E.
1996 Frequency, ordinal position and semantic distance as measures of
cross-cultural stability and hierarchies for action verbs. Acta Psy-
chologica 91: 39-66.
1997 Typicality and context effects in action categories. Proceedings of
the 19
th
Annual Conference of the Cognitive Science Society, 949.
Stanford, CA: Lawrence Erlbaum Associates.
Johansson, Gunnar
1973 Visual perception of biological motion and a model for its analysis.
Perception and Psychophysics 14: 201-211.
Johnson, Mark
1987 The Body in the Mind: The Bodily Basis of Cognition. Chicago, IL:
University of Chicago Press,
Kourtzi, Zoe and Nancy Kanwisher
2000 Activation in human MT/MST by static images with implied motion.
Journal ofCognitive Neuroscience 12 (1): 48-55.
Lakoff, George
1987 Women, Fire and Dangerous Things. Chicago, IL: The University of
Chicago Press.
Langacker, Ronald W.
1987 Foundations of Cognitive Grammar. Vol. I: Theoretical Prerequi-
sites. Stanford: Stanford University Press.
Mandler, Jean
2004 Foundations ofthe Mind. Oxford: Oxford University Press.
Marr, David
1982 Vision. San Francisco, CA: Freeman.
Marr, David and H. Keith Nishihara
1978 Representation and recognition of the spatial organization of three-
dimensional shapes. Proceedings of the Royal Society in London, B
200, 269-294.
Marr, David and Lucia Vaina
1982 Representation and recognition of the movements of shapes. Pro-
ceedings ofthe Royal Society in London, B214: 501-524.
Mervis, Carolyn and Eleanor Rosch
1981 Categorization of natural objects. Annual Review of Psychology 32:
89-115.
Montague,Richard
1974 Formal Philosophy: Selected Papers of Richard Montague. In:
Richmond H. Thomason (ed.), New Haven, CT: Yale University
Press.
Nelson, Katherine
1986 Event Knowledge: Structure and Function in Development.
Hillsdale, NJ: Lawrence ErlbaumAssociates.
Pinker, Steven
1997 How the Mind Works. New York, NY: W. W. Norton and Co.
Port, Robert and Timothy van Gelder (eds.)
1995 Mind as Motion. Cambridge, MA: MIT Press.
Povinelli, Daniel
2000 Folk Physics for Apes: The Chimpanzee's Theory of how the World
Works. Oxford: Oxford University Press.
Pylyshyn, Zenon
1984 Computation and Cognition. Cambridge, MA: MIT Press.
Quinlan, Philip
1991 Connectionism and Psychology: A Psychological Perspective on
New Connectionist Research. New York, NY: Harvester Wheatsheaf.
Rizolatti, Giacomo and Michael Arbib
1998 Language within our grasp. Trends in Neuroscience 21: 188-194.
Rumelhart, David. E. and James L. McClelland
1986 Parallel Distributed Processing, Vols. 1 and 2. Cambridge, MA:
MIT Press.
Rosch, Eleanor
1975 Cognitive representations of semantic categories. Journal ofExperi-
mental Psychology: General 104: 192-233.
1978 Prototype classification and logical classification: The two systems.
In: Ellin Scholnik (ed.), New Trends in Cognitive Representation:
Challenges to Piaget's Theory, 73-86. Hillsdale, NJ: Lawrence Erl-
baumAssociates.
Runesson, Sverker
1994 Perception of biological motion: The KSD-principle and the impli-
cations of a distal versus proximal approach. In: Gunnar Jansson,
Sten-Sture Bergstrom, and William Epstein (eds.), Perceiving Events
and Objects, 383-405. Hillsdale, NJ: Lawrence ErlbaumAssociates.
Runesson, Sverker and Gunnar Frykholm
1981 Visual perception of lifted weights. Journal of Experimental Psy-
chology: Human Perception and Performance 7: 733-740.
1983 Kinematic specification of dynamics as an informational basis for
person and action perception. Expectation, gender recognition, and
deceptive intention. Journal of Experimental Psychology: General
112: 585-615.
Schiffman, Harvey Richard
1982 Sensation and Perception, 2nd edition. New York, NY: John WHey
and Sons.
Sivik, Lars and Charles Taft
1994 Color naming: A mapping in the NCS of common color terms. Scan-
dinavian Journal ofPsychology 35: 144-164.
Smith, Linda
1989 From global similarities to kinds of similarities - the construction of
dimensions in development. In: Stella Vosniadou and Andrew Ortony
(eds.), Similarity and Analogical Reasoning, 146-178. Cambridge:
Stalnaker, Robert
1981 Antiessentialism. Midwest Studies ofPhilosophy 4: 343-355.
Talmy, Leonard
1988 Force dynamics in language and cognition. Cognitive Science 12:
49-100.
Vaina, Lucia
1983 From shapes and movements to objects and actions. Synthese 54: 3-
36.
Vaina, Lucia and YossufBennour
1985 A computational approach to visual recognition of arm movement.
Perceptual and Motor Skills 60: 203-228.
Van Gelder, Timothy
1995 What might cognition be if not computation? Journal ofPhilosophy
92: 345-381.
Winter, Simon and Peter Gardenfors
1995 Linguistic modality as expressions of social power. Nordic Journal
ofLinguistics 18: 137-166.
From pre-representational cognition to language
Takashi Ikegami and Jordan Zlatev
Abstract
In this chapter we argue for a number of qualitative differences in human cognition:
interaction vs. representation, procedural vs. declarative knowledge, dynamical
categories vs. concepts, synaesthesia vs. language, the affecto-imagistic dimension,
necessary for characterizing the meaning of Japanese mimetics, vs. the analytical
dimension (Kita 1997, 2001). Such emphasis is necessary because there is persis-
tent tendency in embodiment theories to "resolve" such oppositions by ignoring the
differences, and thus, in effect, reducing or eliminating the second and more "dis-
embodied" side of the oppositions. At the same time, we explore how structures
and processes of pre-representational cognition such as dynamical categories, inter-
nal meanings space, and synaesthesia can play a role in the "grounding" of mental
representations (concepts) and language.
Keywords: active perception, complex systems, dynamic categorization, internal
meaning space, Japanese mimetics, language, representation, synaesthesia.
1. Introduction
The issue of the nature and role of representations in cognitive science is a
heavily contested one, with e.g. Johnson and Rohrer (this volume) arguing
for a non-representational account of cognition, including language, while
e.g. Gardenfors (this volume) claims that his theory of "conceptual spaces"
is fundamentally representational.
While mental representation was the fundamental concept of "classi-
cal" cognitive science (e.g. Fodor 1981), along with that of computation,
the 1990s witnessed the rise of "second generation" cognitive science (e.g.
Varela, Thompson and Rosch 1991) making heavy use of notions such as
embodiment and interaction and this approach reacted against what was
perceived to be an overextension of the term "representation" to involve
just about any kind of cognitive process. For example, Johnson and Lakoff
198 Takashi Ikegami and Jordan Zlatev
(2002: 249-250) point out that: "As we said in Philosophy in the Flesh, the
only workable theory of representations is one in which a representation is
a flexible pattern of organism-environment interactions, and not some inner
mental entity that somehow gets hooked up with parts of the external world
by a strange relation called 'reference'."
However, there is a serious problem for such interactionist accounts to
the extent that they purport to provide an explanation of language, since
sensorimotor interaction is an inherently non-representational notion (in
any sensible use of the term "representation"), while language is represen-
tational in two different, though related, respects: (a) it has expression-
content structure of signs and (b) statements are about (real or imagined)
states of affairs (see Sonesson this volume; Zlatev this volume).
In this chapter, we hope to clarify this problem, and perhaps even offer
some ingredients to its resolution. The first step involves conceptual analy-
sis, in which we explain how we understand the concept of representation,
and in particular, mental representation. We will distinguish between pre-
representational and representational cognition, and point out their respec-
tive properties. We will suggest that only true representations are properly
regarded as concepts.
In sections 3 and 4, respectively, we will characterize two structures of
pre-representational cognition: dynamical category and internal meaning
space. A dynamical category is an emergent category resulting form senso-
rimotor interaction with the environment. Unlike explicit, "classical" con-
cepts it can not be characterized in terms of necessary and sufficient con-
ditions. In these two respects it resembles two of the central concepts
within Cognitive Linguistics: image schemas, as defined by Johnson
(1987) and Johnson and Rohrer (this volume) and prototypes (Rosch 1973;
Lakoff 1987). However, the concept of dynamical category is most closely
related to Gibson's (1979) ecological psychology (Costall this volume) and
in particular to what is currently known as active perception.
By internal meaning space we mean a cross-modal network of dynami-
cal categories. Similar to its constituent notion of dynamical category, the
notion emphasizes the first-person, or subjective nature of cognition. Sen-
sory input is meaningful for the organism due to its internal (neural) dy-
namics (Freeman 2003), which depend on an intrinsic value system (von
Uexkiill1940 [1982]; Zlatev 2003). At the same time, the notion of internal
meaning space captures the fact that dynamical categories are not inde-
pendent but linked in a "space" of different sensory modalities and dimen-
sions. We will suggest that such a dynamic, internal, value-laden space
From pre-representational cognition to language 199
serves as an essential intermediary between simple, reactive sensory-motor
cognition and representational cognitive functions, including language. It
resembles the "conceptual spaces" of Gardenfors (2000, this volume), but
differs in being dynamic and value-based, and thereby affect-laden. We
will also suggest than an internal meaning space is implied in the phe-
nomenon of synaesthesia (Cytowic 2002).
In section 5, we show how such a pre-representational meaning space
can help account for two aspects of language: synaesthetic metaphors such
as sweet smile, and Japanese mimetics such as suta-suta ('walking hur-
riedly'). We follow the analysis of Kita (1997, 2001), who has argued that
mimetics involve an "affecto-imagistic dimension" that is distinct from the
"analytic dimension" that dominates linguistic meaning. Thus, the mean-
ings of mimetics can be related in more than name to the concept of mi-
metic schemas (Zlatev 2005, this volume), which constitute pre-linguistic
mental representations deriving from bodily imitation. In brief, while we
will argue that language cannot be reduced to structures of pre-
representational cognition, we will also show that such structures may be
necessary for a complete account of it.
The concepts of dynamical category and meaning space may be rather
difficult to grasp, especially if they are only characterized through lan-
guage, which after all is not optimal for talking about pre-representational,
dynamical phenomena with fuzzy boundaries. That is why our approach
will be to use complex systems (Port and van Gelder 1998; Kaneko and
Ikegami 1998; Kaneko and Tsuda 2000) in order to elucidate the nature of
structures of pre-representational cognition. In particular, we will describe
a number of computational simulations involving "artificial creatures"
which "live" in a simulated "environment" and interact with it, whereby
dynamical categories and a pre-conceptual meaning space emerge. To il-
lustrate, we will show how even abstract phenomena such as triangles and
rectangles can be categorized pre-representationally via blind touching, so
that the criterion for their classification is defined by the creatures' styles
of action patterns, rather than by their detached concepts about the figures.
We hasten to note, however, that these experiments should be regarded
as a form of "Weak AI" rather than "Strong AI": there is a radical differ-
ence between simulating cognition and life and duplicating it (Searle 1992;
Ziemke 2001). The "creatures" that are described in this chapter are not
truly animate: they do not have any intrinsic value system, and thus have
no basis for intrinsic intentionality and phenomenal experience (Zlatev
2003). We ask the reader to please remember this, since it will be tedious
to have to point out again and again that when we refer to "creatures" in
connection with the simulations described in Section 3, we are not suc-
cumbing to animism: the attribution of life properties to inanimate matter.
1
Nevertheless, we believe that there are sufficient structural parallels be-
tween the mode of operation of models based on complex systems and real
organisms to make the method justified as an analytical tool for studying
"embodied" cognition.
2. Mental representation
As we understand the concept, a representation is a structure that consists
of three parts: an expression that stands for a content for a given subject. It
is thus identical with the classical definition of a sign (see Emmeche this
volume; Sonesson this volume). A clear example of a representation is a
picture: the depicted apple cannot be eaten, but it represents (in this case
iconically) an apple that can. The painting itself is the expression, and it is
different from, at the same time as it corresponds to, something else.
Whether this "something else" is a real specific apple, a generic apple, or
an imagined apple is not important here: what is important is the expres-
sion-content structure itself. What "connects" the expression and the con-
tent is a process of interpretation: representations do not exist by them-
selves, but only for someone.
This much is fairly uncontroversial. The real controversies begin when
we ask whether there are mental representations, and if so what are they.
"Classical", first-generation cognitive science found them everywhere: in
thought, in language, in perception, in practical action (Gardner 1987).
Some representatives of second generation, "embodied" cognitive science
(Varela, Thompson and Rosch 1991; Johnson and Rohrer this volume)
appear to take the opposite extreme and more or less abolish them.
We believe that these extremes are equally mistaken, and a golden mean
is the best answer, in line with phenomenology (Husserl 1962) and Piaget's
(1945) account of the origin of "symbols" in childhood: to say that a sub-
ject has a mental representation is to say that he can (a) differentiate be-
tween the expression and content, and (b) see the first as corresponding to
the second, pretty much as in the picture example, mentioned above. Son-
1. We are grateful to Goran Sonesson, for helping us emphasize this point and thus
avoid a possible misinterpretation.
esson (1989, this volume) uses this as the major criterion to distinguish
between (true) signs and simple indexicalities (based on contiguity and
factorality, i.e. part-whole) and iconicities (based on resemblance), which
are not signs but only a ground for indexical and iconic signs, respectively.
One may (and usually does) ask: Who is the "someone" doing the differ-
entiation and finding the correspondence? There are three types of an-
swers, only the third of which will do:
(a) An unconscious processor or a "homunculus" in the head; this leads
to infinite regress: we need to account for the ability of the homun-
culus to "see" the expression and content, and "figure out" that the
first stands for the second, and then we need to account for the
mental representations in its head, ad infinitum (Edelman 1992).
(b) The expression is a "symbol" that is associated with a "meaning" for
someone else than the system that is actually using the symbols, as in
a computer. Yes, but then the representation is not intrinsic to the
system, but to the programmer, or whoever else is doing the "inter-
preting" (Searle 1992).
(c) The subject himself (or herself), i.e. the conscious individual who
"owns" a (possibly internalized) mental representation. As when you
close your eyes and imagine an apple: you do not confuse your
imagined apple with the one it represents, you differentiate between
the two. On the other hand, when you see an apple, you do not think
of your perception as a "representation" of an apple: you see the ap-
ple itself. So there are no mental representations in perception, but
only in imagination (Piaget 1945; see also Zlatev this volume; Son-
esson this volume).
Mental representations, as here defined, make an irreducible reference to
consciousness. The notion of consciousness is, of course, vexed with even
more riddles than that of representation, but since it has become again a
(scientifically) respectable topic over the last 20 years,2 considerable ad-
vances have been made in both philosophical discussions, e.g. of the "hard
problem" of the irreducibility of qualitative experience (Chalmers 1997), in
distinguishing between different kinds of consciousness, e.g. affective from
2. As testified by a number of journals including Journal of Consciousness Stud-
ies, Consciousness and Cognition, and PSYCHE: An Interdisciplinary Journal
of Research on Consciousness and annual conferences such as Towards a Sci-
ence ofConsciousness (TSC).
reflective consciousness, as well as in understanding the neural underpin-
nings of this admittedly rather elusive phenomenon (e.g. Edelman 1992;
Damasio 2000). Mental representations involve both types of conscious-
ness mentioned above: they are reflective, since they can be accessed and
thought about independently of whatever they represent, but they are also
affective since they have what phenomenologists call a "affective tone"
(Thompson 2001), a particular "flavor" (to use a metaphor), due to the
intrinsic value system of the subject as a living being (Zlatev 2003). The
"flavor" of my imagined apple is different from that of an imagined rotten
banana.
Finally, since mental representations can be (in principle) accessed,
unless they are "repressed" due to Freudian or other reasons (Searle 1992),
they can be themselves represented, or expressed - in language, pictures,
gestures or some other external medium (more or less accurately). That is
why they can be said to constitute our declarative knowledge, as opposed
to procedural skills such as bicycle-riding, which are not based on mental
representations (contrary to the claims of first-generation cognitivists). The
latter distinction is made clear by Mandler (2004), who consistently distin-
guishes between the two sorts of knowledge in her recent monograph,
pointing out some of their respective characteristics:
Procedural knowledge, both perceptual and motor, is inaccessible to con-
sciousness. [... ] In spite of taking in lots of information at once [... ] it is
also relatively slow to learn, and learning is accomplished by associative
strengthening, typically over a number of trials, as in operand conditioning
or perceptual schema formation. It aggregates frequency information. [... ]
Declarative or conceptual knowledge, in contrast, is accessible to awareness
and is either describable in language, or, with a little analytic training, by
drawing. It requires attention to be encoded into this format; this means that
it is selective. [... ] The system can learn information in a single trial (in
small quantities, of course) simply by being told. In comparison to proce-
dural knowledge, it is relatively context-free. (Mandler 2004: 55)
Similarly to Mandler (2004), we consider only declarative knowledge,
which consists of mental representations, to be conceptual. Pre-
representational cognition is no-less, and probably even more important for
our survival, but it needs to be clearly distinguished from conceptual
knowledge, and hence we will say that its major "building blocks" are sen-
sorimotor schemas and categories, but not concepts.
We can now explain the prefix "pre" in our title: Pre-representational
(pre-conceptual) knowledge precedes representational cognition in (a)
phylogenesis: all animals have it, but it is only certain that human beings
have mental representations - though it is possible that at least some ani-
mals such as the great apes have some forms as well, e.g. dyadic mimesis
(Zlatev this volume), (b) ontogenesis: the sensorimotor cognition of the
young infant is pre-conceptual in the sense of Piaget, though not as long as
Piaget assumed: at least 9-month old babies can recall past experiences,
and that implies representations (Mandler 2004) and (c) microgenesis: pre-
representational schemas and categories typically operate faster than con-
scious representational thought, and continue to serve as a constant back-
drop to representational cognition, even when we have progressed past the
sensorimotor period.
Table 1 summarizes some of the properties of representational cognition
and mental representations, contrasting these with properties of pre-
representational cognition.
Table 1. Comparison of mental representation and pre-representational cognition
along a number of dimensions.
Dimension
Expression-content
differentiation
Consciousness
Major functions
Learning
Evolution
Ontogeny
Guide to behavior
Type ofmemory
Status
Example structures
Mental representations
+
accessible
thought, recall, planning
faster
later
later
lower
declarative
conceptual
mental images, mimetic
schemas, symbols
Pre-representational
cognition
inaccessible
perception, recognition,
self-motion
slower
earlier
earlier
faster
procedural
pre-conceptual
interaction patterns,
sensorimotor schemas
As shown in Table 1, the two sorts of cognition have complementary prop-
erties, and an account of human cognition, and possibly even that of certain
"higher" animals, involves both types and their coordination. While we
have here emphasized their differences in order to make a clear conceptual
distinction, we need to mention two caveats, less we be accused of "di-
chotomizing". First, since mental representations are evolutionarily and
ontogenetically later than pre-representational cognition, they can be said
to "emerge" from it, and hence will not be completely independent from
their pre-representational, "bodily" roots. Second, and as implied above,
any particular form of "higher cognition", such as language use, will in-
evitably involve pre-representational, as well as representational structures
and processes.
Nevertheless, we emphasize that mental representations cannot be re-
duced to pre-representational cognition: they constitute a qualitatively new
ontological level, and in this sense, we are in disagreement with the claims
of strong evolutionary continuity of e.g. Johnson and Rohrer (this volume).
Language is based on mental representations, but involves yet one higher
ontological level: that of consensual social reality, mutual knowledge (It-
konen 1978, 2003; Zlatev this volume). At the same time, it is not com-
pletely independent from the pre-representational, sensorimotor roots of all
cognition. In the remainder of this chapter, we will show some evidence for
this claim by first explicating two related forms of pre-representational
cognition, dynamical categories and internal meaning space, and then show
how they "map" onto certain aspects of language.
3. Dynamical categories
The concept of dynamic categorization originates from Gibson's (1962,
1979) "ecological" theory of perception (see also Costall this volume), and
in particular his emphasis on perception as a form of activity, which is
currently often referred to as active perception. Gibson's insights have
been developed and re-interpreted in multiple ways. Sasaki (2000, 2002)
applied a Gibsonian analysis to various situations such as a blind man's
navigation patterns in a town, people's usage of the visual landscape and
the action structure of breaking an egg. According to Costall (this volume),
the most important characteristics of Gibson' s concept of meaning is that it
is neither equivalent to external sensory input, nor to "representations"
generated in one's brain, but constitutes a dynamical, relational category
that arises as an active perceiver interacts with an environment.
The following aspects are particularly important in re-thinking the con-
cepts of active perception and meaning for the present chapter (see Sasaki
2000,2002; O'Regan and Noe 2001):
i) Perception can emerge via self-movement.
ii) Perceiving the environment means to explore it.
iii) Any action has inherent multiplicity.
An instance of the first aspect is active touch. Gibson (1962) reports on
experiments with blind subjects touching different shapes of cookie cutters.
If the cutter was placed on the subjects' palms, they could only determine
the correct shape with 50% accuracy. When the cutter was pushed ran-
domly on one's palm, the subjects could tell with 72% accuracy. Only by
touching the cutter in a self-guided manner could they recognize the object
in more than 95% of the cases. This study also illustrates point (ii) that
perception is a form of exploration: As we will discuss in this section, ex-
ploration is not just a method to arrive at perception; rather perception is
equivalent to the on-going exploratory process.
The third aspect is especially important for simulations involving "arti-
ficial creatures". An issue that often comes up in this context is how to
select the most appropriate set of actions. However, aspect (iii) implies that
no discrete action set (a "plan") needs to be prepared in advance. Our body
schema (see Gallagher 2005, this volume) has a huge number of degrees of
freedom. Even a simple action pattern (e.g. sitting down on a chair) con-
sists of multiple sub-level actions and the exact sequence will be afforded
by the environment as the action proceeds.
The following example could perhaps clarify the distinction between
"abstract" representational knowledge, and the kind of pre-representational
know-how that is gained through active perception. In Japan it is customary
for children to make paper cranes. We can learn how to make a paper crane
by just looking at a picture of the end state of the process and step-by-step
instructions on how to get there (as in an IKEA manual). But this kind of
knowledge is qualitatively different from that gained by actually making a
paper crane. One's experience of making the paper crane, the way we fold
the paper, how we feel touching origami and hear its rustling, etc.: all these
complex perceptive experiences organize our "embodied" dynamical cate-
gory of a paper crane. While folding origami, we experience trial-and-error
everywhere giving rise to an exploratory process. There is no way to form
this category just by seeing and memorizing an algorithmic instruction.
Therefore not only can representational cognition not be reduced to pre-
representational cognition (as argued in Section 2), bodily skills cannot be
reduced to (mental) representations either (Dreyfus and Dreyfus 1986).
3.1. Experiments with "artificial creatures"
The notion of active perception is often evoked within the "embodied cog-
nition" approach in the field of cognitive robotics (Pfeifer and Scheier
2001). In this context "embodiment" refers to the spatial/temporal dynam-
ics and the physical constraints of the robot's body in interaction with a
given environment. This is what Ziemke (2003a) apparently means by "or-
ganismoid embodiment", which is similar to, but nevertheless different
from the kind of embodiment that characterizes living systems ("organis-
mic embodiment"). Since phenomenology presupposes biology, i.e. only
(some) living beings are capable of qualia (Zlatev 2003; Emmeche this
volume), there is no reason to suppose that such "artificial creatures" have
any kind of phenomenal experience. Nevertheless, they can illustrate how
categorization can result form mastering sensory-motor coordination with
certain aspects of the environment, and in the process creating dynamical
categories of these without forming concepts.
Recently, there have been many studies of dynamic categorization in-
volving "artificial creatures", some of which are summarized in Table 2
below. Some of these studies involve actual, physical robots (Scheier and
Pfeifer 1995), while others involve computational simulations (Morimoto
and Ikegami 2004). While some scholars such as Brooks (1999) and Steels
(1994) insist on the importance of using real physical devices in "embodied
AI", there are good reasons to rely on simulations as well. As Ziemke
(2003b: 390) points out, "instead of focusing on one experiment or a few
experiments with a real robot, many questions are more suitably addressed
through a large number of experiments allowing for more variations on
agent morphologies, control architectures, and or environments." Further-
more, it should be remembered that no matter the type of the experiment,
such studies involve models which simulate rather the duplicate the phe-
nomenon being studied (Searle 1992; Zlatev 2003), and this is more easily
forgotten if the model is a "real" 3-dimensional, physical structure.
Another thing that is often forgotten, or at least not explicitly empha-
sized, but which is central for our discussion, is the difference between the
(models of the) categories of the artificial creatures, and the corresponding
(human) concepts such as LARGE/SMALL. The two are quite different: the
first are "sub-personal" and pre-representational, the second are personal,
(potentially) shared and representational. What the models simulate is
categories of the first type, and it is only we as external observers possess-
ing the respective concepts who can see correspondences between the col-
umns in Table 2.
Table 2. Studies of computer simulations of dynamical categorization. The fIrst
column shows the corresponding categories that the "creatures" formed
on the basis of physical motion. The second column shows a specifica-
tion of the differences in conceptual terms. It is only our interpretation
that links the two.
Dynamic categories
GRASPING/NON-GRASPING
NAVIGATION DIFFERENCES
NAVIGATION DIFFERENCES
FINGER MOTION
DIFFERENCES
APPROACHING/AVOIDING
Concepts
LARGE/SMALL
ROOM AlROOM B
TRIANGLE/RECTANGLE
SPHERE/CUBIC
FAST/SLOW BLINKING
References
Scheier and Pfeifer (1995)
Tani and Nolfi (1999)
Morimoto and Ikegami (2004)
Marocco and Floreano (2003)
Nolfi and Marocco (2002)
Iizuka and Ikegami (2004)
To understand this better, we offer a brief summary of how the "creatures"
in the first two studies of Table 1 performed classification through dynamic
categorization. It is worth having in mind the similarities with Gibson's
(1962) psychological experiment involving blind touch mentioned earlier:
in both cases perception is performed through self-guided motion and ex-
ploration.
Scheier and Pfeifer (1995) conducted studies of robots that learned to
discriminate on the basis of the size of an object by their bodily move-
ments. In the experiment, there were pegs of two different sizes, one with a
large diameter and one with a smaller one, distributed in an arena. A robot
used a gripping wire to pick up pegs after sensing these with its light sen-
sors. The robot could pick up the smaller pegs to carry them over to its nest
but not the larger ones. Using (a model of) an unconscious learning proc-
ess, it came to discriminate small and large pegs. As learning proceeded,
the robot neglected large pegs and only tried to pick up the smaller ones.
On the basis of this, one could be tempted to say that the robot came to
know the concepts of LARGE and SMALL (relative to its own "embodi-
ment"), but that would be a mistake. Rather, by mastering a specific type of
sensory-motor coordination, the robots learned the categories GRASP-
ABLE/NON-GRASPABLE, as they apply in this particular context. For exam-
ple, if we changed the surface texture of the pegs so that some "small" ones
would be smooth, they would become difficult to grasp, and would there-
fore be categorized together with the large ones, showing again the differ-
ence between categories and concepts.
Another experiment demonstrated that robots could discriminate be-
tween two different rooms by their bodily movements (Tani and Nolfi
1999). The two rooms were distinguished due to their different spatial ar-
rangements of corners and walls. A door that connected the two rooms
randomly opened and closed. When the door opened, a robot could move
into the other room by chance. After going back and forth between the two
rooms, a robot came to "know" which room it is situated in: it came to
possess two mutually exclusive neural modules, one for room A and one
for room B, and depending on which one was activated it behaved differ-
ently. We can say the robot categorized some of the spatial characteristics
of its environment, but can we say that it had concepts of the two rooms?
The distinctions made in Section 2 lead to a negative answer. The "neural
modules" corresponding to the two rooms are not mental representations of
the two, since there was no way for the robot to differentiate between the
module and (its perception of) the room. There was therefore no way for it
to imagine room B, while it was in room A, and to "decide" to go there.
3
On the other hand, having concepts of the two rooms implies being able
to link the concept to specific perceptual details such as the color of the
walls or the different light pattern around a corner. Concepts must be
"grounded" in perception - or else they are "empty", as Kant famously
pointed out. Our proposal is that dynamical categories play an important
role in this grounding.
3.2. Dynamic categorization of object shapes: a case study
In order to gain a better understanding of how such models of dynamic
categorization work, and to appreciate both the strengths and the weak-
nesses of dynamical categories, we here describe in some more detail the
study of Morimoto and Ikegami (2004), in which simulated artificial crea-
tures could learn to classify triangles and rectangles through self-guided
motion. The goal of the experiment was to see if geometrically well-
3. The classification of the two rooms was also dependent upon such contingen-
cies as the dynamics of the door between the two rooms. Indeed it turned out
that the open/shut dynamics of the door that connected room A and room B was
essential: randomness was needed to achieve successful classification (Tani,
private communication).
defined concepts such as TRIANGLE and RECTANGLE could be classified
only by local "blind touch" and "exploration".
The "creatures" of Morimoto and Ikegami (2004) "live" in a 2-
dimensional world, populated by objects of various shapes and colors, as
shown in Figure 1. The objects are 2 different kinds of triangles and 4 dif-
ferent kinds of rectangles. As a creature "explores" the environment, it
"touches" some of these objects, and then changes its style of motion.
Since the creature's receptive field is highly limited, it can not perceive the
shape of a whole figure. The creatures were "evolved" using a genetic al-
gorithm, which mimics biological evolution, and after thousands of evolu-
tionary generations, using a "fitness function" that favors creatures that
avoid triangles and explore rectangles, some creatures succeeded in
spending more time in touching rectangles and less time in triangles. The
point is that the creatures were not in any way explicitly trained to distin-
guish triangles from rectangles and the classification was established only
as a result of the actual exploration.
Figure 1. Part of a 2-dimensional environment showing (a) rectangles and
triangles and (b) the spatial trail of a "creature", shown as a line. The
black parts of the figures represent points of touch between creature
and objects (for details, see Morimoto and Ikegarni 2004)
Figure 1 shows part of the environment with (a) several triangles and rec-
tangles randomly distributed and (b) narrow lines penetrating the objects.
The lines represent a creature's spatial trails and we can see that it by-
passes triangles but "explores" rectangles, the black part representing
points of contact between creature and object. In particular, it can be seen
that the creature has densely covered a rotated square in the top-right cor-
ner.
How did the creatures achieve this implicit classification? Each creature
was controlled by a simple artificial neural network, shown in Figure 2, of
the type used in many other similar models (e.g. Ziemke and Thieme
2002). The network receives sensory input in a 3x3 matrix of "neurons",
each one of which has a continuous value from 0 to 1.
~ Output Neuron
/\
( ~ 00 --..\ 0
......1--+--t-
0
-4 ..... ITTIcontext Neuron
Input Neuron
~ sensory stimul i
Figure 2. The design of the neural network that controlled the artificial crea-
ture's "behavior" (Adapted from Morimoto and Ikegami 2004)
A set of connections projects from the sensory input to the motor output,
both directly, and via a number of "context neurons". Each input neuron
has "synaptic" connections to the other input neurons, and similarly for the
context neurons. Each neuron receives an integrated signal from the other
neurons multiplied by the connection weight. The connection weights also
take continuous values, and are initially set by the genetic algorithm. The
integrated signal filtered by a sigmoid function gives a new neural state of
that neuron. This updating schedule (neural states at time T > integration of
weighted activities > neural states at time T + 1) is recursively iterated
while a creature is moving around the environment. The output signals
from the input neurons are integrated to produce motor outputs. There are
three output neurons (L, F and R). The most active output neuron deter-
mines the next action of the creature: turn right (R), go straight (F), or turn
left (L). The weights of the connections were not fixed, but adapted by a
process of Hebbian learning (Hebb 1949), to allow predicting the sensory
stimuli of one step in the future. This plasticity enables dynamic switching
from one navigation mode to another: "wandering", "exploring objects"
and "filling in" (for details, see Morimoto and Ikegami 2004).
Two important observations concerning the classification and the way
in which it was achieved need to be made. First, the creatures could not
learn to discriminate between triangles and rectangles completely accu-
rately, but rather formed prototype-like categories. Figure 3 shows an ex-
ample of a creature's "behavior" towards both different instances of the
same object and different objects, given the same initial neural state (i.e.
connection weights and neuron activations). It can be noticed that a crea-
ture's behavior looks very different depending on where the creature first
encounters the object:
a) The first 10 instances of Figure 3, starting from the top, are triangles.
As can be seen, the creature enters and leaves these objects without
spending much time exploring them.
b) The following 4 instances are squares, and the creature correctly
"fills in" these figures more or less completely.
c) The following 6 instances are all parallelograms. The creature fills in
the insides of the first three quite thoroughly, but not the other three.
d) The remaining 12 instances are trapezoids. The creature's behavior
depends on the orientation of the trapezoid: the first 6 instances are
explored but their reversed images are not, and are thus not differen-
tiated from triangles.
Thus, we can conclude that the creature treated squares as the most proto-
typical type of rectangle, while other rectangular shapes were not consis-
tently distinguished from triangles.
The second observation is that the classification is highly context-
sensitive. Different instances of the same object were treated differently
depending on (a) the creature's internal state and (b) the spatial arrange-
ment of the objects. For example, the order in which the objects were ex-
plored (rectangle to triangle or triangle to triangle) changes the internal
state of a creature so that the same triangle is "perceived" differently at
different points in time.
Figure 3. A creature's "imperfect" categorization of triangles (the fIrst 10
fIgures) and rectangles (the remaining fIgures). (See the text for
discussion)
The key to understanding these features of the model, as well as its relative
success in the discrimination task, were the internal context neurons,
which functioned as a sort of procedural memory. As shown in Figure 2,
the creature's behavior was not simply a product of the coordination be-
tween bare sensory input and motor output, but also involved processed
sensory input that was "stored" in the context neurons and the weights of
the connections to and from these units. Specific analyses showed that a
creature could be sometimes driven by the sensory input (the left circuit)
but sometimes by the context neural states (the right circuit), see Figure 2.
The conclusion is that in order to perform (prototypical, context-sensitive)
triangle/rectangle discrimination, the creature required the combination of
ilraw" input signals and the internal neural states. Without this combina-
tion, the creatures showed much less diverse behavior and lower perform-
ance in discrimination.
3.3. Further studies of dynamic categorization
In order to further explore the nature of dynamic categorization and spe-
cifically the role of internal dynamics, Iizuka and Ikegami (2004) con-
ducted experiments with a new neural network architecture, where "crea-
tures" were able to spontaneously turn on and turn off the signal from the
sensory input. If the signal was "off' then behavior was guided solely by
the internal dynamics (i.e. memory) of the system, which controlled
whether environmental input should be taken in or not. This can be com-
pared to a mechanism that differentially focuses attention on either the
environment or on the memory of past experience. Since such differentia-
tion is the first step towards representation, as argued in Section 2, the
model of Iizuka and Ikegami (2004) could be seen as a model of a step
towards mental representation. But only a relatively small one, since there
is of course no awareness of any correspondence between the internal state
and the environment.
The model was applied to a different task: differentiating between dif-
ferent frequencies of light blinking by approaching, respectively avoiding
the light source. It turned out that the model learned the appropriate be-
haviors much more easily with the help of the "selective attention" mecha-
nism than without it. The reason was that by being able to selectively direct
its attention either to the blinking patterns or to its memory of past interac-
tions, a creature's behavior was not determined so much by the external
stimulus as by the style of its own motion in previous similar circum-
stances.
In sum, the experiments with "artificial creatures" described in this sec-
tion show how dynamical categories can emerge as result of sensorimotor
interactions with various stimuli, without there being any mental represen-
tations (i.e. concepts), or even awareness of these stimuli. We showed that
such categories display features such as prototype effects and context-
sensitivity, which may be advantageous in some circumstances, but disad-
vantageous in others. In particular, we suggested that dynamical categori-
zation can be enhanced through internal "context units" or other mecha-
nisms implementing a sort of (procedural) memory, and even more so by
the ability to selectively pay attention to the internal states rather than only
to the external environment. We use this as departure point for a discussion
of the role of these internal states, or what we call internal meaning space.
4. Internal meaning space
The experiments described in the previous section illustrated some Gibso-
nian features such as the active nature of perception and the "multiplicity
of action", and how they could emerge from a combination of evolutionary
history and a history of spatial exploration. At the same time, we pointed
out that the dynamic internal state of the simulated creatures retains a form
of long-term memory or personal history (which is something underplayed
in Gibsonian theory), and this affects the selection of the current action
pattern. This would imply that a living creature with such a memory would
experience the world differently from creatures with different histories of
structural coupling (Varela, Thompson and Rosch 1991; Ziemke and
Sharkey 2001). In this sense, we can say that a creature with an internal
state is not simply coordinating sensory input and motor output, but coor-
dinates the meaning of the sensory input with self-motion: it "interprets"
its environment, rather than just reacting to it.
4
In section 2 we argued that there is a gap between pre-representational
cognition and (true) mental representations. The experiments with "artifi-
cial creatures" helped clarify this gap, but also showed that the internal
dynamics of the control architectures that govern the behaviors of the
"creatures" were essential for the formation of adequate dynamical catego-
ries. We will here argue that such internal dynamics constitute an internal
meaning space. This concept does not "bridge the gap" between sensori-
4. On the basis of his long-term animal experimental studies, Freeman (2003)
insists that neural activity patterns in the sensory cortex reflect the meaning of
the input rather than the actual stimulus. See also the discussion by Ziemke and
Sharkey (2001) concerning von Uexkiill' s notion of the "historical basis of re-
action".
motor and representational cognition, but nevertheless constitutes an im-
portant (evolutionary) step in the direction of mental representations.
4.1. Quality dimensions, comparison with Gardenfors (2000)
Let us explain the senses in which we are using the constituent terms of the
notion internal meaning space. The fact that it is "internal" does not mean
that it is representational, but only that it is subjective, an emergent catego-
rization of the environment performed by the organism, i.e. its Umwelt
(von Uexkiill 1940 [1982], see Lindblom and Ziemke this volume). The
"meaning" of the space lies in the fact that it consists of a network of dy-
namical categories which are value-laden. But in what way is it useful to
regard it as a "space"?
This metaphorical expression becomes justified if, similarly to Garden-
fors (2000, this volume), we regard the internal meaning space to be struc-
tured by a number of quality dimensions, such as HEIGHT, BRIGHTNESS,
PITCH, VOLUME, WEIGHT and SHARPNESS. These are not just abstract di-
mensions along which stimuli can be classified, but dimensions that matter
to living creatures, in the sense of supporting their self-preservation and are
hence intrinsically meaningful (Zlatev 2003). Also similar to Gardenfors,
we assume that some of these dimensions have been selected for in evolu-
tion and are thus in one sense of the word "innate", while others are
learned, or at least modulated, as a result of experience. A final similarity
is that we find Gardenfors' s notion of "natural concepts" as convex regions
in conceptual space useful and believe that it corresponds to our notion of
dynamical categories: both allow for prototype effects, with the prototype
corresponding to an attractor in the internal meaning space.
The differences between our concepts and those of Gardenfors are basi-
cally two. First, the notion of internal meaning space, with dynamic catego-
ries as attractor states, is considerably more dynamic than what Garden-
fors' model allows (even with the extensions of his model involving forces
presented in the present volume): dynamical categories are constructed
through sensorimotor activity in the manner illustrated by the simulations
in Section 3, and do not simply exist as static "convex regions". Second,
we maintain that Gardenfors conflates pre-representational internal mean-
ing space and true representations by calling both "representations". This is
shown clearly in the quotation that he refers to as well:
Evidence suggests that dimensions that are easily separated by adults such as
brightness and size of a square are treated as fused together by children. . ..
For example, children have difficulty identifying whether two object differ
on their brightness or size, even though they can easily see that they differ in
some way. Both differentiation and dimensionalization occur throughout
one's lifetime. (Goldstone and Barsalou 1998: 252)
We interpret this as evidence that the dimensions of 2 year-old children are
still pre-representational, and that only when they develop the capacity to
differentiate between the dimensions consciously, or what Gardenfors (this
volume) calls "to reason about the dimensions", do they become truly rep-
resentational, or conceptual. Similarly, experientially relevant "regions" in
internal meaning space are for us natural categories, but not (yet) concepts.
This may sound like terminological hair-splitting, but it is not, since we
maintain a difference that Gardenfors does not. This leads Gardenfors (this
volume) into difficulties when it comes to distinguishing between forces as
"psychological constructs" and as "scientific dimensions". Most crucially
for the purpose of the chapter, we emphasize the pre-representational na-
ture of the internal meaning space since we are interested in clarifying, and
to some extent bridging the gap between non-representational cognition
and language, and we suggest that the internal meaning space plays an
important role in this respect.
4.2. Cross-modal transfer and synaesthesia
An important property of the internal meaning space is that it is intrinsi-
cally cross-modal. Indeed, Edelman (1992) argues that at least two mo-
dalities ("a classification couple") are necessary for any kind of natural
categories to emerge. While the experiments described in Section 3 were
multi-modal in the sense that they involved simulations of coordination
between haptic sensation and self-motion, perceptual experience is much
richer than so. The quality dimensions mentioned earlier derive from dif-
ferent modalities: vision (e.g. BRIGHTNESS), auditory system (e.g. PITCH),
kinesthetic sense (e.g. WEIGHT) and haptic sense (e.g. SHARPNESS), and the
major function of the internal meaning space is to provide an integration of
the different modalities.
A good deal of this integration is due to experience, by coordinating the
modalities through exploration, e.g. as proposed by Piaget (1945). But this
cannot explain all the available data, since it appears that some forms of
coordination between modalities, often described in terms of "transfer" or
"mappings" are pre-established by evolutionary processes (analogous to
the "genetic algorithm" in the simulation described in Section 3). Meltzoff
and Borton (1979) showed that infants as young as 1 month looked longer
at pacifiers that they had previously explored only orally, displaying a
transfer between haptic sense and vision. Perhaps even more famously (and
at first controversially), a number of studies (Meltzoff and Moore 1977,
1995) have shown that neonates are able to copy a number of bodily ac-
tions such as tongue protrusion and mouth opening, practically from birth
(see Gallagher this volume). More recently, such displays of "blind imita-
tion" (the infant cannot see its own body part) have been shown for chim-
panzees as well (Myowa-Yamakoshi et al. 2004). These results imply map-
pings between visual perception of body motion and kinesthetic perception
of one's own body and have been linked to specific neural mechanisms
such as "mirror neurons" (e.g. Rizzolatli and Arbib 1998). While not of-
fering evidence of innateness, the perception and classification of types of
body motion (Johansson 1973, see Gardenfors this volume) through visu-
ally minimal information can also be explained as the result of mapping
from vision to the body-schema (Gallagher 2005, this volume).
All these mappings, we suggest, take place in the internal meaning
space. They do not require to be thought about, or inferred, but are "per-
ceived directly" in phenomenological terms. This is in line with our claims
that internal meaning space is pre-representational, in contrast to Garden-
fors (this volume), as well as Meltzoff and Borton (1979) who interpreted
their results as showing that infants "represent" objects in an amodal repre-
sentational format. Rather, we view the internal meaning space as cross-
modal, and initially inaccessible to reflective consciousness.
A phenomenon that can be interpreted in terms of a cross-modal inter-
nal meaning space is synaesthesia (Cytowic 1995, 2002; Baron-Cohen
1996; Ramachandran and Hubbard 2001). Despite controversies on the
mechanisms responsible for it, there is now a consensus that "synaesthesia
(Greek, syn = together +aesthesis = perception) is the involuntary physical
experience of a cross-modal association" (Cytowic 1995). Clinically, sy-
naesthesia is present in (at least) 1 in 20,000 individuals, with a higher rate
for women than men (6:1) and is genetically inherited. Synaesthesia has a
number of characteristics:
(a) It is involuntary and "insuppressible": the subject cannot help but,
for example, see a certain color on hearing a particular tone;
(b) It can involve all modalities, but some are more common than others
(see Section 5 below);
(c) It is usually unidirectional, e.g. different sounds evoke visions, but
vision does not typically evoke sound;
(d) It is "projected", i.e. perceived externally as ordinary perception,
rather than in "the mind's eye";
(e) It is emotional: either disturbing or rewarding, but never neutral.
The involuntary character of synaesthesia is sufficient to distinguish it
from mental representations, and in particular language. While language
may involve some similarities with synaesthesia, as we will point out be-
low, we emphasize again that language, as a conventional-normative semi-
otic system for communication and thought, involves (intersubjective)
representations, and is qualitatively different from synaesthesia. To quote
Cytowic (1995) again: "Its phenomenology clearly distinguishes it from
metaphor, literary tropes, sound symbolism, and deliberate artistic contriv-
ances that sometimes employ the term 'synaesthesia' to describe their
multi-sensory joinings." On the other hand, synaesthesia is obviously not
"out there" in the objective environment. In our terms, it is clearly pre-
representational.
While we follow Cytowic in emphasizing its "distinctness" as a phe-
nomenon, synaesthesia is possibly more continuous to ordinary perception
than so, and in a sense "we are all synaesthetics" to some degree. Maurer
(1993) argues, on the basis of both animal and human evidence, that neo-
natal cross-modal transfer is essentially synaesthetic, but that it normally
disappears after the first few months:
During early infancy - and only during early infancy - [... ] evoked re-
sponses to spoken language (are recorded) not just over the temporal cortex,
where one would expect to find them, but over the occipital cortex as well.
There are similar reports of wide-spread cortical responses to visual stimuli
during the fIrst 2 months of life. [... ] Results such as these suggest that pri-
mary sensory cortex is not so specialized in the young infant as in the adult.
(ibid: 111, quoted in Baron-Cohen 1996)
Such "neonatal synaesthesia" could possibly explain the phenomenon of
neonatal imitation, which we pointed out above is also a form of cross-
modal transfer - as well as its "disappearance" after the first 3 months or
life. However, cross-modal transfer "reappears" later in life, e.g. in the
form of mental simulation (see Svensson, Lindblom and Ziemke this vol-
ume), but we would argue that these effects reflect imagination, rather than
perception, and are thus representational. There is thus room for both dis-
continuity and some continuity between internal meaning space and repre-
sentational cognition, including language, as we suggest in the next sec-
tion.
5. Representation, quasi-synaesthesia and language
What kind of change in internal meaning space is necessary in order to give
rise to mental representations? This is a difficult question to which we will
here offer a simple and preliminary answer: Representation involves a bi-
furcation of the internal meaning space into (a) perceptual consciousness
and (b) imagination (reflective consciousness). The model of Iizuka and
Ikegami (2004), reviewed at the end of Section 3, can serve as an illustra-
tion. To the extent that the subject can attend differentially to aspects of the
internal meaning space itself, separate from the way they mediate the per-
ception of the external world, that subject will be capable of "mental im-
ages". What remains is to understand these images as actually standingfor
something else than themselves. With this the conditions for mental repre-
sentations defined in Section 2 will be fulfilled.
How does this "bifurcation" come about? Piaget (1945) argued that
imitation plays a central role for this in childhood. Zlatev (this volume)
explains this in terms of bodily mimesis: initially the child does not differ-
entiate between the perceived body motion of the other, and the motion of
its own body, as in synaesthesia. But with deferred imitation comes differ-
entiation. And with "representational imitation" comes the ability to (con-
sciously) access the mimetic schema, and use it as a model to guide future
action. However, an apparent problem for these accounts is posed by chil-
dren who have been paralyzed, or severely motorically impaired, from birth
(e.g. Jordan 1972). Hence, actual imitation cannot be a necessary condition
for the emergence of mental representations. Zlatev (2005) attempts to deal
with this in terms of covert imitation, but this does not answer where the
ability to "covertly imitate" derives from.
At present there does not seem to be any good explanation, but we can
here at least offer a description: to have a mental representation is to have
one part of internal meaning space, imagination, standing in correspon-
dence with another, perception, and being able to differentially focus at-
tention to one or the other. This description, however, reminds of the phe-
nomenon of synaesthesia reviewed at the end of the last section. The
important differences are that (a) attention is under voluntary control and
(b) the "synaesthetic experience" is not projected into the perceptual world,
but understood as internal and "unreal" (in normal conditions).
If this conjecture is correct, we would expect to find "quasi-
synaesthetic" experiences: imagined projections between modalities in the
arts, as well as in language. In this section, we will briefly describe two
linguistic phenomena which seem to provide some support for the view
that at least some forms of linguistic representation may involve what we
here call quasi-synaesthesia in order to distinguish it from true synaesthe-
sia as defined in Section 4.2.
5.1. Quasi-synaesthetic metaphors
What we will refer to as quasi-synaesthetic metaphors are more commonly
known as "synaesthetic metaphors" (e.g. Ullman 1964): expressions such
as sweet smile, cold look, soft music, and loud color, where experience
from one sense ("source domain") is projected to, or mapped onto a phe-
nomenon that is primarily perceived in terms of another sense. For the
expressions given above, that would involve: Taste ~ Vision, Temperature
~ Vision, Touch ~ Hearing, and Hearing ~ Vision, respectively. While
some of these expressions may be quite conventional, others may be more
"novel", e.g. bitter chuckles (Tomas Pynchon, Gravity's Rainbow). The
specific types of expressions are language- and culture-specific, but the
phenomenon is apparently universal (Osgood 1959). Furthermore, many
have observed that there appear to be some universal tendencies. Ullman
(1964: 86) writes:
[T]he movement of synaesthetic metaphors is not haphazard but conforms to
a basic pattern. I have collected data for the sources and destinations of such
images in a dozen nineteenth-century poets, French, English and American,
and found three tendencies which stood out very clearly: (1) transfers from
the lower to the more differentiated senses were more frequent that those
that map in the opposite direction: over 80 percept of a total of 2000 exam-
ples showed this "upward trend"; (2) touch was in each case the largest sin-
gle source, and (3) sound the largest recipient.
If these tendencies are found across languages and cultures, then that
would appear to imply some general cognitive motivation of semantic
structure, of the type often evoked in Cognitive Linguistics. At the same
time, "mappings" from less differentiated domains (Taste and Touch) to
more differentiated domain (Vision and Hearing) would appear to go
against the grain of Conceptual Metaphor Theory (Lakoff and Johnson
1980, 1999), where rather the latter is expected: "The greater inferential
complexity of the sensory and motor domains gives the metaphors an
asymmetric character, with inferences flowing in one direction only." (La-
koff and Johnson 1999: 57-58)
Ullman does not specify his evidence in more detail, but Day (1996) re-
ports a study in which he analyzed 1269 (quasi-)synaesthetic metaphors in
English texts ranging from Chaucer to Pynchon, specifying the directional-
ity of the mappings. Table 3, adapted from Day (1996, Table 7), classifies
the 1269 expressions in terms of these mappings between the "six senses".
It can be seen, for example, that Touch was by far the most common source
domain, mapping to Vision 135 times (e.g. hard look), while Hearing was,
also by far, the most common Target domain, exactly as claimed by UII-
man (1964).
Table 3. Classification of the mappings in 1269 quasi-synaesthetic metaphors,
based on Day (1996: Table 7), see text for discussion.
Target Source domain
domain Hearing Taste Smell Temp. Touch Vision # Target
domain
Hearing 149 1 86 540 80 856
Taste 0 0 1 6 0 7
Smell 7 60 3 34 14 118
Temp. 0 19 0 8 4 31
Touch 3 10 0 0 2 15
Vision 26 38 1 42 135 242
# Source 36 276 2 132 723 100 1269
domain
When Day (1996) subtracted the instances when a sense modality was used
as source from those when it was used as target, he arrived at the "sensory
ranking" given in (1):
(1) Touch> Taste> Temperature> Smell> Vision> Hearing
Analogously, Day classified the types of synaesthesia from the 25 subjects
reported by Cytowic (2002) some of who had multiple synaesthesia, giving
a total of 35 projections. These are given in Table 4, adapted from Day
(1996, Table 6). Comparing to Table 3 shows that, similarly, Hearing was
by far the most common target (i.e. when the subjects heard sounds they
also experienced sensations from other modalities), but unlike with the
metaphors, it was Vision that was the most typical source, which is often
stated in descriptions of synaesthesia, typically involving projection of
c%r (e.g. Ramachandran and Hubbard 2001).
Table 4. Classification of 35 proj ections in 25 synaesthetic subj ects, analyzed by
Cytowic (2002) dapted from (Day 1996: Table 5), see text for discus-
sion.
Target Source modality
modality Hearing Taste Smell Temp. Touch Vision # Target
modality
Hearing 2 1 0 2 21 26
Taste 0 0 1 1 1 3
Smell 0 0 1 1 0 2
Temp. 0 0 0 0 0 0
Touch 0 0 0 0 2 2
Vision 0 0 1 0 1 2
# Source 0 2 2 2 5 24 35
modality
Using the same method of "subtraction" provided the ranking given in (2),
which, however, given the small number of subjects and projections should
not be taken as too seriously. Nevertheless, it clearly shows that there are
both similarities between quasi-synaesthetic metaphors and actual synaes-
thesia, above all that Hearing is most often "interpreted" as something else,
and differences: above all the ranking of Vision.
(2) Vision> Touch> Temperature> Smell> Taste> Hearing
The conclusions that we draw from this are in part similar to those drawn
by Day (1996): that while the similarities may reflect universal features of
human consciousness, and its neural underpinnings, or in our terms, the
organization of our internal meaning space, the differences require distin-
guishing between the mechanisms of synaesthesia and semantic processes:
"The meanings for synaesthetic metaphors are not simply there, hard-wired
and innate, but are generated through semantic processes and fashioned by
time and cultural elements, much like other metaphors" (ibid: 20). We
further add that the differences point to the radical difference between pre-
representational cognition and language. The first could possibly motivate
some of its characteristics (e.g. the tendency to consciously relate sound-
based concepts to concepts based on some other modality), but semantics
cannot be reduced to pre-representational cognition, and even less so to
neural structures (against the claims of e.g. Dodge and Lakoff 2005).
Still, we find this line of research intriguing, and pointing to the possi-
ble synaesthetic roots of conscious, mental representations. Once the
"source" and the "target" modalities can be differentiated, the first can be
mapped onto the other, utilizing the quality dimensions of the internal
meaning space. If the mapping between the perception of another's actions
and those of oneself involves similar mechanisms to those involved in sy-
naesthesia, its differentiation and focus on the "internal image" of the ac-
tion would be equivalent to (dyadic) covert mimesis, possibly the original
form of mental representation, as suggested by Piaget (1945) and Zlatev
(2005, this volume).
5.2. Japanese mimetics
Sound-symbolism is another universal phenomenon of language, but the
degree to which particular languages employ it varies. It is comparatively
marginalized in Indo-European languages, and unsurprisingly, it was not
considered a central feature of language from the birth of modem linguis-
tics in Europe (Saussure 1916). Another reason for its de-appreciation is
that it goes against the Saussurian dictum of the "arbitrariness of the lin-
guistic sign". However, while language is conventional, this need not imply
arbitrariness, even though these two concepts are often conflated (see
Zlatev this volume). Japanese mimetics are highly conventional, but "as-
pects of the form meaning relationship are not arbitrary but are motivated
by iconicity." (Kita 2001: 419-420). They are also a central feature of the
language. Ivanova (2001: 2) provides the following informative characteri-
zation of their role in the language:
Japanese is one of the languages with vast sound-symbolic systems [... ] with
more than 2,000 onomatopoeic and mimetic words. These words overwhelm
ordinary speech, literature and the media due to their expressiveness and
load of information. Although they are never used in official documents, it is
not exceptional to hear them in formal situations, too. People of all ages em-
ploy mimetic words in communication, believing that speech that abounds in
such words sounds much more natural and full of life than speech that tends
to avoid them.
Classifications of Japanese mimetics differ, but at least four types can be
distinguished (Martin 1975; Kita 1997; Ivanova 2001; Baba 2003):
(a) gi-sei-go, words imitative of sounds produced by living creatures,
e.g. wan-wan ('bow-bow', the barking of a dog)
(b) gi-on-go, words imitative of sounds produced by the inanimate
world, e.g. ban ('bang', a loud sound produced from an object hit-
ting another)
(c) gi-tai-go, words imitative of physical actions, e.g. koro-koro ('light
object rolling repeatedly')
(d) gi-jyoo-go, words imitative of psychological states, e.g. muka-tsuku
(' irritating')
The first thing that needs to be explained is the sense in which these words
can be said to be "imitative", i.e. their iconicity. In the case of (a) and (b),
which are similar to the onomatopoetic words that we are all familiar with,
this is relatively straightforward: the sound-expression resembles the sound
that is produced in the referential scene. Hamano (1998), who is often
credited for providing one of the most extensive analysis of Japanese mi-
metics, attempts to link particular phonemes (e.g. Ipl vs. Id/) and distinctive
features (e.g. +1- voice) to specific meaning components. Thus, Ipl is for
exampled claimed to be associated with "light, small, fine", /bl with
"heavy, large, course" and /ml with "murkiness". However there are prob-
lems for this analysis. As Ivanova (2001) points out mimetic "words with
initial Imf are maza-maza (clearly, vividly), meki-meki (remarkably, fast),
miQchiri (hard, severe), moya-moya (hazy, murky), muka-muka (retch, go
mad). It is clear that "murkiness" is not their common semantic feature."
Instead, she characterizes the expression-meaning correspondence of 199
mimetics of the (c) and (d) types in terms of more general "phonaesthe-
matic patterns", where "phonaesthematic describes the presence of se-
quence of phonemes shared by words with some perceived common ele-
ment in meaning" (Ivanova 2001). An example of such a pattern is given in
(3) below.
(3) expressIon: g/k + V + chi + g/k + V + chi
meaning: LACK OF FLUIDITY OR SPACE
examples: gichi-gichi ('very tight'), kachi-kachi ('frozen hard',
'dried up'), kochi-kochi ('tense', 'stiff, 'frozen hard')
What this approach, however, leaves out are generalizations that apply
within and across the patterns, such as that contrast +1- voice may be used
to distinguish WEIGHT, e.g. koro-koro vs. goro-goro 'heavy object rolling
repeatedly' or VOLUME, e.g. chara-chara 'few coins rattling' vs. jara-jara
'many coins rattling' - as when the coins "come out of a slot machine
when one hits the jackpot" (Baba 2003: 1868). Finally, neither of these
accounts explains in which way the expression of the mimetic "imitates" or
"resembles" their meanings, a problem that is even more pronounced with
respect to gi-jyoo-go, sometimes also called "psychomimes". How can we
make sense of the suggestion that their expression is mimetic with respect
to psychological states?
We believe that a key to this puzzle is offered by the presence of cross-
modal mappings in internal meaning space. In their investigations of sy-
naesthesia Ramachandran and Hubbard (2001) describe experiments in
which ordinary subjects were given contrasting pictures of objects of dif-
ferent shapes, for example one which was roundish and "soft" and another
which was edgy and "sharp". Then they were given two "names", e.g. kiki
and bouba, and asked to pair name and object. Just as expected, 95% of the
subjects paired kiki with the sharp object and bouba with the roundish
one.
5
Why should this be the case? If we start from the shapes, the cross-
modal mapping between vision and touch would allow them to be per-
ceived as "soft" vs. "sharp", motivating the use of these quasi-synaesthetic
metaphors as a natural way to describe these figures. From the side of the
expressions the production of the velar stop 1kJ, even more so combined
with the front, unrounded vowel lil involves obstructions and narrowings in
the vocal tract, which can similarly be perceived as "sharp" and "edgy". On
the other hand, the shape of the vocal tract and the lips in the production of
Iu/ in bouba, are quite literally "roundish" and the passage of the air is
"soft". The mappings between the senses Vision-Touch-Proprioception-
Sound in internal meaning space thus provides for a correspondence be-
tween the shapes and the labels that would be impossible otherwise. A
robot or a Martian with a very different kind of body (and possibly even a
person lacking haptic sense and proprioception) would not be able to per-
ceive the iconicity involved.
6
5. The experiment was essentially a replication of a classical experiment per-
formed by Kohler (1929), who called the figures takete and baluma.
6. Ramachandran and Hubbard (2001: 18) note that a patient with damage of the
angular gyrus, a cortical structure situated between the temporal, parietal and
occipital lobes "showed no propensity for the boubalkiki effect". This is in-
Returning to Japanese mimetics, we can suggest that they are quasi-
synaesthetic in a similar way to kiki and bouba. This can explain some of
the contrasts, such as +1- voice, since voicing involves a higher degree of
energy, both in terms of perception and production. But notice that even for
kiki and bouba it is not possible to figure out what they would mean only
on the basis of the iconicity of the expressions, but only to do the matching
to the "correct" shape when one is provided a few shapes to choose from.
Sonesson (2001) makes an important distinction between primary iconic-
ity, in which the similarity between A and B can be perceived even without
knowing that A is a representation of B, and secondary iconicity, in which
to perceive any similarity between A and B requires knowing that A is a
representation (sign) of B. Realistic pictures, photographs and pantomime
can be interpreted by virtue of primary iconicity. On the other hand, the
iconicity of diagrams in which "up stands for more" or onomatopoetic
words like "bow-bow" can be appreciated first when one understands their
representational expression-content structure. We would like to suggest
that the distinction between primary and secondary iconicity is more of a
cline, defined by the degree to which the "sign function" (i.e. knowing
what an expression represents) is necessary for perceiving the similarity
involved in iconicity. From this perspective, the bouba/kiki phenomenon is
somewhat intermediary in the cline.
Synchronically speaking, the iconicity of Japanese mimetics must be
secondary rather than primary: once the child learns what concepts they
express as part of the language acquisition process, some of the similarities
could be perceived. It is much more difficult to answer the diachronic
question: how the particular set of mimetics emerged in the first place? But
it is clear that it must have involved a social, collaborative process, and not
just a matter of Japanese speakers spontaneously externalizing their spe-
cific dynamic categories in speech. Furthermore, since Japanese mimetics
are conventional expressions the motivation behind their meaning will be
mediated by cultural norms and analogies to other expressions in the lan-
guage, and hence often difficult to perceive. For example, while the gi-tai-
go mimetic noro-noro ('drag oneself', 'walk slowly'), can possibly be re-
lated to the "laxness" of the nasal 1nl and the round central vowel 10/, the
triguing since the angular gyrus is considered to play a role in cross-modal
transfer, and may even be important for the comprehension of (novel) quasi-
synaesthetic metaphors.
meaning of the gi-jyoo-go mimetic noko-noko ('nonchalantly') is less
transparent, and possibly co-motivated by its analogy to noro-noro.
We can conclude therefore that Japanese mimetics take a somewhat in-
termediary place between iconic representations such as pictures and pan-
tomimes (which can be interpreted even by virtue of primary iconicity),
and fully symbolic and propositionallanguage. While they may bear traces
of their pre-representational roots, in particular by relying on the cross-
modal mappings of the internal meaning space, they are clearly conven-
tionalized linguistic representations, consisting of socially shared expres-
sions and contents. These contents, however, appear to be more subjective,
somewhat difficult to define, and very difficult for second language learn-
ers of Japanese (Ivanova 2002).
This conclusion is fully consistent with the analysis of Japanese mi-
metics presented by Kita (1997, 2001). Kita argues that the meaning of
Japanese mimetics is (primarily) represented in an affect-imagistic dimen-
sion, where "language has direct contact with sensory motor and affective
information" (Kita 1997: 380) and "vivid imagery of perceptual and
physiological experiences" (Kita 2001: 420). In contrast, Kita advocates
that the meaning of non-mimetic expressions constitutes an analytic di-
mension, including "quantifiers, logical operators, and semantic categories
such as agent, patient and action" (Kita 1997: 1863). Apart from the ico-
nicity of Japanese mimetics discussed above, Kita provides the following
types of evidence for the need to evoke two different kinds of representa-
tions for mimetic and non-mimetic expressions.
(a) A mimetic such as suta-suta ('walk hurriedly') does not lead to re-
dundancy when combined with a semantically overlapping non-
mimetic expression such as haya-aruki ('walk hastily') in a single
clause. In comparison, the combination of the latter and another
overlapping adverbial such as isogi-ashi ('hurriedly') does lead to an
impression of "wordiness".
(b) A clause with an (adverbial) mimetic cannot be combined with sen-
tence negation.
(c) The production of a mimetic is highly associated with expressive
intonation and spontaneous iconic gestures: 95% of the mimetics
produced in a study where precisely synchronized with an iconic
gesture, compared to only 36% of the verbs.
At the same time, Kita (2001) clarifies that this evidence concerns only so-
called "adverbial mimetics", which are most common and least integrated
into the grammar of the language, rather than cases when mimetics are
used as verbs, nouns, and noun-modifiers. Thus there is not only close
integration between the two dimensions, as Kita points out, but also evi-
dence for a cline between them, with nominal mimetics being the most
grammaticalized and "analytical".
This interpretation is furthermore supported by a recent pragmatic study
of the use of mimetics in 4 different spoken registers, characterized by
different levels of "emotive intensity" (Baba 2003). It was found that in-
deed the total use of mimetics correlated with the intensity level, and that
gi-jyoo-go (the "psychomimes") where used only when the episode was
narrated from a first-person perspective, involving the highest degree of
subjectivity. At the same time, the nominal use of mimetics was most typi-
cal with the least emotive and most detached of the four levels.
Finally, we can link these studies to the concept of bodily mimesis and
mimetic schema, a mental representation involving bodily simulation that is
prelinguistic, and arguably precedes and "grounds" language in phylogeny
and ontogeny (Zlatev, Person and Gardenfors 2005; Zlatev 2005, this vol-
ume). Mimetic schemas are dynamical structures of consciousness involv-
ing the body image, used in pre-linguistic thought and externalized in body
movements and gestures. The meaning of adverbial Japanese mimetics
would thus appear to correspond rather directly to mimetic schemas, even
more so than the meanings of verbs, while those of nominal and verbal
mimetics, as well as non-mimetic expressions would qualify as "post-
mimetic". Thus our distinction mimetic/post-mimetic appears to corre-
spond quite closely to Kita's distinction between the "affecto-imagistic"
and the "analytic" dimension. Both dimensions are necessary and need to
be integrated for effective communication, especially in literature.
Consider the following passage, taken from the novel And Then by 80-
seki Natsume:
Turning to the head of his bed, he noticed a single camellia blossom that had
fallen to the floor. He was certain he had heard it drop during the night; the
sound had resounded in his ears like a rubber ball bounced off the ceiling.
Although he thought this might be explained by the silence of the night, just
to make sure that all was well with him, he had placed his right hand over
his heart. Then, feeling the blood pulsating correctly at the edge of his ribs,
he had fallen asleep.
7
The meaning conveyed by this passage consists only in part of its proposi-
tional, analytic content, representing subjective and objective states-of-
affairs such as the protagonist turning to the head of the bed, a flower lying
on the floor, his memory of a loud sound, placing the hand over the heart
etc. What we could call the "embodied meaning" (if the phrase were not
overused nowadays) supplements this by the reader identifying with the
protagonist and mimetically experiencing the situation from the protago-
nists' point of view. Notice that Natsume explicitly mentions four different
sensory modalities: Proprioception (turning, placing the hand, blood pul-
sating), Vision (noticing), Hearing (dropping, bouncing), Touch (feeling).
Due to the cross-modal connections of the internal meaning space, there
are quasi-synaesthetic experiences as well: the smell of the camellia blos-
som, and the temperature (warmth) of the blood. It is possible to associate
in the affecto-imagistic dimension further. For example, for one of us (the
native speaker of Japanese), mentioning the pulsating of blood and the
heart brings to mind the color red. Hence the color of the falling blossom is
also perceived (in imagination) as red!8 The chain of subjective quasi-
synaesthetic experiences can run on: the color red induces a memory of a
red sunset. The memory cold air at sunset stimulates the olfactory senses
and that further stimulates the tactile feeling of a cold handraiL .. However,
since these meanings are not conventional, they are bound to remain rather
private associations. For example, the second (and non-Japanese) author
"sees" the fallen flower as white, while a young Swedish poet (reading the
English translation) is convinced that it is pink... Such indeterminacy has
its advantages, as in the interpretation of poetry, but is problematic if it is
essential for "sender" and "receiver" to be able to share similar experi-
ences.
7. This constitutes a literary translation, by Norma Field, of the original passage,
which reads: Makura moto wo miruto, yae no tubaki ga itirin tatami no ueni
otiteiru. Daisuke ha yuube tokono nakade tashikani kono hana no ochiru oto
wo kiita. Kare no miminiha, sore ga gomumari wo tenjyou-ura kara nagetuketa
hodo ni hibiita. Yoru ga fukete, atari ga sizukana seikatomo omottaga, nenno-
tame, miginote wo shinzou no ueni nosete, abara no hazureni tadashiku ataru
chi no oto wo tashikamenagara nemuri ni tuita.
8. And similarly for another native Japanese speaker, Misuzu Shimotori.
Japanese mimetics offer the advantage of being conventional, and thus
less idiosyncratic, while at the same time evoking the affecto-imagistic
dimension. In a passage from another novel, Wayfaring Diary, Natsume
uses a number of mimetic expressions describing the spatial movements of
the first-person protagonist, as well as of other personae.
A flight of stone steps brings back my memories. Some time I strolled
around the "Five Mountains". Just like today I was sluggishly walking up
the stairs which leads to the residence of monks in Engaku temple or some-
where else. Out of the gate appeared a monk in a yellow robe. He had a flat-
crowned head. I ascended while he descended. When we came across, he
asked in a sharp voice, "Where are you going?" I answered, "To see the
Precincts", and stopped. "There's nothing within the precinct", the monk
gave an immediate answer as he left quickly down the stairs.
The mimetics used in place of the highlighted parts are guru-guru ('stroll-
ing around'), nosori-nosori ('sluggishly') and suta-suta ('walk hurriedly')
capturing vividly the contrast between the protagonist's motion, and meta-
phorically his state of mind, and those of the monk.
6. Summary and conclusions
The goal of this chapter has been twofold. On the one hand, we have tried
to emphasize the following qualitative differences in human cognition:
interaction is different from representation,
procedural knowledge is different from declarative knowledge,
dynamical categories are different from concepts,
synaesthesia is different from language,
the meaning of Japanese mimetics (the affecto-imagistic dimension)
is different from the meaning of e.g. quantifiers etc. (the analytical
dimension)
We believe that such emphasis of differences is necessary because there is
persistent tendency in embodiment theories, e.g. the "full embodiment"
approach advocated by Nufiez (1999), to "resolve" such oppositions by
ignoring the differences, and thus, in effect, reducing or eliminating the
second and more "disembodied" side of the oppositions. In our view, such
an approach is inadequate since we believe that conceptual - and in some
cases even ontological - differences between different levels need to be
maintained.
At the same time, our second goal has been to explore how structures
and processes of pre-representational cognition such as dynamical catego-
ries, internal meanings space, and synaesthesia can play a role in the
"grounding" of mental representations (concepts) and language, i.e. pro-
vide evolutionary and ontogenetic prerequisites for the emergence of the
latter. Figure 4 summarizes our general picture of the major different levels
of meaning in a pyramid ofsemiotic development.
The rock-bottom of cognition is life itself, the sine qua non of all
meaning (von Uexkiill 1940 [1982]; Maturana and Varela 1987; Zlatev
2003). Value systems derived from natural selection in evolution control
the behavior and learning of living organisms. Using the complex systems
modeling approach in Section 3 we showed how simple "artificial crea-
tures" can form dynamical categories through sensorimotor coordination
without any representational ability. At the same time it was suggested that
by distinguishing perception and memory of past experience, categoriza-
tion can be enhanced. Cross-modal mappings of various sorts are prevalent
in the internal "space" which defines a coherent, multi-modal world for the
subject. Synaesthesia may be only the tip of the iceberg of this, showing
the importance of both correlating and differentiating modalities.
The crucial step between pre-representational and representational cog-
nition occurs, we suggested, with the bifurcation of the internal meaning
space into a part the focuses on the external world, and another that
"looks" into memory, i.e. recall and the projected future, i.e. planning. A
necessary step for this, as with any representation, is to acknowledge that a
certain "expression" both corresponds to and is different from certain
"content". It is possible that this breakthrough occurred precisely with
bodily mimesis, making the body image the first true "signifier", but this
remains so far only a conjecture. In any case, the presence and role of mi-
metic schemas is most clearly shown in iconic gesturing, which is universal
and ubiquitous. The close synchronization of gesture and speech can be
explained if mimetic schemas underlie both, with iconic gestures, and cer-
tain structures of language such as mimetic expressions, being more di-
rectly related to the mimetic, imagistic dimension, while most of the unique
properties of language are qualitatively distinct from mimesis in being
symbolic, propositional and fully conventional: the top triangle of the
pyramid. Of course, just like the Cat with a Hat of Dr. Seuss, we could cut
up this top in smaller and smaller slices, with the more "abstract" ones
being on the top: written language, mathematics, symbolic logic ... On the
other hand, actual language use does not involve the "top" alone, but the
whole pyramid, and at least the imagistic dimension involved in mental
simulation (imagination) and the quasi-synaesthetic associations of the
internal meaning space. This is, in brief, our view of how language can be
"embodied", while at the same time remaining conceptually and ontologi-
cally irreducible to sensorimotor experience.
Dynamical categories
Autopoiesis and intrinsic value (Life)
Figure 4. From Life to Language. Autopoiesis and intrinsic value preserving
the identity of the organism, essential properties of life, constitute the
primary basis for cognition and consciousness. The interconnected-
ness of dynamical categories along common quality dimensions
provides an internal meaning space. Bifurcations of this space,
possibly arising from self-other separation in overt or covert
imitation gives rise to bodily mimesis and mimetic schmeas, which
are consciously accessible mental representations. These are imagis-
tic and affect-laden, and can be said to ground language in ontogeny
and phylogeny. Only the highest two layers involve mental
representations.
2002
Baba, Junko
2003
Acknowledgements
We wish to thank the collaborators of the first author: Ryoko Uno, Hiro-
yuki Iizuka and Gentaro Morimoto, as well as Goran Sonesson, Tom
Ziemke, Sotaro Kita and Misuzu Shimotori for their valuable comments on
an earlier draft.
The first author was partially supported by a grant-in-aids from The
21st Century COE (Center of Excellence) program (Research Center for
Integrated Science) of the Ministry of Education, Culture, Sports, Science
and Technology, Japan, the Kayamori foundation and the ECAgent project,
sponsored by the Future and Emerging Technologies program of the Euro-
pean Community (IST-1940). The second author was supported by the
Language, Gesture and Pictures in Semiotic Development project at the
Faculty for Humanities and Theology at Lund University, Sweden, and the
EU-project Stages in the Evolution and Development of Sign Use
(SEDSU).
References
Pragmatic function of Japanese mimetics in the spoken discourse of
varying emotive intensity levels. Journal of Pragmatics 35: 1861-
1889.
Baron-Cohen, Simon
1996 Is there a normal phase of synaesthesia in development? PSYCHE,
2(27), June 1996, http://psyche.cs.monash.edu.aulv2/psyche-2-27-
baron cohen.html
Brooks, Rodney
1999 Cambrian Intelligence. Cambridge, Mass.: MIT Press.
Costall, Alan
this vol. Bringing the body back to life: James Gibson's ecology of agency.
Chalmers, David
1997 The Conscious Mind: In Search of a Fundamental Theory. Oxford:
Oxford University Press.
Cytowic, Richard
1995 Synaesthesia: Phenomenology and neuropsychology. PSYCHE,
2(10), July 1995, http://psyche.cs.monash.edu.aulv2/psyche-2-10-
cytowic.html
Synaesthesia: A Union of the Senses. Second edition. Cambridge,
Mass: MIT press.
Damasio, Antonio
2000 The Feeling of What Happens. Body, Emotion and the Making of
Consciousness. New York: Harvester.
Day, Sean
1996 Synaesthesia and synaesthetic metaphors. PSYCHE, 2 (32), July
1996, http://psyche.cs.monash.edu. au/v2/psyche-2-32-day.html
Dodge, Ellen and George Lakoff
2005 On the neural bases of image schemas. In: Beate Hempe (ed.), From
Perception to Meaning: Image Schemas in Cognitive Linguistics.
Berlin: Mouton de Gruyter.
Dreyfus, Hubert and Stuart Dreyfus
1986 Mind over Machine. The Power ofHuman Intuition and Expertise in
the Era ofthe Computer. New York: Free Press.
Edelman, Gerald
1992 Bright Air, Brilliant Fire: On the Matter of the Mind. London: Basic
Books.
Emmeche, Claus
this vo!. On the biosemiotics of embodiment and our human cyborg nature.
Fodor, Jerry A.
1981 Representations. Cambridge, Mass.: MIT Press.
Freeman, WaIter
2003 A neurobiological theory of meaning in perception. International
Journal ofBifurcation and Chaos 13 (9): 2493-2511.
Gallagher, Shaun
this vo!. Phenomenological and experimental contributions to understanding
Gardenfors, Peter
2000 Conceptual Spaces. The Geometry of Thought. Cambridge, Mass.:
MIT Press.
this vo!. Representing actions and functional properties in conceptual spaces.
Gardner, Howard
1987 The Mind's New Science. London: Basic Books.
Gibson, James J.
1962 Observations on active touch. Psychological Review 69: 477-491.
1979 The Ecological Approach to Visual Perception. Boston: Houghton-
Mifflin.
Goldstone, Robert L. and Lawrence W. Barsalou
1998 Reuniting perception and conception, Cognition 65: 231-262.
Hamano, Shoko
1998 The Sound-symbolic System ofJapanese. Stanford, CA: CSLI.
Hebb, Donald O.
1949 The Organization o/Behaviour. New York: John WHey and Sons.
Husserl, Edmund
1962 Phiinomenologische Psychologie. Husserliana IX. The Hague:
Nijhoff.
Itkonen, Esa
1978 Grammatical Theory and Metascience. Amsterdam: Benjamins.
2003 What is Language? Turku: University of Turku Press.
lizuka, Hiroyuki and Takashi Ikegami
2004 Simulating autonomous coupling in discrimination of light frequen-
cies. Connection Science 16 (4): 283-299.
Ivanova, Gergana
2001 On the relation between sound, word structure and meaning in Japa-
nese mimetic words. Iconicity in Language. http://www.trismegis-
tos.com/lconicitylnLanguage/Articles/lvanova.html
Johansson, Gunnar
1973 Visual perception of biological motion and a model for its analysis.
Perception and Psychophysics 14: 201-211.
Johnson, Mark
1987 The Body in the Mind. Chicago: University of Chicago Press.
Johnson, Mark and George Lakoff
2002 Why cognitive linguistics requires embodied realism. Cognitive
Linguistics 13 (3): 245-263.
Johnson, Mark and TimRohrer
this vol. We are live creatures: Embodiment, Pragmatism and the cognitive
organism.
Jordan, N.
1972 Is there an Achilles heel in Piaget's theorizing? Human Development
15: 379-382.
2001
Kita, Sotaro
1997 Two-dimensional semantic analysis of Japanese mimetics. Linguis-
tics 35: 379-415.
Semantic schism and interpretive integration in Japanese sentences
with a mimetic: a reply to Tsujimura. Linguistics 39: 419-436.
Kaneko, Kunihiko and Takashi Ikegami
1998 Evolutionary Scenario o/Complex Systems Studies. Tokyo: Asakura.
Kaneko, Kunihiko and Tsuda Ichiro
2000 Complex Systems: Chaos and Beyond: A Constructive Approach
with Applications in Life Sciences. Berlin: Springer-Verlag.
K6hler, Wolfgang
1929 Gestalt Psychology. New York: Liveright.
Lakoff, George
1987 Women, Fire and Dangerous Things: What Categories Reveal About
the Mind. Chicago: University of Chicago Press.
1999 Philosophy in the Flesh: The Embodied Mind and its Challenge to
this vol. Embodiment and social interration: A cognitive science perspective.
Mandler, Jean
2004 The Foundations of Mind: Origins of Conceptual Thought. Oxford:
Marocco, Davide and Dario Floreano
2003 Active vision and feature selection in evolutionary behavioral sys-
tems. In: Bridget Hallam, Dario Floreano, Jean-Arcady Hallam,
Gillian Hayes and John A. Meyer (eds.), From Animals to Animates
VII: Proceedings of the 7th International Conference on Simulation
ofAdaptive Behavior, 247-255. Cambridge, Mass: MIT Press.
Martin, Samuel, E.
1975 A Reference Grammar of Japanese. New Haven: Yale University
Press.
Maturana, Humberto and Francisco Varela
1987 Tree of Knowledge: The Biological roots of Human Understanding.
Boston: Shambhala.
Maurer, Daphne
1993 Neonatal synaesthesia: implications for the processing of speech and
faces. In: Benedict de Boysson-Bardies (ed.), Developmental Neuro-
cognition: Speech and face processing in the first year of life, 109-
124. Dordrecht: Kluwer.
Meltzoff, Andrew N. and Richard W. Borton
1979 Intermodal matching by human neonates. Nature 282 (5737): 403-
404.
Meltzoff, Andrew and Michael Moore
1977 Imitation of facial and manual gestures by human neonates. Science
198: 75-78.
1995 Infants' understanding of people and things: From body imitation to
folk psychology. In: Jose Bermudez, Naomi Eilan, and Anthony
Marcel (eds.), The Body and the Self, 43-70. Cambridge, Mass.:
MIT Press.
Morimoto, Gentaro and Takashi Ikegami,
2004 Evolution of plastic sensory-motor coupling and dynamic categori-
zation, In: Jordan Pollack, Mark Bedau, HH Husbands, Takashi Ike-
gami and Richard A. Watson (eds.), Artificial Life IX' Proceeding of
Ninth International Conference on the Simulation and Synthesis of
Living Systems, 188-193. Cambridge, Mass.: MIT Press.
Myowa-Yanakoshi, Masako, Tomonaga, Masaki, Tanaka, Masayuki and Tetsuru
Matsuzawa
2004 Imitation in neonatal chimpanzees (Pan troglodytes), Developmental
Science 7 (4): 437-442.
Nolfi, Stefano and Davide Marocco
2002 Active perception: A sensorimotor account of object categorization.
In: Bridget Hallam, Dario Floreano, Jean-Arcady Hallam, Gillian
Hayes and John A. Meyer (eds.), From Animals to Animates VII:
Proceedings of the 7th International Conference on Simulation of
Adaptive Behavior, 266-271. Cambridge, Mass: MIT Press.
Nuiiez, Rafael
1999 Could the future taste purple? Reclaiming mind, body and cognition.
Journal ofConsciousness Studies 6 (11/12): 41-60.
O'Regan, J. Kevin and Alva Noe
2001 A sensorimotor account of vision and visual consciousness, Behav-
ioral and Brain Sciences 24: 939-1011.
Osgood, Charles E.
1959 The cross-cultural generality of visual-verbal synaesthetic tendencies.
Behavioral Science 5: 146-169.
Piaget, Jean
1945 La formation du symbole chez l'enfant. Neuchatel-Paris: Delachaux
et Niestl6; English translation: G. Gattegno and F. M. Hodgson. Play,
Dreams and Imitation in Childhood. New York: Norton, 1962.
2001 Understanding Intelligence. Cambridge, Mass: MIT press.
Port, Robert and Timothy Van Gelder
1998 Mind as Motion: Explorations in the Dynamics ofCognition. Reprint
Edition. Cambridge, Mass.: MIT Press.
Ramachandran Vilayanur S. and Hubbard Edward M.
2001 Synaesthesia: A window into perception, thought and language.
Journal ofConsciousness Studies 8 (12): 3-32.
Rizzolatti, Giacomo and Michael Arbib
Rosch, Eleanor
1973 Natural categories. Cognitive Psychology 4: 328-350.
Sasaki, Masato
2000 Chikaku ha Owaranai [Perception Never Stops - Introduction to
Affordance] Tokyo: Seido-sha.
2002 Sentai-Shinrigaku no Kousou [Grand Design of Affordance] Tokyo:
University of Tokyo Press.
Saussure, Ferdinand de
1916 Cours de Linguistique Generale. Paris: Payot.
Scheier, Christian and RolfPfeifer
1995 Classification as sensory-motor coordination: A case study on
autonomous vehicles. In: Federico Moran, Alvaro Moreno, Juan J.
Merelo and Pablo Chacon (eds.), Proceedings of the 3rd European
Conference on Artificial Life, 657-667. Berlin: Springer-Verlag.
Searle, John
1992 The Rediscovery ofthe Mind. Cambridge, Mass.: MIT Press.
Sonesson, Goran
1989 Pictorial Concepts. Lund: Lund University Press.
2001 From semiosis to ecology. On the theory of iconicity and its conse-
quences for the ontology of the Lifeworld. VISIO 6/2-3: 85-110.
this vo!. From the meaning of embodiment to the embodiment of meaning: A
study in phenomenological semiotics
Steels, Luc
1994 The artificial life roots of artificial intelligence. Artificial Life 1: 75-
IlO.
this vo!. Making sense of embodied cognition: Simulation theories of shared
Tani, Jun and Stefano Nolfi
1999 Learning to perceive the world as articulated: an approach for hierar-
chical learning in sensory-motor systems. Neural Networks 12:
1131-1141.
Thompson, Evan
2001 Empathy and consciousness, Journal of Consciousness Studies 8
(5/7): 1-32.
Ullman, Stephen
1964 Language and Style. Oxford: Basil Blackwell.
Varela, Francisco, Evan Thompson and Eleonor Rosch
von Uexkiill, Jakob
1940 The theory of meaning. Semiotica 42 (1): 25-82. Reprint 1982.
Ziemke, Tom
2001 The construction of 'reality' in the robot. Foundations of Science 6
(1): 163-233.
2003 a What's that thing called embodiment? In: Richard Alterman and
th
baum.
2003 b On the role of robot simulations in embodied cognitive science. AISB
Journal 1 (4): 389-399.
Ziemke, Tom and Noel Sharkey
Semiotica 134 (1/4): 701-746.
Ziemke, Tom and Mikael Thieme
2002 Neuromodulation of reactive sensorimotor mappings as a short-term
memory mechanism in delayed response Tasks. Adaptive Behavior
10 (3/4): 185-199.
Zlatev, Jordan
ory of meaning. Evolution o/Communication 4 (2): 253-296.
this vol. Embodiment, language and mimesis.
Zlatev, Jordan, Tomas Persson and Peter Gardenfors
2005 Bodily mimesis as "the missing link" in human cognitive evolution,
LUCS 121. Lund: Lund University Cognitive Studies.
Making sense of embodied cognition: Simulation
theories of shared neural mechanisms for
sensorimotor and cognitive processes
1
Henrik Svensson, Jessica Lindblom and Tom Ziemke
Abstract
Although an increasing number of researchers are convinced that cognition is em-
bodied, there still is relatively little agreement on what exactly that means. Notions
of what it actually means for a cognizer to be embodied range from simplistic ones
such as "being physical" or "interacting with an environment" to more demanding
ones that consider a particular morphology or a living body prerequisites for em-
bodied cognition. Based on experimental evidence from neuroscience, psychology
and other disciplines, we argue that a key to understanding the embodiment of
cognition is the "sharing" of neural mechanisms between sensorimotor processes
and higher-level cognitive processes. The latter are argued to be embodied in the
sense that they make use of (partial) simulations or emulations of sensorimotor
processes through the re-activation of neural circuitry also active in bodily percep-
tion and action.
Keywords: action, embodied cognition, gesture, intersubjectivity, language, mirror
neurons, perception, simulation theories.
1. Introduction
Although an increasing number of researchers are convinced that cognition
is embodied, there still is relatively little agreement on what exactly that
means. Notions of what it actually means for a cognizer to be embodied
range from simplistic ones such as "being physical" or "interacting with an
environment" to more demanding ones that consider a particular morphol-
ogy or a living body prerequisites for embodied cognition (cf., e.g., Ander-
1. This is an extended and revised version ofSvensson and Ziemke (2004).
242 Henrik Svensson, Jessica Lindblom and Tom Ziemke
son 2003; Chrisley and Ziemke 2003; Rohrer this volume; Wilson 2002;
Ziemke 2003). This lack of agreement or coherence, after two decades of
research on embodied cognition, has unfortunate consequences. Firstly,
critics commonly argue that the only thing that embodied cognitive theo-
ries have in common is, in fact, the rejection of traditional, computational-
ist and supposedly disembodied cognitive science. Secondly, there is a
certain trivialization of embodiment, not least among many AI researchers
who consider as embodied any physical system, or in fact any agent that
interacts with some environment, such that the distinction between com-
putationalist and embodied cognitive theories disappears since, in some
sense, all systems are embodied, and thus cognitive science has always
been about embodied cognition (Chrisley and Ziemke 2003; Ziemke 2004).
Thirdly, there is the "misunderstanding" that perhaps embodiment is only
relevant to sensorimotor processes directly involving the body in percep-
tion and action, while higher-level cognition might very well be computa-
tional in the traditional sense and only dependent on the body in that men-
tal representations ultimately need to be grounded in sensorimotor
interaction with the physical environment.
Instead, we argue that the key to understanding the embodiment of cog-
nition, in an important, non-trivial sense, is to understand the "sharing" of
neural mechanisms between sensorimotor processes and higher-level cog-
nitive processes. Based on experimental evidence from a range of disci-
plines, we argue that many, if not all, higher-level cognitive processes are
body-based in the sense that they make use of (partial) simulations or
emulations
2
of sensorimotor processes through the re-activation of neural
circuitry that is also active in bodily perception and action (cf. Clark and
Grush 1999; Hesslow 2002; Grush 2004). As Barsalou, Solomon and Wu
(2003: 45) put it, the main point is that "simulations of bodily states in
modality specific brain areas may often be the extent to which embodiment
is realized".
The next section describes the idea of cognition as body- and simula-
tion-based in more detail. Section 3 presents empirical evidence that
strongly suggest a role of sensorimotor simulation in mental imagery (3.1
and 3.2), agent-object interaction (3.3), social cognition (3.4 and 3.5) and
2. The terms simulation and emulation are used somewhat interchangeably in this
paper, as in much of the literature, but it should be noted that they are some-
times used differently (e.g., Grush 2004).
Making sense ofembodied cognition 243
language (3.6 and 3.7). The final section then briefly discusses some open
questions and directions for future work.
2. Cognition as body-based simulation
The idea that even higher-level cognitive processes are in a strong sense
grounded in bodily activity and experience is, of course, hardly new. It was
developed already in the 1980s, most influentially by Maturana and Varela
(1980, 1987; cf. Varela, Thompson and Rosch 1991) from a neurobiologi-
cal perspective, and by Lakoff and Johnson (1980, 1999) from a linguistic
perspective. Lakoff (1988: 121) summarized the basic idea as follows:
Meaningful conceptual structures arise from two sources: (1) from the
structured nature of bodily and social experience and (2) from our innate ca-
pacity to imaginatively project from certain well-structured aspects of bodily
and interactional experience to abstract conceptual structures.
Back in the 1980s, however, relatively little was known about exactly how
such an imaginative projection from bodily experience to abstract concepts
might work. In recent years more detailed accounts of how the sensorimo-
tor structures of the brain are involved in cognition have been developed in
several disciplines, often taking into account data from neurophysiological
and neuroimaging studies (cf. Johnson and Rohrer this volume). These
accounts show that the traditional strong division between perception and
action, as well as between sensorimotor and cognitive processes, needs to
be revised.
A particular kind of "embodiment" theory that has emerged in different
contexts are the so-called emulation or simulation theories (e.g., Barsalou,
Solomon and Wu 2003: Decety 1996; Frith and Dolan 1996; Grush 2003,
2004; Gallese 2003a; Hesslow 1994, 2002; Jeannerod 1994, 2001). The
basic idea is that neural structures that are responsible for action and/or
perception are also used in the performance of various cognitive tasks. As
Hesslow (2002) pointed out, this idea is not entirely new; Alexander Bain,
for example, suggested back in 1896 that thinking is basically a covert
form of behavior that does not activate the body and thus remains invisible
to external observers. Today simulation theories, based partly on data from
neuroscience, can further clarify the possible role of simulation in cogni-
tion, and thus explain in a more concrete way than before the embodiment
of cognition.
Hesslow (2002) proposed that cognition can to a large extent be ex-
plained by simulated chains of covert behavior. More precisely, at least
human brains
3
, from a certain age, have the ability to reactivate previous
perceptions and actions in the absence of any sensory input or overt
movement. These simulations of actions or perceptions are through, for
example, conditioning coupled to achieve internal simulations of organism-
environment interactions (Hesslow 2002). A similar, technically more de-
tailed account of the general idea has recently been formulated by Grush in
his emulation theory of representation (Grush 2003, 2004; see also Clark
and Grush 1999). Based on the control-theoretic concept offorward mod-
els (emulators), previously used to account for motor control (e.g., Wolpert
and Kawato 1998), Grush developed an emulation theory for several types
of cognitive processes, including perception, imagery, reasoning and lan-
guage. In a nutshell, he argued that emulation circuits are able to calculate
a forward mapping from control signals to the (anticipated) consequences
of executing the control command. For example, in goal-directed hand
movements the brain has to plan parts of the movement before it starts. To
achieve a smooth and accurate movement proprioceptive/kinesthetic (and
sometimes visual) feedback is necessary, but sensory feedback per se is too
slow to affect control appropriately (Desmurget and Grafton 2000). The
"solution" is an emulator/forward model that can predict the sensory feed-
back resulting from executing a particular motor command. A further pre-
diction is that the emulator circuits are achieved by the reactivation of the
same sensorimotor processes that are used in overt action and perception
(e.g., Grush 2004; Hesslow 2002; Jeannerod 2001)4.
Connecting simulated perceptions and actions by an anticipatory
mechanism might also explain certain ways/types of problem solving
(Hesslow 2002). An example of this kind of simulation is possibly the
problem solving and planning involved in the Tower-of-London problem
(Shallice 1982), a task that requires subjects to manually put objects on top
of each other under certain non-trivial constraints that require planning
ahead. Dagher et al. (1999) found that even seemingly purely mental plan-
3. To which degree animals are capable of so-called "mental time travel", i.e.,
recollection of specific past events or anticipation of the future, is still an open
question. For a detailed recent discussion see Clayton, Bussey and Dickinson
(2003).
4. According to Blakemore, Frith and Wolpert (1999), this is also why for most
people it is not so easy to tickle themselves: the forward model produces pre-
dicted sensory feedback that "prepares" the agent.
ning and problem solving, without physical object manipulation, activated
higher motor areas (premotor cortex, prefrontal cortex) and the basal gan-
glia, which seemed to interact with visual and posterior parietal areas (cf.
Schall et al. 2003). This gives some support to the idea that the subjects
solved the problem by simulating the moving around of objects through the
use of reactivated (or simulated) perceptions and actions (Hesslow 2002).
In other words, the simulation account argues that cognitive processes
are achieved by the reactivation of the same neural structures as used for
physically sensing, moving and manipulating the environment. The fol-
lowing section summarizes a number of the many empirical studies that
support the idea that cognition is body-based, especially as predicted by
simulation theories.
3. Empirical evidence
Several sources of evidence support the basic tenet of the simulation ac-
count, i.e., the idea that perceptual and motor areas of the brain can be
covertly activated either separately or in sequence for use in cognitive pro-
cesses. Moreover, there are a number of accounts that implicate simulation
on multiple levels of cognitive complexity from motor imagery to lan-
guage. The experiments mentioned here range from mental chronometry
studies to functional imaging studies of animal and human brains.
The following subsections review some of the empirical evidence that
suggest that sensorimotor structures of the brain are deeply involved in the
generation of cognitive phenomena, such as imagery and problem solving.
The starting point is the extensive similarities found between the neural
structures activated during preparation (and execution) of an action and
mentally simulating an action (i.e., motor imagery), as well as between
visual perception and visual imagery. These similarities are so striking that
some have posited that internally activated actions and perceptions are the
same as overt ones, but without actual sensory input or overt movement
(e.g., Hesslow 2002; Jeannerod 2001).
3.1. Motor imagery
There has been extensive research in the last couple of decades into the
relation between motor imagery and the preparation and execution of ac-
tions (Jeannerod 2001; Johnson 2000). A large number of behavioral and
neurophysiological experiments have shown that motor imagery and the
mechanisms involved in the planning and production of overt actions share
a large number of properties (e.g., Decety, Jeannerod, and Prablanc 1989;
Jeannerod and Decety 1995; Jeannerod and Frak 1999; for reviews see
Decety 1996, 2002; Jeannerod 1994, 2001).
Motor imagery is usually defined as the recreation of an experience of
actually performing an action, e.g., the person should feel as if he or she
was actually walking (Decety 1996; Jeannerod 1994). It is difficult to en-
tirely distinguish or separate motor imagery from visual imagery, since
actions also involves visual consequences (e.g., Jeannerod 1994; see also
Dechent, Merboldt and Frahm 2004). However, a motor image differs from
a visual image in that it is based mainly on kinesthetic/proprioceptive in-
formation about the action, i.e., the subject feels as if performing the ac-
tion, not necessarily involving a visual iconic representation of the action
or the external visual surroundings. Some examples involving a motor im-
age would be imitating somebody's movements, anticipating the effects of
an action and having kinesthetic or bodily sensations (e.g., muscle contrac-
tions, heart beats) (Jeannerod 1994). In fact, the term motor imagery is
sometimes used in a wider sense to mean an unconscious form of motor
imagery such as when subjects are tested on tasks that, for example, require
making judgments about actions, but not necessarily evoke a feeling of
performing an action (cf., Jeannerod and Frak 1999). For instance, Frak,
Paulignan and Jeannerod (2001) manipulated the positions to place fingers
on a cup of water and asked subjects whether it would be "easy", "diffi-
cult", or "impossible" to grasp the cup and pour the contents into another
container.
5
They found that subjects rated grasp executions near the limits
of what is a physically possible grasping action as difficult, whereas posi-
tions leading to grasp executions that are preferred when subjects actually
perform the action were rated as easy.
Furthermore, the response times increased with the estimated difficulty
of the task. The interpretation was that subjects were mentally simulating
performing the action in order to determine its feasibility. Thus, subjects
seemed to rely on an unconscious form of motor imagery. However, to
avoid confusion this is better described in terms of the subjects relying on
simulations of actions, i.e., reactivations of motor areas responsible per-
forming an action but without any overt movement. Motor imagery should
5. For a closely related study see also Johnson (2000).
be used to denote the, at least, partly conscious act of feeling as if one were
performing an action, which in turn might be based on the more general
(neural) mechanism of simulation of action. In fact, there is considerable
evidence that motor structures are reactivated in several phenomena closely
related to motor imagery, such as intentions to act, judging the feasibility
of an action, determining by observation whether an object is graspable,
and actions in dreams (Gallese 2003a; Jeannerod 2001). Since imagined
actions or motor imagery have been shown to share many properties with
actual actions, that will be the focus our brief review of the current evi-
dence for similarities between overt actions and mental actions.
6
The em-
pirical evidence cited in support for the (partial) equivalence of overt ac-
tions and motor imagery comes mainly from mental chronometry,
physiological responses, measurements of brain activity and lesion studies.
Mental chronometry experiments, which measure the duration of be-
havioral and mental responses, have found that the time needed to mentally
execute actions in several conditions closely corresponds to the time it
takes to actually perform them (Jeannerod and Frak 1999; Papaxanthis,
Pozzo et al. 2002; Papaxanthis, Shieppati et al. 2002; for a review see
Guillot and Collet 2005). For example, Decety and Jeannerod (1996) found
that Fitt's law (i.e., the finding that execution times increase with task dif-
ficulty) also holds for motor imagery. Decety et al. (1989) compared the
durations of walking towards targets (with blindfolds) placed at different
distances and mental simulation of walking to the same targets. In both
conditions times were found to increase with the distance covered.
Besides producing similar reaction times motor imagery has been
shown to produce similar physiological effects, in the form of muscle
6. Johnson (2000) has pointed out that it is important in experiments on motor
imagery not to ask subjects explicitly to use motor imagery in order to minimize
experimenter effects. However, the current issue does not concern if and in
which situations people rely on motor imagery, where Johnson's concern is cor-
rect; rather it is if motor imagery and other related (cognitive) phenomena rely
on simulations of actions as defmed above. Consequently, some studies re-
ported here might be better characterized as employing a covert form of motor
imagery, i.e., simulation of action rather than actual conscious awareness of
imagining an action. Deiber et al (1998) pointed out a related issue concerning
the lack of explicit control of whether visual imagery was involved in the stud-
ies. A hypothesis is that if simulations of actions are coupled with anticipatory
mechanisms, simulation would be a more pervasive property of cognition, not
only related to imagery, as will be seen throughout this chapter.
strength and autonomic responses, as overt actions. Sport psychological
experiments have revealed that mentally practicing a specific action can
enhance the performance when subsequently actually performing the pre-
viously mentally imagined action (Jeannerod 1994). For example, Vue and
Cole (1992; cf. Ranganathan et al. 2004) showed that mentally simulating
oneself contracting a muscle could significantly (by 22%) increase muscle
strength? (actual training produced an increase of 30%). Jeannerod (1994)
pointed out that there have been several interpretations of these results, for
example, that motivational factors increase the physiological arousal and as
result of this performance is increased. Another explanation is that if motor
imagery, at least in part, involves the same neural structures responsible for
overt action it is probable that motor imagery could "train" the neural
structures used in subsequent execution of the previously imagined action
(Jeannerod 2001; Ranganathan et al. 2004; Vue and Cole 1992). Besides
strength increases, autonomic responses, such as the adaptation of heart
and respiratory rates, which are beyond voluntary control, have been
shown to be activated by motor imagery to an extent proportional to that of
actually performing the action, and as a function of mental and actual
physical effort (Decety 1996; Jeannerod 1994; Jeannerod and Decety
1995). For example, Decety et al. (1991) showed that when subjects imag-
ined performing a leg exercise their heart and respiratory rates increased.
Since the first study that investigated motor imagery using regional
cerebral blood flow (rCBF) to indicate active brain areas (Ingvar and
Philipson 1977), there have been many neuroimaging experiments that
confirm the first study's indication that similar brain areas are activated in
overt actions and motor imagery. Together with results from studies on
neurological disorders (e.g., Jeannerod and Decety 1995), these experi-
ments have, with some discrepancies, found motor imagery to involve
structures primarily associated with the execution of actions, such as pri-
mary motor cortex, premotor cortex, supplementary motor area, lateral
cerebellum and the basal ganglia, as well as those primarily associated with
action planning, such as, the dorso-Iateral prefrontal cortex, inferior frontal
cortex and posterior parietal cortex (Grezes and Decety 2001; Jeannerod
2001; Jeannerod and Frak 1999; Schwoebel, Boronat and Coslett 2002).
Even though there is a strong similarity between overt action and simula-
tion of action, the actual degree of overlap is not yet fully understood. For
7. More precisely, the voluntary force production of the fifth digit's metacarpo-
phalangeal joint (Yue and Cole 1992).
example, subjects in mental chronometry experiments tend to over- or un-
derestimate durations of actions in some conditions (Guillot and Collet
2005) and activation of the primary motor cortex is not found in every
neuroimaging study on motor imagery (Dechent, Merboldt and Frahm
2004; Grezes and Decety 2001).
To summarize, there is a substantial amount of research, which suggests
that there are strong psychological and neurophysiological resemblances
between overt actions and motor imagery. Thus, motor imagery and associ-
ated phenomena might best be explained in terms of simulations of actions,
i.e., neural processes normally used to produce overt actions are reactivated
by motor imagery, with the overt movement inhibited.
Although the focus so far has been on actions, it should not be forgotten
that the motor system also integrates sensory information when planning
and executing an action (e.g., Grush 2004; cf. Desmurget and Grafton
2000; Jeannerod 1997). Thus, simulating an action might also involve an
emulator mechanism (forward model) that predicts the proprioceptive
feedback and sometimes visual feedback that would have resulted from the
executed action to produce the (conscious) feeling of mentally imagining
performing an action (Decety 2002; Grush 2004; Jeannerod 1997, 2001).
Also, there are a number of unanswered questions concerning motor im-
agery worth mentioning here (which, however, do not affect the chapter's
main point concerning the embodiment of cognition). For example, to what
degree do actions and mental simulations of actions engage executive mo-
tor structures (such as, the primary motor cortex) (cf., e.g., Decety 2002;
Jeannerod and Frak 1999), and how is the overt movement "hindered"
(Jeannerod 2001; Hesslow 2002)? Although there may not be a complete
overlap between the neural structures involved in real and mentally simu-
lated action, we believe that the evidence suggests that they are not differ-
ent in nature, but only in degree.
3.2. Visual imagery
The discussion of motor imagery can be extended to the visual modality in
that similar types of studies report that perceptual structures can be and are
internally reactivated when, e.g., visually recreating a previous perception.
Many studies in cognitive psychology have found such similarities between
visual perception and visual imagery (Farah 1988; Finke 1989). For exam-
ple, in a seminal study by Shepard and Metzler (1971), subjects had to
determine whether two three-dimensional forms had the same shape or not.
Besides the introspective reports of the subjects that they had mentally
rotated three-dimensional forms to see if they were the same, the results
showed that reaction times increased linearly with the angular difference,
which meant that the imagined rotations were performed at a constant rate
(cf. Finke 1989). Furthermore, they found reaction times not to be longer
for depth rotations than for rotations in the picture plane. These two find-
ings suggest that imagined rotations in some aspects correspond to actual
physical rotations of objects (Finke 1989).
Although alternative explanations are difficult to rule out, such as ex-
perimenter effects or tacit knowledge (cf. Finke 1989), neuropsychological
and neuroimaging studies offer more conclusive evidence of the involve-
ment of similar areas of the brain in mental visual imagery and visual per-
ception (Farah 1988,2000; Hesslow 2002; Kosslyn and Thompson 2000).
However, as in the case of motor imagery, the overlap is neither complete
nor uniform across experiments, which to some extent might be explained
by differences between types of imagery, and resolution of mental images
(Ganis, Thompson and Kosslyn 2004; Kosslyn and Thompson 2000; Tro-
jano et al. 2004; see also Mazard et al. 2004). In line with simulation theo-
ries, Wexler, Kosslyn and Berthoz (1998: 92) suggested that at least visual
mental imagery in the form of mental rotation is achieved by a "prediction
of an about-to-be-executed motor action". Their hypothesis has also been
supported by functional brain imaging studies (Vingerhoets et al. 2002).
The so-called mental imagery debate in cognitive science between those
arguing for a picture theory (e.g., Kosslyn 1994) and those arguing for a
description theory (e.g., Pylyshyn 1981), which has recently reemerged
(Kosslyn, Ganis and Thompson 2003; Pylyshyn 2003; see also Thomas
1999), is not of central importance for the discussion in this chapter since
both perception and imagery may be said to use the same format. That is,
even though there are strong similarities between properties of imagery and
of perception both may be explained using either theory (Block 1983; see
also Pylyshyn 2003). On the other hand, explaining cognition as
reactivation of sensorimotor structures does not (at least in some cases)
rely on the computer metaphor of symbol manipulation (cf. Lindblom and
Ziemke this volume), and thus may offer a novel view that does not see the
vehicle and the content of representations as separate entities but as con-
stitutive of each other (Gallese 2003b; cf. Dreyfus 2002; Thomas 1999).
The representations of embodied cognitive theories are not the same as the
amodal symbol systems proposed by classical theories, but are grounded in
bodily interaction with an environment. According to Gallese (2003b),
canonical neurons in the monkey brain illustrate how the interaction be-
tween an agent and its environment provides an example of such represen-
tations.
3.3. Canonical neurons
The discovery of so called mirror neurons and canonical neurons in the
macaque monkey8 (di Pellegrino et al. 1992; Murata et al. 1997; Rizzolatti
et al. 1996) have resulted in a number of different theories about their role
in primate and human cognition (cf. Johnson and Rohrer this volume; Gal-
lagher this volume; Lindblom and Ziemke this volume). Canonical neurons
(and mirror neurons, see Section 3.5 below) have been found in the rostral
region of the inferior premotor cortex (area F5) of the monkey brain which
contains neurons that are known to discharge during goal directed hand
movements, such as grasping, holding, tearing, or manipulating. However,
they are not responsive to similar movements, but only actions that have
the same "meaning" (di Pellegrino et al. 1992; Rizzolatti et al. 1996; Riz-
zolatti et al. 2002), which is why they are often interpreted as internal rep-
resentations of actions, rather than motor or movement commands (Jean-
nerod 1994; Rizzolatti et al. 1996; Rizzolatti et al. 2002). Gallese (2003b)
emphasized seeing them as coding not physical parameters of movement,
but a relationship between agent and object.
The so-called canonical neurons of area F5 have both motor properties
and sensory properties, and they discharge both during the action they code
and when an object that affords that action in the Gibsonian sense (cf.
Costall this volume; Sonesson this volume) is perceived. Canonical neurons
have a strict congruence between the type of grasping action and the size or
shape of the object they respond to (Gallese 2003b). This implies that they
implement affordances, e.g., code objects that are graspable-in-a-certain-
way, specifying not only perceptual and action aspects but a particular
relationship between agent and environment (cf. Gallese 2003b, see also
8. For practical and ethical reasons it is so far not possible to investigate the exis-
tence of mirror neurons (and canonical neurons) at the single neuron level in
humans. However, many researchers have presented strong arguments for the
existence of a similar system in humans (e.g., Arbib in press; Fadiga et al. 1995;
Grafton et al. 1996; Grezes et al. 2003; Rizzolatti and Arbib 1998; Rizzolatti et
al. 1996).
Dreyfus 2002). This kind of function has also been suggested as part of
human action and motor imagery (Jeannerod 1994). Jeannerod (1994: 233)
argued that a motor representation not only contains the "plan" that gener-
ates the actual kinesthetic movements but also a pragmatic representation
"in which the visual attributes of objects are thought to be processed as
affordances, that is, on the basis of the extent to which they are related to a
given action directed at these objects". The more general implication is that
the common division between outer and inner, or mind and body/environ-
ment, stemming from the debates between rationalism and empiricism, is
not plausible (cf. Lindblom and Ziemke this volume). The generation of
behavior and of cognition always takes place in the interaction with an
environment and if anything it is this interaction that gets represented by
the subject through dynamic brain processes. Similar simulation mecha-
nisms as those thought to be the basic machinery behind mental imagery
have also been suggested to be an essential part of social cognition, where
mirror neurons might be a key example, as discussed in more detail in the
following subsections.
3.4. The body as an intersubjective resonance mechanism
The research mentioned so far enforces yet another dichotomy of cognitive
science, viz., that of individual versus social, by focusing on individual
cognitive processes. However, even though it may be possible to separate
the two for explanatory purposes, in nature humans and many other ani-
mals are essentially social beings. Blakemore, Winston and Frith (2004),
for example, argued that humans are highly social beings such that much of
the brain must have evolved to handle social communication and interac-
tion. The interest in social cognitive neuroscience, the empirical study of
the neural mechanisms underlying social cognitive processes, has increased
rapidly in recent years. Recent work addressing social aspects of embodi-
ment implies that the body has several important roles in social interactions
(cf. Lindblom and Ziemke this volume). Dautenhahn (1997), for instance,
hypothesized that a phenomenological dimension of social understanding
might be founded in embodied mechanisms that allow biological agents (in
particular humans) to read "social signs" and other agent's mind, by simu-
lating the other agent's emotional stance, and she suggested that the agent's
own body can be used as the point of reference. In line with this remark,
simulation mechanisms have also been suggested to play a vital role for
action recognition (e.g., Gallese 2003a), empathy (e.g., Decety and Chami-
nade 2003), social cognition (e.g., Barsalou et al. 2003; Nielsen 2002; cf.
Lindblom and Ziemke this volume) and even language understanding (e.g.,
Glenberg and Kaschak 2002). Yet, social cognition and its connection to
bodily-based simulation processes are not well understood, and they have
not received much attention in embodied/situated cognitive theories. The
evidence reported in the following subsections emphasizes that perceptual
and motor processes are not different in nature at the neural and behavioral
level, but seem to be intimately linked in social cognition, possibly through
simulation mechanisms.
Barsalou et al. (2003) noted that there are at least four types of wel-
lknown phenomena in social psychology experiments, which can be ex-
plained as simulations of bodily states (cf. Lindblom and Ziemke this vol-
ume; Nielsen 2002). Firstly, perceived social stimuli can produce bodily
states (e.g., a more slumped posture in response to negative feedback).
Secondly, social stimuli can induce bodily mimicry (e.g., a smile in re-
sponse to a smile). Thirdly, bodily states can produce and effect emotional
states (e.g., an upright posture tends to have a positive effect). Finally,
compatibility between bodily states and emotional states leads to increased
cognitive performance (e.g., it is easier to pull a lever towards you in re-
sponse to "positive" stimuli than in response to "negative" ones; cf. also
Section 3.7). These phenomena have in common that states of the body,
such as postures, arm movements and facial expressions, change automati-
cally without any conscious mediating knowledge structures in specific
instances of social interaction (cf. Nielsen 2002).
Roughly speaking, these phenomena imply that bodily states are in-
volved in social cognition and that they might constitute the very founda-
tions of the particular social cognitive phenomena in question. An example
of how perception, action and social cognition come together at the level of
single neurons is so-called mirror neurons in macaque monkeys (Decety
and Sommerville 2003).
3.5. Mirror neurons
Beside canonical neurons, area FS of the monkey brain contains so-called
mirror neurons which have sensory properties that become activated both
when performing a specific action and when observing the same goal-
directed hand (and mouth) movements of an experimenter (di Pellegrino et
al. 1992; Rizzolatti et al. 1996; Rizzolatti et al. 2002). Mirror neurons pro-
vide a key example of sensorimotor brain structures also involved in (so-
cial) cognitive processes.
Although different hypotheses exist, many of the theories of the func-
tion of mirror neurons emphasize their role in social cognition (e.g., Gal-
lese and Goldman 1998; Gallese, Keysers and Rizzolatti 2004; Rizzolatti
and Arbib 1998; Rizzolatti et al. 2002). These researchers aclmowledge
that area F5 and mirror neurons can be interpreted as a kind of observation-
execution mechanism or resonance mechanism, which links the observed
actions to actual actions of the subject's own behavioral repertoire. That is,
it enables the agent to understand the meaning of the observed action by
simulating the observed action through its own sensorimotor processes.
Thus, mirror neurons can be interpreted as representations of actions, used
both for performing and understanding actions (e.g., Rizzolatti et al. 1996;
Rizzolatti and Arbib 1998).
Gallese and Goldman (1998) hypothesized that mirror neurons might be
a basic mechanism necessary for "mind-reading", i.e., attributing mental
states to others. In particular, they argued that such mechanisms can ex-
plain how an agent determines what mental states of another agent have
already occurred. When mirror neurons are externally activated by ob-
serving a target agent executing an action (allowing the subject to evaluate
the meaning of the other's action), the subject lmows (visually) that the
observed target is currently performing this very action and thereby "tags"
the "experienced" action as belonging to the target.
Brain imaging experiments with human subjects sitting still observing
others moving have indicated that the mirror system seems to distinguish
between biological and non-biological actions (Blakemore, Winston and
Frith 2004). It is commonly argued that another person's action can influ-
ence one's own actions, and Sebanz et al. (2003) showed that when a sub-
ject carried out a spatial compatibility task, the presence of another person
altered the timing of the response time. Moreover, observation of another
person's actions has an impact on one's own actions, and interference ef-
fects occur when there is a mismatch between one's own actions and the
observed ones (Blakemore, Winston and Frith 2004). However, these inter-
ference effects seem to occur only for observed human actions and not
while observing a robot making interfering actions (Kilner, Paulignan and
Blakemore 2003). Blakemore, Winston and Frith (2004) asked what is
special about human biological actions and why mirror systems require
biological action to be activated. Furthermore little is lmown about how the
subject can distinguish its own actions from those performed by others,
given that to large extent the same neural mechanisms are underlying both
action observation and one's own action (cf. Blakemore, Wolpert and Frith
2002). However, this might be possible to resolve in the future using tech-
niques for dual scanning of two brains which would facilitate the recording
of simultaneous responses of two interacting humans (Blakemore, Winston
and Frith 2004).
3.6. Gesture and language
In addition to action-recognition, mirror neurons are also considered to be
involved in more complex actions, such as gestures. Rizzolatti and Arbib
(1998) suggested that the human mimetic capacity (cf. Donald 1991) is a
natural extension of action-recognition based on mirror neuron mecha-
nisms, allowing human ancestors to communicate to a higher degree than
other primates. For instance, they pointed out that empirical studies suggest
that a mirror system for gesture recognition also exists in humans and is
situated in Broca's area (a homolog to area F5 in the monkey). Premotor
areas are activated both when performing an action and when observing
another person performing an action. According to Rizzolatti and Arbib, a
series of mechanisms are usually activated in order to inhibit the actual (re-
) production of the observed action. Occasionally, however, the premotor
system will allow a tiny aspect of the simulated movement to be executed,
and this short glimpse is recognized by the other person, affecting both the
actor and the observer. That means, the actor recognizes it as an intention
in the observer, and the observer notices that her (involuntary) response, in
turn, affects the behavior of the actor (Rizzolatti and Arbib 1998). That
means, the mirror system provides the causal mechanisms for basic inten-
tional interaction and thus might constitute the foundation for human lan-
guage (cf. Arbib 2005).
Iverson and Thelen (1999) examined the role of embodiment in lan-
guage, noting that gesture is a pan-human ability in communication and
that gestures are tightly connected and synchronized with speech. Further-
more, they pointed out that gestures provide important communicative
information to the listener and even blind people gesture while talking to
others, even when talking to blind listeners (cf. Goldin-Meadow 2003;
Iverson and Goldin-Meadow 1998). Iverson and Thelen (1999) linked
gestures and language together from the perspective of embodiment, and
presented three types of empirical evidence (see also Section 3.7). Firstly,
some language and motor functions share the same underlying brain
mechanisms. Ojemann (1984), for example, demonstrated that there seems
to be a common brain mechanism for sequential movement and speech
production located in the same area in the brain. Moreover, Fried et al.
(1991) pointed out that there are indications that the vocal tract, and the
hands and arms are represented in closely related sites in certain brain ar-
eas. Secondly, some of the brain regions normally associated with motor
functions have been shown to be involved in language tasks (cf. Pulver-
muller et al. 1996). Furthermore, classical "language areas" become acti-
vated during motor tasks (e.g., Bonda et al. 1994; Krams et al. 1998). Fi-
nally, there seems to be a close link between patterns of collapse and
recovery in certain motor and language functions in some type of patients.
For instance, language breakdowns in patients suffering from aphasia show
a parallel dysfunction in gesturing (cf., e.g., Hill 1998). Moreover, there
are close connections between the oral and manual systems in the infant at
birth, e.g., the Babkin reflex which makes newborn babies open their mouth
if pressure is applied to their palm. Furthermore, gesturing has been shown
to have positive effects on language development in infants (cf. Goodwyn
and Acredolo 1998). Taken together, there exists converging empirical
evidence that the systems of hand and mouth movements are not separate
systems; rather they should be viewed as intimately linked in language
production. Thus, from the perspective of embodied cognition, the simula-
tion mechanisms grounded in the mirror system might function as the glue
that binds hand, mouth and language together.
3.7. Language as embodied simulation
Some researchers have argued that conceptualization and language under-
standing cannot be achieved through the manipulation of amodal, arbitrary
symbols alone but have to be grounded in bodily interaction with an envi-
ronment. In particular, Glenberg and Kaschak (2003) have outlined an
explanation of language in line with the ideas of cognition as body-based
simulation as expressed in this paper, suggesting that language is partly
achieved through the same neural structures as used to plan and guide ac-
tion.
Under the heading of the indexical hypothesis they developed an ac-
count of language comprehension partly based on simulation of action.
They argued that the meaning of a sentence is achieved by a process that
indexes words to perceptual symbols, i.e., modal symbols based on records
of the neural states that underlie perception (Barsalou 1999), which in turn
retrieves the available affordances in the situation and determines their
relevance through the particular sentence construction. Thus, the under-
standing of a sentence is essentially achieved through a simulation of ac-
tion using the same neural systems active in overt behavior.
An empirical result that supports the close coupling between language
and action is the "action-sentence compatibility effect" (Glenberg and Kas-
chak 2002). It was found that the sensibility of a sentence is modified by
physical actions. Reaction times increased when subjects read "toward
sentences" that implied action toward the reader, such as "Open the
drawer" and had to give the answer through an incongruent action, i.e.,
moving the hand away from the body. Conversely, when subjects answered
through an action congruent with the sentence, reaction times decreased. It
might be worth noting that Glenberg and Kaschak included not only sen-
tences describing concrete, physical transfers, but also sentences describing
cases of abstract transfer, such as "Liz told you the story" (2002: 560). The
action-sentence compatibility effect was also present when reading these
more abstract sentences.
Further support comes from experiments on language comprehension
and construction that are only explainable by implicating perception and
action systems as predicted by the indexical hypothesis (Glenberg and
Kaschak 2003). To give but one example, Barsalou, Solomon and Wu
(1998) described an experiment which showed that presenting a modifier
that potentially reveals internal features has an effect on feature listing not
predicted by standard amodal approaches. The standard models predict that
listing the features of half a watermelon opposed to a whole watermelon
would only differ with regard to amount, i.e., a half watermelon is smaller
than a whole watermelon. The experiment showed, however, that subjects
listed more internal features such as seeds, which can be explained if the
concepts are based on perceptual symbols. Readers interested in more
comprehensive reviews, also including neurophysiological evidence, of the
coupling between language and action/perception are referred to Glenberg
and Kaschak (2003) or Zwaan (2004) (see also Johnson and Rohrer this
volume; Rohrer this volume; Zlatev this volume).
4. Discussion
This paper has presented an emerging framework of simulation theories,
based on terminology and ideas from control theory as well as data from
psychological, neurophysiological and brain imaging studies, that explains
higher-level cognitive processes as - at least partly - based on reactivations
of sensorimotor brain structures. By reactivating mechanisms used in bod-
ily perception and action together with a predictive mechanism a flexible
inner world emerges that can be used for many different higher-level cog-
nitive tasks (cf. Grush 2004; Hesslow 2002). Crucial to the embodiment of
cognition, according to this account, is not so much the physical nature of a
cognizer's body or its interaction with the environment as such, but the
relation between sensorimotor and higher-level cognitive processes, more
specifically, the way that the latter are fundamentally based on and rooted
in the former at the level of the neural mechanisms underlying both of
them.
According to Rizzolatti and Arbib (1998), during the course of evolu-
tion, the capacity to voluntarily control one's own mirror system to emit
signals, instead of the mere automatic leaking of parts of the mirrored ac-
tions (cf. Section 3.6), was essential for the emergence of a (basic) dia-
logue between two individuals which forms the core of language. Rizzolatti
and Arbib further speculated that this new capacity of the mirror system
was initially based on oro-facial movements, given that all primates mainly
communicate through oro-facial movements. Later on, manual gestures
were added, as a way of complementing the oro-facial ones, since gestures
increased the sender's expressive power. The combination of oro-facial
movements and gestures, according to Rizzolatti and Arbib, strongly im-
plies the importance of controlled vocalization as an extension of oro-facial
movements and gestures. The evolutionary pressure for more complex
sound emission, together with the anatomical possibilities, resulted in a
move of intentional interaction from its oro-facial and gestural origins to
sound emission (cf. Corballis 1999). This might provide a tentative expla-
nation of why and how the human Broca's area emerged from area F5, its
homolog in the monkey. However, much further work is needed in order to
clarify the relation between simulation mechanisms, gestures and the
emergence of language.
Although corroborating evidence comes from several disciplines and
different experimental paradigms, the simulation account is not yet a well
established or coherent theory of cognition in general, and there are many
questions still to be answered. For example, in current accounts it is un-
clear exactly what constitutes the difference between an executed, overt
action and a simulated/imagined, covert one. Can this be accounted for in
terms of simulation theories or are other, presumably higher-level, mecha-
nisms required after all to selectively trigger one or the other?
A closely related question is exactly what it is that simulation accounts
are accounts of? Do they constitute alternative theories of representation
(only), as Grush (2004) seems to argue, or are they intended as more en-
compassing theories of cognition and representation, as probably Hesslow
(2002) would argue? There seem to be good arguments for both positions.
Empirical results, such as Glenberg and Kaschak's (2002) finding of an
action-sentence compatibility effect even for abstract expressions (cf. Sec-
tion 3.7) that cannot directly be explained in terms of perception and ac-
tion, seem to indicate, as does much work in cognitive linguistics (cf. John-
son and Rohrer this volume; Rohrer this volume), that even much, if not all,
abstract thought and language is (metaphorically) grounded in embodied
simulations. On the other hand, Markman and Brendl (2005), for example,
showed that the type of embodiment effects presented by Barsalou et al.
(2003) are not always tied to the subject's body, but sometimes the actions
and corresponding effects are performed in relation to a non-physical in-
stantiation of the self (i.e., moved away from the subject's physical body).
In such cases the mere simulation of actions, according to Markman and
Brendl, is not sufficient for explaining the phenomena, since actions are
usually tied to the subject's body and egocentric perspective.
The perhaps most crucial issue, at least partly also underlying the above
questions, is the problem of the right level ofgranularity or abstraction (cf.
Meltzoff and Prinz 2002; Ziemke, Jirenhed and Hesslow 2005). That is, at
what level of abstraction does the simulation occur? In the case of imagery
it seems that the simulation occurs on a low-level including very many of
the aspects of actually perceiving or acting, as indicated by neuroimaging
studies and the fact that motor imagery also has physiological effects (e.g.,
Jeannerod 2001). In problem solving, on the other hand, more abstract
aspects of actions may be employed, as indicated by the finding that
Tower-of-London problem solving activity seems to activate only higher
motor centers, such as prefrontal and premotor cortex (Dagher et al. 1999;
cf. Section 2). However, this is a speculative interpretation of the neuroi-
maging results. The problem of granularity has also been suggested to be of
importance in robotic models, which so far largely have been limited to the
lowest level (cf. Stening, Jacobsson and Ziemke 2005; Ziemke, Jirenhed
and Hesslow 2005), and in studies of language understanding. Glenberg
and Kaschak (2003), for example, pointed out that it is unlikely that in
comprehending a sentence all aspects of the situation need to be simulated.
But exactly what the crucial aspects are is still unclear.
Hence, in conclusion one might say that, although much work remains
to be done, simulation theories have come a long way in challenging tradi-
tional theories of cognition and representation, and there is an impressive
wealth of corroborating empirical evidence from different disciplines. In
particular simulation accounts clarify what it might mean for cognitive
processes to be embodied in a strong sense, and thus provide an alternative
to more conservative theories that try to integrate physical embodiment as a
mere constraint into the traditional functionalist framework.
References
Anderson, Michael L.
130.
Arbib, Michael A.
2005 From monkey-like action recognition to human language: An evolu-
tionary framework for neurolinguistics. Behavioral and Brain Sci-
ences 28: 105-167.
Barsalou, Lawrence W.
1999 Perceptual symbol systems. Behavioral and Brain Sciences 22: 577-
660.
Ruppert
2003 Social embodiment. In: Brian H. Ross (ed.), The Psychology of
Press.
Barsalou, Lawrence W., Karen Olseth Solomon and Ling-Ling Wu
1999 Perceptual simulation in conceptual tasks. In: Masako K. Hiraga,
Christopher Sinha and Sherman Wilcox (eds), Cultural, Typological
and Psychological Perspectives in Cognitive Linguistics, 209-228.
Amsterdam: John Benjamins.
Blakemore, Sarah-Jayne, Chris D. Frith and Daniel M. Wolpert
1999 Spatio-temporal prediction modulates the perception of self-
produced stimuli. Journal ofCognitive Neuroscience 11: 551-559.
2002
Decety, Jean
1996
Blakemore, Sarah-Jayne, Joel Winston and Uta Frith
2004 Social cognitive neuroscience: where are we heading? Trends in
Blakemore, Sarah-Jayne, Daniel M. Wolpert and Chris D. Frith
2002 Abnormalities in the awareness of action. Trends in Cognitive Sci-
ences 6: 237-242.
Block, Ned
1983 Mental pictures and cognitive science. Philosophical Review 93:
499-542.
Bonda, Eva, Michael Petrides, Stephen Frey and Alan C. Evans
1994 Frontal cortex involvement in organized sequences of hand move-
ments: evidence from positron emission tomography studies. Society
for Neurosciences Abstract 20: 353.
Costall, Alan
agency.
London: Macmillan.
Clark, Andy and Rick Grush
1999 Towards a cognitive robotics. Adaptive Behavior 7 (1): 5-16.
Clayton, Nicola S., Timothy J. Bussey and Anthony Dickinson
2003 Can animals recall the past and plan for the future? Nature Reviews
Neuroscience 4: 685-691.
Corballis, Michael C.
1999 The gestural origin of language. American Scientist 87 (2): 138.
Dagher, Alain, Adrian M. Owen, Henning Boecker and David J.Brooks
1999 Mapping the network for planning. Brain 122: 1973-1987.
Dautenhahn, Kerstin
1997 I could be you: The phenomenological dimension of social under-
standing. Cybernetics and Systems 25 (8): 417-453.
Do imagined and executed actions share the same neural substrate?
Cognitive Brain Research 3: 87-93.
Is there such a thing as functional equivalence between imagined,
observed and executed action. In: Andrew Meltzoff and Wolfgang
Prince (eds.), The Imitative Mind: Development, Evolution and Brain
Bases, 291-310. Cambridge, MA: Cambridge University Press.
Decety, Jean and Thierry Chaminade
2003 Neural correlates of feeling sympathy. Neuropsychologia 41: 127-
138.
Decety, Jean and Marc Jeannerod
1996 Mentally simulated movements in virtual reality. Behavioural Brain
Research 72: 127-134.
Decety, Jean, Marc Jeannerod and Claude Prablanc
1989 The timing of mentally represented actions. Behavioural Brain Re-
search 34: 35-42.
Decety, Jean, Marc Jeannerod, M. Germain and J. Pastene
1991 Vegetative response during imagined movement is proportional to
imagined effort. Behavioural Brain Research 42: 1-5.
Decety, Jean and Jessica A. Sommerville
2003 Shared representations between self and other. Trends in Cognitive
Sciences 7: 527-533.
Dechent, Peter, Klaus-Dietmar Merboldt and Jens Frahm
2004 Is the human primary motor cortex involved in motor imagery? Cog-
nitive Brain Research 19: 138-144.
Deiber, Marie-Pierre, Vincente Ibanez, Manabu Honda, Norihiro Sadato, Ramesh
Raman and Mark Hallet
1998 Cerebral processes related to visuomotor imagery and generation of
simple fmger movements studied with positron emission tomography.
Neurolmage 7: 73-85.
Desmurget, Michel and Scott Grafton
2000 Forward modeling allows feedback control for fast reaching move-
ments. Trends in Cognitive Sciences 4: 423-431.
di Pellegrino, Giuseppe, Luciano Fadiga, Leonardo Fogassi, Vittorio Gallese and
Giacomo Rizzolatti
1992 Understanding motor events. Experimental Brain Research 91: 176-
180.
Donald, Merlin
1991 Origins of the Modern Mind. Cambridge, MA: Harvard University
Press.
Dreyfus, Hubert L.
2002 Intelligence without representation. Phenomenology and the Cogni-
tive Sciences 1: 367-383.
Fadiga, Luciano, Leonardo Fogassi, Giovanni Pavesi and Giacomo Rizzolatti
1995 Motor facilitation during action observation. Journal ofNeurophysi-
ology 73: 2608-2611.
Farah, Martha J.
1988 Is visual imagery really visual? Psychological Review 95 (3): 307-
317.
2000 The neural bases of mental imagery. In: Michael S. Gazzaniga (ed.),
The New Cognitive Neurosciences, 965-974. Cambridge, MA: MIT
Press.
Finke, Ronald A.
1989 Principles ofMental Imagery. Cambridge, MA: MIT Press.
Frak, Victor, Yves Paulignan and Marc Jeannerod
2001 Orientation of the opposition axis in mentally simulated grasping.
Experimental Brain Research 136: 120-127.
Fried, Itzhak, Arieh Katz, Gregory McCarthy, Kimberly J. Sass, Peter Williamson,
Susan S. Spencer and Dennis D. Spencer
1991 Functional organization of human supplementary motor cortex stud-
ied by electrical stimulation. Journal of Neuroscience 11: 3656-
3666.
Frith, Chris and Ray Dolan
1996 The role of the prefrontal cortex in higher cognitive functions. Cog-
nitive Brain Research 5: 175-181.
Gallagher, Shaun
Gallese, Vittorio
2003 a The manifold nature of interpersonal relations: the quest for a com-
mon mechanism. Philosophical Transactions of the Royal Society of
London, B 358: 517-528.
2003 b A neuroscientific grasp of concepts. Philosophical Transactions of
the Royal Society ofLondon, B 358: 1231-1240.
Gallese, Vittorio and Alvin Goldman
Gallese, Vittorio, Christian Keysers and Giacomo Rizzolatti
2004 A unifying view of the basis of social cognition. Trends in Cognitive
Ganis, Giorgio, William L. Thompson and Stephen M. Kosslyn
2004 Brain areas underlying visual mental imagery and visual perception.
Cognitive Brain Research 20: 226-241.
Glenberg, Arthur M. and Michael P. Kaschak
2002 Grounding language in action. Psychonomic Bulletin and Review 9:
558-565.
2003 The body's contribution to language. In: Brian H. Ross (ed.), The
Psychology ofLearning and Motivation 43: 93-126. San Diego, CA:
Academic Press.
2003 Hearing Gesture - How Our Hands Help Us Think. Cambridge, MA:
The Belknap Press of Harvard University Press.
2002
2004
Grush, Rick
2003
Goodwyn, W. Susan and Linda P. Acredolo
1998 Encouraging symbolic gestures: A new perspective on the relation-
ship between gesture and speech. In: Jana M. Iverson and Susan
Goldin-Meadow (eds.), The Nature and Functions of Gesture in
Children's Communication, 61-73. New Directions for Child Devel-
opment, no. 79. San Francisco: Jossey-Bass.
Grafton, Scott T., Michael A. Arbib, Luciano Fadiga and Giacomo Rizzolatti
1996 Localization of grasp representations in humans by positron emission
tomography - 2. Observation compared with imagination. Experi-
mental Brain Research 112: 103-111.
Grezes, Julie, Jorge L. Armony, James Rowe and Richard E. Passingham
2003 Activations related to "mirror" and "canonical" neurons in the human
brain: an fMRI study. Neurolmage 18: 928-937.
Grezes, Julie and Jean Decety
2001 Functional anatomy of execution, mental simulation, observation and
verb generation of actions: A meta-analysis. Human Brain Mapping
12: 1-19.
In defense of some "Cartesian" assumptions concerning the brain and
its operation. Biology and Philosophy 18: 53-93.
The emulation theory of representation. Behavioral and Brain Sci-
ences 27: 377-442.
Guillot, Aymeric and Christian Collet
2005 Duration of mentally simulated movement: A review. Journal of
Motor Behavior 37: 10-20.
Hesslow, Germund
1994 Will neuroscience explain consciousness? Journal of Theoretical
Biology 171: 29-31.
Conscious thought as simulation of behaviour and perception. Trends
in Cognitive Sciences 6: 242-247.
Hill, L. Elisabeth
1998 A dyspraxic deficit in specific language impairments and develop-
mental coordination disorder? Evidence from hand and arm move-
ments. Developmental Medicine and Child Neurology 40: 388-395.
Ingvar, David H. and Lars Philipsson
1977 Distribution of the cerebral blood flow in the dominant hemisphere
during motor ideation and motor performance. Annals of Neurology
2: 230-237.
Iverson, Jana M. and 'Susan Goldin-Meadow
1998 Why people gesture when they speak. Nature 396: 228.
Iverson, Jana M. and Ester Thelen
1999 Hand, mouth and brain - the dynamic emergence of speech and ges-
ture. Journal ofConsciousness Studies 6 (11-12): 19-40.
Jeannerod, Marc
1994 The representing brain. Behavioral and Brain Sciences 17 (2): 187-
245.
1997 The Cognitive Neuroscience of Action. Cambridge, MA: Blackwell
Publishers.
2001 Neural simulation of action. NeuroImage 14: SI03-S109.
Jeannerod, Marc and Jean Decety
1995 Mental motor imagery. Current Opinion in Neurobiology 5: 727-
732.
Jeannerod, Marc and Victor Frak
1999 Mental imagining of motor activity in humans. Current Opinion in
Neurobiology 9: 735-739.
Johnson, Mark and TimRohrer
this vo!. We are live creatures: Embodiment, American Pragmatism and the
cognitive organism.
Johnson, Scott H.
2000 Thinking ahead: the case for motor imagery in prospective action
judgements of prehension. Cognition 74: 33-70.
Kilner, James, M., Yves Paulignan and Sarah-Jane Blakemore
2003 An interference effect of observed biological movement on action.
Current Biology, 13 (6): 522-525.
Kosslyn, Stephen M.
1994 Image and Brain: The Resolution of the Imagery Debate. Cam-
Kosslyn, Stephen, Giorgio Ganis and William L. Thompson
2003 Mental imagery: against the nihilistic hypothesis. Trends in Cognitive
Kosslyn, Stephen M. and William L. Thompson
2000 Shared mechanisms in visual imagery and visual perception: Insights
from cognitive neuroscience. In: Michael S. Gazzaniga (ed.), The
New Cognitive Neurosciences, 975-985. Cambridge, MA: MIT
Press.
Krams, Michael, Matthew S. F. Rushworth, Marie-Pierre Deiber, Richard S. J.
Frackowiak and Richard E. Passingham
1998 The preparation, execution and suppression of copied movements in
the human brain. Experimental Brain Research 120: 386-398.
Lakoff, George
1988 Cognitive semantics. In: Umberto Eco, Marco Santambrogio and
Patrizia Violi (eds.), Meaning and Mental Representations, 119-154.
Bloomington: Indiana University Press.
1980 Metaphors We Live by. Chicago: University of Chicago Press.
this vol. Embodiment and social interaction: A cognitive science perspective.
Markman, Arthur B. and C. Miguel Brendl
2005 Constraining theories of embodied cognition. Psychological Science
16(1): 6-10.
1987 The Tree of Knowledge - The Biological Roots of Human Under-
Mazard, Angelique, Nathalie Tzourio-Mazoyer, Fabrice Crivello, Bemard Mazoyer
and Emmanuel Mellet
2004 A PET meta-analysis of object and spatial mental imagery. European
Journal ofCognitive Psychology 16: 673-695.
Meltzoff, Andrew N. and Wolfgang Prinz
2002 An introduction to the imitative mind and brain. In: Andrew Meltzoff
and Wolfgang Prince (eds.), The Imitative Mind: Development,
Evolution and Brain Bases, 1-15. Cambridge, MA: Cambridge Uni-
versity Press.
Murata, Akira, Luciano Fadiga, Leonardo Fogassi, Vittorio Gallese, Vassilis Raos
and Giacomo Rizzolatti
1997 Object representation in the ventral premotor cortex (area F5) of the
monkey. Journal ofNeurophysiology 78: 2226-2230.
Nielsen, Lisbeth
2002 The simulation of emotion experience. Phenomenology and the Cog-
nitive Sciences 1: 255-286.
Ojemann, A. George
1984 Common cortical and thalamic mechanisms for language and motor
functions. American Journal ofPhysiology, 246: R901-R903.
Papaxanthis, Charalambos, Thierry Pozzo, Xanthi Skoura and Marco Shieppati
2002 Does order and timing in performance of imagined and actual move-
ments affect the motor imagery process? The duration of walking and
writing task. Behavioural Brain Research 134: 209-215.
Papaxanthis, Charalambos, Marco Shieppati, Randolphe Gentili and Thierry Pozzo
2002 Imagined and actual arm movments have similar durations when
performed under different conditions of direction and mass. Experi-
mental Brain Research 123: 447-452.
Pulvermiiller, Friedmann, Hubert Preissl, Wemer Lutzenberger and Niels Birbau-
mer
1996 Brain rhythms of language: Nouns versus verbs. European Journal of
Neuroscience 8: 937-941.
Pylyshyn, Zenon
1981 The imagery debate: analogue media versus tacit knowledge. Psy-
chological Review 88: 16-45.
2003 Return of the mental image: are there really pictures in the brain?
Trends in Cognitive Sciences 7: 113-118.
Ranganathan, Vinoth K., Vlodek Siemionow, Jing Z. Liu, Vinod Sahgal and Guang
H.Yue
2004 From mental power to muscle power-gaining strength by using the
mind. Neuropscyhologia 42: 944-956.
Rizzolati, Giacomo and Michael A. Arbib
Rizzolatti, Giacomo, Luciano Fadiga, Leonardo Fogassi and Vittorio Gallese
2002 Frommirror neurons to imitation: Facts and speculations. In: Andrew
MeItzoff and Wolfgang Prince (eds.), The Imitative Mind: Develop-
ment, Evolution and Brain Bases, 247-266. Cambridge, MA: Cam-
bridge University Press.
Rizzolatti, Giacomo, Luciano Fadiga, Vittorio Gallese and Leonardo Fogassi
1996 Premotor cortex and the recognition of motor actions. Cognitive
Brain Research 3: 131-141.
Rohrer, Tim
Schwoebel, John, Consuelo B. Boronat and H. Branch Coslett
2002 The man who executed "imagined" movements: Evidence for disso-
ciable components of the body schema. Brain and Cognition 50: 1-
16.
Schall, Ulrich, Patrick Johnston, Jim Lagopoulos, Markus Jiiptner, WaIter Jentzen,
Renate Thienel, Alexandra Dittmann-Balcar, Stefan Bender and
Philip B. Ward
2003 Functional brain maps of tower of London performance. NeuroImage
20: 1154-1161.
Sebanz, Natalie, Giinther Knoblich and Wolfgang Prinz.
2003 Representinmg others' actions: just like one's own? Cognition 88
(3): B11-B21.
Shallice, Tim
1982 Specific impairments of planning. Philosophical Transactions of the
Royal Society ofLondon B 298: 199-209.
Shepard, Roger N. and Jacqueline Metzler
1971 Mental rotation of three-dimensional objects. Science 171: 701-703.
Sonesson, Goran
this vol. From the meaning of embodiment to the embodiment of meaning: A
study in phenomenological semiotics.
Stening, John, Henrik Jacobsson and Tom Ziemke
2005 Imagination and abstraction of sensorimotor flow: Towards a robot
model. In: AISB'05: Proceedings ofthe Symposium on Next Genera-
tion Approaches to Machine Consciousness - Imagination, Devel-
opment, Intersubjectivity and Embodiment, 50-58. The Society for
the Study of Artificial Intelligence and the Simulation of Behavior,
UK.
Svensson, Henrik and Tom Ziemke
2004 Making sense of embodiment. In: Proceedings of the 26th Annual
Conference ofthe Cognitive Science Society. Mahwah, NJ: Lawrence
Erlbaum.
Thomas, Nigel J. T.
1999 Are theories of imagery theories of imagination? Cognitive Science
23: 207-245.
Trojano, Luigi, David E. J. Linden, Elia Fromisano, Rainer Goebel, Alexander T.
Sack and Francesco Di Salle
2004 What clocks tell us about the neural correlates of spatial imagery.
European Journal ofCognitive Psychology 16: 653-672.
1991 The Embodied Mind. Cambridge, MA: MIT Press.
Vingerhoets, Guy, Floris P. de Lange, Pieter Vandemaele, Karel Deblaere and Erik
Achten.
2002 Motor imagery in mental rotation: An fMRI study. NeuroImage 17:
1623-1633.
Wexler, Mark, Stephen M. Kosslyn and Alain Berthoz
1998 Motor processes in mental rotation. Cognition 68: 77-94.
Wilson, Margaret
9 (4): 625-636.
Wolpert, Daniel M. and Mitsuo Kawato
1998 Multiple paired forward and inverse models for motor control. Neu-
ralNetworks 11: 1317-1329.
Vue, Guang and Kelly J. Cole
1992 Strength increases from the motor program. Comparison of training
with maximal voluntary and imagined muscle contractions. Journal
ofNeurophysiology 67: 1114-1123.
Ziemke, Tom
David Kirsh (eds.), Proceedings of the 25th Annual Meeting of the
baum.
2004 Embodied AI as science: Models of embodied cognition, embodied
models of cognition, or both? In: Fumiya Iida et al. (eds), Embodied
Artificial Intelligence, 27-36. Heidelberg: Springer Verlag.
Ziemke, Tom, Dan-Anders Jirenhed and Germund Hesslow
2005 Internal simulation of perception: A minimal neuro-robotic model.
Neurocomputing 68: 85-104.
Zlatev, Jordan
Zwaan, Rolf A.
2004 The immersed experiencer: toward an embodied theory of language
comprehension. In: Brian H. Ross (ed.), The Psychology ofLearning
and Motivation 44: 35-62. San Diego, CA: Academic Press.
Phenomenological and experimental contributions
to understanding embodied experience
Shaun Gallagher
Abstract
Much recent work on the relationship between phenomenology, in the Husserlian
tradition, and the contemporary cognitive sciences has focused on the question of
embodiment. In this chapter I suggest that the distinction between body image and
body schema, once clarified on phenomenological grounds, can contribute to an
understanding of embodied experience, and how the body shapes cognition. A clear
'distinction between these two concepts can be both verified and applied in specific
empirical research concerning such issues as intentional action, intermodal percep-
tion, neonate imitation, mirror neurons and pathologies that involve unilateral ne-
glect and deafferentation. A consideration of these phenomena also leads to clarifi-
cations about the nature of intersubjectivity consistent with both phenomenological
insights offered by Husserl and Merleau-Ponty and the most recent neuroscience of
social cognition.
Keywords: action, body image, body schema, deafferentation, intersubjectivity,
neonate imitation, unilateral neglect.
1. Introduction
It is a long-established principle in phenomenology that a strictly physical
(scientific) analysis of the objective body is not sufficient to reveal its
contribution to cognition. Phenomenology proposes an analysis of the body
as we live it. In the context of the cognitive sciences, however, it is not
sufficient to stop with a pure phenomenology of lived experience. And
specifically, in the context of the cognitive sciences, one needs to appeal to
empirical verifications and clarifications that will confirm phenomenologi-
cal insight, and then use that insight to interpret the empirical data - a her-
meneutical circle, to be sure, but not a methodologically vicious one.
272 Shaun Gallagher
Within this circle the relationship between phenomenological analysis of
embodied experience found, for example, in Husserl's writings from 1912-
1915 (Husserl 1952) and in Merleau-Ponty (1945), and recent experimental
research on a variety of issues related to embodiment, can be one of mutual
enlightenment. This is not an uncontroversial claim, and is a matter of on-
going discussion (see e.g., Varela 1996; Gallagher 1997; Gallagher and
Varela 2002; Bayne 2004; Overgaard 2004; Zahavi 2004). Rather than
enter into this debate here, however, I will adopt the following limited
position: a phenomenology that understands intentionality as a form of
being-in-the-world, and recognizes the importance of embodied action for
shaping perception, offers an interpretational framework different from
purely functional or syntactic interpretations of the empirical data. The best
way to explicate this framework is actually to go to work on specific issues
that lend themselves to both phenomenological and experimental ap-
proaches to embodied experience.
In this chapter I will pursue two questions. (1) To what extent and in
what way is one's body part of one's perceptual field? This question is
clearly open to both empirical and phenomenological analysis. (2) How
does the body shape perception, or more generally, cognition? In this case,
phenomenology can point to certain "prenoetic" performances of the body;
but empirical science is required to clarify such performances.
Psychologists already have a relatively developed way of addressing the
first question, about the appearance of the human body in the perceptual
field, or more generally, about the image that a person has of their own
body. In most instances, this is referred to as a body image. The extensive
literature on body image, however, is problematic. It is not only wide-
ranging - the concept is employed and applied in a great variety of fields,
from neuroscience to philosophy, from the medical sciences to the athletic
sciences, from psychoanalysis to aeronautical psychology and robotics -
but as often happens in such cases, the term changes meaning from one
field or discipline to the next, from one author to the next, and sometimes
even within a single author (Gallagher 1986, 2005).
Problems about the meaning of the term body image are also bound up
with the use of another term, body schema. But this is not just a termino-
logical issue. There are more deep-seated conceptual confusions involved.
Precisely such confusions, which can lead to problems involving experi-
mental design and the interpretation of experimental results, motivate some
authors to suggest that we ought to give these terms up, abandon them to
history, and formulate alternative descriptions of embodiment (e.g., Poeck
Phenomenological and experimental contributions 273
and Orgass 1971; Ga11ese 2005). I argue here that with respect to our two
questions, a clear phenomeno10gica11y-based conceptual distinction be-
tween body image and body schema can do some useful work despite the
ambiguity involved in the historical use of these concepts. Each concept
addresses a different sort of question. The concept of body image helps to
answer the first question about the appearance of the body in the perceptual
field; in contrast, the concept of body schema helps to answer the question
about how the body shapes the perceptual field. So these terms and con-
cepts, if properly clarified, provide a way to explicate the role embodiment
plays in conscious and cognitive experience.
2. Body image and body schema
Rather than rehearse the long history of conceptual confusion regarding
these terms (see Ga11agher 1986, 1995, 2005), let me go directly to the
conceptual distinction between body image and body schema. We can then
examine how this distinction contributes to understanding several phenom-
ena studied in the cognitive sciences literature, including unilateral neglect
and deafferentation (i.e., a loss of peripheral sensory input), neonate imita-
tion and our ability to understand others.
Phenomeno10gical reflection tells us that there is a difference between
taking an intentional attitude towards one's own body (having a perception
of, or belief about, or emotional attitude towards one's body) and having a
capacity to move or to exist in the action of one's own body. The concepts
of body image and body schema correspond to this phenomeno10gical dif-
ference.
Body image is a (sometimes conscious) system of perceptions, atti-
tudes and beliefs pertaining to one's own body.
Body schema is a system of processes that constantly regulate posture
and movement: sensory-motor processes that function without reflec-
tive awareness or the necessity of perceptual monitoring.
The distinction between body image and body schema is not an easy one to
make because behaviorally the two systems interact and are highly coordi-
nated in the context of intentional action, and in pragmatic and socially
contextualized situations. A conceptual distinction is nonetheless useful
274 Shaun Gallagher
precisely in order to understand the complex dynamics of bodily movement
d
. 1
an experIence.
The body image, consisting of a complex set of intentional states - per-
ceptions, beliefs and attitudes - in which the intentional object of such
states is one's own body, involves a form of reflexive or self-referential
intentionality. Studies involving body image (e.g., Cash and Brown 1987;
Gardner and Moncrieff 1988; Powers et al. 1987) frequently distinguish
among three of these elements:
a) the subject's perceptual experience ofhislher own body;
b) the subject's conceptual understanding (including folk knowledge
and/or scientific knowledge) of the body in general; and
c) the subject's emotional attitude toward hislher own body.
Although a conceptual understanding and emotional attitude do not neces-
sarily involve an on-going conscious awareness, they are maintained as sets
of beliefs or attitudes, and in that sense form part of an intentional system.
Conceptual and emotional aspects of the body image are no doubt affected
by various cultural and interpersonal factors (see e.g. Roth this volume,
Sonesson this volume, Zlatev this volume). It is also the case, as I will sug-
gest below, that the perceptual content of the body image originates in
intersubjective perceptual experience.
In contrast to the body image, a body schema is not a perception, a be-
lief, or an attitude. Rather it is a system of motor functions or motor pro-
grams that operate below the level of self-referential intentionality. It in-
volves a set of tacit performances - preconscious, subpersonal processes
that play a dynamic role in governing posture and movement. In most in-
stances, movement and the maintenance of posture are accomplished by the
close to automatic performances of a body schema, and for this very reason
the normal adult subject, in order to move around the world, neither needs
1. In making this conceptual distinction, however, I am not reaffrrming the tradi-
tional distinction between perception and action, although I am affrrming a phe-
nomenological distinction between the perception of my body and my bodily
action. Intentional action, for example, clearly involves perception, but does
not clearly involve a perceptual monitoring of my body. Furthermore, I am not
making any claim about the neurological underpinnings of body image and
body schematic processes. There is good reason to think that body image and
body schematic processes are in part underlain by neuronal activity in the same
or similar brain areas (see, e.g., Gallese 2005).
nor has a constant body percept. In this sense the body tends to efface itself
in most normal activities that are geared into external goals. To the extent
that one does become aware of one's own body in terms of monitoring or
directing perceptual attention to limb position, movement, or posture, such
awareness helps to constitute the perceptual aspect of a body image. Such
awareness may then interact with a body schema in complex ways, but it is
not equivalent to a body schema itself.
I said that a body schema operates in a close to automatic way. This
does not mean that its operations are a matter of reflex. Movements con-
trolled by a body schema can be precisely shaped by the intentional experi-
ence or goal-directed behavior of the subject. If I reach for a glass of water
with the intention of drinking from it, my hand shapes itself in a precise
way for picking up the glass, and it does this completely outside my aware-
ness. But the shape that it takes on is in complete conformity with my in-
tention (Jeannerod 1997; Jeannerod and Gallagher 2002). Thus it is im-
portant to note that although a body schema is not itself a form of
consciousness, or in any way a cognitive operation, it can enter into and
support intentional activity, including cognition. In this sense motor action
is not completely automatic; it is usually part of a voluntary, intentional
project. When I walk across the room to greet someone or jump to catch a
ball in the context of a game, my actions may be explicitly willed, and
governed by my perception of objects or persons in the environment. My
attention and even my complete awareness in such cases, however, are
centered on the other person or the ball, and not on the precise accom-
plishment of locomotion. The body moves smoothly and in a coordinated
fashion not because I perceptually monitor or have an image of my bodily
movement, but because of the coordinated functioning of a body schema.
It is also the case that a body image or percept can contribute to the
control of movement. The visual, tactile and proprioceptive attentiveness
that I have of my body may help me to learn a new dance step, improve my
tennis game, or imitate the novel movements of others. In learning a new
movement in such contexts, for example, I may consciously monitor and
correct my movement. In other cases, when there is a physical threat, my
movement may involve a large amount of perceptual monitoring and willed
conscious control. Even in such cases the contribution made to the control
of movement by my perceptual awareness of my body will always find its
complement in capacities that are defined by the operations of a body
schema that continues to function to maintain balance and enable move-
ment. Such operations are always in excess of what I can be aware of.
276 Shaun Gallagher
Thus, a body schema is not reducible to a perception of the body; it is
never equivalent to a body image.
Am I always conscious of my own body as an intentional object, or as
part of an intentional state of affairs? The distinction between consciously
attending to the body and being marginally aware of the body is important.
As I suggested, sometimes we do attend specifically to some aspect or part
of the body. But in much of our everyday experience, and most of the time,
our attention is directed away from the body, toward the environment or
toward some project we are undertaking. Do we remain consciously aware
of some aspect or part of the body even in cases where our attention is not
directed toward the body? Such awareness may vary by degree among in-
dividuals. Some people may be more aware, others, at times, not at all
aware of their body. If I am solving a difficult mathematical problem, am I
also and at the same time aware of the position of my legs or even of my
grip on the pencil, or are these things so much on "automatic pilot" that I
do not need to be aware of them? In any case, to define the difference be-
tween body image and body schema it is not necessary to determine to
what extent we are conscious of our bodies. It suffices to say that some-
times we are attentive to or aware of our bodies; other times we are not. A
body image is inconstant in this sense. When I am marginally aware that I
am moving in certain ways, my awareness may not capture the whole
movement. If I am marginally aware that I am reaching for something, I
may not be aware at all of the fact that for the sake of balance my left leg
has stretched in a certain way, or that my toes have curled against the floor.
Posture and the majority of bodily movements operate in most cases with-
out the help of a body image.
As distinct from body image, the body schema system involves a preno-
etic performance of the body, that is, a performance that helps to structure
our experience, but does not explicitly show itself in the contents of con-
sciousness. That a body schema operates in a prenoetic way means that it
does not depend on a consciousness that targets or monitors bodily move-
ment. This is not to say that it does not depend on consciousness at all. For
certain motor programs to work properly, I need information about the
environment, and this is most easily received by means of perception. In
my intentional actions the body acquires a certain organization or style in
its relations with its environment. For example, it appropriates certain ha-
bitual postures and movements; it incorporates various significant parts of
its environment into its own schema. The carpenter's hammer becomes an
operative extension of the carpenter's hand, or, as Head (1920) noted, the
body schema can extend to the tip of the blind man's cane. The system that
is the body schema allows us to actively engage with our environment
without the requirement of a reflexive conscious monitoring directed at the
body. It is a dynamic, operative performance of the body, rather than a
copy, image, or conceptual model of it.
In so far as I am conscious of what I am doing, the content of my con-
sciousness is not about my body, but is specified in its most pragmatic
meaning. That is, if I were to formulate the content of my consciousness
when reaching to get a drink, it would not be in terms of operating or
stretching muscles, bending or unbending limbs, turning or maintaining
balance; it would not even be in terms of reaching and grasping. Rather, if I
were stopped and asked what I was doing, I would say something like "I'm
getting a drink". The details of bodily movement entailed in the action are
just as much hidden in my pre-reflective experience as they are hidden in
that description. I am aware of my bodily action not as bodily action per se,
but as action at the level of my intentional project. Thus, prenoetic func-
tions underpin and affect my experience, and are subsumed into larger
intentional activities. In this sense, detailed aspects of movement (such as
the contraction of certain muscles), even if we are not aware of them (even
if they are not explicitly intentional), are intentional insofar as they are part
of a larger intentional action.
Establishing a conceptual distinction between body image and body
schema is only the beginning of an explication of the role played by the
body in action and cognition. There are reciprocal interactions between
prenoetic body schemas and cognitive experiences, including normal and
abnormal consciousness of the body. Such behavioral relations between
body image and body schema can be worked out in detail, however, only if
the conceptual distinctions between them are first understood. Armed with
this phenomenological distinction we can seek verifications and clarifica-
tions in the empirical literature.
3. Unilateral neglect and deafferentation
In sciences like psychology a good way to verify that a conceptual distinc-
tion between X and Y is valid is to identify a double dissociation, that is, to
identify a case in which we find X but not Y, and another case in which we
find Y but not X. For the distinction that we have been discussing, impor-
tantly, it is possible to find cases in which a subject has an intact body
278 Shaun Gallagher
image but a dysfunctional body schema, and vice versa. For example, one
finds evidence of an intact body schema but the absence of a completely
intact body image in some cases of unilateral neglect. Denny-Brown and
his colleagues report that a patient, following stroke, who suffers from a
neurologically caused defect in perception related to the left side, fails to
notice the left side of her body. She excludes it from her body image. She
fails to dress her left side or comb the hair on the left side of her head. Yet
there is no motor weakness on that side. Her gait is normal, although if her
left slipper comes off while walking she fails to notice. Her left hand is
held in a natural posture most of the time, and is used quite normally in
movements that require the use of both hands, for example, buttoning a
garment or tying a knot. Thus she uses the motor ability of the neglected
side, to dress the right side of her body (Denny-Brown, Meyer and Horen-
stein 1952; similar cases are reported by Ogden 1996 and Pribram 1999).
In such cases, the patient's body schema system is intact despite her prob-
lems with body image on the neglected side.
Dissociation of the opposite kind can be found in rare cases of deaffer-
entation. A subject (IW) who has lost tactile and proprioceptive input from
the neck down can control his movement only by cognitive intervention
and visual guidance of his limbs. In effect he employs his body image
(primarily a visual perception of his body) in a unique way to make up for
the impairment of his body schema (see Cole 1995; Gallagher and Cole
1995). Proprioception is that bodily sense which allows us to know how
our body and limbs are positioned. If a person with normal proprioception
is asked to sit, close their eyes and point to their knee, it is proprioception
that allows them to successfully guide their hand and find their knee. If IW
is asked to close his eyes and point to his knee, he has some difficulty. If,
in this situation, I move either his knee or his arm, he is unable to point to
his knee since, without vision or proprioception, he does not know where
either his knee or his hand are located. He assumes that they are in exactly
the same location as when he last saw them and he moves his hand so as to
point to where he remembers his knee to have been.
Because of the loss of proprioception and tactile sense IW does not
have a sense of where his limbs are or what posture he maintains without
visual perception. In order to maintain motor control he must conceptualize
his movements and keep certain parts of his body in his visual field. His
movement requires constant visual and mental concentration. In darkness
he is unable to control movement; when he walks he cannot daydream but
must concentrate on his movement constantly. When he writes he needs to
concentrate on both his body posture and on holding the pen. Maintaining
posture is, for him, a task rather than an automatic process.
In terms of the distinction between body image and schema, IW has lost
major aspects of his body schema, and thereby the possibility of normally
unattended movement. He is forced to compensate for that loss by de-
pending on his body image in a way that normal subjects do not. For him,
control over posture and movement are achieved by a partial and imperfect
functional substitution of body image for body schema.
Proprioception is a major source of information for the maintenance of
posture and the governance of movement - that is, for the normal func-
tioning of the body schema. But proprioception is not the only possible
source for the required information. lW, as a result of extreme effort and
hard work, recovered control over his movement and regained a close to
normal life. It is important to understand that he did not do this by recov-
ering proprioceptive sense. In strict physiological terms, he has never re-
covered from the original problem. His proprioception has not been re-
paired. He is able to address the motor problem on a behavioral level,
however, primarily by using an enhanced body image to help control
movement. This case, in terms of the body image-body schema distinction,
is just the opposite to neglect. If the neglect patient is capable of controlled
movement even on the neglected side because of an intact body schema,
lW, who is unable to depend on a body schema, must employ his body
image to guide his movement. In complete contrast to neglect, IW is re-
quired to pay an inordinately high degree of attention to his body. Thus,
cases of unilateral neglect and deafferentation, and the double dissociation
implied, begin to provide logical and empirical reasons for thinking that
there is a useful distinction to be made between body schema and body
image. It is also to be noted, however, that the distinction between body
image and body schema can help to make sense out of such cases.
4. Neonate imitation
Prior to the development of a body image or a body schema in a small
child, is it the case that something like a less embodied consciousness ex-
ists? Less embodied may even mean less structured, along the lines of
William James' (1890) famous phrase about the "blooming, buzzing confu-
sion" of the infant's experience. It is not unusual to find proponents of the
view that conscious experience is in some way the developmental source
280 Shaun Gallagher
for both the body image and the body schema. Indeed, this is the traditional
view in both psychology and philosophy.
An empiricist, for example, might hold that a body image is generated
only on the basis of the prolonged perceptual experience that one has of
one's own body. Conceptual and emotional aspects of the body image, and
the structural aspects that the body image brings with it, are obviously
traceable to certain early and originary experiences that the child may have
in tactile, visual and other sensations of the body. It might also be thought
that a body schema originates only through the conscious experience of
movement. Much as we learn habits through practice, we learn to control
our movements through the practiced experience of movement. This seems
to be the case in examples we referred to before, such as in the learning of
a new dance movement. It seems more obviously true of learning to crawl
and to walk. On this view, then, conscious experience is at the origin of
such things as body image and body schema. Thus, a certain kind of con-
sciousness, primitive and perhaps disorganized, would predate the con-
sciousness that is shaped and structured by embodiment.
This traditional view assumes that the newborn infant has no body im-
age or body schema, and that such things are acquired through prolonged
experience in infancy and early childhood. This view has been worked out
in a number of ways and in a variety of contexts in scientific and philo-
sophical discussions. Up until about thirty years ago this position was the
almost unanimous consensus among developmental theorists. At that time,
however, on several fronts, new evidence was developed in support of a
more nativist position. The idea that body schemas may in fact be innate
was put forward, for example, in studies of phantom limbs in cases of con-
genital absence of limb (for further discussion of aplasic phantoms, see
Gallagher 2005; Gallagher et al. 1998; Gallagher and Meltzoff 1996). In
the 1970s, in studies of neonate imitation (Meltzoff and Moore 1977), fur-
ther evidence was provided to show that certain elements of what previ-
ously were understood to be learned motor behaviors were in fact already
present in the newborn.
The traditional view is that the body schema is an acquired phenome-
non, built up in experience, the product of development. This traditional
view is well represented by the developmental psychologist Marianne
Simmel (1958, 1962, 1966), one of the few psychologists who makes a
clear distinction between body schema and body image. The body schema,
Simmel claims, is
[... ] built up as a function of the individual's experience, i.e., it owes its ex-
istence to the individual's capacity and opportunity to learn. This means that
at some early time in the development of the human organism the schema
has not yet been formed, while later [...] it is present and is characterized by
considerable differentiation and stability. (1958: 499)
This position is also held by Merleau-Ponty (1945), who was greatly influ-
enced by his study of developmental psychology and by the psychologists
and psychological research he cited, including the work of Piaget (1945),
Wallon (1925), Guillaume (1943) and Lhermitte (1939). Although Mer-
leau-Ponty rightly conceives of the body schema as an anterior condition of
possibility, a dynamic force of integration that cannot be reduced to the
sum "of associations established during experience", still, in terms of de-
velopment, the operations of the body schema are "'learnt' from the time of
global reactions of the whole body to tactile stimuli in the baby [...]"
(1945: 101, 122n). The body schema functions as if it were an "innate
complex" (1945: 84), that is, as strongly and pervasively as if it were in-
nate, but, as an acquired habit with a developmental history, it is not actu-
ally innate.
Following Wallon (1925), Merleau-Ponty believed that experience be-
gins by being interoceptive, and that the newborn is without external per-
ceptual ability (1960: 121). James's "blooming, buzzing confusion" does
not begin to be resolved until between the third and sixth month of life
when a collaboration takes place between the interoceptive and exterocep-
tive domains - a collaboration that simply does not exist at the beginning
of life (Merleau-Ponty 1945: 121). On this view, one reason for the lack of
any organized exteroceptive perception is precisely the absence of a
"minimal bodily equilibrium", an equilibrium that must be sorted out be-
tween a developing body schema and the initial and still very primitive
stages of a body image. For Merleau-Ponty, motor experience and percep-
tual experience are dialectically or reciprocally linked. The mature opera-
tion of a body schema depends on a developed perceptual knowledge of
one's own body; and the organized perception of one's own body, and then
of the external world, depends on a proper functioning of the body
schema.
2
2. "Up to that moment [exteroceptive] perception is impossible. [...] The operation
of a postural schema - that is, a global consciousness of my body's position in
space, with the corrective reflexes that impose themselves at each moment, the
282 Shaun Gallagher
The infant does not yet have a body schema, according to Merleau-
Ponty, because of a certain lack of neurological development. The
development of the body schema can happen only in a gradual and
fragmentary way as the central nervous system develops. Motor schemas
are then gradually integrated, and in a reciprocal system with external
perception and sensory inputs, become "precise, restructured and mature
little by little" (1960: 123).
Simmel and Merleau-Ponty are good representatives of the traditional
view that both body schema and body image are acquired through experi-
ence. This view clearly implies what is possible and what is not possible in
the conscious experience of the infant: conscious experience is disorgan-
ized; exteroceptive perception is impossible. Also, according to this view,
the capacity for imitation - an important capacity directly related to ques-
tions about perception, social recognition, the ability to understand another
person and the origins of a sense of self - is non-existent in infants younger
than 12 months.
3
Jean Piaget (1945) expresses this view in the most precise
terms. The question is about a certain kind of imitation called "invisible
imitation". Piaget defines invisible imitation as the child's imitation of
another person's movements using parts of the child's body that are invisi-
ble to the child. For example, if a child does not see its own face, is it pos-
sible for the child to imitate the gesture that appears on another person's
face? Piaget's answer is that at a certain point in development it is possi-
ble; but in early infancy it is not. The reason is that invisible imitation re-
quires the operation of a relatively mature body schema. Thus, according to
Piaget (as well as most other classical theorists of development), invisible
imitation is not possible prior to 8 to 12 months of age.
The intellectual mechanisms of the [child under 8 months] will not allow
him to imitate movements he sees made by others when the corresponding
movements of his own body are known to him only tactually or kinestheti-
cally, and not visually (as, for instance, putting out his tongue). [...] Thus
since the child cannot see his own face, there will be no imitation of move-
ments of the face at this stage. [...] For imitation of such movements to be
possible, there must be co-ordination of visual schemas with tactilo-
kinesthetic schemas [...]. (Piaget 1945: 19, 45)
global consciousness of the spatiality of my body - all this is necessary for
[exteroceptive] perception (Wallon)" (Merleau-Ponty 1960: 122).
3. Merleau-Ponty (1945: 352) does recognize this kind of ability in an infant at 15
months.
Merleau-Ponty follows Guillaume and Piaget in regard to these issues.
Thus, to imitate
[... ] it would be necessary for me to translate my visual image of the other's
[gesture] into a motor language. The child would have to set his facial mus-
cles in motion in such a way as to reproduce [the visible gesture of the
other]. [...] If my body is to appropriate the conducts given to me visually
and make them its own, it must itself be given to me not as a mass of utterly
private sensations but instead by what has been called a "postural" or "cor-
poreal schema". (Merleau-Ponty 1960: 116-117)
In complete contrast to this traditional view, studies on imitation in infants
conducted by Meltzoff and Moore (1977, 1983) show that invisible imita-
tion does occur in newborns. Their experiments, and others that replicate
and extend their results (see Meltzoff and Moore 1994 for summary), show
that newborn infants less than an hour old can indeed imitate facial ges-
tures. A brief review of several of their experiments will help to clarify the
results and their relevance to the issues of body schema, body image and
intermodal perception.
Meltzoff and Moore (1983): 40 normal and alert newborn infants
ranging in age from less than 1 hour to 71 hours were tested. The ex-
perimenter presented each infant with a mouth-opening gesture over a
period of 4 minutes, alternating in 20-second intervals between the
mouth opening and a passive facial appearance. The same procedure
was then followed using tongue protrusion as the target gesture. The
study showed a clear and statistically significant result in terms of both
the frequency and duration of the infants' response gestures, demon-
strating that normal and alert newborn infants systematically imitate
adult gestures of mouth opening and tongue protrusion. Notably, even
the youngest infant in the study, 42 minutes old at the time of the test,
showed a strong imitation effect. Other experiments have extended the
range of gestures that young infants imitate to a wider set, including lip
protrusion, sequential finger movement, head movements, smiling,
frowning and surprised expressions.
Meltzoff and Moore (1977): showed some form of memory to be in-
volved in early imitation. Infants between the ages of 16-21 days imi-
tated facial gestures after a delay. This involved putting a pacifier in the
infant's mouth as it was shown a facial gesture. After the presentation
of the facial gesture was complete, the pacifier was removed and the in-
284 Shaun Gallagher
fant imitated the gesture. Thus, imitative responses were delayed and
only allowed when the gesture had vanished from the perceptual field.
Experiments also show that infants imitated after a delay of up to 24 hours,
and that infants improve their gestural performance over time (Meltzoff
and Moore 1994). Their first attempts at imitation do not necessarily repli-
cate the seen gesture with a high degree of accuracy. When tongue protru-
sion is displayed, infants quickly activate the tongue; but they improve
their motor accuracy over successive efforts.
4
What aspects of embodiment allow for these possibilities in the neo-
nate? There are two things that need to be considered. First, a relatively
developed body schema already existing at birth. If we follow the logic
expressed by the proponents of the traditional view, namely, that imitation
requires a developed body schema, then the studies on newborn imitation
suggest that there is at least a primitive body schema from the very begin-
ning. This is an innate body schema sufficiently developed at birth to ac-
count for the ability to move one's body in appropriate ways in response to
environmental stimuli, and specifically for the possibility of invisible imi-
tation. Here I use the word "innate" to mean, literally, "something existing
prior to birth".
Second, an intermodal sensory system is required to enable the infant to
recognize a structural equivalence between itself and the other person. A
large number of experiments have now been done to show that perception
is intermodal from the very beginning (for summary see Meltzoff 1993;
Gallagher and Meltzoff 1996). In an intermodal system, proprioception and
vision are already in communication with each other. In certain cases, what
I see automatically gets translated into a proprioceptive sense of how to
4. The fmdings of imitation under these experimental conditions rule out "re-
flexes" or release mechanisms as potential mediators of this activity. Re-
flexes and release mechanisms are highly specific - that is, narrowly cir-
cumscribed to limited stimuli. One cannot have a reflex or release
mechanism for imitation in general. As a result, the range of behaviors dis-
played by infants would require the unlikely postulate of distinct reflexes or
release mechanisms for each kind of imitative behavior: tongue protrusion,
tongue protrusion to one side, mouth openings, smile, frown, etc. While it
may not be difficult to imagine how evolution might provide for a reflex
smile, it is difficult to understand why it would furnish a reflex for angular
tongue protrusion. Furthermore, neither delayed response nor improvement
in response is compatible with a simple reflex or release mechanism.
move. Proprioception and vision are intermodally linked in several ways,
and these linkages are part of a more general link between sensory and
motor activities. For example, and quite relevant to the possibility of neo-
nate imitation, both proprioception and vision are integrated with vestibu-
lar information about head motion and orientation.
s
Importantly, these
structures, involving self-awareness, are mature at birth. Thus in the case
of neonate imitation, the imitating subject depends on a complex back-
ground of embodied processes, a body-schema system involving visual,
proprioceptive and vestibular information. In the foreground, what the
infant sees gets translated into a proprioceptive awareness of her own rele-
vant body parts; and proprioceptive information allows her to move those
parts so that her proprioceptive awareness matches up to what she sees.
This intermodal intra-corporeal communication, then, is the basis for an
inter-corporeal communication. Just here we can postulate the beginnings
of a body image - based on the infant's sense that the face of the other
person is like its own face, defined pragmatically, as something it can
move in the same way. This has profound implications for the child's rela-
tions with others.
Meltzoff and Moore (1997) propose a psychological-cognitive model, a
set of theoretical black boxes representing "comparison function", "act
equivalence", "recognition of my own capability", etc. Here I want to sug-
gest that both phenomenology and neuroscience are required to open up
these black boxes and to show that the embodied processes that make imi-
tation possible make sense only in a larger context defined in terms of in-
tersubj ectivity.
5. The vestibular nucleus, a relatively large midbrain structure, serves as a
complicated integrative site where fIrst-order information about head posi-
tion is integrated with whole-body proprioceptive information from joint re-
ceptors and oculo-motor information about eye movement. This integrated,
multimodal information projects to the thalamus, informing connections that
project to cortical areas responsible for control of head movement. Vestibu-
1ar neurons in the parietal lobe respond to vestibular stimulation, but also to
somatosensory and optokinetic stimuli, and more generally there is cortical
integration of information concerning self-motion, spatial orientation, and
visuo-motor functions (Guldin, Akbarian, and Griisser 1992; Jouen and
Gapenne 1995).
286 Shaun Gallagher
5. Intersubjectivity
Husserl, in texts from 1906-1913, suggests that our understanding of oth-
ers involves processes that happen on the level of bodily sensations, and
that this provides access to others that predates or prefigures anything that
would involve inference or analogy. For Husserl, understanding another
person is not a matter of intellectual inference but a matter of sensory acti-
vations that are unified in or by the animate organism or lived body that is
perceiving another animate organism. But Husserl's thought here is offered
more as a question than as a confirmed view. Re asks: "Can what effects
the unitary lived embodiment [Leiblichkeit] extend itself to the separate
and movable bodies in the spatial world?" (1973: 33). And here he sug-
gests that the perception of another person's body as object [Korper-
wahrnehmung] is in some way different from the perception of another
person's lived body [Leibwahrnehmung].
These suggestions are reminiscent of Russerl's writings in Ding und
Raum (1907). There he talks about kinaesthetic sensations that are acti-
vated in perception. When I perceive something, the sensory activation
involved is joined by a corresponding activation of kinaesthetic sensations
in my lived body. If Russerl were Merleau-Ponty he would have put it in
this way: my body reverberates with the things of the world.
Kinaesthesia is the sensory experience of one's own movement and is
closely related to proprioception or position sense. Quite simply, when I
move, I have a pre-reflective and recessed sense of moving. Russerl's
claim, however, is not simply that we have kinaesthetic sensation when we
move, but that we have kinaesthetic sensation when we perceive something
- the something that we perceive registers in a certain way within our pro-
prioceptive-kinaesthetic system, or more generally, within our body
schema. Specifically in regard to intersubjectivity, when we see someone
else act in a certain way, our own body-schematic system is activated. This
kind of process is directly relevant to imitation, and in part, is what pro-
vides us with a primary understanding of the other person.
6
6. For the notion of primary understanding, see Dilthey (1926). Scheler (1913
[1948]: 254) is of the same mind and insists that this primary understanding
is perceptual in nature. "For we certainly believe ourselves to be directly ac-
quainted with another person's joy in his laughter, with his sorrow and pain
in his tears, with his shame in his blushing, with his entreaty in his out-
stretched hands [...] And with the tenor of this thoughts in the sound of his
words. If anyone tells me that this is not 'perception', for it cannot be so, in
What remains somewhat tentative in Russerl is fully developed in Mer-
leau-Ponty. It is consistent with his phenomenological insights to say that
we emerge from pre-natal life immersed in a set of prenoetic, natural proc-
esses and these make up the complex sensory fields upon which appear the
world and others. In regard to intersubjectivity, Merleau-Ponty puts things
in the right order: "The very first of all cultural objects, and the one by
which all the rest exist, is the body of the other person as the vehicle of a
form of behavior" (1945: 348). He takes one's own lived body to be the
locus of intersubjective experience:
I have the world [and others ... ] through the agency of my body as the po-
tentiality of this world. [... ] [B]etween this phenomenal body of mine and
that of another as I see it from the outside, there exists an internal relation
which causes the other to appear as the completion of the system. (1945:
350,352)
And in a much later text he continues the same thought: "My corporal
schema is a normal means of knowing other bodies... " (Merleau-Ponty
1956-1960: 218). This is what he terms intercorporeality.
Recent studies in neuroscience suggest that there are specific neuro-
physiological processes that can account for this intercorporeality, under-
stood as a body-schematic reverberation that depends on the close intermo-
dal connections between visual perception, kinaesthetic-proprioception and
motor behavior. These are body-schematic, and specifically motor proc-
esses that operate prenoetically, as general conditions of possibility for
motor stability and control. They are also directly related to the possibility
of imitation. I refer here to what neuroscientists now describe as processes
that involve mirror neurons and resonant systems (Gallese 1998; Gallese et
al. 1996; Rizzolatti et al. 1996). Mirror neurons link up motor processes
with visual ones in ways that are directly relevant to the possibility of imi-
tation. When I see another person act in a certain way, the neurons acti-
vated in the pre-motor cortex are precisely the same neurons that are acti-
vated when I act in the same way. More generally, overlapping brain areas,
or "shared representations" in the motor, premotor and prefrontal cortexes,
are activated in the following conditions: during motor action, during the
view of the fact that a perception is simply a 'complex of physical sensa-
tions' [...] I would beg him to turn aside from such questionable theories and
address himself to the phenomenological facts." For the relevance of a direct
perceptual or primary understanding in the context of contemporary debates,
see Gallagher (2001, 2004).
288 Shaun Gallagher
observation of another's motor action, and during the imaginative enact-
ment (conscious simulation) of my own or another's motor action, and
during preparation for imitating the other (Georgieff and Jeannerod 1998;
Grezes and Decety 2001; Jeannerod 2001; Ruby and Decety 2001). That is,
the same neuronal areas are activated when I engage in intentional action
and when I see or imagine such action performed by another person.
What both the phenomenology and the neuroscience show is that my
intersubjective understanding of others is not a purely intellectual accom-
plishment. I perceive the emotions and the intentions of the other person in
their bodily movements and gestural expressions, and in doing so, my own
embodiment acts as the template for understanding (see Gallagher 2001).
6. Conclusion
My primary concern has been to show in a partial way, but in sufficient
detail, how embodiment provides certain innate capacities that enable and
condition our experience of ourselves and others. There is much more to
say in regard to all of these issues, but I hope that I have given sufficient
indication of how the distinction between body image and body schema
can be used in a productive way. More specifically, I think the use of these
concepts entails a multi-disciplinary approach that involves both phenome-
nology and the empirical studies of psychology and neuroscience. This is a
two-way process, however. Not only does the phenomenological distinc-
tion between body image and body schema receive verification and clarifi-
cation by appealing to empirical evidence, but the distinction itself goes
some distance towards the clarification of a variety of issues in the empiri-
calliterature. Only through this combination of disciplines can we begin to
map out the details of how the body shapes our mental experience.
References
Bayne, Timothy
2004 Closing the gap? Some questions for neuro-phenomenology. Phe-
nomenology and the Cognitive Sciences 3 (4): 349-364.
Cash, Thomas F. and Timothy A. Brown
1987 Body image in anorexia nervosa and bulimia nervosa: A review of
the literature. Behavior Modification 11: 487-521.
Cole, Jonathan
1995 Pride and a Daily Marathon. Cambridge, MA: MIT Press.
Denny-Brown, Derek, John S. Meyer and Simon Horenstein
1952 The significance of perceptual rivalry resulting from parietal lesion.
Brain 75: 433-471.
Dilthey, Wilhelm
1926 The understanding of other persons and their life-expressions. Trans.
Kurt Mueller-Vollmer. In: Kurt Mueller-Vollmer (ed.), The Herme-
neutics Reader, 152-164. New York: Continuum, 1988.
Gallagher, Shaun
1986 Body image and body schema: A conceptual clarification. Journal of
Mind and Behavior 7: 541-554.
1995 Body schema and intentionality. In: Jose Bermudez, Naomi Eilan and
Anthony Marcel (eds.), The Body and the Self, 225-244. Cambridge:
MIT/Bradford Press.
1997 Mutual enlightenment: recent phenomenology in cognitive science.
Journal ofConsciousness Studies 4 (3): 195-214
2001 The practice of mind: Theory, simulation, or interaction? Journal of
Consciousness Studies 8 (5/7): 83-107.
2004 Hermeneutics and the cognitive sciences. Journal of Consciousness
Studies 11 (10/11): 162-174.
Gallagher, Shaun, George Butterworth, Adina Lew and Jonathan Cole
1998 Hand-mouth coordination, congenital absence of limb and evidence
for innate body schemas. Brain and Cognition 38: 53-65.
Gallagher, Shaun and Jonathan Cole
1995 Body schema and body image in a deafferented subject. Journal of
Mind and Behavior 16: 369-390.
Gallagher, Shaun and Andrew Meltzoff
1996 The earliest sense of self and others: Merleau-Ponty and recent de-
velopmental studies. Philosophical Psychology 9: 213-236.
Gallagher, Shaun and Francisco Varela
2002 Redrawing the map and resetting the time: Phenomenology and the
cognitive sciences. In: Steven Crowell, Lester Embree and Samuel J.
Julian (eds.), The Reach of Reflection: The Future of Phenomenol-
ogy, 17-45. ElectronPress; Reprinted in Canadian Journal of Phi-
losophy, Supplementary Volume 29, 2003: 93-132.
Gallese, Vittorio
1998 Mirror neurons: From grasping to language. Paper read at Tucson III
Conference: Towards a Science of Consciousness (Tucson 1998).
2005 Embodied simulation: From neurons to phenomenal experience.
Phenomenology and the Cognitive Sciences 4 (1): 23-48
290 Shaun Gallagher
Gallese, Vittorio, Luigi Fadiga, Leonardo Fogassi and Giacomo Rizzolatti
1996 Action recognition in the premotor cortex. Brain 119: 593-609.
Gardner R. M. and C. Moncrieff
1988 Body image distortion in anorexics as a non-sensory phenomenon: A
signal detection approach. Journal 0/ Clinical Psychology 44: 101-
107.
Georgieff, Nicolas and Marc Jeannerod
1998 Beyond consciousness of external events: A "Who" system for con-
sciousness of action and self-consciousness. Consciousness and
Cognition 7: 465-477.
Grezes, Julie and Jean Decety
2001 Functional anatomy of execution, mental simulation, observation and
verb generation of actions: A meta-analysis. Human Brain Mapping
12: 1-19.
Guldin, W. 0., SchahramAkbarian and o. J. Griisser
1992 Cortico-cortical connections and cytoarchitectonics of the primate
vestibular cortex: A study in squirrel monkeys (Saimiri sciureus).
Journal o/Comparative Neurology 326: 375-401.
Guillaume, P.
1943 Psychologie. Paris: Presses Universitaires de France.
Head, Henry
1920 Studies in Neurology. Vol. 2. London: Oxford University Press.
Husserl, Edmund
1907 Ding und Raum. Husserliana 16. The Hague: Martinus Nijhoff,
1973; English translation, Richard Rojcewicz. Thing and Space:
Lecturs of1907. Dordrecht: Kluwer Academic.
1952 Ideen zu einer reinen Phiinomenologie und phiinomenologischen
Philosophie, WaIter Biemel (ed.), Husserliana 4. Dordrect: Kluwer.
English translation: Richard Rojcewicz and Andre Schuwer. Ideas
Pertaining to a Pure Phenomenology and to a Phenomenological
Philosophy Second Book: Studies in the Phenomenology of Consti-
tution. Dordrecht: Kluwer. 1989.
1973 Zur Phiinomenologie der Intersubjektivitiit Texte aus dem Nachlaj3.
Erster Teil. 1905-1920.1. Kern (ed.). Dordrecht: Kluwer Academic.
James, William
1890 The Principles ofPsychology. New York: Dover, 1950.
Jeannerod, Marc
1997 The Cognitive Neuroscience of Action. Oxford: Blackwell Publish-
ers.
2001 Neural simulation of action: A unifying mechanism for motor cogni-
tion. NeuroImage 14: 103-109
Jeannerod, Marc and Shaun Gallagher
2002 From action to interaction: An interview with Marc Jeannerod.
Journal ofConsciousness Studies 9 (1): 3-26.
Jouen, Fran90is and Olivier Gapenne
1995 Interactions between the vestibular and visual systems in the neonate.
In: Pierre Rochat (ed.), The Self in Infancy: Theory and Research,
277-301. Elsevier Science B. V.
Lhermitte, J.
1939 L'image de notre corps. Paris: Nouvelle Revue Critique.
Meltzoff, Andrew
1993 Molyneux's babies: Cross-modal perception, imitation, and the mind
of the preverbal infant. In: N. Eilan, R. Mccarthy, and B. Brewer
(eds.), Spatial Representation: Problems in Philosophy and Psy-
chology, 219-235. Oxford: Basil Blackwell.
Meltzoff, Andrew and M. Keith Moore
1977 Imitation of facial and manual gestures by human neonates. Science
198: 75-78.
1983 Newborn infants imitate adult facial gestures. Child Development,
54: 702-709.
1994 Imitation, memory and the representation of persons. Infant Behavior
and Development 17: 83-99.
1997 Explaining facial imitation: A theoretical model. Early Development
and Parenting 6: 179-192.
1945 Phenomenologie de la perception. Paris: Gallimard; English transla-
tion: Colin Smith Phenomenology ofPerception. London: Routledge
and Kegan Paul, 1962.
1956-60 La nature: Notes, cours du College de France. Paris: Editions de
Seuil, 1995. La Nature: Course notes from the College de France.
Trans. Robert Vallier. Evanston: Northwestern University Press,
2003.
1960 Les relations avec autrui chez l'enfant. Paris: Cours de Sorbonne;
English translation: William Cobb, The child's relations with others.
In: James M. Edie (ed.), The Primacy of Perception. Evanston:
Northwestern University Press, 1964.
Ogden, Jenni A.
1996 Fractured Minds: A Case-Study Approach to Clinical Neuropsy-
chology. Oxford: Oxford University Press.
Overgaard, Morten
2004 On the naturalising of phenomenology. Phenomenology and the
Cognitive Sciences 3 (4) 365-379.
292 Shaun Gallagher
1962
1966
1996
Piaget, Jean
1945 La formation du symbole chez l'enfant, Neuchatel-Paris: Delachaux
et Niestl6; English translation: G. Gattegno and F. M. Hodgson. Play,
Dreams and Imitation in Childhood. New York: Norton, 1962.
Poeck, K. and B. Orgass
1971 The concept of the body schema: A critical review and some experi-
mental results. Cortex 7: 254-277.
Powers, P. S., R. G. Schulman, A. A. Gleghom and M. E. Prange
1987 Perceptual and cognitive abnormalities in bulimia. American Journal
ofPsychiatry 144: 1456-1460.
Pribram, Karl H.
1999 Brain and the composition of conscious experience. Journal of Con-
sciousness Studies 6 (5): 19-42.
Rizzolatti, Giacomo, L. Fadiga, M. Matelli, V. Bettinardi, E. Paulesu, D. Perani
and G. Fazio
Localization of grasp representations in humans by PET: 1. Observa-
tion compared with imagination. Experimental Brain Research 111:
246-252.
Roth, Wolff-Michael
this vol. Communication as situated embodied practice.
Ruby, Perrine and Jean Decety
2001 Effect of subjective perspective taking during simulation of action: a
PET investigation of agency. Nature Neuroscience 4 (5): 546-550.
Scheler, Max
1948 Wesen und Formen deer Sympathie. Frankfurt am Main: G. Schulte-
Bumke, 1948; originally published as Zur Phlinomeno-logie der
Sympathiegefuhle und von Liebe und Hass. Halle: Niemeyer, 1913.
English translation: Peter Heath, The Nature of Sympathy. London:
Routledge and Kegan Paul, 1954.
Simmel, Marianne L.
1958 The conditions of occurrence of phantom limbs. Proceedings of the
American Philosophical Society 102: 492-500.
Phantoms - experiences following amputation in childhood. Journal
ofNeurology, Neurosurgery andpsychiatry 25: 69-78.
Developmental aspects of the body scheme. Child Development 37:
83-95.
Sonesson, Goran
this vol. From the meaning of embodiment to the embodiment of meaning. A
Varela, Francisco
1996 Neurophenomenology: A methodological remedy for the hard prob-
lem. Journal ofConsciousness Studies 3 (4): 330-349.
Wallon, Henri
1925 Stades et troubles du developpement psycho-moteur et mental chez
l'enfant. Paris: Alcan.
Zahavi, Dan
2004 Phenomenology and the project of naturalization. Phenomenology
and the Cognitive Sciences 3 (4): 331-347.
Zlatev, Jordan
Section C
Body, language and culture
Embodiment, language, and mimesis
Jordan Zlatev
For years now, leading representatives of theoretical
linguistics have been arguing that humans, being gov-
erned by a blind 'language instinct', can be exhaus-
tively described in physico-biological terms. ... [T]his
conception has been shown to be fundamentally false.
Humans are also, and crucially, social, normative, and
conscious beings, occasionally capable of acts of free
will. Esa Itkonen, What is Language?
Abstract
The present focus on embodiment in cognitive science undervalues concepts such
as convention/norm, representation and consciousness. I argue that these concepts
constitute essential properties of language, and this makes it problematic for "em-
bodiment theories" to account for human language and cognition. These difficulties
are illustrated by examining a particular, higWy influential approach to embodied
cognition, that of Lakoff and Johnson (1999), and exposing the problematic char-
acter of the notion of the "cognitive unconscious". To attempt a reconciliation
between embodiment and language, I turn to the concept of (bodily) mimesis, and
propose the notion of mimetic schema as a mediator between the individual human
body and collective language.
Keywords: bodily mimesis, consciousness, conventions, mimetic schemas, repre-
sentation.
1. Introduction
The main goal of this chapter is to investigate the relationship between
language and the concept of embodiment which has become a central, if
ambiguous, notion within cognitive science (e.g. Varela, Thompson and
Rosch 1991; Clark 1997; Ziemke 2003), the neuroscience of consciousness
(e.g. Edelman 1992; Damasio 1994, 2000), (neuro)phenomenology (e.g.
298 Jordan Zlatev
Varela 1996; Thompson 2001; Thompson and Varela 2001; Gallagher
1995, 2005, this volume), cognitive linguistics! (e.g. Lakoff 1987; Johnson
1987; Zlatev 1997; Svensson 1999; Evans 2003) and to some extent devel-
opmental psychology (e.g. MacWhinney 1999; Mandler 2004). The notion
of embodiment is, indeed, even intended to unite efforts in these different
fields into what is often called "second generation cognitive science"
(Lindblom and Ziemke this volume) or "embodied cognition" (Johnson and
Rohrer this volume). There is much to recommend in this (re)turn to the
body in the study of the mind, especially since in many ways it can be seen
as a justified reaction to the many shortcomings of "classical" information-
processing cognitive science according to which the "mindlbrain" works
essentially as a computer (e.g. Fodor 1981; Jackendoff 1987; Pinker 1994).
There are, however, three major unresolved issues within the current
"embodiment turn" in the sciences of the mind. The first was mentioned in
passing already: there is not one but many different meanings behind the
term "embodiment", both between and within fields, and the corresponding
theories are in general not compatible (Ziemke 2003). In particular, I would
claim, there is no uniform concept of representation within "embodied
cognition", and this is a constant source of (misguided) debate, both be-
tween proponents of embodiment and between them and representatives of
the "algebraic mind" (Marcus 2001). Second, by their nature, embodiment
theories have a strong individualist orientation, and despite recurrent at-
tempts to connect embodiment to social reality and culture (e.g. Palmer
1996; Zlatev 1997; Sinha 1999), there is still no coherent synthesis. In
particular, within the work of those emphasizing the role of the "body in
the mind" there is no adequate notion of convention or norm, which is es-
sential for characterizing both human culture and the human mind. Third,
there is a dangerous tendency to underestimate the role of consciousness in
many - though not all - embodiment theories. There seems to be some sort
of fear that in appealing to anything that is irreducible to either biology or
behavior, one is bound to fall into the clutches of "Cartesian dualism". The
consequence is, however, that such "non-dualistic" approaches run the risk
1. When using small letters, i.e. cognitive linguistics, I will refer to the work of
linguists who regard language and cognition as intimately connected (e.g. It-
konen, Levinson and Jackendoff). When used with capital letters, Cognitive
Linguistics refers to the school of linguistics departing from the work of Lakoff,
Langacker and Talmy. The borders are admittedly fuzzy, but in general, Cogni-
tive Linguistics is a hyponym (extensionally speaking a subset) of cognitive lin-
guistics.
Embodiment, language, and mimesis 299
of one form or another of physico-biological reductionism, which as
pointed out by Itkonen in the motto to this chapter is deeply misguided.
To substantiate these claims in detail would require an extensive review
of the literature, which the allotted space of a book chapter does not permit
me. My strategy will therefore be to single out one of the above mentioned
fields, cognitive linguistics, and even more narrowly, focus on a single
exposition of "embodiment theory": Philosophy in the Flesh (PitF) by
George Lakoff and Mark Johnson (1999). This choice is motivated by the
following reasons: (a) Lakoff and Johnson are two of the foremost propo-
nents of "embodied cognition" not only in (cognitive) linguistics, but in
general, (b) PitF is their most recent extensive joint publication, and it is
often mentioned as one of the three major reference works on embodiment
up to date, along with Varela et al. (1991) and Damasio (1994), and (c)
while philosophically oriented, the work deals with implications from lin-
guistic research, and it is precisely in relation to language that the difficul-
ties of "embodiment theory" are most clearly accentuated.
2
The problem reveals itself when we ask the seemingly simple question:
In what sense can (knowledge of) language be said to be "embodied"?
Prior to answering this question, however, we need to step back and ad-
dress, ifbriefly, the fundamental question: What is language? In the mono-
graph with this title, from which the opening quotation was taken, Esa It-
konen persuasively argues that the nature of language has been commonly
misunderstood in modem "theoretical linguistics" (including both the gen-
erative and the cognitive/functional paradigms). Instead of "instincts",
"cognitive modules", "neural mechanisms" or "usage", Itkonen (1978,
1983, 1991, 2003) offers a very clear and intuitive answer: Language is a
social institution for communicating meanings, a conception with sound
roots in the tradition, e.g. Saussure (1916), Trubetzkoy (1939) and Witt-
genstein (1953). As such, language exists primarily between people rather
than (only) within people. It is "shared" by the members of the community
who speak it - in the strong sense in which people can "share a secret":
they all know it, and they know that they know it, rather than in the weak
sense of "sharing a bottle of wine". But what is it that people share when
they know a language? Above all: linguistically encoded concepts, i.e.
2. I should point out that my own previous work on language and embodiment
(Zlatev 1997) suffers from the same three drawbacks listed above, i.e. it lacks
coherent concepts of representation and convention and, in addition, disregards
their dependence on consciousness. My criticism of "embodiment theory" in the
frrst part of this chapter is therefore also a form of (former-)self-criticism.
300 Jordan Zlatev
lexical meanings, and rules for their combination. In Section 2 of this
chapter I will elaborate on this, and argue that it is impossible to account
for linguistic meaning without the concept of representation. Nearly as
obviously, the conventionality of language, as well as the fact that we fol-
low rules (which we are free to break) rather than mechanical deterministic
procedures shows that our knowledge of language is (in principle) accessi-
ble to consciousness. This also implies that linguistic knowledge involves
declarative, and not only procedural knowledge.
3
This characterization of language in terms of conventionality, repre-
sentation and accessibility to consciousness appears to be on a collision
course with attempts to explain language in terms of "embodiment", since
as pointed out above, it is precisely these three concepts that are at best
underdeveloped, and at worse rejected by proponents of embodied cogni-
tion. In the recent work of cognitive linguists such as Johnson and Lakoff,4
and especially in PitF, this dissonance turns into an outright contradiction.
In Section 3 I analyse the concept of embodiment as explicated within PitF
(with some references to other Cognitive Linguistic work to show that PitF
is by no means an exception), in order to make this contradiction as clear
as possible. In brief: if language has the properties that I claim, and if em-
bodiment has the properties that Lakoff and Johnson claim, then language
can not be embodied. And since language is not just a "module" of the
human mind - something that Cognitive Linguistics emphasizes - but
largely constitutive of it (e.g. Vygotsky 1934; Nelson 1996; Tomasello
1999), then the human mind cannot be embodied either.
3. Mandler (2004) eloquently argues for the need to distinguish between declara-
tive, conceptual knowledge, which is accessible to consciousness, and proce-
dural, sensorimotor skills, which are not (cf. Ikegamie and Zlatev, this volume).
While language learning and use undoubtedly involve both types, it is a mistake
to attempt to reduce all linguistic knowledge to procedural "know-how" as e.g.
done by Zlatev (1997). Consciousness is a multifaceted phenomenon (and con-
cept) but similarly to Mandler, in this chapter I focus on the deliberative aspect
of consciousness, rather than on its qualitative, experiential aspect. Also it
should be noted that in stating that something is accessible to consciousness,
this does not imply that it is, of course, accessed in any particular moment. Con-
sciousness has a center-periphery structure, so of necessity some of the objects
of consciousness will be in the "margins" (Gurwitsch 1964).
4. Though admittedly, this was less obvious in their earlier formulations, such as
their rather inspiring Metaphors We Live By (Lakoff and Johnson 1980), as well
as Johnson (1987).
However, the overall goal of this chapter is not to criticize the short-
comings of "embodiment theory", but to attempt to show how the concept
can be developed in order to resolve the contradiction laid out in the previ-
ous paragraph. The first step is to argue in Section 4 that the PitF notion of
"embodiment" is indeed not viable, and therefore a replacement is re-
quired. Then I proceed in Section 5 with an attempt if not to fill, at least to
minimize the gap between language and embodiment through the concept
of bodily mimesis, understood along the lines of Donald (1991, 2001) as
the volitional use of the body for constructing and communicating repre-
sentations. On this basis, I offer conceptual and empirical support for a
novel theoretical concept, mimetic schemas, which constitute body-based,
pre-linguistic, consciously accessible representations that serve as the
child's first concepts (Zlatev 2005). Furthermore, mimetic schemas possess
a basic intersubjectivity which can serve as the foundation for developing a
conventional-normative semiotic system, i.e. language. In Section 6, I
briefly outline how the concept of mimetic schemas can contribute to the
(hopeful) resolution of a number of puzzles in explaining language evolu-
tion, acquisition and spontaneous gesture. Finally, I summarize the argu-
ment.
2. Language
The claim that language is primarily a social institution for communicating
meanings, stated in the introduction, is customarily met with incomprehen-
sion by linguists and psychologists.
5
To put the objection into the termi-
nology of this volume: what is the "embodiment" of this institution? Part of
it may be in writing systems and other artifacts (Donald 1991; Clark 1997;
Sonesson this volume), but would not language cease to exist if it were not
instantiated within the minds of its users, the individual speakers? Well,
this can be debated since one can argue that "dead languages" are not
really dead if they have been preserved in written texts and especially in a
5. This statement may seem to contradict occasional remarks in the cognitive lin-
guistic literature concerning the "social dimension" of language, and the fre-
quent use of phrases such as "conventional imagery" (Langacker 1987) and
"conventional metaphor" (PitF). The implications of these remarks are, how-
ever, never explored. In particular, it is never explained how is it possible that
individual mental phenomena such as imagery and metaphorical "mappings"
can at the same time be conventional, i.e. social.
302 Jordan Zlatev
grammatical description, because that would allow them to be "recreated"
by studying the texts and grammar, which is more like (collective) remem-
bering than rediscovery. But, of course, it must be granted that language is
an individual as well as a social phenomenon and none (or very few) of the
social accounts of language has ever denied this. However, even as an in-
dividual psychological phenomenon, as say, knowledge of English rather
than the social institution English, language can be shown to consist of
conventional representations accessible to consciousness. Let me try to
explicate. What do I need to know in order to understand (1), which has
been uttered by, say, Peter? Minimally, I would need to know the (social)
facts (2) - (7).
(1) John kissed Mary.
(2) The word kiss means KISS.
(3) The words John and Mary are names of a male and a female human
being, respectively.
(4) The word order shows that John kissed Mary, rather than vice versa.
(5) The past tense signifies that the event described occurred sometimes
in the past relative to the time of utterance.
(6) The sentence (normally) expresses an assertion.
(7) The names John and Mary actually refer to individual X and Y.
But this is not enough to guarantee that I understand Peter. Imagine that I
know (2-7), but Peter, who has had a rather idiosyncratic upbringing,
thinks that kiss means HIT-ON-THE-HEAD. I will then fail to understand the
meaning of (1) as meant by Peter. So I must also know that Peter knows
(2-7). Furthermore, I must know, or at least assume, that Peter knows that I
know (2-7). For if Peter thinks that I've had a strange upbringing, or
maybe as a foreigner I do not have a proper command of English, then he
may not be using (1) in its conventional way, even though he knows (2-7).
If this seems far-fetched, consider only (7), which involves not the meaning
(Sinn) of the names John and Mary but their reference - or Bedeutung ac-
cording to the classical distinction of Frege (1882 [1997]). Here it is easier
to see that unless Peter and I can be quite sure not only that both of us
know who the names refer to in this context, but that Peter knows that I
know, and I know that Peter knows, there might be a misunderstanding. For
instance, I am thinking of Mary Smith, and Peter is thinking of Mary
Smith. But if I don't lrnow that Peter lrnows that I am thinking of Mary
Smith rather than Mary Williams, then I couldn't be sure who he is really
referring to by Mary in uttering (1).
This type of reflexively shared knowledge is known as common knowl-
edge (Itkonen 1978), mutual knowledge (Clark and Marshal 1981) or com-
mon ground (Clark 1996). A convenient way to say that (2-7) are part of
common knowledge is to say that they are conventions (Lewis 1969; Clark
1996), norms (Itkonen 1978) or even rules (Wittgenstein 1953; Searle
1969).6 These closely related terms have rather complementary implica-
tions, so while I will predominantly use the term conventions to refer to our
knowledge of facts such as (2-7), it is crucial to remember that this knowl-
edge is normative, in the sense that one can be right or wrong according to
public criteria of correctness (Wittgenstein 1953; Baker and Hacker 1984),
in one's use of these conventions. This normativity can be on various lev-
els of explicitness and scope ranging from prescriptive grammars for the
"national language" to intuitions about "the way we talk in our family".
However, it is always social and always involves a degree of conscious
awareness, since to be following a convention/norm/rule - as opposed to
the movement governed by a reflex or a blind habit - one must be able to
compare it to actual usage and notice any potential mismatch. It is sense-
less to talk about this noticing of a difference between "should" and "is"
without being aware of the difference and this implies at least a degree of
consciousness. Such conscious processes of noticing and judgment are also
essential for the acquisition of language by pre-verbal children (e.g. Bloom
2000) and by second-language learners (Schmidt 1990). As argued at
length by Mandler (2004: 228), without consciousness, language acquisi-
tion could not come off the ground:
The ability to make an old-new distinction requires awareness of prior oc-
currence or pastness; its loss is one of the hallmarks of amnesia. Amnesiacs
retain the ability to be influenced by past experience and to learn at least
certain new skills, but they have lost the awareness that these experiences
are familiar to them.
6. Unfortunately, all these terms have other (negatively charged) meanings when
applied to language, thus conventional is often identified with "arbitrary". Norm
has bad connotations for linguists since it is associated with "normative gram-
mar", which prescribes rather than describes. Finally, rule is often interpreted as
an explicit, algorithmic, non-creative procedure, which is just about the oppo-
site of what e.g. Wittgenstein (1953) meant by "rule-following".
304 Jordan Zlatev
One of the things that amnesiacs can not learn is a new language, implying
that language can not be acquired by processes of implicit learning of the
type that are modeled by most connectionist models (e.g. Elman 1990),
which do not require conscious awareness.
Thus we can conclude that knowing and learning conventions such as
(2-7) involves making them accessible to consciousness. Notice that I am
not claiming that consciousness is involved in every aspect of language
learning and use: it is beyond doubt that implicit learning and procedural
knowledge are important as well. My claim is that consciousness is at least
essential for (a) the acquisition of concepts and rules, (b) the ability to
notice any "breaking" of the rules and (c) all forms of meta-linguistic
knowledge. It is (b) and (c) that are the basis for all grammaticality judg-
ments and linguistic analysis and thus for traditional or "autonomous" lin-
guistics (Itkonen 1978, 1991). On the other hand, attempts to make lin-
guistic theories "psychologically real" have always attempted to reconcile
the analysis obtained from (b-c) with the learner's perspective in (a).
While there are obvious differences in the three processes (a), (b) and (c),
conscious awareness unites them, and sets them apart from the "auto-
mated" procedures that underlie reflexes and habits of the kind that govern
the behavior of most animals, and which are also important for human
beings.
Language conventions can concern pronunciation (phonology) or the
combinations of words and phrases (morphology and syntax), but the most
important conventions and those that distinguish language from other con-
vention/norm/rule systems such as those in dancing tango, boxing or eating
at a restaurant concern semantics and pragmatics. In all the aforementioned
activities there is a "right" and a "wrong" way of doing things and that is
how we know that they are conventional-normative. But in language (and
some other semiotic systems) one can be right and wrong representation-
ally.
There are two ways in which linguistic utterances like (1) can be prop-
erly regarded as representations. Both are conveniently explicated by the
classical semiotic triangle (Ogden and Richards 1923), displayed using
generic terms for its three relata in Figure 1.
First, the relationship between Expression and Meaning, the latter con-
sidered as conventional context-general content, is that of the classical
Sausserian sign, the first one corresponding to the "signifier", the second to
the "signified". What 100 years of theoretical linguistics and especially
functional/cognitive linguistics (Giv6n 2001; Lakoff 1987) have added to
this basic insight is that the relationship need not be as "arbitrary" as Saus-
sure assumed, especially considering that grammatical constructions are
also a kind of sign, and these are at least to some degree motivated by fac-
tors such as iconicity and indexicality (and are thus not classical Peircean
"symbols"). This, however, does not mean that the mapping between Ex-
pression and Meaning is any less conventional (Zlatev 2003). The first five
of the conventions involved in understanding (1) as an English sentence
(2-6) involve linguistic signs in this sense.
Meaning
Expression Reality
Figure 1. The "semiotic triangle", after Odgen and Richards (1923).
What about the relationship Meaning-Reality? First of all, age-old philo-
sophical problems concerning the "aboutness" of language can be resolved
by noting that it is not the expressions of language that relate directly to
reality (this is implicit in the notion of the semiotic triangle), and not
meaning in the sense of conventional content either, but rather meaning as
illocutionary (speech) acts, performed by speakers and hearers by inten-
tionally imposing illocutionary force on the propositional content of sen-
tences. Or as expressed succinctly by Searle (1999):
Language relates to reality in virtue of meaning, but meaning is the property
that turns mere utterances into illocutionary acts. (ibid: 139) [... ] The con-
ventional intentionality of the words and the sentences of a language can be
used by a speaker to perform a speech act. When a speaker performs a
speech act, he imposes his intentionality on those symbols. (ibid: 141)
306 Jordan Zlatev
There are three important aspects of this process in relation to our discus-
sion of the nature of language that need to be emphasized. First, the "impo-
sition of intentionality" on the part of the speaker (and its interpretation by
the hearer) is clearly dependent on conscious awareness - unless the
speaker is talking in his sleep and thus speaking "non-intentionally", in
both the everyday and the philosophical sense of the word. Second, at least
in the case of assertives including speech acts such as statements, descrip-
tions and classifications which have what Searle calls a "mind-to-world
direction of fit" we have a fairly clear representational relation between
Meaning and Reality: the speech acts are "pictures of reality" that can be
either true of false. This is not representation in the Saussurian sense but
rather in the sense of the Tractatus (Wittgenstein 1923 [1961]), with the
provision that it is utterances spoken by speakers that are true or false, not
sentences - as famously emphasized by Strawson (1950) in his critique of
Russell (1905).
It is this representational relationship that is denied by pragmatism, and
by many representatives of cognitive linguistics (Lakoff and Johnson 1999;
Johnson and Lakoff 2002; Johnson and Rorher this volume). But such ob-
jections seem to be beside the point, since they concern the metaphysics of
an "objective reality" and the epistemology of "objective truth", where
both sense of "objective" are understood as mind-independent. However,
all that is necessary in order to regard the relationship between a statement
and a state-of-affairs (SoAs) as a representation, is for: (a) the first to be
about that SoA, rather than just in association with it, (b) the speaker of the
statement to be aware of (a), and (c) the possibility or the state-
ment/representation to either match or not the SoA.
Nothing in (a-e) requires either the SoA or the matching with the state-
ment to be "mind-independent". These conditions are fulfilled in Lakoff
and Johnson's definition of "embodied truth" (1999: 106), so even in their
account the meaning of a (true) sentence can be regarded as a (matching)
representation of a situation. Even if the representational relation between
linguistic meaning and reality-as-conceived is to be rejected, for whatever
reasons, then there is still the Saussurian representational or "symbolic"
relationship between "the phonological" and "the semantic pole" (Lan-
gacker 1987), i.e. expression and content. In short, representation is simply
inescapable in accounting for language (Sinha 1988, 2005).
Finally, we should note that the "imposition of intentionality" men-
tioned by Searle in the previous quote is not a private, speaker-internal
matter, but is constrained firstly by the conventional meaning of the ex-
pression(s). This is what makes it difficult (though perhaps not impossible)
to express your love by saying I'll kick you. The second constraint is a
more situation-specific and dynamic sort of intersubjectivity, exemplified
by the need to have a "common ground" for figuring out the referent of the
names John and Mary in (7). In order to successfully refer, you need to
formulate your speech act in a way that will make the referent intersubjec-
tively "shared" for you and your hearer, and this requires a fairly keen sen-
sitivity to the norms of the language, to the situation and to your inter-
locutor's state of mind. All this is unthinkable without consciousness, as
also pointed out by Donald (2001), and takes quite some time and effort to
be mastered by children.
To sum up, the discussion in this section has pointed out the following
features that can be regarded as definitional of human language: conven-
tionality, implying normativity; representationality: between expression
and content and between an assertive speech act and reality; accessibility
to consciousness: necessary for the establishment of common knowledge
and for the management of successful communicative action.
7
A characteristic feature of language that has not been discussed is one
that is perhaps most often mentioned in discussions of the "uniqueness of
language" in respect to other human and animal systems of communication
- to the extent of forgetting those listed above - namely, the systematicity
of language (Saussure 1916; Deacon 1997). It is true that this is an essen-
tial feature of language, and something that for example distinguishes lan-
guage from gesture (McNeil 1992; Senghas, Kita and Ozyi1rek 2004). It
should be pointed out, however, that this concerns not the "syntax" of lan-
guage alone, but its general capacity to express an unlimited number of
meanings, both in the sense of content and speech acts. Finally, while the
primary function of language is social interaction, once internalized, it
becomes a representational vehicle of thought, transforming the cognition
of its user (Nelson 1996; Tomasello 1999).
Therefore, a suitable concise definition of language would be: A con-
sciously supervised, conventional representational system for communica-
tive action and thought. This is admittedly terse and different from what
one usually finds in linguistics textbooks, but it is no more than the com-
pact summary of the explication provided in this section. If this explication
7. Though, to remind once again, reflective consciousness need not be involved in
every aspect of learning, producing and understanding language.
308 Jordan Zlatev
has been clear enough, then its relative non-orthodoxy is no reason for it
not to be accepted.
3. Embodiment
Let us now turn to see how embodiment is defined within Cognitive Lin-
guistics, focusing on the recent work of Lakoff and Johnson, and above all
on PitF. Somewhat surprisingly, there is no straightforward definition of
"embodiment" to be found in a 624 page book with the subtitle The Em-
bodied Mind and its Challenge to Western Thought, the closest approxi-
mation being: "... there are at least three levels to what we are calling the
embodiment of concepts: the neural level, phenomenological conscious
experience and the cognitive unconscious" (PitF: 102). What are these ("at
least") three levels?
Starting from the bottom, we are told that "neural embodiment concerns
structures that characterize concepts and cognitive operations at the neural
level" (PitF: 102). It is furthermore claimed that this level "significantly
determines [... ] what concepts can be and what language can be" (PitF:
104). One of the most specific definitions of "an embodied concept" is
provided in terms of this level only: "An embodied concept is a neural
structure that is part of, or makes use of the sensorimotor system of our
brains. Much of conceptual inference is, therefore, sensorimotor inference"
(PitF: 20, original emphasis). However, Lakoff and Johnson make it clear
that they will not deal with the nitty-gritty of neurobiology like "ion chan-
nels and glial cells" (PitF: 103) since the neural level refers to a higher-
level generalization that is heavily dependent on "an important metaphor to
conceptualize neural structure in electronic terms" (PitF: 103). Thus, the
connectionist model of Regier (1996) is given as an instance of "neural
modeling", even though it is quite removed from what is known about the
brain (and even though Regier does not apply the adjective "neural" to the
model himself and repeatedly points out that his model is only inspired by
some aspects of neural systems).
The next level, "phenomenological embodiment", is devoted much less
attention. Its first definition is "[... ] the way we schematize our own bodies
and things we interact with daily" (PitF: 36), with reference to the phe-
nomenological tradition and specifically the work on the body schema and
the body image of Gallagher (1995). The second definition is considerably
broader: "It [i.e. phenomenological embodiment] consists of everything we
can be aware of, especially our own mental states, our bodies, our envi-
ronment and our physical and social interactions. This is the level at which
we speak of the "feel" of experience [... ]" (PitF: 103). What the authors do
not make clear is whether all conscious experience should be considered as
"phenomenological embodiment", and if so, why this is the case. At the
same time, they point out that "phenomenology also hypothesizes noncon-
scious structures that underlie and make possible the structure of our con-
scious experience" (PitF: 103). This heralds the arrival of the main hero of
Lakoff and Johnson's account of embodiment: the "cognitive uncon-
scious."
The cognitive unconscious is the massive portion of the iceberg that lies
below the surface, below the visible tip that is consciousness. It consists of
all those mental operations that structure and make possible all conscious
experience, including the understanding and use of language. (PitF: 103)
This level is said to be "the realm of thought that is completely and irrevo-
cably inaccessible to direct conscious introspection" (PitF: 12) and
(nearly) all-pervasive: the cognitive unconscious constitutes "the 95 per-
cent below the surface of conscious awareness [that] shapes and structures
all conscious thought" (PitF: 13). In case the reader should wonder how
this all-important level (of embodiment) that is "completely and irrevoca-
bly inaccessible" was discovered, Lakoff and Johnson point out that it is
"hypothesized on the basis of convergent evidence, [... ] required for sci-
entific explanation" (PitF: 115) and that "the detailed processes and struc-
tures of the cognitive unconscious (e.g., basic-level categories, prototypes,
image schemas, nouns, verbs, and vowels) are hypothesized to make sense
of conscious behavior" (PitF: 104). So it turns out that this all-important
level of embodiment is a hypothetical theoretical construct. It is clear that
Lakoff and Johnson feel pressed to defend the "reality" of this construct
and they attempt to do so repeatedly. Perhaps the most revealing statement
is "To say that the cognitive unconscious is real is very much like saying
that neural "computation" is real" (PitF: 104). But is neural computation
"real"? We will return to this in the next section.
What can one say of Lakoff and Johnson's notion of embodiment? It is
obviously in contradiction with the account of language presented in Sec-
tion 2. Not only does PitF imply that "95 percent of all thought" and con-
sequently of language is completely below the level of conscious aware-
ness, Lakoff and Johnson's definition of "embodiment" has no real place
for the two central concepts of conventionality and representation. Re-
garding the first, there are frequent references to "conventional mental
310 Jordan Zlatev
imagery" (PitF: 45), but it is not even made clear whether this imagery is
conscious or only part of the "cognitive unconscious" - not to mention the
question of how this imagery would be shared, and furthermore mown to
be shared, which is necessary for it to be conventional. One could say the
same for the use of the term "conventional metaphor" in the cognitive lin-
guistic literature - there is nothing "conventional" about neurally realized
domain-to-domain mappings, at least in any conventional use of the term
convention (e.g. Lewis 1969, see footnote 5).
When Lakoff and Johnson feel pressed to account for shared meanings,
they do point out that "commonalities [... ] exist in the way our minds are
embodied" (PitF: 4) and that "we all have pretty much the same embodied
basic-level and spatial-relations concepts" (PitF: 107). But this is clearly
not enough to give you conventions such as those of (2-7) and to account
for how a simple English sentence such as (1) is understood.
Concerning the concept of representation, Lakoff and Johnson represent
quite clearly the anti-representationalist Zeitgeist within "second genera-
tion" cognitive science (e.g. Varela et al. 1991), which as pointed out in the
introduction eschews the concept of representation in reaction to its over-
use in "classical" cognitive science (e.g. Fodor 1981). In a recent (polemi-
cal) publication of the two authors this is made explicit:
As we said in Philosophy in the Flesh, the only workable theory of repre-
sentations is one in which a representation is a flexible pattern of organism-
environment interactions, and not some inner mental entity that somehow
gets hooked up with parts of the external world by a strange relation called
'reference' . We rej ect such classical notions of representation, along with
the views of meaning and reference that are built on then. Representation is
a term that we try carefully to avoid. (Johnson and Lakoff2002: 249-250)
A similar if not stronger form of anti-representationalism is advanced by
Johnson and Rohrer (this volume: Section 6):
We have been arguing against disembodied views of mind, concepts, and
reasoning, especially as they underlie Representationalist theories of mind
and language. Our alternative view - that cognition is embodied - has roots
in American Pragmatist philosophy and is being supported and extended by
recent work in second-generation cognitive science.
In their urge to dissociate themselves from any "disembodied views of
mind", scholars like Lakoff, Johnson and Rohrer, as well as many other
representatives of second-generation cognitive science (e.g. Brooks 1999)
can be said to overkill (mental) representations. It is one thing to (justly)
argue against "representations" in perception and active involvement, as
done by Dreyfus (1972 [1993]) with support from the phenomenological
tradition (e.g. Merleau-Ponty 1945 [1962]), and quite another to deny that,
say, a picture is a representation of whatever it depicts, irrespective of
whether the latter exists in the "real world" or not (Sonesson 1989, this
volume). It is in this latter sense that some, though not all, language use is
representational. Furthermore, to deny that assertions are a kind of repre-
sentation is to deny that a description of a situation can be either true or
false. As pointed out in Section 2, Lakoff and Johnson should not really
deny this since in their definition of "embodied truth" a person holding a
sentence to be "true" is said to understand the sentence to "accord" with
"with what he or she understands the situation to be" (PitF: 106). This is
clearly a roundabout way of saying that the person understands the sen-
tence to represent the situation correctly. But what is won from such avoid-
ance of the notion? There is nothing "strange" or "metaphysical" in the
concepts of representation and reference once it is understood that these
are performed by conscious speakers (and signers), not by the expressions
in the language themselves. To restrain oneself from using these concepts
in accounting for language is to make it impossible to account for the dif-
ference between language and perception, or between theatre and love-
making. (Though admittedly, the latter may be more fun.)8
In this section I have tried to make it as clear as possible that there is a
contradiction between the account of language presented in Section 2 and
the account of embodiment given by Lakoff and Johnson in PitF, which I
have suggested is not atypical for much of "embodied cognition" or "sec-
ond generation cognitive science". If my account of language and Lakoff
and Johnson's account of embodiment are both accepted, then it follows
that "embodiment theory" cannot account for language, and since language
is a central part of the human psyche, it cannot account for the latter either.
8. Rather more troublesome is the fact that in a pragmatist evolutionary theory
insisting on the "continuity" of all cognition such as that of Johnson and Rohrer
(this volume) there is no place for a qualitative distinction between the cognition
of human beings and ants... Compare: "According to our interactionist view,
maps and other structures of organism-environment co-ordination are prime ex-
amples of non-representational structures of meaning, understanding, and
thought." (ibid: Section 3.3) with "Ant cognition is thus nonrepresentational in
that it is both intrinsically social and situated in organism-environment interac-
tions." (ibid: Section 5)
312 Jordan Zlatev
This negative conclusion can be avoided in one of two ways: Lakoff
and Johnson (and their colleagues) would presumably argue that I have
misconstrued language. The alternative, which (unsurprisingly) I undertake
in the following section, is to argue that the concept of embodiment pre-
sented in PitF is inadequate, as a preliminary to suggesting how the con-
cept of bodily mimesis can contribute to a more adequate notion of human
embodied cognition that naturally combines with the three essential fea-
tures of language: convention, representation and accessibility to con-
sciousness.
4. Embodiment lost? A critique of Lakoff and Johnson (1999)
Let us begin with Lakoff and Johnson's first level of embodiment: the
"neural level". An obvious question to ask is why the exclusive focus on
the brain (and the rest of the nervous system) at the expense of the whole
living body? One reason seems to be that the activity of the brain could
possibly be understood "computationally" - using the "neural computa-
tion" metaphor - while that of the whole bio-chemistry of the body cannot,
in any remotely meaningful way. Another reason seems to be that the "non-
neural" parts of the body are not considered relevant for the "shaping" of
cognition. It seems to be that for Lakoff and Johnson "brain and body are
used as substantially interchangeable" (Violi 2003: 205). Leaving for the
time being phenomenological aspects, this is still deeply problematic. Is a
brain-in-a-vat just as embodied as a living body? There are at least two
good, more or less obvious, reasons to doubt this. First, all sensorimotor
interactions with the environment are performed by using our limbs, mus-
cles, eyes, ears, nose, skin, tongue etc. - not with the somatosensori cortex
itself. Or is it so that Lakoff and Johnson hold that these periphery systems
are merely "transducers" and could equally well be substituted by artificial
correlates managing the input-output of electrical signals to the brain?
Whatever the tenability of this position, it is clearly a very "non-embodied"
way to think of cognition, and, for that part, of the brain itself (see Lind-
blom and Ziemke this volume). The second reason is that the living body
participates not only in interaction with the environment, but in evaluation
of it - at least according to somatic theories of emotion such as that of
Damasio (1994, 2000). According to Damasio certain regions of the brain
constantly monitor the state of the whole body, and depending on its "well-
being" judge external stimuli (though as we all know, people have found
many ways to trick these monitoring systems over the ages, allowing them
to "feel good" while their body is not thriving). If this is still somewhat
speculative, let me simply remind of an aspect of our non-neural bodies
that has a strong effect on our emotional life, and thereby on our thinking:
the hormonal system. What all this points to is that even when regarding
the body from an external, "third-person" perspective, it is a gross simplifi-
cation to consider only the nervous system as relevant for cognition. The
living body as a whole is relevant, and the kind of embodiment this in-
volves could be called simply "biological" or perhaps "organismic"
(Ziemke 2003; Zlatev 2003).
Turning now to the "phenomenologicallevel" of Lakoff and Johnson's
three-level notion of embodiment (in PitF), we can notice the opposite
tendency: if there was an under-extension of the role of the body when
regarding embodiment as a biological phenomenon, there appears now to
be an over-extension by equating bodily awareness with all conscious ex-
perience, i.e. "everything we can be aware of' (PitF: 103). While it is clear
that phenomenal bodily experience is involved in physical interactions,
either with the inanimate environment or in physical social interactions
such as chasing, wrestling, love-making... it is far from obvious what role
the body schema, or even the body image (Gallagher 1995, 2005, this vol-
ume) play in more detached social interactions, such as tax payment -
while I am presumably conscious when I fill in my tax-return forms. Lakoff
and Johnson never address this problem, which is unsurprising since con-
sciousness is on the whole treated by the authors in a rather step-motherly
fashion: tolerated out of necessity but neglected.
It is characteristic that others who have given a more prominent role to
consciousness or "subjectivity" in linguistics and cognitive science do not
view it primarily in terms of embodiment. Thus Talmy (2000) writes
"Meaning is located in conscious experience. In the case of subjective data,
'going' to their location consists in introspection. [... ] Consciousness is
thus often a necessary concomitant at the subject end within cognitive sci-
ence" (ibid: 5-6). It is not obvious that the (phenomenal) body plays any
important role in such introspection. Similarly, in discussing the notion of
perspectivity in language, treated as a form of embodiment by MacWhin-
ney (1999), Violi (2003: 218) writes that "both the perspective a given
grammatical construction imposes on the action, and the perspective con-
nected to interpersonal and social frames, refer to subjectivity more than
embodiment". Notice that I am not saying that this latter claim is necessar-
ily true - it could turn out on closer inspection that the phenomenal body is
314 Jordan Zlatev
implicated in all kinds of social interaction and even in linguistic perspec-
tive-taking. One of the goals of the analysis presented in the next section is
precisely to suggest a greater role for phenomenal embodiment for lan-
guage and cognition. But the elucidation of the role of embodiment for
subjectivity and experience is an enormous task, begun by the classical
phenomenologists like Husserl and Merleau-Ponty (cf. Gallagher this vol-
ume), and continued more empirically by (neuro)phenomenologists such as
Varela (1996), Thompson and Varela (2001) and Gallagher (1995, 2005,
this volume), semioticians (Violi 2001; Sonesson this volume), cognitive
scientists (Donald 1991), etc. One cannot simply call consciousness "phe-
nomenological embodiment" and leave it at that.
However, the major problem with the PitF approach to embodiment is
neither of the above two levels - the "neural" and the "phenomenological"
- but the third, and as shown earlier, crucial, element in Lakoff and JoOO-
son's theory: the "cognitive unconscious". In the remainder of the section I
will argue that this notion is conceptually incoherent and rather than being
amended should be simply disposed of.
First, the notion conflates two very different kinds of entities. On the
one hand are structures such as "domain-to-domain mappings", "neural
computations" and "image schemas" which are hypothesized to operate
with an unconscious causality that one can become as aware of as, say,
synaptic growth or the operation of the immune system, that is, not at all.
On the other hand Lakoff and JoOOson mention "nouns, verbs and vowels"
(PitF: 104), i.e. categories which (nearly) all linguists analyzing all human
languages recognize, by applying standard practices of conscious linguistic
analysis. Since these analyses are not based on generalizations from speak-
ers' "behavior", despite occasional claims to the contrary, but on the basis
of linguistic intuitions (of correctness), it becomes clear that even "naIve"
speakers have consciously accessible knowledge of these categories of
their language. Thus the denizens of the Cognitive Unconscious are of two
different ontological kinds: the first, to repeat, are hypothetical causal
mechanisms, while the second are explications of linguistic knowledge that
are consciously accessible. As expressed by Itkonen (1978) in a different,
but analogous, context:
[W]e have here a confusion between the following two types of entities: on
the one hand, the concept of 'correct sentence ofa language L', which is the
object of conscious knowledge; on the other, utterances of language L,
which are manifestations of unconscious 'knowledge'. In the former case
'knowledge' equals consciousness, while in the latter, 'knowledge' is a hy-
pothetical dispositional concept. (Itkonen 1978: 82)
A second objection is methodological: what is the status of the evidence
for postulating the various structures of the Cognitive Unconscious? Lakoff
and Johnson often refer to "converging evidence", but does this evidence
really converge? On inspection it turns out to be very heterogeneous. On
the one hand is intuition and introspection, resulting in e.g. analyses of
semantic polysemy as "radial categories" (Lakoff 1987) or Talmy's (2000)
grammatical and semantic analyses which are acknowledged to be phe-
nomenological (see above). On the other hand there is psycholinguistic
experimentation involving unconscious mechanisms such as "semantic
priming" (Cuyckens, Sandra and Rice 1997; Tufvesson, Zlatev and van de
Weijer 2004) as well as neurolinguistic studies getting even closer to the
actual causality of the brain processes (e.g. Rohrer 2001; de Lafuente and
Romo 2004). Methodological pluralism is to be applauded, but the task of
combining evidence from disparate sources into a coherent framework is
formidable, and is not made easier by postulating levels that are inaccessi-
ble to both introspection and empirical observation such as the Cognitive
Unconscious. In contrast, the framework of "levels of investigation" pro-
posed by Rohrer (this volume) suggests how different kind of evidence can
be brought together in a nonreductionist manner, without any "cognitive
unconscious".
The third objection is more general (and philosophical). It involves not
just the Cognitive Unconscious postulated by Lakoff and Johnson and the
methodological self-understanding of Cognitive Linguistics, but all forms
of "information processing" psychology and cognitive science that postu-
late the existence of mental phenomena which are completely divorced
from and inaccessible to consciousness. The problem is the following:
without consciousness, there is no basis for distinguishing mental from
non-mental states within an organism. As pointed out by Searle (1992:
154): "not every state of an agent is a mental state, and not even every state
of the brain that functions essentially in the productions of mental phenom-
ena is itself a mental phenomenon". Searle's favorite examples are myeli-
nation and the OVR reflex: both are important for cognition, but in what
sense can they be said to be mental? And if they are, then anything neural
is mental. But in this case we have abolished the distinction mental vs.
neural. Now that may be something that "identity theorists" (e.g. Arm-
strong 1968) and "eliminativists" (e.g. Churchland 1992) in the philosophy
316 Jordan Zlatev
of mind would applaud. However all such proposals have so far run
aground, and the "mind-body problem" remains unsolved (Maslin 2001).9
Within information-processing, "classical" cognitive science a common
way to make the distinction between mental and non-mental without re-
course to consciousness is through the notion of computation: mental proc-
esses are involved in (symbolic) computation, non-mental ones are not (e.g.
Jackendoff 1987; Pinker 1994; Marcus 2001). Despite their overall rhetori-
cal debate with and opposition to information processing theorists, through
their endorsement of "neural computation" Lakoff and Johnson come sur-
prisingly close to the position of their opponents. Unfortunately the "com-
putational" solution to the mental/non-mental distinction does not work for
a very simple reason: there is no intrinsic computation going on in the
brain, as argued at length by e.g. Searle (2002). All talk of neural compu-
tation is metaphorical, in the sense that it is a matter of attribution from the
outside, just as in, say, computational interpretations of the weather proc-
esses or of water flow. And because of that, the "computational level" is
not ontologically or causally distinct from the neural level: "Except in
cases where an agent is actually intentionally carrying out a computation,
the computational description does not identify a separate causal level dis-
tinct from the physical structure of the organism" (Searle 2002: 126). It is
only a matter of "level of description", which is something completely
different: a matter of epistemology rather than ontology.
A possible objection to defining the mental (or the "cognitive") through
consciousness and thereby denying the coherence of the notion of the Cog-
nitive Unconscious is the existence of unconscious mental states, either of
the obvious kinds including our beliefs when we sleep or otherwise not
think about them, and the less obvious kind due to "repression" according
to Freud (1949). The claim would be that when not conscious, unconscious
mental states have some intermediate state of existence - not neural, not
conscious - and when this intermediary realm is granted, then why can't it
be populated by all sorts of mental phenomena, some of which could never
be accessible to consciousness? However, this possibility is rejected by
what Searle calls the connection principle: "all unconscious intentional
9. Furthermore, Lakoff and Johnson (1999, Chapter 7) claim to be neither identity
theorists (reductionists) nor eliminativists with respect to consciousness, so they
would need a principled means to distinguish conscious experience from its
neurallbiological underpinnings.
states are in principle accessible to consciousness" (Searle 1992: 156). In a
nutshell, the argument is the following:
All intentional states have aspectual shape: whatever they are about is
seen from a certain perspective rather than other, so that extensionally
identical entities such as "the Evening Star" and "the Morning Star" (cf.
Frege 1882) have different aspectual shapes.
Aspectual shape cannot be exhaustively characterized in third-person
predicates, either as brain states or as behaviors. This finds support in
Quine's (1960) thesis of the indeterminacy of translation.
When unconscious, mental states exist as neurophysiological phenom-
ena, rather than in a mental space that is kept outside the purview of con-
SCIousness.
On this basis one can draw the conclusion: "The notion of an uncon-
scious intentional state is the notion of a state that is a possible conscious
thought or experience" (Searle 1992: 159). There has been extensive dis-
cussion of this argument in the recent philosophical literature into which I
will not go (cf. Garrett 1995). But suffice it to say that while one can dis-
cuss any of the three premises above in some detail, Searle offers a coher-
ent way to think about unconscious mental states without postulating a
"cognitive unconscious". Since the concept is problematic both ontologi-
cally and methodologically, as suggested earlier, this places a heavy burden
on those who appeal to "unconscious mental processing" that is different
from both neuro-physiological processes and conscious thought to con-
vince us of the reality of their claims.
Lakoff and Johnson are aware of the difficulty, and spend some three
pages arguing for the "causal efficacy" of their construct. However this
defense is far from convincing. Rather it displays the unconventional ways
in which crucial theoretical concepts are used in their work. First, it is
claimed that an unconscious "basic-level concept like chair is both inten-
tional and representational" (PitF: 116). Undoubtedly, but in what way is it
unconscious? If chair is not the concept of a conscious subject, then who is
it that applies the concept to whatever it is about? Intentional states are not
self-interpreting so there must be an unconscious "homunculus" doing the
job, in whose mind there must be yet another etc. Similarly for the claim
that there are unconscious representations - if there is no ability for mis-
representation, error, we cannot speak of representation in any non-
vacuous way. But when there is error, if not earlier, the discrepancy will be
noticed, i.e. brought into consciousness. Notice that I am not stating that
representations need to represent "objective reality" and thus I am not
318 Jordan Zlatev
committing the sin of "objectivism" that is so much abhorred within Cog-
nitive Linguistics (Lakoff 1987) - what is essential however is that there
are criteria for judging the adequacy of the representation, and at least in
the case of language, these need to be public, as shown by Wittgenstein
and pointed out earlier.
So to summarize, Lakoff and Johnson's crucial notion of "the cognitive
unconscious" faces a dilemma: If it is a generalization of neuronal activity,
it is clearly causally efficacious, but then it is not separate from "neural" or
rather "biological" embodiment. On other hand, if it consists of intentional,
representational phenomena such as concepts, nouns and vowels, then each
one of these is (potentially) conscious, and therefore "phenomenological."
In both cases the Cognitive Unconscious is redundant. Furthermore, since
the role of the phenomenal body for cognition and especially for language
is still unclear, we are left with the provisional conclusion that lan-
guage/mind may not be embodied in any interesting, non-trivial way, i.e.
apart from saying that they are "realized in" or "supported by" living mat-
ter.
5. Bodily mimesis
There is, I would argue, another and more productive way of linking the
concept of embodiment to language: one that is based on the concept of
bodily mimesis, understood as the use of the body for representational
means (Donald 1991, 2001; Zlatev 2002, 2003). Unlike in reductionist
approaches such as that of Lakoff and JOhnson (1999) and the similar
sounding but very dissimilar in content "memetics" (e.g. Blackmore 1999)
mimesis has by definition two of the three crucial features of language:
representationality and accessibility to consciousness. This is already obvi-
ous in the most concise definition provided by Donald (1991: 168): "Mi-
metic skills or mimesis rests on the ability to produce conscious, self-
initiated, representational acts that are intentional but not linguistic."
In this section I will first introduce the notion as done by Donald in the
context of cognitive evolution, and elaborate it somewhat. Then I will re-
late it to a very similar concept from developmental psychology: Piaget's
(1945 [1962]) notion of a symbol which plays a crucial role in mediating
between the sensorimotor cognition of the infant, and the language-based
cognition of the verbal child and adult. On this theoretical basis, I will
introduce a relatively novel concept, the mimetic schema (Zlatev 2005),
and show how it can help resolve the apparent contradiction between em-
bodiment and language that I have argued for so far.
5.1. Mimesis in hominid evolution
In Donald's (1991) highly original theory of human origins, early hominids
- most likely belonging to the species Homo ergaster/erectus, considering
the relative jump in brain size and material culture in the hominid line
around 2 million years ago - evolved a new form of cognition based on
mimesis.
lo
This allowed our ancestors to use their bodies to perform elabo-
rated actions that others are observed to be doing (imitation), to represent
external events for the purpose of communication or thought (pantomime,
gesture) and to rehearse a given skill by matching performance to an
imagined goal. These are all capabilities which distinguished hominins
from the common ape-human ancestor, but which precede language and are
thus not dependent on it.
This hypothesis is similar to so-called "gesture theories" of language
origins (Stokoe 2001; Corballis 2002). However, it also differs from them,
since mimesis lacks at least two properties of language (or even "proto-
language") - full conventionality and systematicity, which are likely to
have appeared when vocal calls became recruited for the purpose of dis-
ambiguating gestures {Arbib 2003).11 Thus, mimesis can be seen as serving
as a "missing link" in human evolution. Furthermore it has been suggested
that mimesis can play a similar role in human ontogenetic development
(Nelson 1996; Zlatev 2001, 2003). In order to make the concept more pre-
cise and to distinguish it from other evolutionary and developmental theo-
ries which also emphasize the role of imitation such as that of Tomasello
(1999), the following {re)definition can be given, also adding the adjective
10. Donald's theory is based on evidence from paleontology, archeology, neurobi-
ology and cognitive psychology, that I will not have the space to present, but
Zlatev (2002) and Zlatev, Persson and Gardenfors (2005) offer a brief exposi-
tion of this and other empirical support for the mimetic hypothesis of human
origins.
11. The difference between mimesis and a gestural (proto) language, makes mime-
sis a more likely stepping stone to speech, since if language fIrst emerged in the
manual modality, it is difficult to explain why we do not all use sign languages
today, i.e. what would force language evolution out of the manual-brachial
track.
320 Jordan Zlatev
"bodily" in order to distinguish bodily mimesis from the broader concept
of mimesis with Aristotelean roots (cf. Zlatev, Persson and Gardenfors
2005).
(Det) Bodily mimesis: A particular act of cognition or communication is
an act of bodily mimesis if and only if:
(1) It involves a cross-modal mapping between proprioception and some
other modality (Cross-modality).
(2) It consists of a bodily motion that is, or can be, under conscious
control. (Volition)
(3) The body (part) and its motion are differentiated from and under-
stood to correspond (either iconically or indexically) to some action,
object or event. (Representation)
(4) The subject intends the act to stand for some action, object or event
for an addressee. (Communicative sign function)
But it is not an act of bodily mimesis if:
(5) The act is fully conventional (i.e. a part of mutual knowledge) and
breaks up (semi)compositionally into meaningful sub-acts that sys-
tematically relate to other similar acts. (Symbolicity)
Properties 1 to 5 are assumed to appear in this order in evolution, and logi-
cally build on one another. Thus they form an implicational hierarchy: 1 <
2 < 3 < 4 < 5. (If one has higher level properties, one must have lower-level
ones, but not vice-versa).
Bodily acts that lack either property 2 or 3 (or both), e.g. crying, are ac-
cording to the definition not mimetic. On the other hand, signed language
possesses property 5 and is excluded as well. However, not all forms of
mimesis need fulfill property 4: e.g. pantomime does, but imitation does
not. On this basis we can distinguish between two forms of bodily mimesis:
triadic mimesis which fulfills properties 1-4 in the definition, and dyadic
mimesis, where 4 is missing. Given the implicational hierarchy, it follows
that dyadic mimesis is simpler than triadic mimesis and should precede it
in evolution, and possibly also in ontogeny. Indeed, it is by now clear that
all great apes (orangutan, gorilla, chimpanzee and bonobo) have the capac-
ity for dyadic mimesis, as shown in e.g. mirror self-recognition (Gallup
1982) and imitation of arbitrary gestures (Custance, Whiten and Bard
1995), though in less developed form than human beings. What is espe-
cially difficult for apes, though, is the understanding that representations
can be used communicatively, i.e. by the sender and receiver sharing the
x-v (expression-content) mapping. Language-taught apes achieve this
with some effort (e.g. Patterson 1980), but there is no clear evidence that it
appears spontaneously, the most convincing case being certain "iconic
gestures" involving sexual and play invitations in captive apes (Tanner and
Byrne 1999).
Thus, what distinguishes my reformulation, and corresponding theory,
mostly from that of Donald (1991) is that I hypothesize that it is triadic
mimesis that crucially separated Homo erectus/ergaster from the common
ancestor, allowing a leap in cultural evolution. Triadic imitation implies the
understanding of communicative intentions, and in this way my proposal is
similar to Tomasello's (1999) suggestion that it is the understanding of
others as intentional agents that distinguishes human beings from apes.
However, it differs in emphasizing communicative intentions, and indeed,
recent evidence has granted support to this position, since apes have been
shown to understand that others have "psychological states" such as goals,
at least in competitive, non-communicative contexts (Tomasello, Call and
Hare 2003). On the other hand, understanding a gesture as corresponding
to something presumably came naturally to our predecessors, as suggested
in the following scenario:
Early humans' eyes and brains would naturally have seen that their hands
and their movements pointed directly to other things or reminded them of
other things by looking like them. [... ] Take for instance a gesture meant by
its maker and understood by its watcher to represent "the animal went up the
tree". The hand would point at the animal that both individuals had seen and
move upward as it pointed to the tree. What the brain would have done - a
million or two years ago as now - is interpret the hand's pointing flISt to
mean "that animal" and then to mean "that tree", all the time while inter-
preting the hand and arm movement as "climbing". (Stokoe 2001: 12)
While Stokoe is probably over-interpreting the degree of differentiation of
early representational gesture (in line with his "original gestural language"
hypothesis), the quote captures the essence of triadic mimesis quite accu-
rately. Thus, my evolutionary hypothesis proposes that the common ape-
human ancestor had the basic potential for dyadic mimesis. It was further
selected for as a consequence of living in larger social groups and bipedal-
ism which furthermore provided the niche for the communicative use of
bodily representations, i.e. triadic mimesis. Ontogenetic development, as
shown in the following subsection, can offer some corroborating evidence
to this scenario.
322 Jordan Zlatev
5.2. Piaget's epigenetic theory and mimetic schemas
Epigenesis, the co-determination of ontogenetic development by genes and
environment leading to a spiral of morphologies, with "lower" states serv-
ing as preconditions for "higher" ones, is nowadays nearly unanimously
accepted in biology (see Badcock 2000; Zlatev 2003). What makes epi-
genesis even more central for human development is the fact that the hu-
man infant is born in a highly immature state compared to other mammals.
Furthermore the human enyironment is so culturally rich that "culture"
impinges on "nature" to such a degree that it becomes nearly impossible to
distinguish between the two (Tomasello 1999). The developmental theory
of Piaget (1945, 1953, 1954) is epigenetic in this sense and it can be
showed that Piaget presupposed a role for bodily mimesis in ontogenetic
development that is analogous to the one envisioned for evolution above,
though this seems to have remained hidden due to terminological differ-
ences.
Piaget distinguishes between three different kinds of cognitive struc-
tures: sensorimotor schemas, symbols and signs, emerging in development
in this order. Of the three, the first is best known in the literature, in par-
ticular in relation to theories of embodiment. In previous work (Zlatev
1997) I suggested that sensorimotor schemas, which are goal-directed
structures of practical activity, can provide the "grounding" of language in
experience, thus making them analogous to the "image-schemas" proposed
by many cognitive linguists (Johnson and Rohrer this volume). As dis-
cussed in previous sections, however, this proposal is problematic since
sensorimotor schemas are non-representational, while language is repre-
sentational.
Piaget was very much aware of this difference, and while he acknowl-
edged that sensorimotor schemas play an important part in the "construc-
tion of reality for the child", he claimed that they have inherent limitations,
since "sensorimotor activity involves accommodation only to present data,
and assimilation only in the unconscious practical form of application of
earlier schemas to present data" (Piaget 1945: 278). This prevents them
from being representations, since for Piaget, as in the present account, a
representation needs to be (a) accessible to the consciousness of the sub-
ject for whom it serves as a representation and (b) differentiated from
whatever it represents, i.e. between the "signifier" and the "signified", in
Saussurean terms. Thus, a qualitatively new stage of development emerges
with the attainment of what Piaget calls the symbolicfunction:
This specific connection between "signifiers" and "signified" is typical of a
new function that goes beyond sensorimotor activity and that can be char-
acterised in a general way as the "symbolic function." It is this function that
makes possible the acquisition of language or collective "signs," but its
range is much wider, since it also embraces "symbols" as distinct from
"signs," i.e. the images that intervene in the development of imitation, play,
and even cognitive representations. (ibid: 278)
To understand this quotation, we should emphasize that Piaget is using the
term "symbol" in a sense that is very different from what it implies in the
Anglo-Saxon world: conventionality, systematicity and arbitrariness.
Rather, "symbols" are for Piaget dynamic mental images, more or less
vivid in consciousness, representing non-present actions or events. Cru-
cially, both for Piaget and for my argument, they emerge through imitation:
Hence the image is both interiorised sensorimotor imitation, and the draft of
representative imitation. [... ] It is imitation that has been interiorised as a
draft for future exterior imitation, and marks the junction-point between the
sensorimotor and the representative. (ibid: 279)
Imitation can play this bridging role since it usually emerges through the
following _ontogenetic progression: sensorimotor imitation (the imitated
action of the model is contiguous in time) > deferred imitation (the imi-
tated action is removed in time) > representative imitation - in which "the
interior image precedes the exterior gesture, which is thus a copy of an
"internal model" that guarantees the connection between the real, but ab-
sent model, and the imitative reproduction of it." (ibid: 279)
Two important aspects of Piaget' s account of the rise of representations
or "the symbolic function" should be emphasized in the present context.
The first is that they arise from an overt, public activity - imitation - which
with time becomes internalized. This is reminiscent of Vygotsky's (1978)
"law of cultural development" stating that interpersonal forms of higher
cognition precede their "intrapersonal" realizations (cf. Lindblom and
Ziemke this volume). Second, as pointed out above, this makes possible the
acquisition of language, which both consolidates and conventionalizes
these representations, leading to a new level of cognitive structure: "Verbal
representations constitute, in fact, a new type of representation, the con-
ceptual." (Piaget 1945: 280) In other words these "symbols", i.e. internal-
ized imitations serve as a "missing link" in the acquisition of language.
The analogy to the role of bodily mimesis in phylogeny should be now
obvious. On this basis, as well as a wealth of empirical data provided by
Piaget, but also by many others who have studied infant imitation and ges-
324 Jordan Zlatev
ture since then (Bates et al. 1979; Acredolo and Goodwyn 1994; Zlatev
2002), I have proposed a more fitting term for the structures that Piaget is
(rather confusingly for the modem reader) calling "symbols", namely, mi-
metic schemas (Zlatev 2005).
If we refer to the definition of bodily mimesis provided above, we no-
tice that in the case of representative imitation the first three properties:
Cross-modality, Volition and Representation are fulfilled. Thus the covert
imitation of a child following its "internal model" in executing an action is
at least a case of dyadic mimesis. In order to become triadic, in e.g. panto-
mime ("baby signs") what is necessary is to understand communicative
intentions. This can be seen as a wish to induce others to "activate" in con-
sciousness schemas similar to one's own. In other words, while Piaget
writes of "symbols" (mimetic schemas) as the "signifier" and the actual
model as the "signified", the relation can be reversed, so that a communi-
cative gesture becomes the signifier, while the (shared) mimetic schemas
are the "signified" or perhaps in Peircian terms the "interpretant" (cf. Son-
esson this volume). Let us now summarize some of the properties of mi-
metic schemas.
Mimetic schemas can be used either dyadically (in thought) or triadi-
cally (in communication).
Mimetic schemas are experiential: each schema has a different emo-
tional-proprioceptive "feel", or affective tone (Thompson 2001) to it. For
example, consider the affective contrast between the mimetic schemas
KICK and KISS. Thus, mimetic schemas can be regarded as an (important)
aspect of phenomenological embodiment.
Mimetic schemas are representational: the "running" of the schema is
differentiated from the "model event" which is represented - unlike the
most common explication given to "image schemas" (Johnson 1987; John-
son and Rorher this volume; see Hampe 2005).
Mimetic schemas are, or at least can be pre-reflectively shared: since
my and your mimetic schemas derive from imitating culturally salient ac-
tions and objects, as well as each other, both their representational and
experiential content can be "shared" - though not in the strong sense of
being known to be shared in the manner of (true) symbols or conventions.
They could also be called egocentric: "Imitation, with the help of images,
provides the essential system of ' signifiers' for the purpose of individual or
egocentric representation" (Piaget 1945: 279-280). However, it should be
remembered that for Piaget, this formulation does not imply that mimetic
schemas are private, but rather the contrary: "on the social plane the child
is most egocentric at the age in which he imitates most, egocentrism being
failure to differentiate between the ego and the group, or confusion of the
individual view-point and that ofothers" (ibid: 290, my emphasis).
Mimetic schemas can serve as the basis for the acquisition of language
in two ways: (a) they constitute the first form of (conscious) internal repre-
sentation and help lead to the "insight" that others have internal models - a
prerequisite for communicative intentions and (b) they constitute pre-
linguistic concepts, and in this respect correspond to Mandler's (2004)
characterization of "image schemas" but not to that of Johnson and Rohrer
(this volume; cf. Zlatev 2005).
These properties of mimetic schemas, and particularly the last, can al-
low us to bridge (or at least minimize) the gap between language and em-
bodiment, as discussed in the next section, which also retraces the argu-
ment presented in this chapter.
6. Embodiment regained? Mimetic schemas and language
I started by pointing out three essential properties of (the knowledge of)
language: conventionality, representationality and conscious accessibility-
and proceeded to see if, and if so how, they can be made compatible with
the currently popular conception that the (human) mind is an "embodied
mind". In one of the most influential accounts of "embodiment theory",
especially within Cognitive Linguistics, that of Lakoff and Johnson (1999),
we saw that these three properties were essentially absent. In what fol-
lowed I subjected this version of "embodiment" to criticism, and in par-
ticular its central concept of the Cognitive Unconscious. While this criti-
cism does not automatically generalize to other accounts, it gives us
reasons to worry if embodiment and language can be made compatible, not
the least because of the lack a coherent concept of representation. The
quest for a more adequate notion of embodiment led us to the work of
Donald (1991), and the concept of (bodily) mimesis, which was explicated
and related to Piaget's developmental theory. In particular, I argued for the
need to acknowledge the concept of mimetic schemas, which among other
things:
are structures of the "lived" (phenomenal, experiential) body,
meaning that they are accessible to consciousness;
326 Jordan Zlatev
are representational structures: they are differentiated from what
they stand for, and can be enacted overtly (as pantomime and ges-
ture) or covertly (as mental images);
can be pre-reflectively shared with others since they (usually) arise
from imitation.
But notice that these three characteristics of mimetic schemas correspond
to - without being identical - to the three properties of language under
focus. Thus, the following hypothesis concerning the "embodiment" of
language can be formulated: Public linguistic symbols are Hembodied" in
the sense that part of their meaning is constituted by underlying mimetic
schemas.
If this hypothesis holds true, bodily mimesis can serve not only as a
"missing link" between sensorimotor and linguistic cognition in evolution
as envisioned by Donald (1991) and in ontogenesis as argued by Piaget -
and in rather different ways proposed by Nelson (1996) and Zlatev (2001,
2002) - but as a conceptual, meta-theoretical link between embodiment
and language. Since language is a central aspect of human sociocultural
situatedness, mimetic schemas can help integrate the two major factors that
define the human mind - embodiment and situatedness - in a coherent
framework.
What else can we offer in support of this hypothesis? A proper treat-
ment of this question would require a separate chapter, so here I only men-
tion the following considerations, to be explored in more detail in the fu-
ture (cf. also Zlatev 2005):
First, the existence of pre-linguistic but representational mimetic sche-
mas can help solve the puzzle how "socially shared symbolic systems"
(Nelson and Shaw 2002) emerge in pre-linguistic children. Since young
children lack the meta-linguistic capacity for establishing full-fledged con-
ventions, it is still a mystery how they come from the sensorimotor to the
symbolic (i.e. conventional and systematic) level. Mimetic schemas, with
their implicit sharing, suggest a way out of this impasse.
Second, a particular difficulty in explaining language acquisition is to
account for the learning of actions terms ("verbs"). After having tradition-
ally been considered to follow object terms ("nouns") in child language
(Macnamara 1982), action words have during the past years been shown to
arise simultaneously (Tomasello 1992; Nelson 1996), and if they are
prominent in parental speech, even to precede nouns in some cases
(Gopnik, Choi and Baumberger 1996). It is obvious how mimetic schemas
for concrete, imitable actions (e.g. RUN, EAT, SEAT... ) can serve as a basis
for the acquisition of the corresponding "verbs". Furthermore, the devel-
opment of shared representations for objects that can be manipulated such
as cups, balls, toys, books, food etc. will be also facilitated, and thus un-
derlie the acquisition of the corresponding "nouns".12 Notice that if mi-
metic schemas ground the acquisition of the first words in childhood, the
prediction is that the child's early vocabulary will consist of terms such as
run, sit, eat, cat and toy... and this is indeed the case (Nelson 1996; Bloom
2000).
Third, and conversely to helping explain the ease with which children
acquire language, and in line with Donald's (1991) original proposal, mi-
metic schemas may help explain why language acquisition is so difficult
even for "enculturated" apes: evolution has given us an adaptation for tri-
adic mimesis supporting advanced imitation and gesture that is beyond the
capacities of our nearest relatives in the animal kingdom.
Forth, mimetic schemas as a ground for public symbols can help explain
how both "cognitive" (representational) and "affective" (experiential)
meaning can be communicated through language, since both aspects can be
- to various degrees - shared by communicators, even if the two can be
decoupled in abnormal conditions.
Fifth, the close connection of linguistic symbols and mimetic schemas
is consistent with the accumulating evidence from experimental psychol-
ogy and neuroscience showing that language use engages motor represen-
tations, as well as the corresponding brain regions (Glenberg and Kaschak
2003; Svensson, Lindblom and Ziemke this volume). At the same time,
neither this evidence, nor the present proposal implies a stronger form of
"language embodiment" in which (practically) all symbolic and inferential
processing is carried out by sensorimotor categories and brain regions (La-
koff and Johnson 1999; Johnson and Rohrer this volume). If that were the
case it would be very hard to explain the qualitative difference between
animal and human cognition, in particular with respect to language skills.
To emphasize again, according to the present hypothesis, mimetic schemas
ground, but do not constitute linguistic meaning - which as pointed out in
12. In the case of objects there is also another means to achieve shared reference,
e.g. joint attention (Tomasello 1999), and this would serve to pick out shared
perceptual attributes. But there are problems in explaining how this is done,
conceptual (Quine 1960) as well as empirical (Bloom 2000); mimetic schemas
for acting on the objects can help pick out the relevant properties.
328 Jordan Zlatev
Section 2 is conventional in the strong sense: not just shared, but mutually
known to be shared.
Sixth, the hypothesis is consistent with the recent enthusiasm sur-
rounding "mirror neurons", which are assumed to support action recogni-
tion and imitation, and their role in the evolution of language (Rizzolati
and Arbib 1998; Arbib 2003). Since there appears to be a homology be-
tween area F5 of the monkey brain where mirror neurons for grasping were
originally discovered and Broca's area, it is reasonable to suppose that a
developed mirror neuron system constitutes a (partial) "neural correlate" of
the ability to form and entertain mimetic schemas.
Seventh, and finally, a long lasting debate in the study of spontaneous
co-speech gestures (e.g. McNeil1 1992) is whether they are primarily
"communicative" or "cognitive", i.e. whether they are performed for the
benefit of the speaker, or for the speaker himself (given that even blind
people gesture to each other, as well as more mundanely, people talking on
the telephone). Considering gestures to be realizations of mimetic schemas
allows them to be both. The work of Kita and Ozyiirek (2003), showing the
existence of non-linguistic "spatio-motoric representations" that are to
some extent influenced by the language of the speaker, fits in naturally
with the present proposal.
7. Conclusion
In this chapter I have argued for the following set of interrelated theses:
Language is fundamentally a socio-cultural phenomenon, based of
grammatical and semantic conventions, and therefore it cannot be reduced
to individual minds, and even less so to brains. However, apart from con-
ventionality, language also presupposes representationality and conscious
accessibility and these imply subjectivity.
Qualitative experience is a subjective, "first-person" phenomenon as
well as an interpersonal one, involving emotion and affective tone. Thus a
truly experiential theory of language needs to account for the ability to
communicate through linguistic signs which are shared both representa-
tionally and phenomenologically.
Theories of embodiment such as that of Lakoff and Johnson (1999)
which ignore these characteristics cannot satisfactorily account for lan-
guage. Since language plays an important role in shaping the human mind,
such theories are not capable of accounting for human cognition as well.
The concepts of bodily mimesis, and its derivative concept: mimetic
schemas, can help resolve the contradiction between embodiment and lan-
guage, and thus assist us in the long-term project of (re)integrating body,
language and mind.
Acknowledgments
In writing this chapter, I have benefited from interactions with other mem-
bers of the project Language, Gesture and Pictures in Semiotic Develop-
ment at Lund University and the EU-project Stages in the Evolution and
Development of Sign Use (SEDSU): Goran Sonesson, Peter Gardenfors,
Tomas Persson, Ingar Brinck, and Sara Lenninger. I would also wish to
thank Jorg Zinken, Gorel Sandstrom, Roslyn Frank, Alex Kravchenko;
Lars-Ake Henningsson and two anonymous reviewers for comments on
various earlier drafts. Finally, I wish to dedicate this essay to my friend Esa
Itkonen, for his brave fight for the true nature of language against varieties
ofbio-physical reductionism over the past 30 years, i.e. half his life.
References
Acredolo, Linda and Susan Goodwyn
1994 Sign language among hearing infants: The spontaneous development
of symbolic gestures. In: Virginia Volterra and Carol J. Erting (eds.),
From Gesture to Language in Hearing and Deaf Children, 68-78.
Washington, DC: Gallaudet University Press.
Arbib, Michael
2003 The evolving mirror system: A neural basis for language readiness.
In: Morten Christiansen and Simon Kirby (eds.) Language Evolution,
182-200. Oxford: Oxford University Press.
Armstrong, David M.
1968 A Materialist Theory of the Mind. London: Routledge and Kegan
Paul.
Badcock, Christopher
2000 Evolutionary Psychology. A Critical Introduction. Cambridge: Polity
Press.
Baker, Gordon P. and P.M.S Hacker
1984 Language, Sense and Nonsense: A Critical Investigation into Mod-
ern Theories ofLanguage. Oxford: Basil Blackwell.
330 Jordan Zlatev
1979
How Children Learn the Meaning of Words. Cambridge, Mass.: MIT
Press
Brooks, Rodney
1999 Cambrian Intelligence: The Early History ofthe New AI. Cambridge,
Mass.: MIT Press.
Churchland, Paul M.
1992 A Neurocomputational Perspective: The Nature of Mind and Struc-
ture ofScience. Cambridge, Mass.: MIT Press.
Bates, Elizabeth, Laura Benigni, Inge Bretherton, Luigia Camaioni and Virginia
Volterra
The Emergence of Symbols. Cognition and Communication in In-
fancy. New York: Academic Press.
Blackmore, Susan
1999 The Meme Machine. Oxford: Oxford University Press.
Bloom, Paul
2000
2000
Clark, Andy
1997 Being There: Putting Brain, Body, and World Together Again. Cam-
bridge, Mass.: MIT Press.
Clark, Herbert
1996 Using Language. Cambridge: Cambridge University Press.
Clark, Herbert and Catherine R. Marshall
1981 Defmite reference and mutual knowledge. In: Arivind K. Joshi, Bon-
nie L. Webber and Ivan A. Sag (eds.), Elements ofDiscourse Under-
standing, 10-63. Cambridge: Cambridge University Press.
Corballis, Michael C.
2002 From Hand to Mouth: The Origins of Language. Princeton, NJ.;
Princeton University Press.
Custance, Deborah M., Andrew Whiten and Kim A. Bard
1995 Can young chimpanzees (Pan troglodytes) imitate arbitrary actions?
Hayes and Hayes (1952) revisited. Behavior 132: 839-858.
Cuyckens, Hubert, Dominiek Sandra and Sally Rice
1997 Toward and empirical lexical semantics. In: Birgit Smieja and Meike
Tasch (eds.), Human Contact Through Language and Linguistics,
35-54. Frankfurt: Peter Lang.
Damasio, Antonio
1994 Descartes' Error. Emotion, Reason and the Human Brain. New
York: Grosset/Putnam.
The Feeling of What Happens. Body, Emotion and the Making of
Consciousness. New York: Harvester.
de Lafuente, Victor and Ranulfo Romo
2004 Language abilities of motor cortex. Neuron 41: 178-180.
2001
Deacon, Terry
1997 The Symbolic Species: The Co-Evolution ofLanguage and the Brain.
New York: Norton.
Donald, Merlin
1991 Origins of the Modern Mind. Three Stages in the Evolution of Cul-
ture and Cognition. Cambridge, Mass.: Harvard University Press.
A Mind So Rare. The Evolution of Human Consciousness. New
York: Norton.
Dreyfus, Hubert
1972 What Computers (Still) Can't Do. A Critique of Artificial Reason.
Third revised edition. Cambridge, Mass.: MIT Press. Reprint 1993.
Edelman, Gerald
1992 Bright Air, Brilliant Fire: On the Matter ofthe Mind. London: Basic
Books.
Elman, Jerry, F.
1990 Finding structure in time. Cognitive Science, 14: 179-211.
Evans, Vyv
2003 The Structure ofTime. Language, Meaning and Temporal Cognition.
Amsterdam: Benjamins.
Fodor, Jerry A.
1981 Representations. Cambridge, Mass.: MIT Press.
Frege, Gottlob
1882 [1997] On Sinn and Bedeutung. In: Michael Beaney (ed.), The Frege
Reader, 151-171. Oxford: Blackwell.
Freud, Sigmund
1949 An Outline ofPsycho-analysis. London: Hogarth.
Gallagher, Shaun
1995 Body schema and intentionality. In: Jose Bermudez, Naomi Eilan,
and Anthony Marcel (eds.), The Body and the Self, 225-244. Cam-
bridge: MIT/Bradford Press.
Gallup, Gordon G.
1982 Self-awareness and the emergence of mind in primates. American
Journal ofPrimatology 2: 237-248.
Garrett, Brain, J.
1995 Non-reductionism and John Searle's The Rediscovery of Mind. Phi-
losophy and Phenomenological Research 55 (1): 209-215.
Giv6n, Tom
2001 Syntax, VoI1-2. Amsterdam: Benjamins.
Itkonen, Esa
1978
1983
332 Jordan Zlatev
Glenberg, Arthur M. and Michael P. Kaschak
2003 The body's contribution to language. In: Brian H. Ross (ed.), The
Psychology ofLearning and Motivation 43: 93-126. San Diego, CA:
Academic Press.
Gopnik, Alison, Soonja Choi and Therese Baumberger
1996 Cross-linguistic differences in early semantic and cognitive develop-
ment. Cognitive Development 11 (2): 197-227.
Gurwitsch, Aron
1964 The Field ofConsciousness. Pittsburgh: Duquesne University Press.
Hampe, Beate (ed.)
2005 From Perception to Meaning: Image Schemas in Cognitive Linguis-
tics. Berlin: Mouton de Gruyter.
Grammatical Theory and Metascience. Amsterdam: Benjamins.
Causality in Linguistic Theory. A Critical Investigation into the
Philosophical and Methodological Foundations of "Non-
autonomous" Linguistics. Bloomington: Indiana University Press.
1991 Universal History ofLinguistics. Amsterdam: Benjamins.
2003 What is Language? Turku: University of Turku Press.
Jackendoff, Ray
1987 Consciousness and the Computational Mind. Cambridge, MA: MIT
Press.
Johnson, Mark
1987 The Body in the Mind. Chicago: University of Chicago Press.
Johnson, Mark and George Lakoff
2002 Why cognitive linguistics requires embodied realism. Cognitive
Linguistics 13 (3): 245-263.
Johnson, Mark and Tom Rohrer
this vol. We are live creatures: Embodiment, Pragmatism and the cognitive
organism.
Kita, Sotaro and Asli 6zyiirek
2003 What does cross-linguistic variation in semantic coordination of
speech and gesture reveal? Evidence for an interface representation
of spatial thinking and speaking. Journal of Memory and Language
48: 16-32.
Lakoff, George
1987 Women, Fire and Dangerous Things: What Categories Reveal
About the Mind. Chicago: University of Chicago Press.
1999 Philosophy in the Flesh: The Embodied Mind and its Challenge to
Langacker, Ronald
1987 Foundations of Cognitive Grammar, Vol 1. Stanford, CA: Stanford
University Press.
Lewis, David K.
1969 Convention: A Philosophical Study. Cambridge MA: Harvard Uni-
versity Press.
this vol. Embodiment and social interaction: A cognitive science perspective.

1982 Names for Things. Cambridge, MA: Press.
Brian
1999 The emergence of language from embodiment. In: Brain Whin-
ney (ed.), The Emergence of Language, 213-256. NJ:
Lawrence Erlbaum.
Jean
2004 The Foundations of Mind: Origins of Conceptual Thought. Oxford:
Marcus, Gary F.
2001 The Algebraic Mind: Integrating Connectionism and Cognitive Sci-
ence. Cambridge, Mass.: MIT Press.
Keith
2001 An Introduction to the Philosophy of Mind. MaIden, Mass.: Polity
Press.
McNeill, David
1992 Hand and Mind: What Gestures Reveal about Thought. Chicago:
University of Chicago Press.
Maurice
1945 Phenomenology of Perception. London: Routledge and Kegan Paul.
Reprint 1962.
Nelson, Katherine and Lea Kessler Shaw
2002 Developing a socially shared symbolic system. In: James Byrnes and
Eric Amseli (eds.) Language, Literacy and Cognitive Development,
27-57. Mahwah, NJ: Lawrence: Erlbaum.
Nelson, Katherine
1996 Language in Cognitive Development. The Emergence of the Medi-
ated Mind. Cambridge: Cambridge University Press.
Ogden, C.K. and LA. Richards
1923 The Meaning ofMeaning. London: Routledge and Kegan Paul.
Palmer, Gary
1996 Toward a Theory of Cultural Linguistics. Austin: The University of
Texas Press.
334 Jordan Zlatev
Patterson, Francis
1980 Innovative use of language in a gorilla: A case study. In Katherine
Nelson (ed.) Children's Language, Vol 2,497-561. New York: Gar-
den Press.
1953
Piaget, Jean
1945
philosophical foundations of cognitive linguistics. In Rene Dirven,
ogy: Cognitive Theoretic Approaches: Volume 1, 49-81. Amster-
dam: Benjamins.
Russell, Bemard
1905 On denoting. Mind 14: 479-93.
Saussure, Ferdinand de
1916 Cours de Linguistique Generale [Course in General Linguistics].
Paris: Payot.
Schmidt,Richard
1990 The role of consciousness in second language learning. Applied Lin-
guistics 11: 17-46.
La formation du symbole chez l'enfant, Neuchatel-Paris: Delachaux
et Niestle; English translation: G. Gattegno and F. M. Hodgson. Play,
Dreams, and Imitation in Childhood. New York: Norton, 1962.
The Origin of Intelligence in the Child. London: Routledge and
KeganPaul.
1954 The Construction ofReality in the Child. New York: Basic Books.
Pinker, Steven
1994 The Language Instinct. New York: William Morrow.
Quine, Willard V. O.
1960 Word and Object. Cambridge, Mass.: MIT Press.
Regier, Terry
1996 The Human Semantic Potential: Spatial Language and Constrained
Connectionism. Cambridge, Mass.: MIT Press.
Rizzolatti, Giacomo and Michael Arbib
Rohrer, Tim
2001
Searle, John
1969
1992
1999
2002
Speech Acts. Cambridge: Cambridge University Press.
The Rediscovery ofthe Mind. Cambridge, Mass.: MIT Press.
Mind, Language and Society. Philosophy in the Real World. London:
Weidenfeld and Nicolson.
Consciousness and Language. Cambridge: Cambridge University
Press.
2005
1999
Sinha, Chris
1988
Senghas, Ann, Sotaro Kita and Asli OZyiirek
2004 Children creating core properties of language: Evidence from an
emerging sign language in Nicaragua. Science 305: 1779-1782.
Language and Representation. A Socio-naturalistic Approach to
Human Development. New York: Harverster Press.
Grounding, mapping and acts of meaning. In: Theo Janssen and
Gisela Redeker (eds.), Cognitive Linguistics: Foundations, Scope
and Methodology, 223-255. Berlin: Mouton de Gruyter.
Blending out of the Background: Play, props and staging in the mate-
rial world. Journal of Pragmatics (Special issue on Conceptual
Blending Theory, guest eds. Seana Coulson and Todd Oakley), 37:
1537-1554.
Sonesson, Goran
1989 Pictorial Concepts. Lund: Lund University Press.
this vol. From the meaning of embodiment to the embodiment of meaning: A
Stokoe, William C.
2001 Language in Hand. Why Sign Came before Speech. Washington
D.C.: Gallaudet University Press.
Strawson, Peter F.
1950 On referring, Mind 59: 320-344.
Svensson, Patrik
1999 Number and Countability in English Nouns: An Embodied Model.
Uppsala: Swedish Science Press.
this vol. Making sense of embodied cognition: Simulation theories of shared
TalmY,Len
2000 Toward a Cognitive Semantics, Vol I and Vol If. Cambridge, Mass.:
MIT Press.
Tanner, Joanne E. and Richard W. Byrne
1999 The development of spontaneous gestural communication in a group
of zoo-living lowland gorillas. In: Sue T. Parker, Robert W. Mitchell
and H. Lyn Miles (eds.), The Mentalities ofGorillas and Orangutans
- Comparative Perspectives, 211-239. Cambridge: Cambridge Uni-
versity Press.
Thompson, Evan
2001 Empathy and consciousness, Journal of Consciousness Studies 8
(5/7): 1-32.
336 Jordan Zlatev
Thompson, Evan and Francisco Varela
Tomasello, Michael
1992 First Verbs: A Case Study of Early Grammatical Development.
Cambridge: Cambridge University Press.
1999 The Cultural Origins of Human Cognition. Cambridge, Mass.: Har-
yard University Press.
Tomasello, Michael, Joseph Call and Brian Hare
2003 Chimpanzees understand psychological states - the question is which
ones and what extent. Trends in Cognitive Sciences 7 (4): 153-156.
Trubetzkoy, Nikolay S.
1939 Grudzuge der Phonologie. Gottingen: Vandenhoeck and Ruprecht.
Reprint 1958.
Tufvesson, Sylvia, Jordan Zlatev and Joost van de Weijer
2004 Idiomatic entrenchment and semantic priming, In: Augusto Soares da
Silva, Amadeu Torres, Miguel Gonyalves (eds.), Linguagem, Cultura
e Cogni9iio: Estudos de Linguistica Cognitiva, Vol 1: 309-334.
Coimbra: Almedina.
Varela, Francisco
1996 Neurophenomenology: A methodological remedy for the hard prob-
lem. Journal ofConsciousness Studies 3 (4): 330-350.
Varela, Francisco, Evan Thompson and Eleonor Rosch
Violi, Patrizia
2001 Meaning and Experience. Translated by Jeremy Carden, Blooming-
ton: Indiana University Press.
2003 Embodiment at the crossroads between cognition and semiosis. Re-
serches en communication 19: 199-217.
Vygotsky, Lev S.
1934 Thought and Language. Cambridge, Mass.: MIT Press. Reprint
1962.
1978 Mind in Society. The Development of Higher Psychological Proc-
esses, Cambridge, Mass.: Harvard University Press.
Wittgenstein, Ludwig
1923 The Tracatus Logico Philosophicus. London: Routledge. Reprint
1961.
1953 Philosophical Investigations. Oxford: Basil Blackwell.
Ziemke, Tom
David Kirsh (eds), Proceedings of the 25
th
baum.
Zlatev, Jordan
1997 Situated Embodiment. Studies in the Emergence ofSpatial Meaning.
2001 The epigenesis of meaning in human beings, and possibly in robots.
Minds and Machines 11 (2): 155-195.
2002 Mimesis: The "missing link" between signals and symbols in phylog-
eny and ontogeny. In: Anneli Pajunen (ed.), Mimesis, Sign and the
Evolution of Language, 93-122. Publications in General Linguistics
3. Turku: University ofTurku Press.
Zlatev, Jordan, Tomas Persson and Peter Gardenfors
2005 Bodily mimesis as "the missing link" in human cognitive evolution.
LUCS 121. Lund: Lund University Cognitive Studies.
The body in space: Dimensions of embodiment
Tim Rohrer
Abstract
Recent research from a large number of fields has recently come together under the
rubric of embodied cognitive science. Embodied cognitive science attempts to
show specific ways in which the body shapes and constrains thought. I enumerate
the standard variety of usages that the term "embodiment" currently receives in
cognitive science and contrast notions of embodiment and experientialism at a
variety of levels of investigation. The purpose is to develop a broad-based theoretic
framework for embodiment which can serve as a bridge between different fields. I
introduce the theoretic framework using examples that trace related research issues
such as mental imagery, mental rotation, spatial language and conceptual metaphor
across several levels of investigation. As a survey piece, this chapter covers numer-
ous different conceptualizations of the body ranging from the physiological and
developmental to the mental and philosophical; theoretically, it focuses on ques-
tions of whether and how all these different conceptualizations can form a cohesive
research program.
Keywords: cognitive neuroscience, Cognitive Linguistics, Embodiment, frames of
reference, mental rotation, space.
1. Introduction: Embodiment and experientialism
1.1. Embodiment: The return of the absent body to cognitive science
HUMAN BEINGS HAVE BODIES. Academics of every variety, so often caught
up in the life of the mind, find that simple truth altogether too easy to for-
get. Imagine working late into the night, hotly pursuing another bit of per-
fect prose. But now let there be a power outage and, in the absence of
electric light or the pale glow of the computer screen, imagine how we
grope and fumble to find our briefcase, locate the door, and exit the build-
ing. In such circumstances, the body returns. Whenever we are unexpect-
340 Tim Rohrer
edly forced to move about in the dark, we are forcibly reacquainted with
our bodily sense of space. Problems ordinarily solved beneath the level of
our conscious awareness become dominant in our cognition; we find our-
selves noticing subtle changes in the floor texture underfoot, carefully
reaching out for the next step in the stairwell. It is a most peculiar experi-
ence, one that may well remind us of being young and just learning to walk
down stairs.
Unfortunately for cognitive science, many academics of that particular
variety haven't simply forgotten that human beings have bodies cognitive
scientists have deliberately theorized the body away. For most of its first
fifty years, cognitive science was in the throes of a peculiarly devilish axis
between information theory in computer science and functionalism in the
philosophy of mind and psychology. Within computer science and infor-
mation theory, the problem of building a thinking machine was identified
with just one narrowly specified field of human cognition - computing
mathematical functions (Turing 1950; Hodges 1983). Under the function-
alist paradigm, the mind was treated as if it were a series of modular com-
puter programs - or "black boxes" - whose inputs and outputs could be
specified in symbolic terms. While no one would have argued that the
physical architecture of vacuum tubes and transistors making up the early
computers were identical to the physical architecture of the neural systems
making up the brain, the functionalists did argue that the specific physical
details of how such thinking systems computed were irrelevant to what
they computed. In fact, they argued that as computation could take place
not only in electrical and neural systems but also in mechanical systems
such as a loom or Babbage's steam-powered analytical engine, cognition
was independent of its physical medium. From this perspective, the only
thing that mattered to simulating cognition was getting the inputs of these
black boxes to compute the correct outputs (Cummins 1977). The physical
body - whose architecture was seen as largely irrelevant to cognition - was
redefined as a series of black boxes that computed mathematical functions.
The disembodied computer was the analogical origin of the disembodied
mind.
In recent years however, another strain of cognitive scientists have be-
gun to take their inspiration from the contrarian vision - embodied cogni-
tive science. Unlike the computationalist-functionalist hypothesis, em-
bodiment theorists working in various disciplines argue that the specific
details of how the brain and body embody the mind do matter to cognition.
This broad theoretical approach has been the result of many parallel devel-
The body in space: Dimensions ofembodiment 341
opments in diverse fields ranging from neurobiology and linguistics to
robotics and philosophy. While there are undoubtedly many touchstones
and origins of this approach, Johnson and I (Johnson and Rohrer this vol-
ume) have given a detailed account of some of the neurobiological and
philosophical roots of the embodiment hypothesis in cognitive science with
a particular emphasis on how the contributions of American Pragmatism
anticipated modem cognitive neuroscience. By contrast, in this paper I
intend only to survey the wide variety of manners in which embodied cog-
nitive science is done, including these among others, in order to develop a
general theoretic framework as a backdrop against which these research
projects can be situated.
One of the most central examples of how embodied cognitive science
has revolutionized the field lies in the details of how the mind, brain and
body interact to construct our experience of space. Tracing this example
across the different disciplines of cognitive science will require the whole
of this article, but as a beginning recall the basic finding of the work on
mental rotation (Shepard and Metzler 1971). In their renowned experiment
wherein participants were asked to determine whether one two-dimensional
drawing of a three-dimensional object was identical to or a mirror image of
another, they found that subjects mentally rotated the object at a linear rate
- about 60 degrees per second. In other words, participants were manipu-
lating such images as wholes, preserving their topologies while rotating
them through a series of intermediate depictions. At the time of its publi-
cation, their finding was surprising because the then prevailing computa-
tionalist and functionalist view held that the mind operated in a symbolic
rather than depictive fashion, and therefore argued that any such mental
imagery would be merely epiphenomenal (Pylyshyn 1973). Over the ensu-
ing thirty years, a variety of convergent evidence has established not only
the fact that mental images are rotated in the brain as perceptual wholes
(Kosslyn et al. 1995), but have also specified how that fact impacts our
understanding of exactly what our minds are "computing" (reviewed in
Kosslyn 1994; Kosslyn, Ganis and Thompson 2002).
Consider, for example, how the body - and not just the brain - plays a
role in modifying the rate at which mental rotations take place. Extend
your left arm in front of you and hold your left hand straight out, palm
upwards. Now try to rotate your hand 180 degrees to the left and then to
the right. Notice that the rightward (inward) rotation is relatively easy,
while the leftward is quite difficult, requiring additional shoulder and arm
joint movements. A series of experiments by Parsons (1987ab, 1994; Par-
342 Tim Rohrer
sons et al. 1995) showed that when subjects were asked to perform mental
rotations of images which consisted of line drawings of human hands in-
stead of Shepard-Metzler 2D/3D block diagrams, subjects were quicker
and better at identifying those rotations of the hand that were easier to per-
form, given the kinds of bodily constraints on joint movements we have as
humans. Furthermore, Parsons found that subjects were quicker and better
at judging which hand - left or right - was pictured when imagining rotat-
ing the hand that did not require difficult bodily movements. Given the
details of the way the body works, the motor imagery system actively con-
strains how fast mental imagery is performed.
1
Even more dramatically, consider how patients with chronic arm pain in
one limb perform similar mental hand rotation tasks. For their affected arm
as compared to their uninjured arm patients are much slower to perform the
necessary mental rotations in those conditions where the bodily movements
that would be required for the actual hand rotation involve large arm
movements (Schwoebel et al. 2001). A group of non-patient controls also
showed no such differences between their left and right arms. Not only
does the body affect how our mind works, but the body in pain affects how
the mind works. Of course, this last insight should come as no surprise to
anyone except those cognitive scientists who believe that our minds work
just like disembodied computers.
As a fallback position, a committed computationalist could simply jetti-
son the functionalist claim that our cognition is independent of our neuro-
physiological architecture. One could argue that embodiment means only
that "computations" of a particular kind - analog and iconic, not symbolic;
physiologically embodied and perspectivally situated, not abstractly uni-
versal - are being performed as the body and brain pass topology-
preserving structures forward and backward between the visual and motor
systems. Admittedly, these topologies are not the "outputs" of the compu-
tations of a rigidly modular functionalist architecture, but rather dynamic
activation patterns which imagistically map the perceptual contours of
experience, rippling back and forth through multiple reentrant neuro-
anatomical connections within a web of functionally interrelated neural
regions. These embodied neural "computations" compete to become the
1. Interestingly, if one adds a cylindrical "head" to Shepard-Metzler cube stimuli,
one can produce similar embodied facilitation and inhibition effects to those
produced by more naturalistically "embodied" stimuli such as Parsons' line
drawings of hands (see Amorim, Isableu and Jarraya 2006).
most salient and pragmatically useful mental constructs to address the cur-
rent problem for the organism, whatever that is. While such revisions to
our conception of what counts as cognitive "computations" are certainly
warranted by the evidence and are important steps toward an embodied
cognitive science, we might also inquire whether the focus on embodiment
leads to additional constraints that are not purely physiological.
Suppose that the current problem for the organism is once again the
mental rotation of images. Given the results concerning how the rate of the
mental rotation varies when the stimulus is a hand, are there multiple
strategies for rotating mental objects that could compete to solve such
problems? As one obvious difference between the Parsons stimuli and the
Shepard and Metzler stimuli is that the former are line drawings of body
parts while the latter are line drawings of 3D blocks, it might be possible
that the motor imagery effects Parsons observed are limited to body-part
images. While Kosslyn et al. (1998) had initially argued that there was this
sort of stimulus-determined choice between two separate neural systems
that could perform mental rotations, namely the motor imagery (hand
stimulus) and visual imagery (object stimulus) systems, Kosslyn et al.
(2001) now argue that there are two possible perspectives - or spatial
frames of reference - that influence which strategy for mental rotation is
chosen. In one such frame of reference - a viewer-centered perspective - it
is possible to imagine oneself physically grasping and rotating a 3D object;
while in the other frame of reference - an object-centered frame - it is
possible to imagine viewing something else rotating the 3D object. Kosslyn
and colleagues built wooden 3D constructs of a Shepard-Metzler block
figures, and just prior to the neuroimaging had the subjects either turn by
hand the wooden blocks or observe the blocks rotating on a motor-driven
spindle. Participants were then instructed to imagine rotating the visual
stimuli presented during the neuroimaging task in precisely the same man-
ner. The neuroimaging results showed that the differences in the strength
of activation in the motor imagery (or visual imagery) brain regions corre-
lated with which perspective was obtained on the model via the partici-
pant's socially instructed interaction with it. Their results show that par-
ticipants could voluntarily choose to adopt a particular strategy based on
the frame of reference in which they were told to interact with the object
and not based solely on the type of stimulus image, i.e., body parts or
blocks. In other words, it is not the case that only the details of our physi-
ology matter, such as the constraints of our joints as we imagine rotating
344 Tim Rohrer
our hands. Instead, the socially instructed choice of perspective also mat-
ters to how the embodied mind works.
The Kosslyn group's experiments demonstrate why embodiment in
cognitive science should never be construed as an exclusively physiologi-
cal phenomenon. Even when researchers are measuring physiological
changes such as changes in the blood flow or glucose uptake within the
brain, both socially and environmentally induced factors can play a theo-
retically significant role as to what brain activity is being measured. In-
structing a participant in a neuroimaging experiment to imagine using one
spatial frame of reference or the other - that is, to imagine manipulating
the blocks themselves as opposed to imagining the blocks spinning on their
own - demonstrates how the social context influences the physiological
response. Similarly, constructing a 3D physical version of a heretofore
visually presented 2D stimulus predisposes the participant to interact with
the stimulus using a slightly different mix of sensory modalities - resulting
in a different physiological response. Note that it is not the case that the
"body" enters into the measured response only in the condition where the
participant physically manipulates the 3D object. In each case the body
interacts with the stimuli in different ways (visually or motorically), and
the resulting environmental predispositions to imagine using either the
visual or the motor system are carried into the PET scanner. Unfortunately,
the Kosslyn group has not yet investigated whether the social and environ-
mental influences are separable, but it is reasonable to predict that they are.
One could test this hypothesis using an experimental variation derived
from semantic priming; if some participants were instructed to imagine
operating in the opposite frame of reference during the scan than the one
induced by their pre-scan bodily interaction with the stimulus, one would
expect that their responses would be weaker in activation (inhibited) when
compared to those participants for whom the social instructions coincided
with the embodied environmental interaction.
1.2. Experientialism: "The body" of cognitive science expands
The Kosslyn et al. (2001) experiment is particularly revealing because it
shows that even for those of us who use methodologies dedicated to meas-
uring the body, embodiment means not just the physiological body - or
worse yet, just the physiological brain - but the body-in-space, the body as
it interacts with the physical and social environment. Many of the objects
we interact with every day are in fact cognitive artifacts we have designed
with our bodies in mind. Consider one last set of experiments on mental
rotation, one which compares the mental rotation of hands with the mental
rotation of tools. Vingerhoets et al. (2002) compared the fMRI activation
patterns of right-handed male subjects who were asked to decide whether a
pair of pictures were different or identical (except for being rotated) when
presented with either two pictures of hands (either right or left) or two
pictures of hand tools (a monkey wrench, a pencil sharpener, a can opener
and a soup ladle). While they found pre-motor and motor activation in both
experimental conditions, their key finding was that tools, unlike hands,
activated only the left hemisphere premotor and motor hand cortices -
contralateral to the subject's dominant hand. In other words, when we think
about rotating tools - as opposed to Kosslyn's abstract shapes or Parsons'
pictures of hands - we are mentally "grasping" those tools and rotating
them with the same hand that we would ordinarily use to rotate them in the
physical world. Thus, the body-in-the-brain is not just shaped by the body,
but by the habitual interactions of the body with the environment.
The point is not just that the body shapes the embodied mind, but that
the experiences of the body-in-the-world also shape the embodied mind.
But the experiential worlds with which we interact are more than simply
physical; we are born into social and cultural milieus which transcend our
individual bodies in time. Tools are an excellent example of the elements
of our physical world that come to us already shaped by socio-cultural
forces which predate each individual's body, if not the human body in gen-
eral - for there has certainly been a long process of cultural refinement in
the design of hand tools. Like tools, language is another part of the socio-
cultural milieu within which we exist. Can we investigate how socio-
cultural factors (such as the language into which we are born) shape our
cognition?
Let us begin by considering matters of prepositional structure, perspec-
tive and frames of reference in a linguistic context. In English we can
speak metaphorically about features of the landscape in terms of the body,
such as the face of a mountain, the mouth of a river, the foothills, and on.
Peninsulas can be construed as fingers of land, or as heads (as in Hecata
Head). In other words we understand features of the landscape metaphori-
cally, using our bodies as the grounding frame of reference. Lakoff and
Johnson (1999, 1980) have called such systematic patterns of metaphoric
projection "conceptual metaphors", and have argued that they exhibit a
general tendency to conceptualize more abstract entities in terms of the
346 Tim Rohrer
more bodily ones. We now know that both literal and metaphorical uses of
body-part terms exhibit mental imagery effects similar to those described
in the experimental lines already discussed. In an fMRI study which in-
cluded instances of the LANDSCAPE IS A BODY metaphors, participants'
primary and secondary hand sensorimotor cortices were active during the
comprehension of both literal and metaphoric hand sentences (Rohrer
2001b, 2005). However, languages vary in how they construct space; is it
possible that what is a metaphoric usage in English is the basic frame of
reference habitually used by members of another culture?
Linguists have documented a number of Mayan languages such as
Mixtec, Tzeltal and Zapotec whose prepositional structure is entirely com-
posed of body-part morphemes. For example, saying the stone is under the
table requires saying the stone is proximal to the table's belly (yuu wa hi-
yaa cii-mesa / stone the be-located table-belly) (Lakoff 1987: 313). Within
Cognitive Linguistics Brugman (1985) and Lakoff (1987; see also related
work in MacLaury 1989) have claimed that such languages require pro-
jecting the names for body parts onto objects in the world. They argue that
Mixtec speakers start off with a viewer-centered frame of reference
2
and
then take up the perspective of the object. This change in perspective yields
an object-centered frame of reference for Mixtec spatial relation terminol-
ogy, where tables metaphorically acquire bellies located where a human
belly would be. From a purely neurophysiological conception of the body -
and one strongly influenced by an overly narrow conceptualization of the
brain in terms ofjust the visual system (and not the sensorimotor system)-
one could conclude that this order of events was inevitable, given that in
the visual system we first construe the world in our visual system in
viewer-centered neural maps, and only later in object-centered maps.3
Given the cognitive neuroscience available at that time, Lakoff (1987)
plausibly argued that speakers of such languages were metaphorically pro-
jecting the viewer-centered frame of reference to form another, object-
centered frame of reference.
2. Even though the topic has somewhat shifted to language, I am still using the
term "frame of reference" primarily in its spatial sense, as would be found in
cognitive psychology and cognitive neuroscience. I discuss the complex relation
between linguistic and non-linguistic frames of reference in Section 4.
3. Current studies of the sensorimotor system reveal that there are separable frame
of reference maps for body-centered (i.e., viewer-centered) mental rotation and
object-centered mental rotation (see review in Parsons 2003).
However, related evidence gathered in cross-cultural language acquisi-
tion studies reveals that the embodied mind is being shaped here not simply
by the neurophysiology but by the particular socio-cultural practices that
accompany language acquisition. The metaphoric projection hypothesis
predicts that such terms would be learned first as names for the body parts,
and only later extended to spatial relations terms. In a cross-cultural study
of Danish- and English-speaking children on one hand and Zapotec-
speaking children on the other, Jensen de Lopez and Sinha (1998; Sinha
and Jensen de Lopez 2000; Jensen de Lopez 2002) investigated whether
each culture's children acquire body-part morphemes first as body-part
terms and then only later metaphorically project them as spatial relations
terms. Their results show that Zapotec-speaking children acquire the body-
part morphemes first as spatial relations terms and only later - and seem-
ingly independently - as names for the body parts, while Danish and Eng-
lish children acquire them first as body-part names and only later use them
to indicate spatial relations. Furthermore, Jensen de Lopez and Sinha hy-
pothesize that the difference derives from differing cultural practices of
child-rearing. They note that Zapotec infants spend most of their first two
years in a sling on the mother's back, sharing her spatial perspective, while
Danish and English infants are placed in cribs and encouraged more to
move about on their own. Consequently, joint attentional episodes during
which the child's body parts are named may be less frequent in Zapotec
child-rearing practices than in Danish or English child-rearing practices. In
short, Jensen de Lopez and Sinha suggest that what might have looked like
a projection of viewer-centered body-part terms in order to form an object-
centered frame of reference is instead simply the acquisition of an object-
centered frame of reference through joint attentional episodes focused on
the spatial characteristics of such objects. The work of Jensen de Lopez
and Sinha, along with other cross-cultural language acquisition work
(Bowerman and Choi 2003), is an example of why embodied cognitive
science must include the socio-cultural milieu as one dimension of vari-
ability.
Note how similar the findings of Jensen de Lopez and Sinha are to the
Kosslyn et al. (2001) finding concerning how the actions that directly pre-
cede the mental rotation experiment influence which neural system is cho-
sen to perform the task. In both cases, the social context ofjoint attentional
episodes, whether between caregiver and child or experimenter and par-
ticipant, influences what frame of reference is chosen. To be embodied as a
human being means in part that we are born into a socio-cultural milieu
348 Tim Rohrer
within which we have particular problem-solving strategies reinforced
through experience. Deeply habitual experiences or an immediately prior
atlentional experience can alter which strategy we choose to employ. The
body of embodied cognitive science is not limited to physiological and
neurophysiological influences on mind, nor to that plus the physical body's
interactions with the physical world, but also incorporates the experiences
of the social and cultural body as well. In other words, it has to take ac-
count of the socio-cultural context within which a particular body is situ-
ated.
At this point it is clear that there are a number of different, if interre-
lated, senses of the term "embodiment" at play in the literature. I began this
chapter with the image of how the phenomenological body intrudes upon
the mind when the lights unexpectedly go out and one must fumble to find
the way out of a building, and then traced some examples of how such
experiential considerations motivate experiments that investigate the
physiological and neurophysiological responses to the experienced body
(as in imagining rotating the hand for both normal participants and partici-
pants with chronic arm pain). Continuing to examine the literature on
imagination and mental rotation, I showed how even a focus on measuring
the neurophysiology leads to the realization that both the physical envi-
ronment and the socio-cultural context are factors which impact the em-
bodied mind. Now it is time to begin addressing the meta-theoretic picture
explicitly. How can all these senses of the term hang together as a frame-
work for research on embodied cognition? What are the dimensions of
embodiment that different theorists think it is important to measure and
address? How many dimensions are there, and how do they interact to form
different research clusters?
2. Surveying the dimensions of embodiment
Like most scientists, linguists usually acknowledge that it is a difficult but
admirable goal to begin as descriptively as possible before proceeding pre-
scriptively. By my latest count the term "embodiment" can be used in at
least twelve different important senses with respect to our cognition. Be-
cause theorists often (and sometimes appropriately, given their specific
purposes) conflate two or more of these dimensions, it is important to get a
clear picture of as many of the different dimensions of variability as possi-
ble. This list is not intended to be entirely exhaustive of the term's current
usage, nor are the dimensions necessarily entirely independent of each
other nor even entirely distinct from one another. Thus it is important to
note that, and unlike the argumentative analyses given in Anderson (2003)
or Wilson (2002), this initial survey is not intended to be a prescriptive
definition ofthe term, but instead is intended only to catalogue some of the
contemporary usage of the term in a way that reveals the most relevant
dimensions to which one must be responsive in order to develop a general
theoretic framework for embodiment theory in cognitive science. However,
I do note where some theorists have used the term in several of these
senses simultaneously, and in several cases I argue for making finer dis-
tinctions and recognizing more dimensions than do the original theorists
themselves.
2.1. Dimension 1: Philosophy
First - and perhaps most broadly - "embodiment" is used as a shorthand
term for a counter-Cartesian philosophical account of mind, cognition and
language. Descartes took problems within geometric and mathematical
reasoning (such as the meaning of the term "triangle") as model problems
for study, and concluded that knowledge par excellance is "disembodied"
- that is, fundamentally independent of any particular bodily sensation,
experience, or perspective - all of which are roots of uncertainty. In argu-
ing that the meaning of the term "triangle" consists in the reference rela-
tionship between the word and an object that exists not in the physical,
embodied world but in thought alone, Descartes' thought experiments set
the stage for many thinkers within analytic philosophy, formal semantics,
and early cognitive science. Broadly speaking, such philosophers of lan-
guage typically construe the two central problems of meaning to be (i)
mapping the reference relations between idealised objects of knowledge,
their counterpart symbolic expressions in language and the objects or
"states of affairs" in the real world (as in Fregean semantics), and (ii) ex-
plaining the internal logical structure of the relations which hold between
these idealised objects or their corresponding linguistic symbols (as in
theories of "autonomous syntax"). While Descartes was by no means
unique nor alone within Western philosophy in claiming this position, his
extraordinary clarity has garnered him the laurel of becoming metonymic
for this package of philosophical assumptions (Lakoff and Johnson 1980,
1999; Geeraerts 1985; Johnson 1987; Rohrer 1998). Most such embodi-
350 Tim Rohrer
ment theorists, while perhaps somewhat favouring the empiricist side of
the rationalist-empiricist split, generally try to dissolve such philosophical
problems as hangovers from a bad metaphysics by recasting the problems
in a different metaphysical basis. Many, although not all, of the theorists
objecting to such Cartesian treatments of language, meaning and represen-
tation use "embodiment" in this broadly philosophical sense even as they
work explicitly in one or more of the somewhat narrower dimensions of the
term that follow.
2.2. Dimension 2: The socio-cultural situation
"Embodiment" is also used to refer to the social and cultural practices
within which the body, cognition and language are perpetually situated. In
this sense, "embodiment" is often used to emphasize the particularistic,
rather than the universalistic, tendencies of human cognition; e.g., how a
particular mind in a particular body is shaped by the particular culture
within which it is embedded. One example of a cognitive cross-cultural
language acquisition study would be the previously discussed research by
Sinha and Jensen de Lopez (2000). The cultural variation in child-rearing
practices might well account for differing acquisition sequences of spatial
language terms in English-, Danish-, and Zapotec-speaking children.
Such socio-cultural practices can be given material form in the material
artifacts that aid and manifest cognition - many of which are extensions of
the body (Hutchins 1995, 1999; Fauconnier and Turner 2002). In assessing
the differences between Micronesian and Western traditions of navigation,
Hutchins observes that the
[... ] physical artifacts became repositories of knowledge, and they were
constructed in durable media so that a single artifact might come to repre-
sent more than any individual could know. Furthermore, through the combi-
nation and superimposition of task-relevant structure, artifacts came to em-
body kinds of knowledge that would be extremely difficult to represent
mentally. (1995: 96)
Hutchins cites the example of the medieval astrolabe, a set of rotating disks
that embody the spatial relationships of the celestial bodies at different
latitudes with much greater precision than would be possible from only the
individual navigator's memory. These are set into a frame that represents
the horizon, which is itself inset with a scale marking out the 24-hour day
and/or the 360 degrees of the compass. He notes that the astrolabe embod-
ies socio-cultural practices in two important ways. First, the astrolabe is an
extension of the body in that a skilled navigator physically manipulates it
by rotating its disks in order to predict celestial movements. Second, in its
design the astrolabe is "a physical residuum of generations ofastronomical
practice" (Hutchins 1995: 96-97). Any particular navigator using the as-
trolabe is the intellectual heir of a wide set of social practices which have
been designed into the instrument. In contrast, a Micronesian navigator
eschews such material artifacts, relying successfully instead on cultural
artifacts such as chants that encode the relevant celestial relations for voy-
ages between particular islands (Hutchins 1995: 65-92, 111). However,
such cultural artifacts perform a similar function in that they also embody
generations of knowledge gleaned from navigational practices.
2.3. Dimension 3: Phenomenology
"Embodiment" has a phenomenological sense in which it can refer to the
things we consciously notice about the role of our bodies in shaping our
self-identities and our culture through acts of conscious introspection and
deliberate reflection on the lived structures of our experience (Brandt 2000,
1999). The conscious phenomenology of cognitive semiotics can be prof-
itably contrasted with the cognitive unconscious of cognitive psychology
(see dimension 7). For example, Gallagher (this volume) traces how the
work of phenomenological philosophers such as Husserl and Merleau-
Ponty has contributed to the distinction between the conscious body image
and the largely automatic body schema now emerging in cognitive neurosci-
ence. For Husserl and Merleau-Ponty, embodiment refers not only to the
lived experience of our own bodies but also to the ways in which our expe-
rience of other animate bodies moving differs from our experience of other
moving objects in the physical world. This has found theoretic support
from cognitive neuroscience in the discovery of the "mirror neuron" sys-
tem in the premotor cortex (Gallese et al. 1996; Rizzolatti and Craighero
2004), in which primates have been shown to have neural systems which
are activated not only by their own motor actions but also by witnessing
another's motor action. Gallagher suggests that the emergent sense of "in-
tercorporeality" from mirror neuron activity could be a basis of human
intersubjectivity.
352 Tim Rohrer
2.4. Dimension 4: Perspective
"Embodiment" can also refer to the particular subjective vantage point
from which a particular perspective is taken, as opposed to the tradition of
the all-seeing, all-lmowing, objective and panoptic vantage point. While
this sense of the term can be seen as at least partly philosophical (as in
Nagel 1979: 196-213; Geeraerts 1985; Johnson 1987; Rohrer 1998), the
idea of considering the embodied viewpoint of the speaker has linguistic
implications for the role of perspective in subjective construal (Langacker
1990; MacWhinney 2003), as well as a myriad number of psychological
implications (e.g., Carlson-Radvansky and Irwin 1993; Kosslyn et al.
2002).
For example, consider how the embodied perspective of the subject can
interact with the canonical orientations implicit to construing spatial situa-
tions. When we give directions, we ordinarily assume that one is facing in
the direction of the travel. However, in many subway trains the seats face
in both directions. Should we give directions such as "after the subway
goes above ground, look to your left. When you pass the automobile deal-
ership, exit at the next stop...", imagine the confusion if the addressee
should choose a seat facing opposite in the direction of travel and not make
the adjustment to look to the right. Similarly, not only our bodies but also
many of the objects of our world have canonical orientations that our de-
termined by the ways in which we - that is our bodies - interact with them.
Cups and trash-cans stand upright, while mattresses lie flat. When we say
"the fly is over the trash can", but the trash can is lying on its side, is the
fly above the side of the trash-can or adjacent to its mouth? Buildings such
as cathedrals or ski lodges also have canonical orientations as to their
fronts and backs; one can say "1'11 meet you in the restaurant to the right of
the cathedral at noon", and inadvertently fail to specify if that is the per-
spective of the tourist facing the cathedral or the perspective of the cathe-
dral as it faces the city square. We routinely project the canonical orienta-
tion of our embodiment onto the objects in the world; sometimes we take
up the perspective of inanimate objects, sometimes we take up the per-
spective of animate bodies. Of course, problems of co-aligning frames of
reference are of practical import in areas such as ship navigation practices
and the internal maps built up by robots; as such this dimension frequently
interacts with the other practical senses of the term as well as the meta-
theoretic ones.
2.5. Dimension 5: Development
In yet another important sense "embodiment" is used to refer to the devel-
opmental changes that the organism goes through as it transforms from
zygote to fetus, or from infant to adult. There are at least three ways in
which the developmental sense interacts with the other dimensions of "em-
bodiment". First, certain events in the development of an organism open
windows for the acquisition of a particular skill. Babies are not born
speaking, nor can they handle objects or self-Iocomote at birth. As the in-
fant acquires additional sensorimotor skills, additional patterns become
available to be incorporated into its cognitive functioning. Second, and
perhaps counterintuitively, such events may not expand but instead con-
strain the mappings between the possible patterns of embodied perceptual
structures and the resulting conceptual structures of later developmental
stages. For example, Bowerman and Choi (2003) have shown that while
nine-month old Korean and English speaking infants can make the same
spatial discriminations, at eighteen months their acquisition of language
has solidified their spatial categories enough so that they are no longer able
to make the discriminations which their language does not. Note that such
developmental changes are not purely physiological, but take place within
the relevant socio-cultural and linguistic contexts.
2.6. Dimension 6: Evolution
An equally important temporal sense of the term "embodiment" refers to
the evolutionary course the species of organism has undergone throughout
the course of its genetic history. For example, an account of the gradual
differentiation of the cortex into separate neural maps each representing a
different frame of reference in the visual and tactile systems of mammals
might provide an evolutionary explanation for which multiple frames for
spatial reference were universally found by the typological studies of spa-
tial language and cognition (Majid et al. 2004). Or on an even grander
scale: humans have not always had the capacity for language and so evi-
dence from studies on the evolutionary dimension of embodiment may
often prove crucial to understanding why, for example, language process-
ing in the brain does not appear to be exclusively concentrated as an
autonomous module but instead draws on numerous subsystems from the
354 Tim Rohrer
perceptual modalities (see for treatments Edelman 1992; Donald 1991;
Deacon 1997; MacWhinney 1999).
2.7. Dimension 7: The cognitive unconscious
Additionally "embodiment" can mean those routine cognitive activities that
ordinarily operate too quickly and too automatically for the conscious mind
to focus on them. Lakoff and Johnson (1999: 9-15) have recently called
these the cognitive unconscious. In this sense "embodiment" refers to the
ways in which our conceptual thought is shaped by many processes below
the threshold of our ordinary conscious awareness. As such they are gener-
ally inaccessible to introspection, though they may be measured indirectly
using methods from cognitive and social psychology. Lakoff and Johnson
cite examples ranging from mental imagery to semantic processing to proc-
essing sound into phonemes. They intend that this sense of the term "de-
scribes all unconscious mental operations concerned with conceptual sys-
tems, meaning, inference and language" (1999: 12). While their primary
source of evidence for the cognitive unconscious is cognitive psychology,
Lakoff and Johnson also intend for this sense of embodiment to include the
neural modelling at least some aspects of our neurophysiological embodi-
ment, noting that these "are obviously not independent of one another"
(1999: 104). However, I would argue that these two aspects can be profita-
bly separated out from the cognitive unconscious of experimental cognitive
and social psychology as two additional dimensions of embodiment.
2.8. Dimension 8: Neurophysiology
In a neurophysiological sense, the term "embodiment" can refer to meas-
uring the activity of the particular neural structures and cortical regions
that accomplish feats like object-centered versus viewer-centered frames of
reference in the visual system, metaphoric projection, and so on (Rohrer
2001b, 2005; Coulson and Van Petlen 2002). Such methods would include
single-neuron recording, electroencephalography (EEG) and derivative
measures (EMG and ERP), positron emission tomography (PET), func-
tional magnetic resonance imaging (fMRI), and magneto-encephalography
(MEG), as well as the neuroanatomical organization of the brain and nerv-
ous system. This dimension would comprise a portion of, but not be syn-
onymous with Lakoff and Johnson's use of their term "neural embodi-
ment" (1999: 102-103), in which they lump neurophysiologically-based
methods together with the neurocomputational modelling of both high-
level cognitive tasks (such as temporal aspect in language) and low-level
cognitive tasks (spatial perception). Together with observations on human
physiology, the relevant neurophysiology is sometimes advanced as ex-
plaining certain constraints on the patterns exhibited in linguistic systems,
such as in the regularities in the cross-cultural typology of color words
(Lakoff 1987).
2.9. Dimension 9: Neurocomputational modelling
"Embodiment" can sometimes also refer to research using neurocomputa-
tional models. Such neural networks may be said to be "embodied" in at
least four different ways. First, they may more or less closely model the
actual neurophysiology of the neural circuitry whose function they seek to
emulate. Second, some kinds of neural networks build on better-understood
neurocomputational models of the actual neurophysiology to provide "ex-
istence proofs" that a series of neural nets could in principle account some
kind of cognitive behaviour - as in "structured connectionism". Neuro-
computational models such as those in Lakoff and collaborators' "neural
theory of language" thus are not explicit models of the underlying neuro-
physiology, but instead (and by using as their input structures the output
from better understood perceptual neural structures) they seek to demon-
strate how the known computational facts about the neurophysiology could
produce certain kinds of observable linguistic behaviours (such as the
metaphoric structuring of more abstract experience in terms of perceptual
experience) (Lakoff and Johnson 1999: 569-83; see also Regier 1992,
1995; Feldman and Narayanan 2003). Third, and most often without any
explicit reference to any intermediate structures in the underlying neuroa-
natomy, connectionist neural networks are taken to be models of experien-
tial activity at the conceptual and/or psychological levels of processing, as
in the stochastically-based arguments that there is no "poverty of stimulus"
but instead plenty of experience to account for the acquisition of syntax by
children - (Elman et al. 1996; MacWhinney 2003). Fourth, neural net-
works can be seen as models of how socio-cultural norms can be internal-
ized within a specific "mind" (Zlatev 1997, 2003; Howard 2001). For ex-
ample, by developing neural models of how the age/gender contrasts are
356 Tim Rohrer
marked in English and Spanish, Howard argues that biased socio-cultural
norms of age and gender are partly the result of predispositions of how the
human brain and nervous system learn.
It is important to note that the neurocomputational use of this sense of
"embodiment" is partly motivated by the fact that some other physiologi-
cal, experiential and/or socio-cultural dimension of that term is explicitly
being modeled by the neural network. Yet such models seek to ground their
models not only in what they seek to model, but also in the fact that "neu-
rocomputational embodiment" is explicitly anti-functionalist. All neural
networks are anti-functionalist in that the particular shape of the neural
model is at least partly determined by analogizing some of its computa-
tional properties to the underlying neurophysiology, rather than presuming
that the cognition or behavior to be modeled is computationally independ-
ent of any such bodily constraints (as in functionalist models).
2.10. Dimension 10: Morphology
The terms "embodiment" and "embodied cognition" are now also widely
used in robotics (Chrisley and Ziemke 2002) where any computational
modelling necessarily requires a body of some type for interaction with the
world. While in robotics it is perhaps most saliently associated there with
humanoid robot projects (Brooks and Stein 1994), it can also refer to cases
where the work done by the robot depends on the particular morphological
characteristics of the robot body (Pfeifer and Scheier 1999). For example,
Comell University's Passive Dynamic Walker uses no motors and no cen-
tralized computation but instead relies on gravity, mechanical springs and
cleverly designed limb morphology to "walk". By exploiting the capacities
of the morphology, cognition is offloaded onto the body - a design princi-
ple that is consonant with both evolutionary theory and embodied cognitive
science (Collins, Wisse and Ruina 2001; Bertram and Ruina 2001).
The morphology of the physiological body also yields important con-
straints for measuring cognition in cognitive psychology or cognitive neu-
rophysiology, as in the already discussed studies by Parsons (1987ab,
1994; Parsons et al. 1995) on the mental rotation of line drawings of the
hand and in the Vingerhoets et al. (2002) fMRI study of the activation
courses in response to images of tools.
2.11. Dimension 11: Directionality of metaphor
Within Cognitive Linguistics, the term "embodiment" has two often con-
flated senses that stem from Lakoff and Johnson's (1980: 112) initial for-
mulation of the embodiment hypothesis as a constraint on the directionality
of metaphoric structuring. More accurately, this sense of "embodiment"
could be termed the directionality of metaphor mappings. In this strong
directionality constraint Lakoff and Johnson claim that we normally project
image-schematic patterns of knowledge unidirectionally from a more em-
bodied source domain to understand a less well-understood target domain.
In other words, they claim that each and every mapping between the ele-
ments of the source and the elements of the target is unidirectional; the
logic of the image-schema is projected from the source to the target, and
not from target to source. For example, in their analysis of the metaphors
shaping various theories of visual attention in cognitive psychology, Fer-
nandez-Duque and Johnson claim that:
[... ] each submapping is directional, going from source to target. We under-
stand aspects of the target domain via the source domain structures and not
the reverse. Such unidirectionality shows itself clearly in the reasoning we
do based upon conceptual metaphors. (Femandez-Duque and Johnson 1999:
85)
This constraint has been the source of much controversy within Cognitive
Linguistics. Fauconnier and Turner (1995, 2002) and others have argued
that there is a much greater role for feedback between target and source to
the extent that they have proposed an alternate theory in cognitive seman-
tics, conceptual blending, which is in part a response to the unidirectional-
ity constraint. Non-unidirectional and blending-like phenomena can be
observed with respect to the same theories of visual attention analyzed by
Fernandez-Duque and Johnson. For example, they correctly argue that a
VISUAL ATfENTION IS ASPOTLIGHT metaphor shapes research questions in
cognitive psychology, such as for experiments designed to measure the
speed at which the attentional spotlight moves across the visuo-spatial field
when experimental participants shift the focus of their attention (Shulman,
Remington and McLean 1979) and whether the subject would attend to
intermediary objects in the path of the attentional spotlight (Tsal 1983).
However, the cognitive psychologist Miisseler (1994) observed that al-
though relatively proximal shifts in the focus of visual attention seemed to
behave as if the attentional spotlight did follow an analog path across the
visual field illuminating everything in its path, larger shifts in the focus of
358 Tim Rohrer
visual attention did not follow an analog path. This observation about the
target domain (visual attention) initiated a re-examination of the source
domain. Miisseler initially proposed that the attentional spotlight was "re-
set" during large attentional shifts. However, convergent evidence from
both other psychological studies of attention and from cognitive neuropsy-
chology on colour, motion, shape and other visual subsystems caused an
even more radical shift in the source domain of the metaphor - visual at-
tention began to be understood as an array of multiple spotlights, as would
be found in a theatre (Rohrer 1998). In no case was this process of feed-
back, revision, and accommodation strictly unidirectional; it was always
motivated by observed changes in the target domain of visual attention that
required making changes to the source domain of the spotlight(s).
2.12. Dimension 12: Grounding
Finally, "embodiment" can be used to refer to a particular hypothesis as to
how we might explain how abstract symbolic behaviour is grounded in
experience. Within Cognitive Linguistics even Lakoff and Johnson's origi-
nal formulation (1980: 112) of the embodiment hypothesis contained the
germ of a broad generalisation about the kinds of basic conceptual domains
which were typically serving as source domains for conceptual metaphors,
rather than as explicitly referring to the directionality of projection for each
and every element mapped within a particular metaphor. We might call this
additional sense of embodiment the directionality of explanation in order
to distinguish it from the directionality of metaphor mappings. Lakoff and
Turner specifically acknowledge this sense of embodiment in their
Hgrounding hypothesis ", in which they argued that meaning is grounded in
the sense that we must choose from a finite number of semantically
autonomous source domains to understand more abstract experiences (La-
koff and Turner 1990: 113-120). This sense of the term is related to the
symbol-grounding problem in cognitive science generally (Harnad 1990),
though it is important to note that many embodiment theorists would want
to address that issue without conceding any sort of Cartesian-like split be-
tween words, thoughts or symbols and the worldly things to which they
refer.
This descriptive list is meant to illustrate that the embodied cognitive
science requires thinking through evidence drawn from a multiplicity of
perspectives on embodiment, and therefore drawn from multiple method-
ologies. Of course almost no researcher or research project can attend to all
these different senses of the term at once and still produce scientific find-
ings, but research projects that build bridges or perform parallel experi-
ments across these differing dimensions are of particular interest. However,
once the descriptive work has been done it can be seen that many of these
senses cluster about at least two poles of attraction. Critiques of embodied
cognitive science from within have often given voice to two broad senses
in which the term "embodiment" is used. These two could be well de-
scribed as "embodiment as broadly experiential" and "embodiment as the
physical substrate". In one cluster the term refers to dimensions that focus
on the specifically subjective, contextual, social, cultural and historical
experiences of language speakers. Dimensions (2) through (4) of my enu-
meration of the term's usage would typically cluster in this realm, while
dimensions (7) through (10) would often cluster about the pole which em-
phasizes the physiological and neurophysiological bodily substrate that is
typically associated with supposedly more "objective" methodologies.
Such a division is at best rough and provisional however. Clearly not all
the dimensions of the term can be so clustered, given that the attention to
temporal character which characterizes the developmental (5) and evolu-
tionary (6) dimensions can place them about either pole. Similarly, there
are many interesting studies which bridge the gap between the experiential
and the physiological poles even while largely measuring a dimension typi-
cally construed as mostly one or the other, such as the Kosslyn group's
neuroimaging research into alternate strategies of mental rotation (2001).
Depending on the behaviour modelled, embodiment as neurocomputational
modelling (9) can also cross the line from the physical substrate to more
experiential matters. Finally, the more explicitly meta-theoretical dimen-
sions of the term, (1), (11) and (12), have much traffic with both the expe-
riential and physical substrate poles and also do not lend themselves easily
to such a rough and ready distinction. Given such considerations, at least
two more poles of attraction emerge - temporal and metatheoretical stud-
ies. In the end, an adequate theoretic framework for embodiment theory in
cognitive science will have to acknowledge all of the wide variety of
senses in which the term "embodiment" is being used and provide a non-
reductionistic framework for reconciling research across all these different
dimensions.
360 Tim Rohrer
3. The levels of investigation theoretic framework
The rough and ready distinction between experiential and physiological
embodiment does have the virtue, however, of illuminating how we might
assess the utility of different approaches to embodied cognition. Two other
recent attempts to clarify the uses of the term "embodiment" have come to
diametrically opposed conclusions as to how embodiment theory can con-
tribute to future work. On one tack, the computer scientist Anderson (2003)
proposes evaluating embodiment theory in terms of its practical utility in
reconceiving how our work and our lives can be enhanced. His objectives
are fundamentally technological, inquiring whether embodiment theory has
any import for efforts to offload difficult cognitive tasks onto the social
and cultural environment, such as teamwork or the design of intelligent
material artifacts such as embedded computers. On another tack, theorists
such as the psychologist Wilson (2002) propose an experimentally-based
evaluation of embodiment theory, arguing that we need more investigations
of how offline, subconscious bodily processes structure real-time cognition
while explicitly rejecting efforts to explain cognition as being situated in
and distributed across socio-cultural practices. Clearly, these approaches
differ not only as to what direction future research should take but also in
terms of the physical scale of their research scope. Wilson's concerns are
primarily with how the individual physiological organism interacts with its
environment, arguing that distributed social and cultural patterns of cogni-
tion are too impermanent to constitute a unit of analysis having explanatory
force (2002: 630-631). While Anderson also agrees that insights from how
the embodied individual organism interacts with the physical world are
relevant, he argues that embodied cognition has "social significance, for
the construction of meaning, of the terms through which we encounter the
world, is not generally private, but is rather a shared and social practice"
(2003: 125). In considering whether and how a material artifact such as
patient's medical chart might be extended by embedded computers, he
notes that such efforts should not seek to obliterate important facets of how
the material artifact encodes the fact that cognition is distributed across the
social group caring for the patient, such as how the handwriting indicates
what member of the group made a particular observation or where the chart
is kept indicates what member of the medical team has responsibility for
the patient. On Wilson's argument the scope of the research would be lim-
ited to the size of space within which the individual interacts with the arti-
fact, but for those cognitive scientists interested in improving the func-
tioning of such social groups, the scope of the physical scale expands to
encompass the entire cultural and communicative space of the team.
However, from my own vantage as a cognitive scientist originally
trained in the philosophical tradition of American Pragmatism, both of
these theorists are sailing in the right direction - given that the relevant
operational scale of the phenomena they are studying has changed. The
first challenge for developing a theoretical framework in which we can
address such differing approaches is to propose the adoption of a simple
and well-understood organising criterion. Unfortunately, most previous
proposals have generally accorded an ontological status, rather than an
epistemological or methodological status, to the organisation of their theo-
retic framework. Thus most such frameworks postulate "higher" and
"lower" levels of cognition in ways which imply that the higher levels may
be reduced to operations at the lower levels, ultimately arguing for the
elimination of higher-levels of description in favour of lower levels of de-
scription (Churchland 1981, 1989). One exception is Posner and Raichle's
(1994) schematisation of the levels of investigation in cognitive neurosci-
ence, in which the primary emphasis is given to the methodologies used to
investigate the phenomena rather than their ontological status. Similarly,
Edelman (1992) points out that in the physical sciences, the phenomena are
operationally grouped in levels according to the physical scale of the
methodology with which the phenomena are being studied. Thus the most
basic organising criterion of this theoretic framework is the scale of the
relative physical sizes of the embodied phenomena which produce the dif-
ferent kinds of socio-cultural, cognitive or neural events to be studied.
In Figure 1, physical size is mapped on the y-axis, providing a relative
distribution of the "higher to lower" methodological levels of cognitive
processes. A general name for each level is indicated by boldface type in
the first column. To provide clarification, the next column provides exam-
ples of what the relevant physiological structures are at a given physical
scale. For example, at the communicative, cultural and social level we
study spatial language as it used between people, and hence multiple cen-
tral nervous systems; alternatively, it is possible to measure one individ-
ual's (and hence one central nervous system's) performance on a similar
set of linguistic tasks. Similarly we can examine, with even more granular-
ity, relative changes in cerebral blood flow to regions of the brain in re-
sponse to spatial linguistic tasks; or we can construct neurocomputational
models of those brain regions. However, Posner and Raichle's key
362 Tim Rohrer
Size Physiological Level of Typical Sample Sample Methods
Structures Investigation Cognitive Operative o/Studyand
Science Tasks Theoretic Measurement
Constructs
lm Multiple Communicative Cross-cultural Viewer-centered, Linguistic analysis,
and Central and cultural investigations of obj ect-centered, cross-linguistic
up Nervous systems in mental rotation geo-centered typology, videotaped
Systems anthropology, and frames of frames of refer- interview, cognitive
language, science reference; ence in language; ethnography
and philosophy language child-rearing
acquisition; practices;
conceptual norms as to which
metaphor; gesture spatial frame used
.5m Central Performance Individual Spatial frames of Verbal report,
to Nervous domain; performance on reference, speed observational
2m Systems Cognitive, frames of refer- of mental neurology, discourse
conceptual, ence and mental rotation; analysis, cognitive
gestural and rotation tasks; morphological and Developmental
linguistic systems measuring ability constraints studies examining
as performed by to gesture in reaction time (RT)
individual subjects direction-giving
situations
contrived to
inhibit it
IO-lm
Gross to Neural systems Activation course Body-image, Lesion analysis,
to medium size in somatosensory, motor and visual neurological
IO-2
m neural auditory, and cortices, what- dissociations,
Regions visual processing where pathway; neuroimaging using
(anterior areas when fMRI and PET, ERP
cingulate, processing spatial methods,
parietal lobe, frames of refer- neurocomputational
etc.) ence tasks or simulations
mental rotation
tasks
IO-lm
Neural Neuroanatomy; Neuroanatomical Motor and visual Electrocellular
to networks, Neural circuitry in connections from cortices, parietal recording,
IO-4
m maps and maps, pathways, visual, auditory, topographic neural anatomical dyes,
pathways sheets somatosensory maps neurocomputational
regions to simulations
language areas
IO-J
m Individual Neurocellular Fine neuro- Orientation-tuning Electrocellular
to neurons, systems; Cellular anatomicalorgani- cells; ocular Recording,
IO-6
m cortical and very small sation of particular dominance anatomical dyes,
columns intercellular structures re- columns neurocomputational
structures cruited in lan- simulations
Iguage processing
Less Neuro- Subcellular None-beyond Neurotransmitter, Neuro-
Than transmitters, systems; theoretical scope sYnapse, ion pharmacological,
IO-6
m ion channels, subcellular, mo- channels neurochemical,
sYnapses lecular and and neurophysical
electrophysical methods
Figure 1. Theoretic framework for embodied cognitive science.
insight is that it is important to consider how the basic inquiry changes
given the different tasks and methods at various levels of investigation. All
methodologies have constraints and freedoms which limit or enhance their
scope of investigation and define the theoretical constructs that they de-
velop, and these are a product of the physical scale at which the measure-
ment is taking place. The final two columns acknowledge this by specify-
ing some of the relevant theoretic constructs and the various methodologies
operative at each level of investigation.
This framework can be used to structure studies of various topics of in-
terest to cognitive scientists, such as mental imagery, frames of reference,
metaphor and so on. While this type of theoretic framework is becoming
commoner within much of cognitive neuroscience, most embodiment theo-
rists have been slow to give explicit attention to the problem of how we are
to theoretically situate and reconcile these different levels of investigation,
perhaps due to a fear of appearing to favor reductionism.
I have included just a single level of cultural and communicative analy-
sis, but by no means should this be taken as indicative of its importance
relative to other the other levels. Of course, one could argue for a multi-
plicity of levels embedded within this one, though they might not be clearly
differentiated from one another in terms of physical scale. In choosing to
include a general level situated at a meter and up on the physical size axis I
mean to emphasize only that human beings should be considered not sim-
ply in terms of physiological size, but also in terms of the standard scale of
their interactional distance in speaking and interacting with one another. At
this level of the chart the "physiological structures" column reads "Multi-
ple Central Nervous Systems", but that awkward term is intentionally in-
adequate so as to emphasize that the physiology is less relevant here - what
primarily matters on this level are the social and cultural interactions be-
tween human beings. Investigations at the cultural level are occasionally
given short shrift by some versions of embodied cognitive science, but
generally this has been and should remain a strong thrust of future research
in the field. Note also that difficult phenomena such as cultural and lin-
guistic norms, or individual consciousness and awareness, are situated at
the physical scale at which they are measured and observed, rather than
attempting to place them on (or reduce them to) a lower level of investiga-
tion. Nonetheless, it is certainly possible and sometimes useful to ask, for
example, how long the neural processing of a visual experience takes be-
fore impacting conscious decision-making, or how the linguistic norm of
forming the English past tense might be performed in a neurocomputa-
364 Tim Rohrer
tional model. However, research in embodied cognitive science should not
seek to reduce such phenomena to another level but should instead bridge
across these levels in important ways - for example, the linguistic corpora
used to train the neurocomputational model should be based on naturalistic
recordings of an actual child's utterances rather than text harvested from
internet newsgroups, and so on.
While the chart depicting the theoretic framework is designed to give an
overview of the relationship between body, brain and culture, this repre-
sentation is not as illustrative for issues pertaining to evolutionary and
developmental time scales, which may be considered at any of these levels.
However, this failing is more a limitation of the imagery of a two-
dimensional chart than of the theoretic framework itself. If we were to add
another axis for time perpendicular to the surface plane of the chart, we
could imagine this framework as a rectangular solid. I have omitted repre-
senting this dimension because such an illustration would make it difficult
to label the levels, but I make the point explicit here because both the de-
velopmental and evolutionary time courses of these phenomena are a cen-
tral dimension to understanding them, and their bearing on the embodied
mind.
4. Applications of the theoretic framework
This theoretic framework can help link related research from one level of
investigation to another, providing opportunities to test similar hypotheses
and incorporate insights originally developed at one level of investigation
at another. As in the introduction I have already given several examples of
embodied cognition linking physiological and neurophysiological experi-
ments on cognition, consider three examples from the cultural and perfor-
mative levels of investigation involving mental rotation and frames of ref-
erence.
First, in a series of cross-cultural and cross-linguistic typological studies
on spatial cognition, Pederson et al. (1998) have found that the linguistic
frame of reference
4
which predominates in a language strongly influences
4. As I summarize their typological work I use Levinson's (2003) terminology for
linguistic frames of reference, though I try to indicate a rough alternative term
from the broader literature on spatial frames of reference in the cognitive sci-
ences where their nomenclature may be uninformative to the naIve reader.
However, such indications are intended only as clarifying approximations, and
the spatial cognition problem-solving strategy chosen. In one of several
tasks they use, experimenters place three animal figurines in a row on a
table and ask participants to memorize the scene. The participants are then
rotated 180 degrees and asked to reconstruct the scene on a second table.
Speakers of languages which predominately use a relative frame of refer-
ence normally recreate the scene relative to their own body position, i.e.,
the animal to their left in the original rotation remains the animal on their
left in the new 180-degree rotation. However, speakers of languages in
which an absolute (geo-centric) frame of reference predominates recreate
the same scene relative to the position of the animals with respect to in-
variant features of the landscape, i.e., the animal to the north is still placed
on the north side even though the participant's body position has been ro-
tated 180 degrees from the original scene. Despite numerous challenges,
they have successfully replicated their findings in a variety of experimental
environments and across a wide variety of languages (Majid et al. 2004). A
typological finding, gathered at the cultural level of investigation, has been
transposed onto the performative level of investigation.
Their experimental research has been extended by a second source of
evidence concerning spatial frames of reference which is of substantial
interest to researchers working on embodiment - studies of gesture. By
cleverly manipulating the body position of his experimental subjects rela-
tive to the directions the experimenter requests, Kita (2003) has shown that
in giving directions speakers of languages in which the relative frame of
reference predominates will often make difficult, torso-twisting gestures
across their bodies in an attempt to co-align their current bodily perspective
with the perspective that they would need to have in order to tell if a land-
mark at that point in the directions will be on the right or left. Just as with
the work on mental rotation and mental imagery, in direction-giving the
embodied mind simulates following the path of the directions being given.
This process of mental imagery - the mental gymnastics done by the
speaker's mind as one visualizes how the hearer will need to follow the
directions - surfaces in the form of physical gymnastics - that is, the ges-
tures involving the rotation of the torso. Once again, these experiments link
the cultural and performative levels of investigation.
Third, evidence of the ways in which language and culture embody
spatial frames of reference has also been found in studies linking gesture
readers should refer to Levinson's own work for detailed defmitions of his
terms for the linguistic frames of reference.
366 Tim Rohrer
and conceptual metaphor. In Indo-European languages such as English and
Spanish, time is typically conceptualized using two basic TIME IS SPACE
metaphors - one in which the observer stands still while times pass by
(e.g., "The end of the year is coming up on us soon"; and another in which
the observer is moving through a landscape of times (e.g., "We're coming
up on the end of the year") (Lakoff and Johnson 1980, 1999). However, in
both metaphor systems the observer faces the future, while the past is be-
hind the observer. Nufiez and Sweetser (2006) have shown that for speak-
ers of Aymara, who almost exclusively use a stationary version of the TIME
IS SPACE metaphor, the past is in front and the future comes from behind.
In Aymara, the orientation of the spatial frame of reference is reversed.
Using videotaped interviews with bilingual Spanish and Aymara speakers
both recounting Aymara legends and talking about their own communities'
immediate future, they demonstrate how speakers gesture to areas in front
of them when referring to the past, while gesturing to future events with
over the shoulder motions. Furthermore, the gestures reveal that more re-
cent past events are closer to the speaker's point of view than events in the
more distant past. For example, as he contrasts ancient times with current
events, an informant gestures by pointing outward and upward as opposed
to pointing four times closer to his body. Together with their linguistic
research on the Aymara conceptual metaphors for time, the research of
Nufiez and Sweetser shows that the Aymara map the temporal frame of
reference onto a viewer-centered frame of reference in an inverse direction
to that found in English or Spanish.
These three studies, largely situated at the "top" levels of the theoretic
framework, are excellent examples of how the cultural level of embodi-
ment is expressed in the performances of individual experimental partici-
pants. Not only that, but they also provide clues of what researchers might
ask next at other levels of investigation. For example, one might ask what
sorts of analogous linguistic stimuli would show neural responses similar
to spatial stimuli. One could design neuroimaging experiments investigat-
ing whether navigating directions on a visual display required mental im-
agery rotation in either the visual or motor imagery system. Analogously,
one could also investigate whether or not sentence stimuli concerning a
similar navigational task requiring either a viewer-centered, object-
centered, or geo-centered set of coordinates would activate distinct brain
regions.
There are several convergent neural studies which indicate that this is a
hypothesis worth pursuing. In a recent review article, Parsons (2003) ar-
gues that we can already distinguish between superior parietal cortical
regions active when mentally rotating objects about their own axis (i.e., in
an object-centered frame) and superior parietal regions active when men-
tally rotating one's own body (a viewer-centered frame). Several neuroi-
maging studies using spatial relations terms as linguistic stimuli also show
activation of either left (Damasio et al. 2001) or both (Emmorey et al.
2005; Carpenter et al. 1999) superior parietal cortical areas, though none of
those tasks were designed to elicit distinctions between the frames of refer-
ence. Similarly, there is also convergent evidence from the literature on
attention in visual hemi-neglect (a syndrome typically resulting from dam-
age to the right superior parietal cortex) suggesting that viewer-centered
and object-centered neglect are dissociable phenomena in experimental and
observational neurology (Behrmann and Tipper 1999; see Rohrer 2001a for
discussion). Halligan et al. (2003) reviews the neurological evidence that
neglect of near and far space occurs independently, suggesting that they are
likely to be mapped in separable parietal areas - evidence which suggests
that the geocentric frame of reference in language could be correlated with
the map of far space. Furthermore, while most studies of neglect do not
report language deficits on standard naming or visuo-auditory picture
matching tasks, a recent study by Eden et al. (2003) shows that when dys-
lexic children were asked to draw a clock face they neglected to fill in the
numbers on the left side of the clock face in much the same manner as ob-
served for the standard clock drawing task given visual neglect patients.
Their results suggest that there may be a common underpinning for spatial
representation and language in right superior parietal cortical areas. Yet
more evidence of parietal involvement in spatial language can be found in
Coslett and Lie (2000), who have reported on parietal patients who exhibit
spoken language and comprehension deficits correlated with the spatial
orientation of their body (see also Chatterjee 2001). In sum, while there are
not yet any definitive studies showing that linguistic stimuli can drive dis-
tinct regions of the parietal cortex thought to be responsible for maintain-
ing viewer-centered, object-centered and geo-centered frame of reference
maps, studies of both parietal involvement in language and the parietal
processing of different spatial frames of reference provide good reason to
investigate the possibility. Although the mapping between the frames of
reference observed in linguistic typologies (Majid et al. 2004) and the
frames of reference used in neural processing is not likely to be isomor-
phic, experiments bridging the two lines of investigation could prove fruit-
ful.
368 Tim Rohrer
One could also ask about the role of the parietal regions in understand-
ing metaphoric expressions of spatial relations. If linguistic stimuli can
drive the brain regions demonstrably involved in tasks involving the differ-
ent spatial frames of reference, then would metaphoric linguistic stimuli
drive those same brain areas given in tasks involving a temporal frame of
reference? Similar to the Aymara gestural results, other studies of how
temporal reasoning and spatial reasoning interact (see Matlock, Ramscar
and Boroditsky 2005) have shown that temporal problem-solving uses the
resources of spatial problem-solving. One could design linguistic compre-
hension and linguistic problem-solving tasks using the TIME Is SPACE
metaphor that might differentially activate those parietal areas.
It is also important to see that the dialogue between the levels of inves-
tigation in this theoretic framework is not a one-way street. Similar ques-
tions can also be taken to the cultural and communicative level from the
neurophysiological level. For example, and with respect to multimodal user
interface design, as we know that altering the viewer-centered versus the
object-centered perspective influences which neural system performs
mental rotations (Kosslyn et al. 2001), can we take advantage of that fact to
design better hand controls and visual displays for real-world tasks that
involve real-time mental rotation, such as airplane piloting or air traffic
control? The appropriate display and control interface should prompt for
the appropriate frame of reference. How might such controls help mitigate
errors in frames of reference problems that result from the finding that
people find it more difficult to rotate object-centered frames than ego-
centered frames (Wraga, Creem and Proffitt 1999)? If the speakers of some
languages prefer to use a certain frame of reference, how does that alter
how one might teach technical skills (such as navigation) that involve
mental rotations in a cross-cultural setting? The list of such questions is
endless, for the details of embodiment play an enormous role in every cog-
nitive activity.
5. Conclusions
Human beings have bodies, and those bodies shape and constrain how we
think. At the outset of this chapter, I asked the reader to imagine how an
event as mundane as a power outage can reawaken an awareness of the
body. However, and as I have argued throughout this chapter, the body is
not some other thing to which the mind returns when thinking is inter-
rupted - thinking itself is shot through and through with the body. In the
words of the early American Pragmatist philosopher-psychologist William
James:
[... ] our own bodily position, attitude, condition, is one of the things of
which some awareness, however inattentive, invariably accompanies the
knowledge of whatever else we know. We think; and as we think we feel our
bodily selves as the seat of the thinking. If the thinking be our thinking, it
must be suffused through all of its parts with that peculiar warmth and inti-
macy that make it come as ours. (James 1900, 1: 241-242, emphases origi-
nal)
When we think about rotating a hand in space, we feel that it is more diffi-
cult to rotate in ways which conflict with the natural motions of our joints;
when we have pain in the appropriate joints, we rotate it slower. As James
notes, we often may be inattentive to this sort of awareness, especially
when we are physically fit and habitually lost in thought; at such times we
may need experiments to reacquaint ourselves with the body. But for those
of us who feel pain in their wrists as they imagine gripping and rotating a
hammer, no argumentative reintroduction is necessary - for them, the
awareness that our own thinking is embodied is inescapable.
James' felt sense of the experienced body, or what I have called the
phenomenological dimension of embodiment, is only one prominent di-
mension among the many dimensions of the term discussed in this article.
Over the first fifty or so years of cognitive science the field deliberately
theorized away the many contributions of the body to thinking, a theoreti-
cal failing that is only now beginning to be corrected. Interdisciplinary
research, ranging from linguistics and neuroscience to philosophy and psy-
chology, using methods ranging from the experimental and physiological to
the phenomenological and sociological, has begun to show that there are
multiple dimensions along which the human body shapes human thought.
For example, in classical cognitive science our mental representations were
assumed to be logical and symbolic; however, cognitive neuroscience has
shown them to be embodied and image-like. Like the Pragmatist philoso-
phers, the new approaches see human cognition as action situated within a
practical context, and mental representation as instrumental rather than
absolute. Our ordinary experience of space is not one in which we seek to
discover some absolute external frame of reference, but one in which we
are obsessed with coordinating multiple frames of reference - body-, head-,
or other-centered - in order to solve the practical problem at hand. That
practical problem may be getting directions to an unfamiliar place, helping
370 Tim Rohrer
a patient recover from and cope with visual neglect, a team of navigators
piloting a ship, file management within a computer's graphical user inter-
face, or tracking a lost child through the forest; but all of these examples
share a common thread of understanding how the body establishes frames
of reference and moves in space. As the neuroscientist Antonio Damasio
puts it:
The body, as represented in the brain, may constitute the indispensable
frame of reference for neural processes that we experience as the mind; that
our very organism rather than some absolute external reality is used as the
ground reference for the constructions we make of the world around us and
for the construction of the ever-present sense of subjectivity that is part and
parcel of experiences; that our most refmed thoughts and best actions, our
greatest joys and deepest sorrows, use the body as a yardstick. (Damasio
1994: xvi)
Embodied cognitive science begins with the realization that the body, along
all of the dimensions I have outlined in this chapter, grounds and shapes
human cognition.
Wakened from its long dormant slumber, the body has returned to cog-
nitive science. In this survey chapter I have traced how our physiological,
neurophysiological, interactional and sociocultural embodiment impinges
on how we think. The embodied mind is not something which should be
narrowly identified with anyone of these levels of investigation, nor with
anyone of the dimensions of variability that I have noted. The embodied
mind cannot be reduced only to the brain any more than it can be reduced
to culture. Nor is the embodied mind merely a computer, in any traditional
sense of that term; it can be said to perform "computations", but the sub-
stance and structure of these computations are imagistic due to the par-
ticular kind of bodies we have and the environments we inhabit. We are
just at the beginning phase of understanding the myriad ways in which the
body is in the mind.
References
Amorim, Michel-Ange, Brice Isableu and Mohammed Jarraya
2006 Embodied spatial transformations: "Body analogy" for the mental
rotation. Journal of Experimental Psychology: General 135: 247-
327.
Anderson, Michael L.
130.
Behrmann, Marlene and Steven P. Tipper
1999 Attention accesses multiple reference frames: Evidence from visual
neglect. Journal ofExperimental Psychology 25: 83-101.
Bertram, John E. A. and Andrew Ruina
2001 Multiple walking speed-frequency relations are predicted by con-
strained optimization. Journal ofTheoretical Biology 209: 445-453.
Bowerman, Melissa and Soonja Choi
2003 Space under construction: Language-specific spatial categorization in
fITst language acquisition. In: Dedre Gentner and Susan Goldin-
Meadow (eds.), Language and Mind, 387-427. Cambridge, MA:
MIT Press.
Brandt, Per Aage
1999 Domains and the grounding of meaning. In: Jose Luis Cifuentes
Honrubia (ed.), Estudios de Lingiiistica Cognitiva, 467-478.
Alicante: Department de Filologia Espaiiola, Lingiiistica General y
Teoria de la Literatura, Universidad de Alicante.
2000 Metaphor, catachresis, simile: A cognitive and semiotic approach and
an ontological perspective. Ms., Aarhus, Denmark: Center for Semi-
otic Research, University of Aarhus.
Brooks, Rodney A. and Lynn A. Stein
1994 Building brains for bodies. Autonomous Robots 1 (1): 7-25.
Brugman, Claudia
1985 The use of body-part terms as locatives in Chalcatongo Mixtec. In:
Report No. 4 of the Survey of Californian and Other Indian Lan-
guages, 235-290. Berkeley: University of California.
Carlson-Radvansky, Laura A. and David E. Irwin
1993 Frames of reference in vision and language: Where is above? Cogni-
tion 46: 223-244.
Carpenter, Patricia A., Marcel Adam Just, Timothy A. Keller, William F. Eddy and
Keith R. Thulbom.
1999 Time course of fMRI activation in language and spatial networks
during sentence comprehension. Neuroimage 10: 216-224.
Chatterjee, Anjan
2001 Language and space: Some interactions. Trends in Cognitive Sci-
ences 5: 55-61.
Chrisley, Ronald and Tom Ziemke
2002 Embodiment. Encyclopedia of Cognitive Science, 1102-1108. Mac-
millan Publishers.
372 Tim Rohrer
ChurcWand, Paul
1981 Eliminative materialism and the propositional attitudes. Journal of
Philosophy 78: 67-90.
1989 A Neurocomputational Perspective: The Nature of Mind and the
Structure ofScience. Cambridge, MA: MIT Press.
Collins, Steven H., Martijn Wisse and Andrew Ruina
2001 A 3-D passive-dynamic walking robot with two legs and knees. In-
ternational Journal ofRobotics Research 20: 607-615.
Coslett, H. Branch and Eunhui Lie
2000 Spatial Influences on Language Performance. J. Cognitive Neurosci-
ence (Suppl.): 76.
Coulson, Seana and Cynthia Van Petten
2002 Conceptual integration and metaphor: An event-related potential
study. Memory and Cognition 30: 958-968.
Cummins, Robert
1977 Programs in the explanation of behavior. Philosophy of Science 44:
269-287.
Damasio, Antonio R.
1994 The Feeling of What Happens: Emotion, Reason and the Human
Brain. Cambridge, MA: MIT Press.
Damasio, Hanna, Thomas J. Grabowski, Daniel Tranel, Laura L. B. Ponto, Richard
D. Hichwa and Antonio R. Damasio
2001 Neural correlates of naming actions and of naming spatial relations.
NeuroImage 13: 1053-1064.
Deacon, Terrence
1997 The Symbolic Species: The Co-evolution ofLanguage and the Brain.
New York: W. W. Norton.
Donald, Merlin
1991 Origin ofthe Modern Mind: Three Stages in the Evolution of Culture
and Cognition. Cambridge, MA: Harvard University Press.
Edelman, Gerald M.
1992 Bright Air, Brilliant Fire: On the Matter ofMind. New York: Basic
Books.
Eden, Guinevere F., Frank B. Wood and John F. Stein
2003 Clock drawing in developmental dyslexia. Journal of Learning Dis-
ability 36: 216-228.
Elman, Jeffrey L., Elizabeth A. Bates, Mark H. Johnson, Annette Karmiloff-Smith,
Dominico Parisi and Kim Plunkett
1996 Rethinking Innateness. A Connectionist Perspective on Development.
Emmorey, Karen, Hanna Damasio, Stephen McCullough, Thomas J. Grabowski,
Laura L. B. Ponto, Richard D. Hichwa and Ursula Bellugi
2005 Neural systems underlying spatial language in American Sign Lan-
guage. Neuroimage 17: 812-824.
Fauconnier, Gilles and Mark Turner
1995 Conceptual integration and formal expression. Metaphor and Sym-
bolic Activity 10: 183-204.
2002 The Way We Think. New York: Basic Books.
Feldman, Jerome and Srini Narayanan
2004 Embodied meaning in a neural theory of language. Brain and Lan-
guage 89: 385-392.
Femandez-Duque, Diego and Mark L. Johnson
1999 Attention metaphors: How metaphors guide the cognitive psychology
of attention. Cognitive Science 23: 83-116.
Gallagher, Shaun
this vo!. Phenomenological and experimental contributions to understanding
Gallese, Vittorio, Luciano Fadiga, Leonardo Fogassi and Giacomo Rizzolatti
1996 Action recognition in the premotor cortex. Brain 119: 593-609.
Geeraerts, Dirk
1985 Paradigm and Paradox: Explorations into a Paradigmatic Theory of
Meaning and its Epistemological Background. Leuven, Belgium:
Leuven University Press.
Halligan, Peter, Gereon R. Fink, John C. Marshall and Giuseppe Vallar
2003 Spatial cognition: Evidence from visual neglect. Trends in Cognitive
Sciences 7: 55-61.
Hamad, Steven
1990 The symbol grounding problem. Physica D 42: 335-346.
Hodges, Andrew
1983 Alan Turing: The Enigma. New York: Walker and Co.
Howard, Harry
2001 Age/gender morphemes inherit the biases of their underlying dimen-
sions. In: Esra Sandikcioglu and Rene Dirven (eds.), Language and
Ideology, 165-196. AmsterdamlNew York: John Benjamins.
Hutchins, Edwin
1999 Blending and material anchors. Ms. San Diego, CA: Cognitive Sci-
ence Department, University of California at San Diego.
James, William
1900 Psychology (American Science Series, Briefer Course). New York:
Henry Holt and Co.
2001
1998
Kita, Sotaro
2003
374 Tim Rohrer
Jensen de Lopez, Kristine
2002 Language-specific patterns in Danish and Zapotec children's com-
prehension of spatial grams. In: Eve Clark (ed.), The Proceedings of
the 31st Stanford Child Language Forum: Space in Language Loca-
tion, Motion, Path and Manner, 50-59. Stanford University: Center
for the Study of Language and Information.
Jensen de Lopez, Kristine and Chris Sinha
1998 Corn stomach basket. Spatial language and cognitive development in
Danish- and Zapotec-acquiring children. Paper presented at the Child
Language Seminar, Sheffield University England, September 1998.
Johnson, Mark L.
1987 The Body in the Mind: The Bodily Basis of Meaning, Imagination
and Reason. Chicago: University of Chicago Press.
Johnson, Mark L. and TimRohrer
this vol. We are live creatures: Embodiment, American Pragmatism and the
cognitive organism.
Interplay of gaze, hand, torso orientation and language in pointing.
In: Sotaro Kita (ed.), Pointing: Where Language, Cognition and
Culture Meet, 307-328. Mahwah, NJ: Lawrence Earlbaum.
Kosslyn, Stephen M.
1994 Image and Brain: The Resolution of the Imagery Debate. Cam-
Kosslyn, Stephen M., William L. Thompson, Irene J. Kim and Nathaniel M. Alpert
1995 Topographic representations of mental images in primary visual
cortex. Nature 378: 496-498.
Kosslyn, Stephen M., Gregory J. DiGirolamo, William L. Thompson and Nathaniel
M. Alpert
Mental rotation of objects versus hands: Neural mechanisms revealed
by positron emission tomography. Psychophysiology 35: 151-161.
Kosslyn, Stephen M., William L. Thompson, Mary J. Wraga, and Nathaniel M.
Alpert
Imagining rotation by endogenous versus exogenous forces: Distinct
neural mechanisms. NeuroReport 12: 2519-2525.
Kosslyn, Stephen M., Giorgio Ganis and William L. Thompson
2002 Neural foundations of imagery. Nature Reviews Neuroscience 2:
635-642.
Lakoff, George
1987 Women, Fire and Dangerous Things. Chicago: University of Chi-
cago Press.
Lakoff, George and Mark L. Johnson
1999 Philosophy in the Flesh: The Embodied Mind and Its Challenge to
Lakoff, George and Mark Turner
1990 More than Cool Reason: A Field Guide to Poetic Metaphor. Chi-
cago: University of Chicago Press.
Langacker, Ronald
1990 Concept, Image and Symbol: The Cognitive Basis of Grammar. New
York, NY: Mouton de Gruyter.
Levinson, Stephen C.
2003 Space in Language and Cognition: Explorations in Cognitive Diver-
sity. Cambridge: Cambridge University Press.
MacLaury, Robert
1989 Zapotec body-part locatives: Prototypes and metaphoric extensions.
International Journal ofAmerican Linguistics 55: 119-154.
MacWhinney, Brian
1999 The emergence of language from embodiment. In: Brian MacWhin-
ney (ed.), The Emergence ofLanguage, 213-256. Mahway, NI: Law-
rence Earlbaum.
2003 The emergence of grammar from perspective-taking. Ms., Pittsburgh,
PA: Camegie Mellon University.
Majid, Asifa, Melissa Bowerman, Sotaro Kita, Daniel B. M. Haun and Stephen C.
Levinson
2004 Can language restructure cognition? The case for space. Trends in
Matlock, Teenie, Michael Ramscar and Lera Boroditsky
2005 The experiential link between spatial and temporal language. Cogni-
tive Science 29: 655-664.
Miisseler, Iochen
1994 Position-dependent and position-independent attention shifts: Evi-
dence against the spotlight and premotor assumption of visual fo-
cusing. Psychological Research 56: 251-260.
Nagel, Thomas
1979 Mortal Questions. New York: Cambridge University Press.
NUfiez, Rafael and Eve Sweetser
2006 In Aymara, Next Week Is Behind You: Convergent Evidence from
Language and Gesture in the Crosslinguistic Comparison of Spatial
Construals of Time. Cognitive Science 30: 1-49.
Parsons, Lawrence M.
1987 a Imagined spatial transformations of one's hands and feet. Cognitive
Psychology 19: 178-241.
1987 b Imagined spatial transformation of one's body. Journal of Experi-
mental Psychology 116: 172-191.
376 Tim Rohrer
1994 Temporal and kinematic properties of motor behavior reflected in
mentally simulated action. Journal of Experimental Psychology:
Human Perception and Performance 20: 709-730.
2003 Superior parietal cortices and varieties of mental rotation. Trends in
Cognitive Science 7: 515-551.
Parsons, Lawrence M., Peter T. Fox, J. Hunter Downs, Thomas Glass, Traci B.
Hirsch, Charles C. Martin, Paul A. Jerabek and Jack L. Lancaster
1995 Use of implicit motor imagery for visual shape discrimination as
revealed by PET Nature 375: 54-58.
Pederson, Eric, Eve Danzinger, David Wilkins, Stephen Levinson, Sotaro Kita and
Gunter Senft
1998 Semantic typology and spatial conceptualization. Language 74: 557-
589.
Posner, Michael and Marcus Raichle
1994 Images ofMind. New York: Scientific American.
Pylyshyn, Zenon W.
1973 What the mind's eye tells the mind's brain: a critique of mental im-
agery. Psychological Bulletin 80: 1-24.
Regier, Terry
1992 The acquisition of lexical semantics for spatial terms: A
connectionist model of perceptual categorization. PhD. Dissertation,
University of California at Berkeley.
1995 The Human Semantic Potential. Chicago: University of Chicago
Press.
Rizzolatti, Giacomo and Laila Craighero
2004 The mirror neuron system. Annual Review ofNeuroscience 27: 169-
192.
Rohrer, Tim
1998
2001 a
2001 b
When metaphors bewitch, analogies illustrate and logic fails: Con-
troversies over the use of metaphoric reasoning in philosophy and
science. PhD. dissertation, University of Oregon.
philosophical foundations of cognitive linguistics. In: Rene Dirven,
ogy: Cognitive Theoretic Approaches. Volume 1, 49-81. Amsterdam:
John Benjamins.
Understanding through the Body: fMRI and of ERP studies of meta-
phoric and literal language. Paper presented at the 7th International
Cognitive Linguistics Association Conference, July 2001.
2005 Image schemata in the brain. In: Beate Hampe (ed), From Perception
to Meaning: Image Schemas in Cognitive Linguistics, 165-196. Ber-
lin: Mouton de Gruyter.
Schwoebel, John, Robert Friedman, Nanci Duda and H. Branch Coslett
2001 Pain and the body schema evidence for peripheral effects on mental
representations of movement. Brain 124: 2098-2104.
Shepard, Roger N. and Jacqueline Metzler
1971 Mental rotation of three-dimensional objects. Science 171: 701-703.
Shulman, Gordon L., Roger W. Remington and John P. McLean
1979 Moving Attention Through Visual Space. Journal of Experimental
Psychology: Human Perception and Performance 5: 522-526.
Sinha, Chris and Kristine Jensen de Lopez
2000 Culture and the embodiment of spatial cognition. Cognitive Linguis-
tics 11: 17-41.
Tsal, Yehoshua
1983 Movements of attention across the visual field. Journal of Experi-
mental Psychology: Human Perception and Performance 9: 523-
530.
Turing, Alan M.
1950 Computing machinery and intelligence. Mind 59: 433-460.
Vingerhoets, Guy, Floris P. de Lange, Pieter Vandamaele, Karel Deblaere and Erik
Achten
2002 Motor Imagery in Mental Rotation: An fMRI study. Neuroimage 17:
1623-1633.
Wilson, Margaret
2002 Six views of embodied cognition. The Psychological Bulletin and
Review 9: 625-636.
Wraga, Maryjane, Sarah H. Creem, and Dennis R. Proffitt
1999 The influence of spatial reference frames on imagined object- and
viewer rotations. Acta Psych0 logica 102: 247-264.
Zlatev, Jordan
1997 . Situated Embodiment: Studies in the Emergence ofSpatial Meaning.
2003 Polysemy or generality? Mu. In: Hubert Cuyckens, Rene Dirven and
John Taylor (eds.), Cognitive approaches to lexical semantics, 447-
494. Berlin: Mouton de Gruyter.
On the biosemiotics of embodiment and our human
*
cyborg nature
Claus Emmeche
"my faculty of discussion is equally localized in my
inkstand" Peirce
1
Abstract
This chapter briefly introduces biosemiotics, a new perspective on living systems
based upon standard contemporary biology reinterpreted through a qualitative or-
ganicist tradition in biology inspired by Jakob von Uexkiill and Charles S. Peirce.
To be emphasized is the difference between a living organism as a general semiotic
system with vegetative and self-reproductive capacities ("the body of biology") and
an animal body also with sentience and phenomenal states. The "representational-
ism" invoked by critiques of cognitive science tends to focus only on simplistic
notions of representations. However, these must be distinguished from a Peircean
notion of representation as an embedded ongoing process. The implications of this
distinction for theorizing about the physical, biological, animate, phenomenal and
social body will be made. It will be argued that these very dimensions of embodi-
ment must be accounted for coherently and seen within an evolutionary emergentist
perspective. Such a perspective requires a "Non-standard Neo-Aristotelian Pluralist
notion of Causation" (NNPC), and I will argue that biosemiotics offers an approach
to animate embodiment compatible with NNPC. However, biosemiotics is not
enough to account for the characteristics of human embodiment because of what
shall be called our intrinsic cyborg nature as techno-culturally embedded beings
within a space of meanings that are not only symbolic, but argumentative and so-
cially empowered by different kinds of sociocultural systems.
* The author is indebted to Frederik Stjernfelt, Simo Keppe, Jesper Hoffmeyer,
Charbel Niiio EI-Hani and Joao Queiroz for valuable discussions, and to Tom
Ziemke and an anonymous reviewer for critical comments on earlier versions of
this paper.
1. Peirce ([1931-1958] 1902, CP 7.366)
380 Claus Emmeche
Keywords: animal, biosemiotics, causality, cyborg, embodiment, emergence,
organicism, representation, society.
1. Introduction
The desire we have as persons for knowledge about the body is often redi-
rected to science. If the body is a living organism, the science of the body
should be biology, as one might envision this "science of life" as being
able to account for the nature of human bodies and the embodiment of
human existence. It was only some time after having started training as a
biologist that it became clear to me that biology cannot really be a science
of living bodies. It is a science of organisms and their parts and relations
within systems of organisms and environments, and these two domains of
inquiry, living bodies and organisms, are not co-extensive. Intuitively I had
held the idea that biology because of its generality simply subsumed hu-
mans as a species, just as physics subsumed biological processes in the
sense that any biological theory of these processes has to conform to physi-
cal laws. Embodiment had to be seen as a biological phenomenon, I
thought. This stance was naIve for two reasons, an epistemic one and an
ontological one. The epistemic one is that each science in an important
sense actively constitutes is own paradigmatic objects of investigation, and
thus body and embodiment in biology is different from body and embodi-
ment in anthropology. In fact, embodiment is usually neither considered a
technical term nor a theoretical issue within biology. The ontological rea-
son why embodiment (how to be a body, have a body, use and express
one's body2) is not simply a biological phenomenon is that the human body
in every specific concrete context is a far more complex and multi-Ievel
3
phenomenon than any single scientific perspective can account for.
2. Often expressions in everyday language exemplifying the non-identity between
the notions of self, person and body (like "I moved my body"), appear as if pre-
supposing a metaphysical mind/body dualism, leading to questions like "Do you
use your body to express yourself, or is your body and your self the same ex-
pressive entity?". As Wittgenstein remarked, we are often bewitched by lan-
guage.
3. As we shall see later, "multi-level" means including at least a physical, a general
biological-organismic, a specific animate-zoological, and a human-sociocultural
level. There is a one-way relation of inclusion between the levels (cf. the notion
of a "specification hierarchy" in Salthe 1989). The biological level includes the
On the biosemiotics ofembodiment and our human cyborg nature 381
To understand the full implications of embodiment, one has to grasp at
least the distinction between the concept of an organism and the concept of
a body. This distinction matches the distinction between biology and soci-
ology (roughly speaking, leaving out animals for a moment). Of course,
being an organism is a precondition for having a living body which is nei-
ther a thing not a corpse. We could not, as humans, be embodied - that is,
be material creatures co-evolving with an existential-phenomenal world of
situated activity involving emotional experiences, feelings, cognitive proc-
esses, perception and action, a specific perspective, placed in historical and
biographic time regimens - if we were not rooted in an organic world by
our very being, that is, if we were not biological creatures.
4
Yet in signifi-
cant ways we transcend our biological form of existence by producing,
through culture, language and social institutions, specific dynamic modes
or patterns in which our organic and animate existence is realized. Here we
shall try to bring together some perspectives upon the body that may help
increase theoretical sensitivity to our (minimally double) existence as be-
longing both to a first biological nature bound by animate and metabolic
processes, and a second social and semioticS nature in which the signifi-
cance of the organism's forms of movement can only be comprehended if
zoological - being an animal implies being an organism- but not the other way
around. However, even though humans are animals, zoology as a science is not
adequate and too unspecific to acount for all there is to say about the cultural
life of Homo sapiens, a being socialized in a human society.
4. Some AI researchers, cognitive psychologists and philosophers might consider
that bio-chauvinismif the intended implication is the impossibility of "intrinsic"
machine intentionality. However, machines should not be reified as something
outside a human sociocultural context. Machine intentionality might be possi-
ble, but the action of signs and interpretation (i.e., semiosis) in machines must
be considered as tightly bound to their sociocultural context. Machines are ex-
tensions of human capacities and intrinsic parts of human sociocultural systems;
they do not originate ex nihilo or fall readymade from heaven (Marx 1867;
Ziemke and Sharkey 2001).
5. "Social" can be said to mean concerning human actions within a community (as
studied by sociology, the science of social processes). "Semiotic" is here taken
to mean concerning sign production, information transmission, communication
and interpretation within some system (as studied by semiotics, the general sci-
ence of sign action and interporetation, as founded by the philosopher and sci-
entist C. S. Peirce). The word nature has several meanings but here I rely on my
reader as a competent language user.
382 Claus Emmeche
seen as formed by the norms, social roles and institutions of a human soci-
ety.
However, we shall circumvent the all too common dualism of seeing
these modes as exclusive, or seeing, as often expressed within the humani-
ties, meaning and significance as an exclusively human social phenome-
non, implying non-human nature to be either merely an abstraction or a
boundary concept denoting forms of materiality devoid of any intrinsic
meaning or significance. First, the modes of existence of embodiment are
inclusive; briefly, the human body includes the biological organism.
6
Sec-
ondly, the organism as our "first" biological nature is not something non-
semiotic. On the contrary, the roots of processes involving communication,
information, signs and interpretation (in one word: semiosis) - and more
specifically, human forms of sign action and interpretation (anthro-
posemiosis) - must be found within an evolutionary framework. This
should not, however, simply be standard neo-Darwinism (as this paradigm
during the history of twentieth century biology unfortunately has become
dominated by a physicalist metaphysics), but supplemented with a bio-
semiotic perspective. In Section 2 I shall briefly introduce biosemiotics as
one useful resource to sketch a framework for an overall understanding of
embodiment in its material, processual, externalist as well as internalist
aspects.
A brief comment on the current interest in the phenomena of embodi-
ment may be adequate to discern the specific message ofbiosemiotics. The
concern with embodiment has several sources: (1) the study of human cog-
nition, especially criticisms of the classical functionalism of traditional
cognitive science and AI, alternatives like "enactive cognitive science"
(Varela, Thompson and Rosch 1991) or ideas of "situated and embodied
robots" Brook 2002), which have led to a radical questioning of the need
for representations in cognitive science (Riegler, Peschl and Stein 1999);
(2) the study of human language, especially critiques of modeling language
as a disembodied abstract module, and the alternative movement in cogni-
tive semantics that sees basic metaphoric forms of language as rooted in
the organization of human bodily action, and claims these unconscious
metaphors are based on common bodily experiences (Lakoff and Johnson
1999); and (3), a widespread interest, within the human sciences and inter-
disciplinary cultural studies, in the transformation of the body, in which
6. See also note 3. This notion ofinclusivity (see below) was developed in relation
to emergent levels of reality in Emmeche, Keppe and Stjernfelt (1997).
"the new body" is seen no longer as a "fixed, material entity subject to the
empirical rules of biological science" (Csordas 1996). Instead, the "new
body" is seen as a body with a history, behaving "in new ways at particular
historical moments" (ibid.), characterized by indeterminacy, flux, self-
creating processes. This has lead to a deep questioning of a set of suppos-
edly basic boundaries, namely the boundaries of corporeality itself, and the
boundaries between physical and non-physical, animal and human, and
between organic bodies and machines (cf. Hayles 1999).
Let me state preliminarily and in simple terms what biosemiotics is
about: It is an attempt to study life not only by chemical approaches seeing
cells and organisms as assemblies of molecules, but from the perspective of
semiotics (cf. footnote 5) seeing those same molecules (correctly described
by chemistry and molecular biology) as sign vehicles for information and
interpretation processes.
One main concern of biosemiotics is thus to see the phenomena of sig-
nificance, meaning and interpretation in the human sphere as rooted in and
to some extent continuous with the same very general kind of phenomena
in the non-human sphere. We can take the words of the Peirce scholar T. L.
Short (1982: 298) to be emblematic of this evolutionary yet non-
reductionist perspective ofbiosemiotics:
Of course human speech is unique: it consists of the ability to replicate leg-
isigns in ever new patterns, as well as in the ability to create new legisigns,
not by the slow process of evolution, but within the lifetime of individuals,
and by their own volition. The point, rather, is that the distinctive power of
human speech is not a supernatural gift, but is a remarkable development of
basic principles found elsewhere in nature."
From a biosemiotic point of view, more than one approach to embodiment
is needed for the above intellectual movements (1) and (2) may lead too
hastily to radical anti-representationalist stances, while (3), within the
posthuman or postmodern discourse, they may tend simply to dissolve real
boundaries between the categories of body/machine, human/animal, or
nature/culture, instead of analyzing their dynamics, inclusivity relations, or
how they are being contested. Biosemiotics is critical about these implica-
tions; it builds on a more adequate notion of representation (namely that of
Peirce; see below) than classical cognitive science; and it allows for keep-
ing fundamental distinctions while seeing these in a dynamic evolutionary
perspective which involves a modest dialectics of continuity and emer-
gence. Yet I will also claim that as a scientific perspective, biosemiotics
has a limited area of validity, corresponding more or less to the subject
384 Claus Emmeche
matter of contemporary biology, and it cannot fully capture the specificities
of human embodiment.
The idea to be presented is in some respects a very simple one of emer-
gent levels of embodiment: A human body - e.g., the body of a child, a
soccer player, or a diplomat - includes (in the sense specified in footnote
3) an organism, but is also something more, transcending the mere set of
organismic properties (like metabolism, growth, homeostasis, reproduc-
tion), just as, if we go down one step to a lower level of integration, an
organism is a physical system, yet it transcends the basic physics of that
system. "Transcending" here does not imply some transcendentalism or
metaphysical dualism. The expression "Z transcends Y" has two aspects.
(a) One aspect is epistemic, i.e., "Z's description cannot adequately be
given in terms of a theory generally accounting for Y, even though this Z-
description in no way contradicts a description of the Y-aspects of Z". (b)
The other aspect is ontological, i.e., "crucial properties and processes of Z
are of a completely different category than the ones of Y, even though they
may presuppose and depend on Y".
7
The organism is a material, physical
processual entity with a form of movement so specific that physics alone
(as a science) cannot account for that entity. The organism is a very special
type of physical being, as it includes certain purposeful (functional) part-
whole relations, based upon genuine sign systems of which the genetic
code is the most well-known but not the only example.
It is time to insert a one-paragraph crash-course in semiotics (cf. Peirce
[1931-1958] 1902): A sign is anything that can stand for something (an
object) to some interpreting system (e.g., a cell, an animal, a legal court),
where "standing for" means "mediating a significant effect" (called the
interpretant) upon that system. Thus, semiosis, or sign action, always in-
volves an irreducibly triadic process between sign, object and interpretant.
Just as in chemistry we see the world from the perspective of molecules, in
semiotics (as a general logic of sign action) we see the world from the per-
spective of sign action, process, mediation, purposefulness, interpretation,
generality. Those are not reducible to a dyadic mode of mechanical action-
reaction, or merely efficient causality.8 Thus, organisms are certainly com-
7. Both emergence and variants of supervenience have been suggested as candi-
dates for an ontological dependence relation, but we need not enter the debate
on these technicalities.
8. The details of this argument are more complex in the sense that the distinction
fmallefficient causation and organismic/physical processes are not co-extensive,
and thus, the irreducibility of fmal (teleologic) causation to efficient causation
posed of molecules, but these should be seen as sign vehicles having func-
tional roles in mediating sign action of, e.g., genotype and environment,
upon the phenotype.
Biological organization is emergent upon physical order, i.e., the or-
ganism includes its physical processes (for instance, one can break a leg)
but from the point of view of physics their processual forms are radically
new, unpredictable and realizing processes (e.g., self-reproduction) that are
not strictly deducible from properties found in the non-biological realm.
However, discussing embodiment, one cannot limit oneself to the gross
primary levels of reality including the physical, the biological and the so-
cial level of embodiment, one has to do a more fine-grained analysis of
various biological forms of organismic existence before we can make sense
of a distinction between biological and human embodiment. To this end
biosemiotics is valuable although it cannot alone account for human em-
bodiment.
2. Biosemiotics - a qualitative organicist account of embodiment
Biosemiotics is a growing field that studies the production, action and in-
terpretation of very different types of signs (such as sounds, objects,
smells, movements, but also signs on molecular scales normally not per-
ceived by an organism) in the physical and biological realm, in an attempt
to integrate the findings of biology and semiotics. One goal ofbiosemiotics
is to form a new view of life and meaning as immanent features of the
natural world, rather than as epiphenomena, however complex they might
be. Early pioneers ofbiosemiotics include Charles S. Peirce (1839-1914),
Jakob von Uexkiill (1864-1944), Charles Morris (1901-1979), Thure von
Uexkiill (b. 1908), Heini Hediger (1908-1992), Thomas A. Sebeok (1920-
2001), and Giorgio Prodi (1928-1987). Contemporary scholars who have
contributed to the field or commented extensively include, among others,
biologists Jesper Hoffmeyer, Kalevi Kull, Alexei Sharov, Anton Markos,
Marcello Barbieri and Seren Brier, and semioticians Walther A. Koch,
Floyd Merrell, John Deely, Frederik Stjemfelt, Winfried N6th and Lucia
Santaella (for a historical introduction, see KuII1999).
(sensu Peirce) is not necessarily the same as the irreducibility of organismic
processes to physical processes, as for Peirce, every lawful process has an ele-
ment offmality (see Hulswitt 2002).
386 Claus Emmeche
One of the central characteristics of living systems is the highly organ-
ized nature of their physical and chemical processes. These processes are
based, in part, on the informational and molecular properties of what came
to be known in the 1960s as the genetic code. Some biologists, such as
Emst Mayr (1982), have viewed these program-like properties of the ge-
netic code as distinguishing life from anything else in the physical world,
except computers. However, although the informational teleology (or goal-
directedness based upon a stored informational code) of a computer pro-
gram is not an original form of teleology because the program is designed
by humans to achieve specific goals, the teleology and informational char-
acteristics of organisms are intrinsic, as they evolved naturally through
self-organized evolutionary processes. Traditional biology, at least in
popular and text book versions, and a large part of mainstream philosophy
of biology have regarded such informational processes as in the end purely
physical; unfortunately also adopting a restricted notion of the physical as
having to do with only efficient causation. Thus these traditions have often
aimed at a "naturalization" of teleology (of, e.g., genetic information) and
intentionality (of cognitive information). The admirable purpose of this is
making these phenomena continuous with the rest of nature, and to account
for them in a manner acceptable to physical science. Contemporary sci-
ence, however, is often prejudiced as the only legitimate basis of judg-
ments on the nature of Nature. Taking the animal or human mind and its
teleological and intentional processes to be embodied is often perceived,
within biology and cognitive science, as implying a search for reductive
explanations of mental phenomena in terms of, e.g., neural information
processing, or dynamical systems approaches continuous with the kind of
explanations given for non-linear complex systems within physics. This is
the case even for some of the search for other notions of embodiment (e.g.,
"radical embodiment", Clark 1999). In many programmatic naturalization
strategies, the "natural" seems to be reduced beforehand to restricted meta-
physical versions of nature as only involving certain kinds of properties or
causal processes, like, as said, classical efficient causation.
9
Drawing upon the insights of Peirce who founded semiotics as a logic
and scientific study of dynamic sign action in human and non-human na-
ture, biosemiotics attempts to use semiotic concepts to investigate ques-
9. Partly based on mathematical arguments Scott (2004) convincingly shows that a
reductionist notion of causality is not even defensible for many non-linear com-
plex systems in physics.
tions about the biological and evolutionary emergence of meaning, inten-
tionality and a psychic
IO
world which are difficult if not impossible to an-
swer within a mechanicist or physicalist framework.
II
Biosemiotics see the
evolution of life and the evolution of semiotic systems as two aspects of
the same process. The main tenet of biosemiotics is the belief that the sci-
entific approach to the origin and evolution of life has given us highly
valuable accounts of the external aspects of these processes, but has over-
looked the "inner", qualitative and significant aspects of sign action, thus
leading to a reduced picture of causality (cf. Santaella Braga 1999).
Complex self-organized living systems are governed by formal and final
causality (Van de Vijver, Salthe and Delpos 1998). They are mereologi-
callyI2 governed by formal causality in the sense of a non-temporal
"downward causation" (Andersen et al. 2000) "from" a whole structure
(such as the organism) "to" its individual molecules, constraining their
action but also endowing them with functional meanings in relation to the
whole metabolism. They are governed by final causality in the sense of
their tendency to take habits (find new attractors in phase space) and to
generate future interpretants of present sign actions. In this sense, biosemi-
otics draws upon insights of fields like systems theory, theoretical biology
and the physics of complex self-organized systems.
I3
Particular scientific fields like molecular biology, cognitive ethology,
cognitive science, robotics and neurobiology deal with information proc-
esses at various levels and thus - in that minimal sense of biosemiosis as
informational processes in living organisms - spontaneously contribute to
knowledge about biosemiosis and living sign action, even though these
10. Allow me to use "psychic" instead of "mental", as the former word has less
Cartesian connotations than the latter and is rooted in a realist, neo-Aristotelian
and Peircean philosophical tradition. Intentionality is used here not according to
Brentano's dualist metaphysics but in a way inspired by Peirce to designate both
the representational "aboutness"-aspect of animate sign processes, and their
qualitative, psychic, phenomenal aspects.
11. This applies equally for non-reductive physicalism within philosophy of mind.
12. Mereology, the study of parts and wholes, usually refers to a mathematical or at
least formal theory thereof, such as that of Lesniewski or Goodman (cf. Varzi
2004), developed by the former in the hope of forming an alternative to set the-
ory as a foundation for mathematics.
13. See Hoffmeyer (1996), Emmeche, Kull and Stjernfelt (2002), Hulswitt (2002),
Emmeche (2004a), EI-Hani, Queiroz and Emmeche (2005). (More on causality,
see also section 5 below).
388 Claus Emmeche
findings are not framed within a theory of biosemiotics. However, bio-
semiotics is not so much a specific disciplinary research programme as a
general scientific perspective on living systems. It attempts to integrate and
re-interpret empirical findings, and to build a new foundation for biology
that acknowledges the genetic "information talk" of codes, messenger
molecules, translation etc., as being not simply problematic metaphors but
a symptom of real organic semiosis, emphasizing that a notion of "infor-
mation processing" can be provided, that is theoretically richer than stan-
dard notions of information processing in contemporary cognitive science
(cf. Lindblom and Ziemke, this volume). What is this alternative notion of
information and representation?
According to biosemiotics, the cognitive system is not a dual system of
hardware (a physical brain) and software (an algorithmic symbolic system),
and similarly the organism is not a dual system of "information and flesh",
with a DNA code as governor of all bodily processes. Sign processes are
active components of the overall physiology of the organism, and accord-
ingly everything which may become "a difference that makes a difference"
to the organism (to use Bateson's cybernetic definition of information;
Bateson 1972) is acting as a sign. Just as the mind is not a computer pro-
gram, DNA is not the central governor of the body, and genes are not pre-
formed descriptions of phenotypic characters with a privileged causal
status in every cell, as often assumed in standard genetics and molecular
biology. It is possible to use Peircean semiotics to analyse sign processes
not only in protein synthesis but in any significant process within and be-
tween cells. Though not identical, the notions of sign action sensu Peirce
and of information sensu Bateson are intersecting and not conflicting, if
interpreted correctly: Both notions are generally denoting processes of
production, transmission and interpretation of something of significance
for some interpreting system (like a body, an organism, a cell), and the
important aspects of both notions are relational and semiotic. If informa-
tion is a difference that makes a difference (implied is "to an organism" or
"to some interpreting system"), then by Peircean terminology this can be
restated more precisely in a semiotic definition of information as a triadic
process we may formulate as follows:
Information is a process in which a sign (representamen), simple or
composite, makes a difference (interpretant) to some system (the inter-
preter) by making the interpretant (the effect of the sign as a difference)
stand in a similar relation to something else (the object of the sign) as that
to which the representamen stands or refers, thus mediating object and
interpretant, i.e., conveying to the interpretant a dynamic form that signi-
fies the object.
14
This definition can be exemplified on the level of both molecular biol-
ogy and cognitive ethology: Fine-grained analysis of genes and protein
synthesis as semiotic processes can be made (EI-Hani, Queiroz and Em-
meche 2005) as well as analyses of, for instance, the alarm system in vervet
monkeys (Queiroz and Ribeiro 2002). Here one has to remember the very
general character of the formal sign definition in Peirce, of which the pro-
posed definition is a variant that emphasizes the processual character. That
the meaning of the sign is its causal effects (i.e., the interpretant) implies
that these effects have to be physically, biologically, or culturally embod-
ied in an interpreting system (or in the dialogic process between such sys-
tems; see Petrilli 1999) to the extent that we are dealing with material sys-
tems of a physical, biological or cultural kind. In philosophical jargon,
meaning is supervenient on the physics of the system, not reducible to
physics, yet dependent upon a material basis.
Furthermore, biosemiotics helps to resolve some remnants of Cartesian
dualism that are still haunting philosophers and scientists. It provides an
alternative to the implicit Cartesianist metaphysics in evolutionary biology
that tend to reduce organisms to non-sentient mechanical beings, a reduc-
tion that makes it almost impossible to conceive of a satisfying evolution-
ary account of the qualitative aspects of consciousness (Emmeche 2004a).
By subscribing to a process rather than substance metaphysics, and by de-
scribing the continuity between matter and mind, biosemiotics may also
help to understand higher embodied forms of mind and the evolutionary
roots of cultural phenomena.
In a historical account of "embodiment", Keller (2003) comments upon
the tendency of science writers and journalists to articulate both a quasi-
religious awe when facing the structure of the genetic code or the human
genome, seeing it as "the word becoming flesh", but also repeating, by
such expressions, a dualist framework that has been operative since the
foundation of modem science: "The body as a machine, as a closed system,
14. Cf. one ofPeirce's own defmitions: "A Sign, or Representamen, is a First which
stands in such a genuine triadic relation to a Second, called its Object, as to be
capable of determining a Third, called its Interpretant, to assume the same tri-
adic relation to its Object in which it stands itself to the same Object." ([1931-
1958] 1902, CP 2.274), emphasis in the original. Peirce distinguished between
several kinds of signs, objects, and interpretants, which may be used to develop
further the informational variant of the sign defmition.
390 Claus Emmeche
to which spirit, mind and God, if they exist, were posited as external
agents, dominated the western imagination until the twentieth century"
(ibid., 255). Even though Keller locates this new "gospel of genetics" in a
culture-historical trend towards more inclusive senses of embodiment (de-
riving from not only the process theology of Alfred North Whitehead, but
also from feminist and ecological movements) and thus sees it as contrib-
uting to re-opening the closed system of the body as a machine, these very
rhetorical figures of popular genetics tend to reinforce the dualisms be-
tween mind and body; information and flesh. Here, biosemiotics may
function as one of the resources for critical study of mind/body dualism in
Western science and religion.
The biosemiotic account of embodiment is a form of organicicm, an im-
portant tradition in biological thought. The resolution of the debate be-
tween vitalism and mechanicism in the history of biology looks on the
surface like a victory for mechanicism, but was really a compromise, a sort
of mainstream organicism, exemplified by the writings of such well-mown
biologists as J. Needham, P. Weiss, C. H. Waddington, J. Woodger, E.
Mayr, R. C. Lewontin, R. Levins and S. J. Gould, (see also Gilbert and
Sarkar 2000), and functioning more or less tacitly as a background phi-
losophy of biology (Emmeche 2001). Organicism takes the complexity and
physical uniqueness of the organism as a symptom of the distinctiveness of
biology as a natural science sui generis. As a compromise, although often
framed within a naturalist evolutionary perspective, it was anticipated by
Kant's more critical (non-naturalist) notion of a living organism as an or-
ganized product of nature in which every part is reciprocally purpose (end) .
and means (cf. Van de Vijver, Van Speybroeck and Vandevyvere 2003).
However, within mainstream organicism this teleology is interpreted as a
more or less "mechanical" teleonomy being the result of the forces of blind
variation and natural selection, plus eventually some additional "order for
free" or physical self-organization (Kauffman 2000). Mainstream organi-
cism is thus non-vitalist, ontologically non-reductionist (methodologically
allowing for reductionist research strategies though) and emergentist. What
is studied as emergent properties are common material structures and proc-
esses within several levels of living systems (developmental systems, evo-
lution, self-organizing properties etc.), all of which are treated in the usual
way as objects with no intrinsic experiential properties.
In contrast, qualitative organicism represents a more colored view on
living beings; it emphasizes not only the ontological reality of biological
higher level properties or entities (such as systems of self-reproducing
organisms being parts of the species' historical lineage) but also the exis-
tence of phenomenological or qualitative aspects of at least some higher
level properties. When sensing light or colors, an organism is not merely
performing a detection of some external signals which then are processed
internally (described in terms of neurochemistry or "information process-
ing"); something additional is part of the story, at least if we want the full
story, namely the organism's own experience of the light, and this experi-
ence is seen as something very real.
15
As a scientific position qualitative
organicism is concerned with qualities which are not only of the famous
category of "primary" qualities (roughly corresponding to the scientifically
measurable quanta) including shape, magnitude and number; but also con-
cerned with the "secondary" qualities of color, taste, sound, feeling, etc.
Neither qualitative nor mainstream organicism are fully coherent stances,
theories or paradigms; many authors cannot be consistently categorized as
belonging to either one or the other position exclusively; the important
thing is to recognize that in fact two different conceptions of life are at
stake. To the extent that biosemiotics is based in a Peircean logic of signs
of varying levels of complexity, it is a form of qualitative organicism. We
leave it to others to form a taxonomy of the variants of qualitative organi-
cism. Here we will concentrate on its biosemiotic brand instead.
Biosemiotics recognizes not only the external effects of signs upon their
interpreters, forming new interpretants (remember that the interpretant,
being an effect of the sign, need not be the whole interpreter organism), but
also the internalist and phenomenal aspects of this process. The phenome-
nology of basic biological ("vegetative") processes of life is not yet devel-
oped, and we shall primarily focus on the animal level of biosemiosis. A
living being that is not simply an organism but also an animal, which is
emphatically "animated", that is, it realizes a specific form of movement
which is not reducible to pure physical change or basic biologic processes
(like metabolism). Movement, understood as living change and as semiotic,
when realized by self-moving organisms (i.e., animals), crucially depends
on mediating all the elements of quality, physical force and interpretation -
three general ontological aspects of every semiotic process that correspond
to the triadic structure of categories in Peirce. Single cells, plants and ani-
mals are all, so to speak, semiotic machines (the machine metaphor is not
15. This is the case in Jakob von Uexkiill's biology (von Uexkiill1982) as well as
in Peirce's synechism, i.e., his philosophy of mind/matter continuity (Santaella
Braga 1999).
392 Claus Emmeche
implying any mechanicism here, cf. Noth 2003), nodes in complex webs of
sign interpretation processes.
Every process,I6 even in non-living nature, has an aspect of (i) "tone"
(i.e., possibility, chance and phenomenal quality), (ii) "token" (existence,
force, here-and-now), and (iii) "type" (generality, being governed by a
general law). This is true for sign processes as well. The characteristic
level of sign interpretation processes in plants are intra- and inter-cellular,
with molecular signs representing the immediate environment and long-
term morphogenetic changes (processes like metabolism, growth, differen-
tiation and reproduction). A plant's body is the result of an organism's
vegetative interpreting systems, and irritability is a crucial component of
vegetative embodiment.
I
? Animate embodiment is different. Although it
includes the vegetative form of embodiment (thus an animal's skin may
indeed get irritated, its growth may be disturbed, etc.), it is a form of em-
bodiment where self-movement is a crucial new development of the gen-
eral biological self-control one finds in plants and single cells. In self-
movement, animal agency emerges as a temporary structure of bodily sign
action processes. Animate agency is semiotic, meaning that it mediates the
three dimensions of (i) a qualitative phenomenal feeling, i.e., the "inner"
side of the sign considered not in relation to anything else (thus, a form of
Firstness in Peirce); (ii) the here-and-nowness of signs or stimuli (the
"haecceity" of a stimulus being an example of Secondness) that can have
an environmental or bodily proprioceptive origin, but whose being is exis-
tent and thus simply consists in their immediate effects, action-reaction,
their "actually acting and being acted on" (Peirce [1931-1958] 1902, CP.
6.318); and (iii) the cybernetically organized cycles of organism-
16. Peirce was not satisfied with a standard type/token distinction which only oper-
ates with Thirdness (type, a form of generality) and Secondness (token, exis-
tence), so he added "tone" (quality) to represent Firstness. It is not possible in
brief space to explain his three basic categories, so I only can offer my reader
these hints (see also Hausman 1993; Hulswitt 2002; Peirce [1931-1958] 1902).
17. Irritability is used here in the sense of a general semiotic capacity to feel and
interpret stimuli so as to adapt to changes in the environment or in other parts of
the organism. Compare the term's original denotation, viz. the capacity of cer-
tain parts of the body to contract when stimulated, as introduced by the English
physician Francis Glisson (circa 1597-1677) who saw it as a property of all the
body's fibres independent of consciousness and the nervous system (cf. Law-
rence 1981). It has played an important role in debate between mechanicists and
vitalists over the basic defInition of life.
environment interaction, the "functional cycle" of Jakob von Uexkiill's
biology, describing how the environmental stimuli are integrated and inter-
preted in the Umwelt
18
of the organism - this integrative mediating process
involves emotional and proprioceptive interpretation systems that mediate
the past and present Umwelt into anticipatory schemes for further action
and habit-formation (i.e. the mediative and future-directed category of
Thirdness). Animal embodiment, accordingly, is the graceful integration,
accomplished by a living animal body, of physical change, biologic-
metabolic processes (neural as well as general physiological) and the
emerging interpretative Umwelt relation (of a body within its specific per-
ceived and acted upon environment) representing the animal's own de-
tailed movements in an ever-changing flux of environmental and bodily
stimuli.
3. Triadic representations vs. representationalism
It is important to note that a biosemiotic notion of animal embodied cogni-
tion is not committed to the same kind of functionalist representationalism
often criticized by "new AI" or cognitive semantics, i.e., the notion that
cognition and language consist of rule-governed manipulation of a distinct
set of symbolic representations inside an organism referring to an outside
world (cf. Riegler and Stein 1999; Ziemke and Sharkey 2001). The repre-
sentationalism of classical cognitive science, like the physical symbol sys-
tem hypothesis of functionalism, has a dyadic notion of representation
(tending to disregard the open-ended and processual character of interpre-
tation) and an atomistic concept of symbol, in strong contrast to the triadic,
general and evolutionary nature of sign processes and causality involved in
a Peircean conception of representation. On the level of general semiotics,
signs are forms of mediation (i.e., generals embodied as relational tokens)
that are embedded within a network of local processes of ongoing genera-
tion of new interpretants; this applies to cellular "vegetal" signs as well as
18. The Umwelt is the species-specific subjective universe, or phenomenal world,
of an organism; the part of the environment of a subj ect that it selects with its
species-specific sense organs according to its organization and biological needs.
Everything in the Umwelt is labeled with perceptual cues and effector cues of
the subj ect. According to J. von Uexkiill, every subj ect is the construtor of its
Umwelt; cf. J. von Uexkiill 1940 [1982], with a good introduction by his son
Thure von Uexkiill (1982). See also the volume edited by Kull (2001).
394 Claus Emmeche
cognitive "animate" ones. Embodiment in this basic sense is a sine qua non
for all sign action in nature and culture, and here, the term "embody" des-
ignates a general metaphysical aspect of semiosis: A sign must, so to
speak, materially incarnate its meaning.
19
It is on this level of generality,
we might call it semiotic embodiment, one must answer the question
whether embodiment is necessary or sufficient for meaning or mind: From
the point of view of semiotics, meaning and mind are all-pervasive phe-
nomena; meaning always has to be embodied in signs, but need not be con-
fined to organic bodies.
20
Although Peirce's own semiotics is embedded within his broader views
on the nature of reality, evolution and thought, contemporary Peircean
semiotics and biosemiotics need not subscribe to the total ofPeirce's fasci-
nating blend of pragmaticist philosophy of science, evolutionary cosmol-
ogy and metaphysics, although these inform our understanding of his no-
tion of logic as tightly connected to semiotics. What is real is not merely
what is existing, because the existing "here and now" of individual events
(their Secondness) are always, on the one hand, embedded within the gen-
eralities of law, tendencies, habit formation and thought, that is, the general
logic of things (the category of mediation, Thirdness), and on the other
hand, influenced by genuine chance, random deviations from lawfulness,
spontaneous generation of new possibilities (belonging to the category of
Firstness). Thus, any existing sign, such as this word "hand", is only un-
derstandable as a sign of something to some interpreter, by being a token
(Peirce called it a sinsign) of some general type of sign (a legisign), namely
the kind of word just mentioned. As Santaella states, "The legisign depends
on individual cases to be actualized. The symbolic legisign" - here, the
word hand - "is embodied in individual cases. In this same act of embodi-
19. A physical precondition for signification, so to say. Sign action is mediation,
and as such requires a material medium for this process (as well as energy, at
least under normal thermodynamic conditions; it is well-known that information
processing has physical limits and cost energy, cf. Zurek 1989)
20. "Thought is not necessarily connected with a brain. It appears in the work of
bees, of crystals, and throughout the purely physical world; and one can no
more deny that it is really there, than that the colors, the shapes, etc., of objects
are really there. Not only is thought in the organic world, but it develops there.
But as there cannot be a General without Instances embodying it, so there can-
not be thought without Signs. We must here give 'Sign' a very wide sense, no
doubt, but not too wide a sense to come within our defmition. (Peirce [1931-
1958] 1902, CP 4.551).
ment the individual cases are conformed to the symbol's domain. The sym-
bol functions as a rule for the formation of a certain sub class of sinsigns
which are called replicas. The rules for the formation of the replicas also
involve the interpretative rules of these replicas. Hence, the replica of a
symbol is a special kind of index which acts to apply the general rule or
habit of action or expectance associated to the symbol to a something par-
ticular" (Santaella 2003; cf. Short 1988). Also symbolic representations are
embedded ongoing processes, and thus we see that the notion of symbol in
semiotics is very different from the notion of symbol within classical cog-
nitive science. Regarding the specific human form of embodiment, the
symbol attains a specific role accounting for the social nature of human
consciousness.
4. The body within evolutionary emergent levels
As we have seen, the animate body as a self-moving organism transcends
the simpler category of an organism realizing the vegetative functions of
growth and reproduction. Already at this stage biosemiotics provides not
only an alternative (non-physicalist) "naturalization" of semiotic processes
in general, but also a framework for distinguishing between levels of em-
bodiment corresponding to different degrees of complexity involved in sign
interpretation processes. To clarify the structure of the present argument
for a multiplicity of phenomena of embodiment, a simple schematic list of
levels of embodiment may help. The full meaning of the scheme's concepts
will be explained in the following. Here is the scheme (adapted from Em-
meche 2002a):
1) The body of physics: Dissipative self-organizing structures. Thermo-
teleology.
2) The body of biology: The organism as a vegetative, physiologic-
homeostatic self-organizing structure. Bio-functionality and irrita-
bility.
3) The body of zoology: The animal as an autonomous, self-moving
organism. Intentionality and consciousness.
4) (a) The body of anthropology: The human body as a signifying ani-
mal, incarnating a socio-cultural specific life-world. Desire, his-
tronics and conscience. (b) The body of sociology: A "cybody", i.e.,
396 Claus Emmeche
a societal body dependent upon technology, embedded in a civiliza-
tion. Cosmopolitics, hybridicity, posthumanity.
The scheme indicates ordering relations between some forms of embodi-
ment. Their epistemic dimension is mentioned first, by organizing those
forms according to different domains of science each constituting its own
objects; their ontic dimension is implied by an underlying ontology of lev-
els of organization in Nature. Four such levels are mentioned. The point is
not the exact number of levels (these are contingent upon a historically
relative state of science) but the fact that irreducible levels do exist. E.g.,
the animal as a phenomenon includes an organism, yet the animal form of
embodiment transcends (in the sense explained above) the vegetative form
of embodiment; that of a cell or a plant. No distinction is made between
embeddedness and situatedness, but between their vegetative, animate and
sociocultural forms: Organismic embodiment: The organism is situated in
its ecological nice. Animate embodiment: The animal is additionally em-
bedded in a complementary subjective Umwelt (see below). Anthropic
embodiment: A sociocultural Umwelt constitutes a lifeworld enacting so-
cial roles and expectations (histrionics) as well as conscience.
Thus, different notions of embodiment are partly constituted by differ-
ent areas of science, yet from a point of view of "semiotic realism", science
can indeed capture true aspects of "the joints of nature". Those joints are
inclusive, in the sense that being a blackbird includes being a bird, which in
turn includes being a vertebrate (and so forth), and more generally, being
an animal includes being an organism, and this in turn includes being
physical (cf. footnote 3). The modes of human embodiment are also inclu-
sive: The point of calling non-living entities "bodies" and talking about
"the body of physics" is that we only get a more complete comprehension
of even non-living matter when we realize that it can evolve into higher
forms of organization whose properties transcend mere physical proper-
ties.
21
Such properties are emergent, though still depending on physical
processes.
Physical embodiment. Classical as well as modem physics deal with
three kinds of objects; first, general forces in nature, particles, general
21. This point is reminiscent of Marx' use of Hegel's philosophy. Cf. Marx' apho-
rism "The anatomy of man is a key to the anatomy of the ape" (from Introduc-
tion to Grundrisse) that should be understood within a frame of a radically re-
constructive historical approach as opposed to naively constructive chronologi-
cal historiography (see Stah11975: 59-62).
bodies (matter in bulk), and the principles ("laws") governing their action;
second, more specifically the structural dynamics of self-organized bodies
(galaxies, planets, solid matter clusters, etc.); third, physical aspects of
machines (artefacts produced by human societies and thus only fully ex-
plainable also by use of social sciences, like history of technology). In the
history of science one has often seen attempts to reduce all of physics to a
formalism equivalent to some formal model of a machine, but there are
strong arguments against the completeness of this program (Rosen 1991),
i.e., mechanical aspects of the physical world are only in some respects
analogous to a machine. Some of the general properties of bodies studied
in physics have a teleonomic character (a kind of directedness or finality),
in the scheme called "thermo-teleology", as this phenomenon of directed-
ness is most known from the second law of thermodynamics (a directed-
ness towards disorder), as well as the opposing self-organizing tendencies
in far from equilibrium dissipative systems. "Final causation" indeed is a
pervasive aspect of purely physical processes (Hulswitt 2002).
Organismic embodiment. A biological notion of function
22
is not yet
present in physics, while it is crucial for all biology. Biofunctionality is not
possible unless the living system is self-organizing in a specific way, based
upon a memory of how to make components of the system that meet the
requirement of a functional metabolism of a high specificity. For Earthly
creatures this principle is instantiated as a code-plurality between a "digi-
tal" code of DNA, a dynamic regulatory code of RNA (and other factors as
well, partly digital partly analogic), and a dynamic mode of metabolism
involving molecular recognition networks of proteins and other compo-
nents. This establishes a basic form of living embodiment, the single cell
(as a simple organism) in its ecological niche, which presupposes the
workings of "the physical body" as a thermodynamic system in non-
equilibrium, yet transcends that form by its systematic "memory,,23 of or-
ganism components and organism-environment relations. Biosemiotics
posits that organismic embodiment
24
is the first genuine form of embodi-
22. Or that for which a part works in order to sustain a living whole, cf. Emmeche
(2002 b).
23. This memory is often described as "genetic information" but it includes other
forms of memory as well, i.e., stable inheritance systems located within the or-
ganism (but outside DNA), or within the ecological habitat of the organism
(Jabonka 2001).
24. A concept very similar to the same term in Ziemke 2003, who bases his term
"organismic embodiment" on J. von Uexkiill and the notion of autopoiesis.
398 Claus Emmeche
ment in which a system becomes an autonomous agent "acting on its own
behalf' (cf. Kauffman 2000), i.e., taking action to secure access to avail-
able resources necessary for continued living. It is often overlooked that
the subject-object structure of this active agent is mediated not only ener-
getically by a structured entropy difference between organism and envi-
ronment, but also by signs of this difference; signs of food, signs of the
niche, signs of where to be, what to eat, and how to trigger the right inter-
nal processes of production of organismic components the right time. It is
often forgotten that the active responsitivity of the agent organism (based
upon observable molecular signs) has, as an "inner" dimension, a quality of
feeling,2s implied here by what is called irritability at the level of a single
cell. Irritability is probably both real, logically in accordance with a basic
matter-mind evolutionary continuity, rationally conceivable, though impos-
sible for humans to sense or perceive "from within" or empathetically
know "what it feels like", say, for an amoeba or an E.coli.
Animate embodiment. As the reader might have guessed, the biosemi-
otic idea is that when we consider the realm of animal mind, the intention-
ality of an animal presupposes the simpler forms of feelings and irritability
we stipulate in single cells (including the "primitive" free-living animals,
such as protozoa, lacking, e.g., a nervous system), yet transcends these
forms by the phenomenal qualities of the perceptual spaces that emerge in
functional perception-action cycles as the animal's Umwelt. As already
mentioned, proprioceptive semiosis is a crucial element of phenomenal as
well as functional properties (cf. Sheets-Johnstone 1998). More generally
However, Ziemke's "organismic embodiment of autopoietic, living systems"
(2003: 1306) does not distinguish, as we do here, between the body of biology
and the body of zoology and thus lacks a specific notion of animate embodi-
ment which is characterized by not only (vegetative) autopoiesis but also (ani-
mate, proprioceptive, perception-motor-coupling based) movement.
25. The concept of "feeling" is not quite the same as "qualia" in contemporary
philosophy of mind, but closer to its use in Peirce's phaneroscopy (e.g., [1931-
1958] 1902, CP.1.306-311)."Inner" is not referring to a separation between
brain states and bodily or environmental states (which is problematic, cf.
Thompson and Varela 2001: 422) but to an ontological category of possibility
and quality. As Thompson and Varela do not distinguish organismic from ani-
mate embodiment, the fIrst of their three dimensions of embodiment,
"organismic regulation" (including "sentience" or "the feeling of being alive",
ibid., 424) is a mixture of feeling on this basic organismic level and "primal" or
"core" consciousness at the level of animate embodiment.
(and less controversially), the animal body is a highly complex and specific
kind of a multicellular organism, a kind that builds upon the simpler sys-
tems of embodiment on the level of biology, such as physiological and
embryogenetic regulation of the growth of specific organ systems, includ-
ing the nervous system. These regulatory systems are semiotic in nature,
and rely on several levels of coded communications within the body and
their dynamic interpretations (for details, see Hoffmeyer 1996; Barbieri
2001; Markos 2002). The expression "the body of zoology" in the sche-
matic list is used to emphasize both its distinctness as a level of embodi-
ment, and that zoology instead of being simply part of an old-fashioned
division of the sciences should be the study of animated movement, in-
cluding its phenomenal ("mental") qualities.
Anthropic embodiment. Two forms of human (anthropic) embodiment
should be distinguished, (a) social embodiment universally characteristic of
humans as beings forming distinct language-dependent sociocultural
groups; (b) societal embodiment, forming "cyborg bodies" (Emmeche
2004b), or cybodies. The later is a special form of (a) in which a socio-
cultural body is embedded in a society with several social subsystems (e.g.,
work, education, politics, law, economy, love/reproduction! consumption,
health care, art, science, media), i.e., a civilization, and where the body's
animate mode of being is intrinsically connected to technology (e.g., med-
ico-technology) dissolving sharp body/machine boundaries; see section 6
below. First let us concentrate on the anthropic body of a person as a so-
cietal being, that is, not simply social in the sense of being a social animal,
but emphatically "human-social", being part of a society with division of
labor, institutional subsystems, social roles, culture, etc.; as Aristotle said,
a political animal. It is beyond the scope of this chapter to develop the spe-
cific senses in which humans (i) are embedded within a society, (ii) em-
body distinct social systems with all their peculiarities, (iii) have their
minds constituted by institutional habits, which they (iv) internalize during
socialization. The "habitus" sensu Bourdieu (Jenkins 1992) of a human
being in a social field is the implicit way his or her process of socialization
has enacted a semiotic system of embodied habits, some of which have
even physically formed the very animate body of the person - juxtaposing
people from different times and classes; an attritioned third world coal
miner, a North American teenager, or a 17
th
Century French noblewoman,
may serve as illustration. No doubt, other theoretical notions in sociology
could express culturally specific forms of human embodiment, but Bour-
dieu's concept of habitus suffices here, as it represents a set of class-
400 Claus Emmeche
specific dispositions generating specific practices and patterns of percep-
tion; it is embodied in individuals, and at the same time, it is a collective
and homogeneous phenomenon, mutually adjusted for and by a social
group or class.
26
Thus, on this genuinely anthroposemiotic level, biosemiotics as an ap-
proach does not suffice to capture the complexities and specificities of
human embodiment. Animals have various needs and appetites, including
sexual ones, whereas human desire, though developmentally presupposing
appetites, cannot be reduced to them. Furthermore, consciousness in ani-
mals like apes and monkeys can exhibit quite complex forms of social cog-
nition (Bekoff, AlIen and Burghardt 2002), but without a linguistic system
(for symbolic representation of abstract structures) and a societal con-
sciousness (of different language games, social roles, institutions and nor-
mative systems) even animal social consciousness is not that form of so-
cietal consciousness we find among humans, together with self-
consciousness in its more complex forms. One of those forms is con-
science, internalized culture-specific schemata of right and wrong relations
between self and other. Histrionics, normally a name for dramatic art, is
used in the schematic list to signify the intimate relation of human em-
bodiment and socio-cultural situatedness: "The naked animal" is not really
human unless it realizes itself as a histrion, an actress on a social scene, a
26. Using the concept of habitus should make the distintion between animate and
anthropic forms of embodiment clear. However, one could ask (a) what is em-
bodied in what; (b) what is not embodied, and (c) if "embodied" could simply
be replaced by "physical", "situated" or "embedded". As for (a), sociocultural
history is embodied in social institutions, and the norms and rules of an institu-
tion are again embodied in a person functioning within such an institution.
Thus, a person in some role incarnates a part of sociocultural history. Some-
times but far from always, forms of anthropic embodiment imprint their marks
on the animate body of a person. Question (b) is like asking "what is not cul-
tural?" when dealing with sociocultural systems. The answer depends on the
purpose of the analysis (are we interested in the pervasiveness of cultural proc-
esses in a human lifeworld, or, e.g., in delimiting borders between a civil society
and a state as distinct spheres of influence?). The answer to (b) depend on the
purpose, focus and conceptual tools of any specific analysis. Regarding (c),
similar remarks applies (cf. the schematic list). The distinct levels of organis-
mic, animate, and sociocultural embodiment imply distinct forms of embedment
and situatedness.
person mediating - as if by a social "mask,,27 - a general status in the social
field "in front of the mask" and a narratively structured unique self-
consciousness "behind the mask" (cf. Welker 2000).
5. Causality and emergent levels of interpretation
The above levels of embodiment can be located within an evolutionary
macro-history and further differentiated in detail. The attempts to charac-
terize more precisely what notions of causality should be necessary to ac-
count for the whole series of levels would lead into technicalities we will
eschew here (see, e.g., El-Rani and Emmeche 2000). Instead, let us briefly
describe the intuitions and demands that such a notion of causality should
conform to. The idea from emergence theory (cf. review by Pihlstrom
2002) is that once a new emergent level is formed, it implies both new
properties and other principles than those found on the previous level, but
also has an effect to constrain the possibilities of processes and actions on
the level below. This "downward" influence (Andersen et al. 2000;
Thompson and Varela 2001) may be seen as a formative, structuring cause,
not to be mixed up with the efficient cause of classical physics. Thus, a
biosemiotic perspective demands a "Non-standard Neo-Aristotelian Plu-
ralist notion of Causation (NNPC), that allow for dealing with a complex
system of several levels. Within a NNPC, the classical notion of a temporal
dyadic cause-effect relationship is just one element (often to be used for
within-level explanations) of a more complete set of causes also including
material causes (as answers to question about composition), formal causes
(corresponding to structural constraints of a higher level upon its compo-
nents), and final causes (that may take a level-specific form, and thus may
be interpreted either as the enactment of anticipatory purposeful behavior
within a cybernetic system, or more generally (and Peircean) as a pervasive
form of structural causation of which the physical laws of nature are just an
example).28 For the notion of inclusive levels of embodiment this means,
(I) for the physics-biology border, that for instance the metabolic regulari-
ties of organismic embodiment of a grass constrain the physics of protein
27. Cf. the Latin persona, a mask used by an actor.
28. There is an unsolved question here about the relation between a Peircean con-
ception of causality operating with efficient and formal causation and an even
more pluralist notion as the NNPC. See also Hulswitt 2002; Emmeche 2004a;
Andersen et al. 2000.
402 Claus Emmeche
action within its leaves; (11) for the biology-zoology border, that the cogni-
tive action-reaction cycles of an animal's perception and movement con-
strain the formation of interpretants within its nervous system; and (Ill) for
the zoology-anthropology border, that even societal structures like a market
economy in specific ways may have constraining effects upon the embodi-
ment of not just human freedom and action, but also the very animal exis-
tence of human beings.
Another important distinction is the one between, on the one hand, a
general ontological principle of embodiment stating that general forms
have to be incarnated or realized in particular instances and can hardly
exist disembodied as platonic forms (the general type needs its individual
tokens to exist), and on the other hand, system-specific embodiment, i.e.,
embodiment holding for particular phenomena of mind, consciousness,
meaning and significance, as these phenomena are embodied in particular
system types (therefore the general principle that the mind is materially
embodied must be specified). We have already seen the first principle ex-
emplified by what we called semiotic embodiment: A particular general
sign, such as a symbol, needs particular instances to embody it. The second
principle was illustrated by the explanation of the scheme of levels of em-
bodiment. Seeing different forms of embodiment both in a semiotic per-
spective and as evolutionarily emergent means seeing them as involving
emergent levels of interpretation and understanding. One could learn all
physiological facts about, say, the effects of iron nails against the skin of a
person; perhaps useful for learning surgery, yet without getting to the es-
sence of some phenomenon. Also the biology of wound healing and the
neurophysiology of pain experiences may not satisfy our inquiry. The
question is, what is the phenomenon? Closer to revealing fragments of that
mystery may be developmental psychology, but at this stage we also have
to ask questions about the specific sociocultural context - are we talking
about skin piercing as a fashion of youth culture, or crucifixion as a tool
for torturing political prisoners, or an alleged messiah?
6. The cyborg nature of human embodiment
From the reflections on system-specific forms of embodiment thus far it
transpires that biosemiotics is not enough to account for the characteristics
of human embodiment. Some of these, such as the capacity to handle lan-
guage and language-dependent societal processes, have often been seen as
disembodied, almost Platonic forms. But what may seem to be disembod-
ied societal structures, like a state's constitution, a moral principle, or the
very spoken language, are always specific historical products that indeed
have to be realized in specific social fields and systems to be real. Thus the
general metaphysical principle of embodiment holds, also for the societal
level.
The human animal was from its beginning societal, language-based,
technical and political, but not all these determinations were equally devel-
oped. Let us just briefly look at the technological dimension that only be-
came fully realized in the industrial phases of civilization, when we dis-
cover what we can call the intrinsic cyborg nature of humans, as techno-
culturally embedded beings within a space of meanings that are not only
symbolic, but argumentative and culturally empowered by different kinds
of social systems. Often, in cyber-punk and science fiction literature, the
cyborg is seen either as the idea of a "melding of human and machine" and
a "new era of participant evolution", or as people modified mechanically to
perform specific tasks or redesigned to work in an alien environment (Sta-
bleford 1999). By a strangely familiar kind of reification, the machine is
seen as something alien to the human body, and the melding of body and
machine is then perceived as threatening human dignity or the 'essence' of
being a human subject.
29
The anti-essentialist threat of the possibility, af-
forded by biotechnology, of a complete "hybridization" of a human organ-
ism and a machine has been used critically within feminist thinking (Hara-
way [1991]; Hayles 1999; Flanagan and Booth 2002) which in turn has
inspired phenomenological approaches to technology (Ihde 2002). We will
only use these sources to emphasize that the machine is not any "alien"
entity. Though usually made of inorganic materials, any machine is a prod-
uct of human work and its societal network of skills, ideas, thought, insti-
tutions, etc., and thus, we reify or blackbox the machine when we see it as
a non-human entity.30 Marx was probably the first historian to make this
point which is also a point about embodiment. On the level of human em-
bodiment, tools, technical artefacts, machines, all embody human co-
29. Thus, one tends to forget that "The body in a contemporary society is already a
cyborg body (a partly scientific-technologically governed body) as we are de-
pendent, from birth to death, on the blessings of present-day medical science"
(Emmeche 2002a: 155; see also Clark 2003.
30. This does not mean that there are no crucial differences between organismic and
robotic forms of embodiment, as the work of Tom Ziemke (2003, see also this
volume) clearly shows.
404 Claus Emmeche
operative division of labor, and of course, in our society, knowledge-
intensive machines embody societal forms specific for a "post-industrial,"
"late capitalist," "risk" or "knowledge" society.
Today the use of computers or "thinking machines" has become wide-
spread, and we should be aware of the ease of blackboxing or reifying the
semiotic processes involved in their interpretation. In current Western so-
ciety, it is an aspect of the cyborg nature of human existence that machines
both embody and hide complex social relations, and somehow make us
forget the distributedness of processes like meaning and cognition, as we
are led to commit failures of misplaced concreteness when we localize
thought either in a single machine or a single brain. Our cyborg nature
implies as a concept that technical artefacts, as one of the conditions of
existence of the human form of life, embody human thought processes.
This applies also to simple artefacts as part of the anthroposemiotic net-
work of tools, as Peirce noticed:
A psychologist cuts out a lobe of my brain (nihil animale me alienum puto)
and then, when I fmd I cannot express myself, he says, "You see your fac-
ulty of language was localized in that lobe. " No doubt it was; and so, if he
had filched my inkstand, I should not have been able to continue my discus-
sion until I had got another. Yea, the very thoughts would not come to me.
So my faculty of discussion is equally localized in my inkstand. It is local-
ization in a sense in which a thing may be in two places at once. On the the-
ory that the distinction between psychical and physical phenomena is the
distinction between fmal and efficient causation, it is plain enough that the
inkstand and the brain-lobe have the same general relation to the functions
of the mind. (Peirce [1931-1958] 1902, CP 7.366; see also Noth 2003;
Skagestad 1999).
When we have arrived at a phase of civilization, in which the societal form
of human embodiment
31
within the health care system has made us all ex-
31. Levels of embodiment (cf. the schematic list above) are partly dependent upon
the epistemic interest and analytical perspective; this applies also for the above
mentioned (a) social and (b) societal forms of anthropic embodiment. The kind
of "social embodiment" reviewed in Barsalou et al. (2003) corresponds to (a)
although some of the research reviewed tends to assume that, e.g., facial and
bodily responses (examplifying social embodiment) are cross-cultural univer-
sals. The theory of Barsalou (cf. Lindblom and Ziemke this volume), especially
the notion of simulators, is interesting (in spite of its metaphorical load of the
brain as a computer) but raises this question: To which extent are simulators
(e.g., "the simulator for the social category face", ibid., 67) specific for social
istentially connected to that system (and thus, to the system of tech-
noscience that health care relies on), our cyborg nature takes on a double
form. In a basic dimension, we continue to be that kind of animal, who can
only survive in its present form by being interconnected to a global techno-
societal system. On a supervening dimension, historically specific for a
modem risk society, this interconnected existence change our very animal
bodily appearance by the possibility of, e.g., prolonging life, exchanging
body parts because of ageing and desires to be forever young, tendentially
transforming us into the cyborgs of science fiction - what some has called
"the posthuman condition".32 Due to the inclusivity of the forms of em-
bodiment, the term posthuman is misleading though, as it is only now,
when we come to consider ourselves to be beyond human nature, that our
nature is revealed to us as at once human and cyborgian: To paraphrase
Marx, the anatomy of the cyborg is the key to the anatomy of humans. We
are already cyborgs, hooked-up to our fellow beings by tools and technol-
ogy, and in late modernity, also via the technoscientific health care system.
Human cyborg embodiment can take many forms, not all equally desirable,
and will continue to be a contested issue in the biopolitics of the present
century.
References
Andersen, Peter. B., Niels O. Finnemann, Peder V. Christiansen and Claus Em-
meche (eds.)
2000 Downward Causation. Minds, Bodies and Matter. Aarhus: Aarhus
University Press.
Barbieri, Marcello
2001 The Organic Codes. Cambridge: Cambridge University Press.
embodiment in humans? Could the theory also apply to, say, social embodiment
in a herd of wolfs? And if so, in which sense is it a theory specifically explain-
ing social embodiment in humans, in contrast to more general animal-based
cognitive capacities?
32. Most famous in the posthuman discourse is Fukuyama (2002); see also Hayles
(1999).
406 Claus Emmeche
2003
Ruppert
Social embodiment. In: Brian H. Ross (ed.), The Psychology of
Press.
Bekoff, Marc, Colin AlIen and Gordon M. Burghardt
2002 The Cognitive Animal. Empirical and Theoretical Perspectives on
Animal Cognition. Cambridge, Mass.: MIT Press.
Brooks, Rodney A.
2002 Flesh and Machines. New York: Vintage Books.
Clark, Andy
1999 An embodied cognitive science? Trends in Cognitive Science 3 (9):
345-351.
2003 Natural-Born Cyborgs. Oxford: Oxford University Press.
Csordas, Thomas J.
1996 Body. In: Adam Kuper and Jessica Kuper (eds.), The Social Science
Encyclopedia. 2
nd
ed., 55-56. London: Routledge.
EI-Hani, Charbel Nifio and Claus Emmeche
2000 On some theoretical grounds for an organism-centered biology:
Property emergence, supervenience and downward causation. Theory
in Biosciences 119 (3/4): 234-275.
EI-Hani, Charbel Nifio, Joao Queiroz and Claus Emmeche
2005 A semiotic analysis of the genetic information system. (unpublished
manus., in prep.)
Emmeche, Claus
2001 Does a robot have an Umwelt? Reflections on the qualitative bio-
semiotics of Jakob von UexkiilI. Semiotica 134 (1/4): 653-693.
2002 a Kroppens kaput som organisme. In: Gert Balling (ed.), Homo sapiens
2.0. Nar teknologien kryber ind under huden. 121-156, 238-239.
Kebenhavn: Gads Forlag.
2002 b The chicken and the Orphean egg: On the function of meaning and
the meaning of function. Sign Systems Studies 30 (1): 15-32.
2004 a Causal processes, semiosis and consciousness. In: Johanna Seibt
(ed.), Process Theories: Crossdisciplinary Studies in Dynamic
Categories, 313-336. Dordrecht: Kluwer.
2004 b Alife, organism and body: The semiotics of emergent levels. In:
Mark Bedeau, Phil Husbands, Tim Hutton, Sanjev Kumar and Hide-
aki Suzuki (eds.), Workshop and Tutorial Proceedings. Ninth Inter-
national Conference on the Simulation and Synthesis of Living Sys-
tems (Alife IX), 117-124. Boston Massachusetts.
Emmeche, Claus, Simo Keppe and Frederik Stjernfelt
1997 Explaining emergence - towards an ontology of levels. Journal for
General Philosophy ofScience 28: 83-119.
Emmeche, Claus, Kalevi Kull and Frederik Stjernfelt
2002 Reading Hoffmeyer, Rethinking Biology. (Tartu Semiotics Library 3).
Tartu: Tartu University Press.
Flanagan, Mary and Austin Booth (eds.)
2002 reload: rethinking woman + cyberculture. Cambridge, Mass.: MIT
Press.
Fukuyama, Francis
2002 Our Posthuman Future: Consequences of the Biotechnology Revolu-
tion. New York: Farrer, Strauss and Giroux.
Gilbert, Scott F. and Sahotra Sarkar
2000 Embracing complexity: Organicism for the 21st Century. Develop-
mental Dynamics 219: 1-9.
Haraway, Donna
1985 A cyborg manifesto: science, technology and socialist-feminism in
the late twentieth century. In: Donna Haraway (ed.), Simians, Cy-
borgs and Women: The Reinvention ofNature, 149-181. New York:
Routledge. Reprint 1991.
Hausman, Carl R.
1993 Charles S. Peirce's Evolutionary Philosophy. Cambridge: Cam-
bridge University Press.
Hayles, N. Katherine
1999 How We Became Posthuman. Chicago: The University of Chicago
Press.
Hoffmeyer, Jesper
1996 Signs of Meaning in the Universe. Bloomington: Indiana University
Press.
Hulswitt, Menno
2002 From Causation to Cause. A Peircean Perspective. Dordrecht: Klu-
were
Ihde, Don
2002 Bodies in Technology. Minneapolis: University of Minnesota Press.
Jablonka, Eva
2001 The systems of inheritance. In: Susan Oyama, Paul E. Griffiths and
Russell D. Gray (eds.), Cycles of Contingency, 99-116. Cambridge,
MA: MIT Press.
Jenkins, Richard
1992 Pierre Bourdieu. New York: Routledge.
Kauffman, Stuart
2000 Investigations. Oxford: Oxford University Press.
408 Claus Emmeche
Keller, Catherine
2003 Embodiment. In: J. Wentzel Vrede van Huyssteen (ed.), Encyclope-
dia ofScience and Religion. Vol. I, 254-256. New York: Macmillan
Reference.
Kull, Kalevi (ed.)
2001 Jakob von Uexkull: A paradigm for biology and semiotics. Berlin
and New York: Mouton de Gruyter (= Semiotica 127 (1/4): 1-828).
The Capital. Vol. I. London: Penguin Classics. Reprint 1990.
Kull, Kalevi
1999 Biosemiotics in the twentieth century: A view from biology. Semi-
otica 127 (1/4): 385-414.
1999 Philosophy in the Flesh. The Embodied Mind and its Challenges to
Lawrence, Christopher J.
1981 Irritability/sensibility. In: W.F. Bynum, E.J. Browne and Roy Porter
(eds.), Macmillan Dictionary ofthe History ofScience, 214. London:
Macmillan Press.
Markos, Anton
2002 Readers ofthe Book ofLife. Oxford: Oxford University Press.
Marx, Karl
1867
Mayr, Emst
1982 The Growth ofBiological Thought. Harvard, Cambridge: Belknap.
Noth, Winfried
2003 Semiotic machines. S.E.E.D. Journal (Semiotics, Evolution, Energy
and Development) 3 (3): 81-99.
Peirce, Charles Sanders
1902 Collected Papers of Charles Sanders Peirce. Vol. 1-6, Charles
Hartshome and Paul Weiss (eds.); Vols. 7-8, Arthur W. Burks (ed.).
Cambridge, Mass.: Harvard University Press. [References: CP, fol-
lowed by vol. and paragraph number]. Published 1931-1958.
Petrilli, Susan
1999 About and beyond Peirce. Semiotica 124 (3/4): 299-376.
Pihlstrom, Sami
2002 The re-emergence of the emergence debate. Principia 6 (1): 133-
181. [Universidade Federal de Santa Catarina, Florian6polis, Brasil]
Queiroz, Joao and Sidharta Riberio
2002 The biological substrate of icons, indexes and symbols in animal
communication: a neurosemiotic analysis of Vervet monkey alann-
calls. In: Michael Shapiro (ed.), The Peirce Seminar Papers: Essays
in Semiotic Analysis, Vol. 5, 69-78 New York: Berghahn Books.
Riegler, Alexander, Markus Peschl and Astrid von Stein (eds.)
1999 Understanding Representations in Cognitive Science. Does repre-
sentation need reality? Dordrecht: Kluwer.
Rosen, Robert
1991 Life Itself. A Comprehensive Inquiry Into the Nature, Origin and
Fabrication ofLife. New York: Columbia University Press.
Salthe, Stanley N.
1989 Self-organization of/in Hierarchically Structured Systems. Systems
Research 6 (3): 199-208.
Santaella Braga, Lucia
1999 A new causality for understanding the living. Semiotica 127 (1/4):
497-519.
Santaella, Lucia
2003 What is a symbol. S.E.E.D. Journal (Semiotics, Evolution, Energy
and Development) 3 (3): 54-60.
Scott, Alwyn
2004 Reductionism revisited. Journal of Consciousness Studies 11 (2):
51-68.
Sheets-Johnstone, Maxine
1998 Consciousness: A natural history. Journal ofConsciousness Studies 5
(3): 260-294.
Short, Thomas L.
1988 The growth of symbols. Cruzeiro Semiotico 8 (Janeiro): 81-87.
(Porto, Portugal).
1982 Life among the legisigns. Transactions ofthe Charles Sanders Peirce
Society 18 (4): 285-310.
Skagestad, Peter
1999 Peirce's inkstand as an external embodiment of mind. Transactions
ofthe Charles S. Peirce Society 35: 551-561.
Stableford, Brian
1999 Cyborg. In: John Clute and Peter Nicholis (eds.), The Encyclopedia
ofScience Fiction, 290-291. London: Orbit.
Stahl, Gerry
1975 Marxian Hermeneutics and Heideggerian Social Theory: Interpret-
ing and Transforming our World. Ph.D. thesis, Northwestern Univer-
sity, Evanston, Illinois. (also available at http://www.cs.colorado.edu
/ dissertations/philosophy)
Thompson, Evan and Francisco J. Varela
Cognitive Sciences 5 (10): 418--425.
410 Claus Emmeche
Van de Vijver, Gertrudis, Stanley Salthe and Manuela Delpos (eds.)
1998 Evolutionary Systems: Biological and Epistemological Perspectives
on Selection and Self-Organization. Dordrecht: Kluwer.
Van de Vijver, Gertrudis, Linda Van Speybroeck and Windy Vandevyvere
2003 Reflecting on complexity of biological systems: Kant and beyond?
Acta Biotheoretica 51 (2): 101-140.
Varzi, Achille
2004 "Mereology", The Stanford Encyclopedia of Philosophy (Fall 2004
Edition). In: Edward N. Zalta (ed.), URL = http://plato.stanford.edu/
archives/fa1l2004/entries/mereology/
von Uexkiill, Jakob
1940 Bedeutungslehre. (Bios 10. Johann Ambrosius Barth, Leipzig),
[English translation: The theory of meaning, Semiotica 42 (1): 25-
82, 1982].
von Uexkiill' Thure
1982 Introduction: Meaning and science in Jakob von Uexkiill's concept of
biology. Semiotica 42 (1): 1-24.
Welker, Michael
2000 Is the autonomous person of European modernity a sustainable model
of human personhood? In: Niels Henrik Gregersen, Willem B. Drees
and UlfGorman (eds.), The Human Person in Science and Theology,
95-114. Edinburgh: T&T Clark.
Ziemke, Tom
th
baum.
Semiotica 134 (1/4): 701-746.
Zurek, Wojcieh H.
1989 Thermodynamic cost of computation, algorithmic complexity and
information metric. Nature 341: 119-124.
Embodiment and self-organization of human
categories: A case study for speech
Luc Steels and Bart de Boer
Abstract
The paper considers explanations for the kinds of categories that have been found
to be involved in human behavior. It insists that embodiment not only plays an
important role in shaping these categories, but also that the collective dynamics
generated by social interaction is of equal importance. A case study is developed
for speech sounds. We show through computer simulations how a group of
autonomous agents equipped with a (sufficiently) realistic perceptual and auditory
apparatus can arrive at a shared repertoire of vowels and that these vowel systems
exhibit the same universal trends that are found in human vowel systems. It is sig-
nificant that this happens without innate a priori categories.
Keywords: Embodiment, categorization, self-organization, speech.
1. Introduction
The term "embodied" is being adopted by a growing number of researchers
interested in cognition (Ziemke 2003), but often with different meanings.
At least three senses of embodiment are being used:
The first one emphasizes that cognitive processes, such as those
required for vision or language, are to be implemented in terms of
networks that have neural plausibility, both in terms of their compu-
tational microstructure and in terms of the general architecture of the
brain. While there is of course a wide consensus that cognitive proc-
esses must somehow be mapped onto brain hardware, the "strong
embodiment" hypothesis claims more, namely, that without this neu-
ral realism many cognitive phenomena cannot be understood. This
line of reasoning finds its most enthusiastic defenders among phi-
412 Luc Steels and Bart de Boer
losophers like Searle and those who argue that the brain should not
be viewed as an information processor at all (e.g. Edelman 1990).
The second type of embodiment research first manifested itself in
"behavior-based AI" (Steels and Brooks 1995). It argues that part of
our interaction with the world is not explicitly controlled by any kind
of information processing. Instead, the physical properties of the
body, the physical characteristics of sensors and actuators, the
structure of the task, and the physical properties of the environment
all play an important role in shaping actual behavior. This line of re-
search has proven to be very fruitful for designing and building
autonomous robots that interact with real-world environments in real
time (Pfeifer and Iida 2004). A neat example is the robot Stumpy
that exhibits locomotion without any kind of explicit control or
sensing (Iida, Dravid and Paul 2002).
The third sense of embodiment refers to the grounding and expres-
sion of human concepts. It argues that many concepts, even abstract
mathematical concepts, find their ultimate foundation in grounded
interaction with the world (Lakoff 2000) and that this is still visible
in the metaphors and expressions used to communicate using these
concepts. For example, the temporal preposition "back" (as in "back
in time") is historically derived from a body part (the back) which
first became used to indicate a spatial relation or area (as in the back
of the car) and then a temporal relation (see also Nlifiez 1999). It
follows from this position that conceptualization and concept acqui-
sition must be embedded in a developmental history of embodied
interaction with the world.
This paper explores the second view on embodiment, as manifested in
"embodied AI", and tries to demonstrate what kind of explanatory power
can be drawn from it. We take the domain of speech sounds as a case study
because we believe that without studying concrete cognitive tasks and
showing what embodiment can bring, the discussion may become too di-
vorced from reality. Section 2 contrasts classical theories of speech that do
not take embodiment into account with theories grounded in the function of
speech that do take embodiment into account. Section 3 presents theoreti-
cal considerations about what constitutes an explanation in cognitive sci-
ence as well as an overview of the phonological phenomena under consid-
eration. Section 4 presents a case study in which the role in explaining
universals of speech sounds of embodiment and situatedness are investi-
Embodiment and self-organization ofhuman categories 413
gated with a computer model. Section 5 presents a summary and the con-
clusion.
2. Embodied versus disembodied speech research
The domain of speech is known to be extraordinarily difficult to under-
stand and model because it not only involves complex fine-grained motor
control of the vocal articulators for producing speech sounds, but also
complex acoustic signal processing and categorization for recognizing
speech sounds. Therefore, there is at least the potential for a strong role of
embodiment. At the same time speech also seems to imply abstract catego-
ries (in the sense that some distinctions are relevant in a language and oth-
ers are not) and abstract rules (such as "a voiced consonant becomes un-
voiced at the end of a word", a rule which holds for Dutch but not for
English).
There is a long tradition in linguistics, shaped by the Prague structural-
ists, with Roman Jakobson (Jakobson and Halle 1956) as its most promi-
nent member, and Chomsky and Halle's influential work on abstract pho-
nology (Chomsky and Halle 1968), that completely ignores the embodied
aspects of speech. Instead, these linguists move immediately to an abstract
level in terms of distinctive features (such as +/- voiced, +/- labial, +/-
aspirated, etc.). These distinctive features are assumed to be both grounded
in articulation, as well as in acoustic recognition and categorization. For
example, voiced means that the vocal chords are vibrating while a sound is
being produced, such as in the final consonant of "bad" (voiced) versus
"bat" (voiceless). Yet it is never specified in detail how exactly this
grounding is realized. Abstract phonologists simply use the abstract fea-
tures to formulate phonological constraints (e.g. which combinations of
consonants and vowels are allowed in a language). But it turns out that it is
very hard to recognize these features in the acoustic data and that they are
not sufficiently precise to realistically drive motor control. Moreover this
branch of phonology often assumes that the features are genetically prede-
termined as part of the innate language acquisition device (LAD) and ar-
gues that systematicity is genetically encoded in terms of a set of principles
with parameters to be set by cues from the environment (Dresher 1992).
Abstract phonology is therefore a prime example of disembodied cog-
nitive science and it contrasts with work by another group of phoneticians
and phonologists who employ what can be considered an embodied ap-
414 Lue Steels and Bart de Boer
proach to speech (see e.g. Lindblom and Maddieson (1988); Browman and
Goldstein (1992)). They start from the actual acoustic signals and from
sophisticated articulatory models, taking constraints coming from the
physics of the world and embodiment into account. For example, many co-
articulation phenomena are no longer assumed to be governed by rules but
simply follow from the dynamics of motor control. Grounding now plays a
role in explaining the acquisition of speech sounds or why certain sounds
or sound combinations are occurring (MacNeilage 1998). Perception is
always coupled tightly to production (Plaut and Kello 1999). The research
discussed in this paper fits within this embodied view of speech. Section 3
presents theoretical considerations about what constitutes an explanation in
cognitive science
3. Explanatory theories
Before delving into the concrete case study presented in this paper, it is
worthwhile to make some general methodological remarks about the nature
of theories in cognitive science. It is well known that categorization is a
central topic in cognition; indeed we can say that a behavior becomes cog-
nitive only when some form of categorization is implied. Traditionally,
linguistics, psychology and anthropology have collected a large number of
facts about human categorization. For example, Berlin and Kay (1969)
collected data about the kinds of color categories found in human lan-
guages by doing naming and memory experiments, as first pioneered by
Lenneberg and Roberts (1956). Subjects are shown a series of color sam-
ples (the Munsell chips) and are asked to show the best representative for a
specific color like "red". Berlin and Kay found that there are some sur-
prising universal trends in focal colors and their work has generated a wide
body of literature that seeks to identify additional evidence which confirms
(Kay and Regier 2003) or refutes (Davidoff 2001) these basic human color
categories.
Another domain that has been studied extensively is space. Here again
research has shown that there are universal tendencies in spatial categories.
For example, IN, ON, LEFT OF, etc. are very common categories in hu-
man spatial reasoning and language (Herskovits 1986) and so it has been
argued that they are largely innate. But on the other hand, comparative
studies of the expression of spatial categories has shown significant
intercultural differences. Also, children only gradually acquire spatial cate-
gories under the influence of their native language (Bowerman and Choi
2001).
Similarly the categories (and hence the articulatory gestures) used in
sound systems of human languages show remarkable regularities. Lan-
guages do not use random subsets of the many possible speech sounds, but
rather certain sounds occur more often than others. Although humans are
able to produce and distinguish an amazing number of different speech
sounds, the average number of distinctive speech sounds that is used in the
languages of the world lies between 20 and 37 (Maddieson 1984). Some
sounds, such as [a], [m] or [k] occur almost universally, while others occur
only rarely. For example, English does not use the rather rare vowel [y]
(pronounced as in French "rue") whereas French (and Dutch) do. Phoneme
inventories tend to exhibit symmetries as well. If, for example, a repertoire
contains [0] it is very likely to also contain [e]. Such symmetries are also
found in consonant systems (Lindblom and Maddieson 1988). Languages
furthermore constrain the set of possible sound combinations. For example,
in English [mb] can occur at the end of a word as in "lamb" but not in the
beginning, whereas in some African languages this is possible (as in Swa-
hili "mbali" (far)). So sounds fall into classes and the classes form a com-
binatorial system. Again there appear to be universal tendencies in these
sound combinations although they have been less adequately surveyed.
Regularities like these (and many others can be demonstrated, including
universal tendencies in grammar) demand an explanation. Linguists and
psychologists in the disembodied tradition often propose that these univer-
sal tendencies are based on innate mechanisms (e.g. the distinctive
phonological features and their markedness is assumed to be part of the
innate language faculty), and so they are said to be universally shared by
all human beings, the same way each of us has (normally) five fingers on
each hand. But that means that these theories simply pass the buck to evo-
lutionary biologists. However, biologists are often not so keen about as-
cribing complex human behavior to innate features, partly because it seems
rather implausible that the micro circuitry of the brain is under detailed
genetic control (Edelman 1990), and partly because we would still need an
evolutionary scenario to explain for instance why [y] (as in French "rue")
is not common and [a] (as in English "bar") is, or why a language with four
color terms usually has terms for either green or yellow.
So simply saying that human categories are (to a large extent) innate is
not a sufficient explanation; we also need to show which neural structures
can perform the categorization, how these neural structures could be ge-
netically determined in development and how the relevant genes could
have become universally present in human populations. All of these are
non-trivial tasks that remain to be carried out. Another difficulty for the
genetic position is that without exception, human categorization is not ab-
solutely universal across all human populations and cultures. Rather it
shows significant variation. Although this variation is supposedly captured
in parameters set during development, it is unclear how such a parameter-
ized system could evolve under realistic selection pressures based on lan-
guage communication.
If genetics is not very satisfactory as a source of explanations, we
should perhaps investigate other ways in which certain categories like col-
ors, spatial categories or speech sounds can become shared (universally or
locally). Three possible determining factors have been discussed:
(1) Either human embodiment is so constraining that it is not necessary
to make the categories innate because they could never be perceived
or the motor movements could never be made. For example, the hu-
man vision system restricts perception to a certain range of the spec-
trum; the human vocal tract is limited to certain number of sounds,
the human body suggests a division into front and back, etc. This en-
ables and limits the set of possible solutions for a specific task.
(2) Human beings interact socially and this creates a collective dynam-
ics that may also constrain individual choices, pushing them in the
direction of a shared system. For example, in Britain cars drive on
the left side of the road and in continental Europe they drive on the
right side. This is not because British drivers have "left driving
genes" so that they are born to drive on the left, or that their em-
bodiment makes driving on the right impossible, but rather because a
cultural choice was made and this choice is enforced by the subse-
quent behavior of all involved. Even if somebody would like to drive
on the right in England, they would be quickly sanctioned or maybe
even lose their life.
(3) It could be that the regularities in the environment are such that a
statistical learning process picks them out easily and so there would
be no need to encode the category in the genome. For example, the
chromatic distribution of colors in the world might show enough
statistical regularity that statistical clustering algorithms could detect
them.
In the work being carried out by our research group, we have been explor-
ing explanations for human categorization in which these three factors play
a role. A good example is our research on color categories, in which we
have attempted to explain the universal tendencies in color categories using
embodiment, the collective dynamics generated by social interaction, and
the statistical structures inherent in the world (Steels and Belpaeme 2005).
In the domain of language in general and speech sounds in particular, the
statistical regularity is not present in the world independently of the agents.
So only factors (1) and (2) are relevant, particularly when a totally new
system is being bootstrapped.
In the case study that follows, originally described in (de Boer 1999,
2000), embodiment and the dynamics of language use in a population are
investigated as factors explaining regularities in systems of human speech
sounds. As vowels are the simplest speech sounds to model, this paper
focuses on vowels, but there also exists work on syllables and syllable
structures (Steels and Oudeyer 2000; Redford, Chen and Miikkulainen
2001).
Our main point is that embodiment in itself is not sufficient to explain
many universal tendencies or culturally accepted norms. Earlier embodi-
ment approaches have shown that embodiment constrains the possibilities,
e.g. that there are limitations of actuators and sensors which prevent or
discourage humans from using certain sounds. In addition, these ap-
proaches demonstrate that typical speech categories are in some sense op-
timal (see e.g. Lindblom, MacNeilage and Studdert-Kennedy. 1984). Spe-
cifically the sounds in a language are claimed to have a particular
distribution and structure that is optimized for the task of reliably produc-
ing and perceiving sounds, and hence the sounds are maximally contras-
tive. If that is the case, then embodiment and the constraints of the task are
sufficient to explain why human languages show particular sounds and not
others.
Nonetheless, observations of real phonetic data show that suboptimal
phonetic systems also exist in many languages. For example, the UPSID
database of human sound systems lists two examples with suboptimal 3-
vowel systems, namely with the sounds [e] [a] [0] instead of the more
common optimal [i] [a] [u]. They come from a North-American Indian
language, Alabaman (Rand 1968) and a Peruvian language, Amuesha (Fast
1953). These 3-vowel systems are suboptimal because they do not provide
the maximal contrast in the acoustic space. Therefore embodiment (and the
constraints of the task) are in themselves not sufficient to explain the vowel
systems one finds in human languages. One also has to take the collective
dynamics of language use in a population into account. All users have to
conform to the system of speech sounds that has historically been chosen in
order to achieve communicative success. This system is partly arbitrary and
may be suboptima1.
4. Experiments in self-organization of speech sounds
To examine whether the structure of vowel systems is sufficiently deter-
mined by self-organization in a population under constraints of embodi-
ment, we need to find a mechanism that incorporates these two factors, and
then show that this indeed leads to emergent vowel systems that reflect the
universal tendencies observed in human vowel systems. In our case, we use
a population of artificial agents that can produce, perceive and remember
speech sounds in a human-like way. Each agent is equipped with an ar-
ticulatory synthesizer, a model of human perception for calculating the
distances between different signals, and an associative memory for storing
vowel prototypes. Also, each agent can interact with other agents (follow-
ing a fixed pattern) by imitating them. These interactions are called imita-
tion games, and are in the same line as the "language games" we have used
in our work for the past decade (see Steels 2003). The agents can update
their vowel repertoires depending on the outcome of the interactions in
such a way that the expectation of future imitation success is maximized.
The agents' goal in life is to imitate the other agents as well as possible
with a repertoire of vowels that is as large as possible. However in doing
this, the agents only use local information, and do not carry out any explicit
optimization. The behavior of the agents is summarized in table 1 at the
end of section 4.1.
4.1. Embodiment constraints on perception and production
The repertoire of speech signals that humans can produce is limited by the
physics and the physiology of the vocal tract (figure 1). The repertoire of
sounds that they can distinguish is limited by the properties of the auditory
apparatus. In this example the focus is on vowels, and therefore only the
articulatory and perceptual properties of vowels are integrated. These
properties are also better understood than those of consonants. This has to
do with the fact that vowels are static signals, while many consonants are
dynamic, i.e. their articulation and their signal do not remain constant over
time.
IBlade
-GloWs
:fculators
Alveolum
(teeth ridge)
\
1
1
\
I,
..
Tongue
Teeth
Lips
. ~ ~ -
.........
Tip
Figure 1. Structure of the human vocal tract. Speech sounds are produced by
very fme-grained movements of the articulators
The acoustic signal of a vowel can be described relatively straightfor-
wardly by the resonances of the vocal tract. These resonances cause promi-
nent peaks in the frequency spectrum of a vowel signal. The frequencies at
which these peaks occur are called the formant frequencies. This is illus-
trated for the vowel [a] (as in "father") by the black line in figure 2. For
different vowels, these peaks occur in different places. Vowels can there-
fore be uniquely characterized by their formant frequencies. In practice,
only the three or four resonances at the lowest frequencies are relevant.
In the simulations discussed here, no acoustic signals are generated, al-
though it would be straightforward to add the extra complexity. Vowels are
represented by their first four formant frequencies. If an agent wants to
generate a given vowel, it synthesizes these formants using an articulatory
synthesizer. This synthesizer takes as inputs the three major vowel pa-
rameters: tongue position, tongue height and lip rounding (Ladefoged and
Maddieson 1996, Ch. 9). The outputs of the synthesizer are the first four
formant frequencies of the corresponding vowel. The inputs are modeled as
real values in the range 0 to 1. For tongue position, 0 means most to the
front and 1 means most to the back. For tongue height, 0 means lowest and
1 means highest. For lip rounding, 0 means least rounded and 1 means
most rounded. Thus (0, 0, 0) corresponds to the vowel [a] and to the for-
mant frequencies (708, 1517, 2427, 3678) Hertz. The parameters (1, 1, 1)
correspond to the vowel [u] and to the formant frequencies (276, 740,
2177,3506) Hertz. Formant values were taken from (ValIee 1994). This
synthesizer is able to generate all possible basic vowels.
2..,.---------------------,
o 700 H4380 Hz
.. -2 \./ A
4 \ ' Hz

-!---".....-....r----,,--.....,..--..,.,....----l
o 1000 2000 3D 4000 0000 5000
1, q
Figure 2. The fIrst four fonnants of the vowel [a] black line, and its perception
in tenns of the effective second fonnant (gray line)
Humans also perceive vowels based on their formant frequencies. These
can be used to calculate a perceptual distance between vowels. Unfortu-
nately, the distance calculation is not as simple as calculating a Euc1idean
distance between two formant vectors. As the bandwidth of the sound re-
ceptors at higher frequencies is greater than those at lower frequencies,
blurring of spectral detail takes place at higher frequencies. This means
that the formant peaks at higher frequencies can generally be compressed
into one broader peak. The center frequency of this peak is called the ef-
Embodiment and selforganization ofhuman categories 421
fective second formant. The first peak in the spectrum is usually perceived
as remaining in the same place, and is still referred to as the first formant.
The perception in terms of the first and effective second formant of the
signal [a] is shown as the gray line in figure 2. Note that this is an idealized
view to illustrate the process.
While there are several ways to calculate the effective second formant,
the one adopted in the research described here has been developed by
Schwartz et al. (1997b). It is a non-linear weighted average of the 2nd, 3rd
and 4th formant frequencies. In order to calculate distances between vowel
signals, the first and effective second formant are expressed in the Bark
frequency scale. The Bark scale is a perceptually inspired frequency scale
that can be considered to be logarithmic for the frequencies that are rele-
vant to formants. Equal frequency differences in Bark are perceived as
equal intervals, in contrast to the way frequencies in Hertz are perceived.
20 games 500 games 2000 games 10000 games
21
..~ .
~
..
a\.
2
.-:..... ~ .
~ ...
;
1
~ 4 ~ 4 ~ 41
,.
~ . ~ 4
...
..
-:1
~
I
~ !!!. !!!.
~
LL....
:'
LL9""
t
LL9""
g
LL'"
'. "
6! 6
1
;:
.0.
.J.
'" : I , .......
8 8 8 8
16 14 12 10 8 16 14 12 10 8 16 14 12 10 8 16 14 12 10 8
F
2
(Bark) F
2
(Bark) F
2
(Bark) F
2
(Bark)
Figure 3. Emergence of a vowel system in a population of twenty agents.
Using the first and effective second formant frequencies in Bark, the dis-
tance between two vowels can be calculated as an ordinary weighted
Euclidean distance. As differences in the effective second formant are per-
ceptually less important than differences in the first formant, the effective
second formant is multiplied by 0.3 when calculating distances.
These distances are then used to determine which vowel in an agent's
repertoire is recognized. This is done by calculating the distance between a
perceived signal and all the vowels in an agent's repertoire. The vowel that
is closest to the perceived signal is considered to be the one that the agent
heard.
Initiator Imitator
1 If ( V = )
Add random vowel to V
Pick random vowel v from V
u
v
u
v
+ 1
Produce signal A
1
: A
1
ac
v
+ noise
Receive signal A
1
.
If ( V = )
Find phoneme( v
new
, A
1
)
V Vv
new
Calculate v
rec
:
Produce signal A
2
: A
2
ac
vrec
+ noise
2
3 Receive signal A
2
.
Calculate v
rec
:
If ( v
rec
= v )
Send non-verbal feedback: success.
s
v
s
v
+ 1
Else
Send non-verbal feedback: failure.
Receive non-verbal feedback.
Update V according to feedback
signal.
4
5 Do other updates of V. Do other updates of V. 5
( )
2
2 1 2 1
:
( , ) ( , )
rec
v vrec
v V v
v V D A ac D A ac

<
( )
2
2 2 2 2
:
( , ) ( , )
rec
v vrec
v V v
v V D A ac D A ac

<
4.2. Results
The agents all start out with an empty vowel repertoire. By playing imita-
tion games with each other, the agents have to develop a vowel system that
is as large as possible, that allows for successful communication and that
should be realistic if self-organization and embodiment are really factors in
explaining the structure of human vowel systems.
4.2.1. Emergence ofa vowel system
The emergence of a vowel system in a population of twenty agents under
10% acoustic noise is shown in figure 3. In each of the frames of the fig-
ure, the acoustic aspects of the prototypes of the agents' vowels in the
population are plotted in acoustic space. In this particular acoustic space
(based on the first and effective second formant frequencies of a vowel
signal), equal distances between points correspond to equal perceptual
distances. Each vowel of each agent in the population is represented by a
dot. Note that due to articulatory constraints, only a roughly triangular area
of the acoustic space is available to the agents. This is indicated in the
fourth frame of figure 3.
From the figure it is clear that after the first 20 games the agents still
only have very few vowels. The vowels that exist are more or less ran-
domly dispersed through the acoustic space, although some of them already
show a tendency to cluster. This is caused by the fact that all agents start
out with an empty vowel repertoire. In order to get the imitation games
started, random vowels are inserted. However, the imitating agents in the
games try to make imitations that are as close as possible and add these to
their vowel repertoires. This accounts for the clustering. After some 500
imitation games, shown in the second frame, the clustering has become
more pronounced. The most important process at this moment is the com-
pacting of the clusters due to the fact that the agents move their vowel
prototypes closer to the signals they perceive. However, there is still suffi-
cient room in the auditory space for extra vowels, so the random addition
of new vowels also plays a role. After 2000 games, the available vowel
space becomes filled more evenly with vowels and the shape of the vowel
system becomes more realistic. After 10.000 imitation games, the available
acoustic space has become more or less filled up with vowels and the
vowel system has become realistically symmetric and dispersed. After this
has happened, the vowel system remains stable. However, it is not static.
The vowel prototypes of agents (and therefore the clusters) tend to move,
and it is even possible that they merge or that new clusters are formed (if
they do not interfere with other clusters).
Type A (30/54) Type B (10/54) Type C (7/54)
8
i 101
,
.
.- I'J/
...
lit ...... / 1
,.:: u
i
/fB/
2
8 1...- ------'
16 14 12 10
F
2
(Bark)
8
.
..
e :. /01
1/ .,... u......
6
lel
/
',1 " ..
.::. : lu!
2
lal :-.:
8 '---------.;.,....--"--..........,
16 14 12 10
F
2
(Bark)
2
6

\ -a/ul
lal ift..f 4
, UI
IfJ ,.:.- I'JI e?.
. .p u...... 6
lal .-.,. ..\

8 '--- """--__----..J
16 14 12 10 8
F
2
(Bark)
Type D (7/54) Type E (3/54) Type F (1/54)
I'JI
/01 4
lul
.Ial
/tl
lce/
/if
2
Jul
.
.
I'JI
hI
lel .t.
li! .,
2
,. lul
lif 2
lal a
8 1...- ...0-- ------'
16 14 12 10 8
F2 (Bark)
la/
8 L--
16 14 12 10 8
F
2
(Bark)
101
8 I...- """'-----"-_----.J
16 14 12 10 8
F2 (Bark)
Figure 4. Emerged vowel systems with six vowels.
4.2.2. Evaluation ofthe emerged vowel system
Are the vowel systems that emerge realistic? If in the emerged systems the
same types of vowel systems are found in the same proportions as in hu-
man languages, they are realistic. The classifications of human vowel sys-
tems that were used as a reference were the ones by Crothers (1978) and by
Schwartz et al. (1997a).
An example of a classification of emerged vowel systems containing six
vowels is shown in figure 4. The data in this figure were obtained by run-
ning the simulation a hundred times for a given parameter setting (acoustic
noise set to 12%). In each run 25 000 imitation games were played.
Note that although the frames in figure 4. look similar to the figures of
vowel systems shown previously in figure 3, there is a crucial difference.
In previous figures, the agents shown in one frame were all members of the
same population (and had therefore interacted with each other). In the pre-
sent figure, all agents shown are members of different populations. The
fact that there still is a large amount of similarity between the vowel sys-
tems of agents from different populations is a strong demonstration of how
self-organization can make populations converge towards similar vowel
systems.
The emerging systems are realistic. Most of them conform to the uni-
versals that Crothers (1978) found for human vowel systems. When the
percentages with which the different emerged systems occur are compared
to the percentages with which human vowel systems occur, a good match is
found as well. Schwartz et al. (1997a) have measured the occurrence of
different vowel system types in the different languages in the UCLA
Phonological Segment Inventory Database (UPSID, a database based on
speech sound data of 451 languages, Maddieson 1984; Maddieson and
Precoda 1990). They find 60 vowel systems with 6 vowels. Although their
classification is not exactly the same as the classification shown in figure
4, there is good agreement. Of the systems they found in UPSID, 43% is of
type A, 20% is of type B, 5% is of type C, 7% is of type D and 20% is of
type E. No systems of type F were found, and two of the systems from
UPSID (3%) cannot easily be fitted into the classification used here, but
are probably of type A. Equally good agreement was found for systems of
4 and 5 vowels and reasonable agreement was found for systems of 7 and 8
vowels (de Boer 1999, ch. 6). It seems that the simulation is capable of not
only predicting the most frequently found vowel systems in human lan-
guage (as was already possible with systems that optimize acoustic dis-
tinctiveness), but also of predicting the less frequently occurring vowel
systems and, approximately, their relative abundance.
5. Conclusions
The results of the simulation show that vowel systems can emerge through
self-organization in a population of agents whose production and percep-
tion is constrained by their embodiment. Although the agents start with an
empty vowel repertoire and do not have any constraints on the kinds of
vowel systems they can learn, the vowel systems that emerge in the popu-
lation tend to be symmetric and dispersed, and optimized for acoustic dis-
tinctiveness. The model also predicts the types of vowel systems that occur
less frequently and their abundance with a reasonable degree of accuracy.
This illustrates that dynamics and embodiment play an important role in
determining the structure of phonological and phonetic systems. The im-
portance of embodiment both for constraining the space of possibilities and
for pushing towards optimality in the task was known already (see e.g.
Lindblom, MacNeilage and Studdert-Kennedy 1984) and the computer
model presented here provides additional empirical support for this thesis.
At the same time, the model clearly shows that embodiment can (and has to
be) complemented with the collective dynamics flowing from social inter-
action. As the imitation success of agents is partly determined by how well
the agents' vowel repertoires conform to the repertoires that are used by
the other agents in the population, different (sub-optimal) configurations
can emerge and be maintained. Some configurations can be considered
stronger attractors than others in the dynamical system that is defined by
the agents and their interactions. The same universal tendencies that are
found in human vowel systems are found in the systems that emerge. This
indicates that innate rules and constraints are not necessary in order to
explain why these tendencies are present; neither is it necessary to assume
that they have to be genetically encoded.
More generally (and based on other works such as reported in Steels
and Belpaeme 2005) we argue that the universal tendencies found in hu-
man categorization, as well as the cognitive systems that use these catego-
rizations for action and decision-making, are a function of multiple forces,
and in particular embodiment and collective dynamics. In domains where
the statistical structure inherent in the real world is relevant (such as color
categories), sensitivity to this structure will also play a role. In domains
where there is genuine choice (like driving to the right or left) the regular-
ity is generated by the agents and cannot be explained on the basis of pre-
existing statistical structure. All this does not exclude the possibility that
certain categorizations could not have become genetically assimilated.
However, in the case of cultural systems, like language, which undergo
relatively rapid change, this seems less likely than is often assumed.
2001
2000
Davidoff, J.
2001
Acknowledgement
The Sony Computer Science Laboratory in Paris funds the work by Luc
Steels, with additional funding from the OHLL CNRS project, the OMLL
ESF project, and the EU FET ECAgents project. The work ofBart de Boer
reported here took place at the University of Brussels AI lab and was
funded by a GOA project.
References
Berlin, B. and P. Kay
1969 Basic color terms: Their universality and evolution. University of
California Press.
Bowerman, Melissa and Soonja Choi
2001 Shaping meanings for language: Universal and language specific in
the acquisition of spatial semantic categories. In: Melissa Bowerman
and Stephen C. Levinson (eds.), Language Acquisition and Concep-
tual Development, 475-511. Cambridge: Cambridge University
Press.
Browman, Catherine P. and Louis Goldstein
1992 Articulatory phonology: An overview. Phonetica 49: 155-180.
Chomsky, Noam and Morris Halle
1968 The Sound Patterns ofEnglish. Cambridge, Mass: MIT Press.
Christophe, Anne, Jacques Mehler and Nuria Sebastian-Galles
2001 Perception of prosodic boundary correlates by newborn infants. In-
fancy 2: 285-394.
Crothers, John
1978 Typology and universals of vowel systems. In: Joseph H. Greenberg,
Charles A. Ferguson and E. A. Moravcsik (eds.), Universals of Hu-
man Language, Volume 2 Phonology, 93-152. Stanford: Stanford
University Press.
Language and perceptual categorisation. Trends in Cognitive Sci-
ences 5 (9): 382-87.
de Boer, Bart
1999 Self Organisation in Vowel Systems, PhD Thesis, AI-Lab, Vrije
Universiteit Brussel.
Self organization in vowel systems. Journal of Phonetics 28 (4):
441-465.
The Origins of Vowel Systems. Oxford: Oxford University Press.
Dresher, Elan
1992 A learning model for a parametric theory in phonology. In: Robert
Levine (ed.), Formal Grammar: Theory and Implementation, 290-
317. Oxford: Oxford University Press.
Edelman, Gerald
1990 Neural Darwinism. New York: Basic Books.
Fast, Peter W.
1953 Amuesha (Arawak) phonemes. International Journal of American
Linguistics 19: 191-194.
Herskovits, Annette
1986 Language and Spatial Cognition. Cambridge: Cambridge University
Press.
Iida, Fumiya, Raja Dravid and Chandana Paul
2002 Design and control of a pendulum driven hopping robot. Proceedings
ofInternational Conference on Intelligent Robots and Systems 2002
(IROS 02) Lausanne, Switzerland, 2141-2146.
Jakobson, Roman and Morris Halle
1956 Fundamentals ofLanguage. The Hague: Mouton and Co.
Kay, P. and T. Regier
2003 Resolving the question of color naming universals. Proceedings of
the National Academy of Sciences 100 (15): 9085-9089.
Ladefoged, Peter, and Ian Maddieson
1996 The Sounds ofthe World's Languages. Oxford: Blackwell.
1980 Metaphors We Live by. Chicago: University of Chicago Press.
Lindblom, Bjom, Peter MacNeilage and Michael Studdert-Kennedy
1984 Self-organizing processes and the explanation of phonological uni-
versals. In: Brian Butterworth, Bernhard Comrie and Osten Dahl
(eds.), Explanations for Language Universals, 181-203. Berlin:
WaIter de Gruyter.
Lindblom, Bjom and Ian Maddieson
1988 Phonetic universals in consonant systems. In: Larry Hyman and
Charles N. Li (eds.), Language, Speech and Mind, 62-79. London:
Routledge.
Lenneberg, Eric H. and John M. Roberts
1956 The language of experience: A study in methodology. International
Journal ofAmerican Linguistics memoir 13.
MacNeilage, Peter
1998 The frame/content theory of evolution of speech production. Behav-
ioral and Brain Sciences 21: 499-548.
Maddieson, Ian
1984 Patterns ofSounds. Cambridge: Cambridge University Press.
Maddieson, Ian and Kristin Precoda
1990 Updating UPSID. UCLA Working Papers in Phonetics 74: 104-111.
NUflez, Rafael
1999 Could the future taste purple? Reclaiming mind, body and cognition.
Journal ofConsciousness Studies 6 (11-12): 41-60
Pfeifer, Rolf and Fumiya Iida
2004 Embodied artificial intelligence: Trends and challenges, In: Fumiya
Iida, RolfPfeifer, Luc Steels and Yasuo Kuniyoshi (eds.), Embodied
Artificial Intelligence, Lecture Notes in Computer Science 3139: 1-
26. Berlin: Springer Verlag.
Plaut, D. C. and C.T. Kello
1999 The emergence of phonology from the interplay of speech compre-
hension and production: A distributed connectionist approach. In:
Brian MacWhinney (ed.), The emergence of language: 381-415.
Mahwah, NJ: Erlbaum.
Rand, Earl
1968 The structural phonology of Alabaman, a Muskogean language. In-
ternational Journal ofAmerican Linguistics 34: 94-103.
Redford, Melissa A., Chun C. Chen and Risto Miikkulainen
2001 Constrained emergence of universals and variation in syllable sys-
tems. Language and Speech 44: 27-56.
Schwartz, Jean-Luc, Louis-Jean Boe, Nathalie Vallee and Christian Abry
1997 a Major trends in vowel system inventories. Journal of Phonetics 25:
233-253.
1997 b The dispersion-focalization theory of vowel systems. Journal of
Phonetics 25: 255-286.
Steels, Luc
2003 Evolving grounded communication for robots. Trends in Cognitive
Science. 7 (7): 308-312.
Steels, Luc and Tony Belpaeme
2005 Coordinating perceptually grounded categories through language.
Behavioral and Brain Science. 28 (4): 469-529.
Steels, Luc and Rodney Brooks
1995 The Artificial Life Route to Artificial Intelligence. Building Embod-
ied Situated Agents. New Haven: Lawrence Erlbaum.
Steels, Luc and Oudeyer, Pierre-Yves
2000 The cultural evolution of syntactic constraints in phonology. In: M.
A. Bedau, J. S. McCaskill, N. H. Packard and S. Rasmussen (eds.),
Proceedings of the VIf
h
artificial life conference (alife 7): 382-394.
Cambridge (MA): MIT Press.
Vallee, Nathalie
1994 Systemes vocaliques: de la typologie aux predictions. These. ICP,
Grenoble.
Ziemke, Tom
2003 What's that thing called embodiment? In: R. Alterman and D. Hirsch
(eds.), Proceedings o/the 25th Annual Meeting o/the Cognitive Sci-
ence Society: 1305-1310. Hillsdale, NI: Lawrence Erlbaum.
Communication as situated, embodied practice
Wolff-Michael Roth
Abstract
Researchers generally are preoccupied with written language as the paradigm of
communication. This paradigm makes communication and what is communicated
appear as something that exists apart from invariably embodied and situated action.
A large part of human communication involves participants in face-to-face conver-
sation. In such situations, participants attend to semiotic resources other than words
produced and attended to, including gestures, body positions, and structures in the
environment. To avoid the dualism that characterizes much of the scholarship on
communication, I provide a dialectical account of human activity, one component
of which is communication, itself characterized by the dialectical relation of differ-
ent modes (speech, gesture, structures in environment). The analyses of an episode
from a school science classroom exemplify the central role of the body in commu-
nication, which by far exceeds what participants articulate in so many words. Its
relationship to environmental features, however, is dialectical, making communica-
tion a (materially, socially) situated and embodied practice.
Keywords: agency I structure dialectic, communication, embodiment, gestures,
situated practice.
1. Introduction
Since the late 1980s, it has become fashionable to talk about knowing and
learning in terms of embodied cognition Of, alternatively, in terms of dis-
tributed or situated cognition. All of these terms imply that knowing ex-
ceeds that what can be found in the brain - viewed as computational wet-
ware - and that has been the preoccupation of psychologists.! Like many
others concerned with cognition and development, I had been involved in
1. Etymologically, to know and cognize have the same origins, generally held to
have the same root as can.
432 Wolff-Michael Roth
building models of human cognition based on Piaget's work, which came
to be associated with information processing. However, once I conducted
research in the attempt to understand knowing and learning in real time and
particularly the evolution of language, the then valid measures of reasoning
ability and short-term memory were unable to predict performances in the
real-world settings that I attempted to explain. Theoretical approaches that
included or were based on social processes seemed to be much more
promising, as the turn to social constructivism in disciplines such as social
studies of science (e.g., Latour and Woolgar 1979), anthropology, and edu-
cation (e.g., Lave and Wenger 1991) seemed to imply.
In the course of the 1990s, I had taken hundreds of hours of videotape
first in classrooms and then among scientists and technicians at work. It
became increasingly clear to me that analyzing only the words they used
was insufficient to understand what participants knew, what they commu-
nicated to one another, and what they knew to be going on. It was impossi-
ble to model communication unless one also accounted for the bodily ori-
gin of communication and nature of knowing, on the one hand, and for
material cultural processes, on the other. There was a problem, however,
which I understood only afterward: the individual (body) and the sociality
of culture cannot be thought together in dualistic models, because each
approach ends up reducing one to the other. It was at then that dialectical
approaches to communication became salient in my work.
In this chapter, I exemplify the kind of communicative action that I
found among students and professionals of science. Drawing on an episode
from a science classroom, I exemplify the (socially, materially) situated
and embodied nature of communicative action. I begin by articulating a
dialectical framework for activity and action that allows me to treat com-
municative action and situated cognition in a non-dualist way. I move on to
provide a brief description of the ethnographic and analytic background,
before providing an extended analysis of one typical episode.
2. Integrating body, mind and culture
Body, mind, and culture can easily be integrated into the same unit within a
dialectical approach (e.g., Leont'ev 1978). In the following, I first sketch a
general theory resting on the dialectic of agency and structure, and then
articulate communicative action as one aspect within this framework.
Communication as situated, embodiedpractice 433
2.1. Agencylstructure dialectic
The embodied, situated, and cultural nature of cognition in general and
communication of a conscious agent more specifically can be understood
within an agencylstructure dialectic (Sewell 1992). Agency denotes the
capacity to act. It is immediately clear that there is no agency without
structure, material or cognitive. For living organisms, structure is every-
where. Bodies are structured in a material sense, exhibiting particular ar-
ticulations, symmetries, and sizes, all of which mediate the kind of move-
ments organisms are capable of. The world in which we live is structured,
too, with its characteristic forces and articulations among things, but
structured differently for different organisms (von Uexkiill 1972/1928).
Scholars attend less to the structured ways in which we see, hear, feel,
move, and do things, including thinking about them. These structured ac-
tions, including perception, have arisen from our bodily engagement with
the world (Merleau-Ponty 1945; see also Gallagher this volume; Lindblom
and Ziemke this volume). The very cognitive structures that allow us to
perceive the world in structured ways are themselves a consequence of
interactions of a structured body with a structured world (Bourdieu 1980).
But there would not be the need for speaking of structures in the world
unless there were organisms to which these structures are behaviorally
relevant. To distinguish the two forms of structure, the notions of objec-
tively given (sociomaterial) resources and (mental) schema have been pro-
posed (Sewell 1992). At any moment in activity, the given structures, i.e.,
sociomaterial resources and schema, enable and constrain actions.
In any consideration of human activity, the two forms of structures have
to be considered dialectically. Any object of activity simultaneously exists
as a vision or hypothesis of the forthcoming product (which implies con-
sciousness), a set of material conditions including materials and tools, and
knowledge of sociomaterial conditions of the productive activity (Saari and
Miettinen 2001). Therefore, neither sociomaterial resources nor schema
individually or together determine action; rather, actions emerge in re-
sponse to the contradictions embodied in the unit of analysis, which com-
prises the identity of (the non-identical) sociomaterial resources and their
appearance to conscious mind in action.
Any analysis of activity simultaneously has to consider three levels,
which integrate the bodily and cultural nature of engagement with the
world: activity, action, and operation (Leont'ev 1978). Activities, such as
schooling, farming, or researching are always collective endeavors but
concretely intended and realized in varying form, and are directed toward
some generalized (conscious) object or motive. Concretely realization oc-
curs by means of sequencing of particular conscious actions, each directed
toward a specific conscious goal formulated by the acting subject. While
researching, a professor might decide to measure a certain variable or to do
an ANOVA. The relation of activity and action is dialectical: the activity
determines the sense of an action - an ANOVA done by a pollster is likely
to have a different sense than the professor's - but actions concretely real-
ize the activity. Properly sequenced smaller units, operations, constitute
actions; operations are unconsciously produced responses to conditions,
such as when the professor keys the statistical model into the computer but
is not consciously aware of hitting particular keys but focuses on the equa-
tions appearing on the screen. The relation of action and operation is,
again, dialectical: the action is a generalized referent for the proper se-
quencing of operations, but operations concretely realize actions. Opera-
tions have been shaped in and are the outcome of previous interactions of
the body with the sociomaterial world, they constitute a concrete realiza-
tion of cultural possibilities; they are a form of unconscious collective con-
sciousness, that is, an embodiment of culture. This approach corresponds to
that taken by Gallagher (this volume), where actions are goal-directed and
conscious but realized by unconscious operations brought about by body
schema. Communication is but one form of actions that develop the current
activity, without necessarily being about the activity.2 This approach there-
fore contrasts the claims of those scholars (e.g., Zlatev, this volume) who
fail to recognize that (a) language-in-use (parole) is most frequently pro-
duced without prior planning (i.e., conscious supervision) and frequently
does not follow the conventions grammarians have adopted and (b) com-
municative action and thought are frequently produced in real time rather
than reproduced through speech.
2. Space limitations do not allow me to elaborate the relation between resources
and schema on the one hand, and activity, action, and operation on the other
hand. Suffice it to say that resources exist externally to the agent, at the inter-
face of activity and action; and schemas exist internally to the agent, at the inter-
face of action and operation.
2.2. Communicative action
Communicative action is more than sending information (words) from a
sender (speaker) to a receiver (listener), which would require equal tuning
of the two participants to code and encode concrete material sound parcels.
Because of the dialectical relation material resources and schema, the
identical tuning of speaker and listener is never given in human communi-
cation. A dialectical approach to human activity leads us to a different
conception of communication. Each action produces a change in the cur-
rent setting, thereby providing changes to the semiotic resources available
for the conduct of activity in general and face-to-face communication in
particular. Speech act theory allowed us to understand that utterances, too,
have effects in the setting, they constitute resources that can be part of any
future action, current or subsequent speaker. Artifacts (Streeck 1996), dia-
grams (Goodwin 2000), and other structured aspects in the setting, includ-
ing various forms of gestures (e.g., Bavelas, Chovil, Coates and Roe 1995),
pointing and body orientation (Hindmarsh and Heath 2000), and pitch
(Goodwin, Goodwin and Yeager-Dror 2002), constitute resources produced
and used by speakers and listeners alike. To understand communication in
everyday settings, all of these structures need to be accounted for simulta-
neously (Roth 2004).
These different structures that are apparent in communicative action
(words, gestures, things pointed to) are not simply additive but dialectically
related. Thus, for example, speech and iconic gestures constitute two ways
in which thought is not merely made public but actually finds its expres-
sion as productive activity (McNeill and Duncan 2000). Speech and ges-
tures are dialectically related because they constitute different expressive
forms for the same intentional stance toward the world. This stance is not
some idea somehow behind or underlying and therefore expressing (trans-
lating) speech. Rather, it is my bodily me that gestures and speaks, and
thereby signifies thought and intention for others and myself (Merleau-
Ponty 1945).
Making sounds, speaking individual words, gesturing, nodding the head,
turning the body and so forth are operations that together constitute a
communicative act (Mikhailov 1980). The production and recognition of
these operations always requires a concrete body.3 But each communica-
3. Gallagher (this volume) and Lindblom and Ziemke (this volume) elaborate on
the link between production and recognition of body-produced movements
tive act does not have a sense in itself; rather, sense emerges from the rela-
tion between the speech act and the overall, collective activity. Communi-
cative action is therefore at the same time a concretely embodied (intra-
psychological) and sociocultural (inter-psychological) phenomenon; its
meaning emerges from the simultaneous grounding of action in cultural
activity and bodily operations. Conscious action therefore lies at the inter-
face between the unconscious operations that realize them and the cultural
processes that give them their sense.
3. Ethnographic context
The data analyzed here derive from an in-situ naturalistic study of commu-
nication in a seventh-grade science classroom where students learned about
the physics of simple machines largely by designing machines. To make
the learning environment as authentic as possible, the specifications for the
design tasks were articulated in a call for proposals. As part of their sub-
mission, the students included paper-and-pencil sketches (e.g., Figure 1)
and prototypes. They subsequently presented their prototypes to the student
collective, which instantiated a form of peer review. This curriculum there-
fore provided many opportunities for students to engage in conversations
with others, both during the design of the prototype and the review ses-
sions. Two cameras and two tape recorders were used to record small-
group and whole-class conversations.
To begin with, all recordings were roughly but word for word tran-
scribed to capture all utterances as completely as possible under the pre-
vailing conditions (e.g., with students hammering and using power tools).
In a second pass, the VHS videotapes were digitized (Macintosh iMovie);
offprint images capturing characteristic moments were imported into the
transcripts. Each lesson subsequently was broken into episodes, which
were exported to QuickTime versions. These digitized clips were then
analyzed using Peak DV 3.21, which allows the graphical representation
of waveform (for determining time, relative loudness, and duration) and its
Fast Fourier Transform (for determining pitch changes). Individual utter-
ances were timed with an accuracy of O.OOl seconds but rounded to the
(signs) and the role of mirror neurons to bring about this integration between
perception and action.
nearest 10 milliseconds and coordinated with the frames that allowed tim-
ing to 33 milliseconds (30 frames/s rate of the VHS video).
4. Language and body in design activity
Linguistic analyses generally are concerned with language independent of
communicative praxis. This leads directly leads to the grounding of lan-
guage, that is, its connection to things and situations, which is continuously
realized in practice but has been a problem for traditional cognitive science
(Harnad 1990). Furthermore, most linguistic studies are hardly relevant to
speech, for the language of writing is a fundamentally different cognitive
phenomenon than language in practical face-to-face communication (Ong
1982). Here, I am concerned with language as it appears in face-to-face
communication alongside other semiotic resources humans produce in
interaction and for interactive purposes - body orientation, body move-
ments, and hand gestures. Following the presentation and brief gloss of a
23-second episode, analyses are provided of the concrete role played by
human bodies in communication.
4.1. A moment in design activity
The following episode was recorded at the moment when the three girls
were deliberating how to go about building the tower that would hold an
elevator, the first element in their design of a Rube Goldberg machine
(Figure 1).4 Leanne (far left in offprints) had previously explained that they
needed to build a two-sided tower; Amanda (far right) and Bella (center)
went along so that it appeared that there was agreement on this design.
Following Leanne's suggestion, the three had decided to cut the wood
needed to build that part of their design. At one moment, Leanne talked
about cutting two pieces of wood for the two sides of the structure. This
4. Rube Goldberg was a cartoonist who drew complicated machines that accom-
plished the most mundane everyday actions. Constructing a Rube Goldberg ma-
chine is a popular design activity of middle school students around the world.
More information is available at http://www.rube-goldberg.com/.
was when the following episode began.
5
(I first provide the raw transcript
and then a verbal description of the events.)
Figure 1. The students had decided to have a tower and a chute as the frrst two
elements in their Rube Goldberg machine, a complicated device that
would accomplish the simple task of feeding a cat. The episode con-
cerns the construction of the tower part.
5. To get a better sense for the changes in body positions and hand gestures in-
volved, readers should visually scan the images before reading the transcript.
The following transcription conventions have been used in line with traditional
conversation analysis: (0.41) : time in seconds; [ttwo sides] : square brackets
enclose speech overlapped by gesture or other body movement; [ t ( ( r ~ lH
moves up)) : description of gesture (rH, ill = right and left hand, respectively),
body movement; * : the asterisk aligns speech and video offprints; = : equal sign
shows latching, that is, two utterances are not separated by the normal pause;
.,?! : punctuation is used to indicate speech features, such as rising intonation
heard as a question, or falling intonation to indicate the end of an idea unit
(sentence); - : the n-dash indicates stop in utterance without voice inflection in-
dicating end of idea unit.
01 Bella:
02
03 Leanne:
04
05 Leanne:
06
07
08
09
10 Bella:
11 Bella
12
13 Leanne:
14 Bella:
15 Leanne:
Communication as situated, embodied practice 439
Okay, which two sides? *
(0.47)
Okay, you know [\how we are doing
that thing where the el]\bevator thing -
comesh bup*
[l(body rotates to the left, rH places
saw on table))
[z((rH moves upward, IH slightly up-
ward))
b((begins turning upper body,
(0.43)h
b((head turns toward board to right,
rHpoints to board))
[4these * are the sides]
[4((rH single stroke along the board))
[sthat like keep it] * [6standing].
[s((body turns to left, both hands come
up palms facing one another to pinna-
cle oftrajectory))
[6((two-handed gesture halfway down,
slightly up, then completely down))
(0.59)
hO.60]
h((two-handed up down gesture, at half
height))
[8The two sides?] *
[8((two-handed up gesture))
[9(0.88)]
=[9((nods three times, at the end, turns
to her right toward the piece ofboard))
[\oUh, okay.]
[1O((turns to her right toward the piece
ofboard))
440 WolfJ-Michael Roth
16 Leanne:
17
18
19 Amanda:
20
21 Leanne:
22 Amanda:
23 Leanne:
24
25 Leanne:
26 Bella:
27 Leanne:
[111 think it'll work. But if it doesn't, I _ " - : ~ M ! I "
know what we can-
[lI((PicAJ up saw, begins sawing))
(0.42)
fix how we can fix it.
Don't we need four?*
(0.56)
No,just-
Just two.
Two.
(0.44)
[12
Ri
ght?]*
[12((turns upper body, head to gaze at
Bella))
=[13((nods twice))] 11
[14((turns head to gaze at saw and
wood))
At the beginning of this episode, Bella, apparently confused about just how
the part was to look, then asked, "Okay, which two sides?" (turn 01). A
brief pause followed Bella's question, before Leanne began to answer,
"you know how we are doing that thing where the elevator thing comes up"
(turn 03). While uttering this description of the current state of affairs, she
turned her body toward Bella, laid down the saw she had been operating,
and moved her right arm and hand upward (turn 03). In the ensuing pause
in the verbal articulation, Leanne turned to her right, her hand moved for-
ward initially pointing to the board on the table, and then moving her hand
about 5 centimeters along one of the edges. It was precisely at the place
where she had previously shown with the saw where and how they needed
to cut the word (Figure 2). Contemporaneous with the gesture, she uttered,
"these are the sides" (turn 06). She oriented toward Bella again and, in a
continuous motion, produced two two-handed up-down hand movements,
palms facing, suggesting two pieces facing one another, while uttering,
"that like keep it standing" (turn 07).
After a longish pause, Bella mimetically reproduced Leanne's double-
handed gesture, before repeating the gesture all the while uttering, with
rising pitch, "The two sides?" At the verbal level, there followed a longer,
0.88-second pause (turn 10). But Leanne began the first of three head nods
just as Bella had completed her utterance (turn 13). That is, as a turn, her
head nod latched onto the ending of Bella's turn. Bella responded with a
sign of understanding and idea completion (falling pitch), "Uh okay,"
while Leanne turned to the piece of board from which she had earlier indi-
cated the intention to cut two narrow pieces. She explained that she
thought her idea would work, and ifnot, she knew an alternative (turns 16-
18). She had begun to saw and would not stop for the remainder of the
episode.
Figure 2. The iconic gesture produced here with the saw moving along the
board where a cut was to be made was reproduced later in an episode
analyzed in this chapter.
At this point, Amanda, who had sat on the table next to her two peers, en-
tered the conversation for the first time in this episode. Up to this point, she
had faced away and now, turning her head only slightly to her right, asked,
"Don't we need four?" (turn 19). Continuing to saw and interrupted by
Amanda's own articulation "two," Leanne responded, "No, just two" (turns
21,23). After a brief pause and without stopping to saw, Leanne turned her
upper body (slightly) and head to gaze squarely at Bella in the process
uttering with rising inflection, "Right?" (turn 25). She had barely com-
pleted when Bella began to nod twice, upon which Leanne turned her head
back to look toward the board that she was working on.
In terms of the overall project, the episode was important in that what
had appeared to be a common design vision of the tower and the elevator it
subtended did not exist but was brought into alignment. Although the three
girls previously had talked about and seemed to agree on the design, and
although Leanne had explicated the two-sided version of the tower once
before, it was apparent that both of her two peers wondered in the course
of this episode why she wanted to cut two pieces of board rather than some
other number, including the four Amanda articulated. This alignment,
however, was not produced merely by talk. Rather, pointing (deictic) and
iconic gestures, head nods, body turns were important aspects of the com-
municative exchange. Without participants' attention to the body of the
respective other, they would find it difficult to produce the alignment
achieved here. The latter part of the episode exemplifies this: Amanda, not
having had perceptual access to the gestures and body orientations of the
other two girls, did not understand why there ought to be two sides to the
tower despite the fact that Leanne just explained it. Thus, Amanda began
the second part of the episode with fundamentally the same question that
Leanne had just answered in response to Bella's query.
In this episode, therefore, alignment of at least two of the three partici-
pants with respect to a future state of their activity was achieved. The epi-
sode had begun with a query about the number of sides for the tower, al-
though this issue seeme,d to have been settled during a previous interaction.
Twice during the episode, Bella aclmowledged the two-sided version of the
tower, once by means of an iconic gesture and confirming utterance (turns
10, 14) and then by means of a head nod. As Amanda acquiesced (turn 22),
we have no indication about her personal vision of what their collective
project was to be. In the following, I analyze different aspects of this epi-
sode, which bring to the foreground different dimensions of the relation-
ship of body and language in action.
4.2. Orienting the body and gesturing mediate turn taking
Communication is an interactive phenomenon requiring the analysis of
pairs of turns. A body of studies in conversation analysis provided an un-
derstanding of mostly audiotaped conversations (ten Have 1999). Whoever
speaks, holds the floor; others normally do not speak at the same time.
However, participants do not own pauses in speech. If the current speaker
has the intention to continue, he or she has to provide other semiotic re-
sources that indicate this intentionality. However, videotaped interactions
reveal that in face-to-face situations, there are many other semiotic re-
sources that participants bring to in play that mediate turn taking. In this
situation, those who want a turn at talk may provide bodily and gestural
resources that indicate this intentionality (Goodwin 2000). Mutual aware-
ness of the interaction participant's body orientation, perceptual access,
and (hand, head) gestures mediates the rules of interaction in ways that
traditional conversation analysis could not reveal. Videotaped materials, as
used here, reveal the central role of the body in face-to-face interaction.
The role of body orientation and perceptual access was quite apparent
in this episode, which had started with Bella's question about the nature of
the tower sides. Leanne responded, followed by a renewed query and af-
firmation that there would be two sides. During the interaction, both had
produced gestures in the context of changing body orientations. Amanda
had been sitting on the table facing away from the two. Thus, she had no
access to all those parts of the communicative interchange between her
peers other than the verbal utterances. In particular, she had not been in the
position to perceive either Leanne or Bella's two-handed production of an
image that pertained both to the role of the two pieces of board that they
were in the process of cutting and the design drawing. Her question "Don't
we need four?" essentially repeated Bella's question about the relationship
between raw materials, design drawing, and their vision of the emerging
artifact. The words alone had been insufficient to bring Amanda in align-
ment with what was to be the collective vision for the unfolding design
artifact.
Pauses are interactive resources that play a particular role in the situated
evaluation of communicative action (Bourdieu 1980). As a pause in talk is
lengthening, it increasingly becomes a resource for any participant to begin
speaking. Following a question, however, a pause becomes an increasing
liability for the addressee, as not answering can be heard as an intentional
exclusion of the querying person from interaction. This happened repeat-
edly in this group, where Leanne did not answer queries that Amanda di-
rected to her, whereas she answered all questions that Bella posed. Thus,
the lengthening pause following Amanda's question (turn 19) opened up
the possibility that this situation, too, would be one in which Amanda did
not receive a response.
Perceptual access to the interlocutor's body changes the rules identified
in conversation analytic studies. Body movements required by the interac-
tion may take time, but the very fact that they are enacted articulates com-
municative intention. So actions other than speech also articulate the intent
to contribute, and therefore that the current turn lies with the acting person.
For example, after Leanne first articulated the sides of the tower ("that
thing where the elevator thing comes up"), there was a pause at the verbal
level. However, at that time she turned her body and moved arm and hand
nearly 90 degrees around to her right. The motion ended in the pointing
gesture and led into the gestural and verbal articulation of the target as "the
sides." Later (turn 13), she nodded three times in response to Bella's query
for confirmation of the two-walled nature of the future artifact. The head
nodding was therefore a different form of turn taking than what has been
available to traditional conversation analysis. In the head nod, the body
articulated an affirmation, which in a different spatial arrangement might
have come forth through an activation of the vocal cords. The relationship
between pause and orientation was also evident in the following situation.
In the final part of the episode (turns 25-27), Leanne had answered
Amanda's question about the number of sides of the tower. There followed
a pause. Leanne then uttered with rising pitch, "Right?" (turn 25) simulta-
neously turning her head to Bella (still continuing to saw). Leanne thereby
held Bella accountable for not having confirmed the answer, not only in
asking for a confirmation but also, by turning her gaze in the direction of
Bella, designating her as the addressee of the question. Bella did not re-
spond verbally, but, immediately following the end of Leanne's head
movement, nodded twice. Without a word, Leanne returned her head to
gaze again at the saw, board, and sawing action. Bella had answered with a
double nod, which would not have made sense had she not been aware of
the fact that Leanne could see her. Leanne's subsequent turn was an ac-
knowledgment that the request for confirmation had been satisfied, and
therefore that she had seen the nods. Bella, too, could know that Leanne
was satisfied with her non-verbal response, for she could have expected a
conversational repair if this had not been the case.
A converse case existed earlier in the episode when Bella acknowledged
the confirmation received to her query about the two sides (turn 14). Bella
was moving her head downward as part of a grooming gesture and Leanne
was turning around. Bella therefore was forced to utter "Ub okay" rather
than enacting acknowledgment in another way (head nod, eye movement,
literal "thumb up"). First, because Bella turned her gaze away from
Leanne, she was not able to see whether her peer was in the position of
seeing her gesture. Second, because Leanne was moving, she could not
have seen the gesture either. It is likely that the verbal utterance was form-
ing at the same moment that the downward head movement was. In yet
another instance (turn 10), Bella had produced a two-handed gesture while
looking down at her hands; she reproduced the gesture this time uttering
questioningly, "The two sides?" while simultaneously raising her gaze to
Leanne's face. This provided a resource for Leanne to respond by nodding
but did not require a verbal equivalent, which, here, was not provided.
These analyses exemplify that the mutual bodily orientation of the par-
ticipants mediated how communication is articulated, that is, verbally or
gesturally (e.g., hand, arm, head). Participants are not only aware of their
own positions and orientations but also of the positions and orientations of
their interlocutors. Speech is the preferred mode if there is evidence that
the listener does not have access to bodily gestures and that both speaker
and addressee are aware of this. The speaker may produce gestures when
the listener has access, even though the speaker herself does not face the
listener. These rules also mediate the pauses between verbal utterances. If
participants have full perceptual access to one another, a lot of the commu-
nicative work is generally done by producing non-verbal semiotic re-
sources, such as indicating intent to take the turn, articulating ideas, ori-
enting the communicative partner toward, and so forth.
4.3. Pointing
Pointing is classified on the spontaneous-idiosyncratic-movement end of a
gesture continuum that reaches to fully developed linguistic systems (e.g.,
American Sign Language) on the other end (Kendon 1988). Acts of point-
ing occur in situations that involve at least two participants, one of whom
is, in the act of pointing, producing resources for the joint establishment of
a particular space as a common focus (Haviland 2000). This common focus
constitutes a resource for the organization of subsequent situated action
and cognition. Pointing is jointly contextualized by semiotic resources that
include, at a minimum, a body that visibly performs the act, a dialectically
related speech act, the material properties of the in pointing targeted entity,
the orientation of relevant participants toward one another and toward the
target space, and the larger activity within which the act of pointing (itself
only part of the communicative act) is embedded (Goodwin 2003).
In this episode, there were repeated instances of verbal deixis ("that,"
"it," and "these"). One of these instances, "these," immediately followed a
deictic gesture (turn 05) and overlapped with an iconic gesture whereby the
hand moved along the piece of wood (turn 06). Such acts of pointing con-
stitute prime examples where the double nature of the object in activity
becomes apparent: from the speaker's perspective, it appears twice, once as
material and once in perception. The body oriented itself toward an entity
other than itself (piece of wood), which it was perceptually isolating from a
more inchoate ground. The body, here, repeatedly pointed to this entity,
verbally by means of the utterance "these" and gesturally by the stretched
arm and edge of the hand coming to rest on the piece of wood, the current
object of attention.
The deictic term "these" locates the entity pointed to near the speaker
(Hanks 1992), in the present case, within reach. The deictic term "these"
not only instructed Bella to attend to something beyond the speech act, the
targeted material structure, but also articulated that what she pointed to as a
multiplicity (i.e. "these" rather than "this"). Furthermore, by using "these"
rather than an articulation of a point in space (here, there), Leanne formu-
lated the target as material things rather than spaces or sounds.
There existed, however, a contradiction: there was only one piece of
wood, and Leanne' s subsequent iconic gesture produced the image of one
rather than two cuts. But her pointing gesture was not only to the piece of
wood but also to the previous interaction where she had gesturally enacted
twice the iconic gesture of two cuts. Because these previous gestures were
part of the girls' interactional history, they constituted production re-
sources "once-removed," which make connections over larger chunks of
the ongoing activity (Erickson 1982). This pointing (turn 05) therefore
indexed the earlier sawing gesture with the edge of the right hand at the
same place where she had earlier moved the saw (Figure 2).
In a continuous motion, the pointing configuration then moved along a
line about five centimeters from the edge where she had earlier indicated
that they needed to cut (Figure 2). Although most classification systems
categorize deictic and iconic gestures differently, there are situations where
the same movement enacts both types, for example, when the pointing
body part traces out some shape. Thus, Leanne's initial deictic gesture
toward the piece of wood (turn 05) also became an iconic gesture (turn 06),
because its trace took the shape of the projected linear cut. Here, move-
ment along a line that separated the object pointed to from what was to be
taken as diffuse ground enhanced salience. The single iconic gesture along
the board constituted a reproduction of an earlier gesture showing where
she was to cut the board (Figure 2). In fact, at that earlier moment, Leanne
produced the gesture twice, holding the saw where the cuts were to be
done. That is, the distance of the saw from the edge doubled for the second
movement in the first pair of gestures; in the second pair, she produced for
practical purposes identical gestures at either end of the board, again indi-
cating that they needed two pieces of board for the tower part. Despite the
iconic relation between the earlier and present gestures, the contradiction
between the gestural (one) and verbal articulations (more than one) was
only resolved in the subsequent iconic gestures (turn 07) that articulated a
vision of two upright boards, as it was further materialized in a subsequent
episode (Figure 3).
From the listener's perspective, the associated task is to perceptually
isolate the entity pointed out by the speaker's body (orientation, deictic
gesture). This is not necessarily an easy task, for even in the sparsest set-
tings, it may not be evident to the listener what the salient feature is for the
speaker. Different speakers may use the same word for denoting different
material entities or may use different words for denoting the same material
entities (e.g., Roth 1995). Other, semiotic resources are required as con-
straining factors to assist the listener in perceptually articulating the target
structure. Such constraining factors are inculcated culturally marked object
or field, or may be found in the history of the activity and associated talk.
In the present episode, the perceptual field articulated in Leanne' s pointing
was not culturally marked, but marked by having been the focus of previ-
ous interactions. By turning to the piece of wood, pointing it out as a re-
source, and then returning to the initial body orientation facing Bella,
Leanne actually brought the pieces of wood into the space between them,
where she gesturally enacted a vision of the future artifact. In the present
case, Leanne had already used repeatedly an iconic gesture to point out
where and how many times they needed to cut for getting the required
pieces in the building of the tower part. The nature of the entity pointed to
was further constrained by the talk that immediately followed, where
Leanne's two-handed palm-facing gesture articulated a vision of boards
facing one another (turn 07, Figure 3).
Figure 3. The two-handed gesture, which had fIrst appeared in a conversation
about what the tower would look like is here reproduced but here in a
holding movement, fIrst by Leanne (a), then by both students (b).
In the present situation, Leanne was not just pointing somewhere, but she
rotated her entire body. The sequence of the first four offprints in the tran-
script (turns 01-07) shows that she had first rotated away from an orienta-
tion to the board (turn 01) in order to face Bella (turn 03), then rotated back
to face the board while pointing to it (turn 06), and then rotated again to
face Bella (turn 07). She was aware that Bella shifted her gaze from facing
Leanne (turn 03) toward the board (turn 06), the endpoint of the gesture,
and then back toward the space between them, where she could see both
Leanne into the face and her two-handed gesture. This intentional pointing
therefore was instantiated not merely the hand but in fact by the entire
body (person) under the auspices of her knowing that the listener Bella was
following her changing orientations.
4.4. Iconic gesturing
Iconic gestures belong to the same class of gestures as pointing. They in-
clude those hand-arm movements that bear a perceptual relation with con-
crete or projected entities and events that the speaker is oriented to (e.g.,
McNeill 1992). In contrast to a deictic gesture, however, an iconic gesture
is itself the focal entity. From a speaker's perspective, the iconic gesture
articulates an immanent or unfolding sense that stands in a dialectical rela-
tion both with concurrent speech and the material entity in the setting so
articulated (Roth 2004). Iconic gestures are important semiotic resources in
interaction (e.g., Ochs, Jacoby and Gonzales 1994). Research in school
science classrooms showed that for students of all ages, iconic gestures are
important aspects of communication, especially while the learners are be-
coming familiar with the phenomena at hand and do not yet have a stable
(viable) language to talk about them (Roth 2003). Iconic gestures also were
important interactional resources continuously produced and reproduced in
this episode.
The first iconic gesture occurred while Leanne reminded Bella that they
were currently working on the first part ("thing") of their Rube Goldberg
machine where the elevator ("thing") moves (turn 03). The rising right
hand palm facing down evoked the image of something moving upward,
like the elevator in the design (Figure 1). More so, while they had talked
about how to realize the tower, both Leanne and Bella had gesturally fol-
lowed the vertical outline of the tower, indicated by the orientation of the
hands. These gestures therefore bore iconic relations to other material
forms, apparent as tower and elevator in the design drawing. At the verbal
level, the upward movement of the hand was iconic to the articulated
"comes up." It is a concurrent gestural and verbal concretization of the up-
down or verticality schema (Johnson 1987, Roth and Lawless 2002b). In
this case, the timing between the verbal and gestural modality was precise
(simultaneous), consistent with the contention that gesture and speech are
simultaneous formulations of an immanent or emerging idea by one and the
same body (McNeill 1992). However, the left hand only moved upward
slightly, about one-quarter the way of the right hand, and given Bella's
gaze direction, probably invisible to her.
In the same part of the episode, one can also note that the idea had be-
gun to form earlier. While uttering that they were "doing that thing,"
Leanne still had the saw in her hand, nearing the end of the body rotation.
She placed the saw and then, in a continuous motion, moved her right hand
up just as the utterance evoked an upward moving elevator. That is, the
placement of the saw was in preparation of the subsequent gesture. But
even before uttering the locative "up," her body and hand already began to
rotate sideways toward enacting the deictic gesture that pointed to the
wood to be used in the tower construction, thereby preparing the subse-
quent gesture.
After the intervening (already analyzed) pointing episode, Leanne' s
whole body turned back toward Bella and, in a continuous motion, both
hands moved upward while she uttered, "that like keep it" (turn 07). The
hands then descended halfway to the table, but, as if hesitating moved
three-quarter the width of her hand upward, before completely descending
to the table level; in the process, she completed the word "standing." The
two-handed gesture produced the image of two upright boards facing one
another, which, according to the verbal part, "keep it [elevator] standing."
The two-handed gesture not only produced an iconic expression of the
design vision, but also produced (a) an image of the two boards that pro-
vided the scaffold for the elevator in their design drawing and (b) an image
of the boards that she had just pointed to in their raw-material state.
Through the two-handed gesture, the design drawing (Figure 1) was com-
ing alive while simultaneously implicating the raw materials to be used: it
was a bodily realization of the vision. Later on, very similar hand move-
ments included pieces of wood now standing for themselves as the "sides"
(Figure 3a). At that point, the design was even further concretized but not
yet an independent artifact: Leanne's body was still implicated by holding
the pieces. Only subsequently, after adding more materials and using fas-
tening resources would the emerging design become self-supporting and
exist independent of the body. In this situation, we observe an opposite
movement to my previous research on gesture in learning science (Roth
and Lawless 2002a): here, ideas and fleeting images are initially concre-
tized in body position and hand gestures whereas the inverse was observed
in learning by experimenting.
At this point, Leanne had apparently ended her response to Bella's
query. There was a long pause in the talk. During the second half of this
pause, however, Bella reproduced a mimetic image of the two-handed
gesture Leanne had just completed - and, similarly to her peer, used the
same hand movements to hold up the two pieces of wood (Figure 3b). She
then reproduced the same two-handed gesture uttering with increasing
pitch, "The two sides?" (turn 11). Here, although the hands mimetically
reproduced Leanne's gesture, thereby confirming that she had perceived it,
the verbal utterance indicated uncertainty. Why should there be uncer-
tainty? What must have appeared in doubt was the relation to the diagram,
that is, their ongoing activity and its motive to construct a Rube Goldberg
machine, with an elevator as its first part. That is, from the receiver's part,
Leanne's production was intelligible as such (Bella reproduced it), but she
had not yet found the relation Leanne' s gestures bore to the envisioned
object of activity, partially concretized in the diagram.
Facing Bella, who looked straight at her, Leanne nodded three times;
she did not produce a verbal affirmation. As shown previously, Bella was
in a position where her sign of affirmation had to be produced verbally.
Leanne then turned around to continue where she had left off. Leanne had
articulated her version of the design in response to Bella' s query about the
sides involved in the elevator tower. Thus, although they had talked about
this before, that is, although Leanne had articulated this vision in a previ-
ous episode, there was apparently a lack of alignment or common ground
between the two. Bella's query articulated this uncertainty about the nature
of tower in what was to be their common design. The iconic and mimetic
character of Leanne's two-handed gesture thus constituted a solution to the
problem of establishing common ground with respect to an initially
ephemeral vision of the object of activity, which was increasingly concre-
tized in the unfolding of the design artifact. Bella enacted an independent
concrete version of what is taken to be the common object at hand, always
under revision in future action.
5. Body, mind and culture
In this chapter, I articulate and exemplify the embodied and situated nature
of human communication drawing of a non-dualist, dialectic framework
that integrates mind and culture, body and mind, speech and gesture, mate-
riality and conscious nature of human experience, and so on. The analyses
show that the separate, interlocking displays of conversational participants
form a unit that is greater than the sum of its parts, a particular kind of
participation framework. Bella and Leanne treated each other's bodies as
resources in the ongoing production of intentional action; Amanda, though
she was close and overheard the two, did not have access to the same re-
sources, as she faced away. For conversation participants, the body is a
complex resource that enables different kinds of articulations that stand in
a dialectically relation to, and therefore elaborate one another. The body
therefore is a different type of entity than the piece of wood Leanne
pointed to, the saw she used, or the paper-and-pencil diagram they had
collectively produced, although it is of the same material kind.
6
Partici-
pants in communicative interaction must not only attend to multiple per-
ceptual arrays (body orientation, gesture, speech, entities in the setting),
but also arrays that are significantly different in their structure.
In the featured episode, there were repeated instances of thoughts in the
process of forming concurrently with the bodily productions (speech, ges-
ture, orientation). Thus, Bella had already produced a two-handed gesture
that enacted the image of the two-walled tower before she began to utter
the question, "The two sides?" while simultaneously reproducing the two-
handed gesture. In the first instance, one sees the question forthcoming, not
yet formed in its entirety; it only became the full interactional resource
when both gesture and speech aligned in the articulation. In fact, the first
hand movement had a shorter trajectory, as if Bella had stopped it to repro-
duce it in full simultaneous with the utterance. This process was not
planned, as gesticulations are generally produced unconsciously. The con-
tinuous unfolding of a thought was also noticeable in Leanne's perform-
ance of her response to Bella's initial question. It had started with a refer-
ence to the diagram, continued by articulating the pieces of board that they
were currently working on, and then articulated the design vision that
brought the two earlier pieces together into one unit. The analyses showed
how her entire body performed this response.
The analyses also showed that the body, through forms of both sponta-
neous and idiosyncratic gesture, is tied to its environment so that commu-
nication is not only an embodied (production of speech, gesture) but also a
situated (distributed) phenomenon. In the deictic gesture, the identity of the
6. Gallagher (this volume) elaborates on the differences between the perceptions
of material objects and human bodies (that of others and one's own) and the
implications this has for intersubjectivity.
perceptual (intra-psychological) and material (extra-psychological) aspects
of the object becomes apparent and articulated. It is in a reflexive move-
ment that this identity is destroyed, that is, that subject becomes separated
from object and object from subject; the reflexive movement, however,
only returns the idea or thought but not experience of a body (Merleau-
Ponty 1945). The body is therefore not only a different type of entity, but
also it cannot be the object proper in the phenomenological or activity-
theoretic formulation. The proper analytic starting point is the dialectical
unity of activity - bodies, thoughts, experiences, and even emotions and
motivations emerge as the effects of the subject I object dialectic that con-
stitutes the activity (Roth 2006).
The analyses also provide a hint at the fact that body, mind, and culture
are inextricably related and cannot be reduced to one another. Thus, each
utterance, gesture, body orientation, and so on produced by one participant
not only provided a resource for future action to herself but also to the
other; each communicative act was explicitly produced for the other. That
is, the idea of a semiotic resource inherently requires the other participant
to be aware of the semiotic potential of these resources. At the moment that
the body articulates itself by producing semiotic resources, these already
have to exist for the recipient other. An individual bodily me that articu-
lates itself has to draw on resources that are already intelligible by the
other and therefore part of the cultural possibilities. This, however, should
not come as a surprise, if we recall that operations (and therefore schemas)
are unconscious collective consciousness. All actions are therefore synthe-
sized from operations, the concrete realizations of cultural (generalized)
possibilities, and therefore are simultaneously the person's and the collec-
tive other's.
The present analyses of one brief event testify to the complexity of
communication, which cannot be reduced to language. The analyses ar-
ticulated communication as an embodied and situated phenomenon and
showed how ideas, questions, and responses emerged in real time. The
episode analyzed here was not different than those preceding or following
it during the six-hour project, or, for that matter, any other design activity
by these students or their peers. If 23 seconds of situated activity are al-
ready so complex, how much more complex will be entire forms of activi-
ties that stretch over days, months, years, and generations? Because com-
munication, ideas, questions, and responses are distributed across different
bodily modes of expressions and across the salient aspects of the setting,
the structure of thought cannot be linguistic structure. Thought cannot be
reduced to language; nor can any other form of expression be reduced to
the body. Rather, thought exists in the dialectical unity of all forms of ex-
preSSIon.
The present analyses make salient the special role of the body, seat and
source of agency, in design activity, constituted in the unfolding ephemeral
ideas into concrete artifacts that we had also observed among environ-
mental activists (Lee and Roth 2001). The body was the first site where
ideas took shape, where they were articulated; it was then augmented by
the introduction of materials and subsequently emerging stages of the arti-
fact. Still later, the body removed itself as additional actions allowed the
artifact, the concrete rendering and development of the initial ideas, to take
on a life of its own. Communication was but one form of action that moved
the unfolding activity along its trajectory toward completion. This role of
gestures complements the findings of our earlier research (e.g. Roth 2002),
which showed the mediation gesturing plays in the development of lan-
guage following manipulations of objects. That is, hand gestures in par-
ticular again played an important role in the transition between two repre-
sentational poles, here, ephemeral design ideas sketchily articulated in
speech, on the one hand, and concrete artifacts, on the other.
There are direct practical consequences of the way of thinking about
and analyzing human communication. For example, in lecture-type situa-
tions, students generally copy whatever the lecturer notes on the inscription
device present; they may also note some of the utterances that do not get
publicly recorded. However, students generally do not record all the other
semiotic resources that the lecturers provide in their performances. It is
therefore not surprising that students experience difficulties to reproduce
the sense of what has been communicated from their notes alone. It also
should not surprise that so many students experience difficulties in under-
standing the lecture content at some later point, although during the lecture
it all had made sense. The perspective developed here suggests that these
students are in fact missing most of the resources that were available to
them in the lecture but have neither been recorded in their notebooks nor
were recoverable from the existing entries.
Acknowledgments
This study was made possible in part by a grant from the Social Sciences
and Humanities Research Council of Canada. I thank Sylvie Boutonne,
Michelle K. McGinn, and Carolyn Woszczyna for the help they provided
during data collection.
References
Bavelas, Jan B., Nicole Chovil, Linda Coates and Lori Roe
1995 Gestures specialized for dialogue. Personality and Social Psychology
Bulletin 21: 394-405.
Bourdieu, Pierre
1980 Le sens pratique. Paris: Les Editions de Minuit.
Erickson, Frederic
1982 Money tree, lasagna bush, salt and pepper: Social construction of
topical cohesion in a conversation among Italian Americans. In:
Deborah Tannen (ed.), Analyzing Discourse: Text and Talk, 43-70.
Washington, DC: Georgetown University Press.
Goodwin, Charles
2000 Action and embodiment within situated human interaction. Journal of
Pragmatics 32: 1489-1522.
2003 Pointing as situated practice. In: Sotaro Kita (ed.), Pointing: Where
Language, Culture and Cognition Meet, 217-241. Mahwah, NJ:
Lawrence ErlbaumAssociates.
Goodwin, Charles, Marjorie H. Goodwin and Malcah Yaeger-Dror
2002 Multi-modality in girls' game disputes. Journal of Pragmatics 34:
1621-1649.
Hanks, William F.
1992 The indexical ground of deictic reference. In: Alessandro Duranti and
Charles Goodwin (eds.), Rethinking Context: Language as an Inter-
active Phenomenon, 43-76. Cambridge: Cambridge University
Press.
Hamad, Stevan
1990 The symbol grounding problem. Physica D 42: 335-346.
Haviland, John B.
2000 Pointing, gesture spaces, and mental maps. In: David McNeill (ed.),
Language and Gesture, 13-46. Cambridge: Cambridge University
Press.
Hindmarsh, Jon and Christian Heath
2000 Embodied reference: A study of deixis in workplace interaction.
Journal ofPragmatics 32: 1855-1878.
Johnson, Mark
1987 The Body in the Mind: The Bodily Basis ofImagination, Reason, and
Meaning. Chicago: University of Chicago Press.
Kendon, Adam
1988 Goffman's approach to face-to-face interaction. In: Paul Drew and
Anthony Wooton (eds.), Erving Goffman: Exploring the Interaction
Order, 14-40. Cambridge: Polity Press.
Latour, Bruno and Steve Woolgar
1979 Laboratory Life: The Social Construction ofScientific Facts. Beverly
Hills, CA: Sage.
Lave, Jean
1988 Cognition in Practice: Mind, Mathematics and Culture in Everyday
Life. Cambridge: Cambridge University Press.
Lave, Jean and Etienne Wenger
1991 Situated Learning: Legitimate Peripheral Participation. Cambridge:
Lee, Stuart and Wolff-Michael Roth
2001 How ditch and drain become a healthy creek: Representations,
translations and agency during the re/design of a watershed. Social
Studies ofScience 31: 315-356.
Leont'ev, Alexei N.
1978 Activity, Consciousness and Personality. Englewood Cliffs, NJ:
Prentice Hall.
McNeill' David
1992 Hand and Mind: What Gestures Reveal about Thought. Chicago:
University of Chicago Press.
McNeill, David and Susan D. Duncan
2000 Growth points in thinking for speaking. In: David McNeill (ed.),
Language and Gesture, 141-161. Cambridge: Cambridge University
Press.
1945 Phenomenologie de la perception. Paris: Gallimard.
Mikhailov, Feliks
1980 The Riddle ofSelf. Moscow: Progress.
Ochs, Elenor, Sally Jacoby and Patrick Gonzales
1994 Interpretive journeys: How physicists talk and travel through graphic
space. Configurations 2: 151-171.
Ong, WaIter J.
1982 Orality and Literacy: The Technologizing of the Word. New York:
Routledge.
Roth, Wolff-Michael
1995 Affordances of computers in teacher-student interactions: The case of
Interactive Physics. Journal of Research in Science Teaching 32:
329-347.
2002 From action to discourse: the bridging function of gestures. Journal
ofCognitive Systems Research 3: 535-554.
2003 Gesture-speech phenomena, learning and development. Educational
Psychologist 38: 249-263.
2004 Perceptual gestalts in workplace communication. Journal of Prag-
matics.
2006 Identity as dialectic: Making and Re/making self in urban schooling.
In: Joe L. Kincheloe, kecia hayes, Karel Rose and Phil M. Anderson
(eds.), The Praeger Handbook of Urban Education: 143-153. West-
port, eT: Greenwood.
Roth, Wolff-Michael and Daniel Lawless
2002 a Signs, deixis, and the emergence of scientific explanations. Semiotica
138: 95-130.
2002 b When up is down and down is up: Body orientation, proximity and
gestures as resources for listeners. Language in Society 31: 1-28.
Sewell, William H.
1992 A theory of structure: duality, agency and transformation. American
Journal ofSociology 98: 1-29.
Streeck, Jiirgen
1996 How to do things with things: Objets trouves and symbolization.
Human Studies 19: 365-384.
ten Have, Paul
1999 Doing Conversation Analysis: A Practical Guide. London: Sage.
Uexkiill, Jakob von
1972 Theoretische Biologie. Frankfurt/M: Suhrkamp. (Originally pub-
lished in 1928)
Index
action 1,3,8,20,21,26,34,36,41,
42,45,46,58,72,109,132,136,
138, 139, 147, 152, 167, 169,
179-183,185,186,188-190,
199,200,204,205,211,214,
219,241-251,253-259,271-
275,277,287,307,313,320,
323,324,326,328,351,369,
381,382,384-388,392,394,
395,397,398,402,426,431-
436,442-445,450-453
active perception 197, 198, 204-
206,214
affordances 7,8,55,68,69,72-74,
76,98,99,186,251,257
agencYlstructure dialectic 431, 433
animal 43, 44, 48, 49, 65, 66, 69,
71,72,76,94-96,98,101,102,
134-138,140,141,168,177,
189,214,218,245,307,321,
327,365,379-381,383,384,
386,391-393,395,396,398-
400,402,403,405
biosemiotics 7, 9, 85, 101, 102, 105,
109,379,380,382,383,385-
391,394,395,400,402
bodily mimesis 219,228,231,232,
297,301,312,318,320,322-
324,326,329
body 2-4,6-9,17,18,22,24,25,
30,35,37,38,40,41,43,47,48,
55-59,62-64,67,69,70,73,75-
77,85-91,93,96,101,108-114,
116-120,130-134,137,138,
142-145, 147-149, 151-153,
169,179,180,183,189,205,
206,217,219,225,228,231,
241-243,252,253,257-259,
271-288,297,298,301,308,
312,313,318,320,325,329,
339-348,350,351,356,364-
370,379-384,388,389,392,
395-400,403,405,412,414,
416,431-435,437-447,449-453
body image 8, 217, 228, 231, 271-
283,285,288,308,313,351
body schema 8, 205, 217, 271-284,
286,288,308,313,351,434
categorization 167, 169, 175, 176,
178,190,206-208,212,214,
215,231,371,376,411,413-
415,417,426
causality 65, 314, 315, 380, 384,
386,387,393,401
cognitive linguistics 1, 6-8, 17, 85,
113,119,120,198,220,259,
298-300,304,306,308,315,
318,325,339,346,357,358
cognitive neuroscience 49, 144,
183,185,252,339,341,346,
351,361,363,369
cognitive science 1, 3-5, 7, 9, 10,
17-21,47,48,81,86,92,98,
119, 129-131, 133, 141-151,
154,168,197,200,242,250,
252,271,273,297,310,311,
313,315,316,339-341,343,
344,347-349,356,358,359,
362-364,369,370,379,382,
383,386,387,393,395,412-
414,437
communication 9, 43, 44, 46, 47,
115, 130, 131, 138, 152, 153,
208,218,223,228,252,255,
458 Index
284,307,319,320,324,381,
382,416,423,431-433,435-
437,444,448,450-453
complex systems 197, 199, 200,
231,386
conceptual spaces 8,167,168,171,
173,175,179,182,185,187,
190, 191, 197, 199
consciousness 2, 6, 7, 9, 22, 85-89,
91,97,98,100,102,107-109,
113,119,135,136,201,202,
217,219,222,228,232,275-
277,279-281,297-300,302-
304,307,309,312,313,315-
318,322-325,363,389,392,
395,398,400,402,433,434,452
conventions 297, 303-305, 310,
324,326,328,434,438
cyborg 379,380,399,402-405
deafferentation 271, 273, 277-279
dynamic categorization 197, 204,
206-208,213
ecology 7,55,56,64,75,85,100,
101
embodied cognition 1, 3-8, 10, 18,
48, 129, 131, 144, 145, 148-151,
154,206,241-243,249,256,
258,297-300,311,312,348,
356,360,364,393,431
emb0dnnent 1-4,6-9,17,18,21,
48,55,56,76,85-89,91,109,
118-120,129-133,137,143-
146, 148-153, 185, 187, 197,
206,207,230,241-243,249,
252,255,258-260,271,272,
280,284,286,288,297-301,
308,309,311-314,318,319,
322,324-329,339,340,342,
344,348-354,356-360,363,
365,366,368-370,379-386,
389,390,392,394-404,411-
414,416-418,423,425,426,
416-418,423,425,426,431,
434,453
emergence 23,86,139,219,231,
258,380,383,384,387,401,423
evolution 3, 80, 85, 114, 116, 134,
209,215,231,258,284,301,
318-322,326-328,383,387,
390,394,403,432
force dynamics 8,167,169,172,
179-183,186-191
frames of reference 339, 343, 345,
346,352,354,362-365,367-369
gesture/-s 6, 9, 129, 130, 134, 153,
202,227,228,231,241,255,
258,282-284,301,307,319-
321,323,324,326-329,362,
365,366,415,431,435,437-453
Gibson, James 7, 55, 56, 61, 64-69,
71-78,79,97-100,102,108,
122, 156, 182, 183, 186, 189,
193,198,204,205,207,234,261
history of cognitive science 129-
131, 133, 138
image schema/-s 17,21,32-37,42,
44,45,85,86,90,92,109,111,
114, 120, 167, 187, 189, 191,
198,309,314,324,325
information 7, 48, 55, 62, 68, 69,
72,76,77,105,115,142,146,
154,168,178,181-183,185,
189,227,246,249,255,315,
316,340,382,383,386-388,
390,391,394,397,412,432,435
interactive technology 129, 133,
151, 153, 154
internal meaning space 8, 197, 198,
204,214-217,219,222,223,
225,227,229,231,232
intersubjectivity 8, 133, 241, 252,
271,285-288,301,307,351,451
Japanese mimetics 197, 199, 223,
224,226-228,230
language 1,2,4,7-10,17,19,20,
22,23,31,36,37,41,42,44,45,
47,48,59,61,63,66,69,87,93,
96,99,101,104,106,107,110,
115-118, 120, 130, 131, 133,
134,139,140,148,152,173,
174,197-200,202,204,216,
218-220,223,226-228,230-
232,241,243-245,253,255-
260,283,297-301,303-314,
318-323,325-329,339,345-
347,349,350,353-355,359,
361,362,364,365,367,380-
382,393,400,402,404,411,
413-418,425,426,431,432,
437,442,448,452,453
Lifeworld 85,87,88,99-101,107,
109,110,120
memory 58,59,85,94,114-119,
142,184,203,212-214,229,
231,283,350,397,414,418,432
mental rotation 9, 250, 339, 341-
343,345-348,356,359,362,
364,365,368
metaphor 4, 5, 7, 9, 17, 20, 37-42,
48,55,61-63,98,111,129,130,
133,142,144,202,218,250,
301,308,310,312,339,357,
358,362,363,366,368,391
mimetic schemas 114, 180, 199,
203,228,231,297,301,322,
324-329
mirror neurons 8, 133, 147, 148,
150,189,217,241,251-255,
271,287,328,436
mutualism 55,75-77
Index 459
nature-culture dualism 55-57, 69,
76, 77
neonate imitation 271, 273, 279,
280,284,285
neurobiology 17,21,26,29,48,88,
91,308,319,341,387
organicism 380, 390, 391
perception 8, 23, 27, 32, 33, 35, 36,
44,60,61,73-75,78,88,95,96,
98,99,106,108,117,132,138,
147,149,167,169,170,172,
182, 183, 186, 190, 193, 195,
200,201,203-205,207,208,
214,217-219,223,226,231,
241-245,249,250,253,257-
259,271-276,278,281-284,
286,287,311,355,381,400,
402,416,418,420,421,425,
433,436,445
phenomenology 3,7,65,85-91,97,
120,150,200,206,218,271,
272,285,288,297,309,351,391
picture 5,44,67,76,85,96,98,99,
116,130,200,205,231,250,
306,311,329,348,367,387
pragmatism 17-21,32,37,39-42,
47,48,306
proprioception 55, 74, 225, 278,
279,284,286,320
radical embodiment 3, 129, 147,
149-151,386
representation 8, 9, 20, 25, 30, 43,
46,69,86,92,96,104,114,115,
144,147,149,152,168,170,
173,177-180,182-184,186,
190,197,198,200,201,203,
213,219,220,223,226,228,
230,231,244,246,252,259,
260,297-300,306,309-312,
317,322-325,350,364,367,
460 Index
369,379,380,383,388,393,
400,436
Representationalism 17, 19-21, 30-
33,35,36,42-48
self-organization 390, 411, 418,
423,425
semantics 8, 17, 167, 176, 187, 188,
191,192,223,304,349,357,
382,393
semiotic function 85, 93-95
semiotics 7, 85, 89-92, 94, 99, 105,
132,351,381,383-386,388,
393,394
sign 85,86, 92-109, 114-116, 118,
120,200,223,226,304,319,
320,381-384,386-389,391-
395,402,438,440,450
simulation theories 5, 12, 147, 241,
243,245,250,258-260
situated practice 431-433, 450-453
social interaction 7, 9, 43, 47, 48,
129-133, 137, 140, 144, 145,
147,148,151-153,252,253,
307,309,313,314,411,417,426
society 141, 381, 382, 397, 399,
400,403-405
space 8,9,23,28,29,33,38,66,
67, 86, 87, 93, 110, 113, 119,
167-178,183,185,186,188,
190,197-199,214-217,219,
224,225,231,232,281,299,
317,319,339-341,346,360,
367,369,414,417,423,426,
445-448
speech 9,47,59,140,174,181,
223,226,231,255,305-307,319,
326,411-418,425,431,434-438,
442,443,445,448,450,451,453
synaesthesia 8, 197, 199, 216-223,
225,230,231
unilateral neglect 271, 273, 277-
279

Body, Language and Mind Vol.1 - Embodiment

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Body, Language and Mind Vol.1 - Embodiment

Загружено:

Авторское право:

Доступные форматы

Body, Language and Mind

Cognitive Linguistics Research

Вам также может понравиться