APA Handbook of

Nonverbal
Communication

APA Handbook of Nonverbal Communication, edited by D. Matsumoto, H. C. Hwang,

and M. G. Frank
Copyright © 2016 American Psychological Association. All rights reserved.
 APA Handbooks in Psychology® Series

APA Handbook of Industrial and Organizational Psychology—three volumes

Sheldon Zedeck, Editor-in-Chief
APA Handbook of Ethics in Psychology—two volumes
Samuel J. Knapp, Editor-in-Chief
APA Educational Psychology Handbook—three volumes
Karen R. Harris, Steve Graham, and Tim Urdan, Editors-in-Chief
APA Handbook of Research Methods in Psychology—three volumes
Harris Cooper, Editor-in-Chief
APA Addiction Syndrome Handbook—two volumes
Howard J. Shaffer, Editor-in-Chief
APA Handbook of Counseling Psychology—two volumes
Nadya A. Fouad, Editor-in-Chief
APA Handbook of Behavior Analysis—two volumes
Gregory J. Madden, Editor-in-Chief
APA Handbook of Psychology, Religion, and Spirituality—two volumes
Kenneth I. Pargament, Editor-in-Chief
APA Handbook of Testing and Assessment in Psychology—three volumes
Kurt F. Geisinger, Editor-in-Chief
APA Handbook of Multicultural Psychology—two volumes
Frederick T. L. Leong, Editor-in-Chief
APA Handbook of Sexuality and Psychology—two volumes
Deborah L. Tolman and Lisa M. Diamond, Editors-in-Chief
APA Handbook of Personality and Social Psychology—four volumes
Mario Mikulincer and Phillip R. Shaver, Editors-in-Chief
APA Handbook of Career Intervention—two volumes
Paul J. Hartung, Mark L. Savickas, and W. Bruce Walsh, Editors-in-Chief
APA Handbook of Forensic Psychology—two volumes
Brian L. Cutler and Patricia A. Zapf, Editors-in-Chief
APA Handbook of Clinical Geropsychology—two volumes
Peter A. Lichtenberg and Benjamin T. Mast, Editors-in-Chief
APA Handbook of Human Systems Integration—one volume
Deborah A. Boehm-Davis, Francis T. Durso, and John D. Lee, Editors-in-Chief
APA Handbook of Men and Masculinities—one volume
Y. Joel Wong and Stephen R. Wester, Editors-in-Chief
APA Handbook of Psychology and Juvenile Justice—one volume
Kirk Heilbrun, Editor-in-Chief
APA Handbook of Nonverbal Communication—one volume
David Matsumoto, Hyisung C. Hwang, and Mark G. Frank, Editors-in-Chief
APA Handbooks in Psychology

APA Handbook of

Nonverbal
Communication

David Matsumoto, Hyisung C. Hwang,

and Mark G. Frank, Editors-in-Chief

American Psychological Association • Washington, DC

Copyright © 2016 by the American Psychological Association. All rights reserved. Except as permitted under the
United States Copyright Act of 1976, no part of this publication may be reproduced or distributed in any form or
by any means, including, but not limited to, the process of scanning and digitization, or stored in a database or
retrieval system, without the prior written permission of the publisher.

Published by
American Psychological Association
750 First Street, NE
Washington, DC 20002-4242
www.apa.org

To order
APA Order Department
P.O. Box 92984
Washington, DC 20090-2984
Tel: (800) 374-2721; Direct: (202) 336-5510
Fax: (202) 336-5502; TDD/TTY: (202) 336-6123
Online: www.apa.org/pubs/books/
E-mail: order@apa.org

In the U.K., Europe, Africa, and the Middle East, copies may be ordered from
American Psychological Association
3 Henrietta Street
Covent Garden, London
WC2E 8LU England

American Psychological Association Staff

Gary R. VandenBos, PhD, Publisher
Julia Frank-McNeil, Senior Director, APA Books
Theodore J. Baroody, Director, Reference, APA Books
Patricia D. Mathis, Reference Editorial Manager, APA Books
Lisa T. Corry, Project Editor, APA Books

Typeset in Berkeley by Cenveo Publisher Services, Columbia, MD

Printer: Sheridan Books, Ann Arbor, MI

Cover Designer: Naylor Design, Washington, DC

Library of Congress Cataloging-in-Publication Data

APA handbook of nonverbal communication / David Matsumoto, Hyisung C. Hwang, and

Mark G. Frank, editors-in-chief. — First edition.
   pages cm. — (APA handbooks in psychology series)
  Includes bibliographical references and index.
  ISBN 978-1-4338-1969-8 — ISBN 1-4338-1969-4
  1. Nonverbal communication. 2. Body language. 3. Facial expression. I. Matsumoto, David Ricky.
II. Hwang, Hyi Sung. III. Frank, Mark G. IV. American Psychological Association.
V. Title: American Psychological Association handbook of nonverbal communication.
  BF637.N66A63 2016
 153.6′9—dc23
2015002620

British Library Cataloguing-in-Publication Data

A CIP record is available from the British Library.

Printed in the United States of America

First Edition

http://dx.doi.org/10.1037/14669-000
Contents

About the Editors-in-Chief���������������������������������������������������������������������������������������������������� vii

Contributors��������������������������������������������������������������������������������������������������������������������������� ix
Series Preface��������������������������������������������������������������������������������������������������������������������������� xi
Foreword�������������������������������������������������������������������������������������������������������������������������������xiii
Introduction ������������������������������������������������������������������������������������������������������������������������� xix

Part I. Overview and History����������������������������������������������������������������������������������������������� 1

Chapter 1. A History of Research on Nonverbal Communication: Our Divergent
Pasts and Their Contemporary Legacies���������������������������������������������������������������� 3
Valerie Manusov
Chapter 2. T
 he Life and Times of Nonverbal Communication Theory and
Research: Past, Present, Future���������������������������������������������������������������������������� 17
Caroline F. Keating

Part II. Factors of Influence����������������������������������������������������������������������������������������������� 43

Chapter 3. Evolution and Nonverbal Communication���������������������������������������������������������� 45
Mark G. Frank and Allison Z. Shaw
Chapter 4. The Cultural Bases of Nonverbal Communication���������������������������������������������� 77
David Matsumoto and Hyisung C. Hwang
Chapter 5. The Developmental Arc of Nonverbal Communication:
Capacity and Consequence for Human Social Bonds���������������������������������������� 103
Caroline F. Keating
Chapter 6. Gender and Nonverbal Behavior������������������������������������������������������������������������ 139
Marianne LaFrance and Andrea C. Vial
Chapter 7. Personality���������������������������������������������������������������������������������������������������������� 163
Elysia R. Todd and David C. Funder

Part III. Sources of Messages������������������������������������������������������������������������������������������� 187

Chapter 8. The Physical Environment and Nonverbal Communication ���������������������������� 189
Miles L. Patterson and Susanne Quadflieg
Chapter 9. Appearance and Physiognomy �������������������������������������������������������������������������� 221
Daniel E. Re and Nicholas O. Rule

v
Contents

Chapter 10. Facial Expressions�������������������������������������������������������������������������������������������� 257

Hyisung C. Hwang and David Matsumoto
Chapter 11. The Voice: From Identity to Interactions �������������������������������������������������������� 289
Sophie Scott and Carolyn McGettigan
Chapter 12. Gesture ������������������������������������������������������������������������������������������������������������ 307
Erica A. Cartmill and Susan Goldin-Meadow
Chapter 13. Eye Behavior and Gaze ������������������������������������������������������������������������������������ 335
Reginald B. Adams Jr. and Anthony J. Nelson
Chapter 14. Signs, Signals, and Symbols in Olfactics���������������������������������������������������������� 363
Jeannette Haviland-Jones, Patricia Wilson, and Robin Freyberg
Chapter 15. The Body: Postures, Gait, Proxemics, and Haptics������������������������������������������ 387
David Matsumoto, Hyisung C. Hwang, and Mark G. Frank
Chapter 16. N
 onverbal Communication in Primates: Observational
and Experimental Approaches ������������������������������������������������������������������������ 401
Lisa A. Parr, Jérôme Micheletta, and Bridget M. Waller

Part IV. Methodology ����������������������������������������������������������������������������������������������������� 423

Chapter 17. M
 easuring the Dynamic Stream of Display: Spontaneous
and Intentional Facial Expression and Communication���������������������������������� 425
Ross Buck and Michael Miller
Chapter 18. Measuring the Voice���������������������������������������������������������������������������������������� 459
Andrew Rosenberg, Frank Enos, and Julia Hirschberg
Chapter 19. Measuring Gesture ������������������������������������������������������������������������������������������ 499
R. Breckinridge Church, Spencer D. Kelly, and Elizabeth Wakefield
Chapter 20. Measuring Eye Behavior ���������������������������������������������������������������������������������� 525
Frank M. Marchak
Chapter 21. Methods in Olfactory Research������������������������������������������������������������������������ 539
Robin Freyberg, Patricia Wilson, and Jeannette Haviland-Jones
Chapter 22. M
 easuring Body Movement: Current and Future Directions
in Proxemics and Kinesics�������������������������������������������������������������������������������� 551
Nele Dael, Nadia Bianchi-Berthouze, Andrea Kleinsmith,
and Christine Mohr
Chapter 23. Measuring Nonverbal Sensitivity���������������������������������������������������������������������� 589
Ronald E. Riggio and Annick Darioly
Index������������������������������������������������������������������������������������������������������������������������������������ 607

vi
About the Editors-in-Chief

David Matsumoto, PhD, is an internationally acclaimed author and psychologist. He received

his bachelor’s degree from the University of Michigan in 1981 with High Honors in Psychol-
ogy and Japanese. He subsequently earned his master’s degree (1983) and doctoral degree
(1986) in psychology from the University of California at Berkeley. He is currently professor of
psychology and director of the Culture and Emotion Research Laboratory at San Francisco
State University, where he has been since 1989. He is also director of Humintell, LLC, a com-
pany that provides research, consultation, and training on nonverbal behavioral analysis and
cross-cultural adaptation. Dr. Matsumoto has studied culture, emotion, social interaction, and
communication for more than 30 years. His books include well-known titles such as Culture
and Psychology, the Cambridge Dictionary of Psychology, and Cross-Cultural Research Meth-
ods in Psychology. He is the recipient of many awards and honors in the field of psychology,
including being named a G. Stanley Hall lecturer by the American Psychological Association.
He is the series editor for the Cambridge University Press series on Culture and Psychology
and former editor-in-chief for the Journal of Cross-Cultural Psychology.

Hyisung C. Hwang, PhD, is an adjunct professor at San Francisco State University. Her research
interests are emotion, nonverbal behaviors, facial expressions, and deception. She is the author
of numerous scientific articles, book chapters, and conference presentations on these topics. She
is also coeditor of the book Nonverbal Communication: Science and Applications.

Mark G. Frank, PhD, is a professor and director of the Communication Science Center at the
University at Buffalo, State University of New York, and he is also the sole proprietor of Mark
G. Frank, LLC. Dr. Frank received his doctoral degree in social psychology from Cornell Univer-
sity in 1989. Afterward, he received a National Research Service Award from the National
Institute of Mental Health to do postdoctoral research with Dr. Paul Ekman in the Psychia-
try Department at the University of California at San Francisco Medical School. In 1992, he
joined the School of Psychology at the University of New South Wales in Sydney, Australia,
where he worked for 4 years until he joined the Communication Department at Rutgers
University in New Jersey. In 2005, he accepted a position in his hometown at the School of
Informatics at the University at Buffalo, where he created and directs the Communication
Science Center. He has published numerous research articles on facial expressions, emotion,
interpersonal deception, and also violence in extremist groups. He has had research funding
from the National Science Foundation, the U.S. Department of Homeland Security, and the
U.S. Department of Defense to examine deception, aggression, and hidden emotion

vii
About the Editors-in-Chief

behaviors in checkpoint, law enforcement, and counterterrorism situations. He is also the

codeveloper of a patented automated computer system to read facial expressions, for which he won
a Visionary Innovator Award from the University at Buffalo. Dr. Frank has used these findings
to lecture, consult with, and train virtually all U.S. federal law enforcement/intelligence
agencies as well as local/state and select foreign agencies such as the Canadian Security
Intelligence Service, the Australian Federal Police, and Scotland Yard (United Kingdom).
He is also one of the original members and senior fellow of the Federal Bureau of Investiga-
tion’s Terrorism Research and Analysis Project. He has presented briefings on deception and
counterterrorism to the U.S. Congress as well as the National Academies of Sciences. He
has also given workshops to the U.S. Federal Judiciary, various state courts, and foreign judges and
magistrates.

viii
Contributors

Reginald B. Adams Jr., PhD, Department of Psychology, Pennsylvania State University,

University Park
Nadia Bianchi-Berthouze, PhD, UCL Interaction Centre, University College London,
London, United Kingdom
Ross Buck, PhD, Department of Communication, University of Connecticut, Storrs
Erica A. Cartmill, PhD, Department of Anthropology, University of California, Los Angeles
R. Breckinridge Church, PhD, Department of Psychology, Northeastern Illinois University,
Chicago
Nele Dael, PhD, Institute of Psychology, University of Lausanne, Lausanne, Switzerland
Deborah D. Danner, PhD, Department of Behavior Science and Sanders-Brown Center on
Aging, University of Kentucky, Lexington
Annick Darioly, PhD, Les Roches International School of Hotel Management, Bluche,
Switzerland
Frank Enos, PhD, D. E. Shaw Group, New York, NY
Robin Freyberg, PhD, Department of Psychology, Stern College for Women, Yeshiva
University, New York, NY
Wallace V. Friesen, PhD, University of California (Retired)
David C. Funder, PhD, Department of Psychology, University of California, Riverside
Susan Goldin-Meadow, PhD, Department of Psychology, University of Chicago, Chicago, IL
Jeannette Haviland-Jones, PhD, Professor Emeritus, Department of Psychology, Rutgers
University, Piscataway, NJ
Julia Hirschberg, PhD, Department of Computer Science, Columbia University, New York, NY
Caroline F. Keating, PhD, Department of Psychology, Colgate University, Hamilton, NY
Spencer D. Kelly, PhD, Department of Psychology, Colgate University, Hamilton, NY
Andrea Kleinsmith, PhD, Department of Computer and Information Science and
Engineering, University of Florida, Gainesville
Marianne LaFrance, PhD, Department of Psychology, Yale University, New Haven, CT
Valerie Manusov, PhD, Department of Communication, University of Washington, Seattle
Frank M. Marchak, PhD, Veridical Research and Design Corporation, Bozeman, MT
Carolyn McGettigan, PhD, Department of Psychology, University of London, Egham,
United Kingdom
Jérôme Micheletta, PhD, Department of Psychology, University of Portsmouth, Portsmouth,
United Kingdom

ix
Contributors

Michael Miller, PhD, Department of Communication, University of Connecticut, Storrs

Christine Mohr, PhD, Institute of Psychology, University of Lausanne, Lausanne, Switzerland
Anthony J. Nelson, Doctoral Candidate, Department of Psychology, Pennsylvania State
University, University Park
Lisa A. Parr, PhD, Department of Psychiatry and Behavioral Sciences, Center for
Translational Social Neuroscience, Yerkes National Primate Research Center, Emory
University, Atlanta, GA
Miles L. Patterson, PhD, Department of Psychology, University of Missouri, St. Louis
Susanne Quadflieg, PhD, School of Experimental Psychology, University of Bristol, Bristol,
England
Daniel E. Re, PhD, Social Perception and Cognition Lab, University of Toronto, Toronto,
Ontario, Canada
Ronald E. Riggio, PhD, Kravis Leadership Institute, Claremont McKenna College,
Claremont, CA
Andrew Rosenberg, PhD, Department of Computer Science, Queens College, City
University of New York, New York
Nicholas O. Rule, PhD, Department of Psychology, University of Toronto, Toronto, Ontario,
Canada
Sophie Scott, PhD, Institute of Cognitive Neuroscience, University College London,
London, United Kingdom
Allison Z. Shaw, PhD, Department of Communication, University at Buffalo, State
University of New York, Buffalo
Elysia R. Todd, PhD, Department of Psychology, University of California, Riverside
Andrea C. Vial, Doctoral Candidate, Department of Psychology, Yale University, New
Haven, CT
Elizabeth Wakefield, PhD, Department of Psychology, University of Chicago, Chicago, IL
Bridget M. Waller, PhD, Department of Psychology, University of Portsmouth, Portsmouth,
United Kingdom
Patricia Wilson, PhD, Department of Psychology, La Salle University, Philadelphia, PA

x
Series Preface

The APA Handbook of Nonverbal Communication is the 19th publication to be released in the
American Psychological Association’s APA Handbooks in Psychology® series, instituted in
2010. The series comprehends both single volumes and multivolume sets focused on core
subfields or on highly focused content areas and emerging subfields. A complete listing of
the series titles to date can be found on p. ii.
Each publication in the series is primarily formulated to address the reference interests
and needs of researchers, clinicians, and practitioners in psychology. Each also addresses
the needs of graduate students for well-organized and highly detailed supplementary texts,
whether to “fill in” their own specialty areas or to acquire solid familiarity with other spe-
cialties and emerging trends across the breadth of psychology. Many of the sets additionally
bear strong interest for professionals in pertinent complementary fields (i.e., depending on
content area), be they corporate executives and human resources personnel; psychiatrists;
doctors, nurses, and other health personnel; teachers and school administrators; counselors;
legal professionals; and so forth.
Under the direction of small and select editorial boards consisting of top scholars in the
field, with chapters authored by both senior and rising researchers and practitioners, each
reference commits to a steady focus on best science and best practice. Coverage converges
on what is currently known in the particular topical area (including basic historical reviews)
and the identification of the most pertinent sources of information in both the core and
evolving literature. Volumes and chapters alike pinpoint practical issues; probe unresolved
and controversial topics; and highlight future theoretical, research, and practice trends. The
editors provide guidance to the “dialogue” among chapters through internal cross-referencing
that demonstrates a robust integration of topic. The user is thus offered a clear understanding
of the complex interrelationships within each field.
With the imprimatur of the largest scientific and professional organization representing
psychology in the United States and the largest association of psychologists in the world, and
with content edited and authored by some of its most respected members, the APA Handbooks
in Psychology series is an indispensable and authoritative reference resource for researchers,
instructors, practitioners, and field leaders alike.

Gary R. VandenBos
APA Publisher

xi
Foreword

The table of contents of this handbook offers the reader a sufficient overview and provides
adequate overall integration. In this foreword, we have chosen to provide ideas and observa-
tions about nonverbal behavior and its scientific study on the basis of our years of involve-
ment in the field. In this narrative, we do not attempt to reference every article, presentation,
or conversation—as no amount of effort could do justice to those who contributed to our
decades of learning. We focus broadly on constructs, methods, and processes that are impor-
tant to nonverbal studies in any area. We assume some variation in terminology here, but
these same constructs and methods are discussed in more detail in the chapters that follow.
Without doubt, from the beginning of time, the survival of human and many nonhuman
species has depended on attending to nonverbal behavior. For the hunter, understanding the
patterns of the behavior of prey was essential for providing food. For the prey, detecting the
behavior of the hunter was essential to staying alive. To defeat and survive, the ancient warrior,
like the modern athlete, had to quickly study the adversary, assessing strength, agility, and
action patterns along with deceptiveness to anticipate and defend against probable actions.
Like the ancients, we hone our observational skills to understand the context and the
likely intent of the other and to interpret their physical characteristics and nonverbal
actions. We are particularly attentive to nonverbal behaviors in contexts in which a shared
language is not available, such as with infants who cannot yet speak, with dementia patients
who have lost the capacity of speech, or with animals whose communicative mode we do not
fully comprehend (see Chapter 5, this handbook, on development as well as Chapter 16 on
nonverbal behavior in nonhuman primates). Additionally, we focus on nonverbal behavior
when a speaker is unable or unwilling to relate facts, emotional states, or attitudes in words,
such as when emotionally loaded memories are repressed. Clearly, nonverbal behaviors and
patterns provide essential information when the other has the motive and intent to deceive.
In these contexts, the psychotherapist or interrogator may devote special attention to facial
expressions, hand and foot movements, and posture and vocalizations to uncover what is
not spoken in words.
The distinctions between communicative, indicative, and instrumental nonverbal behav-
iors are important when designing nonverbal research and examining the results of reported
research. A primary difference between these actions is conscious intent. Communicative
and instrumental actions are generally intentional. In contrast, indicative actions provide the
if, when, and how communicative or instrumental actions are performed. As discussed later,
these indicative behaviors may or may not be performed with conscious intent. For example,

xiii
Foreword

upon waking from a night’s sleep, whether one first walks to the bathroom or to the kitchen
is indicative of which need is most urgent. Likewise, the speed of pace to one’s destination is
indicative of urgency. In other words, the if, when, and how of early morning instrumental
actions carry indicative information. Although such mundane actions are useful for illustrat-
ing the conceptualization of nonverbal behavior, they are rarely, if ever, the focus of scien-
tific study.
Communicative acts range from highly ritualized behaviors (e.g., Matsumoto & Hwang,
2013) to unique improvisations. The taxonomy of hand movements during interaction
offered by Ekman and Friesen (1972) applies equally to facial actions and vocal behavior
and, to a limited degree, to leg and foot movements and posture. Actions in these modalities
may emphasize or illustrate speech as well as substitute for words. Likewise, these modali-
ties can be used to regulate the back-and-forth flow of speaker turns in face-to-face interac-
tion. Which modality is used for communicative action is largely dependent on context, in
particular, the accessibility of the recipient’s senses, and the expectation of the recipient’s
ability to decode the message. Indicative acts potentially reveal inner emotional or attitudinal
states and can be expressed through these modalities: facial expressions; head and eye posi-
tions and movements; hand, arm, leg, and foot positions and movements; torso posture; and
the vocal modulation of speech (see Chapters 10, 11, 12, 13, and 15, this handbook). It is
important to note that these modalities involve muscles that may be innervated without the
actor’s intent yet may also be controlled intentionally and deliberately by the sender to com-
municate an accurate or deceptive message. Thus, what is observed in another’s nonverbal
behavior may indicate an unspoken inner state or may communicate the sender’s intended
performance in much the same manner as the spoken word. In general, as receivers, past
knowledge of the sender or of the current context facilitates determination of the sender’s
intent.
Physical characteristics are relatively static and free of intentionality but, nevertheless,
create impressions for the viewer that may or may not be accurate. Throughout a lifetime,
maturation and aging alter physical appearance, sometimes dramatically. Dyes, makeup,
facial hair, manner of dress, and even cosmetic surgery may alter appearance. Physical char-
acteristics of gender, race, or ethnicity are the core of bigotry, intolerance, and misguided
expectations. When designing nonverbal research or interpreting findings, the degree to
which such beliefs are shared in a study group must be anticipated, and the generalization
of research findings must be limited accordingly. Unfortunately, shared beliefs about how
physical characteristics are indicative of the personality or character of an individual may
influence the perceptions of nonverbal behaviors in ways that are not anticipated by the
researcher. This is particularly problematic in studies that rely exclusively on the judgments
of observers to determine the meaning of nonverbal behaviors.
Collective wisdom about nonverbal behavior took large steps with the writings of Sigmund
Freud and Charles Darwin during the second half of the 19th century and the early 20th
century (see also Chapter 3, this handbook, on evolution). Whereas Darwin (1872) specifi-
cally described the facial expressions of six innate, basic emotions and how facial expres-
sions of emotion evolved from functional, survival actions of the facial muscles, Freud was
less specific, but equally inspiring, in finding repressed and suppressed inner states oozing
from every pore of the body (Freud & Riviere, 1935). When a specific nonverbal behavior
is observed as indicative of an internal state, there is a temptation to assign concrete mean-
ing and apply that meaning across persons and contexts. However, such generalizations
rarely stand the test of scientific investigation. As with most categorical distinctions, the

xiv
 Foreword

boundaries between indicative, communicative, and instrumental nonverbal actions are

often blurred.
During the 20th century, perhaps the most astute observers of nonverbal behaviors and
how they are interpreted have been drama coaches, theater directors, movie producers,
and directors. In theater, attention to what is communicated about an attitude, emotion, or
intention evolved as the audience was brought closer to the actor through advances in the
recording of sound and visual images. Improved technology created the need for actions
to become increasingly accessible and refined. Early theater actors exaggerated nonverbal
behaviors to communicate to a distant audience. Early movies built upon these exaggera-
tions to convey emotions, attitudes, and intentions until the actors could be brought closer
to the audience and the spoken word could be recorded and synchronized with nonverbal
actions. Increasingly, it became the task of the drama coach and movie director to attend to
what nonverbal behaviors communicated specific messages and to demand of their actors
that these actions be accurately performed and coordinated with what was said. Actors were
trained to simulate the spontaneous, indicative nonverbal behaviors that conveyed emo-
tions as well as confidence, timidity, honesty, and deceptiveness. Actors whose stature and
physiognomy conveyed particular nonverbal messages were often selected for roles when
these traits were advantageous. Taking their cue from Hollywood and Broadway, politicians,
business leaders, and other public figures sought training from coaches whose astute obser-
vations and teaching techniques would allow them to convey a desired image and to avoid
unintentionally revealing their actual intentions, beliefs, and attitudes when this disclosure
was disadvantageous to them.
The scientific study of nonverbal behavior progressed as the recording and preservation
of visual and auditory evidence became accessible and as techniques were developed for
objectively and comprehensively measuring facial behavior (Facial Action Coding System;
see Ekman & Friesen, 1978) and later other behavioral channels (see also Chapter 17, this
handbook, on facial measurement). Although there has been progress, comparable compre-
hensive measurement technology is available for measuring the vocal characteristics of fun-
damental pitch, pitch variability, and speed of speech, but measurement tools are needed for
the actions and positions of the hands, feet, legs, and torso (see also Chapters 18 and 22, this
handbook). Moreover, these measurement tools need to examine how age, gender, and race
may alter veridical measurement.
Despite advances in technology and measurement, human beings still must detect
nuances in nonverbal behavior and then derive the meaning and messages (or both) from
those behaviors. This process is fraught with imagined meanings of messages, actual mean-
ings of messages, or some combination therein. The field of nonverbal communication still
suffers from these interpretational issues. A lesson on the importance of the observations,
understanding, and interpretation of nonverbal behavior was quite unexpectedly offered to
one of the authors by a nonhuman species whose communication skills turned out to be sur-
prisingly refined. The lessons of this encounter offer examples of the variety of information
from nonverbal actions that are possible when verbal exchange is not possible and the desire
to communicate is strong.

THE SPARROW AND HIS LESSONS IN NONVERBAL COMMUNICATION

When checking one of our backyard fountains, I spotted a sparrow floating in the black
water of the catchment pool, opening and closing his yellow beak, silently screaming for

xv
Foreword

help. Quickly, I picked him up and while stroking his head and back carried him to a sunny
area of the patio while blowing on him to dry his feathers. Realizing that blowing on his
soaked feathers would not dry him quickly enough to save his life, I put him on a patio
bench and told him to stay there while I went for a hair dryer. After making certain the
dryer was blowing only warm and not hot air, I directed the air on his head, back, sides, and
underside. After a while, he listed to one side. Alarmed that he was losing his strength and
would die, I gently stroked his head and back and quietly spoke words of encouragement.
He responded by lifting the wing opposite to the direction of his leaning, allowing me to
direct the air to the underside of his wing and the exposed side of his body. Any doubt about
his intent was removed when he shifted his weight to the other side and lifted his other wing
so that I could dry the underside of the other wing. After 20 to 30 min, he began fluttering
his wings, so I carried him to the bush where the flock of birds roost and placed him on a
limb in the sun, instructing him to stay there and rest for a while. Stroking his head and
back, I assured him that he would be okay. When checking the bush, after 10 min or so he
was still on the limb in the sun, but later he had moved into the thicket out of sight.
That evening as I sat on the patio where I could see the fountain where the sparrow had
almost drowned, a sparrow flew toward me then changed direction and flew to the edge of
the catchment pool. Standing on the edge of the pool he rapidly nodded his head six to eight
times in the direction of the exact spot where I had rescued him from the water. He then
flew to the top of the fountain, then to the bush, 25 ft away. During hours of watching birds,
I had never before observed that many rapid head bobs, much less head bobs directed at any-
thing other than food. I integrated the unusual actions of the sparrow with previous informa-
tion and understood the sparrow’s intended message to be as follows: “I’m the sparrow you
rescued from this water, thank you, and I am okay.”
First, the sparrow made certain that I could see the actions of his message and took a
flight path that caught my attention, then he combined the most basic components of non-
verbal messages—getting the other’s attention, pointing (head bobs directed at the area of
focus; “I’m the bird you rescued from this place”), and instrumental actions (flying; “I’m
okay”). Although he did not use the basic component of vocalizing, his initial flight path
from the bush to the fountain’s edge was sufficient to get my attention. At other times, birds
have flown from the bushes to the fountain for a drink of water, but his path was directly
toward me before veering sharply off toward the fountain.
While I was drying my sparrow, he used instrumental actions and nonaction to convey
that he understood my intent and trusted my actions. He made no attempt to leave the bench
during the several minutes it took to go into the house for the hair dryer, and the noise from
the hair dryer did not cause him to move away. He understood that drying only the top of
his wings and body was insufficient for him to fly again and that I would understand if he
lifted his wings. Most important of all, seeing me he used the only modality (his beak) to get
my attention. Also, he trusted my holding and stroking him—understanding my intent to
help, not hurt. This alone is remarkable, as it is unlikely that he had ever been touched by a
human prior to being lifted from the water.
For the next two evenings, two sparrows flew from the bushes to a small tree near where
I was sitting and for several minutes sang a duet. In all the hours I have spent watching the
birds at the distant feeders, no bird had come to that tree for more than a few seconds, and
none had ever made a sound prior to these evening performances. In the context of the pre-
vious improvised communicative actions, the duet seemed to be a song of thanks from my
sparrow and from one of his friends who he must have told it was safe to be as close as the

xvi
 Foreword

small tree and to stay a few minutes while joining him in song. I have no way to distinguish
one sparrow from the flock that eats at the feeders, and I have no idea how they communi-
cate with each other. However, the uniqueness of my sparrow’s actions in the unique context
of the rescue leaves little doubt for me that my sparrow intended to communicate with me
and did so successfully.

HOW THE SPARROW CAN INFORM THEORY AND RESEARCH IN

NONVERBAL COMMUNICATION
Before dismissing this story of the sparrow as an anthropomorphic fable, let’s look at what
we can learn from it. My sparrow offered two possible lessons: First, conceptualizing emo-
tion as simply fight and flight is too simplistic for even the common sparrow, and, second,
communication is important, whether or not essential for survival, and can be accomplished
nonverbally with great success. This capacity of the bird is supported by a 10-year study
of the bird brain in which more similarities than differences were found between the bird
and the mammalian and human brains (Chen, Winkler, Pfenning, & Jarvis, 2013; Jarvis
et al., 2013; Karten et al., 2013; Montiel & Molnár, 2013). Not only is it clear that a bird has
the capacity of the oversimplified flight/fight concept of emotion but the recent evidence
strongly suggests that a bird’s brain has the capacity for much higher functions, such as dis-
cerning the difference between a threat and help situation.
Regardless of the bird’s capacity or intent to communicate, we remain confronted with
the veridicality of our interpretation of the sparrow’s behavior. This is a problem with
human nonverbal behaviors when a study relies exclusively on the perceivers’ interpreta-
tions and remains regardless of the sample size of the perceivers. Large groups share miscon-
ceptions and myths about physical characteristics of a sender and possibly even his or her
nonverbal actions. With the coaches and directors of a performance, the popular acceptance
of the performance might be used to confirm the director’s accuracy about the nonverbal
behaviors. However, the popular acceptance of the performance may be due more to the plot
than the embedded nonverbal performance, and the director’s claims of expertise may be
unwarranted. The stream of nonverbal behaviors is rapid, with each event often occurring
in mere seconds and creating a microcontext for succeeding events. The perceptions of the
receiver(s) do not measure the nonverbal behaviors but merely reflect the understanding of
the perceiver. Unfortunately, the internal context or subjective consciousness of perceivers
that influence understanding is currently not measurable beyond self-report, which is the
most unreliable of the semireliable means to measure such things.
The same is the case for the sender. The lack of reliability of self-report has made intro-
spection a limited source of data. A lack of reliability is inevitable, as the task of reporting
one’s internal status in retrospect is a context that cannot be duplicated when asked to repeat
the self-report at another time or place. Nevertheless, for the scientific study of nonverbal
behaviors, such as emotion-related responses to external stimuli, it is probably beneficial to
return to the advice of the founders of psychology and to use trained introspection as repu-
table data. In this manner, it may be possible to come closer to understanding the internal
psychological status of both sender and receiver when studying nonverbal communication.
This technique, of course, has its advantages and disadvantages. We may gain better insight
into humans, but we may also end up with no reliable scientific means to verify that insight.
Thus, it is important, despite our technological and methodological advances, not to lose
sight of our common humanity and the essence of any given human being as exposed by his

xvii
Foreword

or her nonverbal communication. One of the earliest lessons in our study of psychology was
the difference between reliability and validity, yet a lack of reliability has been used to reject
the potential validity of data that might prove valuable. Finding the conceptual level that
balances validity and reliability of self-report remains one of the most challenging issues for
future studies of nonverbal behavior. Fortunately, psychology has solved similar issues that
involved complex human behavior. Yet, this issue cannot be solved if attempts are automati-
cally rejected on the basis of earlier attempts that failed to find reliability or if there is only
curiosity about the subjective consciousness when an individual exhibits psychopathology
or criminal behavior.
With millions of people having ready access to the recording of human behavior in natural
contexts, there will be a wealth of recorded behavior, and automation for the measuring of
behavior will help define the variety and limits of nonverbal behaviors. In the not too distant
future, there will be opportunities to use what is learned from this handbook to study sponta-
neous nonverbal behavior in contexts not previously accessible. We hope that preconceived
assumptions will not prevent the exploration of the inner workings of the minds of those whose
nonverbal actions are studied and for those who react to the behaviors they see and hear.

Wallace V. Friesen and Deborah D. Danner

References
Chen, C.-C., Winkler, C. M., Pfenning, A. R., & Jarvis, E. D. (2013). Molecular profiling of the
developing avian telencephalon: Regional timing and brain subdivision continuities. Journal of
Comparative Neurology, 521, 3666–3701. http://dx.doi.org/10.1002/cne.23406
Darwin, C. (1872). The expression of emotion in man and animals. New York, NY: Oxford University
Press.
Ekman, P., & Friesen, W. V. (1972). Hand movements. Journal of Communication, 22, 353–374. http://
dx.doi.org/10.1111/j.1460-2466.1972.tb00163.x
Ekman, P., & Friesen, W. V. (1978). Facial Action Coding System: Investigator’s guide. Palo Alto, CA:
Consulting Psychologists Press.
Freud, S., & Riviere, J. (1935). A general introduction to psycho-analysis: A course of twenty-eight
lectures delivered at the University of Vienna. New York, NY: Liveright.
Jarvis, E. D., Yu, J., Rivas, M. V., Horita, H., Feenders, G., Whitney, O., . . . Wada, K. (2013).
Global view of the functional molecular organization of the avian cerebrum: Mirror images and
functional columns. Journal of Comparative Neurology, 521, 3614–3665. http://dx.doi.org/10.1002/
cne.23404
Karten, H. J., Brzozowska-Prechtl, A., Lovell, P. V., Tang, D. D., Mello, C. V., Wang, H., & Mitra,
P. P. (2013). Digital atlas of the zebra finch (Taeniopygia guttata) brain: A high-resolution photo
atlas. Journal of Comparative Neurology, 521, 3702–3715. http://dx.doi.org/10.1002/cne.23443
Matsumoto, D., & Hwang, H. C. (2013). Cultural similarities and differences in emblematic gestures.
Journal of Nonverbal Behavior, 37, 1–27. http://dx.doi.org/10.1007/s10919-012-0143-8
Montiel, J. F., & Molnár, Z. (2013). The impact of gene expression analysis on evolving views of avian
brain organization. Journal of Comparative Neurology, 521, 3604–3613. http://dx.doi.org/10.1002/
cne.23403

xviii
Introduction

Nonverbal communication (NVC) has been referred to by many labels in the past, one of
the most popular of which is body language, a term that has been widespread since the pub-
lication of Julius Fast’s (1970) book of the same name years ago. Researchers, however,
have defined NVC differently, embracing the idea that NVC encompasses almost all of
human communication except the spoken or written word (Knapp, 1972). In this hand-
book, we define NVC as the transfer and exchange of messages in any and all modalities
that do not involve words. As such, NVC refers to a broad array of channels, sources, and
messages that inform and influence the thoughts, emotions, and behaviors of others. Two
important ways of understanding NVC—and that frame how we approached structuring
this handbook—concern knowing the sources of the different types of messages that can be
communicated nonverbally and understanding the functions of those messages. We describe
these next and refer back to them later when discussing the organization and structure of
this handbook.

THE SOURCES OF Nonverbal communication

Although there are many ways of categorizing the multiple sources of messages that com-
pose NVC, we broadly arrange them into three categories. One source of messages is the
environment or context. Nonverbal messages communicated by an environment can help
guide the behaviors that occur within that environment. For example, different places send
different messages about their occupants and about what kinds of behaviors are appropriate.
This is accomplished through the use of color, lighting, heat, fabric textures, photographs,
type of furniture, layout, and so forth. The effects of these aspects of the environmental
context can be seen in houses, restaurants, churches, casinos, and all other kinds of person-
made enclosures. Fast-food restaurants use active, bright colors—such as orange, yellow,
and red—in a well-lit environment with hard plastic seating, sending subtle messages that
urge diners to eat more food more quickly and not to lounge around too much afterward.
In contrast, fine-dining restaurants use dimmer lighting, softer and darker colors, and more
comfortable chairs to give a more intimate impression, subtly urging diners to feel comfort-
able and stay around for dessert and coffee. Designers of gambling casinos also know well
about the power of creating an environment to send a message; there is a reason why casinos
are usually dark, with lots of colorful lights, ringing sounds, and no clocks: Patrons can just
lose themselves and their sense of time and stay as long as possible. Not surprisingly, people

xix
Introduction

can make relatively accurate judgments of the occupants’ personalities just by viewing a
room (Gosling, Ko, Mannarelli, & Morris, 2002). Much of the existing knowledge in this
area is reviewed in Chapter 9 of this handbook.
Another source of nonverbal messages is one’s physical characteristics or appearance.
These refer to the static physical appearance or smell of a person, including one’s height and
weight; skin color; hair; eyebrows; cheeks; chin; proportion of eye, nose, and chin size; as
well as odors. For years, within this framework, psychologists and laypersons alike believed
that different body types were predictive of personality, despite the lack of reliable scientific
or empirical evidence. On the basis of social Darwinism, for example, different body types
denoted different types of personalities: endomorphs (heavier, obese, rounder, softer looking)
were sociable and pleasant, mesomorphs (angular, muscular, harder looking) were leaders
and strong-willed, and ectomorphs (thin, frail, brittle looking) were withdrawn, smart, and
nervous (Sheldon, 1940). Even today, mass media capitalize on this perceived linkage by
casting actors, news anchors, and so on accordingly.
One physical characteristic that has received considerable attention is the face, and stud-
ies in this area have examined the relationship between facial structure or physiognomy (as
opposed to facial expressions, which are produced by movements of the facial muscles) and
judgments of personal characteristics. Chinese face reading, for example, is based on obser-
vations of the structure of a person’s face. The Chinese are not the only ones to do this: In
the late 1800s, Europeans believed that they could characterize criminal personalities on
the basis of the heaviness of one’s eyebrows and jaw (Gould, 1981), although there is no
evidence that one can accurately identify criminals by their facial appearance. Research in
the 1980s found that adult humans who have more “baby faces”—a higher forehead, propor-
tionally larger eyes, and smaller nose—are seen as more naive and honest, and they are less
likely to be picked as leaders (Berry & McArthur, 1986). Since that time, much research has
been conducted, and this is reviewed in Chapter 9 of this handbook.
Another physical characteristic that has received increasing research attention is odor.
Odors also send messages, both at a conscious and unconscious level. At a conscious level,
perfumes, aftershaves, and body odor send messages about hygiene in North America,
although such messages are not so clear in other cultures. At a nonconscious level, humans
send pheromones that, when placed under the nose of a woman or man, send signals of
greater attractiveness and appeal. Infants can also recognize the smell of their mothers and
will show strong preferences for items that smell of mom. Many adults will also note how
they are comforted by the smell of loved ones (reviewed in Knapp & Hall, 2006). Much of
the research in this area is reviewed in Chapter 14 of this handbook.
Physical appearance clues also include what are termed artifactual clues, such as jewelry,
clothes, glasses, and so forth. People wearing glasses are seen as being smarter. Jewelry sends
messages about one’s socioeconomic or marital status. For example, North Americans signal
their married status by wearing a ring on their left “ring” finger, whereas Europeans often
wear this signal on the right ring finger. Clothing also sends messages about income, group
membership, and even respect for others. At a formal event, most people would judge a per-
son who wears a t-shirt and jeans differently than a person who wears a suit.
A third source of NVC occurs in the dynamic actions of the face, voice, and body. These
are known as nonverbal behaviors (NVBs) and include the behaviors that occur during
communication or interaction episodes that do not include verbal language. Messages are
transmitted through multiple nonverbal channels, which include facial expressions, voice,
gestures, body postures, interpersonal distance, touching, and gaze. We call these channels

xx
 Introduction

because, like channels on a television, they are each capable of sending their own distinct
message. Biology, learning, and culture all influence these actions, which is why we com-
missioned chapters on evolution, culture, and development (see Chapters 3, 4, and 5, this
handbook).
Conceptually, NVC and NVB are often confused with each other, and researchers and
laypersons often use the terms interchangeably. We consider NVC to be a broader category
than NVB, encompassing the way one dresses, the placement of one’s office within a larger
building, the use of time, the sweat stains in one’s armpits, the distance people stand when
they converse, or the design and arrangement of one’s room (Henley, 1977). In contrast,
NVB is a subcategory of NVC and refers more specifically to the dynamic actions and behav-
iors that occur when people are interacting with one another or with the environment.

THE FUNCTIONS OF Nonverbal communication

There are many ways of understanding and classifying the various functions of NVC; in
this handbook, we do so in four ways: First, NVC can define communication by providing
the backdrop for communication and by explaining or characterizing the context or setting
within which people will interact and behave. For instance, a quiet, dimly lit room suggests
to people that interactions should be subdued (church, mosque, temple). Brightly lit rooms,
with active colors such as yellow and orange, communicate active, upbeat activities.
Second, NVC can comment on verbal communication—that is, the actual words
used—because NVB can occur when people are also talking. NVB can supplement
information missing in the words, complement the information in the words, qualify
the verbal information, or contradict the verbal information. Each of these different
verbal–nonverbal combinations has different implications for what is going on in the com-
municator’s mind. These various combinations make communication not only interesting
but also complex.
Third, NVC can regulate our interaction episodes. Much of our conversations are regu-
lated by nonverbal cues so subtle that the average person does not notice them. Nodding,
smiling, looking concerned or empathetic are all NVBs that occur during conversations and
signal to the talker that the listener is listening and tracking the conversation. The “umms,”
“ahhs,” and other nonverbal signals that occur during conversations are called back channel
communication because they are not the main focus of communication; instead, they function
at the periphery of communication. Turn-taking is regulated by NVBs: When people fin-
ish talking, they drop their voice tone and dynamics to let the listener know they have been
given the floor.
Finally, NVC can be the message itself because it can occur without any words being spo-
ken simultaneously. A smile often indicates joy, pleasantness, or politeness. A frown indi-
cates unhappiness. A wave of the hand signifies “goodbye.” Raising your index finger to your
lips signifies “shhh” or “be quiet.” None of these actions require any words, thus highlight-
ing one of the important functions of NVC.
Understanding the functions of NVC requires one to consider the function of commu-
nication itself. We believe that the function of communication is to allow for the sharing of
social intentions, which facilitates social coordination. The overall function of NVC, there-
fore, is to facilitate this overall purpose of communication—to assist in the sharing of social
intentions and to facilitate social coordination. NVC is phylogenetically older than verbal
communication (see Chapter 3, this handbook) and fulfills this function in many animal

xxi
Introduction

species. For this reason, it plays a crucial role in the maintenance of any society and culture
(see Chapters 4 and 16, this handbook).

OVERVIEW AND ORGANIZATION OF THIS HANDBOOK

Scholars have long acknowledged the crucial importance of NVC and NVB, and they have
long been topics of scientific inquiry and writing. Over the past few decades, there have
been scores of studies on the sources and functions of NVC and NVB, and in many different
contexts. Correspondingly, several scholarly titles on NVC and NVB have appeared over the
years, including a few handbooks published by notable publishers. These handbooks have
covered topics such as research methodologies in studying NVC/NVB, theoretical founda-
tions of NVC/NVB, the factors of influence and the functions of NVC/NVB, and the various
contexts and consequences in which NVC and NVB occur.
Thus, there is a need and a place for state-of-the-art, scholarly presentations, reviews, and
theories of the research world of NVC and NVB. Studies in this area are booming, and new
and improved technologies that allow for the recording, capture, and analysis of behaviors
continually push the field into new findings and new directions. For that reason, the American
Psychological Association (APA) commissioned us to produce this handbook, a project on
which we gladly and wholeheartedly embarked.
Such an endeavor cannot occur without much thought and planning. Our first and
foremost consideration was our intended audience. Without a doubt, we have organized
this handbook as an academically based, scholarly work, whose primary audience is
researchers specializing in this area. As such, we anticipate that the work can and should
be used as a primary resource by fellow researchers as well as in graduate-level classes on
NVC or NVB.
Given that audience, we then considered what would be the most compelling and
important information for that audience to have. The answer to that question was previewed
earlier. We felt that the most crucial information for scholars to have in a reference book
would be material that explored the sources of NVC messages as well as the functions of
those messages in depth. As a scholarly reference for researchers, we wanted the book to
have more than a cursory coverage of the ever-evolving research methodologies associated
with the study of NVC and NVB. Moreover, we wanted researchers to have a good idea of
the history of research in this area to pay respect and homage to the pioneers in the area
and their works. Unfortunately, the political climate of today’s psychological sciences often
encourages contemporary researchers to forget our history and to ignore the vast literatures
that preceded us. We wanted to take steps to correct that.
Thus, we organized this handbook around four broad themes, each of which led to a dif-
ferent section in this handbook. The first concerns the history of the field: Overview and
History, which includes two chapters providing an overview and history of the area, written
by very senior researchers (Valerie Manusov and Caroline F. Keating) with many years of
experience. Indeed, it was an honor to have the contribution of these senior researchers and
pioneers to delineate the background of the field of NVC.
The second theme that we considered concerns the factors of influence of NVC and NVB.
The study of NVC and NVB has encompassed and affected many different theoretical and
foundational perspectives in psychology, and it is impossible today to understand the import
of NVC and NVB fully without equally acknowledging and comprehending the vast theoreti-
cal and conceptual frameworks within which it occurs. For that reason, we present

xxii
 Introduction

five chapters in Part II, Factors of Influence, that discuss the important roles of evolution
and phylogeny (Chapter 3), culture (Chapter 4), development and ontogeny (Chapter 5),
gender (Chapter 6), and personality (Chapter 7). All of the chapter authors were asked to
provide a broad theoretical and conceptual framework to understand how each of their
respective topics influenced, and is influenced by, NVC and NVB.
The third theme that we considered concerns the separate sources of NVC and NVB that
have been studied in the past. Given our understanding of the three sources of messages
concerning NVC described earlier, in Part III, Sources of Messages, we included a chapter on
the physical environment (Chapter 8), a chapter on appearance and physiognomy (Chapter 9),
and a chapter on olfactics and odor (Chapter 14). We also included chapters on each of the
specific NVB channels for which there was a sufficient body of research from which to pro-
vide state-of-the-art reviews. These include reviews of facial expressions (Chapter 10), voice
(Chapter 11), gesture (Chapter 12), and eye behavior and gaze (Chapter 13). We condensed
research on postures, gait, proxemics, and haptics into one chapter on the body (Chapter 15).
We are particularly excited to include a chapter on NVC in nonhuman primates, a burgeoning
area of research (Chapter 16). All of the chapter authors were asked to provide a state-of-
the-art review of the main findings in the scholarly literatures in their areas as well as a road-
map for future research that would overcome current empirical or theoretical limitations.
Finally, given our intended audience of researchers, we would be remiss without having
a section on methodology. We did not, however, want to give only superficial coverage to
methods by including only a single chapter. For that reason, in Part IV, Methodology, we
present seven chapters on research methods that are specific to the various channels of NVB
covered in Part III. Here, readers will find chapters describing methods for measuring and
analyzing facial expressions (Chapter 17), the voice (Chapter 18), gesture (Chapter 19), eye
behavior (Chapter 20), olfactics (Chapter 21), body movement (Chapter 22), and nonverbal
sensitivity (Chapter 23). These chapter authors were asked not only to provide measurement
and analysis overviews and guidelines for researchers in these areas but also to review the
state-of-the-art technologies that may currently exist that allow for recording or analysis of
each of the various channels of behavior.
What readers will not find in this coverage are the important studies in various different
contexts in which NVC and NVB occur, such as within dyads, relationships, marriages,
health care settings, education, the workplace and organizations, and the like. They will also
not find chapters that focus exclusively on specific applications of NVC and NVB, such as
rapport building, therapist–client interactions, or deception. Their noninclusion does not
indicate that we think that these areas of study are not important in their own right. They
are certainly important. We did, however, decide that the material described earlier was nec-
essary and sufficient to form the contents of a foundational work on NVC in the APA Hand-
books in Psychology® series, and for that reason we limited ourselves to that selection.

ACKNOWLEDGMENTS
There are so many people who made this handbook possible and to whom we want to offer
our thanks. First, we would like to extend our deep appreciation and gratitude to the con-
tributors, who gave their time and effort to provide us with chapters. Their expertise gave
this book a special meaning that readers will not find anywhere else, and the authors all went
above and beyond the call of duty not only in drafting their chapters but in working with us
through a very detailed editing process that required sometimes multiple revisions to get to

xxiii
Introduction

the best format for readers to enjoy and from which to benefit most. We truly hope that the
readers of this handbook will recognize the great insights and experiences that the authors
bring to the work and that they will be inspired to do better science in the future. We are
confident that the content reported here cannot be found anywhere else in a single volume.
We also would like to thank all of the staff at APA. Lisa Corry has been a gem throughout
the entire process, keeping all of us—editors and chapter authors alike—in line with her
amazing project-coordination skills. APA Books Director of Reference, Ted Baroody, has
provided useful guidance in all aspects of the project, and he has been an invaluable resource
to us throughout the entire project. We also appreciate the support of our acquisitions editor,
Maureen Adams, as well as the APA production staff, especially Anna Reinhart, our produc-
tion editor.
Although there are many people in our lives who have contributed to the creation of our
ideas and the conduct of our research—and so many who have contributed to the planning,
writing, production, and distribution of this handbook—any mistakes that are in it are ours
and only ours.
David Matsumoto, Hyisung C. Hwang, and Mark G. Frank
Editors-in-Chief

References
Berry, D. S., & McArthur, L. Z. (1986). Perceiving character in faces: The impact of age-related
craniofacial changes on social perception. Psychological Bulletin, 100, 3–18. http://dx.doi.
org/10.1037/0033-2909.100.1.3
Fast, J. (1970). Body language. New York, NY: M. Evans.
Gosling, S. D., Ko, S. J., Mannarelli, T., & Morris, M. E. (2002). A room with a cue: Personality
judgments based on offices and bedrooms. Journal of Personality and Social Psychology, 82, 379–398.
http://dx.doi.org/10.1037/0022-3514.82.3.379
Gould, S. J. (1981). The mismeasure of man. New York, NY: Norton.
Henley, N. M. (1977). Body politics: Power, sex, and nonverbal communication. Englewood Cliffs, NJ:
Prentice Hall.
Knapp, M. L. (1972). Nonverbal communication in human interaction. New York, NY: Holt, Rinehart &
Winston.
Knapp, M. L., & Hall, J. A. (2006). Nonverbal communication in human interaction (6th ed.). New
York, NY: Harcourt Brace.
Sheldon, W. H. (1940). The varieties of human physique: An introduction to constitutional psychology.
New York, NY: Harper & Brothers.

xxiv
 Part I

Overview and History

 Chapter 1
A History of Research on
Nonverbal Communication:
Our Divergent Pasts and Their
Contemporary Legacies
Valerie Manusov
We are fortunate as nonverbal communication
scholars to be a part of a truly interdisciplinary
endeavor. Whereas most contemporary researchers
contributing to our large and diverse area have their
disciplinary homes in psychology and communica-
tion, others whose work informs the study of non-
verbal communication come from fields that range
from anthropology to zoology. This set of traditions
provides rich and fertile soil for the growth of our
understanding of what nonverbal communication
is, what it does, where it originates, how it unfolds,
and what it affects. It also means that a “history”
of our work has many different origins and often
divergent—even competing—answers to some
­fundamental questions.
In this chapter, I work to paint a picture of this
set of histories, or what I call heritages, emphasiz-
ing those scholars who have played a particularly
important role in shaping our research traditions
and illuminating some of the creativity and contro-
versies that have arisen based, in part at least, on the
myriad places from which we have come. Reading
across the other chapters in this handbook provides
additional evidence for the many origins of our
work. To help structure these ideas, however, I offer
a set of categories for this chapter that allows me
to discuss these diverse histories—and what I see
to be some of the legacies of these traditions—in a
focused way. I label these our rhetorical, linguistic,
sociological, cultural, ethological, and psychologi-
cal heritages. Some of these are tied to the intel-
lectual fields for which they are named; others span
The author wishes to thank her colleagues Leah Ceccarelli and Christine Harold for their advice on the rhetoric citations.
http://dx.doi.org/10.1037/14669-001
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
disciplines. All, however, can be used by scholars
across areas to better understand who we are, where
we came from, and to what effect.
Other writers may well have created different
labels or an organization that frames our field in
a way other than what I have here. Moreover, the
categories I suggest should not be seen as mutually
exclusive; they, like our history from where they
emerge, are messier and have more crossover than
they may seem at first. Nonetheless, I believe what
follows offers a useful perspective on from where
contemporary work emerged and who we are now as
scholars interested in nonverbal means of communi-
cating. It also provides a way of understanding—and
respecting—the places where our ideas converge
and where they diverge. Moreover, my aim is to
show that each of these traditions provides only part
of the overall picture that is the nonverbal commu-
nication system and that reading broadly and openly
will further our sensitivity to the value of nonverbal
communication and its study.
RHETORICAL HERITAGE
Mark L. Knapp, in his 2006 chapter that also
provides a history of research in our area, has
asserted that people have been “researching”
nonverbal cues since the dawn of time. The move-
ments that we make with our bodies, the expres-
sions on our faces, the clothes that we wear, and the
smells that we prefer have long made an impres-
sion on people—artists, writers, philosophers, and
3
Valerie Manusov
politicians—who seek to understand—and in some
cases manipulate—human behavior. More focused
scholarly discussion of nonverbal cues is attributed
at around the same time to Confucius in the East
and Aristotle in the West, with the latter expounding
on the importance of what we now call nonverbal
cues as foundational to the canon of delivery.
Knapp (2006) has argued, however, that “it was
the Roman orators and teachers who [first] refined,
clarified, categorized, and expanded on” nonverbal
behaviors (p. 4) in a quest to make oration a more
persuasive and, for some at least, a more ethical
practice. In addition to categorizing nonverbal cues
that were part of oration, this early tradition, what
I am calling part of our field’s rhetorical heritage,
made claims about what “good” (i.e., proper, appro-
priate, persuasive) nonverbal cues used during
public speech should look or sound like. Many
of these propositions centered on the importance
of consistency; that is, nonverbal cues should be
used in a way that, using Quintilian’s (90 CE/1922)
words, must be “harmonious” (i.e., go together or
be congruent with the other cues occurring with
them) to be effective.
This set of claims, along with the elocutionists
who followed starting in the mid-1700s, put in place
several paths that have been followed since by many
who seek to understand the nonverbal communica-
tion system. For example, it tied nonverbal com-
munication to language inherently, often in what
may appear to be a subordinate position. Centuries
later, Paul Ekman (1965), in his foundational model
of nonverbal communication, described nonverbal
cues as they function in relation to what is said and
in a way that reinforced, even if unwittingly, the
rhetors’ assumptions of its status vis-à-vis language.
Ekman argued that there are six ways in which
nonverbal cues interact with spoken words: In our
interactions, nonverbal communication may repeat,
conflict with, complement, substitute for, accent
or moderate, and regulate what is said. This set of
categories has been repeated—usually in an overly
simplistic way inconsistent with the complexity that
Ekman initially provided—in almost every general
discussion of what nonverbal cues are and how they
work, perpetuating the belief that nonverbal cues
are important, largely in their relationship with
4
language. Indeed, in many texts on interpersonal
communication, for instance, Ekman’s categoriza-
tion is used to define nonverbal communication and
how it functions. For those readers, then, nonverbal
communication is important only in how it works
alongside language.
On the other hand, the view that nonverbal cues
are—often at least—tied inherently to language is
the precursor to some very important research look-
ing more closely at this integration. Much of this
work centers on the careful analysis of the coordina-
tion of gestures and facial expressions into speech
acts and references the cues as one larger system,
not as verbal and nonverbal behavior (e.g., Kendon,
1985; McNeill, 1985; see also Chapter 12, this hand-
book). For Bavelas and Chovil (2006), the system-
atic study of the interplay of interaction behaviors
began in the 1950s (see the Linguistic Heritage and
Sociological Heritage sections of this chapter). They
noted that scholars have offered many labels for the
integration of interaction cues, such as mixed syn-
tax, comprehensive communication act, multichan-
nel process, composite signal, integrated message,
and multimodal communication. Others p ­ refer to
call the whole system of interconnected cues “lan-
guage” rather than make what they argue are erro-
neous distinctions between verbal and nonverbal
behavior (see, e.g., Streeck & Knapp, 2002).
The legacy just discussed—tying together all forms
of communication—is not seen typically as an ances-
tor of rhetorical traditions. However, contemporary
rhetorical work has a direct lineage to this heritage.
Whereas rhetoricians study language use (in its more
traditional sense) most commonly, particularly as a
means for persuasion, some focus their interpretive
lens on nonverbal cues. The subfield of visual rhetoric,
for instance, centers on the interpretation and critique
of images (see, e.g., Edwards & Winkler, 1997).
Material rhetoric is even more closely aligned to
nonverbal cues in that it reflects on the signification
of material things, such as the use of space, structure,
and environment as consequential modes of commu-
nication. Carole Blair is perhaps most well-known for
her work in this area, specifically that which centers
on the meanings in and around public memorials
(e.g., Blair, Balthrop, & Michel, 2011; Blair,
Dickinson, & Ott, 2010).

Other rhetoricians focus on the human body
as a communicative vehicle. In her book, Mov-
ing Bodies: Kenneth Burke at the Edges of Language,
Hawhee (2009) has revealed many of the ways in
which Burke, one of the best known contempo-
rary rhetoricians, portrayed rhetoric as more than
studying language form and features but, rather, as
symbolic action. In particular, Burke looked to the
body, its appearance and its movements, as what
he called a “somatic” (physical, body-based) line
of inquiry with tremendous representational and
transformational value. That is, without anything
being said, a person’s physical form and action
have poignant meaning value for the person, those
who engage with him or her, and the larger cul-
tural world in which that body is embedded. For
example, Burke (like Erving Goffman in his 1963
work, Stigma: Notes on the Management of Spoiled
Identity) explored the ways in which body anoma-
lies or “deviance” becomes part of—and creates
challenges to—our symbol system. In his book,
About Face, for instance, Cole (1999) has provided
examples of ­people who have lost the ability to
move their faces or who have facial deformities that
affect how “human” others think they are. Others,
such as Selzer and Crowley (1999), have likewise
put their focus on bodies, their movements, and talk
about body as part of body rhetoric. Bixler (2010),
as an exemplar, did an in-depth study of breast
cancer walks and the ways in which moving their
body—rather than just giving money—became an
important symbolic activity for the walkers, provid-
ing greater participative understanding of the jour-
ney that those with breast cancers undertake.
Together, this heritage reaches back into antiq-
uity to highlight the recognition of nonverbal means
of communicating as part of a larger set of actions.
For some, this set of actions was a speech, a rhetori-
cal moment when a speaker worked to persuade his
audience, and nonverbal cues were seen as essential
to doing this well. The tie between language and
nonverbal acts worked its way through time to those
whose work looks not at nonverbal communication
specifically but, rather, at larger communication
systems, where myriad cues work together as people
engage with one another. It has also encouraged
scholars to focus on the rhetorical (persuasive)
A History of Research on Nonverbal Communication
value of nonverbal cues that are as diverse from one
another as bodies and buildings. The result is an
emphasis on the structure and function of nonverbal
engagement.
LINGUISTIC HERITAGE
The focus on the communicative forms and pro-
cesses of the body—or of being embodied—and the
focus on structure have parallels in other traditions
for the study of nonverbal communication. Particu-
larly well-known among researchers of nonverbal
communication and, in particular, the structure
of nonverbal systems is Ray Birdwhistell, who, in
1970, published Kinesics and Context: Essays on
Body Motion Communication. Birdwhistell opened
his work by stating “these essays are based on the
conviction that body motion is a learned form of
communication, which is patterned within a culture
and which can be broken down into an ordered
system of isolable elements” (p. xi). Using a film
repertoire, he and colleagues from a range of fields
reviewed their data corpus, which ranged from a
mother changing her baby’s diapers to a scene of
a couple, where the woman lights and smokes her
cigarette. From these in-depth observations, Bird-
whistell proposed an elaborate set of categories that
characterized the movements they witnessed, which
he called kinesics.
Just as is true with other heritages described
in this chapter, Birdwhistell was affected by his
assumption that certain communication systems are
“language-like” and can be described by their units,
their combined units, and the ways that they can be
structured together with a particular syntax. Indeed,
even the label kinesics suggests that body move-
ments comprise a communicative code that can be
studied, just as languages are studied within linguis-
tics. However, for Birdwhistell (1970), meaning is
created in more than an a priori way, suggested by
models of what is “good” or “persuasive” inherently
(p. 27), as early studies of oration dictated. Rather, it
is understood within its context, and it is very much
a social endeavor conducted, as Goffman (1959)
also has contended, between people.
Birdwhistell’s (1970) conception of nonverbal
communication as an identifiable social action
5
Valerie Manusov
provided an important grounding for many more
recent projects on nonverbal communication.
One legacy of conceptualizing nonverbal cues as a
­structured and identifiable communication system is
the creation of labels that mirror linguistics, such as
calling touch cues haptics or tacesics or referencing
the communicative features of time as chronemics,
though Birdwhistell did not think that all behavior
systems were equally complex and structured as
is the body. Within this legacy, however, work by
Edward T. Hall on proxemics, or the use of space as
communication, is particularly germane (see also
Chapter 15, this handbook). Like Goffman (1959,
1963), E. T. Hall (1977) relied on extensive obser-
vation to propose that the way we use our personal
space and our territories—and how we come to
interpret the meanings for these space uses—is
part and parcel of the culture in which they occur.
Whereas there is universality to the idea that space
use is rule-governed and meaningful, the specific
ways it becomes so and the forms that it takes vary
across groups and cultures. These differences are
often the source of cross-group or cross-cultural
misunderstanding.
Although not making the linguistics reference,
Ekman also followed the premise that nonverbal
cues are formed from identifiable units that, when
combined, become socially meaningful. In 1978,
Ekman and his collaborator Wallace V. Friesen cre-
ated the Facial Action Coding System, an elaborate
research tool that has been used for, among other
things, making the distinction between the muscles
used in felt (genuine) and false (artificial) smiles
(see Ekman & Rosenberg, 1997, for a summary of
many of the studies that used this methodology up
until their publication date; see, also, Chapter 10,
this handbook). Later, John M. Gottman and his
­colleagues (e.g., Gottman, McCoy, & Coan, 1996)
built upon the Facial Action Coding System with
their Specific Action Coding System and have used
it successfully, alongside other measures, across
studies to help predict divorce.
The linguistic heritage, then, helps the depth
of our understanding about the complexity of the
behavioral repertoires available to us as communi-
cators and as scholars. Rather than the systematic
connections or the means through which actions
6
become meaningful that are the legacies of our
rhetorical heritage, this approach to the study of
nonverbal cues shines its light on the bones and
skeletons of our communicative systems. In doing
so, it brings to our attention the complexity of the
individual systems that make up the larger processes
in which we engage.
SOCIOLOGICAL HERITAGE
The focus on the specific behaviors that make
up nonverbal communicative systems works as a
bridge to other research traditions important to our
area. Albert Scheflen (1973, 1974), for example,
provided some of the foundational work on non-
verbal communication, also focusing initially on
kinesics. Scheflen argued that, ironically, the intro-
duction of kinesics as a language of the body was
distorted elsewhere into the “study” of body lan-
guage, with the assumption that one’s behaviors are
a direct reflection of that person and/or his state of
mind. For him, this psychologically oriented move
was problematic (and it also belies the much more
sophisticated work that has been done by psycholo-
gists, as will be seen). Scheflen’s grounding premise
focused instead on social order and meaning and
therefore helped forge what I call here the socio-
logical heritage of nonverbal research (though he
referenced it as a “communicational point of view”;
Scheflen, 1973, p. xiii).
Much of the work involving nonverbal
­communication stemming from a sociological tra-
dition is, like the two heritages already identified,
tied to language. Perhaps the most notable is the
body of scholarship using conversation analysis, and
here I return to the discussion from the rhetorical
heritage that does not delineate between separate
verbal and nonverbal forms of communicating.
Although conversation analysis is a method used
first to study the structure and form of language-
in-use ­(Schegloff, 1984; Schegloff & Sacks, 1973),
several conversation analysts have focused their
lens on the study of nonverbal cues as they occur in
everyday interaction with an emphasis on how the
behaviors “act” in interaction and what they do for
the interactants. Robinson (2006), for example, rec-
ognized “the inseparability of nonverbal and verbal

behavior” (p. 442) by showing the ways in which
participants in doctor–patient interactions orient
to one another through gaze and do so differently
at the beginning, middle, and end of conversations.
He argued that nonverbal cues are part of the “cause
and effect” of interactions and can be understood
best by observing carefully the sequence of behav-
iors as they unfold.
The idea that nonverbal communication does
things for us in interaction is also at the heart of
Goffman’s work. A sociologist himself, he wrote
prolifically and engagingly about an array of social
customs. In Stigma, for instance, Goffman (1963)
commented on the ways in which “problematic”
bodies alter the social system and the forms of
engagement within them. For instance, when we
see a person whose leg has been amputated compet-
ing in a skiing competition, we are encouraged to
alter our view about what counts as a “good” body.
In doing so, Goffman’s analysis offered an impor-
tant critical lens to our understanding of nonverbal
cues, which has been picked up by some scholars
(e.g., Coupland, 2003).
More germane, perhaps, to the contemporary
study of nonverbal communication, however, are
Goffman’s (1959) ideas relayed in The Presentation
of Self in Everyday Life. In this book, Goffman has
provided a view of the social world that furthered
Mead’s (1934) introduction of social interaction-
ism in Mind, Self, and Society (see also, Blumer,
1969). He argued, among other things, that people
are meaning-making creatures who develop and
alter their sense of themselves and the world
through their engagements with others. Whereas
Goffman (1959) discussed language, he also illus-
trated the salient role that nonverbal cues play by
themselves in how we present ourselves and are
confronted by others. In doing so, Goffman has
brought our specific attention to the interactive
functions that nonverbal cues may serve in our
ongoing interactions.
Whereas his focus in that treatise was on self-
presentation, other scholars have delineated a
larger range of communicative functions served by
nonverbal cues, reflecting even more the heurism
of Goffman’s work. One of those functions, interac-
tion management, can be seen in the conversation
A History of Research on Nonverbal Communication
analysis work (e.g., the gaze patterns of physicians
and their patients) discussed earlier in this section;
other scholars—such as Street and Cappella (1985);
Bernieri and Rosenthal (1991); Giles, Coupland,
and Coupland (1991)—have offered different
forms of inquiry that help detail the ways in which
interactants’ behaviors influence one another’s. For
example, some of this work brings to life the inter-
connectedness between a mother’s vocal cues and
her infant’s smiles and the ways in which gestures
imply that one person cannot yet take the speaking
floor from another. Such analyses suggest that schol-
arship often needs to center on what occurs between
people, rather than only individually, to fully under-
stand the nature of nonverbal communication in
interaction.
Another function, discussed most elaborately
by Judee K. Burgoon, involves nonverbal cues that
reflect for ourselves and others how interactants
appraise the relationship between them, what
Burgoon and Hale (1984) called relational messages
(see also, Burgoon & Le Poire, 1999). Among other
relational definitions, such messages comment on
interactants’ intimacy (see Andersen, Guerrero, &
Jones, 2006; Noller, 2006) or their power vis-à-vis
one another (Burgoon & Dunbar, 2006). For many,
the ability to be able to define, reflect, and some-
times change relationships via nonverbal means is
one of the most powerful social capacities nonverbal
cues carry (Docan-Morgan, Manusov, & Harvey,
2013). Additional vital functions that researchers
have identified include deception, emotional
expression, person perception, and persuasion
(see Patterson, 1991, for more on the functional
approach to nonverbal communication generally).
Together, the sociological heritage offers us
information about the social value of nonverbal
means of communicating. It helps us understand
what nonverbal means of communicating can do for
us as communicators, how the cues “act,” and what
those actions provide for us in our interactions.
Scholars from this heritage also emphasize that
we communicate nonverbally in tandem with our
interaction partners such that each communicator’s
behaviors affect and reflect what the other is doing,
forming a kind of unique communication system
between the interactants.
7
Valerie Manusov

CULTURAL HERITAGE realm. Much of this comparative work can be traced

to E. T. Hall, who, in 1959, published The Silent
As noted, Goffman did most of his research through
Language to engage his ideas around variance in
observation of behaviors in their context, providing
nonverbal displays based in culture. In that book,
rich descriptions of what was occurring and giv-
and others (E. T. Hall, 1966, 1977), he referenced
ing commentary on what he observed. This form of
the idea of context as imperative for interpreting
analysis is consistent with ethnographic research,
nonverbal cues (see also Chapters 4 and 16, this
studies that are conducted in situ and allow for
handbook).
the naturalistic observation of behaviors as they
For E. T. Hall, this reference was used to
unfold. Gerry Philipsen is credited with bringing
­delineate broad level differences between cultures,
the ethnographic form to the study of communica-
including the degree to which the members of a
tion more specifically, and his work, like those who
culture rely more or less on nonverbal cues in their
followed him, is referred to as the Ethnography of
interactions. Specifically, he noted that high-context
Communication and is a part of the cultural heri-
cultures are particularly nonverbal in that less of
tage of nonverbal communication scholarship (see
their social meaning is encoded in what they say to
Philipsen, 2009). This heritage, like the sociologi-
one another. Cues available in the larger context,
cal heritage, focuses on the larger systems in which
such as each interactant’s status, become a primary
nonverbal cues are embedded (see also Chapter 4,
way of understanding behavior and determining
this handbook).
what social actions are appropriate. Myriad studies
Following Dell Hymes and others, ethnographers
have been produced to test the observable differ-
who provide understanding about nonverbal cues
ences of people from high- and low-context cultures
do so with the assumption that communicative cues
(e.g., Kim, Pan, & Park, 1998). In high-context
both reflect and affect culture. That is, nonverbal
cultures, such as China and Malaysia, for instance,
cues become meaningful within their larger cultural
knowing cultural rules and meanings is imperative.
context, which typically is the communicative or
As Salleh (2005) stated, giving the gift of a clock to
speech community that uses—and makes mean-
someone from China amounts to telling the receiver
ing for—those cues. The community’s rules of use,
that the giver wants him or her to have a short life,
their meanings, and changes that they undergo are
whereas in Malaysia, the clock-as-gift exemplifies
part and parcel of the larger set of cultural norms
friendship. E. T. Hall’s (1966) work also delineated
and values held by the communicative community.
cultures into contact and noncontact groups, with
Most specifically, studying nonverbal cues in their
those in contact cultures more likely to engage in
cultural contexts speaks to that culture and what it
touch and have smaller proxemic zones, a set of
believes and finds important. Carbaugh, Berry, and
behaviors referred to elsewhere as immediacy cues
Nurmikari-Berry (2006), for instance, have offered
(Mehrabian, 1981; see also Chapter 4, this hand-
evidence of a Finnish cultural code that values
book). The legacy of research on immediacy is
silence so much so that it is considered, by those in
itself vast (see Andersen et al., 2006).
the community, to be a “natural way of being.” Like-
Another cultural dimension that has influ-
wise, Levine (1997), using a different methodology
enced the study of nonverbal communication is
but highlighting similar cultural links to nonverbal
the i­ ndividualism/collectivism distinction (Hofstede,
cues, showed the ways in which time is a culturally
1980). Although often criticized for its overreaching
rule-governed system.
claims, the idea that some cultures focus more on
The legacy of a cultural heritage includes stud-
personal achievement and responsibility and others
ies that are more about cultural differences than
use the group norms and values as more determi-
the cultural way of communicating emphasized in
nant of behavior has generated a large set of studies
the Ethnography of Communication. For instance,
(e.g., Kowner & Wiseman, 2003; Ozdemir, 2008).
research that looks at how much touch one culture
Others have argued that there are people within
uses compared to another would fall within this

8

cultures who are more individualist or more col-
lectivistic. Matsumoto and Kupperbusch (2001), for
example, found that U.S. women who were more
collectivistic tended to mask their negative emotions
when communicating with others, a collectivist
tendency toward harmony, even though they did
not differ in their expression when alone compared
to their individualist counterparts.
As can be seen, this heritage draws our attention
to the ways in which our larger social and cultural
groups provide framings for how to use and under-
stand nonverbal communication. It encourages us
to think about nonverbal behaviors as learned and
as understood largely within the communicative
community in which it is based. The emphasis is
on what makes people within one community alike
and potentially at odds with others who do not have
the cultural knowledge needed to understand the
behavior from the perspective of those who use it.
In so doing, it suggests places where communica-
tion between people can go “wrong,” and it pro-
vides means for increasing the chances that people
can communicate well through greater cultural
sensitivity.
ETHOLOGICAL HERITAGE
As noted, the cultural heritage of nonverbal com-
municative inquiry highlights the centrality of our
social environment in our understanding of non-
verbal communication. In doing so, it provides a
very different vantage point than another of our key
heritages. This other set of work, while sometimes
looking at culture, functions more commonly to
determine what is universal in our use and interpre-
tation of nonverbal cues. In his text, The Expression
of the Emotions in Man and Animals, Charles
Darwin (1872) helped to set a trajectory for an
ethological heritage, one that relies on the study of
animals, at least to some degree and usually within
context, to understand human behavior (see also
Chapter 4, this handbook).
One of the primary legacies of this tradition is
the very active contemporary research focus on the
biological origins of nonverbal cues, although not all
of it ties to animal behavior directly. The closest to
A History of Research on Nonverbal Communication
Darwin’s observations, however, is work that Floyd
(2006) suggested takes an evolutionary approach to
nonverbal research, one “brand” within an etho-
logical heritage. For Floyd, it is Darwin’s theory of
evolution by means of natural selection that is most
salient as a foundation for contemporary scholarship
on the biology of nonverbal communication. Work
on the biological bases of attraction (e.g., Bradley,
Miccoli, Escrig, & Lang, 2008; Janssen & Everaerd,
1993) also are offshoots of the evolutionary
approach.
Indeed, the legacies for this heritage are many.
One is the argument that nonverbal cues have
­adaptive value for us and that they can be tied to
who we are as a species. These kinds of studies,
whether explicitly calling themselves ethologi-
cal or evolutionary, assume that certain nonverbal
cues will occur universally, as they are based in
who we are as human. The most well-known of
these focuses on the universal expression of emo-
tions, not surprising given Darwin’s influence. In
particular, Ekman, Matsumoto, and their colleagues
(e.g., Ekman, 1993; Matsumoto, 2006; Matsumoto,
Keltner, Shiota, Frank, & O’Sullivan, 2008) have
shown that people from an array of cultural back-
grounds all can decode particular facial expressions
accurately.
In a recent review, Burgoon, Guerrero, and
Manusov (2011) summarized related research that
has also identified basic or primary emotions that are
expressed on people’s face the same way across cul-
tures (see, e.g., Izard, 1977; Tomkins, 1963), with
happiness, sadness, anger, fear, surprise, and disgust
encoded most consistently. Additional support for
the universal and biological bases of nonverbal cues
comes from studies of child development, which
have shown that typically developing children fol-
low relatively set stages of emotional development
and exhibit like expressions at each successive stage.
Other evidence comes from studies of children who
are hearing or sight impaired or who are limbless
and not able to experience certain touch senses. Yet,
the children, who cannot learn emotional displays
through sensory experience, still express universally
recognized emotions (see, e.g., Eibl-Eibesfeldt, 1973;
Galati, Scherer, & Ricci-Bitti, 1997).
9
Valerie Manusov
Perhaps the biggest legacy of this heritage is,
however, controversy regarding the degree to which
emotional expressions are as universal and biologi-
cally based as they may appear. Indeed, some of the
research grounded in other heritages, such as our
cultural and (to some extent) the sociological tradi-
tions, features the social and cultural influences that
complement—or supercede—biology. Furthermore,
many who work to show universality also highlight
the role of culture (e.g., Ekman, 1977, particularly
in his introduction of display rules). Still others,
however, question the degree to which our facial
displays in interaction are best characterized as
emotional expressions. For example, Fridlund and
Russell (2006) have argued that, rather than seeing
our facial movements as displays of emotions, we
should view them as a sort of “social tool” (p. 299)
that allow us to communicate things such as atti-
tude, acknowledgment, agreement, and the like and
that therefore play a large role in how our interac-
tions unfold. Motley (1993) has provided evidence
that our facial cues are most typically in the form of
communicative interjections (i.e., suggesting agree-
ments, queries, concerns) that may be picked up
and acted on by another. Perhaps more than any
heritage, this controversy has provided an ongoing
legacy and a large body of scholarship (for further
discussion, see Chapter 10, this handbook).
A related legacy is the argument that, and con-
trasting with the belief of one integrated system,
there are multiple forms of behavior that occur in
our interactions, each of which has a more biologi-
cal or more social basis. Buck and VanLear (2002),
for instance, say that there are three co-occurring
communicative streams: spontaneous (nonverbal
cues that are automatic and biologically based),
pseudospontaneous (cues that appear like sponta-
neous cues but are adapted to the communicative
context, such as opening up one’s eyes and raising
one’s brows to show the other that one is surprised),
and symbolic (cues that are wholly arbitrary and
socially defined, including emblems like an “okay”
thumbs up in the United States). In his earlier work,
Buck (1984) has provided evidence that the differ-
ent streams of behavior are neurologically processed
differently. Given this same grounding, Andersen
(2008) has made the case for why we should only
10
call those cues that are spontaneous “nonverbal
communication,” with the rest better seen as part of
“language.”
This heritage, then, provides salience to the
inherited nature of nonverbal cues. Whereas those
working in this tradition point out that some
behaviors are culturally derived and understood,
their emphasis is instead on those actions that
have adaptive value. They are also more likely to
emphasize the universality (and, sometimes, the
cross-species similarity) and automaticity of certain
ways of communicating and to locate the biological
origins of some (or, for certain scholars, much) of
what we do nonverbally.
PSYCHOLOGICAL HERITAGE
The decision to look at the neurological basis of
nonverbal cues is also at the heart of a large body
of work in the study of nonverbal communica-
tion. According to Lakin (2006), “understanding
nonverbal communication relies, to some extent,
on appreciating its cognitive foundation” (p. 59).
Given this, it is not surprising that researchers have
queried the ways in which nonverbal cues are tied
to cognitive structures and processes, with a partic-
ular interest in the degree to which nonverbal cues
are processed automatically or with greater aware-
ness and control (see, e.g., Bargh & Chartrand,
1999), creating a relatively heated battle over the
nature of nonverbal behaving.
Not all work from the psychological heritage
regards mental processing per se. More common
are those research areas that tie nonverbal cues to
important individual traits or interaction outcomes
(see also Chapter 23, this handbook). Gifford
(2006), in his review of research on personality
and nonverbal communication, has taken us back
to Aristotle, and others, as the start of this heritage.
In particular, he has argued for an early connec-
tion made between our physical bodies and our
psychological selves, particularly what is now seen
as personality. According to Gifford, “from Aristo-
tle’s time, physiognomists [those who judge human
character from observing the face] have been cer-
tain that they can discern personality solely from a
person’s facial features” (p. 160).

Whereas Gifford has concluded rightfully that
such an approach has been firmly “discredited”
over time (but see important work on such things
as eye color and temperament; e.g., Rosenberg &
Kagan, 1989), it has a strong and important legacy
of research investigating the many ways that psy-
chological processes explain and are related to our
nonverbal cues. Part of this is showing, on the one
hand, just how complex the relationship—where
it exists—is between facets of the personality and
nonverbal cues expressing those facets (Gifford,
2006; see also Chapters 9 and 10, this handbook,
for static facial clues). Part of it is also research
methodologies that reflect, on the other hand, how
little it can take to judge certain states or traits
accurately from observing “thin slices” of inter-
action (Ambady, Krabbenhoft, & Hogan, 2006;
Ambady & Rosenthal, 1993).
An additional implication is the degree to which
states of mind—and states of relationship—can
emerge from the nonverbal behaviors used during
interaction. Among these states is rapport. Accord-
ing to Tickle-Degnen (2006), “rapport is used to
indicate a meaningful human experience of close
and harmonious connection that involves com-
mon understanding” (p. 381). There are several
nonverbal cues or skills that have been found to
be part of establishing rapport. These include non-
verbal expressivity (Boone & Buck, 2003), positive
affect (depending on the context; Tickle-Degnen &
Rosenthal, 1990), and the coordination of behavior
(Burgoon, Stern, & Dillman, 1995), discussed ear-
lier as a function of nonverbal cues. Moreover, the
establishment of rapport has been found to have a
wide range of social benefits.
There is additional work from this heritage that
emphasizes the social implications of nonverbal
behavior. Robert Rosenthal (1974), for example, is
well-known for showing that people have expec-
tancies for others that show up in their nonverbal
communication. In particular, he found that such
expectancies occur commonly in the classroom and
in the research laboratory (see Rosenthal, 2003).
More poignantly, these communicated expectan-
cies may influence the behaviors and outcomes of
the people about whom the expectancies are held.
That is, they may work as a self-fulfilling prophecy
A History of Research on Nonverbal Communication
(Rosenthal, 1974). Part of Rosenthal’s concern had
to do with the sometimes subtle persuasive ability of
nonverbal cues. Particularly in light of atrocities that
occurred before and during World War II, many
researchers from the psychological tradition aimed
to discern how people can become convinced to do
something that they may otherwise not have done.
One of these factors was status and its relation to
power, as expressed by the dress and demeanor of
an experimenter (e.g., Milgram, 1974).
A legacy of this line of research is a focus on sex
differences and similarities (see also Chapter 6, this
handbook). In 1977, Nancy M. Henley put forward
the provocative subordination hypothesis proposing
that differences in the behaviors of males and females
is aligned with differences in behaviors of people in
high and low status or power. Furthermore, some
studies have supported this (see review by Burgoon,
Guerrero, & Floyd, 2010). However, Judith A.
Hall and her colleagues (J. A. Hall, 2006; J. A. Hall,
Coats, & LeBeau, 2005) asserted that the idea that
there is a “vertical dimension” (i.e., power structure)
that explains sex differences in behavior has little
support. They argued instead that what differences
there are—and there are many similarities—can be
explained better by the ways in which males and
females are “skilled” differently by both biological
and social means. For instance, girls are taught to be
more relationally oriented and are therefore encour-
aged to attend to others’ nonverbal cues.
This finding is consistent with a research concern
regarding nonverbal communication as a skill, both
as something that can be learned and also something
that, if impaired, can have important consequences.
Riggio and Riggio (2005) summarized many of the
measures that have developed to access the degree
to which people can, for example, express or decode
emotions. Others, such as Duke and Nowicki
(2005), have focused on particular, often congenital,
conditions, such as dyssemia, that make commu-
nicating nonverbally difficult. Research on autism
also falls into this category (e.g., Yoder, Stone,
Walden, & Malesa, 2009). Likewise, other scholars
(e.g., Segrin, 2000) have focused on more transitory
states, such as depression, and the influence it has
on people’s use of nonverbal skills. Such influence is
often the exacerbation of such conditions.
11
Valerie Manusov
Overall, the psychological heritage encourages
investigation of the individual (or the group) and
the ways in which nonverbal cues reflect the person
and his or her skills, dispositions, and motivations.
It has allowed us to see where real differences lie
and also to show where stereotypes may exist. That
is, work from this heritage points to where people
believe there are ties between personality and non-
verbal cues (perception studies) and what links
there may actually be. Often, we perceive as com-
municators much more connection than research
suggests exists, and this is exacerbated in the media,
which often misconstrues research findings about
perceptions to actual ways in which nonverbal cues
are associated with dispositions. As Gifford (2006)
noted, the psychological origins of certain nonverbal
cues make up a “complex conundrum.”
CONCLUSION
As this chapter—and this handbook—shows, the
study of nonverbal communication has a rich
history, one that runs both deep and wide. Each
heritage delineated here takes us along certain paths
in our exploration of nonverbal communication;
however, walking along those paths provides only
a partial view of our large and varied field. Given
that certain heritages are also tied to certain intel-
lectual areas (e.g., the rhetorical heritage is most
well-known and its paths followed most often by
communication scholars; the psychological heritage
is more likely to follow the assumptions of and be
the basis for work by psychologists), it is important
to have at least a glimpse of other traditions to know
what we include—and what we may be missing—in
our individual inquiries. When seen as a broad set of
research programs meant to understand the myriad
modes of communicating, these heritages and their
legacies are all worth illuminating.
In some ways, however, in this chapter I have
only scratched the surface in highlighting some of
these traditions, the people and ideas that created
and stem from these traditions, and the areas where
our divergent histories may help explain the differ-
ences in perspective on answering some of the truly
foundational questions about human behavior that
are queried within our field. My hope was to reflect
12
the many heritages that inform our contemporary
understanding of nonverbal communication and,
perhaps for some readers, broaden the sense of what
counts as “nonverbal research.” As noted, I also
wanted to reflect the respect that I have—and that I
hope we all share—for the diversity of the
terrain we, as a group of scholars, have charted as
we explore the vast message potential and the possi-
ble and poignant consequences of nonverbal means
of communicating.
References
Ambady, N., Krabbenhoft, M. A., & Hogan, D. (2006).
The 30-sec sale: Using thin-slice judgments to
evaluate sales effectiveness. Journal of Consumer
Psychology, 16, 4–13. http://dx.doi.org/10.1207/
s15327663jcp1601_2
Ambady, N., & Rosenthal, R. (1993). Half a minute:
Predicting teacher evaluations from thin slices of
nonverbal behavior and physical attractiveness.
Journal of Personality and Social Psychology, 64,
431–441. http://dx.doi.org/10.1037/
0022-3514.64.3.431
Andersen, P. A. (2008). Nonverbal communication: Forms
and functions (2nd ed.). Long Grove, IL: Waveland
Press.
Andersen, P. A., Guerrero, L. K., & Jones, S. M. (2006).
Nonverbal behavior in intimate interactions and
intimate relationships. In V. Manusov & M. L.
Patterson (Eds.), The Sage handbook of nonverbal
communication (pp. 259–278). http://dx.doi.org/
10.4135/9781412976152.n14
Bargh, J. A., & Chartrand, T. L. (1999). The unbearable
automaticity of being. American Psychologist, 54,
462–479. http://dx.doi.org/10.1037/
0003-066X.54.7.462
Bavelas, J. B., & Chovil, N. (2006). Nonverbal and verbal
communication: Hand gestures and facial displays
as part of language use in face-to-face dialogue.
In V. Manusov & M. Patterson (Eds.), The Sage
handbook of nonverbal communication (pp. 97–115).
http://dx.doi.org/10.4135/9781412976152.n6
Bernieri, F. J., & Rosenthal, R. (1991). Interpersonal
coordination: Behavior matching and interactional
synchrony. In R. S. Feldman & B. Rimé (Eds.),
Fundamentals of nonverbal behavior (pp. 401–432).
Cambridge, England: Cambridge University Press.
Birdwhistell, R. L. (1970). Kinesics and context: Essays
on body motion communication. Philadelphia:
University of Pennsylvania Press.
Bixler, N. R. (2010). Walk me home: How bodies move
and are moved in the breast cancer walk (Unpublished

doctoral dissertation). University of Washington,
Seattle.
Blair, C., Balthrop, V. W., & Michel, N. (2011).
The arguments of the tombs of the unknown:
Relationality and national legitimation.
Argumentation, 25, 449–468. http://dx.doi.org/
10.1007/s10503-011-9216-9
Blair, C., Dickinson, G., & Ott, B. L. (2010). Rhetoric/
memory/place. In G. Dickinson, C. Blair, & B. L.
Ott (Eds.), Places of public memory: The rhetoric of
museums and memorials (pp. 1–54). Tuscaloosa:
University of Alabama Press.
Blumer, H. (1969). Symbolic interactionism: Perspective
and method. Englewood Cliffs, NJ: Prentice Hall.
Boone, R. T., & Buck, R. (2003). Emotional expressivity
and trustworthiness: The role of nonverbal behavior
in the evolution of cooperation. Journal of Nonverbal
Behavior, 27, 163–182. http://dx.doi.org/10.1023/
A:1025341931128
Bradley, M. M., Miccoli, L., Escrig, M. A., & Lang, P. J.
(2008). The pupil as a measure of emotional
arousal and autonomic activation. Psychophysiology,
45, 602–607. http://dx.doi.org/10.1111/j.1469-
8986.2008.00654.x
Buck, R. (1984). The communication of emotion.
New York, NY: Guilford Press.
Buck, R., & VanLear, C. A. (2002). Verbal and nonverbal
communication: Distinguishing symbolic, spontaneous,
and pseudo-spontaneous communication. Journal of
Communication, 52, 522–541. http://dx.doi.org/
10.1111/j.1460-2466.2002.tb02560.x
Burgoon, J. K., & Dunbar, N. E. (2006). Nonverbal
expressions of dominance and power in
human relationships. In V. Manusov & M. L.
Patterson (Eds.), The Sage handbook of nonverbal
communication (pp. 279–297). http://dx.doi.
org/10.4135/9781412976152.n15
Burgoon, J. K., Guerrero, L. K., & Floyd, K. (2010).
Nonverbal communication. Boston, MA: Allyn & Bacon.
Burgoon, J. K., Guerrero, L. K., & Manusov, V. (2011).
Nonverbal signals. In M. L. Knapp & J. A. Daly
(Eds.), The Sage handbook of interpersonal
communication (4th ed., pp. 239–280). Thousand
Oaks, CA: Sage.
Burgoon, J. K., & Hale, J. L. (1984). The fundamental
topoi of relational communication. Communication
Monographs, 51, 193–214. http://dx.doi.
org/10.1080/03637758409390195
Burgoon, J. K., & Le Poire, B. A. (1999). Nonverbal
cues and interpersonal judgments: Participant
and observer perceptions of intimacy, dominance,
composure, and formality. Communication
Monographs, 66, 105–124. http://dx.doi.org/10.1080/
03637759909376467
A History of Research on Nonverbal Communication
Burgoon, J. K., Stern, L. A., & Dillman, L. (1995).
Interpersonal adaptation: Dyadic interaction patterns.
http://dx.doi.org/10.1017/CBO9780511720314
Carbaugh, D., Berry, M., & Nurmikari-Berry, M. (2006).
Coding personhood through cultural terms and
practices: Silence and quietude as a Finnish “natural
way of being.” Journal of Language and Social
Psychology, 25, 203–220. http://dx.doi.org/
10.1177/0261927X06289422
Cole, J. (1999). About face. Cambridge, MA: Bradford.
Coupland, J. (2003). Ageist ideology and discourses of
control in skincare product marketing.
In J. Coupland & R. Gwyn (Eds.), Discourse, the
body and identity (pp. 127–150). Basingstoke,
England: Plagrave Macmillan.
Darwin, C. R. (1872). The expression of the emotions in man
and animals. http://dx.doi.org/10.1037/10001-000
Docan-Morgan, T., Manusov, V., & Harvey, J. (2013).
When a small thing means so much: Nonverbal cues
as turning points in relationships. Interpersona, 7,
110–124. http://dx.doi.org/10.5964/ijpr.v7i1.119
Duke, M., & Nowicki, S. (2005). The Emory Dyssemia
Index. In V. Manusov (Ed.), The sourcebook of
nonverbal measures: Going beyond words (pp. 32–42).
Mahwah, NJ: Erlbaum.
Edwards, J., & Winkler, C. (1997). Representative
form and the visual ideograph: The Iwo Jima
images in editorial cartoons. Quarterly Journal of
Speech, 83, 289–310. http://dx.doi.org/10.1080/
00335639709384187
Eibl-Eibesfeldt, I. (1973). Expressive behaviour of the
deaf and blind born. In M. von Cranach & I. Vine
(Eds.), Social communication and movement
(pp. 163–194). New York, NY: Academic Press.
Ekman, P. (1965). Communication through nonverbal
behavior: A source of information about
interpersonal relationship. In S. S. Tomkins &
C. E. Izard (Eds.), Affect, cognition, and personality
(pp. 390–442). New York, NY: Springer.
Ekman, P. (1977). Biological and cultural contributions
to body and facial movement. In J. Blacking (Ed.),
Anthropology of the body (pp. 34–84). London,
England: Academic Press.
Ekman, P. (1993). Facial expression and emotion.
American Psychologist, 48, 376–379. http://dx.doi.
org/10.1037/0003-066X.48.4.384
Ekman, P., & Friesen, W. V. (1978). The Facial Action
Coding System: A technique for the measurement
of facial movement. Palo Alto, CA: Consulting
Psychologists Press.
Ekman, P., & Rosenberg, E. L. (1997). What the face
reveals: Basic and applied studies of spontaneous
expression using the Facial Action Coding System
(FACS). New York, NY: Oxford University Press.
13
Valerie Manusov
Floyd, K. (2006). An evolutionary approach to
understanding nonverbal communication.
In V. Manusov & M. L. Patterson (Eds.), The Sage
handbook of nonverbal communication (pp. 139–158).
http://dx.doi.org/10.4135/9781412976152.n8
Fridlund, A. J., & Russell, J. A. (2006). The functions of
facial expressions: What’s in a face? In V. Manusov &
M. L. Patterson (Eds.), The Sage handbook of
nonverbal communication (pp. 299–320). http://
dx.doi.org/10.4135/9781412976152.n16
Galati, D., Scherer, K. R., & Ricci-Bitti, P. E. (1997).
Voluntary facial expression of emotion: Comparing
congenitally blind with normally sighted encoders.
Journal of Personality and Social Psychology, 73,
1363–1379. http://dx.doi.org/10.1037/
0022-3514.73.6.1363
Gifford, R. (2006). Personality and nonverbal behavior:
A complex conundrum. In V. Manusov &
M. L. Patterson (Eds.), The Sage handbook of
nonverbal communication (pp. 159–179). http://
dx.doi.org/10.4135/9781412976152.n9
Giles, H., Coupland, N., & Coupland, J. (1991).
Accommodation theory: Communication, context,
and consequence. In H. Giles, J. Coupland, &
N. Coupland (Eds.), Contexts of accommodation:
Developments in applied sociolinguistics (pp. 1–68).
http://dx.doi.org/10.1017/CBO9780511663673.001
Goffman, E. (1959). The presentation of self in everyday
life. New York, NY: Doubleday.
Goffman, E. (1963). Stigma: Notes on the management of
spoiled identity. Englewood Cliffs, NJ: Prentice Hall.
Gottman, J. M., McCoy, K., & Coan, J. (1996). The
Specific Affect Coding System. In J. M. Gottman (Ed.),
What predicts divorce? The measures (pp. 1–169).
Hillsdale, NJ: Erlbaum.
Hall, E. T. (1959). The silent language. Garden City, NY:
Doubleday.
Hall, E. T. (1966). The hidden dimension. Garden City,
NY: Doubleday.
Hall, E. T. (1977). Beyond culture. Garden City, NY:
Anchor Books.
Hall, J. A. (2006). Women’s and men’s nonverbal
communication: Similarities, differences, stereotypes,
and origins. In V. Manusov & M. L. Patterson (Eds.),
The Sage handbook of nonverbal communication
(pp. 201–218). http://dx.doi.org/10.4135/
9781412976152.n11
Hall, J. A., Coats, E. J., & LeBeau, L. S. (2005). Nonverbal
behavior and the vertical dimension of social
relations: A meta-analysis. Psychological Bulletin,
131, 898–924. http://dx.doi.org/10.1037/
0033-2909.131.6.898
Hawhee, D. (2009). Moving bodies: Kenneth Burke at the
edges of language. Columbia: University of South
Carolina Press.
14
Henley, N. M. (1977). Body politics: Power, sex, and
nonverbal communication. Englewood Cliffs, NJ:
Prentice Hall.
Hofstede, G. (1980). Culture’s consequences: International
differences in work-related values. Beverly Hills, CA: Sage.
Izard, C. E. (1977). Human emotions. New York, NY: Plenum.
Janssen, E., & Everaerd, W. (1993). Determinants of male
sexual arousal. Annual Review of Sex Research, 4,
211–245.
Kendon, A. (1985). Uses of gestures. In D. Tannen &
M. Saville-Troike (Eds.), Perspectives on silence
(pp. 215–234). Norwood, NJ: Ablex.
Kim, D., Pan, Y., & Park, H. S. (1998). High-versus low-
context culture: A comparison of Chinese, Korean,
and American cultures. Psychology and Marketing,
15, 507–521. http://dx.doi.org/10.1002/(SICI)1520-
6793(199809)15:6<507::AID-MAR2>3.0.CO;2-A
Knapp, M. L. (2006). An historical overview of nonverbal
research. In V. Manusov & M. L. Patterson (Eds.),
The Sage handbook of nonverbal communication (pp.
3–19). http://dx.doi.org/10.4135/9781412976152.n1
Kowner, R., & Wiseman, R. (2003). Culture and
status-related behavior: Japanese and American
perceptions of interaction in asymmetric dyads.
Cross-Cultural Research, 37, 178–210. http://dx.doi.
org/10.1177/1069397103037002002
Lakin, J. L. (2006). Automatic cognitive processes
and nonverbal communication. In V. Manusov &
M. L. Patterson (Eds.), The Sage handbook of
nonverbal communication (pp. 59–78). http://dx.doi.
org/10.4135/9781412976152.n4
Levine, R. V. (1997). A geography of time: The temporal
misadventures of a social psychologist. New York, NY:
Basic Books.
Matsumoto, D. (2006). Culture and nonverbal behavior.
In V. Manusov & M. L. Patterson (Eds.), The Sage
handbook of nonverbal communication (pp. 219–236).
http://dx.doi.org/10.4135/9781412976152.n12
Matsumoto, D., Keltner, D., Shiota, M., Frank, M., &
O’Sullivan, M. (2008). Facial expressions of emotion.
In M. Lewis, J. Haviland, & L. Feldman-Barrett
(Eds.), Handbook of emotion (pp. 211–234).
New York, NY: Guilford Press.
Matsumoto, D., & Kupperbusch, C. (2001). Idiocentric
and allocentric differences in emotional expression,
experience, and the coherence between expression
and experience. Asian Journal of Social Psychology,
4, 113–131. http://dx.doi.org/10.1111/j.1467-
839X.2001.00080.x
McNeill, D. (1985). So you think gestures are nonverbal?
Psychological Review, 92, 350–371. http://dx.doi.
org/10.1037/0033-295X.92.3.350
Mead, G. H. (1934). Mind, self, and society. Chicago, IL:
University of Chicago Press.

Mehrabian, A. (1981). Silent messages: Implicit
communication of emotions and attitudes (2nd ed.).
Belmont, CA: Wadsworth.
Milgram, S. (1974). Obedience to authority: An
experimental view. New York, NY: Harper Collins.
Motley, M. (1993). Facial affect and verbal context in
conversation: Facial expression as interjection.
Human Communication Research, 20, 3–40. http://
dx.doi.org/10.1111/j.1468-2958.1993.tb00314.x
Noller, P. (2006). Nonverbal communication in
close relationships. In V. Manusov & M. L.
Patterson (Eds.), The Sage handbook of nonverbal
communication (pp. 403–420). http://dx.doi.org/
10.4135/9781412976152.n21
Ozdemir, A. (2008). Shopping malls: Measuring
interpersonal distance under changing conditions
and across cultures. Field Methods, 20, 226–248.
http://dx.doi.org/10.1177/1525822X08316605
Patterson, M. L. (1991). A functional approach to
nonverbal exchange. In R. S. Feldman & B. Rime
(Eds.), Fundamentals of nonverbal behavior
(pp. 458–495). Cambridge, England: Cambridge
University Press.
Philipsen, G. (2009). Researching culture in contexts
of social interaction: An ethnographic approach,
a network of scholars, illustrative moves. In D.
Carbaugh & P. M. Buzzanell (Eds.), Distinctive
qualities in communication research (pp. 87–105).
New York, NY: Routledge.
Quintilian, M. F. (1922). The institution oratoria, book XI
(H. E. Butler, Trans.). Cambridge, MA: Harvard
University Press. (Original work published circa 90 CE)
Riggio, R. E., & Riggio, H. R. (2005). Self-report
measures of emotional and nonverbal expressiveness.
In V. Manusov (Ed.), The sourcebook of nonverbal
measures: Going beyond words (pp. 105–111).
Mahwah, NJ: Erlbaum.
Robinson, J. (2006). Nonverbal communication and
physician–patient interaction. In V. Manusov &
M. L. Patterson (Eds.), The Sage handbook of
nonverbal communication (pp. 437–460). http://
dx.doi.org/10.4135/9781412976152.n23
Rosenberg, A. A., & Kagan, J. (1989). Physical and
physiological correlates of behavioral inhibition.
Developmental Psychobiology, 22, 753–770. http://
dx.doi.org/10.1002/dev.420220802
Rosenthal, R. (1974). On the social psychology of the self-
fulfilling prophecy: Further evidence for Pygmalion
effects and their mediating outcomes. New York, NY:
M. S. S. Information Corporation Modular
Publication.
A History of Research on Nonverbal Communication
Rosenthal, R. (2003). Covert communication in
laboratories, classrooms, and the truly real world.
Current Directions in Psychological Science, 12,
151–154. http://dx.doi.org/10.1111/1467-8721.
t01-1-01250
Salleh, L. M. (2005). Characteristics of high/low context
in communication: The Malaysian Malay culture.
Journal of Intergroup Relations, 32, 40–55.
Scheflen, A. (1973). Communicational structure: Analysis
of a psychotherapy interaction. Bloomington: Indiana
University Press.
Scheflen, A. (1974). How behavior means. Garden City,
NJ: Doubleday.
Schegloff, E. (1984). On some gestures’ relation to talk.
In J. M. Atkinson & J. Heritage (Eds.), Structures of
social action (pp. 266–296). Cambridge, England:
Cambridge University Press.
Schegloff, E., & Sacks, H. (1973). Opening up closings.
Semiotica, 8, 289–327. http://dx.doi.org/10.1515/
semi.1973.8.4.289
Segrin, C. (2000). Social skills deficits associated with
depression. Clinical Psychology Review, 20, 379–403.
http://dx.doi.org/10.1016/S0272-7358(98)00104-4
Selzer, J., & Crowley, S. (Eds.). (1999). Rhetorical bodies.
Madison: University of Wisconsin Press.
Streeck, J., & Knapp, M. L. (2002). Culture, meaning,
and interpersonal communication. In M. L.
Knapp & J. A. Daly (Eds.), Handbook of interpersonal
communication (pp. 286–319). Thousand Oaks, CA:
Sage.
Street, R. L., & Cappella, J. N. (Eds.). (1985). Sequence
and pattern in communicative behaviour. London,
England: Edward Arnold.
Tickle-Degnen, L. (2006). Nonverbal behavior and
its function in the ecosystem of rapport. In V.
Manusov & M. L. Patterson (Eds.), The Sage
handbook of nonverbal communication (pp. 381–400).
http://dx.doi.org/10.4135/9781412976152.n20
Tickle-Degnen, L., & Rosenthal, R. (1990). The nature
of rapport and its nonverbal correlates. Psychological
Inquiry, 1, 285–293. http://dx.doi.org/10.1207/
s15327965pli0104_1
Tomkins, S. (1963). Affect imagery consciousness: Vol. 2.
The negative affects. New York, NY: Springer.
Yoder, P., Stone, W. L., Walden, T., & Malesa, E. (2009).
Predicting social impairment and ASD diagnosis in
younger siblings of children with autism spectrum
disorder. Journal of Autism and Developmental
Disorders, 39, 1381–1391. http://dx.doi.org/10.1007/
s10803-009-0753-0
15
 Chapter 2
The Life and Times of
Nonverbal Communication
Theory and Research: Past,
Present, Future
Caroline F. Keating
When I entered social psychology graduate school
in the 1970s, information about nonverbal com-
munication was more likely found in the popular
press than in professional journals. According to
most of our professors, this was just where such
material belonged. They took a dim view of the
subject matter, seeing little scientific merit in what
appeared to be either extraneous movements of
the body or exotic habits of people not typically
found in the university subject pool. In those years,
graduate seminars focused instead on mathematical
formulae for attitude change, learning and reinforce-
ment theory, language, intelligence testing, and
cognition. Even the study of face-to-face interac-
tions favored verbal over nonverbal interpersonal
processes—small wonder that the so-called group
“risky shift” turned out to be a mirage. Only a hand-
ful of graduate programs offered courses in non-
verbal communication at the time (Knapp, 2006).
Furthermore, even though seminars on emotion
were part of the general psychology curricula, that
subject matter was taught as if it resided in a paral-
lel universe, distant from mainstream problems in
cognition and communication. The daily lesson was
that “real” psychology had little to do with the non-
verbal and everything to do with considered thought
and language processes.
Late in the evenings, however, when graduate
students congregated in the dark hallways of aca-
demic buildings, a very cool, older graduate student
in my program—a trim fellow who dressed in tight
suede vests and cowboy boots—would occasionally
pass around a rolled-up copy of the glossy, popular
http://dx.doi.org/10.1037/14669-002
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
magazine called Psychology Today. He claimed he
read it for the articles: “Communication With-
out Words” (Mehrabian, 1968), “The New Truth
Machine: Does Your Voice Give You Away?”
(Rice, 1978), and “How Well Do You Read Body
Language?” (Archer & Akert, 1977). Surreptitiously
shared among us graduate students like some sort of
porn, the magazine would eventually make its way
around to me. The intrigue alone was enough to get
anyone hooked, and I was no exception.
It was a timely addiction. After largely slumber-
ing through the post-Darwinian years of the 19th
century, the scientific study of nonverbal commu-
nication was on the cusp of reawakening and revo-
lution; fittingly, it had begun to stir in the United
States during the 1960s. The intellectual climate
of the time was swept up in a cultural storm that
encouraged anticonventional thinking and a rejec-
tion of cultural values, all of which were expressed
in the sensibilities of the day: uncut hair, limited
use of deodorant but heavy musk in perfumes,
beads and moccasins, altered states of conscious-
ness, and music featuring throbbing base tones and
drums that pulsed through the entire body and
made it move. The 1960s were about getting back
to our animal nature and tribal roots in a very
nonverbal way.
In academic circles, evidence of this awakening
was flagged by the reprinting of key volumes around
that time. Charles Darwin’s (1872) publication,
The Expression of the Emotions in Man and Animals,
was republished by The University of Chicago Press
in 1965. Efron’s (1941) interesting read on gesture,
17
Caroline F. Keating
race, and culture was reprinted in 1972. Huber’s
(1931) work on facial anatomy and expression was
also reprinted in 1972. The buds of a revived inter-
est in nonverbal communication could be found in
academe.
Outside of academe the bloom in interest was
on, nourished by the widening influence of televi-
sion. Small black and white screens and later, larger,
color ones sprang up in living rooms and kitchens
everywhere. Shows such as To Tell the Truth and
Candid Camera entertained viewers by putting their
nonverbal communication sensitivities to the test.
Television served a primal instinct, making a big,
impersonal world seem more face-to-face. From
home, viewers watched as national and international
leaders, politicians, and celebrities were freshly
disrobed by the camera’s capture of each spon-
taneous twitch of the body accompanying every
crafted word. The would-be influencers had to be
actors—good ones—or be crushed by television’s
power to reveal telltale nonverbal signs of insincer-
ity and vulnerability. Moreover, nonverbal commu-
nication skill could melt the coolness of the medium
(McLuhan, 1964), enabling good performers to
stoke passions and fan beliefs. Television opened
a door to popular interest in understanding primal
channels of communication and there developed a
hunger for science to enter the fray.
In the decades of the 1950s, 1960s, and 1970s,
the academic pioneers who took up the study of
nonverbal communication often resorted to the
popular press to promote their ideas. As early as
1951, Sheldon’s (1940) very visual work connect-
ing human physique and character (The Varieties
of Human Physique) appeared in popular form on
the slick pages of Life magazine. Biologists and
ethologists probed Darwinian connections between
animal and human communication in paperback
books: for example, Desmond Morris (1967) in The
Naked Ape, Ashley Montagu (1971) in Touching,
Jane Goodall (1971) in In the Shadow of Man, and R.
Dale Guthrie (1976) in Body Hot Spots. Psychologist
Robert Sommer’s (1969) paperback, Personal Space,
described how humans and other species commu-
nicate through the use of space. Publications writ-
ten from a cultural perspective included those by
anthropologists Ray L. Birdwhistell (1970, Kinesics
18
and Context) and Edward T. Hall (1959, The Silent
Language; 1966, The Hidden Dimension) as well as by
symbolic interactionist Erving Goffman (e.g., 1959,
The Presentation of Self in Everyday Life; 1967, Inter-
action Ritual: Essays on Face-to-Face Behavior), who
saw language, posture, gaze, and gesture as a sort of
orchestral arrangement.
The popularity of these paperbacks was a sign
of the times: Some early editions even showed up at
Woodstock. However, popular interest in nonverbal
communication was not just counterculture. Jour-
nalist Julian Fast’s (1970) Body Language was a New
York Times bestseller (Weitz, 1974). Books by engi-
neer-turned-psychologist Albert Mehrabian (1971,
Silent Messages) and psychiatrist Albert E. Scheflen
(1972, Body Language and Social Order; 1974, How
Behavior Means) also flew off bookstore shelves. In
many a mid-1970s graduate student’s office, tow-
ers of paperbacks served as bookends for hardcover
textbooks, professional handbooks, and journals.
Academic outlets began to warm to the infant
science of nonverbal communication. By the late
1960s, Sommer (1967) published his research on
interpersonal space and communication in Psycho-
logical Bulletin, and Mehrabian’s (1969) studies on
status signaling via posture and position appeared
on its pages along with Duncan’s (1969) article,
“Nonverbal Communication.” The Journal of Person-
ality and Social Psychology featured work on paralan-
guage by Dittmann and Llewellyn (e.g., 1967, 1969),
and in the early 1970s, it showcased work by Dun-
can (1972). In Acta Psychologica, Kendon (1967,
1970) wrote on gaze and on the nonverbal aspects of
what would be called rapport today (Tickle-Degnen,
2006). Watson’s (1972) work on proxemics was
published in the Journal of Communication. Articles
by Mehrabian (1972b), Ekman (1972), and Exline
(1972) appeared in the prestigious Nebraska Sym-
posium on Motivation (Cole, 1972). At about the
same time, three textbooks were published with
“nonverbal communication” in their titles (i.e.,
Eisenberg & Smith, 1971; Knapp, 1972; Mehrabian,
1972a). Articles on nonverbal behavior showed up
with some regularity in flagship psychology journals
as the 1970s progressed, and Semiotica—largely
devoted to nonverbal research—was launched. Jour-
nals dedicated to verbal discourse (e.g., Journalism

Quarterly, Communication Monographs, and Human
Communication Research) began to welcome articles
on nonverbal communication (e.g., Burgoon, 1978;
Burgoon & Jones, 1976), and Burgoon and Saine
(1978) published the text, The Unspoken Dialogue.
Still, serious, sustained work on nonverbal commu-
nication lagged the intense interest reflected in the
popular press.
A notable exception to the gap in interest came
from researchers studying emotion. Scholars such as
Silvan S. Tomkins had persisted in studying human
emotion from a Darwinian perspective, despite the
epic events of World War II and Skinnerian psychol-
ogy. Tomkins (1962, 1963) tied emotion to expres-
sion, and his ideas gathered steam in the late 1960s
and 1970s as Paul Ekman, Cal Izard, and their
colleagues synergized the field. In addition, Ekman,
Sorenson, and Friesen’s (1969) seminal research
on cross-cultural facial expressions of emotion was
published in Science. Izard (1971), who conducted
developmental as well as cross-cultural research,
published his findings in The Face of Emotion.
Altogether, these theorists shaped much of what
is believed about emotion and expression today
(see also Chapter 10, this handbook).
Meanwhile, scholars operating from an ethologi-
cal perspective blended knowledge from animal
communication studies with human research.
Iranis Eibl-Eibesfeldt (1972), a student of ethologist
Konrad Lorenz, published his extensive work on
culture and nonverbal communication in Robert A.
Hinde’s (1972) edited volume, Nonverbal Communi-
cation. That compendium of work drew connections
between primate and human expression, including
van Hooff’s (1972) compelling argument that the
primate grin face and human smile are homologous
(Keating, Mazur, Segall, et al., 1981). Hewes (1973)
contributed important cross-species work in Current
Anthropology, arguing that gesture was at the root
of human language, an idea that survives in new
forms today (e.g., Bavelas & Chovil, 2006; Goldin-
Meadow, 2005; Kelly, Hansen, & Clark, 2012; Kendon,
2004; D. McNeill, 1992; see also Chapter 12, this
handbook). Blurton Jones (1972), Konner (1972),
and McGrew (1972) applied ethological techniques
to understand children’s nonverbal communication.
Thus, the squeeze of evolutionary thinking stuck
The Life and Times of Nonverbal Communication
human to nonhuman communication, upping
the ante for those researchers betting on spoken
language and cultural artifact alone.
Even bigger ideas linking all of human and
animal social behavior arrived with fanfare and
controversy upon publication of Harvard biologist
E. O. Wilson’s (1975) weighty and explosive vol-
ume, Sociobiology. Its central message—that human
and nonhuman social behavior fit patterns shaped
by natural selection—was exquisitely complemented
by philosopher Daniel C. Dennett’s (1978) Brain-
storms. Among other things, Dennett’s patient
argumentation maintained that the behavioral
displays of animals reflect social intentions akin to
those inferred for humans. These arguments were
made more appetizing by skillful, cross-over writers
such as Stephen J. Gould (1973, Ever Since Darwin)
and Richard Dawkins (1976, The Selfish Gene),
who commanded both the scientific and the popular
stage. Thus, the intellectual backdrop framing the
new popularity of functional approaches to human
nonverbal communication was set in the broad,
scientific milieu of the time.
OVERVIEW
My task from here on in this chapter is to trace the
scholarly lineages of theory and research in non-
verbal communication and to identify the thrust of
their projections into its future. This is a daunting
quest—impossible, really—because many creative
minds from different areas of expertise over decades
and even centuries contributed to our present-day
understanding of human nonverbal communication.
Scholarly works on gesture and rhetoric can be
found in the writings of philosophers, teachers, and
politicians ranging from Confucius in the 6th cen-
tury, BC, to Aristotle (around 350 BCE), to Cicero
and other Roman orators centuries later (Knapp,
2006). Ideas from these early times permeate much
of how we conceptualize human psychology, includ-
ing nonverbal communication. This neat find, a
quote from Socrates, makes the point:
Nobility and dignity, self-abasement and
servility, prudence and understanding,
insolence and vulgarity, are reflected in
the face and in the attitudes of the body
19
Caroline F. Keating

whether still or in motion. (Xenophon, mimicry, and synchrony (Buck, 1984; Buck & Ren-
Memorabilia [III.x]) fro Powers, 2006; Van Vugt, Hogan, & Kaiser, 2008;
Wiltermuth & Heath, 2009). Signaling intentions
The idea that static as well as dynamic nonver-
and status was a way to avoid costly, physical con-
bal channels have communicative value has out-
frontations and to enhance biological fitness because
lived centuries and is represented in the research
signaling was less risky and energetically cheap
reviewed here.
(Alexander, 1974; Caryl, 1979; Dawkins & Krebs,
Our analysis fast-forwards to comparatively
1978; Mazur, 1985).
recent developments in nonverbal communication
As civilizations developed on different conti-
theory and research. While acknowledging intel-
nents, humans ritualized nonverbal messaging by
lectual inspirations from the more distant past, our
developing gestural protocols, cosmetics and other
pursuit of major themes begins in the 1960s, the
body adornments, costumes, music, and dance
decade that Knapp (2006) considered a “tipping
aimed at deepening social roots and expanding
point” (p. 11) in the modern incarnation of nonver-
social branches (Eibl-Eibesfeldt, 1972, 1975; Etcoff,
bal communication research, and Knapp, Hall, and
Stock, Haley, Vickery, & House, 2011; W. H.
Horgan (2014) described as a “nuclear explosion
McNeill, 1995). Today, nonverbal communication
of the topic” (p. 23). We focus on general, theoreti-
skill is believed essential to emotional intelligence
cal perspectives that influenced research through
and, for better or worse, to charismatic leadership
decades. Rather than cataloging years of literature,
(Goleman, Boyatzis, & McKee, 2002; Keating, 2011;
I have excised from its corpus theoretical struc-
Riggio & Riggio, 2010; see also Chapter 23, this
tures that gave direction to contemporary, nonver-
handbook). Therefore, it seems that, at many levels,
bal communication research and that evoke new
humans understand the mighty power of nonverbal
questions.
communication quite well.
Important elements of this science story are
At the scientific level, the processes guiding
missing from these pages. Theoretical models
nonverbal communication are less well under-
related to particular research endeavors (e.g., mod-
stood. Over the decades, behavioral scientists have
els related to intimacy, parent–infant interaction,
approached the problem from two distinguishable
and race relations) are not covered. Also missing
traditions. One emphasizes affective states underly-
from this chapter is discussion of the role that
ing expression. The second emphasizes communi-
nonverbal communication plays in embodied per-
cation as an interactive process, contextualized by
ception and emotion regulation. In addition, the
social relationships and social motives. Biological
crucial role of advances in the technologies used to
and cultural elements characterize both. Although
study nonverbal communication receives only brief
the central foci of each tradition are discernibly
mention.
different, they share substantial overlap: Some
theorists have operated within each tradition,
APPROACHES TO NONVERBAL
whereas others have blended them. The two large
COMMUNICATION
circles in Figure 2.1 represent each emphasis and
Like any form of communication, nonverbal com- the overlap between them. This chapter proceeds by
munication is social; it makes information accessible exploring some of the major theoretical approaches
to others (W. J. Smith, 1977). At some level, nonver- developed under each scholarly tradition, plotted so
bal communication causes a change in the recipient; as to reflect its emphasis (or emphases).
the expresser, of course, is hoping for the intended
change. Long before the pen was believed to be Emotion and Expression
mightier than the sword, the expressive, nonverbal At the very least, nonverbal behavior can be said to
arts carried the day, enabling coordinated action enable the expression of genuine, internal states, as
among group-living prehominids and humans neurophysiological evidence has shown over the
through signaling, modeling, display, contagion, past 30 years (see Dalgleish, 2004, for a review).

20

CONCEPTUAL EMPHASIS ON
EMOTION
BASIC EMOTIONS
DARWIN 1872
TOMKINS 1962
EKMAN 1969
IZARD 1971
MATSUMOTO 1990
FIGURE 2.1.  A depiction of differing conceptual emphases on emotion versus communi-
cation in nonverbal communication research. The theories and models listed serve
as examples of the differing degrees to which “basic” emotions are imputed from
nonverbal behavior.
Just as importantly, displays of these states
draw affective responses from others (Hatfield,
Cacioppo, & Rapson, 1994; Lishner, Cooter, &
Zald, 2008; Moody, McIntosh, Mann, & Weisser,
2007; Ruys & Stapel, 2008), which is likely why
we often do not keep our affect to ourselves. The
contagion of affect may be partly responsible for
the evolution of the emotion communication platform
from primitive nervous systems that delivered fast,
reflexive or relatively automatic responses designed
to save skins, to more elaborated, conscious, differ-
entiated, and complexly communicative systems that
helped save the skins—and genes—of kin and coali-
tion partners (de Gelder, 2006; Keltner & Haidt,
1999; Tamietto & de Gelder, 2010). In fact, as
knowledge of brain systems expands, the more evi-
dence there is that cognitive mechanisms are integral
to human emotion processes, and the more difficult
it is to argue that emotion leads to expression in any
simple way (Pessoa & Adolphs, 2010; Posner, Rus-
sell, & Peterson, 2005; Salzman & Fusi, 2010).
The Life and Times of Nonverbal Communication
COMMUNICATION
INTERACTIONISTS
of the 1950s, 60s, 70s
LENS MODELS
GIFFORD 1994
SCHERER 2003
AFFORDANCES
ZEBROWITZ 1983
MONTEPARE
TICKLE-DEGNEN 2006
STATUS CUES
MAZUR 1973
KEATING 1985
BEHAVIORAL ECOLOGY
VIEW
FRIDLUND 1994
PARALLEL PROCESS
MODEL
PATTERSON 2006
No surprise, then, that how emotion and expres-
sion relate remains controversial. For that matter,
conceptualizing “emotion” stirs debate, though
most theorists would probably agree that emotional
reactions to environmental events cascade through
multiple processes: physiological responses, action
tendencies, subjective feelings, expression, cognitive
appraisal, and learned habits among them, but not
necessarily in that order (Mulligan & Scherer, 2012).
Over the decades, models of emotion have differed
in the degree to which they conceive of emotions
as a function of internal states versus component
processes (Moors, Ellsworth, Scherer, & Frijda,
2013; Scherer & Ellgring, 2007). Some theorists
view emotions as basic, discreet categories (Darwin,
1872/1965; Ekman, 1972; Ekman et al., 1969; Izard,
1971; Tomkins, 1962, 1963). Others construe emo-
tions along dimensions of internal experiences (Pos-
ner et al., 2005; Russell, 1980; Schlosberg, 1954;
Wundt, 1897). There are prototype models of emo-
tion (Plutchik, 1980; Shaver, Schwartz, Kirson, &
21
Caroline F. Keating
O’Connor, 1987) and appraisal theories of emotion
(e.g., Arnold, 1960; Lazarus, Averill, & Opton,
1970; Scherer, 2001; Scherer, Clark-Polner, & Mor-
tillaro, 2011). The study of nonverbal communica-
tion is impacted by these differing views of emotion,
and nonverbal displays implicate important distinc-
tions among models of emotion, as I show later.
Did Darwin conjure these complexities in the
emotion communication story? Darwin spent the
mid-19th century studying emotion and expres-
sion, and he published his exceptional work in The
Expression of the Emotions in Man and Animals in
1872. He envisioned a set of core, human expres-
sive categories, among them joy, grief, fear, sur-
prise, anger, contempt, disgust, pride, and shame.
The latter Darwin (1872/1965) linked to blushing,
which he considered to be “the most human of all
expressions” (p. 309). Darwin took the core set of
expressions to be universal, offering as evidence
observations they each (a) appeared early in life,
(b) were evident among blind-born people, and
(c) could be found in every culture. He attributed
their ubiquity to genetic inheritance. Darwin’s
observations have since been corroborated by
formal studies, and many human emotion theo-
rists concur with Darwin’s basic assumptions (see
Ekman, 2003).
Darwin (1872/1965) described an evolution of
expression separate from emotion itself. He argued
that elements of human and animal expressions
and gestures evolved sometimes as a function of
the nervous system (e.g., blushing), sometimes as
associated habits that proved useful or serviceable
when responding (e.g., weeping when sad), and
sometimes because display elements made counter
responses unlikely (e.g., smiling’s incompatibility
with biting). Darwin observed that two expressive
hallmarks of disgust—the wrinkling of noses, which
closed the nostrils to defend against bad odors, and
tongue protrusions, which facilitated the expulsion
of bad food—were linked to common sources for
feelings of disgust. To Darwin, then, facial expres-
sion and emotion had collateral but not necessar-
ily shared origins (Fridlund & Russell, 2006) that
may have adaptively converged over time. Argu-
ments both for and against that proposition can
be unearthed from the pages of Darwin’s extensive
22
observations and analyses. Either way, what Darwin
observed was fuel for the evolution of communica-
tion because, as geneticist Ernst Mayr (1988) put it,
“Behavior is the pacemaker of evolution” (p. 408).
Even epigenetic influences can drive evolution (Rid-
dihough & Zahn, 2010).
A century after Darwin, differing interpretations
of his scholarship resulted in what Fridlund and
Russell (2006) described as a “bifurcation” (p. 300)
of the foundational explanations for facial expres-
sion. Actually, it was a fork in the road along the
modern theoretical landscape of nonverbal com-
munication theory and research more generally.
Psychologists studying human emotion adopted a
view whereby expressions evolved expressly because
they signaled basic emotions. Thus, emotion and its
expression were seen as one: If you had an emotion,
at some level you produced its expression (Ekman,
1972; Izard, 1971).
In contrast, scholars from ethological or eco-
logical perspectives interpreted expressions and
gestures as conveyers of intentions not necessarily
linked to underlying, emotion states. You could
intend and signal something but not feel it, or feel
something and neither signal it or intend to carry
it out. Researchers from cross-species traditions
were especially hesitant to infer causative, unitary
drive states from outward, expressive behaviors
(Hinde, 1959), though many recognized evolu-
tionary continuity between the expressive behav-
iors of human and nonhuman primates (Andrew,
1963; Chevalier-Skolnikoff, 1973; Hewes, 1973;
Pitcairn & Eiblesfeldt, 1976; Redican, 1982; van
Hooff, 1972). More recently, however, primatolo-
gist Frans B. M. De Waal (2003) attributed human-
like felt emotion to the expressions of select great
apes. In contrast, others drawn to the ethological
perspective included systems-theory interaction-
ists, for whom the meaning of behavior derived
not from internal states but from interaction best
studied in its natural, social habitat (e.g., Scheflen,
1972). Whether from the emotion or ethological/
ecological tradition, the decades-long theoretical
divide between the degree to which researchers
imputed basic emotion to expression spawned
divergent approaches to the study of nonverbal
communication.

Thus, to human emotion researchers, facial
expressions provided a window on emotion itself:
its phylogeny (e.g., Darwin, 1872/1965; Tomkins,
1962), ontogeny (e.g., Darwin, 1877/1956; Izard,
1971; Oster, 2005), typology (e.g., Darwin,
1872/1965; Ekman, Levenson, & Friesen, 1983),
intensity (e.g., Matsumoto & Ekman, 1989), and
cultural universality or specificity (Ekman &
Friesen, 1971; Matsumoto, 1989; Matsumoto, Olide,
Schug, Willingham, & Callan, 2009). A core set of
emotions was believed temporally linked not only to
phenomenological experience and discreet, bodily
responses but also to sets of specific, facial muscu-
lature movements that express them (Ekman, 1972;
Izard, 1971). Precise study of the facial movements
accompanying different emotions led to the eventual
development of the Facial Affect Coding System
(Ekman & Friesen, 1978), which subsequently
proved to be a valuable coding tool for research-
ers studying a variety of expressive behaviors (e.g.,
Ekman & Rosenberg, 2005). In essence, facial
expression, though not always visible to the naked
eye, was considered part and parcel of a congruent
response system peculiar to a particular emotion:
It is seen this way by many researchers today.
Both Ekman and Izard took strong, universalist
positions, arguing that “basic” categories of emo-
tion were common to humankind and, therefore,
had shared expressive features. The basic emotion
categories identified and tested by these theorists
through exhaustive, empirical research overlapped
with Darwin’s expressive categories: happiness
(or joy), surprise, anger, fear, sadness, and disgust.
Izard (1971), who studied infants, included an emo-
tion he designated as interest. Eventually Ekman,
following Darwin’s lead, added contempt to his basic
six (Ekman & Friesen, 1986). The idea was that
fundamental affects could be expressed in pure form
or blended with one another, thereby accounting for
the more nuanced emotions and expressions evident
in everyday life (Ekman, 1972; Izard, 1971).
Culture is a natural laboratory for testing
arguments for and against the universality of
human emotions. Over decades of cross-cultural
and cross-national research, human antennae have
been found to be sensitive to the six or seven basic
facial expressions of emotion that Ekman and Izard
The Life and Times of Nonverbal Communication
defined (Ekman, 1972; Ekman et al., 1969; Izard,
1971, 1997). When given the opportunity to assign
posed expressions of emotions to prelabeled emo-
tion categories, agreement has been remarkably high
across cultures. Furthermore, when either asked
to pose basic emotions or when stimulated with
particular, emotion-eliciting stimuli, participants
from Western-centric and non-Western cultures as
far-flung as Papua New Guinea produce expressions
recognizable to the majorities of perceivers from all
backgrounds (see Matsumoto, 2009, for a review).
Overall, the evidence was taken to mean that people
from very different cultural backgrounds express and
interpret basic categories of emotion in similar ways.
Some evidence from brain, physiological, and
anatomical studies is largely consistent with the
construal of the proposed basic, emotion categories.
Neural pathways carry emotion signals to and from
different response centers quickly, often without
much cortical involvement (e.g., LeDoux, 1996; cf.
Dalgleish, 2004). Developmentally, the brain seems
tuned to the basic, facial expressions of emotions
early in life (Leppänen & Nelson, 2012). Brief expo-
sure to basic emotional expressions triggers matched
facial behavior (as measured by electromyography)
and reported emotion in adults (Lishner et al.,
2008). Adult brain responses to facial displays of
fear, anger, happiness, sadness, disgust, and
surprise register distinct patterns of process-
ing activity that are rapid and appear automatic
(Batty & Taylor, 2003). Patterned, physiological
responses differentiate emotion from reflexes such
as the startle (Ekman, Friesen, & Simons, 1985) and
also distinguish emotion types (e.g., Ekman et al.,
1983; Levenson, 1992). For example, temperature
rises in response to anger and falls in response to
surprise (Ekman et al., 1983). Anatomical studies of
cadavers reveal that the facial musculature needed to
express the basic emotions is reliably present, sug-
gesting that natural selection favored these specific
emotions (Waller, Cray, & Burrows, 2008). Thus, to
some it appears that humans come prefabricated to
encode and decode specific types of emotion-related,
nonverbal communication. Perhaps more impor-
tantly, the idea that emotion categories organize
human emotional experience and expression has
mustered neuroanatomical “legs.”
23
Caroline F. Keating
Theorists posited early on that humans came
biologically prepared not simply to express affec-
tive states but also to control their display. From a
basic emotions perspective, researchers identified
cultural habits of expression or display rules that
modified the communication value of expressions
(Ekman & Friesen, 1969; Matsumoto, 1990; also see
Matsumoto & Hwang, 2013). These modifications
were often induced by social contexts. Interactants
and audiences, both real and imagined, were able to
attenuate a person’s expression, and to sometimes
amplify it, and at other times alter the type of emo-
tion conveyed (e.g., Manstead, Fischer, & Jakobs,
1999). Genuine emotion is sometimes nowhere to
be (easily) seen.
When it comes to out-and-out lying, the implica-
tions of theoretically tying emotion to expression
with a tight knot are profound, for to catch a liar
means detecting hidden signs of genuine feeling.
Research pioneers began by analyzing frame-by-
frame videotapes of clinical patients (Ekman &
Friesen, 1969). More recent success has been
achieved by identifying split-second, microexpres-
sions that presumably flag true, underlying emotion
(Ekman, 2009; Frank & Svetieva, 2013; Porter &
ten Brinke, 2008). The idea that both face and
body are potential sources of “leaked” emotion and
deception clues has pop culture appeal, as evidenced
by the popular Fox TV series, Lie to Me, which is
based on the Ekman group’s research on deception.
Though face and body offer important clues to
true feelings (Ekman & Friesen, 1969), the preemi-
nence of the face as a signaling channel for emotion
was only occasionally challenged during decades
of research on emotional expression. One early
exception was Davitz (1964), who found that lis-
teners could perceive emotions from U.S. speakers
instructed to perform different emotional recitations
of the English alphabet. Since then, evidence has
shown that emotion categories can be derived from
vocal communication across languages and cultures
(Fridlund, 1994; Sauter, Eisner, Ekman, & Scott,
2010; Scherer, 2003; Scherer, Banse, & Wallbott,
2001), as Darwin (1872/1965) anticipated (see also
Chapter 11, this handbook).
More recently, the emotion categories approach
has been applied to nonverbal channels beyond face
24
and voice. Five basic emotions (anger, fear, hap-
piness, sadness, and disgust) plus three prosocial
emotions (love, gratitude, and sympathy) were
successfully encoded and decoded (enacted and
categorized consistently) through touch (Herten-
stein, Holmes, McCullough, & Keltner, 2009). Body
posture and body expression convey basic emotions
(de Gelder, 2006), augment facial affect displays
(Van den Stock, Righart, & de Gelder, 2007), and
are possibly best at signaling extreme emotions
(Aviezer, Trope, & Todorov, 2012). Different kinds
of laughter convey different kinds of emotion as well
(Szameitat et al., 2009). Emotion has a fleet of non-
verbal vehicles to convey it, and researchers have
only begun to catch the ride (see also Chapter 17,
this handbook).
Questions and Controversies
Despite evidence for universalist positions on
human emotion and expression, there is a pervasive
worry that the methodologies used to test the ubiq-
uity of Western-derived human emotion categories
fall short in meaningful ways (e.g., Elfenbein &
Ambady, 2002; Russell, 1994; Scherer et al., 2011).
Much of the supportive research relies on precon-
ceived emotion categories developed in the West,
forced choice response formats, heavy reliance on
recognition protocols, recognition of posed (not
spontaneous) emotion, still photographs, a preoccu-
pation with faces, emotion judgments of culturally
unfamiliar faces (often professional actors), and lim-
ited samples of international respondents in terms
of social class and education (Elfenbein & Ambady,
2002; Scherer et al., 2011). In addition, variability
in emotion encoding and decoding within cultures
has seldom been studied (see Safdar et al., 2009, for
an exception). The uniformity of the cross-cultural
results depends to some degree on the methodology
used; generally, the more constrained the proce-
dures, the greater the agreement (see Elfenbein &
Ambady, 2002, for a meta-analysis and review).
Thus, some question whether the most basic, uni-
versalist assumptions—that humans “chunk” and
experience emotions similarly, and express and
interpret emotional experience in a consistent
fashion—are fully supported by the available data
(Fridlund & Russell, 2006; Scherer et al., 2011).

Has research on emotion and facial expres-
sion been blinded by the bright lines of Western,
imposed-etic construction of emotion categories?
Early, open-ended efforts to elicit non-Western
conceptualizations and translations of affective
experience and expression seem to have been short-
circuited, perhaps by the apparent success of the
Western, etic approach. Alternative, affect-related
models of nonverbal processes that focused on gen-
eral arousal received little traction (e.g., Andersen,
Guerrero, Buller, & Jorgensen, 1998; Argyle &
Dean, 1965; Cappella & Green, 1984). Meanwhile,
evidence confirming that emotion elicitors, apprais-
als, and expressions were aptly described by the six
or seven basic emotion categories was undeniably
strong. Yet, when researchers go looking for emo-
tion in far-off places, they tend to bring their basic
categories with them or to match what they find to
them, without offering competing conceptualiza-
tions. Stimuli, response formats, and measurement
techniques in laboratory settings are operational-
ized in such a way as to favor the anticipated, basic
emotion categories. Furthermore, there is reason to
believe that we may need to step outside them.
Expressions gathered from real-world situations
and real-world reports of experienced emotion are
sometimes poorly related (Carroll & Russell, 1996;
Parkinson, 2005); what people claim to feel is no
guarantee for how they appear to feel, or even how
they think they appear to feel, or how they theoreti-
cally should feel, given situational events (Reisen-
zein, Studtmann, & Horstmann, 2013). Tickling
produces Duchenne (pleasure) smiles experienced
as unpleasant (Harris & Alvarado, 2005). Difficulty
distinguishing between the smiles of victorious and
disappointed Olympic athletes (Fernández-Dols &
Ruiz-Belda, 1995) and the tears of winning game
show contestants are high-profile examples of the
emotion/expression discontinuity—others are more
mundane (Camras, 1992). When affect is expressed
spontaneously, it rarely matches the posed, emo-
tion prototypes seen in manuals and used as stimuli,
leading to concerns about their ecological valid-
ity (Russell & Carroll, 1999). There is disconnect
between expression and expected experience.
Wilhelm Max Wundt (1897) proposed that emotions could be described by three dimensions: “pleasurable versus unpleasurable,” “arousing versus
1
subduing,” and “strain versus relaxation.”
The Life and Times of Nonverbal Communication
Meanwhile, expressions of embarrassment,
shame, and pride are common in everyday life, and
each is expressed in highly choreographed ways
(e.g., Keltner, 1995; Tracy & Robins, 2008). How-
ever, none are considered basic emotions (Ekman &
Rosenberg, 2005; Izard, 2007; Matsumoto, Frank, &
Hwang, 2013). Instead, these displays are believed
to draw their expressive elements from the basic
emotions that we rarely see in prototypical form.
Perhaps shame draws from sadness (Ekman &
Rosenberg, 2005)—or given the evolution of our
social brain (Dunbar & Shultz, 2007), could it be
the other way around?
Kinks in the chain linking feeling, expression, and
action may lead to new ways of thinking about basic
emotion. For example, 4- to 9-year-old children in
the United States tend to describe prototypical facial
poses of disgust as anger, suggesting that translating
the world into basic categories of emotions requires
some learning (Widen & Russell, 2010); synchro-
nous emotional exchanges between parents and
infants may be an early teacher (Feldman, 2007).
There are quirky aspects to the expres-
sive components of faces that require expla-
nation. For instance, expressions of extreme
pleasure—specifically, pleasure produced by
orgasm—are nearly indistinguishable from expres-
sions of extreme pain when facial affect units are
compared using the Facial Affect Coding System
(Fernández-Dols, Carrera, & Crivelli, 2011; cf.
Hughes & Nicholson, 2008). Apparently, displays
of ecstasy and excruciating pain occupy adjacent
positions on the signal configuration dial but oppo-
site ends on any approach/avoidance or pleasant/
unpleasant dimension (see Figure 2.2). This puzzle
seems not well explained by the current categorical
model of emotion and could supply theorists inter-
ested in dimensional models of affect opportunities
to map it.1 Different kinds of laughter, for example,
have emotional meanings mapped slightly better
by dimensions than categories (Szameitat et al.,
2009). Similarly, dimensional rather than categori-
cal construal of emotion may aid in understanding
anger, a drive state in which the motivational com-
ponent, approach, is conceptually separable from its
25
Caroline F. Keating

B.
A.

D.

C.

FIGURE 2.2.  Pain or pleasure? The elements of facial expression in response to extreme
pain and extreme sexual pleasure overlap. Picture A: male pleasure; Picture B: male pain;
Picture C: female pleasure; Picture D: female pain. Reprinted from “Sex Differences in
the Assessment of Pain Versus Sexual Pleasure Facial Expressions,” by S. M. Hughes and
S. E. Nicholson, 2008, Journal of Social, Evolutionary, and Cultural Psychology, 2, p. 297.
Copyright 2008 by the American Psychological Association.

(negative) valence (Carver & Harmon-Jones, 2009),
and whose facial expression is most often con-
fused with that of disgust, which signals avoidance.
Expression is trying to tell us something about the
nature of emotion that we do not yet understand.
Approaches Emphasizing
Communication Processes
Scholarly traditions that emphasized communication
as an interactive process unleashed expression from
emotion categories. Those attracted to studying
nonverbal communication as social process
included a new wave of linguists and an array of
neo-Darwinian psychologists bent on escaping the
constraints of reinforcement theory (see Figure 2.1).
Who would have imagined that researchers study-
ing human language from a cultural point of view
and others keen on cross-species analyses would
end up at roughly the same place—not at the same
theoretical table but at least at the same data buffet?
Such was the hunger for exploring the social nature
of human interaction in the “back-to-nature” 1960s.

26

System-Theory Approaches to Interaction
A few rogue linguists, some inspired by Goffman’s
work, were early converts to the idea that meaning
could be extracted from interactional, nonverbal
behavior. The idea became popular with structural
linguists who, from a strictly cultural point of view,
had adopted a systems-theory approach to commu-
nication in the 1950s. At Stanford, anthropologists
Gregory Bateson and Ray L. Birdwhistell and, later,
psychiatrist Albert E. Scheflen, embraced this new,
holistic perspective on human interaction (Scheflen,
1972, 1974). Frustrated by linguists for whom
communication meant only words, Scheflen (1974)
expressed the obvious: “We could not find meaning
in words, any more than we could find conscious-
ness in a nerve cell or love in a gonad” (p. 2).
Scheflen’s angst fueled his informal but insightful
analyses of nonverbal exchanges between bosses
and workers, between therapists and clients, and
between the romantically inclined, thereby breath-
ing life into the study of kinesics, proxemics, and
paralanguage (Scheflen, 1972, 1974). Whether these
converts worked from a structural approach, seeking
language-like rules for nonverbal behavior, or from
an external variable approach, in which features of
the communicator and situation were examined, the
focus was on communication (Duncan, 1969).
Contrary to many of his predecessors and cohorts
(e.g., Birdwhistell, Efron, Goffman, and Hall),
Scheflen’s (1972, 1974) analyses of human nonver-
bal communication incorporated what was known at
the time about nonhuman primate communication.
Similar to Sommer (1967, 1969), who analyzed the
use of personal space, Scheflen saw kinship between
the nonverbal behaviors that structured the social
worlds of humans and other primates. In combina-
tion with culture, he viewed nonverbal commu-
nication through the lens of interactive processes.
Scheflen’s synthesis was handwriting on the wall:
By the 1980s and 1990s, many linguists had come
to believe that face-to-face dialogue, not written texts
or monologues, represented the fundamental arena
within which human language evolved and could be
understood (see Bavelas & Chovil, 2006).
The elevation of interactive dialogue set the
study of kinesics and paralanguage in motion. In
fact, motion—and timing—was everything: Meaning
The Life and Times of Nonverbal Communication
was imbued in split-second sequences connecting
dynamic expressions and gestures with interac-
tion sequences and dialogue. That gestures and
speech co-occur seemed obvious, but what links
them? Researchers deciphered patterns of nonver-
bal behaviors that regulated interaction (including
greeting and leave-taking) and conversation
(e.g., Dittmann, 1972; Duncan, 1969; Exline, 1972;
Kendon, 1970). Some researchers proposed formal
models, most notably equilibrium theory (Argyle &
Dean, 1965) and interaction adaptation theory
(e.g., Burgoon, Stern, & Dillman, 1995); their ele-
ments are evident in contemporary social-cognitive
models described later in this chapter.
Interactionists also recognized that the timing of
gestures provided meaning on its own. An analysis
of more than 1,700 gestures embedded in speech
revealed that 70% of them were associated with
words simply by timing and not by intrinsic mean-
ing (Bavelas & Chovil, 2006, p. 106). Human court-
ship behavior is a lot about timing: Characteristics
of motion, such as the speed of offset and onset of
behaviors rather than the specific behaviors them-
selves, predict female interest in males (Grammer,
Kruck, & Magnusson, 1998). Though the dots are a
long way from being connected, the interactive
dialogue approach reveals that nonverbal communi-
cation should be studied as a dynamic process.
Behavioral Ecology Approaches
At about the same time linguists began exploring
synergies between nonverbal and verbal behavior,
behavioral ecologists developed dynamic-system
approaches to the evolution of nonverbal commu-
nication. In true, neo-Darwinian style, signaling
systems were presumed to evolve by optimizing
the biological fitness of both signaler and recipi-
ent. For instance, when an aggressive chin thrust
or stare makes a competitor back down, each
adversary avoids the energetic costs and possible
injury resulting from physical confrontation; each
accrues genetic benefits. However, the engines of
natural selection are dynamic. If recipients are able
to detect that the signaler is bluffing, raising the
ante by threatening back could win the battle of dis-
play and earn access to resources. Aspects of game
theory apply: It behooves the signaler to develop
27
Caroline F. Keating
convincing displays (honest or otherwise), and
recipients to hone superior detection skills. These
combined forces conspire to create a frequency-
dependent, signaling arms race in which communica-
tion skills are shaped by the habits and abilities of
those most likely to be found in the social environ-
ment (Alexander, 1974; Caryl, 1979; Dawkins &
Krebs, 1978; Otte, 1974). The situation is iterative;
better detection encourages better signaling, and so
forth, so that some mismatch between cue validity
and cue utilization is expected. To remain in balance,
signalers ultimately benefit from the communica-
tion, but recipients also share in the spoils.
Unchaining expressions and displays from
basic emotions has advantages in understanding
animal and human communication. In the animal
world, cowbirds combine signs of aggression with
appeasement signals within a single display: First,
they assertively rush at invading birds near their
nests and, next, suddenly freeze in a submissive-
looking, bowed head position. Invaders typically
respond by preening. After a series of experimen-
tal manipulations, Rothstein (1980) determined
that the appeasement elements in the signalers’
display allowed them to appraise the aggressive-
ness of invaders. Thus, the meaning of the display
was unlocked: Cowbirds were assumed to feel
neither angry nor fearful, though they acted like
it. Cowbirds combined expressive elements to
assess other birds’ intentions and to reconfigure
their own. No wonder it is difficult to detect an
“angry” bird!
A confluence of signals also surfaces when
humans flirt (Moore, 2010). Observations of women
in bars reveals that flirting combines nonverbal
signals of dominance and submissiveness that sig-
nal coyness and are likely meant to invite qualified
approach. Though signs of “happiness” (smiles,
laughter), “surprise” (raised brows, raised voice
pitch), “contempt” (asymmetrical muscle contrac-
tions around the mouth, dismissive vocalizations)
parade across flirtatious faces and bodies, the lens of
emotion is not very useful in deriving the meaning
of the communication. From an ecological perspec-
tive, the clearest understanding of what is being
communicated is reached by figuring out its social
motive—in this case, inviting qualified approach via
28
graded status signals as part of a choosey, mating
strategy.
Behavioral ecology view.  The role of social
motives is key to Fridlund’s behavioral ecology view
of facial and vocal expression, which applies behav-
ioral ecology principles to human nonverbal com-
munication (Fridlund, 1994; Fridlund & Russell,
2006). Recently, Fridlund and Russell (2006) argued
that social motives are better predictors of facial
displays in real-life interactions than are emotions.
For these theorists, imputing emotions is unneces-
sary, whereas understanding “the projected plan of
action and its goal” is central to predicting the future
behavior of the displayer (Fridlund & Russell, 2006,
p. 311). Under this scheme, the momentary costs
and benefits of signaling are taken into account,
including calculation of the potential cost of
honesty. “Automatic readouts” of emotion would
be expected to be rare and deception much more
common (Fridlund & Russell, 2006, p. 310). When
social goals instead of emotions are presumed to
be the primary drivers of expression, facial expres-
sions of “emotion” are interpreted quite differently,
as Table 2.1 depicts. Most behavioral ecologists
and human ethologists would be more comfortable
reporting the contents on the right-hand side of the
table than the left.
Status cues perspective.  Communication patterns
extracted from behavioral ecology are also central to
the status cues perspective (Keating, 1985a, 1985b,
2002; see also Mazur, 1985, 2005, for his biosocial
model). Status information is embedded in both
static and dynamic face and body aspects in most
species, including humans. Static, age-related signals
in the structure and features of the face, for instance,
are carriers of status information that convey domi-
nance, which signals threat, and submissiveness,
which invites approach. Sensitivity to status cues
is fundamental to group-living species, as ignoring
them can pose existential threats or missed oppor-
tunities. Evidence of their importance to humans
comes from studies showing that threat judgments
of expressionless male faces are made in less than 39
ms and automatically access speedy, high-priority
(i.e., low-frequency) perceptual processing channels
in the visual system (Bar, Neta, & Linz, 2006).

TABLE 2.1
Interpretations of Facial Expressions From Emotion Category and Social Intention Points of View
Facial expression program
(facial expressions of emotion)
“Felt” (Duchenne) smile (readout of happiness)
“False” smile (feigned happiness)
“Sad” face (readout of sadness)
“Anger” face (readout of anger)
“Leaked” anger (inhibited anger)
“Fear” face (readout of fear)
“Contentment” face (readout of contentment)
“Contempt” face (readout of contempt)
“Poker” face (suppressed emotion)
Note. From The Sage Handbook of Nonverbal Communication (p. 305), by V. Manusov and M. L. Patterson, 2006,
Thousand Oaks, CA: Sage. Copyright 2006 by Sage. Reprinted with permission.
The status cues approach assumes that static
facial signals evolved to operate independently of
age, as is the case for other organisms (Keating,
1985a, 2002). Thus, on any age face, pedomorphic-
looking (neotenous) facial traits (e.g., large eyes,
pudgy lips, and round chins) transmit qualities
associated with submissiveness, including warmth,
weakness, femininity, and honesty. Mature-looking
facial characteristics (e.g., narrow eyes, thin lips,
and square jaws) elicit attributions associated with
dominance, such as strength, cunning, masculin-
ity, and leadership. Early, cross-cultural findings for
adult perceivers (Keating, Mazur, & Segall, 1981)
and for U.S. preschoolers showed that perceivers
selected male and female adult faces with broad jaws
and receded hairlines as people who “tell others
what to do” (Keating & Bai, 1986). And do they?
Are these “honest” signals (Keating, 2002)? Some
recent finding suggest that they are. Facial width-
to-height ratios predict perceptions of competence
and leadership among corporate leaders (Rule &
Ambady, 2010). Fortune 500 corporate profits can
be predicted from the facial widths of their chief
executive officers (Wong, Ormiston, & Haselhuhn,
The Life and Times of Nonverbal Communication
Behavioral ecology view
(signification of intent and verbal equivalent)
Readiness to affiliate or play (“Good to see you,” “Let’s play [keep
playing],” or “Let’s be friends”)
Readiness to appease (“Whatever you say,” or “I give in”)
Recruitment of succor (“Take care of me,” “Hold me,” or “Look at
what you [he/she] did to me”)
Readiness to attack (“Back off or I’ll attack”)
Conflict about attacking (“I want to attack and I don’t want to
attack” or “I’m close to attacking you”)
Readiness to submit or escape (“Don’t hurt me!” or “Take what
you want!”)
Readiness to continue current interaction (“Everything [you’re
doing] is just fine”)
Declaration of superiority (“I can’t even bother with you,” or
“You’re not worth the trouble”)
Declaration of neutrality (“I’m taking no position [on what you’re
doing/saying]”)
2011). Intriguing, correlational studies such as
these have fueled a recent growth spurt in research
on human, physiognomic signaling systems that is
compatible with ecological approaches.
The status cues perspective on human nonverbal
communication also applies to dynamic signaling
systems (Keating, 2011; Keating, Mazur, Segall,
et al., 1981) and helps make sense of what are called
the self-conscious emotions (Matsumoto et al., 2013)
or the social and moral emotions (Buck & Renfro
Powers, 2006). Typically included among them
are pride, shame, and embarrassment, and they
appear to be universal displays (Keltner, 1995;
Tracy & Robins, 2008; Tracy, Shariff, Zhao, &
Henrich, 2013). Each is characterized by a distinct,
choreographed sequence of nonverbal expressions
and postures. They have no simple link to basic
emotions, but their elements are laced with status
cues—specifically, slight smiles (Keating, Mazur,
Segall, et al., 1981), shrugs, head cants (Costa,
Menzani, & Bitti, 2001), and blushing (Gerlach,
Wilhelm, Gruber, & Roth, 2001). Consequently,
embarrassment and shame convey appeasement
and invite approach after transgression; the goal
29
Caroline F. Keating
seems to be reentry into the group (Keltner, 1995;
Martens, Tracy, & Shariff, 2012). Pride contains ele-
ments of dominance such as erect posture, which
slows approach or reverses its direction; it appears
to serve power motives (Keltner, 1995; Martens
et al., 2012; Tracy et al., 2013). The self-conscious
emotions are perhaps better thought of as expressive
of status cues in the service of social motives rather
than emotions per se.
Lens Model Approaches
Nonverbal communication ultimately enhances
reproductive fitness but serves proximate goals (e.g.,
for social approval or resource acquisition) in its
more immediate wake. On a day-to-day basis, the
art of nonverbal communication requires anticipa-
tion of the fit between signal and social context
or ecology. Early functionalists such as Brunswik
(1955) applied these ideas to perception and social
perception (e.g., to face cues, nations). Brunswik
argued that perceptual systems were adapted to their
environments by expectancies established through
experience in a particular environment or ecology.
Probablistic functionalism meant that perception
operated probabilistically; cue perception was biased
toward interpretations that had worked before (e.g.,
Segall, Campbell, & Herskovitz, 1966). Because
probabilistic judgments proved correct most of the
time in particular ecologies, these perceptual hab-
its were thought of as adaptive (Brunswik, 1955).
Perceivers need only beat the odds some percentage
of the time to come away with a winning, percep-
tual hand. Thus, a certain degree of error was to be
expected in social perception; some degree of slip-
page was anticipated in the match-up between cue
validity and cue utilization.
The slippage showed when Gifford (1994)
applied Brunswik’s lens model to the nonverbal com-
munication habits of people with different personal-
ity types. Gifford first uncovered valid cues, finding
that communicators who scored high on personal-
ity measurements of warmth and agreeableness
frequently nodded their heads during interactions.
However, when he next had naive observers judge
communicators’ traits from their nonverbal behav-
ior, he discovered that the valid cue of head nods
was not the only cue they used. Observers actually
30
deployed several postural cues that were unassoci-
ated with the dispositions of communicators.
It was as if observers hedged their bets when
forming impressions based on nonverbal cues.
The appeal of Brunswik’s model has been wide-
spread in recent decades. Theorists have applied
lens models to several nonverbal communication
problems, including vocal quality signals (Scherer,
1978), rapport (Bernieri, Gillis, Davis, & Grahe,
1996), mimicry (Hess, Philippot, & Blairy, 1999),
and deception (Hartwig & Bond, 2011). Because the
approach has the potential to distinguish between
what is expressed and what is detected, it has also
been applied to emotional expression.
Scherer and his colleagues applied a modified
version of Brunswik’s lens model to their appraisal
theory of emotion (Scherer et al., 2011; Scherer,
Johnstone, & Klasmeyer, 2003). The cornerstone
of their argument separates expression from its mea-
surement, perception: Using perceptions of emotion
as a ruler to measure expression of emotion, they
argued, conflates the two. The modified lens model
allows the signaler’s expressive behavior to be con-
sidered separately from the beholder’s perception/
inference. Unlinking them enables closer inspection
of the “appraisal” processes implicated in the encod-
ing and decoding of emotion.
Based on these processing distinctions, Scherer
and colleagues demonstrated that expressors
(encoders) do not display, and perceivers (decoders)
do not utilize, facial action units at the level of preci-
sion or reliability that theorists such as Ekman pre-
dict (Scherer et al., 2011). Nevertheless, recognition
accuracy for emotions remained high. Consistent
with appraisal theories of emotion, interpretation
and inference seemed embedded in message pro-
cessing, and the processes themselves seemed cumu-
lative and dynamic. Appraisals are likely affected
by development and culture, and future research
framed by this paradigm seems directed toward
testing their effects.
The Ecological Theory of Social
Perception: Affordances
That ecology shapes social perception is also con-
sistent with the ecological theory of social perception
developed in the 1980s (McArthur & Baron, 1983).

Over several decades, the idea that direct perception
of affordances (Gibson, 1977) could be extracted
from static and dynamic nonverbal elements has
been championed by Leslie A. Zebrowitz, Joanne
Montepare, and their colleagues in an extensively
developed series of research projects (e.g., Berry &
McArthur, 1986; Liang, Zebrowitz, & Zhang, 2010;
Montepare & Zebrowitz, 1998; Zebrowitz, 1997,
2003; Zebrowitz & Collins, 1997; Zebrowitz &
Montepare, 2008).
Consistent with the ecological theory of percep-
tion, these researchers have argued that facial struc-
ture conveys affordances, defined as opportunities
for certain types of interactions. Sensitivity or attun-
ement to these signals, although adaptive, can be
overgeneralized. In particular, affordances proffered
by “age-related physical qualities” (Montepare &
Zebrowitz, 1998, p. 95), such as babyish facial
cues, may be overgeneralized when displayed by
adults and may influence impressions of social traits
(Zebrowitz, Luevano, Bronstad, & Aharon, 2009).
Hence, baby-faced adults are perceived as having
characteristics associated with babies: They are
judged to be dependent, weak, and kind, whereas
mature facial appearances make individuals look
dominant, strong, and cunning (Berry & McArthur,
1986; Montepare & Zebrowitz, 1998; Zebrowitz,
1997). So potent are these perceptual habits that
they can become self-fulfilling prophecies, as tar-
gets conform to social expectations created by their
physical appearances (Zebrowitz & Montepare,
2008; see also Chapter 10, this handbook).
The affordances perspective has a fine record
of research with a long reach. Researchers from
this perspective have continued to unearth evi-
dence to support their views by launching devel-
opmental, cross-species, and cross-cultural studies
(e.g., Zebrowitz et al., 2011, 2012). They have
incorporated connectionist modeling techniques
(Zebrowitz, Fellous, Mignault, & Andreoletti, 2003;
Zebrowitz et al., 2009) and have used dynamic as
well as static stimuli. They have reached across from
their home domain of social cognition and percep-
tion to emotion, lending support to the thesis of
Marsh, Adams, and Kleck (2005) that expressions of
emotion were designed to mimic age-related aspects
of facial structure (Zebrowitz, Kikuchi, & Fellous,
The Life and Times of Nonverbal Communication
2007). They have also suggested that appearances
of facial dominance may result from the overgen-
eralization of facial likenesses to anger expressions
(Zebrowitz, Kikuchi, & Fellous, 2010). Recently,
the affordances perspective has been a useful tool in
the development of a model of rapport, conceptual-
ized as an optimal, harmonious flow revealed during
interaction and very much dependent on nonverbal
behavior (Tickle-Degnen, 2006).
Social-Cognitive Perspectives on
Nonverbal Communication
Ecological perspectives on nonverbal communica-
tion include functional, social-cognitive approaches
that identify proximate social goals or motives as
the energizers of nonverbal communication. Social
motives are construed as powerful drive states that
set in motion behavior, emotion, and cognitions
aimed at satisfying basic social needs (Fiske, 2010).
These fundamental human needs or goals generally
include belonging, controlling, and enhancing self,
depending on whom you read (Baumeister, 1993;
Bugental, 2000; Fiske, 2010). The development of
these needs and the behavioral strategies used to sat-
isfy them are ultimately linked to biological fitness
(Barkow, Cosmides, & Tooby, 1992; Buss, 1999).
Rosenfeld (1965, 1966) conducted early, elegant,
experimental demonstrations coupling nonverbal
communication with social goals. In a series of stud-
ies, he simply instructed participants to interact with
unfamiliar others (sometimes a confederate), and he
gave them different social goals to be enacted non-
verbally. Sometimes the social goal required seeking
approval (“get the other person to like you”), and
sometimes it did not (“let the other person know
you do not want to be friends with them”). Partici-
pants were chameleon-like; their gestures, postures,
expressions, and use of interpersonal space were
transformed by the explicit, social goal operating at
the time. They “knew” precisely how to nonverbally
present themselves to successfully negotiate the
differing social terrains. Developmental research
suggests that nonverbal social competencies such
as these are learned early in life (Saarni, 1989;
Saarni & Weber, 1999). Throughout life, nonverbal
communication is essential to self-presentational
goals, whether heartfelt, deceptive (DePaulo, 1992;
31
Caroline F. Keating
Feldman, 2009; Keating, 2006), or self-deceptive
(von Hippel & Trivers, 2011).
Patterson (1999, 2006) applied some of this
thinking to his parallel process model of nonverbal
communication. This model attempts to expose the
ways controlled and automatic processes operate in
tandem in the production of nonverbal communica-
tion. According to Patterson, goal-related processes
at or below the level of consciousness routinely trig-
ger the sending and receiving nonverbal messages.
Nonverbal responses and judgments made from
other’s nonverbal behavior occur in parallel, effi-
ciently “managed” by automatic processes during
interaction. Patterson’s model suggests that cogni-
tive and affective filters operate differently for indi-
viduals high and low in traits such as social anxiety
or self-monitoring. It also suggests when and how
cognitive resources would be redirected, for
example, under different status conditions or for
different age groups or in different cultures. To date,
components of the parallel processing model have
heuristic value, but specific hypotheses await
formulation and testing.
SUMMARY AND FUTURE DIRECTIONS
Two traditions of theory and research were identi-
fied in this chapter, one emphasizing emotions
inherent in expressions, and the other emphasizing
communication processes. Each spawned valuable
contributions to understanding human nonverbal
communication. The emphasis on emotion has
achieved relatively precise neuroanatomical mea-
surement and classification of expressions as well
as an understanding of their evolution, distinctive
qualities, and underlying biological substrates. The
emphasis on communication reveals a cross-species
logic to signaling systems that is not frozen in time.
It suggests how interactive, nonverbal processes can
lead to flawed communication.
Though each tradition has added value to the
enterprise of nonverbal communication science,
they can both be faulted for proceeding in ways
that do not make likely credible tests of alterna-
tive possibilities outside their theoretical domain.
Decades ago, William McGuire warned that focus
on testing particular hypotheses from a singular
32
approach may leave worthy alternative models in
the dark (Jost, 2013).
Exceptions to singular approaches have value.
Andersen et al. (1998) tested the explanatory power
of three different theoretical approaches to imme-
diacy behavior. Research on whether different kinds
of laughter convey different emotions was tested
under two emotion models, one that classified
laughter into basic categories and the other
that used dimensions. Each showed some success,
but one showed more success than the other
(Szameitat et al., 2009). Sacco and Hugenberg
(2009) blended basic emotions with morphologi-
cal cues on the pallet of the face, revealing impor-
tant interactive effects of emotion and status cues.
Empirical approaches such as these have the poten-
tial to reveal the relative strengths and weaknesses
of theories and models.
More specific lessons derived from the theories
and models reviewed here include the following:
■■ From interactionists comes the perspective that
nonverbal communication should be studied
in its natural habitat—social interaction—and
that clusters of signals and responses should
be mapped across space and time as a dynamic
flow. Increasingly sophisticated technologies
(e.g., high-speed, high-definition digital record-
ing; computer software that alters and/or records
motion stimuli; high-speed cameras with built-in
motion detection; virtual reality headsets; even
Google glasses) make this feasible.
■■ Social-cognitive approaches offer that conscious
and nonconscious behaviors and goals shape
nonverbal processes during interaction. Evidence
of nonconscious processes that can be temporally
linked to behavior is possible through recent
advances in measuring physiological and brain
activity in moving subjects.
■■ Lens perspectives as well as the Gibsonian affor-
dances position contribute an understanding
of the mechanisms by which nonverbal com-
munication systems “short” us, presumably in
predictable ways as determined by ecologies and
environments. Experimentally induced virtual
environments could be used to test competing
propositions generated by these models.
 The Life and Times of Nonverbal Communication

■■ Both the basic emotions and Gibsonian affor- Basic emotion theorists would be tempted to
dances theoretical positions confirm that a good challenge the very premise of all that error with piles
argument is a thorough one. Each has harnessed of data showing how good people are at identifying
the power of cross-cultural, cross-species, basic emotions. They would be partly right: Basic,
developmental, and multidisciplinary approaches prototypical emotion facial expressions generate
to make their case. impressive agreement and consistency in how they
are perceived across cultures. However, express-
Nonverbal communication is no longer the
ers around the world rarely display them in actual,
“disdained, nameless, orphan child of psychology”
social situations.
(Burgoon, Buller, & Woodall, 1996, p. xviii) that
From a behavioral ecology perspective,
Burgoon described it to be in 1980. Today, it belongs
constraints on precision in messaging would be
to the family of important, human psychological
expected (i.e., Fridlund & Russell, 2006; Keating,
processes. However, if nonverbal communication is
2002): Why would individuals put all their expres-
so important to our species, why are imperfections
sive cards on the table and tell others the full truth
built into its systems? Why do we make the mis-
of how they feel, what they intend, or what capaci-
takes we do when judging the feelings, personalities,
ties they truly have? Is precision an advantage (see
intentions, lies, and truths of conspecifics? Only
M. L. Smith, Cottrell, Gosselin, & Schyns, 2005)?
brief samples of nonverbal behavior, thin slices last-
Why not appear sort of angry or kind of friendly,
ing just seconds, are needed to create impressions of
like those birds and flirtatious women described
one another (Ambady, Bernieri, & Richeson, 2000).
earlier? Graded signals such as these are prevalent in
Once created, perceivers seem generally bent on
the communication patterns of nonhuman species
confirming what they already believe—even if it is
(not to mention in human politics). Expressions
wrong. For example, thin slices lead to consensus
signal intentions (Adams, Ambady, Macrae, &
about the likely trustworthiness and altruism of
Kleck, 2006), and nuanced, nonverbal messages
others (Fetchenhauer, Groothuis, & Pradel, 2010;
that leave open multiple possibilities could be an
Todorov, Pakrashi, & Oosterhof, 2009; van’t Wout &
advantage in negotiating the social world—disguise
Sanfey, 2008; Willis & Todorov, 2006); the trouble is
and conquer, at least to a point.
that the impressions themselves are not always to be
Moreover, inaccuracy in expressions is strategic:
trusted (Rule, Krendl, Ivcevic, & Ambady, 2013).
Being a “hard read”—especially to “outsiders”—
Future research should investigate error patterns
might lend an evolutionary advantage, both proxi-
and imprecision in nonverbal communication to
mately and ultimately. This view predicts ingroup
better understand its complex role in human socia-
advantages to encoding and decoding nonverbal
bility. From the Gibsonian affordances platform,
signals, as is evident in humans (e.g., Elfenbein &
error is the result of overgeneralization; perceivers
Ambady, 2002) and as is seen in the special, non-
perseverate in making inferences based on attune-
verbal “dialects” that develop among family groups
ments appropriate in other contexts (age, anger).
in other species (Briefer & McElligott, 2012)—an
Lens model theorists argue that the sloppy, probabi-
animal version of code-switching, perhaps.2 Thus,
listic shaping of encoding and decoding systems is
discrepancies and error produced by what appears to
to blame for mistakes. Critical tests between these
be imprecision in nonverbal communication may be
theoretical points of view have yet to distinguish
part of the design and worthy of future study.
their relative, explanatory merits. Though both offer
ideas about how the social–perceptual mechanisms
THE IMPORTANCE OF QUESTIONS
of nonverbal expressive systems might go off-kilter,
they do not address why. What is in it for signalers My early plan as a graduate student in the 1970s
and perceivers, and do they ever wise up? was to begin the study of nonverbal communication
Interestingly, cross-species communication might be immune to such defenses, as the dog’s ability to read human emotion seems to suggest
2

(Andics, Gácsi, Faragó, Kis, & Miklósi, 2014).

33
Caroline F. Keating
by photographing a range of emotions on the vis-
ages of everyday people during their everyday lives.
I departed for the field with a borrowed 35-mm
camera and telephoto lens to capture what everyday
life dished out in terms of emotion. Funny thing
was, I could not find any. My lens captured only
one instance of true emotion—the stricken face of
a man standing outside his burning house, his dog
still inside. The rest of my images consisted of exag-
gerated expressions, diminished expressions, neu-
tralized expressions, or expressions that no human
could read. Once in a while, I would see flashes of
the emotional expressions depicted in research man-
uals on the faces of the psychiatric patients at the
hospital where I volunteered. I learned the following
lessons: Studying public expression would teach me
little about emotion but much about communica-
tion; observing raw emotion would tell more about
emotion than communication. Which approach is
best depends on what the question is, as philoso-
phers have noted (Dennett, 1978; Toulmin, 1972).
References
Adams, R. B., Jr., Ambady, N., Macrae, C. N., & Kleck,
R. E. (2006). Emotional expressions forecast
approach–avoidance behavior. Motivation and
Emotion, 30, 177–186. http://dx.doi.org/10.1007/
s11031-006-9020-2
Alexander, R. D. (1974). The evolution of social
behavior. Annual Review of Ecology and Systematics,
5, 325–383. http://dx.doi.org/10.1146/annurev.
es.05.110174.001545
Ambady, N., Bernieri, F. J., & Richeson, J. A. (2000).
Toward a histology of social behavior: Judgmental
accuracy from thin slices of the behavioral stream.
In M. P. Zanna (Ed.), Advances in experimental social
psychology (Vol. 32, pp. 201–271). San Diego, CA:
Academic Press.
Andersen, P. A., Guerrero, L. K., Buller, D. B., &
Jorgensen, P. F. (1998). An empirical comparison
of three theories of nonverbal immediacy exchange.
Human Communication Research, 24, 501–535. http://
dx.doi.org/10.1111/j.1468-2958.1998.tb00429.x
Andics, A., Gácsi, M., Faragó, T., Kis, A., & Miklósi,
A. (2014). Voice-sensitive regions in the dog
and human brain are revealed by comparative
fMRI. Current Biology, 24, 574–578. http://dx.doi.
org/10.1016/j.cub.2014.01.058
Andrew, R. J. (1963, November 22). Evolution of facial
expression. Science, 142, 1034–1041. http://dx.doi.
org/10.1126/science.142.3595.1034
34
Archer, D., & Akert, R. M. (1977). How well do you read
body language? Psychology Today, 11(5), 68–69, 72,
119–120.
Argyle, M., & Dean, J. (1965). Eye contact, distance, and
affiliation. Sociometry, 28, 289–304. http://dx.doi.
org/10.2307/2786027
Arnold, M. B. (1960). Emotion and personality. New York,
NY: Columbia University Press.
Aviezer, H., Trope, Y., & Todorov, A. (2012, November 30).
Body cues, not facial expressions, discriminate between
intense positive and negative emotions. Science, 338,
1225–1229. http://dx.doi.org/10.1126/science.1224313
Bar, M., Neta, M., & Linz, H. (2006). Very first
impressions. Emotion, 6, 269–278. http://dx.doi.
org/10.1037/1528-3542.6.2.269
Barkow, J., Cosmides, L., & Tooby, J. (1992). The adapted
mind: Evolutionary psychology and the generation of
culture. New York, NY: Oxford University Press.
Batty, M., & Taylor, M. J. (2003). Early processing
of the six basic facial emotional expressions.
Cognitive Brain Research, 17, 613–620. http://dx.doi.
org/10.1016/S0926-6410(03)00174-5
Baumeister, R. (1993). Self-presentation: Motivational,
cognitive, and interpersonal patterns. In G. van
Heck, P. Bonaiuto, I. J. Deary, & W. Nowack
(Eds.), Personality psychology in Europe (Vol. 4,
pp. 257–280). Tilburg, the Netherlands: Tilburg
University Press.
Bavelas, J. B., & Chovil, N. (2006). Nonverbal and verbal
communication: Hand gestures and facial displays
as part of language use in face-to-face dialogue.
In V. Manusov & M. L. Patterson (Eds.), The Sage
handbook of nonverbal communication (pp. 97–116).
http://dx.doi.org/10.4135/9781412976152.n6
Bernieri, F. J., Gillis, J. S., Davis, J. M., & Grahe, J. E.
(1996). Dyad rapport and the accuracy of its judgment
across situations: A lens model analysis. Journal of
Personality and Social Psychology, 71, 110–129.
http://dx.doi.org/10.1037/0022-3514.71.1.110
Berry, D. S., & McArthur, L. Z. (1986). Perceiving
character in faces: The impact of age-related
craniofacial changes on social perception.
Psychological Bulletin, 100, 3–18. http://dx.doi.
org/10.1037/0033-2909.100.1.3
Birdwhistell, R. L. (1970). Kinesics and context: Essays on
body motion communication. Philadelphia: University
of Pennsylvania Press.
Blurton Jones, N. G. (Ed.). (1972). Ethological studies
of child behavior. Cambridge, England: Cambridge
University Press.
Briefer, E. F., & McElligott, A. G. (2012). Social
effects on vocal ontogeny in an ungulate, the goat,
Capra hircus. Animal Behaviour, 83, 991–1000.
http://dx.doi.org/10.1016/j.anbehav.2012.01.020

Brunswik, E. (1955). Representative design and
probabilistic theory in a functional psychology.
Psychological Review, 62, 193–217. http://dx.doi.
org/10.1037/h0047470
Buck, R. (1984). The communication of emotion. New
York, NY: Guilford Press.
Buck, R., & Renfro Powers, S. (2006). The biological
foundations of social organization: The dynamic
emergence of social structure through nonverbal
communication. In V. Manusov & M. L. Patterson
(Eds.), The Sage handbook of nonverbal communication
(pp. 119–138). http://dx.doi.org/10.4135/
9781412976152.n7
Bugental, D. B. (2000). Acquisition of the algorithms
of social life: A domain-based approach.
Psychological Bulletin, 126, 187–219. http://dx.doi.
org/10.1037/0033-2909.126.2.187
Burgoon, J. K. (1978). Attributes of the newscaster’s
voice as predictors of his credibility. Journalism
Quarterly, 55, 276–300. http://dx.doi.org/10.1177/
107769907805500208
Burgoon, J. K., Buller, D. B., & Woodall, W. G. (1996).
Nonverbal communication (2nd ed.). New York, NY:
McGraw-Hill.
Burgoon, J. K., & Jones, S. B. (1976). Toward a theory
of personal space expectations and their violations.
Human Communication Research, 2, 131–146. http://
dx.doi.org/10.1111/j.1468-2958.1976.tb00706.x
Burgoon, J. K., & Saine, T. J. (1978). The unspoken
dialogue: An introduction to nonverbal communication.
Boston, MA: Houghton Mifflin.
Burgoon, J. K., Stern, L. A., & Dillman, L. (1995).
Interpersonal adaptation: Dyadic interaction patterns.
http://dx.doi.org/10.1017/CBO9780511720314
Buss, D. M. (1999). Evolutionary psychology: The new
science of the mind. Needham Heights, MA: Allyn &
Bacon.
Camras, L. (1992). Expression development and basic
emotions. Cognition and Emotion, 6, 269–283.
http://dx.doi.org/10.1080/02699939208411072
Cappella, J. N., & Green, J. O. (1984). The effects of
distance and individual differences in arousability on
nonverbal involvement: A test of discrepancy-arousal
theory. Journal of Nonverbal Behavior, 8, 259–286.
http://dx.doi.org/10.1007/BF00985983
Carroll, J. M., & Russell, J. A. (1996). Do facial
expressions signal specific emotions? Judging
emotion from the face in context. Journal of
Personality and Social Psychology, 70, 205–218.
http://dx.doi.org/10.1037/0022-3514.70.2.205
Carver, C. S., & Harmon-Jones, E. (2009). Anger is an
approach-related affect: Evidence and implications.
Psychological Bulletin, 135, 183–204. http://dx.doi.
org/10.1037/a0013965
The Life and Times of Nonverbal Communication
Caryl, P. G. (1979). Communication by agonistic
displays: What can game theory contribute to
ethology? Behaviour, 68, 136–169. http://dx.doi.
org/10.1163/156853979X00287
Chevalier-Skolnikoff, S. (1973). Facial expression of
emotion in nonhuman primates. In P. Ekman (Ed.),
Darwin and facial expressions (pp. 11–89). New York,
NY: Academic Press.
Cole, J. K. (Ed.). (1972). Nebraska symposium on
motivation. Lincoln: University of Nebraska Press.
Costa, M., Menzani, M., & Bitti, P. E. R. (2001).
Head canting in paintings: An historical study.
Journal of Nonverbal Behavior, 25, 63–73.
http://dx.doi.org/10.1023/A:1006737224617
Dalgleish, T. (2004). The emotional brain. Nature
Reviews Neuroscience, 5, 583–589. http://dx.doi.
org/10.1038/nrn1432
Darwin, C. (1956). Biographical sketch of an infant. In
M. Bates & P. S. Humphrey (Eds.), The Darwin
reader (pp. 403–411). New York, NY: Scribner.
(Original work published 1877)
Darwin, C. (1965). The expression of the emotions in man
and animals. Chicago, IL: University of Chicago
Press. (Original work published 1872)
Davitz, J. R. (1964). The communication of emotional
meaning. New York, NY: McGraw-Hill.
Dawkins, R. (1976). The selfish gene. New York, NY:
Oxford University Press.
Dawkins, R., & Krebs, J. R. (1978). Animal
signals: Information or manipulation? In J. R.
Krebs & N. B. Davies (Eds.), Behavioural
ecology: An evolutionary approach (pp. 282–312).
Sunderland, MA: Sinauer.
de Gelder, B. (2006). Towards the neurobiology of
emotional body language. Nature Perspectives, 7,
242–249. http://dx.doi.org/10.1038/nrn1872
Dennett, D. C. (1978). Brainstorms: Philosophical essays
on mind and psychology. Cambridge, MA: MIT Press.
DePaulo, B. M. (1992). Nonverbal behavior and self-
presentation. Psychological Bulletin, 111, 203–243.
http://dx.doi.org/10.1037/0033-2909.111.2.203
De Waal, F. B. M. (2003). Darwin’s legacy and the study
of primate visual communication. In P. Ekman, J.
Campos, R. J. Davidson, & F. B. M. de Waal (Eds.),
Emotions inside out: 130 years after Darwin’s The
Expression of the Emotions in Man and Animals
(pp. 7–31). New York, NY: New York Academy of
Sciences.
Dittmann, A. T. (1972). The body movement–speech
rhythm relationship as a cue to speech encoding.
In A. W. Siegman & B. Pope (Eds.), Studies in
dyadic communication (pp. 131–151). http://dx.doi.
org/10.1016/B978-0-08-015867-9.50011-3
35
Caroline F. Keating
Dittmann, A. T., & Llewellyn, L. G. (1967). The
phonemic clause as a unit of speech decoding.
Journal of Personality and Social Psychology, 6,
341–349. http://dx.doi.org/10.1037/h0024739
Dittmann, A. T., & Llewellyn, L. G. (1969). Body
movement and speech rhythm in social conversation.
Journal of Personality and Social Psychology, 11,
98–106. http://dx.doi.org/10.1037/h0027035
Dunbar, R. I. M., & Shultz, S. (2007, September
7). Evolution in the social brain. Science,
317, 1344–1347. http://dx.doi.org/10.1126/
science.1145463
Duncan, S., Jr. (1969). Nonverbal communication.
Psychological Bulletin, 72, 118–137. http://dx.doi.
org/10.1037/h0027795
Duncan, S., Jr. (1972). Some signals and rules for
taking speaking turns in conversations. Journal
of Personality and Social Psychology, 23, 283–292.
http://dx.doi.org/10.1037/h0033031
Efron, D. (1972). Gesture, race, and culture. The Hague, the
Netherlands: Mouton. (Original work published 1941)
Eibl-Eibesfeldt, I. (1972). Similarities and differences
between cultures in expressive movements. In R. A.
Hinde (Ed.), Nonverbal communication (pp. 297–312).
New York, NY: Cambridge University Press.
Eibl-Eibesfeldt, I. (1975). The biology of behavior (2nd
ed.). New York, NY: Holt, Rinehart, & Winston.
Eisenberg, A. M., & Smith, R. R. (1971). Nonverbal
communication. New York, NY: Bobbs-Merrill.
Ekman, P. (1972). Universal and cultural differences in
facial expressions of emotion. In J. K. Cole (Ed.),
Nebraska symposium on motivation (pp. 207–283).
Lincoln: University of Nebraska Press.
Ekman, P. (Ed.). (2003). Emotions inside out: 130 years
after Darwin’s The Expression of the Emotions
in Man and Animals. New York, NY: New York
Academy of Sciences.
Ekman, P. (2009). Lie catching and microexpressions.
In C. Martin (Ed.), The philosophy of deception (pp.
118–133). New York, NY: Oxford University Press.
Ekman, P., & Friesen, W. V. (1969). The repertoire of
nonverbal behavior: Categories, origins, usage, and
coding. Semiotica, 1, 49–98.
Ekman, P., & Friesen, W. V. (1971). Constants
across cultures in the face and emotion. Journal
of Personality and Social Psychology, 17, 124–129.
http://dx.doi.org/10.1037/h0030377
Ekman, P., & Friesen, W. V. (1978). Manual for the facial
action coding system. Palo Alto, CA: Consulting
Psychologists Press.
Ekman, P., & Friesen, W. V. (1986). A new pan-cultural
facial expression of emotion. Motivation and Emotion,
10, 159–168. http://dx.doi.org/10.1007/BF00992253
36
Ekman, P., Friesen, W. V., & Simons, R. C. (1985).
Is the startle reaction an emotion? Journal of
Personality and Social Psychology, 49, 1416–1426.
http://dx.doi.org/10.1037/0022-3514.49.5.1416
Ekman, P., Levenson, R. W., & Friesen, W. V.
(1983, September 16). Autonomic nervous system
activity distinguishes among emotions. Science,
221, 1208–1210. http://dx.doi.org/10.1126/
science.6612338
Ekman, P., & Rosenberg, E. L. (Eds.). (2005). What
the face reveals (2nd ed.). New York, NY: Oxford
University Press.
Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969,
April 4). Pan-cultural elements in facial displays
of emotion. Science, 164, 86–88. http://dx.doi.
org/10.1126/science.164.3875.86
Elfenbein, H. A., & Ambady, N. (2002). On the
universality and cultural specificity of emotion
recognition: A meta-analysis. Psychological Bulletin,
128, 203–235. http://dx.doi.org/10.1037/
0033-2909.128.2.203
Etcoff, N. L., Stock, S., Haley, L. E., Vickery, S. A., &
House, D. M. (2011). Cosmetics as a feature of the
extended human phenotype: Modulation of the
perception of biologically important facial signals.
PLoS ONE, 6, e25656. http://dx.doi.org/10.1371/
journal.pone.0025656
Exline, R. V. (1972). Visual interaction: The glances of
power and preference. In J. K. Cole (Ed.), Nebraska
symposium on motivation (pp. 163–206). Lincoln:
University of Nebraska Press.
Fast, J. (1970). Body language. New York, NY:
M. Evans & Company.
Feldman, R. (2007). On the origins of background
emotions: From affect synchrony to symbolic
expression. Emotion, 7, 601–611. http://dx.doi.org/
10.1037/1528-3542.7.3.601
Feldman, R. (2009). The liar in your life: The way to
truthful relationships. New York, NY: Twelve.
Fernández-Dols, J.-M., Carrera, P., & Crivelli, C.
(2011). Facial behavior while experiencing
sexual excitement. Journal of Nonverbal Behavior,
35, 63–71. http://dx.doi.org/10.1007/s10919-
010-0097-7
Fernández-Dols, J.-M., & Ruiz-Belda, M.-A. (1995).
Are smiles a sign of happiness? Gold medal winners
at the Olympic Games. Journal of Personality and
Social Psychology, 69, 1113–1119. http://dx.doi.
org/10.1037/0022-3514.69.6.1113
Fetchenhauer, D., Groothuis, T., & Pradel, J. (2010).
Not only states but traits—Humans can identify
permanent altruistic dispositions in 20 s. Evolution
and Human Behavior, 31, 80–86. http://dx.doi.
org/10.1016/j.evolhumbehav.2009.06.009

Fiske, S. T. (2010). Social beings. New York, NY: Wiley.
Frank, M., & Svetieva, E. (2013). Deception.
In D. Matsumoto, M. Frank, & H. S. Hwang
(Eds.), Nonverbal communication: Science and
applications (pp. 121–144). http://dx.doi.
org/10.4135/9781452244037.n6
Fridlund, A. J. (1994). Human facial expression: An
evolutionary view. New York, NY: Academic Press.
Fridlund, A. J., & Russell, J. A. (2006). The functions
of facial expressions: What’s in a face? In V.
Manusov & M. L. Patterson (Eds.), The Sage
handbook of nonverbal communication (pp.
299–320). http://dx.doi.org/10.4135/
9781412976152.n16
Gerlach, A. L., Wilhelm, F. H., Gruber, K., & Roth, W.
T. (2001). Blushing and physiological arousability
in social phobia. Journal of Abnormal Psychology,
110, 247–258. http://dx.doi.org/10.1037/0021-
843X.110.2.247
Gibson, J. J. (1977). The theory of affordances.
In R. Shaw & J. Bransford (Eds.), Perceiving,
acting, and knowing: Toward an ecological psychology
(pp. 67–82). Hillsdale, NJ: Erlbaum.
Gifford, R. (1994). A lens-mapping framework for
understanding the encoding and decoding of
interpersonal dispositions in nonverbal behavior.
Journal of Personality and Social Psychology,
66, 398–412. http://dx.doi.org/10.1037/0022-
3514.66.2.398
Goffman, E. (1959). The presentation of self in everyday
life. Garden City, NY: Doubleday.
Goffman, E. (1967). Interaction ritual: Essays on face-
to-face behavior. Chicago, IL: Aldine.
Goldin-Meadow, S. (2005). Hearing gesture: How our
hands help us think. Cambridge, MA: Harvard
University Press.
Goleman, D., Boyatzis, R., & McKee, A. (2002).
Primal leadership: Realizing the power of emotional
intelligence. Cambridge, MA: Harvard Business
School Press.
Goodall, J. (1971). In the shadow of man. Boston, MA:
Houghton Mifflin.
Gould, S. J. (1973). Ever since Darwin. New York, NY:
Norton.
Grammer, K., Kruck, K. B., & Magnusson, M. S.
(1998). The courtship dance: Patterns of nonverbal
synchronization in opposite-sex encounters. Journal
of Nonverbal Behavior, 22, 3–29. http://dx.doi.
org/10.1023/A:1022986608835
Guthrie, R. D. (1976). Body hot spots. New York, NY: Van
Nostrand Reinhold.
Hall, E. T. (1959). The silent language. Greenwich, CN:
Fawcett.
The Life and Times of Nonverbal Communication
Hall, E. T. (1966). The hidden dimension. New York, NY:
Doubleday.
Harris, C. R., & Alvarado, N. (2005). Facial expressions,
smile types, and self-report during humour, tickle,
and pain. Cognition and Emotion, 19, 655–669.
http://dx.doi.org/10.1080/02699930441000472
Hartwig, M., & Bond, C. F., Jr. (2011). Why do lie-
catchers fail? A lens model meta-analysis of human
lie judgments. Psychological Bulletin, 137, 643–659.
http://dx.doi.org/10.1037/a0023589
Hatfield, E., Cacioppo, J. T., & Rapson, R. (1994).
Emotional contagion. Cambridge, England:
Cambridge University Press.
Hertenstein, M. J., Holmes, R., McCullough, M., &
Keltner, D. (2009). The communication of emotion
via touch. Emotion, 9, 566–573. http://dx.doi.
org/10.1037/a0016108
Hess, U., Philippot, P., & Blairy, S. (1999). Mimicry:
Facts and fiction. In P. Philippot, R. S. Feldman, &
E. J. Coats (Eds.), The social context of nonverbal
behavior (pp. 213–241). Cambridge, England:
Cambridge University Press.
Hewes, G. W. (1973). Primate communication and the
gestural origin of language. Current Anthropology,
14, 5–24. http://dx.doi.org/10.1086/201401
Hinde, R. A. (1959). Unitary drives. Animal Behaviour,
7, 130–141. http://dx.doi.org/10.1016/0003-
3472(59)90002-8
Hinde, R. A. (Ed.). (1972). Nonverbal communication.
Cambridge, England: Cambridge University Press.
Huber, E. (1972). Evolution of facial musculature and facial
expression. New York, NY: Arno Press. (Original
work published 1931)
Hughes, S. M., & Nicholson, S. E. (2008). Sex differences
in the assessment of pain versus sexual pleasure
facial expressions. Journal of Social, Evolutionary,
and Cultural Psychology, 2, 289–298. http://dx.doi.
org/10.1037/h0099338
Izard, C. E. (1971). The face of emotion. New York, NY:
Appleton-Century-Crofts.
Izard, C. E. (1997). Emotions and facial expressions:
A perspective from differential emotions theory.
In J. A. Russell & J. M. Fernández-Dols (Eds.),
The psychology of facial expression (pp. 57–77).
http://dx.doi.org/10.1017/CBO9780511659911.005
Izard, C. E. (2007). Basic emotions, natural kinds,
emotion schemas, and a new paradigm. Perspectives
on Psychological Science, 2, 260–280. http://dx.doi.
org/10.1111/j.1745-6916.2007.00044.x
Jost, J. T. (2013). Introduction to “An Additional
Future for Psychological Science.” Perspectives on
Psychological Science, 8, 414–423. http://dx.doi.
org/10.1177/1745691613491270
37
Caroline F. Keating
Keating, C. F. (1985a). Gender and the physiognomy
of dominance and attractiveness. Social
Psychology Quarterly, 48, 61–70. http://dx.doi.
org/10.2307/3033782
Keating, C. F. (1985b). Human dominance signals:
The primate in us. In S. L. Ellyson & J. F. Dovidio
(Eds.), Power, dominance, and nonverbal behavior
(pp. 89–108). http://dx.doi.org/10.1007/978-1-
4612-5106-4_5
Keating, C. F. (2002). Charismatic faces: Social status
cues put face appeal in context. In G. Rhodes & L. A.
Zebrowitz (Eds.), Facial attractiveness: Evolutionary,
cognitive, and social perspectives (pp. 153–192).
Westport, CT: Ablex.
Keating, C. F. (2006). How and why the silent self speaks
volumes: Functional approaches to nonverbal
impression management. In M. L. Patterson & V.
L. Manusov (Eds.), The Sage handbook of nonverbal
communication (pp. 321–340). http://dx.doi.
org/10.4135/9781412976152.n17
Keating, C. F. (2011). Channeling charisma through
face and body status cues. In D. Chadee & A. Kostic
(Eds.), Social psychological dynamics (pp. 93–111).
Mona, Jamaica: University of the West Indies Press.
Keating, C. F., & Bai, D. L. (1986). Children’s
attributions of social dominance from facial cues.
Child Development, 57, 1269–1276. http://dx.doi.
org/10.2307/1130449
Keating, C. F., Mazur, A., & Segall, M. H. (1981).
A cross-cultural exploration of physiognomic
traits of dominance and happiness. Ethology
and Sociobiology, 2, 41–48. http://dx.doi.
org/10.1016/0162-3095(81)90021-2
Keating, C. F., Mazur, A., Segall, M. H., Cysneiros, P. G.,
Kilbride, J. E., Leahy, P., . . . Wirsing, R. (1981).
Culture and the perception of social dominance
from facial expression. Journal of Personality and
Social Psychology, 40, 615–626. http://dx.doi.
org/10.1037/0022-3514.40.4.615
Kelly, S. D., Hansen, B. C., & Clark, D. T. (2012).
“Slight” of hand: The processing of visually
degraded gestures with speech. PLoS ONE, 7, e42620.
http://dx.doi.org/10.1371/journal.pone.0042620
Keltner, D. (1995). Signs of appeasement: Evidence
for the distinct displays of embarrassment,
amusement, and shame. Journal of Personality
and Social Psychology, 68, 441–454. http://dx.doi.
org/10.1037/0022-3514.68.3.441
Keltner, D., & Haidt, J. (1999). Social functions of
emotions at four levels of analysis. Cognition and
Emotion, 13, 505–521. http://dx.doi.org/10.1080/
026999399379168
Kendon, A. (1967). Some functions of gaze-direction
in social interaction. Acta Psychologica, 26, 22–63.
http://dx.doi.org/10.1016/0001-6918(67)90005-4
38
Kendon, A. (1970). Movement coordination in
social interaction: Some examples described.
Acta Psychologica, 32, 101–125. http://dx.doi.
org/10.1016/0001-6918(70)90094-6
Kendon, A. (2004). Gesture: Visible action as utterance.
Oxford, England: Cambridge University Press.
Knapp, M. L. (1972). Nonverbal communication in human
interaction. New York, NY: Holt, Rinehart, & Winston.
Knapp, M. L. (2006). An historical overview of nonverbal
research. In V. Manusov & M. L. Patterson (Eds.),
The Sage handbook of nonverbal communication
(pp. 3–20). Thousand Oaks, CA: Sage.
Knapp, M. L., Hall, J. A., & Horgan, T. G. (2014).
Nonverbal communication in human interaction (8th
ed.). Boston, MA: Wadsworth Cengage Learning.
Konner, M. J. (1972). Aspects of the developmental
ethology of a foraging people. In N. B. Jones (Ed.),
Ethological studies of child behaviour (pp. 285–304).
Oxford, England: Cambridge University Press.
Lazarus, R. S., Averill, J. R., & Opton, E. M. (1970).
Towards a cognitive theory of emotion. In M. B. Arnold
(Ed.), Feelings and emotions (pp. 207–232). http://
dx.doi.org/10.1016/B978-0-12-063550-4.50023-1
LeDoux, J. (1996). The emotional brain. New York, NY:
Simon & Schuster.
Leppänen, J. M., & Nelson, C. A. (2012). Early
development of fear processing. Current Directions
in Psychological Science, 21, 200–204. http://dx.doi.
org/10.1177/0963721411435841
Levenson, R. W. (1992). Autonomic nervous system
differences among emotions. Psychological Science, 3,
23–27. http://dx.doi.org/10.1111/j.1467-9280.1992.
tb00251.x
Liang, X., Zebrowitz, L. A., & Zhang, Y. (2010).
Neural activation in the “reward circuit” shows a
nonlinear response to facial attractiveness. Social
Neuroscience, 5, 320–334. http://dx.doi.org/10.1080/
17470911003619916
Lishner, D. A., Cooter, A. B., & Zald, D. H. (2008). Rapid
emotional contagion and expressive congruence
under strong test conditions. Journal of Nonverbal
Behavior, 32, 225–239. http://dx.doi.org/10.1007/
s10919-008-0053-y
Manstead, A. S. R., Fischer, A. H., & Jakobs, E. B.
(1999). The social and emotional functions of facial
displays. In P. Philippot, R. S. Feldman, & E. J.
Coats (Eds.), The social context of nonverbal behavior
(pp. 287–316). Cambridge, England: Cambridge
University Press.
Marsh, A. A., Adams, R. B., Jr., & Kleck, R. E. (2005).
Why do fear and anger look the way they do? Form
and social function in facial expressions. Personality
and Social Psychology Bulletin, 31, 73–86. http://
dx.doi.org/10.1177/0146167204271306

Martens, J. P., Tracy, J. L., & Shariff, A. F. (2012). Status
signals: Adaptive benefits of displaying and observing
the nonverbal expressions of pride and shame.
Cognition and Emotion, 26, 390–406. http://dx.doi.org
/10.1080/02699931.2011.645281
Matsumoto, D. (1989). Cultural influences on the
perception of emotion. Journal of Cross-Cultural
Psychology, 20, 92–105. http://dx.doi.org/10.1177/
0022022189201006
Matsumoto, D. (1990). Cultural similarities and
differences in display rules. Motivation and Emotion,
14, 195–214. http://dx.doi.org/10.1007/BF00995569
Matsumoto, D. (2009). Culture and emotional
expression. In C. Y. Chiu, Y. Y. Hong, S. Shavitt, &
R. S. Wyer (Eds.), Problems and solutions in
cross-cultural theory, research, and application
(pp. 271–287). New York, NY: Psychology Press.
Matsumoto, D., & Ekman, P. (1989). American–Japanese
cultural differences in intensity ratings of facial
expressions of emotion. Motivation and Emotion,
13, 143–157. http://dx.doi.org/10.1007/BF00992959
Matsumoto, D., Frank, M. G., & Hwang, H. S. (Eds.).
(2013). Nonverbal communication: Science and
applications. Thousand Oaks, CA: Sage.
Matsumoto, D., & Hwang, H. S. (2013). Facial
expressions. In D. Matsumoto, M. G. Frank, &
H. S. Hwang (Eds.), Nonverbal communication:
Science and applications (pp. 15–52). Thousand
Oaks, CA: Sage.
Matsumoto, D., Olide, A., Schug, J., Willingham, B., &
Callan, M. (2009). Cross-cultural judgments of
spontaneous facial expressions of emotion. Journal
of Nonverbal Behavior, 33, 213–238. http://dx.doi.
org/10.1007/s10919-009-0071-4
Mayr, E. (1988). Toward a new philosophy of biology:
Observations of an evolutionist. Cambridge, MA:
Harvard University Press.
Mazur, A. (1985). A biosocial model of status in face-
to-face primate groups. Social Forces, 64, 377–402.
http://dx.doi.org/10.1093/sf/64.2.377
Mazur, A. (2005). Biosociology of dominance and deference.
Lanham, MD: Rowman & Littlefield.
McArthur, L. Z., & Baron, R. M. (1983). Toward an
ecological theory of social perception. Psychological
Review, 90, 215–238. http://dx.doi.org/10.1037/
0033-295X.90.3.215
McGrew, W. C. (1972). An ethological study of children’s
behavior. New York, NY: Academic Press.
McLuhan, M. (1964). Understanding media: The extensions
of man. New York, NY: McGraw Hill.
McNeill, D. (1992). Hand and mind: What gestures reveal
about thought. Chicago, IL: University of Chicago
Press.
The Life and Times of Nonverbal Communication
McNeill, W. H. (1995). Keeping together in time: Dance
and drill in human history. Cambridge, MA: Harvard
University Press.
Mehrabian, A. (1968). Communication without words.
Psychology Today, 2(9), 52–55.
Mehrabian, A. (1969). Significance of posture and
position in the communication of attitude and status
relationships. Psychological Bulletin, 71, 359–372.
http://dx.doi.org/10.1037/h0027349
Mehrabian, A. (1971). Silent messages. Belmont, CA:
Wadsworth.
Mehrabian, A. (1972a). Nonverbal communication.
Chicago, IL: Aldine-Atherton.
Mehrabian, A. (1972b). Nonverbal communication.
In J. K. Cole (Ed.), Nebraska symposium on motivation
(pp. 107–161). Lincoln: University of Nebraska Press.
Montagu, A. (1971). Touching. New York, NY: Harper &
Row.
Montepare, J. M., & Zebrowitz, L. A. (1998). Person
perception comes of age: The salience and
significance of age in social judgments. In M.
P. Zanna (Ed.), Advances in experimental social
psychology (Vol. 30, pp. 93–161). http://dx.doi.org/
10.1016/S0065-2601(08)60383-4
Moody, E. J., McIntosh, D. N., Mann, L. J., & Weisser,
K. R. (2007). More than mere mimicry? The
influence of emotion on rapid facial reactions
to faces. Emotion, 7, 447–457. http://dx.doi.
org/10.1037/1528-3542.7.2.447
Moore, M. M. (2010). Human nonverbal courtship
behavior—A brief historical review. Journal
of Sex Research, 47, 171–180. http://dx.doi.
org/10.1080/00224490903402520
Moors, A., Ellsworth, P. C., Scherer, K. R., & Frijda,
N. H. (2013). Appraisal theories of emotion:
State of the art and future development.
Emotion Review, 5, 119–124. http://dx.doi.
org/10.1177/1754073912468165
Morris, D. (1967). The naked ape. New York, NY:
McGraw-Hill.
Mulligan, K., & Scherer, K. R. (2012). Toward a
working definition of emotion. Emotion Review,
4, 345–357. http://dx.doi.org/10.1177/
1754073912445818
Oster, H. (2005). The repertoire of infant facial
expressions: An ontogenetic perspective. In J.
Nadel & D. Muir (Eds.), Emotional development:
Recent research advances (pp. 261–292). New York,
NY: Oxford University Press.
Otte, D. (1974). Effects and functions in the evolution
of signaling systems. Annual Review of Ecology and
Systematics, 5, 385–417. http://dx.doi.org/10.1146/
annurev.es.05.110174.002125
39
Caroline F. Keating
Parkinson, B. (2005). Do facial movements express
emotions or communicate motives? Personality and
Social Psychology Review, 9, 278–311. http://dx.doi.
org/10.1207/s15327957pspr0904_1
Patterson, M. L. (1999). The evolution of a parallel
process model of nonverbal communication. In
P. Philippot, R. S. Feldman, & E. J. Coats (Eds.), The
social context of nonverbal behavior (pp. 317–347).
Cambridge, England: Cambridge University Press.
Patterson, M. L. (2006). The evolution of theories
of interactive behavior. In V. Manusov & M. L.
Patterson (Eds.), The Sage handbook of nonverbal
communication (pp. 21–40). http://dx.doi.
org/10.4135/9781412976152.n2
Pessoa, L., & Adolphs, R. (2010). Emotion processing
and the amygdala: From a “low road” to “many roads”
of evaluating biological significance. Nature Reviews/
Neuroscience, 11, 773–783. http://dx.doi.org/10.1038/
nrn2920
Pitcairn, T. K., & Eiblesfeldt, I. (1976). Concerning the
evolution of nonverbal communication in man. In
M. E. Hahn & E. C. Simmel (Eds.), Communicative
behavior and evolution (pp. 81–113). New York, NY:
Academic Press.
Plutchik, R. (1980). Emotion: A psychoevolutionary
synthesis. New York, NY: Harper & Row.
Porter, S., & ten Brinke, L. (2008). Reading between
the lies: Identifying concealed and falsified emotions
in universal facial expressions. Psychological
Science, 19, 508–514. http://dx.doi.org/10.1111/
j.1467-9280.2008.02116.x
Posner, J., Russell, J. A., & Peterson, B. S. (2005).
The circumplex model of affect: An integrative
approach to affective neuroscience, cognitive
development, and psychopathology. Development
and Psychopathology, 17, 715–734. http://dx.doi.
org/10.1017/S0954579405050340
Redican, W. K. (1982). An evolutionary perspective on
human facial displays. In P. Ekman (Ed.), Emotion
in the human face (pp. 212–280). New York, NY:
Cambridge University Press.
Reisenzein, R., Studtmann, M., & Horstmann, G. (2013).
Coherence between emotion and facial expression:
Evidence form laboratory experiments. Emotion Review,
5, 16–23. http://dx.doi.org/10.1177/1754073912457228
Rice, B. (1978). The new truth machine: Does your voice
give you away? Psychology Today, 12(1), 61–78.
Riddihough, G., & Zahn, L. M. (2010, October 29).
Introduction: What is epigenetics? Science, 330, 611.
http://dx.doi.org/10.1126/science.330.6004.611
Riggio, H. R., & Riggio, R. E. (2010). Appearance-based
trait inferences and voting: Evolutionary roots and
implications for leadership. Journal of Nonverbal
Behavior, 34, 119–125. http://dx.doi.org/10.1007/
s10919-009-0083-0
40
Rosenfeld, H. M. (1965). Effect of an approval-seeking
induction on interpersonal proximity. Psychological
Reports, 17, 120–122. http://dx.doi.org/10.2466/
pr0.1965.17.1.120
Rosenfeld, H. M. (1966). Instrumental affiliative
functions of facial and gestural expressions. Journal
of Personality and Social Psychology, 4, 65–72. http://
dx.doi.org/10.1037/h0023514
Rothstein, S. (1980). The preening invitation or
head-down display of parasitic cowbirds: II.
Experimental analysis and evidence for behavioural
mimicry. Behaviour, 75, 148–182. http://dx.doi.
org/10.1163/156853980X00384
Rule, N., & Ambady, N. (2010). First impressions of
the face: Predicting success. Social and Personality
Psychology Compass, 4, 506–516. http://dx.doi.
org/10.1111/j.1751-9004.2010.00282.x
Rule, N. O., Krendl, A. C., Ivcevic, Z., & Ambady, N.
(2013). Accuracy and consensus in judgments
of trustworthiness from faces: Behavioral and
neural correlates. Journal of Personality and Social
Psychology, 104, 409–426. http://dx.doi.org/10.1037/
a0031050
Russell, J. A. (1980). A circumplex model of affect.
Journal of Personality and Social Psychology, 39,
1161–1178. http://dx.doi.org/10.1037/h0077714
Russell, J. A. (1994). Is there universal recognition of
emotion from facial expressions? A review of the
cross-cultural studies. Psychological Bulletin, 115,
102–141. http://dx.doi.org/10.1037/0033-2909.
115.1.102
Russell, J. A., & Carroll, J. M. (1999). On the bipolarity
of positive and negative affect. Psychological
Bulletin, 125, 3–30. http://dx.doi.org/10.1037/
0033-2909.125.1.3
Ruys, K. I., & Stapel, D. A. (2008). Emotion elicitor or
emotion messenger? Subliminal priming reveals
two faces of facial expressions. Psychological Science,
19, 593–600. http://dx.doi.org/10.1111/j.1467-9280.
2008.02128.x
Saarni, C. (1989). Children’s understanding of
strategic control of emotional expression in social
transactions. In C. Saarni & P. Harris (Eds.),
Children’s understanding of emotion (pp. 181–208).
New York, NY: Cambridge University Press.
Saarni, C., & Weber, H. (1999). Emotional displays and
dissemblance in childhood: Implications for self-
presentation. In P. Philippot, R. S. Feldman, & E. J.
Coats (Eds.), The social context of nonverbal behavior
(pp. 317–347). Cambridge, England: Cambridge
University Press.
Sacco, D. F., & Hugenberg, K. (2009). The look of fear
and anger: Facial maturity modulates recognition
of fearful and angry expressions. Emotion, 9, 39–49.
http://dx.doi.org/10.1037/a0014081

Safdar, S., Friedlmeier, W., Matsumoto, D., Yoo, S. H.,
Kwantes, C. T., Kakai, H., & Shigemasu, E. (2009).
Variations of emotional display rules within and
across cultures: A comparison between Canada, USA,
and Japan. Canadian Journal of Behavioural Science/
Revue canadienne des sciences du comportement, 41,
1–10. http://dx.doi.org/10.1037/a0014387
Salzman, C. D., & Fusi, S. (2010). Emotion, cognition,
and mental state representation in amygdala and
prefrontal cortex. Annual Review of Neuroscience,
33, 173–202. http://dx.doi.org/10.1146/annurev.
neuro.051508.135256
Sauter, D. A., Eisner, F., Ekman, P., & Scott, S. K. (2010).
Cross-cultural recognition of basic emotions through
nonverbal emotional vocalizations. Proceedings of the
National Academy of Sciences, USA, 107, 2408–2412.
http://dx.doi.org/10.1073/pnas.0908239106
Scheflen, A. E. (1972). Body language and social order.
Englewood Cliffs, NJ: Prentice Hall.
Scheflen, A. E. (1974). How behavior means. Garden City,
NY: Anchor Books.
Scherer, K. R. (1978). Personality inference from voice
quality: The loud voice of extraversion. European
Journal of Social Psychology, 8, 467–487. http://
dx.doi.org/10.1002/ejsp.2420080405
Scherer, K. R. (2001). Appraisal considered as a process
of multi-level sequential checking. In K. R. Scherer,
A. Schorr, & T. Johnstone (Eds.), Appraisal processes
in emotion: Theory, methods, research (pp. 92–120).
New York, NY: Oxford University Press.
Scherer, K. R. (2003). Vocal communication of
emotion: A review of research paradigms. Speech
Communication, 40, 227–256. http://dx.doi.
org/10.1016/S0167-6393(02)00084-5
Scherer, K. R., Banse, R., & Wallbott, H. (2001).
Emotion inferences from vocal expression correlate
across languages and cultures. Journal of Cross-
Cultural Psychology, 32, 76–92. http://dx.doi.
org/10.1177/0022022101032001009
Scherer, K. R., Clark-Polner, E., & Mortillaro, M.
(2011). In the eye of the beholder? Universality
and cultural specificity in the expression and
perception of emotion. International Journal of
Psychology, 46, 401–435. http://dx.doi.org/
10.1080/00207594.2011.626049
Scherer, K. R., & Ellgring, H. (2007). Multimodal
expression of emotion: Affect programs or
componential appraisal patterns? Emotion,
7, 158–171. http://dx.doi.org/10.1037/
1528-3542.7.1.158
Scherer, K. R., Johnstone, T., & Klasmeyer, G. (2003).
Vocal expression of emotion. In R. J. Davidson, K.
R. Scherer, & H. H. Goldsmith (Eds.), Handbook
of affective sciences (pp. 433–456). New York, NY:
Oxford University Press.
The Life and Times of Nonverbal Communication
Schlosberg, H. (1954). Three dimensions of emotion.
Psychological Review, 61, 81–88. http://dx.doi.
org/10.1037/h0054570
Segall, M. H., Campbell, D. T., & Herskovitz, M. J.
(1966). The influence of culture on visual perception.
Indianapolis, IN: Bobbs-Merrill.
Shaver, P., Schwartz, J., Kirson, D., & O’Connor, C.
(1987). Emotion knowledge: Further exploration
of a prototype approach. Journal of Personality and
Social Psychology, 52, 1061–1086. http://dx.doi.
org/10.1037/0022-3514.52.6.1061
Sheldon, H. (1940). The varieties of human physique.
New York, NY: Harper.
Smith, M. L., Cottrell, G. W., Gosselin, F., & Schyns,
P. G. (2005). Transmitting and decoding facial
expressions. Psychological Science, 16, 184–189.
http://dx.doi.org/10.1111/j.0956-7976.2005.00801.x
Smith, W. J. (1977). The behavior of communicating.
Cambridge, MA: Harvard University Press.
Sommer, R. (1967). Small group ecology. Psychological
Bulletin, 67, 145–152. http://dx.doi.org/10.1037/
h0024201
Sommer, R. (1969). Personal space: The behavioral basis of
design. Englewood Cliffs, NJ: Prentice Hall.
Szameitat, D. P., Alter, K., Szameitat, A. J., Darwin, C.
J., Wildgruber, D., Dietrich, S., & Sterr, A. (2009).
Differentiation of emotions in laughter at the
behavioral level. Emotion, 9, 397–405. http://dx.doi.
org/10.1037/a0015692
Tamietto, M., & de Gelder, B. (2010). Neural bases
of the non-conscious perception of emotional
signals. Nature Reviews Neuroscience, 11, 697–709.
http://dx.doi.org/10.1038/nrn2889
Tickle-Degnen, L. (2006). Nonverbal behavior and
its functions in the ecosystem of rapport. In
M. L. Patterson & V. L. Manusov (Eds.), The
Sage handbook of nonverbal communication
(pp. 381–402). http://dx.doi.org/10.4135/
9781412976152.n20
Todorov, A., Pakrashi, M., & Oosterhof, N. N. (2009).
Evaluating faces on trustworthiness after minimal
time exposure. Social Cognition, 27, 813–833.
http://dx.doi.org/10.1521/soco.2009.27.6.813
Tomkins, S. S. (1962). Affect, imagery, and consciousness:
Vol. 1. The positive affects. New York, NY: Springer.
Tomkins, S. S. (1963). Affect, imagery, and consciousness:
Vol. 2. The negative affects. New York, NY: Springer.
Toulmin, S. (1972). Human understanding: The collective
use and evolution of concepts. Princeton, NJ: Princeton
University Press.
Tracy, J. L., & Robins, R. W. (2008). The nonverbal
expression of pride: Evidence for cross-cultural
recognition. Journal of Personality and Social
41
Caroline F. Keating
Psychology, 94, 516–530. http://dx.doi.
org/10.1037/0022-3514.94.3.516
Tracy, J. L., Shariff, A. F., Zhao, W., & Henrich, J.
(2013). Cross-cultural evidence that the nonverbal
expression of pride is an automatic status signal.
Journal of Experimental Psychology: General, 142,
163–180. http://dx.doi.org/10.1037/a0028412
Van den Stock, J., Righart, R., & de Gelder, B. (2007).
Body expressions influence recognition of emotions
in the face and voice. Emotion, 7, 487–494. http://
dx.doi.org/10.1037/1528-3542.7.3.487
van Hooff, J. A. R. A. M. (1972). A comparative
approach to the phylogeny of laughter and smiling.
In R. A. Hinde (Ed.), Nonverbal communication
(pp. 209–238). Cambridge, England: Cambridge
University Press.
van’t Wout, M., & Sanfey, A. G. (2008). Friend or foe:
The effect of implicit trustworthiness judgments in
social decision-making. Cognition, 108, 796–803.
http://dx.doi.org/10.1016/j.cognition.2008.07.002
Van Vugt, M., Hogan, R., & Kaiser, R. B. (2008).
Leadership, followership, and evolution: Some
lessons from the past. American Psychologist,
63, 182–196. http://dx.doi.org/10.1037/0003-
066X.63.3.182
von Hippel, B., & Trivers, R. (2011). The evolution
and psychology of self-deception. Behavioral and
Brain Sciences, 34, 1–56. http://dx.doi.org/10.1017/
S0140525X10001354
Waller, B. M., Cray, J. J., Jr., & Burrows, A. M. (2008).
Selection for universal facial emotion. Emotion, 8,
435–439. http://dx.doi.org/10.1037/1528-3542.8.3.435
Watson, O. M. (1972). Conflicts and directions
in proxemics research. Journal of
Communication, 22, 443–459. http://dx.doi.
org/10.1111/j.1460-2466.1972.tb00167.x
Weitz, S. (1974). Nonverbal communication: Readings with
commentary. New York, NY: Oxford University Press.
Widen, S. C., & Russell, J. A. (2010). The “disgust face”
conveys anger to children. Emotion, 10, 455–466.
http://dx.doi.org/10.1037/a0019151
Willis, J., & Todorov, A. (2006). First impressions:
Making up your mind after a 100-ms exposure to
a face. Psychological Science, 17, 592–598. http://
dx.doi.org/10.1111/j.1467-9280.2006.01750.x
Wilson, E. O. (1975). Sociobiology: The new synthesis.
Cambridge, MA: Belknap Press.
Wiltermuth, S. S., & Heath, C. (2009). Synchrony and
cooperation. Psychological Science, 20, 1–5. http://
dx.doi.org/10.1111/j.1467-9280.2008.02253.x
Wong, E. M., Ormiston, M. E., & Haselhuhn, M. P.
(2011). A face only an investor could love: CEOs’
facial structure predicts their firms’ financial
42
performance. Psychological Science, 22, 1478–1483.
http://dx.doi.org/10.1177/0956797611418838
Wundt, W. M. (1897). Outlines of psychology. Retrieved
from http://psychclassics.asu.edu/Wundt/Outlines
Zebrowitz, L. A. (1997). Reading faces: Window to the
soul? Boulder, CO: Westview Press.
Zebrowitz, L. A. (2003). Overgeneralization effects in
perceiving nonverbal behavior. Journal of Nonverbal
Behavior, 27, 133–138. http://dx.doi.org/10.1023/
A:1023966526601
Zebrowitz, L. A., & Collins, M. A. (1997). Accurate social
perception at zero acquaintance: The affordances
of a Gibsonian approach. Personality and Social
Psychology Review, 1, 204–223. http://dx.doi.org/
10.1207/s15327957pspr0103_2
Zebrowitz, L. A., Fellous, J. M., Mignault,
A., & Andreoletti, C. (2003). Trait impressions as
overgeneralized responses to adaptively significant
facial qualities: Evidence from connectionist
modeling. Personality and Social Psychology
Review, 7, 194–215. http://dx.doi.org/10.1207/
S15327957PSPR0703_01
Zebrowitz, L. A., Kikuchi, M., & Fellous, J. M.
(2007). Are effects of emotion expression on trait
impressions mediated by babyfaceness? Evidence
from connectionist modeling. Personality and Social
Psychology Bulletin, 33, 648–662. http://dx.doi.
org/10.1177/0146167206297399
Zebrowitz, L. A., Kikuchi, M., & Fellous, J. M. (2010).
Facial resemblance to emotions: Group differences,
impression effects, and race stereotypes. Journal
of Personality and Social Psychology, 98, 175–189.
http://dx.doi.org/10.1037/a0017990
Zebrowitz, L. A., Luevano, V. X., Bronstad, P. M., &
Aharon, I. (2009). Neural activation to
babyfaced men matches activation to babies.
Social Neuroscience, 4, 1–10. http://dx.doi.
org/10.1080/17470910701676236
Zebrowitz, L. A., & Montepare, J. M. (2008). Social
psychological face perception: Why appearance
matters. Social and Personality Psychology
Compass, 2, 1497–1517. http://dx.doi.org/
10.1111/j.1751-9004.2008.00109.x
Zebrowitz, L. A., Wadlinger, H. A., Luevano, V. X.,
White, B. M., Xing, C., & Zhang, Y. (2011).
Animal analogies in first impressions of faces.
Social Cognition, 29, 486–496. http://dx.doi.
org/10.1521/soco.2011.29.4.486
Zebrowitz, L. A., Wang, R., Bronstad, P. M., Eisenberg,
D., Undurraga, E., Reyes-García, V., & Godoy,
R. (2012). First impressions from faces among
U.S. and culturally isolated Tsimane’ people
in the Bolivian rainforest. Journal of Cross-
Cultural Psychology, 43, 119–134. http://dx.doi.
org/10.1177/0022022111411386
 Part II

FACTORS OF INFLUENCE
 Chapter 3
Evolution and Nonverbal
Communication
Mark G. Frank and Allison Z. Shaw
Scientists tend to concur that nonverbal commu-
nication is phylogentically older than verbal com-
munication. Although scientists debate whether
language emerged 400,000 years ago (Johansson,
2011) or 200,000 years ago (McCrone, 1991), or
whether it was an impoverished version of our
modern language (P. Lieberman, 2008) or in fact
was expressed like Sign Language through gestures
(Corballis, 2002; McNeill, 1992), it is clear that our
human ancestors were navigating the planet before
the advent of spoken language (up to 6 million
years ago; White et al., 2009). Language of course
is efficient, flexible, and adaptable; allows one to
represent objects and concepts symbolically; and
can span space and time. One merely needs to try to
communicate a request for future action—for exam-
ple, please send me a 10-slide PowerPoint presenta-
tion that explains fractals next Thursday—using
only nonverbal communication to understand the
efficiency of verbal communication. In fact, the
superiority of a complex language as a tool for com-
munication is what some scholars attribute the
success of the less robust and weaker Homo sapiens
over the more robust and stronger Neanderthals in
Europe and the Middle East (P. Lieberman, 2007).
Despite the capability of complex language,
the phylogenetically older nonverbal communica-
tion system still exists and thrives in human beings
today. Although we might argue that nonverbal
communication is a definitive signaling system,
designed to send messages like spoken language, we
intend on expanding the discussion in this chapter
to show that nonverbal cues can communicate a
http://dx.doi.org/10.1037/14669-003
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
wider variety of states, including those more associ-
ated with health and reproductive fitness. The actual
“signs,” “signals,” and “symbols” from these states
are not as clear as to whether they evolved as part of
a signaling system or just happen to co-occur and
became useful to the observant.
It was only in the last 150 years that scientists
began to address systematically the role of biology in
nonverbal communication, starting with Duchenne
de Boulogne’s (1862/1990) mellifluous notations
concerning differentiating smiles that are “put into
play by the sweet emotions of the soul” from those
that “obey the will” (p. 72). We recognize that many
of the authors in this handbook have tied their
review chapters on faces, bodies, voices, odors, and
so forth into various biological substrates. Histori-
cally, this integration has not always been the case
for nonverbal communication (see Chapter 1, this
handbook), and thus we consider it a strength of
this book. In this chapter, we hope to flip the frame-
work from that of examining nonverbal communi-
cation through each individual channel (e.g., face,
body, eyes) into that of examining the role of evolu-
tion on human survival and reproduction, and how
the process of evolution may be reflected in the vari-
ous nonverbal signs and signals emitted by people.
It will become apparent that some of these nonver-
bal communicative cues may have been selected as
messaging systems with specific meanings, whereas
other cues coevolved with other internal states that
one can argue have survival value but without seem-
ing to have a clear message plan or meaning. One
such mechanism may be to signal health or other
45
Frank and Shaw
states that make one desirable to reproduce with
(known as reproductive fitness). However, we warn
that for most of these nonverbal cues, we can only
deduce whether they were selected for as messages
or simply indicators of other internal states.
EVOLUTION OVERVIEW
The basic premise of Darwin’s (1859) theory of nat-
ural selection is that the genes that are responsible
for characteristics that ensure survival are passed on
to the next generation. This inheritance is accom-
plished through a number of processes: First, more
individuals are produced than an environment can
typically support, thus competition emerges; sec-
ond, genes vary within populations through mating
and random mutations and those genes—and their
resultant characteristics—are selected for because
they are conducive to winning the competition
within a particular environment ensuring the sur-
vival of the gene carrier; and third, those individuals
who have survived are in a position to procreate,
and then they have the opportunity to pass those
genes onto the next generation. Genes and their
resultant characteristics that hinder adaptation to a
particular environment are typically not passed on
due to the gene carrier not surviving long enough to
procreate. It is because of these processes that Dar-
win called it natural (nature) selection (the natural
environment dictated what worked for survival, not
any grand plan or possible Supreme Being). Thus,
conspecifics with slow, inefficient communica-
tion systems that do not permit speedy warning
against predators are eliminated by such predators
as they are easy prey. Conspecifics with speedier
communication systems, on the other hand, permit
many members of the group to escape the preda-
tor through an early warning; thus, group members
survive and then later reproduce. However, fleeing
from predators is just one survival task; organisms
must also compete and fight, must also feed to live,
and must also be in position to procreate more often
than other conspecifics to increase the odds of their
particular set of genes surviving through to the next
generation.
The implication of this system is that there are
two superordinate “tasks” that evolution requires
46
to pass down genes to the next generation. First,
and foremost, is survival to the age of reproduction.
Survival past the reproductive age is somewhat less
relevant, with the exception being the possibil-
ity of additional caretakers/hands within a society
(e.g., Caspari & Lee, 2004), which may facilitate
the survival of the prereproductive-age individuals
who share some of the genes of the caregiver (e.g.,
nephews/nieces). The second, and entirely depen-
dent on the first, is reproductive fitness—meaning
the reproductive age of survivors whose character-
istics are most desirable are vied for and thus
reproduce at a greater rate, which increases the
frequency of their genes within a population,
which, in turn, makes those genes more likely to
appear in ensuing generations.
Where Nonverbal Communication Fits
Nonverbal communication is instrumental in both
these tasks. Broadly speaking, nonverbal commu-
nication helps to ensure the primary task of sur-
vival to reproduction by expressing danger (fear),
imminent attack (anger), contamination (disgust),
needed attention (distress), and novelty (surprise).
Thus, members of the group can be forewarned of
danger, can sidestep a conflict, can avoid a poten-
tially poisonous food, and can attend to members
in distress. Note that the word “expressing” con-
notes communication, and it is communication
that allows for cooperation and coordinated behav-
iors. Nonverbal communication also facilitates
reproductive fitness by indicating who to affiliate
with (friend or kin); potential approachability
or danger from others; and health, fertility, and
strength/dominance. Regardless of whether these
behaviors are deliberate expressions, or after-
effects from other biological processes, they still
communicate information and appear to influence
the behavior of others. We argue that regardless
of whether these immutable characteristics and
behaviors were selected for, or at least were not
detrimental to the survival and reproductive fitness
of a species, these characteristics and behaviors
communicate relevant information for survival and
reproductive fitness, and they facilitate decision
making regarding initiating, maintaining, and ter-
minating interactions.

The extent to which nonverbal communication
is adaptive to the world of today is likely happen-
stance, as the vast majority of human existence on
the planet has been in the hunter–gatherer world of
more than 100,000 years ago (Tooby & Cosmides,
1990). That does not mean that nonverbal com-
munication is not adaptive in today’s world; it only
means that the behaviors we see today did not
evolve to meet the challenges of today such as elec-
tronic communication. However, nonverbal cues
studied today can provide insight into the most
stringent challenges for survival and reproduction
throughout the history of humanity. Thus, we can
argue that the emotion of fear evolved to motivate
humans to flee danger. It can be argued that in the
ancient world, threats to survival that induced fear
were mainly predators and fellow humans, but that
same emotion (i.e., fear) turns out to be helpful in
motivating us to flee things that did not exist in the
ancient world such as handguns (e.g., Öhman &
Mineka, 2001). In other words, although emotions
today may serve proximal needs, we speculate on
the distal causes for such behaviors.
Signs, Signals, Symbols, and Their
Voluntariness
To proceed, we must distinguish signs from signals
from symbols, and for the sake of consistency, we
start in a manner similar to Haviland-Jones, Wilson,
and Freyberg (see Chapter 14, this handbook).
They defined signs as part of an experience, or the
physical reaction to a stimulus, absent of meaning
or information. Signals, alternatively, communi-
cate information of one’s experience or reaction to
a stimulus. From a communication point of view,
such a definition makes no claim as to the design,
or intentionality, of the sign as a message to oth-
ers; however, typically, signs are not intentionally
produced by a sender. Signals, on the other hand, at
times, are intentionally construed. Sometimes, signs
are produced without intentional meaning; how-
ever, receivers infer information from the sign. Later
in this section, we discuss emotional signs (e.g.,
raising eyebrows out of fear) and how they can be
interpreted as signals (e.g., there is something to be
afraid of present) by others, although the producer
involuntarily produced the sign with no conscious
Evolution and Nonverbal Communication
intention of it being a signal to others. Alterna-
tively, signals can also be intentionally produced.
For example, in the context of nonverbal behavior,
deliberately fabricating signs associated with par-
ticular emotional states (e.g., anger) to signal a dif-
ferent but not felt emotional state (e.g., happiness) is
a well-studied instance of intentionally produced sig-
naling (i.e., deception). Finally, Haviland-Jones et al.
defined symbols as signs external to the individual
used to connote symbolic information. Extending
our example using fear, the sign (e.g., raising one’s
eyebrows out of fear) can both signal information
(e.g., danger) and represent symbolic information
(e.g., Run!). The determination of a sign, signal, and
symbol is often times in the eyes of the beholder,
making it challenging at times to determine when a
sign is a signal or a symbol.
Thus, in pulling back the window of observation
from Haviland-Jones et al.’s (see Chapter 14, this
handbook) phenomenon of study (odor) to nonver-
bal communication in general, it appears that we run
into an inference problem—voluntary versus invol-
untary actions. In the case of odor, Haviland-Jones
et al. have pointed out that much of it—the sign—is
involuntary; they have identified some exceptional
situations when individuals in some cultures will
consume products to change their odors—the
signal—or the less exceptional practice of apply-
ing substances to change odor (e.g., perfume; the
symbol). However, individuals cannot consciously
make their natural odors change without some sort
of exogenous agent. In nonverbal communication in
general, there are some behaviors where the individ-
ual can move in a deliberate way that can almost per-
fectly mimic an involuntary movement; for example,
one can smile and fool others into thinking one is
experiencing enjoyment when in fact one is not (e.g.,
Frank, Ekman, & Friesen, 1993). However, such an
instance is endogenously driven action, unlike the
odor change scenarios described.
Finally, we organize the sections by those non-
verbal signs, signals, and symbols that may have
aided survival (i.e., dealing with the immediate chal-
lenges of life) and those dealing more with repro-
ductive fitness. We look at the processes underneath
all three, and we examine how they are expressed
in the faces, eyes, voices, bodies, and even odors of
47
Frank and Shaw
people. These nonverbal cues can be dynamic and
quick, as in a facial expression; can be slow, as in
signs of aging or odors; or can be static, as in one’s
neutral facial appearance or body shape (cf. Ekman,
1978). We realize that some of these cues might be
seen as dynamic or slow, or slow and static; thus,
their categorization may overlap, but it will provide
a starting point upon which to examine the impor-
tance of nonverbal communication in evolution,
and evolution in nonverbal communication. Regard-
less, they are all nonverbal and are likely selected
for because they aided our survival or increased our
chances of reproducing.
SURVIVAL
Our modern, suburban lives are designed to reduce
the dangers of life; we have traffic lights, law
enforcement agencies, laws concerning food purity,
and so forth. Such was not the case in the ancient
world. Analyses of the skeletons of humans found
in ancient cities suggest that close to 30% of males
died from homicide (LeBlanc, 2003). Even up until
the 1970s, anthropologists noted that in preliterate,
indigenous rain forest people, up to 25% of the
males died from homicide (e.g., the Yanamamo
people; Chagnon, 1988). Recent summaries of
human violence have argued convincingly that
humans have become less violent with the passage
of time (Pinker, 2011); thus, regardless of the
extent to which we believe we live in a dangerous
world today, it is not nearly as dangerous as the
ancient world.
It seems two characteristics of human beings
are most amenable to our survival. One is having a
hard-wired response system that motivates behavior
without much contemplation; thus, individuals can
respond quickly and efficiently to potential threats
(Tooby & Cosmides, 2005). The other is living in
groups, which permits aid when ill, allies in war,
shared resources such as food, and remote warnings
for danger. Both of these characteristics are embod-
ied in emotion.
Emotion
Matsumoto and Hwang (2013) defined emotions as
“transient, bio-psycho-social reactions to events that
48
have consequences for our welfare and potentially
require immediate action” (p. 17). Thus, emo-
tions are intimately tied to our biology, affect our
thinking, and have implications for those around
us. Emotions affect our autonomic nervous system
(ANS), which by altering our heart rate, blood
pressure, blood flow, and sweat glands produces
the sensations we know as emotions (Levenson,
Ekman, & Friesen, 1990). Specific emotions appear
to have specific patterns of ANS activity (Ekman,
Levenson, & Friesen, 1983), and these patterns seem
to be universal, as research has revealed that the
Minangkabau people—a matrilineal, Muslim society
in Indonesia—show the same ANS patterns for the
same emotions as university students in the United
States (Levenson, Ekman, Heider, & Friesen, 1992).
In other words, the ANS reorganizes the body’s
priorities to enable quick action, and these patterns
are found in all humans studied to date. Moreover,
other work has shown that these emotions also have
distinct central nervous system patterns (reviewed
in Phan, Wager, Taylor, & Liberzon, 2002; Vytal &
Hamann, 2010). Such patterns of emotional
response would allow us to presume that these reac-
tions are part of the wiring of our species and must
have been helpful to our survival and thus selected
for. We also add that research shows that these
emotions are not necessarily exclusive to humans;
research has shown that chimpanzees, rats, and
other mammals also have emotions and correspond-
ing brain circuits (e.g., Panksepp, 1998). The ter-
minology scientists use for animals is often slightly
different than what we use for humans—for exam-
ple, panic and attack are terms used for animals,
whereas fear and anger are the analogous terms used
for humans—and this terminology shift is likely
due to scientists having to use nomenclature that
describes the animal’s behavior that is presumed to
be driven by emotions, as animals have not been as
forthcoming as humans in describing their inner
states and experiences. Regardless, the basic emo-
tion patterns are similar. The other aspect of the
definition that is important from an evolution point
of view is that emotions are reactions to events.
Of course, in animals that have memories, the recall
of a past event that had triggered an emotion in the
past can retrigger an emotion, and the imagining of

some yet-to-occur event can also trigger an emotion.
Regardless, at their core, emotions are at the inter-
face of organisms with their environment (Ekman,
2003; LeDoux, 1996; Matsumoto & Hwang, 2013;
Panksepp, 1998). This interaction comes into play
in more detail later when we discuss the signs, sig-
nals, and symbols of these emotions.
Research shows that there are anywhere from
five to nine “basic” human emotions, with most
researchers settling on seven—anger, contempt,
disgust, fear, enjoyment, sadness/distress, and sur-
prise (e.g., Ekman, 2003). Some scholars include
interest (e.g., Izard, 1977); others include embar-
rassment (e.g., Keltner, 1995), pride (Tracy &
Robins, 2008), or shame and guilt (e.g., Keltner &
Buswell, 1997). Regardless, these seven emotions
are referred to as basic because they are universal in
physiological activation pattern, and their signals
are recognized across our species (Ekman, 1993).
Universality and the physiological mediators are
just two criteria for emotions being evolutionarily
selected for; we would also expect to see some
form in lower animals, and we would expect to
see the emotion develop regardless of social learn-
ing opportunities (e.g., in those born blind, which
research evidence supports; e.g., Galati, Miceli, &
Sini, 2001; Galati, Sini, Schmidt, & Tinti, 2003;
Matsumoto & Willingham, 2009). Ekman and
Cordaro (2011) went as far as to stipulate up to 12
criteria for determining the evolutionary under-
pinnings of an emotion; these include distinctive
universal signals; distinctive physiology; automatic
appraisal, tuned to distinctive universals in ante-
cedent events; distinctive appearance develop-
mentally; presence in other primates; quick onset;
brief duration; unbidden occurrence; distinctive
thoughts, memories, and images; and distinctive
subjective experience. Regardless, we would need
to be able to make an argument that the presence
of that emotion would facilitate the survival of the
organism, and the absence of the emotion would
be detrimental to the organism. Therefore, an argu-
ment can be made for the survival facilitating ability
of each of the basic seven emotions by what it does
to the body to facilitate an action (action tendency)
that might save an organism (Frijda, 1986). For
example, anger moves blood to the upper body and
Evolution and Nonverbal Communication
increases muscle tension to prepare to fight and/
or attack; disgust slows the heart rate and prepares
the body to eject some ingested item, or stops them
from ingesting altogether; fear moves blood to the
large muscles of the legs to facilitate escape and
increases sweat to improve the grip in hands and
feet; and so forth (see Levenson, 1999). Moreover,
Tooby and Cosmides (2005) argued further for
emotions as the evolutionarily functional interface
of individuals and the environment. Specifically,
emotions that evolved in the ancestral environment
toward events that recurred, which required coordi-
nating the body actions to overcome the event, had
a repeated and consistent pattern of action that also
signaled when it was engaged—and that not hav-
ing such a reaction could cost the individual his or
her life, which of course prevents procreation. Only
those individuals with such behavioral response
patterns would survive to reproduce and, thus,
distribute this reaction pattern across the species.
Given that these observations can be made in other
mammals as well suggests the biological and evolu-
tionary roots of emotion.
Signs—Face.  Given the emphasis on nonverbal
communication, we turn our focus to the signs
that accompany emotional reactions. Ekman
(1999) suggested that these signs are important
because they signal important information to con-
specifics; this information might indicate danger,
or impending attack, and so forth. Many of the
physiological signs of these emotions are not so
easy to see; it is hard to detect a change in heart
rate or slight increase in sweat by simply observing
another person. Occasionally, one can see change in
blood pressure through changes in skin color due to
flushing in the face of light-complexioned individu-
als (as with anger or embarrassment; Keltner, 1995),
but except under the most extreme circumstances,
these signs are hard to detect.
However, the predominant nonverbal sign of
an emotion is the movement of muscles in the face
into particular configurations for each emotion; the
configuration is called the facial expression (see
Chapter 10, this handbook, for the mechanics of the
musculature underlying the facial expression; note
that the facial muscles are the only muscles in the
49
Frank and Shaw
body that connect bone to skin; thus, one purpose
must be to move that skin in a way to be seen by
others). Darwin (1872/1998) identified three princi-
ples underlying the production of facial expressions
that underscore their evolutionary roots; first, he
discussed the principle of serviceable habits, which
argued that the precise configuration of the expres-
sion served the purpose of the particular emotion.
He also discussed the principle of antithesis, which
argued that for some emotions, their adaptation was
to appear opposite, or at least distinctly different, to
another emotion so as to not confuse the two. His
third principle was that of nerve force, which meant
that the impulse for the emotion expression is invol-
untary. All three of these principles ultimately were
considered essential to survival and, thus, were
selected for evolutionarily.
The principle of serviceable habits works along
the same ideas as the ANS activation, where the
physiology facilitates the behavioral action (e.g.,
Levenson, 1999). Thus, in the emotion of anger,
the eyebrows lower, and the upper eyelids raise
in a glare; this response narrows the focus onto
the object or individual that needs to be removed
or potentially attacked. Likewise, the lips narrow,
which functions to move them out of the way so
that a bite can be executed. In contrast, fear raises
the eyebrows and raises the upper eye lids; this
response permits more information to enter the per-
ceptual realm of the individual to enable the maxi-
mum information processing so as to make the best
assessment of any potential danger. Disgust involves
the wrinkling of the nose and raising of the upper
lip; this response is purported to reduce the sensory
information of a potentially sickening ingestible
item. Recent work has supported many of these
assertions. For example, when individuals pose a
fear expression, the amount of sinus cavity surface
area (a marker of potential sensory input) expands
compared to an individual’s neutral, nonexpres-
sive face. Furthermore, a posed disgust face reduces
the sinus cavity surface area, thus reducing sensory
input, just as Darwin had predicted (Susskind et al.,
2008). Anecdotally, it also appears that making a
disgust face while having food in one’s mouth moves
the food to the front of the mouth, thus making it
easier to eject. This reaction may account for why
50
the first author had to reestablish his nonverbal
expertise credentials with his laughing wife after
their infant children, who made disgust expressions
when introduced to new foods, requested more of
the food after swallowing. It seemed that the chil-
dren were testing it, and the disgust face permitted
testing a smaller sensory sample in preparation to
eject it if it was potentially distasteful.
The principle of antithesis, however, works
opposite to serviceable habits. Darwin (1872/1998)
used the example of the dog in attack mode; the
dog’s hair raises at the shoulder, and its ears and
tail go erect (to make it look bigger); its lips curl
to display the teeth as weapons as well as to permit
effective biting. However, the dog who is show-
ing submission (the opposite of attack), as when
greeting its pack leader (or human owner), does
not raise its hair; instead, its shoulders, ears, and
tail go low, thus making the dog look smaller. This
behavior is the antithesis of attack, and its appear-
ance is clearly distinct from attack, thus making
such a posture very hard to confuse with the attack
posture. Similarly, the appearance of the human
smile may be a function of this antithesis principle;
the lips turn up versus down in anger, fear, and
sadness. In fact, when judging posed expressions
of emotion, the smile is least confused with other
expressions (e.g., Ekman et al., 1987) and also is
the expression that can be seen from the farthest
distance away (Hager & Ekman, 1979). Others have
expanded this idea of antithesis to argue that it not
only applies to the direct comparison of happiness
(lip corners up) and sadness (lip corners down) but
also to anger (eye brows down) and fear (eyebrows
up; Weisfeld & Goetz, 2013). This conjecture is
confirmed by analyses of confusions in judgments,
where anger and fear, much like happy and sad, are
rarely confused with each other (Ekman et al., 1987;
Matsumoto, 1993).
The principle of nerve force argues that these
expressions, when driven by emotion, are invol-
untary reactions and, in fact, are part of the entire
coherent emotional reaction that features the ANS,
central nervous system, and body and vocal actions
(see Chapter 10, this handbook). This principle is
supported by 74 published studies between 1972
and 2007 that have demonstrated the link between

the ANS or central nervous system and facial expres-
sions of emotion (Matsumoto, Keltner, Shiota,
O’Sullivan, & Frank, 2008). Darwin further argued
that due to this nerve force, one could not fully
mimic a genuine emotional expression, as some
facial muscles are harder to move deliberately. That
also meant facial muscles will be harder to inhibit if
they are activated; specifically, Darwin (1872/1998)
said the following:
Some actions ordinarily associated
through habit with certain states of mind
may be partially repressed through the
will, and in such cases the muscles which
are least under the separate control of
the will are the most liable still to act,
causing movements which we recognize
as expressive. In certain other cases the
checking of one habitual movement
requires other slight movements; and
these are likewise expressive. (p. 34)
Thus, emotional facial expressions are signs, and
a plethora of research has repeatedly high agreement
rates across all cultures between particular facial
expressions and their emotion sign (see Chapter
10, this handbook, for a thorough review of this
work). However, Darwin also has left open the door
here that facial expressions are not just signs of an
emotion, and thus a particular behavior is likely
to follow, but can also be mimicked without being
felt (activating the physiological response typical
of that emotion), thus moving them into the realm
of a deliberate, consciously chosen signal, which is
addressed later.
Signs—Body.  The other behaviors that support
the action tendency of the emotion can also be
detected by observers. The body shows signs, such
that the movement of angry individuals is different
from sad, fearful, or joyful individuals (Atkinson,
Dittrich, Gemmell, & Young, 2004; Crane & Gross,
2007; Montepare, Goldstein, & Clausen, 1987;
Wallbott, 1998). Thus, anger is associated with more
forward body lean and greater arm swing (attack),
whereas fear has shorter steps and less forward lean
(to retreat; de Meijer, 1989). One can even extend
Darwin’s principle of antithesis to body action; the
Evolution and Nonverbal Communication
body expression of pride (which we can infer also
conveys dominance) features upward body posture
and arms/hands upward, whereas defeat or shame
features the body bent downward (Tracy, Shariff, &
Cheng, 2010; Weisfeld & Dillon, 2012). However,
we do suggest some caution here, as the major-
ity of these studies asked individuals to pose these
emotions.
Signs—Voice.  Other signs of emotion involve
the voice. Due to the nature of human emotion,
research on vocal expression of emotions parallels
many of the same issues involving facial expressions
of emotion. First off, other animals feature vocal-
izations that communicate threats, danger, nature
of relationships, as well as their emotional states
(e.g., Kitchen, Cheney, & Seyfarth, 2003). Darwin
(1872/1998) also described many paralinguistic cor-
relates of various emotional states. However, unlike
the muscles in the face, which receive direct nerve
impulses to produce the expression (see Chapter
10, this handbook), the larynx and sound produc-
tion apparatus of humans do not receive direct nerve
connections from the subcortical structures of the
brain associated with emotion (see Chapter 11, this
handbook). These connections tend to innervate the
structures that surround the sound-making appara-
tus. Thus, researchers encounter a conundrum—like
that in facial research, voice research shows greater
than chance agreement in judging emotion from
the voice but without as clear a consensus as to the
vocal characteristics of each emotion (see review
by Scherer, 2003). Most studies show increase in
pitch for anger and fear, but the universal signa-
ture for each emotion is elusive (again, see Chapter
11, this handbook). Despite that, a comprehensive
review of the literature showed that the vocal signs
of anger, fear, happiness, and sadness were judged
accurately both within and across cultures (Juslin &
Laukka, 2003). For both within and across cultures,
the agreement pattern was the same—the highest
levels of agreement were for the vocal expressions
of anger and sadness, and the lowest level was for
happiness—which is opposite to what researchers
note in judgment studies of the face, as per what
the principle of antithesis would suggest (Juslin &
Laukka, 2003). Although anger, fear, happiness,
51
Frank and Shaw
and sadness have received the most study, smaller
numbers of studies have examined agreement for
emotions—such as disgust, contempt, boredom,
embarrassment, guilt, and shame—and many shades
of positive emotion, including amusement, relief,
contentment, and so forth. However, the results
for these emotions do not show the uniform high
levels of agreement found in the facial expression
judgments (Sauter, Eisner, Calder, & Scott, 2010;
Simon-Thomas, Keltner, Sauter, Sinicropi-Yao, &
Abramson, 2009). Thus, to generalize, the prepon-
derance of the research on the vocal signs of emo-
tion shows cross-cultural agreement on emotion
patterns in the voice, even when the groups are rela-
tively isolated from Western culture (e.g., Bryant &
Barrett, 2008; Scherer, Banse, Wallbott, & Goldbeck,
1991). Two other vocal actions also require mention,
and both seem to have cross-cultural signs. The first
is laughter, which is seen in all cultures—yet, the
actual acoustic properties vary greatly, suggesting
that laughter may serve a few different functions,
ranging from extreme joy to nervousness (Grammer &
Eibl-Eibesfeldt, 1990). Crying is also found in all
cultures, and it tends to be mostly associated with
sadness (Klineberg, 1940)—but it can be seen at
times of supposedly happy occasions (Hendriks,
Nelson, Cornelius, & Vingerhoets, 2008).
We can envision why audible signals of emotion
might have been selected for. First, audible signals
can transmit information under poor viewing condi-
tions, such as distance, darkness, or even when the
individual has his or her face turned away. These
signs of emotion are also found in the animal calls as
well (e.g., Hauser, 1997; Kitchen et al., 2003; Pank-
sepp, 1998). It makes sense to have a redundant sig-
naling system that can augment or substitute for the
primary visually transmitted facial expression.
Signals—Face, body, and voice.  Neuroanatomical
research confirms Darwin’s observation that facial
expressions can be biologically driven, involuntary,
and harder to control (as in the case of the basic
emotions) and, thus, can be considered signs of that
emotion. Given that during the experience of emo-
tions the face, body, and voice produce signs, all of
these signs can serve as a signal as long as a receiver
receives, knowingly or not, information associated
52
with that sign. However, and likely more interest-
ingly, emotions can also be voluntarily posed to func-
tion as a signal to represent that emotion, without
the individual actually experiencing that emotion.
Although Darwin (1872/1998) proposed that facial
expressions are part of an involuntary emotional
impulse (nerve force), research and observation
show that they are not solely the product of invol-
untary emotional impulses. In fact, almost all the
judgment studies of facial expressions of emotion,
starting with Darwin (1872/1998) and restarting
with Tomkins (1962, 1963), used posed expressions
for judgments. In these studies, individuals posed
with facial expressions that resemble anger, con-
tempt, disgust, fear, happiness, sadness, and surprise
even though they were not actually experiencing
those emotions. There are times, however, in social
situations where people attempt to squelch, conceal,
or mask their emotional expression by trying to
show a different emotion (e.g., showing happiness
when someone truly feels sadness). Falsifying emo-
tional expressions may be a socially expedient move,
or it may be an attempt to mislead another. The rea-
son deliberate signaling of emotion is possible is that
there are two distinct neural pathways that mediate
facial expressions, each one originating in a different
area of the brain. The pyramidal motor system drives
the voluntary facial actions and originates in the cor-
tical motor strip, whereas the extrapyramidal motor
system drives the more involuntary, emotional facial
actions and originates in the subcortical areas of
the brain (Meihlke, 1973; Myers, 1976; Tschiassny,
1953). The research documenting these differences
was reliable enough (see review by Rinn, 1984) that
prior to modern soft tissue imaging technology,
observations of posed and spontaneous facial expres-
sions served as the primary diagnostic criteria for
certain brain lesions (DeMyer, 1980).
Research in deception detection, however,
demonstrates clearly that individuals are not
always adept at fabricating emotions. In instances
of fabricated emotions, both the pyramidal and
extrapyramidal motor systems can be activated
simultaneously. When an emotion is triggered,
the subcortical area of the brain sends an involun-
tary ballistic-like signal to the facial nerve(s). To
conceal this response, the individual recruits his

or her voluntary cortical motor strip area of the
brain, which sends a signal to suppress, amplify,
or disguise his or her expression in a socially and
culturally acceptable way. These competing systems
create a “tug of war” over control of the face, and
when the subcortical impulse is strong enough, the
expression will leak onto the face for a very brief
time before the voluntary motor systems regain con-
trol of the expression. This competitive confluence
of signals produces an emotional facial expression
that is shorter in duration than the 0.5–5-s dura-
tion originally identified (Ekman & Friesen, 1982).
This process is the biology of a microexpression
(Ekman & Friesen, 1969a; Frank & Svetieva, 2014;
Haggard & Isaacs, 1966); these expressions are so
brief because the individual intends to conceal this
involuntary action, thus preventing them from being
a sign—and, subsequently, a signal—to others.
This process is also a test of Darwin’s nerve force
principle, and later work has shown that individu-
als do show emotions even when they deliberately
attempt to conceal them (e.g., Ekman, O’Sullivan,
Friesen, & Scherer, 1991; Frank & Ekman, 2004;
Hurley & Frank, 2011; Porter & ten Brinke, 2008;
ten Brinke, Porter, & Baker, 2012). A second rea-
son these expressions are hard to mimic is that not
only do voluntary and involuntary facial actions
differ by neural pathway, but the actions mediated
by these pathways manifest themselves differently.
In a normal person, voluntary pyramidal motor-
system-based movements are limited solely by the
will. However, extrapyramidal motor-system-based
facial actions are characterized by synchronized,
smooth, symmetrical, consistent, and reflex-like or
ballistic-like actions on the part of the component
facial muscles (Ekman & Friesen, 1982; see review
by Rinn, 1984). Relatively speaking, these actions
appear to be less under the deliberate control of peo-
ple. Thus, these spontaneously expressed emotional
expressions tend to have dynamic qualities differ-
ent from deliberate, false emotional signals, such as
having smoother onsets, more symmetry, and a cir-
cumscribed duration lasting between 0.5 and 5 s in
length (Ekman & Friesen, 1982; Frank et al., 1993).
Signaling body emotions that one is experienc-
ing or not aware of is also readily done. As we noted
earlier, similar to the facial expression research, the
Evolution and Nonverbal Communication
vast majority of the research that attempted to iden-
tify the signs of emotion in the body actions have
asked individuals to pose those gross bodily expres-
sions (e.g., Montepare et al., 1987; Wallbott, 1998).
Anecdotally, however, the second author often is
asked by others if she needs to use the bathroom
because she is shaking her leg or foot unbeknownst
to her. Often she does not and notes that she must
be feeling anxious about something. Fidgeting is
often a sign of anxiety and signals to others the
experience of that emotion. It is reasonable to pre-
sume gross body movements that are in service to
the emotion action tendency—thus, forward lean
in the approach emotions such as anger and happy,
and backward lean in the avoidant
emotions such as fear, disgust, and contempt.
Signaling emotions by voice is also possible.
Similar to the research on facial expressions, the vast
majority of the research on recognizing emotions
from the voice has come from individuals mim-
icking the emotions in one’s voice (e.g., Scherer,
2003). Thus, a stimulus subject may be asked to
say a phrase “these pretzels are making me thirsty”
to sound disgusted, or to sound angry, or to sound
happy, or to sound surprised, and so forth. In this
instance, emotions are not elicited but are acted.
However, the research shows that this acting is
quite effective, and people can recognize the emo-
tion attempted at rates greater than 70% agreement
(reviewed by Frank, Griffin, & Maroulis, 2013).
Vocally signaling emotions that one does not
feel is not unique to humans. The ethological lit-
erature is replete with examples of animals man-
aging their threat calls to indicate the location of
the threat, or to juveniles delivering false distress
calls to bring in an adult to chase away other juve-
nile competitors (reviewed by Hauser, 1997). One
primary phenomenon that accounts for signaling
emotions that one may not feel, or concealing emo-
tions one does feel, is the cultural display rules (for
a much more in depth discussion of culture and
facial expression, see Chapters 4 and 10, this hand-
book). That is, to smooth social interaction, each
culture establishes rules, hierarchies, and so forth
to prevent chaos and to permit group action—and
learning to manage one’s emotional output is an
essential feature of that process. Smoothing social
53
Frank and Shaw
interaction is what allows individuals to cooper-
ate in groups and to thus live in groups, with the
resultant advantages of that lifestyle (allies in war,
food, other shared resources, etc.).
Symbols—Face, body, and voice.  Moreover,
people can also use their face to display symbolic
gestures (Ekman & Friesen, 1969b), such as raising
one’s eyebrow to indicate skepticism, or winking to
indicate “I’m kidding.” These facial expressions are
culturally specific, learned like language (Ekman,
1977), and tend to be more variable in their dura-
tion on the face than emotional expressions (Frank
et al., 1993). Therefore, they are not symbols of
emotional states but symbols for other concepts
or items. One can also use hand gestures, known
as emblems (Efron, 1941), to substitute for words
such as insults, agreement, negation, confusion, and
even speech itself through American Sign Language.
Vocally, people may use vocal signs such as fake
coughing to symbolically represent sarcasm.
Survival value.  There are good reasons why it
would be important, from an evolution point of
view, to show signs, as well as signals and symbols,
of these emotional states. The research shows that
there do appear to be sections of the brain that
respond to these particular signs of emotion (e.g.,
Sabatinelli et al., 2011). Thus, we can argue that
not only have these signs evolved as part of an
emotional reaction but so has a “receiver” to detect
these emotion signs. Because both of the systems
of producing and processing a sign resulted from
the same distal process, we can now speculate as
to how showing signs of these emotions may have
aided our species. Let’s start with anger. It may
seem disadvantageous to show anger, as that would
seem to give one’s opponent an early warning; thus,
the angered attacker loses the element of surprise,
which may tip the fight to the attacker’s advantage.
However, research has shown that the vast majority
of our interactions are with ingroup members, and
showing anger has the advantage of often prevent-
ing a fight and, thus, obtaining compliance without
the risk of injury or infection. In fact, social conflict
researchers have known that threats are effective
means for obtaining compliance and are inexpen-
sive to deliver (unless one has to follow through;
54
Pruitt & Kim, 2004). However, there are dangers
to following through on the threat. Infections are
a huge complication associated with fighting, and
not just in the ancient world but as recent as 100
years ago. For example, historians estimate that
62 soldiers died from infection per 1,000 in the
American Civil War; with the advent of antibiot-
ics, that rate dropped to 0.6 per 1,000 in World
War II (Gilchrist, 1998). Thus, a fight that could
result in an open wound could be fatal to both win-
ner and loser. The expression of anger is often the
first to be noticed in a crowd (Hansen & Hansen,
1988; Williams & Mattingley, 2006; but see also
Calvo & Nummenmaa, 2008), which speaks to its
importance for social interaction. An anger expres-
sion causes others to react, including stopping
a transgressor from violating social norms (e.g.,
Averill, 1983), stopping their actions in general
(Winkielman, Berridge, & Wilbarger, 2005), caus-
ing others to engage in avoidance behaviors (Marsh,
Adams, & Kleck, 2005), showing fear (Dimberg &
Öhman, 1996; Esteves, Dimberg, & Öhman, 1994),
and/or triggering a submissive action—such as an
apology—which research shows quickly dissipates
the emotion of anger (Ohbuchi, Kameda, & Agarie,
1989). Thus, it may have been more advantageous
to “win” by showing anger than to surprise attack
individuals—particularly if most of them were
within the group, which historically meant that
most of them would be related to the aggressor. This
nonattack, in turn, would affect the transmission of
some of one’s genes, as those relatives share genes
who were not attacked with the angry individual
(Chagnon, 1988; Diamond, 1997).
Contempt would be important to signal as well;
indicating the relative status of individuals main-
tains the social hierarchy. Societies with strong
social hierarchies are more stable; and societies that
are more stable typically feature less internal vio-
lence (Pruitt & Kim, 2004).
Disgust too would be imperative to signal for
survival value; the ancient world did not feature
health department ratings of food found on the
ground, on trees and bushes, or animals running
away from our ancestors. Thus, to communicate
to members of your group the fact that a particular
item is not ingestible is a life or death matter; rancid

meat and poisonous berries were among the dangers
that were major threats to survival (even though
they are still with us today, but through science and
controlled farming practices they have been greatly
reduced). Even in the 20th century, food-borne ill-
nesses from potentially lethal typhoid fever dropped
in the population from 100 per 100,000 in the 1920s
to less than 2 per 100,000 in 1999 (Centers for
Disease Control and Prevention, 1999). We can only
imagine the prevalence of food-borne illnesses in the
ancient world. Given that starvation was a constant
survival threat, we can envision individuals attempt-
ing to eat anything looking marginally palatable
under such starvation conditions; thus, the ultimate
palatability would be important to signal to others
(Diamond, 1997).
Fear also is important to express; fear usually
indicates danger or signals to an angry conspecific
that one does not intend to fight (thus preventing
a fight). Additionally, it would seem apparent that
receiving advance notice from other group members
of an imminent threat such as a predator would
provide individual members of the group a better
chance of escaping such a threat. Furthermore, even
if the individual who stumbles onto a snake or other
predator does not survive the encounter, he or she
will have been able to signal to others either through
his or her facial and vocal signs of fear to flee before
the predator or other threat can take them as well.
Vocalizing fear allows the message to transmit over
a longer distance, and this communication may help
bring allies to battle the threat or to give those far-
ther away from the threat additional time to flee.
Happiness, or enjoyment, or other positive emo-
tions are also important to signal, as they indicate
approachability and nonthreating actions. Although
some have argued for many types of positive emo-
tion (e.g., Ekman, 2003), it may be the case that
there is no need to differentiate among those states,
as there is no survival value in differentiating plea-
sure from relief from enjoyment from amusement.
Regardless, an individual who shows a genuine sign
of a positive emotion (compared to when this same
individual signals an emotion with a posed smile) is
seen as more pleasant, more genuine, and generally
more positive (Frank et al., 1993). Furthermore,
given the homicidal nature of the ancient world,
Evolution and Nonverbal Communication
knowing someone was approachable and friendly
would be a true life or death judgment.
Sadness and/or distress would be important
to signal, as they demonstrate that an individual
needs attention. Such expressions are essential in
infancy, as an infant can only show distress non-
verbally, but the emotion lasts well into adulthood.
Adults who display expressions of distress are
able to generate help and sympathy from others
(Eisenberg et al., 1989).
Finally, surprise can show novelty and alert
others to the novel event to obtain assistance to help
comprehend it. Infants will react to a mother’s
surprise expression (Hiatt, Sotomayor, Sanchez,
Zambrana, & Knight, 1979), but the inherent neu-
trality of valence for the emotion of surprise often
affects its interpretation. For example, if one gets
presented with a box, and upon opening sees a gift
of $10,000, one would be surprised and happy;
if instead, upon opening one sees the decapitated
head of a cat, one would be surprised and disgusted.
There exists some controversy regarding the nature
of surprise (Reisenzein, Bördgen, Holtbernd, &
Matz, 2006), but when surprise is disambiguated as
either valenced positively or negatively, it is readily
recognized (Cheal & Rutherford, 2013).
The importance of being able to express emotions
in social life is demonstrated through examining
adults who do not have the ability to express through
their faces, such as those with Moebius syndrome.
Such individuals still feel and experience emotions,
but their social lives are often disrupted by their
inability to produce signs, thus signaling to others
their emotional states (Bogart & Matsumoto, 2010;
Coulson, O’Dwyer, Adams, & Croxson, 2004). A
controlled study examining a Rhesus monkey that
had its facial nerve surgically severed, then returned
to a social group, showed that the monkey lost its
place in the social hierarchy and had more aggression
turned toward it (Izard, 1977). Similar social impair-
ments are noted for those suffering from autism or
other disorders that disrupt the expression or per-
ception of emotional cues in either the face or voice
(Baron-Cohen, Wheelwright, Hill, Raste, & Plumb,
2001; Heaton et al., 2012). Thus, in an ancient envi-
ronment, the inability to process such information
may lead to slowed movements, reactions, and thus
55
Frank and Shaw
potentially lethal outcomes. Regardless, the ability
to sign, signal, and use symbols when appropriate,
emotions through face, body, and voice seems criti-
cal to survival, not only through identifying threats
but also through smoothing social interaction, estab-
lishing hierarchies, calling for aid, and so forth. The
evidence—particularly in light of perceptual centers
in the brain for these expressions and the develop-
mental progression among children and among the
blind—strongly suggests that they are part of our
wiring, thus instrumental to our survival as species.
Identity
Given the work showing how lethal our fellow
human beings were in the ancient world, with
more than 25% of male deaths due to homicide
(e.g., LeBlanc, 2003), it becomes clear that recogniz-
ing kin, and by extension, individual identity, could
be a life or death judgment. There are life and death
parallels in current animal behavior; for example,
troops of male chimpanzees will patrol the boundar-
ies of their territory, looking to expand it by assault-
ing or killing any males belonging to neighboring
groups they come across and by stealing females to
impregnate (Goodall, 2000; Wrangham & Peterson,
1996). Darwin himself argued for the selection
pressure for humans to protect against others or
other groups by banding together, as this would
confer an evolutionary advantage:
A tribe including many members who,
from possessing a high degree the spirit
of patriotism, fidelity, obedience, cour-
age, and sympathy, were always ready to
aid one another, and to sacrifice them-
selves for the common good, would be
victorious over most other tribes, and
this would be natural selection.
(Darwin, 1872/1998, p. 132)
Therefore, being able to distinguish “us” from
“them” would be an important capability. In fact,
two well-established lines of research speak to
this issue: first, is work on group behavior, and
second—related to the first—is person recognition.
Group behavior.  One of the most replicated find-
ings in all of social science is the finding that once
56
individuals are placed in groups, almost regardless
of the placement criteria, they will view members of
their own group—the ingroup—as being better than
members of other groups—the outgroup (Tajfel &
Turner, 1986). Some have argued that group mem-
bership is a driving force in the evolution of our
brains (Byrne & Whiten, 1988). Pursuant to this
idea is the male warrior hypothesis (Van Vugt, De
Cremer, & Janssen, 2007). This hypothesis argues
that evolution has keyed the male brain, more
so than the female brain, to engage in intergroup
aggression, to derogate or infrahumanize outgroups,
to form coalitions in the presence of outgroup
threats or competition, to demonstrate more loyalty
to the group, to be more willing to accept violence to
solve political struggles, to prefer social hierarchies,
and generally to define themselves more by the
groups they belong to rather than individual char-
acteristics (Harvey, White, Hood, & Sherif, 1961;
Van Vugt, 2011). This group behavior is a means
for defense but is also a means for offense to attack
and defeat potential rivals; many of whom would
try to kill the individual if they too had a numerical
or tactical advantage. As mentioned previously, this
behavior is observed in chimpanzees (Wrangham &
Peterson, 1996). Interestingly, research has shown
that in the presence of allies, men are more likely
to down-estimate the formidableness of potential
opponents (Fessler & Holbrook, 2013). Thus, this
coalitional, cooperative group behavior can be a life
or death game, but it is a game played more heavily
by males than females. It strongly argues that pro-
pensity for banding into groups was evolutionarily
selected for, but this group formation would not be
possible without a corresponding ability to recog-
nize who is in your group and who is not.
Person recognition.  We seem particularly attuned
to recognizing people. The fact that we put pictures
of our faces on identification cards, and not our
hands, feet, or other body parts attests to this con-
jecture. There is a robust literature showing that
there are sections in the human brain that respond
directly to faces (Hadjikhani, Kveraga, Naik, &
Ahlfors, 2009; Ro, Russell, & Lavie, 2001; also see
Chapter 9, this handbook). We not only can rec-
ognize identity but we can also recognize group

affiliation; for example, Rule, Garrett, and Ambady
(2010) showed that Mormons can recognize from
still photos whether the individual depicted is a fel-
low Mormon or not, at rates greater than chance.
However, other research shows that we are also
quite good at recognizing individuals from their
voices (Hollien, Majewski, & Doherty, 1982; also
see Chapter 11, this handbook) and also from their
odors (see Chapter 14, this handbook). We are also
quite reliable in recognizing not only a person’s
gender from simply observing him or her walk-
ing via point-light displays (Kozlowski & Cutting,
1977) but also if we know him or her (Cutting &
Kozlowski, 1977). This person recognition can be
taken one step further, as it also appears that we are
able to recognize kin from nonkin (D. Lieberman,
Tooby, & Cosmides, 2003). Identifying kin from
nonkin means recognizing individuals who look
and smell like us. The evolutionary advantage of
banding together with those who share our genes
is apparent—even if we do not survive to repro-
duction, the survival of the kin-group ensures that
at least some of our genes will be passed on. This
ability to recognize faces is surprisingly robust; par-
ticipants playing a Prisoner’s Dilemma game against
a researcher-generated opponent are more likely
to choose the cooperative move when they see a
photo of the other individual that has been morphed
with a photo of the participant him- or herself, thus
making the opponent look more similar to the par-
ticipant (DeBruine, 2002). Taken together, we can
argue that we seem biologically wired to recognize
individuals, to recognize when they are kin, and to
band together with them to defeat nonkin competi-
tors (see Chapter 9, this handbook). Furthermore,
as mentioned previously, this phenomenon seems
particularly robust in males (Van Vugt, 2011).
Other nonverbal clues relevant to survival.  We
may also make an evolutionary argument that we
can identify individuals who are formidable threats
and, thus, avoid them (e.g., Fessler, Holbrook, &
Snyder, 2012). There are a number of factors associ-
ated with this phenomenon, but they involve esti-
mating the individual’s formidableness from his or
her body size, voice, and even face, facial hair, and
gazing behaviors (Puts, 2010). For example, people
Evolution and Nonverbal Communication
can estimate body size from facial appearance
(see Chapter 9, this handbook) as well as body
strength (Sell et al., 2009). Dominance as conveyed
from static facial clues was associated with the per-
ception of the individual as being more threatening
(Oosterhof & Todorov, 2008), and this character-
istic can be assessed reliably with images shown
at tachistoscopic speeds of less than 40 ms (Carré,
McCormick, & Mondloch, 2009). This suggests
that dominance was a very important characteristic
to note in individuals, and likely throughout our
evolutionary past, as the more dominant-appearing
faces and body sizes are associated with being for-
midable potential opponents and thus to be avoided
in confrontations. Thus, physical size and estimated
strength from static nonverbal cues in the face, body,
and voice could dissuade one from engaging in
conflict, much like the facial expression of anger
mentioned earlier.
At the other end, research shows that the non-
verbal behavior displayed through one’s gait can
signal to others that one is not only nonformidable
but is also a potential target of aggression. Individu-
als imprisoned for mugging others agreed on which
individuals, whose gait was shown on video, they
would choose to victimize (Grayson & Stein, 1981).
Moreover, Book, Costello, and Camilleri (2013)
showed that individuals who scored higher on psy-
chopathy were able to recognize, from gait alone,
which individuals had been previously victimized
by an assault. Thus, one’s body actions seem to draw
the attention of the more violent predator-like indi-
viduals in society, thus increasing the chances that
the potential victim will either not survive to the age
of reproduction, and if they do, they will likely not
be vied for as a reproductive partner. We turn next
to this evolution issue.
REPRODUCTIVE FITNESS
Having successfully navigated the threats from preda-
tors, tainted food, disease, and fellow humans (par-
ticularly males), an individual is now in position to
reproduce and, thus, pass on his or her genes to the
next generation. The likelihood of that happening,
or the frequency of it happening, is also influenced
by nonverbal cues. In fact, the nonverbal signs are
57
Frank and Shaw
what comprise attractiveness—but attractiveness is a
catch-all category for signs or traits that are desirable,
or to rephrase, those highest in reproductive fitness.
Within the category of attractiveness, we sub-
divide into those characteristics associated with
perceptions of health, fertility, sexual receptivity or
availability, and dominance. At least in humans, it
appears that these characteristics are more gender-
linked, as fertility seems to apply exclusively to
females, and dominance exclusively to males. Health
applies equally to both genders, but signals of avail-
ability also seem to favor females over males, as
females are the limiting sex. Finally, we note that
the distinctions between signs, signals, and symbols
are a bit more muddled in reproductive fitness, as
many of the cues associated with the signs, signals,
and symbols have only been articulated recently
(see Chapters 9 and 14, this handbook). The reader
will notice that much of the work, but not all, on the
signs, signals, and symbols of reproductive fitness
demonstrate that individuals vary in their desire for
particular features or traits, suggesting that those
features and traits are in fact signals, which result
from signs of reproductive fitness. Subsequently,
research in this area inherently commits the fallacy
of the converse (i.e., affirming the consequent);
therefore, the propositions that follow provide evi-
dence that such possibilities are valid but should be
interpreted with caution. We note instances when
enough research evidence exists to suggest a valid
relationship among signs, signals, and symbols.
Health
The state of a potential reproductive partner’s health
is instrumental in the survival of his or her joint
offspring, as healthy individuals produce offspring
healthy enough to stand the greatest chance of
survival. Thus, any sign that communicates, or
signals, the health status of an individual would
seem desirable for mating.
Face—Signs, signals, and symbols.  The past few
years have seen a number of advances in identify-
ing and understanding the signs of genetic health in
the human face. The best review to date is found in
Chapter 9, this handbook, and the reader is directed
there. In addition to the signs identified by Re and
58
Rule (Chapter 9, this handbook), facial symmetry,
specifically, has been noted as a sign of health as well
as genetic health, but that is discussed later when
referring to the overall health of bodies.
In contrast to what is implied by symmetry, high
levels of testosterone have been shown to affect the
immune system negatively, and thus male faces
with characteristically high testosterone markers
(heavy brow ridges, stronger jaw line) may indicate
a slightly compromised immune system (Muehlen-
bein & Bribiescas, 2005). Exactly how such a
tradeoff might occur is outlined by the immuno-
competence handicap hypothesis, which posits that
hormones responsible for sexual differentiation may
also compromise one’s immune system (Folstad &
Karter, 1992). In other words, as men display
more masculinized features, they take a greater
risk of harming their own health. Therefore, only
the most physically capable men can afford poten-
tially compromising their own health to display
more masculinized features. Given this hypothesis,
sexual selection models in which “good genes” are
selected for predict that women will prefer hyper-
masculinized traits (Scheib, Gangestad, & Thornhill,
1999). Therefore, facial features such as heavy brow
ridges and strong jaw lines are signs of compro-
mised immune systems that females may interpret
as signaling strength and facilitate their decision to
mate. In fact, women, under certain circumstances,
will prefer males with such an appearance despite
the health implications because it suggests the male
is of high quality if he can overcome that high tes-
tosterone handicap (see Chapter 9, this handbook;
also see Zahavi, 1975). The “good gene” hypothesis
posits that physical characteristics are perceived as
attractive to the extent “healthy” individuals can dis-
play these characteristics, but “unhealthy” individu-
als cannot, or at least not to the same extent (e.g.,
Andersson, 1994; Gangestad & Simpson, 2000;
Thornhill & Gangestad, 1999). Therefore, men
with high levels of testosterone who have survived
to reproductive age may signal greater fitness than
the men with lower levels of testosterone. Given the
good gene and handicap hypotheses, it is difficult
to determine exactly what masculinized facial signs
signal, but it is clear that the information signaled
from the signs is used in determining mate quality.

There are other facial appearance variables that
affect perceptions and, hence, reproductive fitness.
To quickly summarize (see Chapter 9, this hand-
book, for a more thorough review), individuals with
more homogenous skin texture and distribution of
color in their faces are seen as healthier and more
attractive (e.g., Fink & Matts, 2008). Moreover, Re
and Rule (see Chapter 9, this handbook) have identi-
fied the reason why faces that hue (i.e., sign) toward
red and yellow and that are lighter are also seen as
more desirable. Their basic premise is that redness
signals cardiovascular health, yellow signals con-
sumption of healthy carotenoid bearing vegetables
and fruits, and lightness signals the allowance of
more ultra-violet rays through to enhance produc-
tion of Vitamin D, all of which help to ensure health-
ier offspring. Moreover, they have discussed the
role of adiposity, or subcutaneous fat, in a face (i.e.,
signs) and have reported that moderate levels were
associated with higher levels of health, whereas low
fat or too much fat were seen as less attractive and
were associated with poorer health outcomes (i.e.,
signal; Coetzee, Perrett, & Stephen, 2009). These
are considered signals because they convey informa-
tion regarding the health of the individual, and in
this case, it appears that the relationship between the
sign (i.e., facial fat) and signal (i.e., healthy) is valid.
It also goes without saying that individuals adorn
themselves with objects to enhance beauty, and
these are often culturally specific (lip plates, tattoos,
etc.; again, see Chapter 9, this handbook). Interest-
ingly, males will groom or shave facial hair to look
more attractive, whereas females will apply makeup
to appear more attractive. The pattern of makeup
is also not random: Eye liner, mascara, and eye
shadow function to highlight the eyes (as larger eyes
are seen as more feminine), whereas the use of foun-
dation, or cover-up, is used to generate the appear-
ance of homogeneous skin tone, texture, and color
that were noted earlier as being attractive. The use
of blush on the cheeks of Caucasian women mimics
sexual arousal (McKinney & Sprecher, 1991). Facial
adornments serve as symbolic representations of
health, whether valid or not.
Body—Signs, signals, and symbols.  One of the
most researched areas in health is the symmetry of
Evolution and Nonverbal Communication
the body, including the face. Facial and body
symmetry describes the extent to which, when com-
pared along the sagittal plane, the right hemisphere
of one’s face, or body, matches the left hemisphere of
one’s face, or body. Conversely, fluctuating asymme-
try is defined as the random deviations from symme-
try in bilateral traits (e.g., eyes; Van Valen, 1962; also
see Chapter 9, this handbook). The specific body
markers used to determine symmetry or fluctuating
asymmetry vary from study to study; however, typi-
cal measurements include ear length and breadth,
as well as elbow, wrist, hand, knee, ankle, foot, and
finger length (see Rhodes & Simmons, 2007, for a
review), from which a composite measure is derived
(e.g., Gangestad, Bennett, & Thornhill, 2001).
Unfortunately, fluctuating asymmetry is much more
complicated to determine, and does not exhibit a
strong negative correlation with symmetry as one
might expect (Rhodes & Simmons, 2007). One rea-
son fluctuating asymmetry is difficult to determine
is that the extent of the differences between the right
side and left side is similar to what one might expect
from measurement error (Palmer & Strobeck, 2003).
An additional problem with measuring fluctuating
asymmetry is that there are some types of asym-
metry that are a normal part of human development
(e.g., Dufour & Weatherhead, 1996; Gangestad
et al., 2001). For example, it is typical to have one
foot slightly larger than the other foot.
Generally, the more symmetry one displays, the
more attractive he or she is perceived by others (i.e.,
signal; e.g., Grammer & Thornhill, 1994; Watson &
Thornhill, 1994). This effect tends to be stronger
for faces (effect size = .30) than for the body (effect
size = .14), possibly because facial features are more
closely aligned allowing for comparisons of features
to be made more easily than body parts that are fur-
ther apart (Rhodes & Simmons, 2007; also see Wade,
2010, for a review). Additionally, this effect dissipates
when one half of an individual’s face is reflected to
generate the other half (this is referred to as chimeric
faces in Chapter 9, this handbook; see also Kowner,
1996; Langlois, Roggman, & Musselman, 1994).
Ratings of attractiveness for chimeric faces do not
reach the same levels as naturally occurring sym-
metrical faces. Findings regarding fluctuating asym-
metry have been mixed. Given some of the issues
59
Frank and Shaw
surrounding the measurement of fluctuating asym-
metry, Rhodes and Simmons (2007) were unable to
determine conclusively, through meta-analytic
techniques, the relationship between fluctuating
asymmetry and ratings of attractiveness.
One possible evolutionary explanation for why
symmetrical faces, and bodies, are seen as attractive
is that they could be perceived as indicators of over-
all genetic health (i.e., a sign). In fact, Zaidel, Aarde,
and Baig (2005) found that assessments of targets’
health were influenced by their facial symmetry;
in other words, the more symmetrical the target’s
face, the healthier he or she was perceived by
observers. Research has demonstrated that depar-
tures from facial symmetry indicate lower genetic
health and greater susceptibility to perturbations
during development (e.g., Gangestad, Thornhill, &
Yeo, 1994; Livshits & Kobyliansky, 1989; Møller &
Alatalo, 1999). These findings suggest that facial
symmetry is in fact a valid sign of genetic health.
Therefore, despite researchers’ inability to measure
fluctuating asymmetry, it appears that people are
able to detect the signals of fluctuating asymmetry to
make judgments of attractiveness.
Movements in the body, particularly from
females, that are seen as “natural” are rated as more
attractive (Brown, Cash, & Noles, 1986). Male dance
movements can also be a signal as to mate “quality”
and health, and females agree on which types of
movements indicate such (Fink, Weege, Manning, &
Trivers, 2014). These findings suggest that dancing
is a valid sign of health, but further research needs to
determine the association between dancing as a valid
sign and signal of health. Conversely, there is strong
evidence suggesting that disruption in movements
(i.e., sign) signals injury or illness. For example,
research on Parkinson’s disease suggests that gait is
controlled by two areas of the brain associated with
emotional and volitional movement—the basal gan-
glia, which also controls emotion, as well as higher
cortical areas that control volitional movement (e.g.,
Lewis, Byblow, & Walt, 2000; Thaut, McIntosh,
Prassas, & Rice, 1992)—and that the effects of the
disease produce measureable changes in movement.
Specifically, depressed Parkinson’s patients show
decreased stride length, decreased coordination of
arms in movement, and slower velocity (Lemke,
60
Wendorff, Mieth, Buhl, & Linnemann, 2000).
Parkinson’s patients also show reduced stride, but
they can be trained to expand the stride by employ-
ing various visual clues, again suggesting some
volitional control (Lewis et al., 2000), although it is
these disruptions in movement that are the hallmark
of Parkinson’s disease. Of course other disruptions
such as limping signal injury, which by definition
is a reflection of current, but possibly not future,
health. The current state of research in the area of
body movement does suggest that body movement
is a valid sign of genetic health that signals health
status to others; however, more research needs to
be conducted to elucidate the relationship between
body movement as a valid sign of health.
As noted elsewhere in this chapter, odor is a
nonverbal sign, and the use of perfumes and other
scents to cover more noxious odors is one way to
symbolically represent health (see Chapter 14, this
handbook). Often foul odors indicate infection,
and individuals with such odors will be ostracized.
For example, women in sub-Sahara Africa often
suffer from obstetric fistulas, where tears in the
vagina during childbirth push through to the blad-
der or rectum allowing the waste products to leak
into the vagina (often young, small child brides);
this causes intensely foul odors to emanate from
these women, who are subsequently socially ostra-
cized (Kimani, Ogutu, & Kibe, 2014). Therefore,
perfume and other scents serve as a symbol of
health because they are used to signal health to
others.
Clothing is another nonverbal cue, and despite
their practical origins (protection against cold and
weather), styles and colors are chosen specifically
to symbolically represent the wearer’s personal-
ity or social status, which may indirectly link back
to health. Individuals can also choose clothes to
conceal health-related issues, such as rashes, acne,
scars, loss of hair, as well as infections of various
sorts. Females will often wear undergarments
that help shape the body toward the most attrac-
tive ratios of hips to waist, which beside signaling
health, also signal fertility (Singh, 1994). All of these
trappings serve a symbolic function, as they are
external to the individual but are intended to signal
the health of the individual.
 Evolution and Nonverbal Communication

Fertility shift among women is the widening of their hips due

Fertility is closely associated with health, but it is to the accumulation of gynoid fat on the thighs, legs,
a special subcategory that applies more to women buttocks, and so forth (Kenrick, 1987; Singh, 1993,
than men. That is, the healthier female is pre- 1995). Given that the widths of one’s waist and hips
sumed to be a more fertile female. However, a line are influenced by the concentration of sex hormones
of research has explored fertility per se, and this is (Björntorp, 1988; Singh & Young, 1995), measure-
where we turn. ments of one’s waist and hips are valid signs of
fertility (e.g., reproductive capabilities).
Face—Signs, signals, and symbols.  The extent
In fact, WHR correlates with health outcomes
to which a female face looks feminine, the more
(i.e., sign) as well as fertility perceptions (i.e., sig-
attractive she is perceived (see Chapter 9, this hand-
nal). For instance, among women, WHR greater
book). A face appears more feminine—larger eyes,
than .85 is related to increased health risk factors
fuller lips, proportionally smaller chin and nose,
such as increased heart disease (Larsson, 1985) and
and higher cheekbones—under higher levels of the
Type II diabetes (Barbieri, 1990; Björntorp, 1988),
primary female hormone estrogen (see Chapter 9,
because women with such a ratio exhibit greater
this handbook); and higher levels of estrogen have
concentration of visceral adipose tissue (fat depos-
been associated with higher rates of conception
its; Lemieux, Prud’homme, Tremblay, Bouchard, &
(Lipson & Ellison, 1996). This nonverbal sign,
Després, 1996). These risk factors also affect fer-
therefore, may be a sign of fertility and subse-
tility (Wass, Waldenström, Rössner, & Hellberg,
quently makes such features valid signals of fertility.
1997; Zaadstra et al., 1993). Therefore, one should
In fact, women with more feminine faces—when
be able to use WHR as an indicator of fitness and
other variables were controlled, such as use of
make a decision regarding mating based, in part,
birth control—did in fact produce more offspring
on WHR. Singh (1993, 1994, 1995, 2002; Singh &
(Pflüger, Oberzaucher, Katina, Holzleitner, &
Luis, 1995; Singh & Young, 1995) demonstrated in
Grammer, 2012). A parallel line of research argued
a series of studies the importance of WHR on one’s
that adult females with more baby-faced propor-
physical attractiveness. He concluded that although
tions were also seen as more attractive, because the
body weight of ideal standards of beautiful women
appearance of youth suggested fertility (Berry &
has changed since the 1930s (as indicated by Miss
Zebrowitz-MacArthur, 1988), although there is
America contestants and Playboy playmates), pref-
likely a curvilinear relationship between youthful-
erence for a specific WHR has varied little. Addi-
ness, or age, and fertility (i.e., too young or too old
tionally, Singh (1993) found that men as old as 85
unable to bear children).
years of age found women with smaller WHRs to
It does seem that these signs of fertility signal
be more attractive and believed them to be more
to males that females are fertile. Female rhesus
reproductively capable than women with higher
macaques signal their fertility with a more lumines-
WHR. Men in America and European countries
cent face (i.e., sign), and males familiar with these
view a WHR of a 0.7, regardless of overall weight,
females can identify when they are about to become
as more attractive (Singh & Young, 1995), presum-
fertile (Higham et al., 2011). Younger women were
ably because it is a valid signal of fertility. In fact,
also judged more attractive and fertile from their
Singh and Young (1995) found that men believed
faces (and also their voices) than older women, and
that women with a WHR that was closer to 0.7 were
again femininity seemed to drive the process (Röder,
more capable of reproduction than women with a
Fink, Feinberg, & Neave, 2013).
WHR deviating further from 0.7.
Body—Signs, signals, and symbols.  One of the When a female is more fertile, she will produce
most researched body signs of fertility is a woman’s odors that will trigger more sexual-related thoughts
waist-to-hip ratio (WHR). During puberty, men and in men, will cause men to perceive the woman as
women undergo hormonal, physiological, and mor- more aroused, and will cause men to be more likely to
phological changes. One apparent morphological mimic the female’s behaviors (Miller & Maner, 2011).

61
Frank and Shaw
Given that there exist signs and signals of fertil-
ity, individuals may use external signs (i.e., sym-
bols) to convey fertility. However, in such instances,
it is likely that individuals are signaling sexual
receptivity, or availability, and therefore this topic is
discussed next.
Sexual Receptivity or Availability
Unlike normal adult males who are always sexually
receptive and fertile, females have the ability to sepa-
rate sexual receptiveness from their fertility due to
their ovulatory cycle. Although females are capable
of engaging in sexual activities at any point during
her menstrual cycle, females are most sexually recep-
tive, or available for sex, during ovulation. At least in
humans, and most mammals, ovulation occurs mid-
way through the menstrual cycle (typically around
Days 10–18 of a normal 28-day menstrual cycle).
During this time, the release of hormones (e.g.,
luteinizing hormone, follicle-stimulating hormone)
induces ovulation, or the release of an ovum (i.e.,
egg) to be fertilized. Although somewhat debated,
there is strong evidence suggesting that during ovula-
tion, females are most sexually receptive (e.g., Pills-
worth, Haselton, & Buss, 2004; Roberts et al., 2004).
Exactly what signs fluctuate across the menstrual
cycle to produce fertility signals have not been clearly
identified, but a few possibilities are highlighted here.
Face—Signs, signals, and symbols.  Within the
face of humans, research examining specific signs
that indicate availability during ovulation has not
been well researched. However, during courtship, a
woman signals interest in a romantic partner with
her face through increased blood flow to the face
resulting in blush on the cheeks and a swelling
of the lips (which makeup mimics; McKinney &
Sprecher, 1991), and her pupils dilate—which stud-
ies show makes a woman look more attractive (Hess,
1965). Note these are all involuntary actions and,
therefore, are signs and valid signals of sexual recep-
tivity. Although never directly assessed, it is possible
that increased eye contact is also another sign of
sexual receptivity. Research in the area of immediacy
behaviors demonstrates that increased eye contact
increases liking (e.g., Argyle, 1988). Given that
human females have control over when they are
62
perceived as sexually receptive, irrespective of where
they are in the menstrual cycle, they may exploit
signs of receptivity to their advantage to secure mat-
ing partners. In the Renaissance era, women in Italy
also used belladonna, a low grade poison, to artifi-
cially dilate their pupils, which also replicated the
pupillary reaction when someone is sexually inter-
ested, and which to observers makes the face look
more attractive (Hess, 1965).
Body—Signs, signals, and symbols.  The literature
is replete with examples from the animal kingdom
of valid signs and signals displayed in the body com-
municating sexual receptivity. Primates often show
red (blood engorged) areas in their chests or but-
tocks to signal sexual receptivity (Dixson, 1983).
One of the most well-researched sign of sexual
receptivity is lordosis, which is displayed by most
rodents. Lordosis is a body posture, controlled by
neural circuits and hormones, that is associated with
sexual receptivity because it facilitates copulation
(Pfaff & Sakuma, 1979).
Among humans, women will stand closer,
stand with a more direction face-to-face orienta-
tion, will lean in more toward the person of interest,
and will include more casual touch (see Knapp &
Hall, 2010, for a review) to indicate receptivity to a
romantic partner. It is not clear whether these actions
change across the menstrual cycle or if they are vol-
untary or involuntary; regardless, they are valid sig-
nals of receptivity, and future research would benefit
from examining how they change across the men-
strual cycle. Likewise, a woman’s production of pher-
omones in general can suggest sexual receptivity (see
Chapter 14, this handbook). Moreover, men seem
to be able to detect ovulation through not just odors
but also through the women’s voices and behaviors
(reviewed by Haselton & Gildersleeve, 2011).
In terms of symbols used to signal sexual recep-
tivity, it has been demonstrated that during ovula-
tion, women are more likely to self-ornament by
grooming more thoroughly and by wearing more
attractive clothing (Haselton, Mortezaie, Pillsworth,
Bleske-Rechek, & Frederick, 2007). Women are also
more likely to wear revealing clothing (Durante,
Li, & Haselton, 2008), to use more appearance-
related beauty products (Durante, Griskevicius,

Hill, Perilloux, & Li, 2011; Saad & Stenstrom,
2012), and to spend more time applying these
products (Guéguen, 2012). Similar to their primate
counterparts, during ovulation, women are likely to
display sexual receptivity with signs of red—women
are more likely to wear red or pink attire and
makeup during ovulation (Beall & Tracy, 2013),
and men are more attracted to them when they do
(Elliot & Niesta, 2008). Additionally, women, not
controlling for where in their menstrual cycle they
were, who were interested in casual sexual relation-
ships chose to display the color red in their web
profile pages (Elliot & Pazda, 2012).
Other—Signs, signals, and symbols.  Other signs,
signals, and symbols of sexual receptivity may
include voice and affiliation. Pipitone and Gallup
(2008) found that women’s voices varied across
their menstrual cycle, and men preferred women’s
voices when they were ovulating. S. E. Hill and Buss
(2008) demonstrated that men found women less
desirable when in the presence of males than when
in the presence of females or alone; however, unpub-
lished data from the second author have revealed
no effect of the presences of others based on gender,
but instead desirability is based on attractiveness.
Regardless, it is possible that the presence of oth-
ers (i.e., symbol) communicates receptivity. Males
or females in the presence of opposite sex others
may be viewed as less sexually receptive to others
because they are sexually receptive to those
immediately present.
Dominance
Whereas fertility and sexual receptivity were
almost exclusively about female nonverbal cues,
dominance is much more about male nonverbal
cues. The key driver of the dominance nonverbal
cues is the hormone testosterone, which has effects
on facial appearance, body appearance, and
behavioral actions.
Testosterone emerges during the two critical
periods of neural and physical development (i.e.,
prenatal, pubertal). Although prenatal development
ultimately sets one’s hormonal, neurological, and
physiological development on a trajectory, puber-
tal development is necessary to complete sexual
Evolution and Nonverbal Communication
development. On average, puberty begins at the age
of 10–11 for females and 11–12 for males, and is
complete by the age of 17 for both sexes. Puberty
for both sexes involves a number of hormonal
changes that result in morphological shifts, such as
the development of secondary sex characteristics
(e.g., development of breasts, body hair). Similar to
prenatal sexual differentiation, pubertal sexual dif-
ferentiation can be attributed to differences in the
circulation and sensitivity of sex hormones,
such as androgens and estrogens (Abbassi, 1998;
MacGillivray, Morishima, Conte, Grumbach, &
Smith, 1998). By sexual maturity, signs displayed by
the face and body become sexually dimorphic and
signal dominance and submissiveness. Some of the
more notable changes that both males and females
undergo are changes to their skeletal musculature
associated with the body and face, which begin to
masculinize.
Face—Signs, signals, and symbols.  Weston,
Friday, and Liò (2007) found that by adulthood,
the face is sexual dimorphic, with males having
shorter anterior upper faces than females (i.e., fore-
head). Weston et al. found that the structure of the
face seems to be independent of body size, and the
growth trajectories of the length and width of the
face for males and females seem to diverge around
puberty. The result is that males have faces wider
than they are tall, in comparison to females (again,
see Chapter 9, this handbook). This sex difference in
facial ratio (i.e., sign) does not emerge until puberty,
when males experience a higher concentration of
testosterone than females. In adults, it has been
demonstrated that salivary testosterone concentra-
tion was positively correlated with facial masculin-
ity (Penton-Voak & Chen, 2004), suggesting that
androgens are responsible for masculinizing faces.
These findings suggest that facial ratio is a valid sign
of testosterone exposure.
Although facial ratio has not been studied in this
capacity, Weston et al. (2007) have argued that one
reason for this differentiation is due to sexual selec-
tion pressures, both inter- and intrasexual. Indeed,
as stated earlier, many studies demonstrate that the
greater the perceived masculinity of a male face (but
not necessarily a more masculinized facial ratio), the
63
Frank and Shaw
more attractive it appears to women (e.g., Rhodes,
2006; Rhodes et al., 2007). Additionally, masculin-
ized faces may contribute to intrasexual selection
processes. Rival males perceive more masculinized
faces as more socially and physically dominant
(Mazur & Mueller, 1996; Swaddle & Reierson,
2002), potentially deterring male rivals from com-
peting with males that possess greater genetic fitness.
It is possible that facial structure (i.e., sign) is a
valid signal of dominance. Research has found that
males with more masculinized faces are more likely
to “cheat” in Prisoner’s Dilemma games, and observ-
ers are able to accurately identify these “cheaters”
(Verplaetse, Vanneste, & Braeckman, 2007). It has
been demonstrated that facial ratio, as measured
with similar procedures to Weston et al. (2007),
predicts aggressive behavior (Carré & McCormick,
2008), and observers’ ratings of predicted aggres-
sion is correlated with that same facial ratio (Carré
et al., 2009). Finally, within the context of eco-
nomic and social dilemma games, wider male faces
are related to how individuals interact with their
counterpart. Specifically, Stirrat and Perrett (2010)
found that facial ratio, as measured using the same
procedures as Carré and McCormick (2008), pre-
dicted men’s likelihood to reciprocate in economic
games. Additionally, males with wider faces were
more likely to engage in unethical behavior such as
exploiting their partner (Stirrat & Perrett, 2010) as
well as using deception and cheating (Haselhuhn &
Wong, 2012). These findings, however, may be
moderated by whether group affiliation is salient.
Stirrat and Perrett (2012) found that men with more
masculinized facial ratios were more likely to use
cooperative strategies with ingroup members than
outgroup members, when group membership was
made salient.
Although there are a number of studies that
suggest the importance of facial structure in some
capacity, however, there still is a dearth of infor-
mation. More recently, several studies have been
published that have challenged whether there exists
a sexual dimorphism in facial ratios (e.g., Lefevre
et al., 2012; Özener, 2012). Additionally, the rela-
tionship between facial ratios and aggression has
been challenged, suggesting that when controlling
for body size, this relationship dissipates (Deaner,
64
Goetz, Shattuck, & Schnotala, 2012). It is possible
that other allometric properties contribute to peo-
ple’s perceptions of social and physical dominance.
Beards and other adornments can exacerbate
the appearance associated with dominance. For
example, beards enhance the jaw line, moustaches
can create the appearance of a downturned lip
(thus not appearing to smile, which those males
who are lower in testosterone, as well as women in
general, smile more; Dabbs, 1997; Deutsch, 1990).
Increasingly common place among prisoners are
teardrop tattoos on the face, just beneath the eye
of one’s cheek. This tattoo has become a symbolic
representation of the number of people one has
killed. The placement of such a tattoo on the face
indicates that wearers hope to signal dominance to
others very clearly.
Body—Signs, signals, and symbols.  One of the
first things one can notice is the size of an individual
(i.e., sign). Earlier we discussed the concept of the
formidable male—and that males who are more for-
midable are likely to be avoided in confrontations.
Females have a preference for larger males, or at
least males who are larger than them (Pawlowski,
Dunbar, & Lipowicz, 2000). The “male taller” social
norm is embedded in almost all cultures, and it is
reflected in the fact that an adult human male is typ-
ically 10% larger than a human female, with a higher
ratio of muscle to fat than a female (Fairbairn,
1997). In fact, women who live in more dangerous
areas, compared to those who live in safer areas,
show a significantly stronger preference for more
formidable males (Snyder et al., 2011). These find-
ings suggest that body size signals dominance.
A second body sign is very subtle, and it is found
in the hands–digit ratio. Digit ratio refers to the
relative length of digits, or fingers, measured from
the superior tip of the finger to the midpoint of the
inferior crease of the finger. The ratio between one’s
second digit (2D; index finger) and fourth digit (4D;
ring finger) is of particular interest to researchers
(see Manning, 2002, for a review). It has been dem-
onstrated that the 2D:4D is a proxy for sex hormone
levels during critical periods of neural development
(Manning, Scutt, Wilson, & Lewis-Jones, 1998). To
date, there has been strong evidence suggesting that

prenatal androgen exposure is partly due to recep-
tors’ sensitivity to androgens (the class of hormones
that testosterone falls under). More evidence impli-
cating the importance of androgen receptor sensitiv-
ity in 2D:4D variation can be found from examining
the digit ratio of genetic males (XY) with androgen
insensitivity syndrome, a condition that results in
dysfunctional androgen receptors but with normal
functioning hormonal circulation systems; such
individuals display feminized digit ratios (Beren-
baum, Bryk, Nowak, Quigley, & Moffat, 2009).
In fact, 2D:4D is correlated with verbal aggression
(Shaw, Kotowski, Boster, & Levine, 2012) and
overall aggression in males (Bailey & Hurd, 2005;
Benderlioglu & Nelson, 2004). As an aside, there
also appears to be evidence that 2D:4D may be a
valid sign of health (Fink, Manning, & Neave, 2006;
Manning & Bundred, 2001) and fertility among
males (Manning et al., 1998).
Additionally, more dominant men will touch
others more, and will invade personal space more,
than less dominant men (Knapp & Hall, 2010). The
importance of body size as a signal for dominance
can be seen in the importance that men place in
increasing their muscle mass to appear larger. Men
will typically workout to develop the musculature
in their bodies as well to further enhance their
dominant appearance (Wienke, 1998). As with
fertility and health, individuals can wear cloth-
ing that enhances their dominance. For example,
more aggressive sports teams wear black uniforms
(Frank & Gilovich, 1988). It has also been sug-
gested that the color red (i.e., sign) signals aggres-
siveness to others and affects performance (R. A.
Hill & Barton, 2005). R. A. Hill and Barton (2005)
found that among combat sports in the 2004 Olym-
pic Games, contestants wearing red won more often
than their opponents wearing blue, suggesting that
the color red is a signal of dominance and aggres-
sion to others who are then more willing to succeed.
Other items on clothing, or even body art/tattoos,
may include aggressive logos of motorcycles, skulls,
military style items (guns, tanks, knives), and so
forth. However, men may wear shoe lifts or adopt
hair styles that enhance their height; in other cul-
tures, individuals wear headdresses or other orna-
ments to enhance size. Epaulets on the shoulders
Evolution and Nonverbal Communication
enhance the perceived shoulder size and also
exaggerate the shoulders to suggest more upper
body strength.
CONCLUSION
The signs, signals, and symbols of nonverbal com-
munication cross a number of boundaries, from
the voluntary action to the involuntary reflex, and
with a number of points in between. Table 3.1 is
our hypothesized usage of these clues based on
their survival or reproductive fitness potential and
their sign, signal, or symbol status. This is not com-
prehensive, and it likely oversimplifies, but it is an
attempt to get the taxonomy rolling so others, with
future research programs, can fill in these blanks or
move a behavior from one category to another, and
so forth.
It is clear that these nonverbal behaviors are
useful to us now and were likely more useful to us
in the ancient environment (e.g., Tooby & Cosmides,
1990). Our modern world has taken some of these
and mimicked them (e.g., makeup in women to
mimic sexual arousal) or enhanced them (shoes
with heels to enhance height and thus dominance,
shoulder epaulets to broaden shoulders), and this
is not accidental—it is capitalizing on these basic
human signs and signals recognized by other
humans. This is why we do not see blush applied
to the middle of a woman’s forehead, we do not
see jacket designs that deliberately narrow a man’s
shoulders, and so forth. Moreover, humans learn to
mimic the involuntary dynamic actions; we learn to
smile when we are not happy, to bluff with anger,
and to pose any and all emotional states even when
we are not experiencing them.
Nonverbal communication is essential to our sur-
vival as a species. It enabled us to better execute and
manage conflict, establish hierarchies, avoid danger-
ous foods, escape danger, call for aid, and so forth.
It also enabled us to identify the most desirable
individuals within our species, those most healthy,
fertile, dominant, and interested in us. We can argue
that verbal communication can accomplish many
of these goals; however, the fact that verbal com-
munication is entirely voluntary renders it suspect
as a clear definitive sign of our health, fertility, or
65
Frank and Shaw

TABLE 3.1

Hypothesized Evolutionary Role for Various Nonverbal Signs, Signals, and Symbols

Channel Characteristic Survival? Reproductive fitness? Sign? Signal? Symbol?

Face
Static features Yes Yes Yes: Identity Yes: Formidable No
Yes: Gender
Color in face No Yes Yes: Health Yes: Health Yes: Makeup
Yes: Overcome
handicap (male)
Yes: Fertility (female)
Yes: Sexual arousal
Emotion Yes Yes Yes: Basic emotions Yes: Imminent No
expression actions
Other expression No Yes No Yes: Language Yes: Mimic
(emblems) emotions
Yes: Feigned
emotions
Eye gaze Yes Yes No Yes: Dominance Yes: Emblems
Yes: Sexual interest
Body
Static size Yes Yes Yes: Identity Yes: Formidable
Yes: Gender Yes: Health
Adornments No No No No Yes: Enhance
other clues
Finger ratio No No Yes: Testosterone No No
Waist-to-hip ratio No Yes Yes: Estrogen Yes: Fertility No
Yes: Health
Movements Yes Yes Yes: Emotions Yes: Imminent Yes: Sign language
action
Yes: Arousal
Yes: Dominance Yes: Sexual interest
Yes: Gender Yes: Weakness Yes: Emblems
Odor No Yes Yes: Identity Yes: Imminent Yes: Perfume
actions
Yes: Health
Yes: Fertility
Yes: Mood Yes: Sexual interest
Voice
Tone Yes Yes Yes: Basic emotions Yes: Imminent Yes: Mimic
actions emotions
Yes: Health
Yes: Identity Yes: Dominance
Style No Yes Yes: Identity Yes: Dominance Yes: Mimic identity
Yes: Health Yes: Words

approachability. The fact that the nonverbal com-
munication cues are more likely to be involuntary
suggests that they are more genuine markers of our
internal states, thus our genetic fitness. Further-
more, they appear to be that way because of the long
evolutionary history of our species, the majority
of it accomplished without language—as the past
150,000–200,000 or so years with articulate lan-
guage is a drop in the bucket compared to our 6
million years on the planet. Thus, the 5.8 million
year head start that the signs and signals of nonver-
bal communication got on verbal communication
has helped it maintain its presence within us, and it
continues to influence us today.

66

References
Abbassi, V. (1998). Growth and normal puberty.
Pediatrics, 102, 507–511.
Andersson, M. B. (1994). Sexual selection. Princeton, NJ:
Princeton University Press.
Argyle, M. (1988). Bodily communication (2nd ed.). New
York, NY: Methuen.
Atkinson, A. P., Dittrich, W. H., Gemmell, A. J., &
Young, A. W. (2004). Emotion perception from
dynamic and static body expressions in point-light
and full-light displays. Perception, 33, 717–746.
http://dx.doi.org/10.1068/p5096
Averill, J. R. (1983). Studies on anger and aggression:
Implications for theories of emotion. American
Psychologist, 38, 1145–1160. http://dx.doi.
org/10.1037/0003-066X.38.11.1145
Bailey, A. A., & Hurd, P. L. (2005). Finger length ratio
(2D:4D) correlates with physical aggression
in men but not in women. Biological Psychology,
68, 215–222. http://dx.doi.org/10.1016/
j.biopsycho.2004.05.001
Barbieri, R. L. (1990). The role of adipose tissue
and hypersulinemia in the development of
hyperandrogenism in women. In R. E. Frisch (Ed.),
Adipose tissue and reproduction (pp. 42–57). Basel,
Switzerland: Karger.
Baron-Cohen, S., Wheelwright, S., Hill, J., Raste, Y., &
Plumb, I. (2001). The “Reading the Mind in the
Eyes” Test Revised Version: A study with normal
adults, and adults with Asperger syndrome or
high-functioning autism. Journal of Child Psychology
and Psychiatry, 42, 241–251. http://dx.doi.org/
10.1111/1469-7610.00715
Beall, A. T., & Tracy, J. L. (2013). Women are more
likely to wear red or pink at peak fertility.
Psychological Science, 24, 1837–1841. http://dx.doi.
org/10.1177/0956797613476045
Benderlioglu, Z., & Nelson, R. J. (2004). Digit length
ratios predict reactive aggression in women, but
not in men. Hormones and Behavior, 46, 558–564.
http://dx.doi.org/10.1016/j.yhbeh.2004.06.004
Berenbaum, S. A., Bryk, K. K., Nowak, N., Quigley, C.
A., & Moffat, S. (2009). Fingers as a marker of
prenatal androgen exposure. Endocrinology, 150,
5119–5124. http://dx.doi.org/10.1210/en.2009-0774
Berry, D. S., & Zebrowitz-McArthur, L. (1988). What’s
in a face? Facial maturity and the attribution
of legal responsibility. Personality and Social
Psychology Bulletin, 14, 23–33. http://dx.doi.
org/10.1177/0146167288141003
Björntorp, P. (1988). The associations between obesity,
adipose tissue distribution and disease. Acta Medica
Scandinavica, 723, 121–134.
Evolution and Nonverbal Communication
Bogart, K. R., & Matsumoto, D. (2010). Facial mimicry
is not necessary to recognize emotion: Facial
expression recognition by people with Moebius
syndrome. Social Neuroscience, 5, 241–251.
http://dx.doi.org/10.1080/17470910903395692
Book, A., Costello, K., & Camilleri, J. A. (2013).
Psychopathy and victim selection: The use of gait
as a cue to vulnerability. Journal of Interpersonal
Violence, 28, 2368–2383. http://dx.doi.org/
10.1177/0886260512475315
Brown, T. A., Cash, T. F., & Noles, S. W. (1986).
Perceptions of physical attractiveness among college
students: Selected determinants and methodological
matters. Journal of Social Psychology, 126, 305–316.
http://dx.doi.org/10.1080/00224545.1986.9713590
Bryant, G. A., & Barrett, H. C. (2008). Vocal emotion
recognition across disparate cultures. Journal of
Cognition and Culture, 8, 135–148. http://dx.doi.
org/10.1163/156770908X289242
Byrne, R. W., & Whiten, A. (1988). Machiavellian
intelligence: Social expertise and the evolution of
intellect in monkeys, apes and humans. Oxford,
England: Oxford University Press.
Calvo, M. G., & Nummenmaa, L. (2008). Detection
of emotional faces: Salient physical features guide
effective visual search. Journal of Experimental
Psychology: General, 137, 471–494. http://dx.doi.
org/10.1037/a0012771
Carré, J. M., & McCormick, C. M. (2008). In your face:
Facial metrics predict aggressive behaviour in the
laboratory and in varsity and professional hockey
players. Proceedings of the Royal Society of London,
Series B: Biological Sciences, 275, 2651–2656.
http://dx.doi.org/10.1098/rspb.2008.0873
Carré, J. M., McCormick, C. M., & Mondloch, C. J.
(2009). Facial structure is a reliable cue of aggressive
behavior. Psychological Science, 20, 1194–1198.
http://dx.doi.org/10.1111/j.1467-9280.2009.02423.x
Caspari, R., & Lee, S. H. (2004). Older age becomes
common late in human evolution. Proceedings
of the National Academy of Sciences, USA, 101,
10895–10900. http://dx.doi.org/10.1073/pnas.
0402857101
Centers for Disease Control and Prevention. (1999).
Achievements in public health, 1900–1999: Changes
in the public health system. Morbidity and Mortality
Weekly Report, 48, 1141–1147.
Chagnon, N. A. (1988, February 26). Life histories,
blood revenge, and warfare in a tribal population.
Science, 239, 985–992. http://dx.doi.org/10.1126/
science.239.4843.985
Cheal, J. L., & Rutherford, M. D. (2013). Context-dependent
categorical perception of surprise. Perception, 42, 294–301.
http://dx.doi.org/10.1068/p7130
67
Frank and Shaw
Coetzee, V., Perrett, D. I., & Stephen, I. D. (2009).
Facial adiposity: A cue to health? Perception, 38,
1700–1711. http://dx.doi.org/10.1068/p6423
Corballis, M. C. (2002). From hand to mouth: The origins
of language. Princeton, NJ: Princeton University Press.
Coulson, S. E., O’Dwyer, N. J., Adams, R. D., &
Croxson, G. R. (2004). Expression of emotion and
quality of life after facial nerve paralysis. Otology
and Neurotology, 25, 1014–1019. http://dx.doi.
org/10.1097/00129492-200411000-00026
Crane, E., & Gross, M. (2007, September). Motion capture
and emotion: Affect detection in whole body movement.
Paper presented at the Affective Computing and
Intelligent Interaction Conference, Lisbon, Portugal.
Cutting, J. E., & Kozlowski, L. T. (1977). Recognizing
friends by their walk: Gait perception without
familiarity cues. Bulletin of the Psychonomic Society, 9,
353–356. http://dx.doi.org/10.3758/BF03337021
Dabbs, J. M., Jr. (1997). Testosterone, smiling, and facial
appearance. Journal of Nonverbal Behavior, 21, 45–55.
http://dx.doi.org/10.1023/A:1024947801843
Darwin, C. (1859). On the origin of species by means of
natural selection, or the preservation of favoured races
in the struggle for life. http://dx.doi.org/10.5962/bhl.
title.59991
Darwin, C. (1998). The expression of the emotions in man
and animals (3rd ed.). http://dx.doi.org/10.1037/
10001-000 (Original work published 1872)
Deaner, R. O., Goetz, S. M. M., Shattuck, K., & Schnotala,
T. (2012). Body weight, not facial width-to-height
ratio, predicts aggression in pro hockey players.
Journal of Research in Personality, 46, 235–238.
http://dx.doi.org/10.1016/j.jrp.2012.01.005
DeBruine, L. M. (2002). Facial resemblance enhances
trust. Proceedings of the Royal Society of London,
Series B: Biological Sciences, 269, 1307–1312.
http://dx.doi.org/10.1098/rspb.2002.2034
de Meijer, M. (1989). The contribution of general features
of body movement to the attribution of emotion.
Journal of Nonverbal Behavior, 13, 247–268.
http://dx.doi.org/10.1007/BF00990296
DeMyer, W. (1980). Technique of the neurological
examination. New York, NY: McGraw-Hill.
Deutsch, F. M. (1990). Status, sex, and smiling: The effect
of role on smiling in men and women. Personality
and Social Psychology Bulletin, 16, 531–540.
http://dx.doi.org/10.1177/0146167290163011
Diamond, J. (1997). Guns, germs, and steel: The fates of
human societies. New York, NY: Norton.
Dimberg, U., & Öhman, A. (1996). Behold the wrath:
Psychophysiological response to facial stimuli.
Motivation and Emotion, 20, 149–182. http://dx.doi.
org/10.1007/BF02253869
68
Dixson, A. F. (1983). The hormonal control of sexual
behaviour in primates. Oxford Reviews of Reproductive
Biology, 5, 131–219.
Duchenne de Boulogne, G.-B. (1990). The mechanism
of human facial expression or an electro-physiological
analysis of the expression of the emotions. http://
dx.doi.org/10.1017/CBO9780511752841
(Original work published 1862)
Dufour, K. W., & Weatherhead, P. J. (1996). Estimation
of organism-wide asymmetry in red-winged
blackbirds and its relation to studies of mate
selection. Proceedings of the Royal Society of London,
Series B: Biological Sciences, 263, 769–775. http://
dx.doi.org/10.1098/rspb.1996.0115
Durante, K. M., Griskevicius, V., Hill, S. E., Perilloux,
C., & Li, N. P. (2011). Ovulation, female
competition, and product choice: Hormonal
influences on consumer behavior. Journal of
Consumer Research, 37, 921–934. http://dx.doi.
org/10.1086/656575
Durante, K. M., Li, N. P., & Haselton, M. G. (2008).
Changes in women’s choice of dress across the
ovulatory cycle: Naturalistic and laboratory
task-based evidence. Personality and Social
Psychology Bulletin, 34, 1451–1460. http://dx.doi.
org/10.1177/0146167208323103
Efron, D. (1941). Gesture and environment. Oxford,
England: Kings Crown Press.
Eisenberg, N., Fabes, R. A., Miller, P. A., Fultz, J., Shell,
R., Mathy, R. M., & Reno, R. R. (1989). Relation
of sympathy and personal distress to prosocial
behavior: A multimethod study. Journal of Personality
and Social Psychology, 57, 55–66. http://dx.doi.
org/10.1037/0022-3514.57.1.55
Ekman, P. (1977). Biological and cultural contributions
to body and facial movement. In J. Blacking (Ed.),
Anthropology of the body (pp. 39–84). London,
England: Academic Press.
Ekman, P. (1978). Facial signs: Facts, fantasies, and
possibilities. In T. Sebeok (Ed.), Sight, sound,
and sense (pp. 124–156). Bloomington: Indiana
University Press.
Ekman, P. (1993). Facial expression and emotion.
American Psychologist, 48, 384–392. http://dx.doi.
org/10.1037/0003-066X.48.4.384
Ekman, P. (1999). Basic emotions. In T. Dalgleish & T.
Power (Eds.), The handbook of cognition and emotion
(pp. 45–60). Sussex, England: Wiley.
Ekman, P. (2003). Emotions revealed: Recognizing faces
and feelings to improve emotional life. New York, NY:
Times Books.
Ekman, P., & Cordaro, D. (2011). What is meant by
calling emotions basic. Emotion Review, 3, 364–370.
http://dx.doi.org/10.1177/1754073911410740

Ekman, P., & Friesen, W. V. (1969a). Nonverbal leakage
and clues to deception. Psychiatry: Journal for the
Study of Interpersonal Processes, 32, 88–106.
Ekman, P., & Friesen, W. V. (1969b). The repertoire of
nonverbal behavior: Categories, origins, usage, and
coding. Semiotica, 1, 49–98.
Ekman, P., & Friesen, W. V. (1982). Felt, false, and
miserable smiles. Journal of Nonverbal Behavior, 6,
238–252. http://dx.doi.org/10.1007/BF00987191
Ekman, P., Friesen, W. V., O’Sullivan, M., Chan, A.,
Diacoyanni-Tarlatzis, I., Heider, K., . . . Tzavaras, A.
(1987). Universals and cultural differences in the
judgments of facial expressions of emotion. Journal
of Personality and Social Psychology, 53, 712–717.
http://dx.doi.org/10.1037/0022-3514.53.4.712
Ekman, P., Levenson, R. W., & Friesen, W. V. (1983,
September 16). Autonomic nervous system
activity distinguishes among emotions. Science,
221, 1208–1210. http://dx.doi.org/10.1126/
science.6612338
Ekman, P., O’Sullivan, M., Friesen, W. V., & Scherer, K.
R. (1991). Face, voice, and body in detecting deceit.
Journal of Nonverbal Behavior, 15, 125–135. http://
dx.doi.org/10.1007/BF00998267
Elliot, A. J., & Niesta, D. (2008). Romantic red: Red
enhances men’s attraction to women. Journal of
Personality and Social Psychology, 95, 1150–1164.
http://dx.doi.org/10.1037/0022-3514.95.5.1150
Elliot, A. J., & Pazda, A. D. (2012). Dressed for sex:
Red as a female sexual signal in humans. PLoS
ONE, 7, e34607. http://dx.doi.org/10.1371/journal.
pone.0034607
Esteves, F., Dimberg, U., & Öhman, A. (1994).
Automatically elicited fear: Conditioned skin
conductance responses to masked facial expressions.
Cognition and Emotion, 8, 393–413. http://dx.doi.
org/10.1080/02699939408408949
Fairbairn, D. J. (1997). Allometry for sexual size
dimorphism: Pattern and process in the coevolution
of body in males and females. Annual Review of
Ecology and Systematics, 28, 659–687. http://dx.doi.
org/10.1146/annurev.ecolsys.28.1.659
Fessler, D. M. T., & Holbrook, C. (2013). Friends
shrink foes: The presence of comrades decreases the
envisioned physical formidability of an opponent.
Psychological Science, 24, 797–802. http://dx.doi.
org/10.1177/0956797612461508
Fessler, D. M. T., Holbrook, C., & Snyder, J. K. (2012).
Weapons make the man (larger): Formidability is
represented as size and strength in humans. PLoS
ONE, 7, e32751. http://dx.doi.org/10.1371/journal.
pone.0032751
Fink, B., Manning, J. T., & Neave, N. (2006). The
2nd–4th digit ratio (2D:4D) and neck circumference:
Evolution and Nonverbal Communication
Implications for risk factors in coronary heart
disease. International Journal of Obesity, 30, 711–714.
http://dx.doi.org/10.1038/sj.ijo.0803154
Fink, B., & Matts, P. J. (2008). The effects of skin colour
distribution and topography cues on the perception
of female facial age and health. Journal of the
European Academy of Dermatology and Venereology,
22, 493–498. http://dx.doi.org/10.1111/j.1468-
3083.2007.02512.x
Fink, B., Weege, B., Manning, J. T., & Trivers, R. (2014).
Body symmetry and physical strength in human
males. American Journal of Human Biology, 26,
697–700. http://dx.doi.org/10.1002/ajhb.22584
Folstad, I., & Karter, A. J. (1992). Parasites, bright
males, and the immunocompetence handicap.
American Naturalist, 139, 603–622. http://dx.doi.
org/10.1086/285346
Frank, M. G., & Ekman, P. (2004). Appearing truthful
generalizes across different deception situations.
Journal of Personality and Social Psychology, 86,
486–495. http://dx.doi.org/10.1037/0022-3514.
86.3.486
Frank, M. G., Ekman, P., & Friesen, W. V. (1993).
Behavioral markers and recognizability of the
smile of enjoyment. Journal of Personality and
Social Psychology, 64, 83–93. http://dx.doi.
org/10.1037/0022-3514.64.1.83
Frank, M. G., & Gilovich, T. (1988). The dark side
of self- and social perception: Black uniforms
and aggression in professional sports. Journal of
Personality and Social Psychology, 54, 74–85.
http://dx.doi.org/10.1037/0022-3514.54.1.74
Frank, M. G., Griffin, D. J., & Maroulis, A. (2013).
The voice. In D. Matsumoto, M. G. Frank, & H. C.
Hwang (Eds.), Nonverbal communication: Science and
applications (pp. 53–74). Thousand Oaks, CA: Sage.
Frank, M. G., & Svetieva, E. (2014). Microexpressions
and deception. In M. K. Mandal & A. Awasthi (Eds.),
Understanding facial expressions in communication:
Cross-cultural and multidisciplinary perspectives
(pp. 227–242). New York, NY: Springer.
Frijda, N. (1986). The emotions. Cambridge, England:
Cambridge University Press.
Galati, D., Miceli, R., & Sini, B. (2001). Judging
and coding facial expression of emotions in
congenitally blind children. International Journal of
Behavioral Development, 25, 268–278. http://dx.doi.
org/10.1080/01650250042000393
Galati, D., Sini, B., Schmidt, S., & Tinti, C. (2003).
Spontaneous facial expressions in congenitally blind
and sighted children aged 8–11. Journal of Visual
Impairment and Blindness, 97, 418–428.
Gangestad, S. W., Bennett, K. L., & Thornhill, R. (2001).
A latent variable model of developmental instability
69
Frank and Shaw
in relation to men’s sexual behaviour. Proceedings
of the Royal Society of London, Series B: Biological
Sciences, 268, 1677–1684. http://dx.doi.org/10.1098/
rspb.2001.1675
Gangestad, S. W., & Simpson, J. A. (2000). The evolution
of human mating: Trade-offs and strategic pluralism.
Behavioral and Brain Sciences, 23, 573–587. http://
dx.doi.org/10.1017/S0140525X0000337X
Gangestad, S. W., Thornhill, R., & Yeo, R. A. (1994).
Facial attractiveness, developmental stability and
fluctuating asymmetry. Ethology and Sociobiology,
15, 73–85. http://dx.doi.org/10.1016/0162-3095
(94)90018-3
Gilchrist, M. R. (1998). Disease and infection in the
American Civil War. American Biology Teacher, 60,
258–262. http://dx.doi.org/10.2307/4450468
Goodall, J. (2000). In the shadow of man. Boston, MA:
Houghton Mifflin Harcourt.
Grammer, K., & Eibl-Eibesfeldt, I. (1990). The ritualisation of
laughter. In W. A. Koch (Ed.), Natürlichkeit der Sprache
und der Kultur [Naturalness of language and culture]
(pp. 192–214). Bochum, Germany: Brockmeyer.
Grammer, K., & Thornhill, R. (1994). Human (Homo
sapiens) facial attractiveness and sexual selection:
The role of symmetry and averageness. Journal of
Comparative Psychology, 108, 233–242. http://dx.doi.
org/10.1037/0735-7036.108.3.233
Grayson, B., & Stein, M. I. (1981). Attracting
assault: Victims’ nonverbal cues. Journal of
Communication, 31, 68–75. http://dx.doi.
org/10.1111/j.1460-2466.1981.tb01206.x
Guéguen, N. (2012). Tattoos, piercings, and alcohol
consumption. Alcoholism: Clinical and Experimental
Research, 36, 1253–1256. http://dx.doi.org/10.1111/
j.1530-0277.2011.01711.x
Hadjikhani, N., Kveraga, K., Naik, P., & Ahlfors,
S. P. (2009). Early (M170) activation of face-
specific cortex by face-like objects. NeuroReport,
20, 403–407. http://dx.doi.org/10.1097/
WNR.0b013e328325a8e1
Hager, J. C., & Ekman, P. (1979). Long distance
transmission of facial affect signals. Ethology
and Sociobiology, 1, 77–82. http://dx.doi.
org/10.1016/0162-3095(79)90007-4
Haggard, E. A., & Isaacs, K. S. (1966). Micro-momentary
facial expressions as indicators of ego mechanisms in
psychotherapy. In L. A. Gottschalk & A. H. Auerbach
(Eds.), Methods of research in psychotherapy
(pp. 154–165). http://dx.doi.org/10.1007/978-
1-4684-6045-2_14
Hansen, C. H., & Hansen, R. D. (1988). Finding the face
in the crowd: An anger superiority effect. Journal
of Personality and Social Psychology, 54, 917–924.
http://dx.doi.org/10.1037/0022-3514.54.6.917
70
Harvey, O. J., White, B. J., Hood, W. R., & Sherif, C. W.
(1961). Intergroup conflict and cooperation: Vol. 10.
The Robbers Cave experiment. Norman: University of
Oklahoma Press.
Haselhuhn, M. P., & Wong, E. M. (2012). Bad to the
bone: Facial structure predicts unethical behaviour.
Proceedings of the Royal Society of London, Series B:
Biological Sciences, 279, 571–576.
Haselton, M. G., & Gildersleeve, K. (2011). Can
men detect ovulation? Current Directions in
Psychological Science, 20, 87–92. http://dx.doi.
org/10.1177/0963721411402668
Haselton, M. G., Mortezaie, M., Pillsworth, E. G., Bleske-
Rechek, A., & Frederick, D. A. (2007). Ovulatory
shifts in human female ornamentation: Near
ovulation, women dress to impress. Hormones and
Behavior, 51, 40–45. http://dx.doi.org/10.1016/
j.yhbeh.2006.07.007
Hauser, M. D. (1997). The evolution of communication.
Cambridge, MA: MIT Press/Bradford Books.
Heaton, P., Reichenbacher, L., Sauter, D., Allen, R.,
Scott, S., & Hill, E. (2012). Measuring the effects of
alexithymia on perception of emotional vocalizations
in autistic spectrum disorder and typical
development. Psychological Medicine, 42, 2453–2459.
http://dx.doi.org/10.1017/S0033291712000621
Hendriks, M. C. P., Nelson, J. K., Cornelius, R. R., &
Vingerhoets, J. J. M. (2008). Why crying improves
our well-being: An attachment-theory perspective on
the functions of adult crying. In J. J. M. Vingerhoets,
I. Nyklicek, & J. Denollet (Eds.), Emotion regulation:
Conceptual and clinical issues (pp. 87–96). http://
dx.doi.org/10.1007/978-0-387-29986-0_6
Hess, E. H. (1965). Attitude and pupil size. Scientific
American, 212, 46–54. http://dx.doi.org/10.1038/
scientificamerican0465-46
Hiatt, R. A., Sotomayor, Z. R., Sanchez, G., Zambrana,
M., & Knight, W. B. (1979). Factors in the
pathogenesis of acute Schistosomiasis mansoni.
Journal of Infectious Diseases, 139, 659–666. http://
dx.doi.org/10.1093/infdis/139.6.659
Higham, J. P., Hughes, K. D., Brent, L. J. N., Dubuc, C.,
Engelhardt, A., Heistermann, M., . . . Stevens, M.
(2011). Familiarity affects the assessment of female
facial signals of fertility by free-ranging male rhesus
macaques. Proceedings of the Royal Society of London,
Series B: Biological Sciences, 278, 3452–3458. http://
dx.doi.org/10.1098/rspb.2011.0052
Hill, R. A., & Barton, R. A. (2005, May 19). Psychology:
Red enhances human performance in contests.
Nature, 435, 293. http://dx.doi.org/10.1038/435293a
Hill, S. E., & Buss, D. M. (2008). The mere presence
of opposite-sex others on judgments of sexual
and romantic desirability: Opposite effects

for men and women. Personality and Social
Psychology Bulletin, 34, 635–647. http://dx.doi.
org/10.1177/0146167207313728
Hollien, H., Majewski, W., & Doherty, E. T. (1982).
Perceptual identification of voices under normal,
stress, and disguise speaking conditions. Journal of
Phonetics, 10, 139–148.
Hurley, C. M., & Frank, M. G. (2011). Executing facial
control during deception situations. Journal of
Nonverbal Behavior, 35, 119–131. http://dx.doi.
org/10.1007/s10919-010-0102-1
Izard, C. E. (1977). Human emotions. http://dx.doi.
org/10.1007/978-1-4899-2209-0
Johansson, S. (2011). Constraining the time when
language evolved. Linguistic and Philosophical
Investigations, 10, 45–59.
Juslin, P. N., & Laukka, P. (2003). Communication
of emotions in vocal expression and music
performance: Different channels, same code?
Psychological Bulletin, 129, 770–814. http://dx.doi.
org/10.1037/0033-2909.129.5.770
Keltner, D. (1995). The signs of appeasement: Evidence
for the distinct displays of embarrassment,
amusement, and shame. Journal of Personality
and Social Psychology, 68, 441–454. http://dx.doi.
org/10.1037/0022-3514.68.3.441
Keltner, D., & Buswell, B. N. (1997). Embarrassment:
Its distinct form and appeasement functions.
Psychological Bulletin, 122, 250–270. http://dx.doi.
org/10.1037/0033-2909.122.3.250
Kenrick, D. T. (1987). Gender, genes, and the
environment: A biosocial interactionist perspective.
In P. Shaver & C. Hendrick (Eds.), Sex and gender
(pp. 14–43). Newbury Park, CA: Sage.
Kimani, Z. M., Ogutu, O., & Kibe, A. (2014).
The prevalence and impact of obstetric fistula
in women of Kaptembwa-Nakuru, Kenya.
International Journal of Applied Science and
Technology, 4, 273–287.
Kitchen, D. M., Cheney, D. L., & Seyfarth, R. M. (2003).
Female baboons’ responses to male loud calls.
Ethology, 109, 401–412. http://dx.doi.org/10.1046/
j.1439-0310.2003.00878.x
Klineberg, O. (1940). Social psychology (Vol. 5). http://
dx.doi.org/10.1037/13603-000
Knapp, M. L., & Hall, J. A. (2010). Nonverbal
communication in human interaction (7th ed.). Boston,
MA: Wadsworth, Cengage Learning.
Kowner, R. (1996). Facial asymmetry and attractiveness
judgment in developmental perspective. Journal
of Experimental Psychology: Human Perception
and Performance, 22, 662–675. http://dx.doi.
org/10.1037/0096-1523.22.3.662
Evolution and Nonverbal Communication
Kozlowski, L. T., & Cutting, J. E. (1977). Recognizing
the sex of a walker from a dynamic point-light
display. Perception and Psychophysics, 21, 575–580.
http://dx.doi.org/10.3758/BF03198740
Langlois, J. H., Roggman, L. A., & Musselman, L. (1994).
What is average and what is not average about
attractive faces? Psychological Science, 5, 214–220.
http://dx.doi.org/10.1111/j.1467-9280.1994.
tb00503.x
Larsson, B. (1985). Obesity and prospective risk for
associated diseases with special reference to the
importance of adipose tissue distribution. In J.
Vague, P. Björntorp, B. Guy-Grand, M. Rebuffé-
Scrive, & P. Vague (Eds.), Metabolic complications
of human obesities (pp. 21–29). Amsterdam, the
Netherlands: Excerpta Medica.
LeBlanc, S. A. (2003). Constant battles: The myth of the
peaceful, noble savage. New York, NY: St. Martin’s
Press.
LeDoux, J. (1996). The emotional brain: The mysterious
underpinnings of emotional life. New York, NY:
Touchstone.
Lefevre, C. E., Lewis, G. J., Bates, T. C., Dzhelyova, M.,
Coetzee, V., Deary, I. J., & Perrett, D. I. (2012). No
evidence for sexual dimorphism of facial width-to-
height ratio in four large adult samples. Evolution
and Human Behavior, 33, 623–627. http://dx.doi.
org/10.1016/j.evolhumbehav.2012.03.002
Lemieux, S., Prud’homme, D., Tremblay, A., Bouchard,
C., & Després, J. P. (1996). Anthropometric
correlates to changes in visceral adipose tissue over 7
years in women. International Journal of Obesity, 20,
618–624.
Lemke, M. R., Wendorff, T., Mieth, B., Buhl, K., &
Linnemann, M. (2000). Spatiotemporal gait patterns
during over ground locomotion in major depression
compared with healthy controls. Journal of
Psychiatric Research, 34, 277–283. http://dx.doi.
org/10.1016/S0022-3956(00)00017-0
Levenson, R. W. (1999). The intrapersonal functions of
emotion. Cognition and Emotion, 13, 481–504. http://
dx.doi.org/10.1080/026999399379159
Levenson, R. W., Ekman, P., & Friesen, W. V. (1990).
Voluntary facial action generates emotion-
specific autonomic nervous system activity.
Psychophysiology, 27, 363–384. http://dx.doi.
org/10.1111/j.1469-8986.1990.tb02330.x
Levenson, R. W., Ekman, P., Heider, K., & Friesen, W.
V. (1992). Emotion and autonomic nervous system
activity in the Minangkabau of West Sumatra. Journal
of Personality and Social Psychology, 62, 972–988.
http://dx.doi.org/10.1037/0022-3514.62.6.972
Lewis, G. N., Byblow, W. D., & Walt, S. E. (2000). Stride
length regulation in Parkinson’s disease: The use of
71
Frank and Shaw
extrinsic, visual cues. Brain: A Journal of Neurology,
123, 2077–2090. http://dx.doi.org/10.1093/
brain/123.10.2077
Lieberman, D., Tooby, J., & Cosmides, L. (2003). Does
morality have a biological basis? An empirical test
of the factors governing moral sentiments relating
to incest. Proceedings of the Royal Society of London,
Series B: Biological Sciences, 270, 819–826. http://
dx.doi.org/10.1098/rspb.2002.2290
Lieberman, P. (2007). Current views on Neanderthal
speech capabilities: A reply to Boe et al. (2002).
Journal of Phonetics, 35, 552–563.
Lieberman, P. (2008). Old-time linguistic theories.
Cortex, 44, 218–226. http://dx.doi.org/10.1016/
j.cortex.2007.10.004
Lipson, S. F., & Ellison, P. T. (1996). Comparison
of salivary steroid profiles in naturally occurring
conception and non-conception cycles. Human
Reproduction, 11, 2090–2096. http://dx.doi.
org/10.1093/oxfordjournals.humrep.a019055
Livshits, G., & Kobyliansky, E. (1989). Study
of genetic variance in the fluctuating
asymmetry of anthropometrical traits. Annals
of Human Biology, 16, 121–129. http://dx.doi.
org/10.1080/03014468700006972
MacGillivray, M. H., Morishima, A., Conte, F.,
Grumbach, M., & Smith, E. P. (1998). Pediatric
endocrinology update: An overview. The essential
roles of estrogens in pubertal growth, epiphyseal
fusion and bone turnover: Lessons from mutations
in the genes for aromatase and the estrogen receptor.
Hormone Research, 49(Suppl. 1), 2–8. http://dx.doi.
org/10.1159/000053061
Manning, J. T. (2002). Digit ratio: A pointer to fertility,
behavior, and health. New Brunswick, NJ: Rutgers
University Press.
Manning, J. T., & Bundred, P. E. (2001). The ratio
of second to fourth digit length and age at
first myocardial infarction in men: A link with
testosterone? British Journal of Cardiology, 8,
720–723.
Manning, J. T., Scutt, D., Wilson, J., & Lewis-Jones, D.
I. (1998). The ratio of 2nd to 4th digit length: A
predictor of sperm numbers and concentrations of
testosterone, luteinizing hormone and oestrogen.
Human Reproduction, 13, 3000–3004. http://dx.doi.
org/10.1093/humrep/13.11.3000
Marsh, A. A., Adams, R. B., Jr., & Kleck, R. E. (2005).
Why do fear and anger look the way they do? Form
and social function in facial expressions. Personality
and Social Psychology Bulletin, 31, 73–86. http://
dx.doi.org/10.1177/0146167204271306
Matsumoto, D. (1993). Ethnic differences in affect
intensity, emotion judgments, display rule attitudes,
72
and self-reported emotional expression in an
American sample. Motivation and Emotion, 17,
107–123. http://dx.doi.org/10.1007/BF00995188
Matsumoto, D., & Hwang, H. C. (2013). Facial
expressions. In D. Matsumoto, M. G. Frank, & H. C.
Hwang (Eds.), Nonverbal communication: Science and
applications (pp. 15–52). Thousand Oaks, CA: Sage.
Matsumoto, D., Keltner, D., Shiota, M. N., O’Sullivan,
M., & Frank, M. G. (2008). Facial expressions of
emotion. In M. Lewis, J. M. Haviland-Jones, &
L. F. Barrett (Eds.), Handbook of emotions (3rd ed.,
pp. 211–234). New York, NY: Guilford Press.
Matsumoto, D., & Willingham, B. (2009). Spontaneous
facial expressions of emotion of congenitally
and noncongenitally blind individuals. Journal of
Personality and Social Psychology, 96, 1–10. http://
dx.doi.org/10.1037/a0014037
Mazur, A., & Mueller, U. (1996). Facial dominance. In
A. Somit & S. Peterson (Eds.), Research in biopolitics
(pp. 99–111). London, England: JAI Press.
McCrone, J. (1991). The ape that spoke: Language and the
evolution of the human mind. New York, NY: William
Morrow.
McKinney, K., & Sprecher, S. (1991). Sexuality in close
relationships. Princeton, NJ: Erlbaum.
McNeill, D. (1992). Hand and mind: What gestures reveal
about thought. Chicago, IL: University of Chicago
Press.
Meihlke, A. (1973). Surgery of the facial nerve.
Philadelphia, PA: Saunders.
Miller, S. L., & Maner, J. K. (2011). Ovulation as a male
mating prime: Subtle signs of women’s fertility
influence men’s mating cognition and behavior.
Journal of Personality and Social Psychology, 100,
295–308. http://dx.doi.org/10.1037/a0020930
Møller, A. P., & Alatalo, R. V. (1999). Good-genes effects
in sexual selection. Proceedings of the Royal Society
of London, Series B: Biological Sciences, 266, 85–91.
http://dx.doi.org/10.1098/rspb.1999.0607
Montepare, J. M., Goldstein, S. B., & Clausen, A. (1987).
The identification of emotions from gait information.
Journal of Nonverbal Behavior, 11, 33–42. http://
dx.doi.org/10.1007/BF00999605
Muehlenbein, M. P., & Bribiescas, R. G. (2005).
Testosterone-mediated immune functions and
male life histories. American Journal of Human
Biology, 17, 527–558. http://dx.doi.org/10.1002/
ajhb.20419
Myers, R. E. (1976). Comparative neurology
of vocalization and speech: Proof of a
dichotomy. Annuals of the New York Academy
of Sciences, 280, 745–757. http://dx.doi.
org/10.1111/j.1749-6632.1976.tb25537.x

Ohbuchi, K., Kameda, M., & Agarie, N. (1989). Apology
as aggression control: Its role in mediating appraisal
of and response to harm. Journal of Personality
and Social Psychology, 56, 219–227. http://dx.doi.
org/10.1037/0022-3514.56.2.219
Öhman, A., & Mineka, S. (2001). Fears, phobias,
and preparedness: Toward an evolved module of
fear and fear learning. Psychological Review, 108,
483–522.
Oosterhof, N. N., & Todorov, A. (2008). The functional
basis of face evaluation. Proceedings of the National
Academy of Sciences, USA, 105, 11087–11092. http://
dx.doi.org/10.1073/pnas.0805664105
Özener, B. (2012). Facial width-to-height ratio in a
Turkish population is not sexually dimorphic
and is unrelated to aggressive behavior. Evolution
and Human Behavior, 33, 169–173. http://dx.doi.
org/10.1016/j.evolhumbehav.2011.08.001
Palmer, A. R., & Strobeck, C. (2003). Fluctuating
asymmetry analyses revisited. In M. Polak
(Ed.), Developmental instability (DI): Causes and
consequences (pp. 279–319). Oxford, England:
Oxford University Press.
Panksepp, J. (1998). Affective neuroscience: The
foundations of human and animal emotions. New York,
NY: Oxford University Press.
Pawlowski, B., Dunbar, R. I. M., & Lipowicz, A. (2000,
January 13). Tall men have more reproductive
success. Nature, 403, 156. http://dx.doi.org/
10.1038/35003107
Penton-Voak, I. S., & Chen, J. Y. (2004). High salivary
testosterone is linked to masculine male facial
appearance in humans. Evolution and Human
Behavior, 25, 229–241. http://dx.doi.org/10.1016/
j.evolhumbehav.2004.04.003
Pfaff, D. W., & Sakuma, Y. (1979). Facilitation of the
lordosis reflex of female rats from the ventromedial
nucleus of the hypothalamus. Journal of Physiology,
288, 189–202.
Pflüger, L. S., Oberzaucher, E., Katina, S. T., Holzleitner,
I. J., & Grammer, K. (2012). Cues to fertility:
Perceived attractiveness and facial shape predict
reproductive success. Evolution and Human
Behavior, 33, 708–714. http://dx.doi.org/10.1016/
j.evolhumbehav.2012.05.005
Phan, K. L., Wager, T., Taylor, S. F., & Liberzon, I.
(2002). Functional neuroanatomy of emotion: A
meta-analysis of emotion activation studies in PET
and fMRI. NeuroImage, 16, 331–348. http://dx.doi.
org/10.1006/nimg.2002.1087
Pillsworth, E. G., Haselton, M. G., & Buss, D. M.
(2004). Ovulatory shifts in female sexual desire.
Journal of Sex Research, 41, 55–65. http://dx.doi.
org/10.1080/00224490409552213
Evolution and Nonverbal Communication
Pinker, S. (2011). The better angels of our nature: Why
violence has declined. New York, NY: Viking.
Pipitone, R. N., & Gallup, G. G., Jr. (2008).
Women’s voice attractiveness varies across the
menstrual cycle. Evolution and Human Behavior,
29, 268–274. http://dx.doi.org/10.1016/
j.evolhumbehav.2008.02.001
Porter, S., & ten Brinke, L. (2008). Reading between the
lies: Identifying concealed and falsified emotions in
universal facial expressions. Psychological Science, 19,
508–514. http://dx.doi.org/10.1111/j.1467-9280.
2008.02116.x
Pruitt, D. G., & Kim, S. H. (2004). Social conflict:
Escalation, stalemate, and settlement (3rd ed.).
New York, NY: McGraw-Hill.
Puts, D. A. (2010). Beauty and the beast: Mechanisms of
sexual selection in humans. Evolution and Human
Behavior, 31, 157–175. http://dx.doi.org/10.1016/
j.evolhumbehav.2010.02.005
Reisenzein, R., Bördgen, S., Holtbernd, T., & Matz, D.
(2006). Evidence for strong dissociation between
emotion and facial displays: The case of surprise.
Journal of Personality and Social Psychology, 91,
295–315. http://dx.doi.org/10.1037/0022-3514.
91.2.295
Rhodes, G. (2006). The evolutionary psychology of
facial beauty. Annual Review of Psychology, 57,
199–226. http://dx.doi.org/10.1146/annurev.
psych.57.102904.190208
Rhodes, G., & Simmons, L. W. (2007). Symmetry,
attractiveness, and sexual selection. In R. I. M.
Dunbar & L. Barrett (Eds.), Oxford handbook of
evolutionary psychology (pp. 333–364). Oxford,
England: Oxford University Press.
Rhodes, G., Yoshikawa, S., Palermo, R., Simmons, L.
W., Peters, M., Lee, K., . . . Crawford, J. R. (2007).
Perceived health contributes to the attractiveness
of facial symmetry, averageness, and sexual
dimorphism. Perception, 36, 1244–1252. http://
dx.doi.org/10.1068/p5712
Rinn, W. E. (1984). The neuropsychology of facial
expression: A review of the neurological and
psychological mechanisms for producing facial
expressions. Psychological Bulletin, 95, 52–77.
http://dx.doi.org/10.1037/0033-2909.95.1.52
Ro, T., Russell, C., & Lavie, N. (2001). Changing faces:
A detection advantage in the flicker paradigm.
Psychological Science, 12, 94–99. http://dx.doi.
org/10.1111/1467-9280.00317
Roberts, S. C., Havlicek, J., Flegr, J., Hruskova, M., Little,
A. C., Jones, B. C., . . . Petrie, M. (2004). Female
facial attractiveness increases during the fertile
phase of the menstrual cycle. Proceedings of the
Royal Society of London, Series B: Biological Sciences,
73
Frank and Shaw
271(Suppl. 5), S270–S272. http://dx.doi.org/
10.1098/rsbl.2004.0174
Röder, S., Fink, B., Feinberg, D. R., & Neave, N.
(2013). Facial visualizations of women’s voices
suggest a cross-modality preference for femininity.
Evolutionary Psychology, 11, 227–237.
Rule, N. O., Garrett, J. V., & Ambady, N. (2010). On
the perception of religious group membership from
faces. Public Library of Science ONE, 5, e14241.
http://dx.doi.org/10.1371/journal.pone.0014241
Saad, G., & Stenstrom, E. (2012). Calories, beauty, and
ovulation: The effects of the menstrual cycle on
food and appearance-related consumption. Journal
of Consumer Psychology, 22, 102–113. http://dx.doi.
org/10.1016/j.jcps.2011.10.001
Sabatinelli, D., Fortune, E. E., Li, Q., Siddiqui, A., Krafft,
C., Oliver, W. T., . . . Jeffries, J. (2011). Emotional
perception: Meta-analyses of face and natural scene
processing. NeuroImage, 54, 2524–2533. http://
dx.doi.org/10.1016/j.neuroimage.2010.10.011
Sauter, D. A., Eisner, F., Calder, A. J., & Scott, S.
K. (2010). Perceptual cues in nonverbal vocal
expressions of emotion. Quarterly Journal of
Experimental Psychology, 63, 2251–2272.
http://dx.doi.org/10.1080/17470211003721642
Scheib, J. E., Gangestad, S. W., & Thornhill, R. (1999).
Facial attractiveness, symmetry and cues of good
genes. Proceedings of the Royal Society of London,
Series B: Biological Sciences, 266, 1913–1917.
http://dx.doi.org/10.1098/rspb.1999.0866
Scherer, K. R. (2003). Vocal communication of
emotion: A review of research paradigms. Speech
Communication, 40, 227–256. http://dx.doi.
org/10.1016/S0167-6393(02)00084-5
Scherer, K. R., Banse, R., Wallbott, H. G., & Goldbeck,
T. (1991). Vocal cues in emotion encoding and
decoding. Motivation and Emotion, 15, 123–148.
http://dx.doi.org/10.1007/BF00995674
Sell, A., Cosmides, L., Tooby, J., Sznycer, D., von Rueden,
C., & Gurven, M. (2009). Human adaptations for
the visual assessment of strength and fighting ability
from the body and face. Proceedings of the Royal
Society of London, Series B: Biological Sciences, 276,
575–584. http://dx.doi.org/10.1098/rspb.2008.1177
Shaw, A. S., Kotowski, M. R., Boster, F. J., & Levine,
T. R. (2012). The effect of prenatal sex hormones
on the development of verbal aggression. Journal
of Communication, 62, 778–793. http://dx.doi.
org/10.1111/j.1460-2466.2012.01665.x
Simon-Thomas, E. R., Keltner, D. J., Sauter, D., Sinicropi-
Yao, L., & Abramson, A. (2009). The voice
conveys specific emotions: Evidence from vocal
burst displays. Emotion, 9, 838–846. http://dx.doi.
org/10.1037/a0017810
74
Singh, D. (1993). Adaptive significance of female physical
attractiveness: Role of waist-to-hip ratio. Journal
of Personality and Social Psychology, 65, 293–307.
http://dx.doi.org/10.1037/0022-3514.65.2.293
Singh, D. (1994). Is thin really beautiful and good?
Relationship between waist-to-hip ratio (WHR)
and female attractiveness. Personality and
Individual Differences, 16, 123–132. http://dx.doi.
org/10.1016/0191-8869(94)90116-3
Singh, D. (1995). Female health, attractiveness,
and desirability for relationship: Role of breast
asymmetry and waist-to-hip ratio. Ethology
and Sociobiology, 16, 465–481. http://dx.doi.
org/10.1016/0162-3095(95)00073-9
Singh, D. (2002). Female mate value at a glance:
Relationship of waist-to-hip ratio to health, fecundity
and attractiveness. Neuroendocrinology Letters,
23(Suppl. 4), 81–91.
Singh, D., & Luis, S. (1995). Ethnic and gender
consensus for the effect of waist-to-hip ratio on
judgment of women’s attractiveness. Human Nature,
6, 51–65. http://dx.doi.org/10.1007/BF02734135
Singh, D., & Young, R. K. (1995). Body weight, waist-
to-hip ratio, breasts and hips: Role in judgments
of female attractiveness and desirability for
relationships. Ethology and Sociobiology, 16,
483–507. http://dx.doi.org/10.1016/0162-3095
(95)00074-7
Snyder, J. K., Fessler, D. M. T., Tiokhin, L., Frederick, D.
A., Lee, S. W., & Navarrete, C. D. (2011). Trade-offs
in a dangerous world: Women’s fear of crime predicts
preferences for aggressive and formidable mates.
Evolution and Human Behavior, 32, 127–137. http://
dx.doi.org/10.1016/j.evolhumbehav.2010.08.007
Stirrat, M., & Perrett, D. I. (2010). Valid facial cues
to cooperation and trust: Male facial width and
trustworthiness. Psychological Science, 21, 349–354.
http://dx.doi.org/10.1177/0956797610362647
Stirrat, M., & Perrett, D. I. (2012). Face structure predicts
cooperation: Men with wider faces are more generous
to their in-group when out-group competition is
salient. Psychological Science, 23, 718–722. http://
dx.doi.org/10.1177/0956797611435133
Susskind, J. M., Lee, D. H., Cusi, A., Feiman, R., Grabski,
W., & Anderson, A. K. (2008). Expressing fear
enhances sensory acquisition. Nature Neuroscience,
11, 843–850. http://dx.doi.org/10.1038/nn.2138
Swaddle, J. P., & Reierson, G. W. (2002). Testosterone
increases perceived dominance but not attractiveness
in human males. Proceedings of the Royal Society of
London, Series B: Biological Sciences, 269, 2285–2289.
http://dx.doi.org/10.1098/rspb.2002.2165
Tajfel, H., & Turner, J. C. (1986). The social identity
theory of intergroup behaviour. In S. Worchel &

W. G. Austin (Eds.), Psychology of intergroup
relations (pp. 7–24). Chicago, IL: Nelson-Hall.
ten Brinke, L., Porter, S., & Baker, A. (2012). Darwin
the detective: Observable facial muscle contractions
reveal emotional high-stakes lies. Evolution and
Human Behavior, 33, 411–416. http://dx.doi.
org/10.1016/j.evolhumbehav.2011.12.003
Thaut, M. H., McIntosh, G. C., Prassas, S. G., &
Rice, R. R. (1992). Effect of rhythmic auditory
cuing on temporal stride parameters and EMG
patterns in normal gait. Journal of Neurologic
Rehabilitation, 6, 185–190. http://dx.doi.
org/10.1177/136140969200600403
Thornhill, R., & Gangestad, S. W. (1999). Facial
attractiveness. Trends in Cognitive Sciences, 3, 452–460.
http://dx.doi.org/10.1016/S1364-6613(99)01403-5
Tomkins, S. S. (1962). Affect, imagery, and consciousness:
Vol. 1. The positive affects. New York, NY: Springer.
Tomkins, S. S. (1963). Affect, imagery, and consciousness:
Vol. 2. The negative affects. New York, NY: Springer.
Tooby, J., & Cosmides, L. (1990). The past explains the
present: Emotional adaptations and the structure of
ancestral environments. Ethology and Sociobiology,
11, 375–424. http://dx.doi.org/10.1016/0162-3095
(90)90017-Z
Tooby, J., & Cosmides, L. (2005). Conceptual
foundations of evolutionary psychology. In D. Buss
(Ed.), The handbook of evolutionary psychology
(pp. 5–67). Hoboken, NJ: Wiley.
Tracy, J. L., & Robins, R. W. (2008). The nonverbal
expression of pride: Evidence for cross-cultural
recognition. Journal of Personality and Social
Psychology, 94, 516–530. http://dx.doi.
org/10.1037/0022-3514.94.3.516
Tracy, J. L., Shariff, A. F., & Cheng, J. T. (2010). A
naturalist’s view of pride. Emotion Review, 2, 163–177.
http://dx.doi.org/10.1177/1754073909354627
Tschiassny, K. (1953). Eight syndromes of facial paralysis
and their significance in locating the lesion. Annals
of Otology, Rhinology, and Laryngology, 62, 677–691.
http://dx.doi.org/10.1177/000348945306200306
Van Valen, L. (1962). A study of fluctuating
asymmetry. Evolution, 16, 125–142. http://dx.doi.
org/10.2307/2406192
Van Vugt, M. (2011). The male warrior hypothesis. In
J. Forgas, A. W. Kruglanski, & K. D. Williams (Eds.),
The psychology of social conflict and aggression (Vol. 13,
pp. 233–248). New York, NY: Psychology Press.
Van Vugt, M., De Cremer, D., & Janssen, D. P. (2007).
Gender differences in cooperation and competition:
The male-warrior hypothesis. Psychological Science,
18, 19–23. http://dx.doi.org/10.1111/j.1467-9280.
2007.01842.x
Evolution and Nonverbal Communication
Verplaetse, J., Vanneste, S., & Braeckman, J. (2007).
You can judge a book by its cover: The sequel.
A kernel of truth in predictive cheating detection.
Evolution and Human Behavior, 28, 260–271. http://
dx.doi.org/10.1016/j.evolhumbehav.2007.04.006
Vytal, K., & Hamann, S. (2010). Neuroimaging support
for discrete neural correlates of basic emotions:
A voxel-based meta-analysis. Journal of Cognitive
Neuroscience, 22, 2864–2885. http://dx.doi.
org/10.1162/jocn.2009.21366
Wade, T. J. (2010). The relationship between symmetry
and attractiveness and mating relevant decisions and
behavior: A review. Symmetry, 2, 1081–1098. http://
dx.doi.org/10.3390/sym2021081
Wallbott, H. G. (1998). Bodily expression of emotion.
European Journal of Social Psychology, 28, 879–896.
http://dx.doi.org/10.1002/(SICI)1099-0992
(1998110)28:6<879::AID-EJSP901>3.0.CO;2-W
Wass, P., Waldenström, U., Rössner, S., & Hellberg,
D. (1997). An android body fat distribution in
females impairs the pregnancy rate of in-vitro
fertilization-embryo transfer. Human Reproduction,
12, 2057–2060. http://dx.doi.org/10.1093/
humrep/12.9.2057
Watson, P. J., & Thornhill, R. (1994). Fluctuating
asymmetry and sexual selection. Trends in
Ecology and Evolution, 9, 21–25. http://dx.doi.
org/10.1016/0169-5347(94)90227-5
Weisfeld, G. E., & Dillon, L. M. (2012). Applying the
dominance hierarchy model to pride and shame,
and related behaviors. Journal of Evolutionary
Psychology, 10, 15–41. http://dx.doi.org/10.1556/
JEP.10.2012.1.2
Weisfeld, G. E., & Goetz, S. M. M. (2013). Applying
evolutionary thinking to the study of emotion.
Behavioral Sciences, 3, 388–407. http://dx.doi.
org/10.3390/bs3030388
Weston, E. M., Friday, A. E., & Liò, P. (2007).
Biometric evidence that sexual selection has shaped
the hominin face. PLoS ONE, 2, e710.
White, T. D., Asfaw, B., Beyene, Y., Haile-Selassie, Y.,
Lovejoy, C. O., Suwa, G., & WoldeGabriel, G.
(2009, October 2). Ardipithecus ramidus and the
paleobiology of early hominids. Science, 326, 64,
75–86. http://dx.doi.org/10.1126/science.1175802
Wienke, C. (1998). Negotiating the male body: Men,
masculinity, and cultural ideals. Journal of Men’s
Studies, 6, 255–282.
Williams, M. A., & Mattingley, J. B. (2006). Do angry
men get noticed? Current Biology, 16, R402–R404.
http://dx.doi.org/10.1016/j.cub.2006.05.018
Winkielman, P., Berridge, K. C., & Wilbarger, J. L.
(2005). Unconscious affective reactions to masked
75
Frank and Shaw
happy versus angry faces influence consumption
behavior and judgments of value. Personality and
Social Psychology Bulletin, 31, 121–135. http://dx.doi.
org/10.1177/0146167204271309
Wrangham, R. W., & Peterson, D. (1996). Demonic males:
Apes and the origins of human violence. Boston, MA:
Houghton Mifflin Harcourt.
Zaadstra, B. M., Seidell, J. C., Van Noord, P. A.
H., te Velde, E. R., Habbema, J. D., Vrieswijk,
B., & Karbaat, J. (1993). Fat and female
fecundity: Prospective study of effect of body fat
76
distribution on conception rates. British Medical
Journal, 306, 484–487. http://dx.doi.org/10.1136/
bmj.306.6876.484
Zahavi, A. (1975). Mate selection—A selection for
a handicap. Journal of Theoretical Biology, 53,
205–214. http://dx.doi.org/10.1016/0022-5193
(75)90111-3
Zaidel, D. W., Aarde, S. M., & Baig, K. (2005).
Appearance of symmetry, beauty, and health in
human faces. Brain and Cognition, 57, 261–263.
http://dx.doi.org/10.1016/j.bandc.2004.08.056
 Chapter 4
The Cultural Bases of
Nonverbal Communication
David Matsumoto and Hyisung C. Hwang
Culture has strong and pervasive influences on
nonverbal communication. In America and Europe,
people greet each other with a handshake and a
smile. In East Asia, people bow to each other with
their hands at their sides, whereas in Thailand,
people bow with their hands in front of them as
if in prayer. In the Middle East, people bow with
their hands on their hearts. This chapter addresses
the question of how and why these cultural dif-
ferences occur—the cultural bases of nonverbal
communication—focusing on cultural influences
on nonverbal behaviors.
We begin this chapter with a discussion of the
origin and definition of culture because we believe
that understanding the cultural bases of nonverbal
communication requires a working definition of
culture. Adopting a working definition of culture
requires, in turn, a discussion of the origins of cul-
ture. We then describe how culture influences non-
verbal behaviors through the important mediating
variable of context. Culture does so by allowing
people to create rules that are known as cultural
norms that provide guidelines for appropriate
behavior in specific contexts. Cultures also help
people to create sanctions and punishments for
norm transgressions. We believe that the func-
tion of culture is to provide guidelines for behavior
that promote social coordination and reduce the
potential for social conflict, which, in turn, facilitate
group efficiency and ultimately aid survival.
With this basic information concerning culture
and its functions, we then discuss the cultural
regulation of nonverbal communication. We argue
http://dx.doi.org/10.1037/14669-004
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
that the function of communication is to allow for
the sharing of social intentions, which facilitates
social coordination. Cultural norms provide rules
for the regulation of expressive behaviors, includ-
ing nonverbal behaviors, to allow for the sharing
of social intentions as part of communication.
We further argue that this underlying function of
nonverbal communication vis-à-vis the function
of culture is universal; the cultural norms and the
manifestation of those norms in actual behav-
ior, however, are different because of the various
adaptations different human groups have made to
survive in their ecological contexts. That is why
there are differences across cultures. We review
literature examining cultural differences in various
channels of nonverbal communication and link
those differences to the function of cultural norms.
Behaviors make sense when viewed from the per-
spective of the culture in which the behavior is
produced; they facilitate social coordination and
reduce social chaos in those contexts. However, cul-
tural differences in behavior can produce conflict
in intercultural situations, a topic with which we
end this chapter.
WHAT IS CULTURE?
The Origins of Culture
Gaining a better understanding of culture requires
first a framework for understanding the sources
of its origins. We believe that there are four such
sources: ecology, resources, group life, and the
evolved human mind and brain.
77
Matsumoto and Hwang

Ecology.  Humans exist in specific ecologies, and land in that area—the amount of land on which food
those ecologies have a major impact on the devel- can grow to sustain the people in that area.
opment of culture. One aspect of ecology that influ- There are other ecological factors that influence
ences culture and that has received much research the development of culture as well. For instance,
attention is climate, and in particular deviation from global changes in climate across history have
temperate climate (Van de Vliert, 2009). Humans affected human evolution (Behrensmeyer, 2006),
need to regulate their body temperatures and have as has the incidence and prevalence of infectious
an easier time doing so in temperate climates, diseases in different regions of the world (Murray &
which happens to be around 22 °C (about 72 °F). Schaller, 2010). Contact with other cultures will
Much colder or hotter climates make life more dif- also affect a culture; this is especially true for
ficult and demanding, and these harsher climates immigrants, who come to a land with an already
require people to do more to adapt to their natural existing culture and must deal with the process of
environment. Harsher climates create greater risks acculturation.
of food shortage and spoilage, stricter diets, and
Resources.  A second source that influences the
more health problems; furthermore, infectious and
development of cultures is resources. These include
parasitic diseases tend to be more frequent in hotter
natural resources such as the presence or absence of
climates, and death from exposure is more frequent
water or land to grow food or to raise animals.
in colder climates (Matsumoto & Fletcher, 1996).
A land void of natural resources will encourage
Demanding climates require special clothing, hous-
teamwork and community spirit among its mem-
ing, and working arrangements as well as special
bers and interrelationships with other groups that
organizations for the production, transportation,
have abundant resources in order to survive and
trade, and storage of food. People in hotter climates
meet essential needs. In a land with plentiful natural
tend to organize their daily activities more around
resources, a group will have less need for attributes
shelter, shade, and temperature changes that occur
linked to social coordination or cooperation; in a
during the day. For example, Spanish culture
land with fewer natural resources, groups require
encourages the stopping of work in the midaft-
greater cooperation and cohesion.
ernoon, during the hottest time of the day, and
A major type of resource that influences cul-
reopening later, pushing back the working hours.
tures today is money or affluence, which itself is
There, it is not uncommon for people to have
a human cultural creation (Van de Vliert, 2009).
dinner outside at 11:00 p.m. or even midnight.
Abundant money can help buffer the consequences
People who live nearer the poles may organize their
of a lack of resources and harsh climates, which, in
lives around available sunlight. In psychological
turn, have interesting psychological consequences.
terms, more demanding cold or hot climates elicit a
People and groups with more money can afford to
chain of needs shared by all inhabitants in an area
be less in sync with others, as cooperation is not as
(Van de Vliert, 2009).
essential for survival. People and groups with less
Another ecological factor that influences the
money need to cooperate to survive. Importantly,
development of culture is population density. This
the interaction between ecological factors and afflu-
is the ratio of the number of people that live in an
ence drives the development of cultures (Van de
area relative to the size of the area. Some areas have
Vliert, 2009). Groups that live in harsh climates but
lots of people living in a very small space, like New
that have the resources to deal with them produce
York City, Tokyo, Hong Kong, or Mexico City; that
ways of living that are different than groups that live
is, they have large population density. Other areas
in harsh climates but that do not have the resources
have only a few people living in a very large area,
(i.e., money) to deal with those climates.
like Alaska, Greenland, or the northern island of
Hokkaido in Japan; they have low population den- Group life.  A third source for the origin of human
sity. What affects cultures even more is the number culture is the fact that humans are social animals and
of people in an area relative to the amount of arable live in groups. Humans learned many hundreds of

78

thousands of years ago that living in groups was better
than living alone. However, there are consequences to
group life. On one hand, a major advantage of group
life is that groups are more efficient because they
allow members to divide labor. This allows groups to
accomplish more than any one individual can, which
is functional and adaptive for all group members,
thereby increasing the potential for survival. On the
other hand, a disadvantage is the potential for social
conflict and chaos because individual members are
very different. Social coordination, therefore, becomes
very important to reduce the possibility of conflict
and chaos and to increase efficiency.
The evolved human mind and brain.  A fourth
source for the origin of human cultures is the
evolved human mind and brain. One major com-
ponent of the human mind is the fact that humans
have basic needs that are ultimately related to repro-
ductive success (Boyer, 2000; Buss, 2001). These
include physical needs—the need to eat, drink,
sleep, handle waste, and reproduce—and safety
and security needs—the need for hygiene, shelter,
and warmth (recall our earlier discussion about the
importance of climate). These needs are universal to
all people of all cultures.
Over the course of history, people needed to
solve a host of distinct social problems to adapt
and, thus, to achieve reproductive success. These
social problems include forming and then negotiat-
ing complex status hierarchies, forming successful
work and social groups, attracting mates, fighting
off potential rivals for food and sexual partners, giv-
ing birth and raising children, and battling nature
(Buss, 1988, 1991, 2001). In fact, we need to do
these things in our everyday lives today as well.
Thus, basic needs are associated with social motives
(Hogan, 1982; Sheldon, 2004), which include the
motive to achieve and the motive to affiliate with
others, and these social motives are part of the
evolved human mind and brain.
Nature and evolution endowed humans with
a set of cognitive capacities that aid humans in
adapting to environments and addressing their
needs. One of these is language. Humans have the
unique ability to symbolize their physical and meta-
physical world (Premack, 2004), to create sounds
The Cultural Bases of Nonverbal Communication
representing those symbols (morphemes), to create
rules connecting those symbols to meaning (syn-
tax and grammar), and to put this all together in
sentences. Moreover, humans have developed writ-
ing systems to reduce oral expressions to words on
paper. This book is a uniquely human product.
Other cognitive abilities that humans are
endowed with are those that allow for complex
social cognition. One of the most important
thinking abilities that humans have is the abil-
ity to believe that other people are intentional
agents—that is, that others have wishes, desires, and
intentions to act and behave (Tomasello, Carpenter,
Call, Behne, & Moll, 2005). Furthermore, people
know that others know that they have intentions.
Being in the “public eye,” therefore, takes on special
meaning for humans, because they know that oth-
ers can make judgments about them. Humans also
have causal beliefs, which form the basis for attribu-
tions. Morality, a uniquely human product, is rooted
in this unique human cognitive ability. This ability
apparently turns on in humans around 9 months of
age (Tomasello, 1999), which is a critical time of
development of many cognitive abilities.
Other animals can and do view themselves as
somewhat intentional agents. However, one thing
that differentiates humans from other animals is
the fact that we have the cognitive ability to share
our intentions with others. One of the major func-
tions of language is to allow for us to communicate a
shared intentionality (Tomasello & Herrmann, 2010;
we return to this point later in this chapter). Shared
intentionality may be at the heart of social coordina-
tion, which allows for the creation of human culture
(Fiske, 2000).
Another important ability that humans have is
the ability to continually build upon improvements.
When humans create something that is good, it
usually evolves to a next generation in which it is
even better. This is true for computers, cars, audio
music players, and, unfortunately, weapons. Toma-
sello, Kruger, and Ratner (1993) have called this the
ratchet effect. Like a ratchet, an improvement never
goes backward; it only goes forward and continues
to improve on itself. The ratchet effect does not
occur in other animals; monkeys may use twigs to
catch insects, but they never improve on that tool.
79
Matsumoto and Hwang
Our cognitive skills also include memory, which
allows for the creation of histories, which, in turn,
allows for the creation of traditions, customs, and
heritage (Balter, 2010; Liu et al., 2005, 2009;
Paez et al., 2008; Wang, 2006; Wang & Ross, 2007).
Our cognitive skills also include the ability to think
hypothetically and about the future. This not only
allows us to plan things but also to worry about the
uncertainty of the future, both of which form the
basis of important cultural practices.
People are also equipped with the ability to have
emotions. Emotions are rapid, information process-
ing systems that have evolved to aid humans in
reacting to events that require immediate action and
that have important consequences to one’s welfare
with minimal cognitive processing (Cosmides &
Tooby, 2000). Although emotions are part of an
archaic, biologically innate system that we share
with other animals, they also combine with our
advanced cognitive abilities to produce uniquely
human emotions that coevolved to aid humans in
solving complex social problems (Matsumoto &
Hwang, 2012).
Finally, people come equipped with personality
traits. Humans around the world share a core set
of traits that give them predispositions to adapt to
their environments, to solve social problems, and
to address their basic needs. Most research in this
area has focused on what is known as the Big Five
set of personality traits: Extraversion, Openness,
Neuroticism, Conscientiousness, and Agreeableness
(McCrae & Costa, 1999; McCrae & Terracciano,
2005). Other traits may also be universal (Cheung,
van de Vijver, & Leong, 2011).
Summary.  Humans live in groups, and groups
exist in different ecologies and resources. Humans
have needs that must be met to survive. Fortunately,
humans do not come to the world as blank slates;
they come preequipped with an evolved, naturally
selected set of abilities and aptitudes that allows
them to adapt and survive. Groups of people need
to adapt their behaviors to their ecologies to maxi-
mize the use of their available resources to meet
their needs; the abilities and aptitudes in the evolved
human mind and brain give them the tools to adapt.
These adaptations produce behaviors, ways of living,
80
ways of thinking, and ways of being. These ways
become part of the contents of a group’s culture.
Group life requires social coordination, which
facilitates group efficiency. If groups are not
coordinated, there is social chaos. Thus, humans
need to keep social order and be coordinated to
accomplish tasks efficiently and to survive. Human
groups produce rules for behavior to facilitate coor-
dination, and create sanctions and punishments for
behavior that threatens coordination. To achieve
social order and coordination and to avoid chaos,
human groups therefore must create rules, or sys-
tems of living, and ways of being. This is culture.
Culture, therefore, provides a system that facilitates
social coordination while reducing social chaos
to maximize group efficiency and to ultimately
facilitate survival and well-being. At the same time,
cultural differences occur because different groups
produce different solutions to the problem of adap-
tation because of differences in the contexts in
which they exist.
A Definition of Culture
The term “culture” is an abstract metaphor for the
ways that each group develops to meet needs related
to survival. As a metaphor, culture refers to a mul-
titude of concepts. Culture can be used to describe
activities or behaviors, refer to heritage or tradi-
tion of a group, describe rules and norms, describe
learning or problem solving, define the organiza-
tion of a group, or refer to the origins of a group
(Berry, Poortinga, Segall, & Dasen, 1992; Kroeber &
Kluckhohn, 1952/1963). Culture can refer to gen-
eral characteristics; food and clothing; housing and
technology; economy and transportation; individual
and family activities; community and government;
welfare, religion, and science; and sex and the life
cycle (Barry, 1980; Berry et al., 1992; Murdock,
Ford, & Hudson, 1971). Laypersons use the concept
of culture to describe and explain a broad range of
activities, behaviors, events, and structures.
Over the years, many scholars have attempted
to define culture. Tylor (1865) defined culture as
all capabilities and habits learned as members of a
society. Linton (1936) referred to culture as social
heredity. Kroeber and Kluckhohn (1952/1963)
defined culture as patterns of and for behavior

acquired and transmitted by symbols, constituting
the distinct achievements of human groups, includ-
ing their embodiments in artifacts. Rohner (1984)
defined culture as the totality of equivalent and
complementary learned meanings maintained by a
human population, or by identifiable segments of
a population, and transmitted from one generation
to the next. Jahoda (1984) argued that culture is a
descriptive term that captures not only rules and
meanings but also behaviors. Pelto and Pelto (1975)
defined culture in terms of personality, whereas
Geertz (1975) defined it as shared symbol systems
transcending individuals. Berry et al. (1992) defined
culture simply as the shared way of life of a group of
people, and Baumeister (2005) defined culture as an
information-based system that allows people to live
together and to satisfy their needs.
Culture has been defined in many diverse
ways because the concept of culture covers broad
domains related to almost anything and everything
about human activities or products. Moreover,
contemporary cultures are changing and evolving
quickly; thus, defining cultures is challenging. We
define culture as a unique meaning and information
system, shared by a group and transmitted across
generations, that allows the group to meet basic needs
of survival, pursue happiness and well-being, and
derive meaning from life.
Culture exists first to enable groups to meet basic
needs of survival. Culture helps people meet others,
procreate and produce offspring, put food on the
table, provide shelter from the elements, and care
for our daily biological essential needs. However,
human culture also allows for complex social net-
works and relationships, and provides rules to regu-
late and reduce (but not eliminate) the inevitable
conflicts that emerge. It allows humans to enhance
the meaning of normal, daily activities. It allows
the pursuit of happiness. It allows humans to be
creative in music, art, and drama; to seek recreation
and to engage in sports and organized competition,
whether in the local community Little League or
the Olympic Games; to search the sea and space;
and to create mathematics, an achievement no other
species can claim, as well as an educational system.
Human culture allows people to go to the moon, to
create a research laboratory on Antarctica, and to
The Cultural Bases of Nonverbal Communication
send probes to Mars and Jupiter. Unfortunately, it
also allows humans to have wars, to create weapons
of mass destruction, and to create terrorists.
Human culture does all this by creating and
maintaining complex social systems, institutionaliz-
ing and improving cultural practices, creating beliefs
about the world, and communicating the meaning
system to other humans and subsequent genera-
tions. It is the product of the evolution of the human
mind and complex cognitive abilities in response
to the specific ecologies in which groups live and
the resources available to them to live. Culture as
a meaning and information system results from
the interaction between universal biological needs
and functions, universal social problems created to
address those needs, and the ecological environ-
ment in which people live. Culture is a solution to
the problem of groups’ adaptations to their contexts
to address their social motives and biological needs.
As adaptational responses to the environment, cul-
tures help to select behaviors, attitudes, values, and
opinions that may optimize the tapping of resources
to meet survival needs. Those guidelines are passed
along from one generation to the next so that future
generations do not have to keep reinventing the
wheel. Cultural products are always ratcheted up,
never down.
The Elements of Culture
The elements of culture can be divided roughly into
two major categories—the objective and subjec-
tive elements (Kroeber & Kluckhohn, 1952/1963;
Triandis, 1972). The objective elements of culture
involve objective, explicit elements that are physi-
cal, such as architecture, clothes, foods, art, eating
utensils, and the like; objective elements of culture
typically survive people as physical artifacts. The
subjective elements of culture include all those
parts of a culture that do not survive people as
physical artifacts; thus, they include psychological
processes such as attitudes, values, beliefs, norms,
and such. Understanding the cultural bases of
nonverbal communication requires a focus on the
subjective elements of culture.
Many domains of the subjective elements of
culture have been studied, such as values (Hofstede,
2001; Schwartz, 2004), beliefs (M. H. Bond et al.,
81
Matsumoto and Hwang
2004; Leung et al., 2002), attitudes, worldviews and
self-concepts (Heine & Hamamura, 2007; Sedikides,
Gaertner, & Vevea, 2005), and norms. Of these
various domains, norms are the most relevant for
our discussion. Norms are generally accepted stan-
dards of behavior for any cultural group. Norms
comprise behaviors that members of a culture have
defined as the most appropriate in any given situ-
ation. Recent research has uncovered norms for
describing the behaviors of people of other cul-
tures (Shteynberg, Gelfand, & Kim, 2009) as well
as norms for controlling one’s expressive behavior
when emotional (Matsumoto, Yoo, et al., 2009;
Matsumoto et al., 2008).
Not only do norms proscribe socially appro-
priate, desirable behaviors but they also define
what is inappropriate and undesirable. Thus,
cultures generate social sanctions and punish-
ments for norm transgressions. These sanctions
and punishments may be explicit, as in the letter
of the law and jail time, to implicit, such as social
isolation.
Normal behavior is related to social rituals in
cultures. Rituals are culturally prescribed conduct
or any kind of established procedure or routine.
These include rituals for greetings, daily functions,
or religious ceremonies. Rituals are important
because they reinforce cultural meaning systems.
Some rituals are related to politeness, and many
cultures reify norms of politeness in shared behav-
ioral patterns called etiquette. This is a code of
behavior that describes expectations for social
behavior according to contemporary cultural
and conventional norms within a cultural group.
Etiquette is a large part of many cultures, although
cultures often differ in what is polite and, thus,
appropriate and “good.” Etiquette-related behav-
iors are considered signs of maturity and sanity
within each culture.
An important dimension of cultural vari-
ability with respect to norms involves a concept
known as tightness versus looseness (Pelto, 1968).
Tightness–looseness has two key components: the
strength of social norms, or how clear and pervasive
norms are within societies, and the strength of sanc-
tioning, or how much tolerance there is for deviance
from norms within societies. Pelto (1968) argued
82
that traditional societies varied in their expression
of and adherence to social norms. In his work, the
Pueblo Indians, Hutterites, and the Japanese are
examples of tight societies in which norms were
expressed very clearly and in which severe sanc-
tions were imposed on those who deviated from
norms. By contrast, he identified the Skolt Lapps of
northern Finland and the Thais as loose societies, in
which norms were expressed through a wide variety
of alternative channels and in which there was a
general lack of formality, order, and discipline and a
high tolerance for deviant behavior (see also Gelfand
et al., 2011).
CULTURAL INFLUENCES ON BEHAVIORS:
THE IMPORTANCE OF CONTEXT
Out of all the myriad behaviors possible in the
human repertoire, cultures help to focus people’s
behaviors and attention on a few limited alterna-
tives to maximize their effectiveness, given their
resources and their environment (Poortinga, 1990).
Culture can be viewed as influencing behaviors
in several ways, depending on whether culture is
viewed as antecedent to or a consequence of behav-
ior, and whether culture is viewed as an enabler
or restrictor of behavior (Lonner & Adamapolous,
2000). Culture can be understood as enabling some
behaviors or restricting others, and via the
construction of rules that enable behavior or restrict
behavior (see Table 4.1).
Culture influences behavior by creating norms
through an important mediator: context. As inher-
ently social animals, humans live in a succession
of multiple, different contexts. Cultures produce
norms by imbuing contexts with specific meaning
and information concerning appropriate and inap-
propriate ways of thinking, feeling, and behaving.
There are many different aspects of context, includ-
ing time, place, interactants, the content of activities
or conversations, the reasons why the interactions
are occurring in the first place, and the possibility
of any future interactions between the same inter-
actants. All of these factors combine to produce the
unique contexts in which we live our lives. Next, we
focus on two major components of context: settings
and social roles.

TABLE 4.1
Different Types of Cultural Influences on Behaviors in Context
Culture viewed as Inventing behavior
Restricting behavior
Note. From Handbook of Culture and Psychology (p. 29), by D. Matsumoto, 2000, New York, NY: Oxford University Press.
Copyright 2000 by Oxford University Press. Adapted with permission.
Settings
Cultures ascribe meanings to settings. Because dif-
ferent settings are associated with different cultural
meanings, these differences influence behavior.
Being in public, for example, means something dif-
ferent than being in private, and individuals regulate
their behaviors much more in public than in private
(Matsumoto, 2012). The regulating effect of public
settings may be due to the fact that being in public
is associated with the cognitive representation that
others have knowledge of oneself and can make
causal attributions and judgments about oneself
(Baumeister, 2005; Tomasello, 1999). Thus, people
watch what they do because they are concerned
about how they will be judged by others. This con-
cern is also likely activated in the mirror effect—the
fact that individuals often regulate various aspects
of their behavior when they see themselves as others
see them (Mor & Winquist, 2002).
Despite the widespread acknowledgment of the
importance of setting, there has been very little
research in psychology on this topic. Years ago,
Altman (1975) differentiated between primary
and secondary private settings, the former being
those that people feel belong to them exclusively
and are central to their identities, the latter being
public settings that are used with such regularity
that one develops a proprietary orientation toward
them (see also Chapter 8, this handbook).
We believe that the cultural meaning of settings
revolves around three components: beliefs about being
watched, uncertainty, and anxiety. Settings are associ-
ated with a set of beliefs about the degree to which
The Cultural Bases of Nonverbal Communication
Culture viewed as
Antecedent to behavior Consequence of behavior
Culture as an enabling cause of Culture as the construction of
behavior rules that enable behavior
Culture as a restrictive cause of Culture as the construction of
behavior rules that restrict behavior
others may be watching or judging oneself and oth-
ers associated with oneself (e.g., one’s family, friends,
others of the same ethnic group, etc.). These may also
include beliefs concerning the degree to which the
judgments of others have potentially positive or nega-
tive consequences for oneself or others, and conse-
quences to future relationships with oneself or others.
Settings are also associated with some degree of
uncertainty, especially concerning one’s knowledge
about how others may think, feel, or act. Because
of this uncertainty, different settings are associated
with different degrees of anxiety about how others
may evaluate oneself. Uncertainty and ambigu-
ity inherent to settings, and the anxiety associated
with them, are likely to produce regulatory effects
on behavior. Guerin (1986) suggested that inhibi-
tory contexts, in which the emotions, behaviors,
cognitions, and intentions of others are uncertain,
influence individuals to be more cautious of their
behaviors. Inhibitory contexts are those in which
one’s behaviors are under the watchful eyes of
unknown or less familiar others, and such contexts
produce increased conformity in behaviors to
cultural norms because of the observation.
Because settings are associated with beliefs that
others may be judging oneself and that those beliefs
have potentially important consequences for oneself
and others, they are likely to produce a high degree
of regulatory effects on behavior. Norms are cre-
ated to provide this regulation. Settings associated
with the belief that one’s behavior is not judged,
that judgments of others have no consequences for
oneself, and/or in which there is little ambiguity of
83
Matsumoto and Hwang
the actions of others are likely to produce much less
regulatory effects on behavior (e.g., the anonymity
of the Internet or a dark theater). Being in a crowded
company conference room with one’s prospective
employers is a very different setting than being in an
anonymous Internet chat room, and because of these
psychological differences in the meaning of these
settings, they produce different effects on behavior.
Interactants and Social Roles
Cultures ascribe meaning to individuals and their
social roles. Social roles define the expectations for
behavior for individuals who occupy a position in a
social system. These expectations define the behaviors
that each culture expects its members to engage in to
achieve the goal of living in a culturally appropriate
fashion. Social roles are like scripts in a play (Goffman,
1959), as they delineate the types of behaviors that are
expected in specific situational contexts based on the
specific meanings ascribed to that context. Because
cultures define the meaning of situational contexts,
the scripts associated with the contexts are cultur-
ally dependent. Cultural differences in the meaning
of specific situational contexts reflect cultural differ-
ences in the specific role expectations associated with
different situations across cultures.
The interaction between individuals with
different social roles produces different culturally
prescribed meanings to interactions. At least two
studies have examined what these meanings may
be. Marwell and Hage (1970), for instance, sug-
gested the existence of three dimensions to describe
the nature of role dyads: Intimacy, Visibility, and
Regulation. McAuley, Bond, and Kashima (2002)
obtained ratings of role dyads in Australia and
Hong Kong and demonstrated the existence of four
dimensions used by persons of both cultures to
organize these relationships: Complexity, Equality,
Adversarialness, and Containment.
Summary
Mental processes and behaviors do not occur in a
vacuum, even in the laboratory (or, perhaps, espe-
cially in the laboratory); rather, they occur in a
particular setting with specific interactants, both
of which have been imbued with cultural meaning.
Norm-driven human behaviors can be considered
84
role performances within a specific context. Cul-
tures create overall and context-specific norms to
aid humans to achieve social coordination and to
avoid social chaos in a socially complex environ-
ment. Cultural norms are learned rules for think-
ing, feeling, and behaving in specific contexts that
provide guidelines and rewards for acceptable
behaviors, and sanctions and punishments for unac-
ceptable behavior. Many norms have to do with the
regulation of expressive behavior, which is associ-
ated with nonverbal behaviors and nonverbal com-
munication. Nonverbal communication cannot exist
without some basis of culture as we communicate
with others verbally and nonverbally to convey
information and to share intentions. Thus, culture
and nonverbal communication are dependent on
and influential to each other simultaneously.
CULTURAL REGULATION OF NONVERBAL
COMMUNICATION
Understanding how and why cultures regulate non-
verbal communication requires a basic understand-
ing of the function of communication in general.
Humans have a very complex and differentiated sig-
nal system in the body, including face, voice, gesture,
and whole body movements. Although other animals
share the same channels of behavior, human capabil-
ities in these channels, especially the face and voice,
are considerably more elaborate than those of non-
human animals. Moreover human communication
involves the unique use of verbal language. As men-
tioned earlier, the evolution of these communicative
abilities coincide with the evolution of the human
brain and correspond to the evolution of self–other
knowledge and the ability to know that other people
not only are intentional agents but can share inten-
tions (Tomasello et al., 1993). Therefore, we under-
stand the function of communication to be for the
purpose of facilitating the sharing of social inten-
tions. Further, we assume that evolved capabilities
for language and other modes of expressive behavior
occurred to facilitate human sharing of social inten-
tions. The sharing of social intentions allows humans
to achieve social coordination and reduces the
possibility of social chaos, in turn facilitating group
efficiency and ultimately impacting survival.
 The Cultural Bases of Nonverbal Communication

Cultural influences on communication should
affect both verbal and nonverbal communication.
With regard to the latter, we propose that cultures
produce norms that provide guidelines for the
appropriate expression of nonverbal behaviors and
other forms of nonverbal communication in specific
contexts to facilitate the sharing of social intentions.
Culturally moderated nonverbal behaviors, in turn,
increase social coordination and reduce social chaos,
thereby facilitating group efficiency and survival
(see Figure 4.1).
In broad terms, cultures have norms related to
overall expressivity that is encouraged or discour-
aged in specific cultures (see Table 4.2). Expressive
cultures are likely to facilitate the greater use of
nonverbal behaviors. These cultures have developed
norms that encourage the broad, outward expres-
sion of nonverbal behaviors, such as more animated
facial expressions and gestures, voices with higher
intensity and range, direct gaze, relaxed and open
postures, and closer interpersonal spacing. These
norms have evolved over time because they served
a purpose in specific ecological and environmen-
tal contexts that facilitated group efficiency, social
coordination, and the sharing of intentions, all of
which ultimately impacted survival. Reserved cul-
tures are more likely to facilitate the relatively less
use of nonverbal behaviors. These cultures have
developed norms that encourage restricting one’s
expressive behaviors, facial expressions, and ges-
tures and that use softer voices and avoid direct
gaze, with more rigid, closed postures at relatively
greater distances. These norms also evolved over
time because they served a purpose in specific eco-
logical and environmental contexts that facilitated
group efficiency, social coordination, and the shar-
ing of intentions, all of which ultimately impacted
survival. Nonverbal behaviors serve the same general
function across cultures (i.e., to facilitate the sharing
of social intentions); however, the norms governing
those behaviors (i.e., to be expressive or reserved)
differ depending on the ecological–cultural context
within which those behaviors occur.
The distinction between Expressive and Reserved
cultures is related to Hall’s (1966, 1973) distinction
of high- and low-context cultures, as well as Watson’s
(Watson, 1970; Watson & Graves, 1966) classifica-
tion of contact and noncontact cultures. Our dis-
tinction is different, however, as we believe there is
sufficient evidence to suggest that cultural differences
encompass the entire constellation of nonverbal
behaviors involved in interaction. At the same time,
we do not believe that there is a unidimensional, pos-
itive relationship among all of the various channels

Social
Complexity
Increases Decreases

Need for
Social Order

Culture as a
Socially Appropriate
Group Life Meaning and
Behavior
Information System

Norms Regarding
Nonverbal
Communication

Nonverbal Facilitates
Behaviors

FIGURE 4.1.  The function of nonverbal communication vis-à-vis the function of culture.

85
Matsumoto and Hwang

TABLE 4.2

A Typology of Broad Cultural Differences in Nonverbal Behaviors

Type of cultural norms

Channel Expressive Reserved
Face Animated facial expressions, frequent Fewer facial expressions, less emotions,
displays of emotions, use of face to more controlled expressions, less
amplify and illustrate speech animated expressions
Gesture More speech illustrating gestures, larger Fewer speech illustrating gestures, smaller
motions, higher frequency of emblem motions, lower frequency of emblem
usage usage
Voice Louder voices, larger range, higher speech Softer voices, diminished range, lower
rates speech rates
Gaze More direct gaze during interactions Less direct gaze during interactions
Interpersonal space and touch Closer distances in interaction, more Farther distances in interaction, less
frequent touching frequent touching
Posture More relaxed, open postures More rigid, closed postures

of nonverbal behaviors or that cultural norms exist
only for overall expressivity. Cultures also facilitate
more or less behaviors differentially across channels
(facial expressions, eye behavior, body behavior), and
there are specific norms for specific channels in spe-
cific contexts, and these depend on the channel and
behavior being regulated and their function. We next
discuss what the available evidence suggests in terms
of channel and context specificity for cultural norms.
Cultural Norms Concerning Emotions and
Emotional Expressions
Culturally moderated emotions.  The channel that
has been studied the most with regard to cultural differ-
ences and norms is that of emotional expressions, espe-
cially in the face. Cultures create norms concerning
the regulation of emotion and emotional expressions
to facilitate social coordination through the sharing of
intentions because emotions are primary motivators of
behavior (Tomkins, 1962, 1963) and have important
social functions (Keltner & Haidt, 1999). By regulat-
ing emotions via norms, cultures ensure that behaviors
follow culturally prescribed scripts, increasing social
coordination and decreasing social chaos.
Culturally driven expression regulation occurs
in multiple ways. First, culture regulates expres-
sive behaviors by calibrating the emotion system to
culturally available events so that individuals learn
what to become emotional about in the first place.
Culturally moderated emotions provide guidelines
to individuals for the kinds of emotions they should
have in specific contexts, in turn regulating the kinds
of expressive behavior that will be produced because
of differences in emotional reactions. Walking on
dark streets late at night will evoke fear in some cul-
tures but not in others, and these different emotional
reactions occur because of culturally learned differ-
ences about the meaning of that specific context.
Differences in expressive behavior, in turn, fall out of
these differences in evoked emotions.
Cultural display rule norms.  Culture also regu-
lates the nonverbal expressive behaviors that occur
as a result of emotion elicitation so that individuals
learn what kinds of emotional reactions to have after
an emotion is elicited and the range of acceptable
behaviors for individuals to engage in after emotions
occur. The norms governing these rules of expressive
displays are known as cultural display rules (Ekman &
Friesen, 1969), and they account for cultural differ-
ences in emotional expression. These are rules learned
early in childhood that help individuals manage
and modify their emotional expressions depending
on social circumstances. They were first invoked to
explain the cultural differences in expressive behavior
that occurred in Friesen’s (1972) study comparing
the facial expressions of emotion of American and
Japanese students as they watched emotionally evoca-
tive films (see also Chapter 10, this handbook).

86

After the original inception of the concept of
display rules, cross-cultural research on them was
dormant until Matsumoto’s (1990) study examin-
ing display rules in Americans and Japanese, and
a similar study documenting differences in display
rules among four ethnic groups within the United
States (Matsumoto, 1993). Later, Matsumoto and col-
leagues created the Display Rule Assessment Inven-
tory, where participants choose one of six behavioral
responses when they experience different emo-
tions with family, friends, colleagues, and strangers
(Matsumoto, Takeuchi, Andayani, Kouznetsova, &
Krupp, 1998; Matsumoto, Yoo, Hirayama, &
Petrova, 2005). They demonstrated cultural differ-
ences in display rules, and they provided evidence
for its internal and temporal reliability and for its
content, convergent, discriminant, external, and
concurrent predictive validity with personality.
Individuals of all cultures learn some degree of
overall expression regulation through display rules.
The existence of overall expression regulation via
Zimbabwe
0.51
Overall Expressivity Endorsement
Mexico Brazil
0.48 Portugal
People’s Republic of China
Turkey Japan
0.45
South Korea Lebanon Germany
Croatia
Malaysia
Russia
Indonesia
0.42
Hong Kong
0.39
20 40
Individualism
FIGURE 4.2.  Graphical representation of the relationship between indi-
vidualism and overall expressivity endorsement. From “Mapping Expressive
Differences Around the World: The Relationship Between Emotional Display
Rules and Individualism Versus Collectivism,” by D. Matsumoto et al., 2008,
Journal of Cross-Cultural Psychology, 39, p. 66. Copyright 2008 by Sage.
Reprinted with permission.
The Cultural Bases of Nonverbal Communication
display rule norms across contexts is consistent
with the idea of Expressive versus Reserved cultures
described earlier and with worldwide data concern-
ing display rules. Matsumoto et al. (2008) examined
universal and culture-specific aspects to display rules
in more than 30 countries, reporting several pancul-
tural effects. Individuals of all cultures reported that
they suppress their expressions in some contexts,
exaggerate their expressions in others, and express
their feelings as is in others. Individuals of all cul-
tures had a display rule norm for greater expressivity
toward ingroups than toward outgroups. However,
there were also cultural differences, especially linked
to culture-level individualism versus collectivism.
Collectivistic cultures were associated with less
expressivity overall than individualistic cultures,
suggesting that overall expression management for
all emotions is central to the preservation of social
order in these cultures (see Figure 4.2). This find-
ing is commensurate with the behavioral results
from previous studies (Friesen, 1972; Matsumoto &
Canada USA
Australia
Denmark
New Zealand
Belgium
India Hungary
Netherlands
Poland
Israel Italy
Switzerland
60 80 100
87
Matsumoto and Hwang
Kupperbusch, 2001; Matsumoto, Willingham, &
Olide, 2009). Individualism was positively associ-
ated with higher expressivity norms in general and
for positive emotions in particular.
Furthermore, there were culture- and context-
specificity in display rule norms. Individualism was
positively associated with endorsement of expres-
sions of all emotions toward ingroups but was nega-
tively correlated with negative emotions and was
positively correlated with happiness and surprise
toward outgroups. Thus, people in individualistic
cultures learn to express their emotions to their
ingroup members, whereas people in collectivistic
cultures learn to suppress or regulate their expres-
sions, even to their ingroups. Toward outgroups,
people in individualistic cultures learn to express
less negative and more positive emotions, whereas
people in collectivistic cultures learn to express
more negative and less positive ones. Cumulatively,
these findings suggest a fairly nuanced view of the
relationship between culture and display rules that
varies as a function of type of emotion, context, and
culture (Matsumoto et al., 2008).
Cultural Norms Regarding the Perception
of Other’s Emotional Expressions
As discussed in Chapter 10, there are many cross-
cultural similarities in how facial expressions of
emotions are perceived. Not only are the seven uni-
versal facial expressions panculturally recognized
(Elfenbein & Ambady, 2002; Matsumoto, 2001) but
there is pancultural similarity in judgments of rela-
tive intensity among faces; that is, when comparing
expressions, people of different countries agree on
which is more strongly expressed (Ekman et al.,
1987; Matsumoto & Ekman, 1989). There is also
cross-cultural agreement in the association between
perceived expression intensity and inferences
about subjective experiences (Matsumoto, Kasri, &
Kooken, 1999) and in the secondary emotions por-
trayed in an expression (Biehl et al., 1997; Ekman
et al., 1987; Matsumoto & Ekman, 1989). This agree-
ment may exist because of overlap in the semantics of
the emotion categories, antecedents and elicitors of
emotion, or in the facial configurations themselves.
However, there are cultural differences in
emotion judgments as well, and these differences
88
presumably occur because of differential cultural
norms that govern how the emotional expressions
of others should be interpreted. For example there
are cultural differences in the absolute levels of rec-
ognition across cultures; Americans typically have
higher agreement rates when judging emotions than
observers from other countries (Biehl et al., 1997;
Elfenbein & Ambady, 2002; Matsumoto, 1989, 1992;
Matsumoto et al., 2002). There are also cultural dif-
ferences in ratings of the intensity of expressions; for
example, Japanese tend to rate expressions lower in
intensity than Americans (Biehl et al., 1997; Ekman
et al., 1987; Matsumoto, 1990, 1993; Matsumoto
et al., 1999, 2002). Presumably some cultures facilitate
the perception and interpretation of others’ emotional
expressions, whereas others do not. These cultural dif-
ferences likely occur in service to communication style
preferences across cultures that allow for the sharing
of social intentions but in different ways.
One aspect of communication that is culturally
moderated refers to the degree to which cultures
moderate the relative contributions of context when
judging others’ emotions. Despite the fact that facial
expressions always occur in context in real life,
most mono- or cross-cultural judgment studies pre-
sented them fairly acontextually. Writers have long
debated the relative contribution of face and context
in contributing to emotion messages by studying
congruent and incongruent face–context combina-
tions (Bruner & Tagiuri, 1954; Ekman & O’Sullivan,
1988; Fernberger, 1928; Russell & Fehr, 1987).
One type of study examines the linkage between an
emotion-eliciting context and a facial expression,
which we have called response linkage (Matsumoto &
Hwang, 2010). Studies involving congruent response
linkages have found an additive effect (Bruner &
Tagiuri, 1954; Knudsen & Muzekari, 1983), which
probably occurred because of the increased signal
clarity in the overall emotion message when two
different signal sources provide the same message.
Interestingly, studies involving incongruent response
linkages have generally demonstrated a face pri-
macy effect, indicating that the signals in the face
tend to override the signals provided by the context
(Ekman & O’Sullivan, 1988; Ekman, O’Sullivan, &
Matsumoto, 1991; Frijda, 1969; Goldberg, 1951;
Nakamura, Buck, & Kenny, 1990).

Context effects also exist. Masuda et al. (2008)
presented faces depicting emotions imbedded
within a group of other faces also depicting emo-
tions, and they asked American and Japanese
observers to label the emotion of the central figure.
Americans were more likely to produce labels con-
sistent with the central figure despite the emotions
portrayed by the others, whereas Japanese were
more influenced by the others’ expressions when
labeling the central figure.
To clarify this literature, Matsumoto, Hwang,
and Yamada (2012) conducted two studies involv-
ing observers from the United States, Japan, and
South Korea who judged facial expressions of anger,
sadness, and happiness presented together with a
congruent or incongruent emotion-eliciting con-
text. When faces were congruent with contexts, the
agreement rates in judgments were near perfect,
with no cultural differences. This suggested that
previously documented cultural differences in emo-
tion recognition rates may have been the result of
methodological artifacts because observers were
asked to make judgments of emotion solely from
faces. In reality, such judgments are made from cues
from both faces and contexts, and when multiple
cues are given, cultural differences are eliminated.
When faces and contexts were incongruent, there
were both face and context effects, and the relative
contributions of each were moderated by culture.
American judgments were more influenced by faces,
whereas Japanese and South Korean judgments were
more influenced by context. The results provided
a more nuanced view of how culture and type of
emotion moderate judgments of faces in context by
showing how face and context effects occur simulta-
neously, and how both cultural similarities and
differences existed in the judgments.
Cultures create norms concerning the regulation
of emotion and emotional expressions to facilitate
social coordination by sharing social intentions
because emotions are primary motivators of behav-
ior (Tomkins, 1962, 1963) and have important
social functions (Keltner & Haidt, 1999). Cultures
achieve emotional expression regulation by moder-
ating the types of emotions people have in specific
contexts in the first place, by regulating people’s
expressive behaviors when emotions are elicited
The Cultural Bases of Nonverbal Communication
(via cultural display rules), and by regulating the
way in which people perceive the emotional expres-
sions of others. Norms facilitating these cultural
differences are often specific to the type of emotion
(because different emotions have different func-
tions and prime different behaviors) and context
(setting, interactant) in which emotions are elicited,
and they are even specific to gender (see Chapter 6,
this handbook). This degree of specificity occurs in
response to the complexity of the emotion system
and its functions. Cultural norms regulating this
system serve the purpose of allowing individual
members to share social intentions, thereby
facilitating social coordination.
Cultural Norms Concerning Gestures
As described in Chapter 12 in this handbook,
gestures are body movements (primarily hands,
although they occur in head and face as well) that
are used as part of the human communication sys-
tem. Gestures are interesting because they are a form
of embodied cognition; that is, they are movements
that express some kind of thought or the process of
thinking (Kinsbourne, 2006). They likely coevolved
with adaptations in our physical anatomy and cogni-
tive and language capabilities (Bouissac, 2006). This
allowed for more rapid and efficient communication
systems that went beyond words and verbal language
(Capirci & Volterra, 2008).
Two types of gestures are those that co-occur
with speech, called speech illustrators, and those that
can occur independent of speech, called emblems.
The functions of these two types of gestures dif-
fer; thus, the functions associated with the cultural
regulation (via cultural norms) of these two types of
gestures differ.
Cultural norms for speech illustrators.  Speech
illustrators are movements that are directly tied
to speech and illustrate or highlight what is being
said. There are different types of speech illustrators
(Efron, 1941; Freedman & Hoffman, 1967); all are
associated with verbal behavior on a moment-to-
moment basis (Kita et al., 2007) and are directly
tied to speech content, verbal meaning, and voice
volume. They likely occur outside of or with
minimal conscious awareness and intention.
89
Matsumoto and Hwang
The study of culture and gestures has its roots
in the work of Efron (Boas, Efron, & Foley, 1936;
Efron, 1941), who examined the gestures of Sicil-
ian and Lithuanian Jewish immigrants in New York
City. Efron found that there were distinct gestures
among immigrant Jews and Italians who adhered to
the traditional culture but that those gestures dis-
appeared as people were more assimilated into the
larger American culture, and their children adopted
the illustrators typical of Americans.
The meaning and function of illustrators are
likely similar across cultures and are likely a biologi-
cally innate product of our evolved capability for
speech. However, cultures differ in norms concern-
ing the appropriateness of both the amount and
form of illustrative gestures, and in the frequency
of illustrator usage, expansiveness, and duration.
Expressive cultures, such as Latin American and the
Middle Eastern cultures, encourage the use of large,
illustrative gestures when speaking; they are highly
animated in their gesticulation (Kendon, 1992,
1995). In Italy, for instance, one is expected to
“speak with your hands.” Other cultures are much
more reserved in their use of gestures. The British,
for example, gesticulate less than Italians when
speaking (Graham & Argyle, 1975), and large ges-
tures are considered impolite in British culture. East
Asian cultures discourage the use of such gestures,
especially when in public; thus, Asians are even
more reserved in their gesticulation. As mentioned
previously, differences in cultural norms for overall
expressivity are likely related to preferences in com-
munication styles developed in different ecologies to
facilitate the social sharing of intentions.
Cultural differences also exist in the forms of
illustrators. When counting, for example, Germans
use the thumb for one, whereas Canadians and
Americans use the index finger (Pika, Nicoladis, &
Marentette, 2009). People of different cultures
also use different gestures while describing motion
events (Kita, 2000; Kita & Ozyurek, 2003; McNeill,
2000). When pointing, people in the United States
and many Western European cultures use the index
finger. People of some other cultures, however,
learn to point with their middle finger, which of
course resembles an obscene gesture in many cul-
tures. Each of these different forms likely developed
90
for a particular reason in a culture, with the goal
of facilitating the social sharing of intentions. Even
though that reason has probably long disappeared,
the forms of the gesture likely remain through
cultural transmission across generations.
Hybrid gestures refer to gestures that are origi-
nally associated with one language but come to be
used with another, and they occur among immi-
grants and bi- or multilingual individuals. They
were first observed by Efron (1941), who reported
about a U.S.-born, Eastern Jewish individual who
used a classical Eastern Jewish culture gesture
(fist clenched, thumb outstretched, describing a
scooping motion in the air as if digging out an idea)
even when speaking English. Another type of hybrid
gesture was described by Morris, Collett, Marsh,
and O’Shaughnessy (1980), who described the com-
bining of two different gestures (the flat hand chop
threat gesture of Tunisia combined with the A-OK
ring gesture to produce a ring-chop hybrid gesture).
Other studies have documented that immigrants
often use gestures from their original culture when
using their second language (Scheflen, 1972). This
cross-linguistic transfer of gestures seems to occur
from a high-frequency gesture culture to a low-
frequency culture (Pika, Nicoladis, & Marentette,
2006) and, again, occurs because of the need to
share intentions with others. As the cultural back-
grounds of the others with whom intentions are
shared are different, the gestures also need to evolve.
Cultural norms for emblems.  Emblems are ges-
tures that convey verbal meaning independent of
words. These are also known as symbolic gestures
or emblematic gestures. Just as every culture has its
own verbal language, every culture develops its own
emblem vocabulary in gestures. Emblematic ges-
tures are culture specific (and some are gender spe-
cific; see Ekman, 1976; Friesen, Ekman, & Wallbott,
1979; Morris et al., 1980). Unlike illustrators,
emblems can occur with or without speech.
Because emblems are like speech, cultures pro-
duce norms for their development and usage just
as they do for words. Like words, emblems are pro-
duced to facilitate the sharing of social intentions.
Furthermore, just as different cultures facilitate the
differential use of vocal chords to produce different

sounds and different symbols to produce words,
they produce different symbolic gestures to rep-
resent words nonverbally, such as the peace sign
(forefinger and middle finger up, palm facing out-
ward or inward) or OK (thumb up, hand in fist)
in North American culture.
Because emblems are culture-specific, their mean-
ings across cultures are often different and some-
times offensive. The North American A-OK sign, for
example, means “OK” in some cultures, an orifice
having sexual implications in some cultures, “money”
in some cultures, and “zero” in some cultures (Morris
et al., 1980). Placing both hands at the side of one’s
head and pointing upward with the forefingers signals
that one is angry in some cultures; in others it refers
to the devil; and in others it means that one wants sex
(is horny). The reversed peace sign—forefinger and
middle finger up in a “V” shape, with the palm facing
inward—is an insult in England and Australia mean-
ing “screw you.” Thus, when emblems are interpreted
across cultures, there is large probability that indi-
viduals will misinterpret the sender’s meaning of the
emblem. Yet, they serve a functional purpose within
the culture in which it is used, and that is to facilitate
the social sharing of intentions.
In addition to the hands, humans also gesture
with their heads, the most common of which are
the emblems “yes” and “no.” In the United States, as
in many cultures of the world, these head gestures
are nods and shakes of the head. However, whereas
most people of most cultures nod their head yes and
shake their head no, some cultures of the world do
not do so. We also gesture with our bodies. In the
United States and many other cultures, the emblem
for “I don’t know” is a shrug. Shrugs are often dis-
played in our shoulders but also by our hands or
even our faces.
Although the movements associated with emblems
are culture specific, there is universality in the con-
tent themes, functions, and reasons why cultures have
a rich vocabulary of emblems (Matsumoto & Hwang,
2013a; Morris et al., 1980). Rituals concerning greet-
ings and salutations, references to locomotion or
mental states, and insults are aspects of life that occur
in all cultures, and for which it would be convenient
to be able to signal without words. Thus, all cultures
develop some emblems for these universal aspects of
The Cultural Bases of Nonverbal Communication
life. The specifics of the movements associated with
each emblem are different, however, as these are
influenced by national and linguistic boundaries,
cultural influx across history due to wars or
immigration, and the richness of the word or phrase
signaled in the verbal dictionaries of the cultures
(Morris et al., 1980).
Morris et al. (1980) argued that some emblems
arose from gesturing particular symbols. For
example, the crossed fingers for good luck was origi-
nally a surreptitious “sign of the cross” to signal to
another one was a Christian, and then it became just
the sign of the cross to ward off Satan, and now it is
just “good luck.” Interestingly, the crossed fingers
emblem did not occur in non-Christian cultures in
Morris et al.’s study.
Morris et al. (1980) called other emblems “relic”
emblems in that they were trace representations of
specific behaviors; for example, the Greek “moutza”
is an insult emblem that involves a forward hand
gesture, palm outward, with fingers spread upward.
It was the original “talk to the hand” that we now
see with younger people. The moutza is a represen-
tation of tossing garbage or urine, or possibly wip-
ing cinders or other effluent on the face of another.
Its origins were thought to be in ancient Greece
where the public would toss their garbage or urine
on prisoners as they were transported through the
streets. This no longer occurs, but that gesture
remains as a relic of that action, and today is used as
an insult or a curse.
Some emblems are becoming recognized across
cultural boundaries despite differences in origin,
such as come, go, hello, goodbye, yes, and no
(Matsumoto & Hwang, 2013a). These results are
likely being driven by the strong influence of mass
media around the world, particularly television,
movies, and the Internet, where people can view the
behaviors of others of different cultures and learn
how to decode behaviors. These technologies may
be helping to homogenize gestures into a world-
wide emblem dictionary, and if so, it may be only a
matter of time that a homogenized, universal set of
emblematic gestures replaces culture-specific ones.
Regardless of how emblems evolve across time,
however, they will continue to serve the function of
facilitating the social sharing of intentions.
91
Matsumoto and Hwang
Many emblems are associated with nonverbal
greetings. In most cultures, forms of greetings are
important social rituals that bond people together.
Many cultures differ, however, in the specific form
of the greeting, such as the eyebrow flash, the hand-
shake, the bow, bringing the hands together as if in
prayer and bowing, and the like. These differences in
forms probably arose because of context-specific dif-
ferences in the past that had some kind of meaning at
that time and that have been transmitted across gen-
erations and continue to currently survive. Regardless
of the differences in form, however, there is underly-
ing universality in the function of these nonverbal
social greetings as an important part of every culture.
Cultural Norms for Gaze
Gaze is a powerful nonverbal behavior, most likely
because of its evolutionary roots in animals (see also
Chapter 13, this handbook). Gaze is associated with
dominance, power, or aggression in both humans
and nonhuman animals (Fehr & Exline, 1987) as
well as affiliation and nurturance (Argyle & Cook,
1976). The affiliative aspects of gazing begin in
infancy (Fehr & Exline, 1987), as infants attend to
adults as their source of care and protection.
All cultures create norms concerning gaze
because both aggression and affiliation are behav-
ioral tendencies that are important for group stabil-
ity and maintenance. Cultures differ according to
whether they encourage or discourage power and
status differences in a hierarchy, with hierarchical
cultures affording power to status, but egalitarian
cultures not (Hofstede, 2001; Schwartz & Bardi,
2001). Commensurately, cultures differ in the
amounts of gaze considered appropriate when inter-
acting with others. Arabs, for example, gaze much
longer and more directly at their partners than do
Americans (Hall, 1963; Watson & Graves, 1966).
Watson (1970), who classified 30 countries as either
a contact culture (those that facilitated physical
touch during interaction) or a noncontact culture,
found that contact cultures engaged in more gazing
and had more direct orientations when interacting
with others, less interpersonal distance, and more
touching. Within the United States, ethnic groups
differ in gaze and visual behavior (Exline, Jones, &
Maciorowski, 1977; LaFrance & Mayo, 1976).
92
Status-oriented, hierarchical cultures have
norms for members to avert gaze when interacting
with higher status others as a form of deference.
Egalitarian cultures tend to have norms for members
to “look the other in the eye” when talking. These
cultural differences can have dire consequences,
especially in credibility assessment or deception
detection. Around the world, a commonly held
belief is that when people are not looking one
straight in the eye, they are likely to be lying
(Global Deception Research Team, 2006). These
beliefs influence actual judgments; in one study
(C. F. Bond, Omar, Mahmoud, & Bonser, 1990),
American and Jordanians were videotaped while
telling lies and truths, and the videotapes were
shown to other Americans and Jordanians who
made truth/lie judgments. In both cultures, indi-
viduals who avoided eye contact were judged to be
deceptive. However, there is little or no empirical
support for the belief that gaze is reliably associated
with lying (DePaulo et al., 2003).
Cultural Norms for Vocal Behavior
Nonverbal vocal cues are called paralinguistic cues
and include the tone of voice, intonation, pitch,
speech rate, use of silence, and volume (see Chap-
ter 11, this handbook). Early work on paralinguistic
cues provided evidence that some specific emotional
states were conveyed through the voice across cultures
(Beier & Zautra, 1972; Matsumoto & Kishimoto,
1983; McCluskey & Albas, 1981; Scherer, 1986)—a
view that has garnered more support in recent work
(Sauter & Eimer, 2010; Sauter, Eisner, Ekman, &
Scott, 2010; Simon-Thomas, Keltner, Sauter,
Sinicropi-Yao, & Abramson, 2009). Anger produces
a harsh edge to the voice; the voice gets louder, and
speech rates increase. Disgust produces “yuck” or
gagging sounds, whereas fear produces higher pitch
and sudden inhalations. Sadness produces softer
voices and decreased speech rates. Furthermore,
like some elements of emotional expression in the
face, there are elements in the emotion expression
in the voice that seem to be involuntary (Sauter &
Eimer, 2010).
The voice is used to illustrate and amplify
speech, and much like speech illustrating gestures,
cultures produce norms that moderate the use of

these vocal characteristics in social interaction.
Expressive cultures encourage the use of relatively
louder voices with higher speech rates, whereas
reserved cultures use softer voices with lower
speech rates, especially in specific contexts.
Additionally, pronunciations of some languages
require the production of different sounds and
rhythms in the voice that may be associated with
different emotions (e.g., the guttural quality of some
Germanic languages, the up and down rhythms
of Mandarin). Although these vocal cues sound
normal in the cultures in which they originated,
in other cultures, it is easy to have negative reac-
tions to these because they sound different and
are associated with aversive emotions.
Unfortunately, little cross-cultural research on
this topic exists. In one of the very few studies in
this area, Matsumoto and Hwang (2013b) compared
the vocal characteristics of European Americans
with Chinese, Hispanic, and Middle Eastern immi-
grants as they participated in various interviews
concerning a mock crime. Across all questions
and interviews, the Hispanic and Middle Eastern
immigrants consistently spoke with higher vocal
intensity and with higher speech rates than the Chi-
nese and European Americans. These cultural dif-
ferences open the door to examining how they may
be related to the facilitation of the social sharing of
intention differentially in different cultures.
Cultural Norms for Interpersonal Space
and Touch
The use of space in interpersonal interactions is
called proxemics (see Chapter 15, this handbook).
Hall’s (1966, 1973) classic work in this area speci-
fied four levels of interpersonal space use depend-
ing on social relationship type: intimate, personal,
social, and public. He suggested that interpersonal
distance helps to regulate intimacy by control-
ling sensory exposures, because the possibility
of sensory stimulation (smells, sights, touch) is
enhanced at closer distances. Hall suggested that in
the United States, intimate distances are less than
18 in. (45.72 cm), personal distances range from 18
in. (45.72 cm) to 4 ft (1.2 m), social distances range
from 4 ft (1.2 m) to 12 ft (3.66 m), and pubic dis-
tances are greater than 12 ft (3.66 m).
The Cultural Bases of Nonverbal Communication
This meaning and function of space is a universal
aspect of life that exists across cultures. Thus, cul-
tures must regulate the use of space, as such regula-
tion is necessary for social coordination; violations
of space bring about aversive reactions (Sussman &
Rosenfeld, 1978). People of all cultures appear to
use space according to these four distinctions, but
they differ in the size of the spaces they attribute to
them. A study of people from five different cultures
(American, Swedish, Greek, Southern Italian, and
Scottish), for example, showed that the cultures
were similar in the order of the distances for differ-
ent types of transactions but that there were signifi-
cant mean differences in the actual distances used
(Little, 1968). Cultures around the Mediterranean,
Middle East, or of Latin origin interact at closer dis-
tances. Arab males tend to sit closer to each other
than American males, with more direct, confronta-
tional types of body orientations (Watson & Graves,
1966). They also use greater eye contact and speak
in louder voices. Arabs, at least in the past, learned
to interact with others at distances close enough to
feel the other person’s breath (Hall, 1963, 1966).
Latin Americans tend to interact more closely than
do individuals of European backgrounds (Forston &
Larson, 1968), and Indonesians tend to sit closer
than Australians (Noesjirwan, 1977, 1978). Italians
interact more closely than either Germans or Ameri-
cans, and Colombians interact at closer distances
than do Costa Ricans (Shuter, 1976). When interact-
ing with someone from their same culture, Japanese
sat the farthest away, Venezuelans the closest, with
Americans somewhere in the middle (Sussman &
Rosenfeld, 1982). Interestingly, in the same study,
when the nonnative English speakers spoke in
English, they adopted the American conversational
distance compared to when speaking with others
from their home country in their native language.
Cultural differences in the use of space even occur
when individuals set dolls to interact with each
other (Little, 1968).
In addition to cultural norms, another major
factor determining the amount of space used in
interpersonal interactions is the relationship of the
interactants; the specific content or affective tone
of the interaction is also important (Little, 1968).
Furthermore, much of the information we have
93
Matsumoto and Hwang
about the use of interpersonal space involves dyadic
interaction. We have much less research informa-
tion about the use of space among groups of people,
especially strangers, across cultures.
Cultures also regulate touch. The meaning and
function of touch is likely similar across cultures.
Cultures differ in the amounts of touching behavior
deemed acceptable, which facilitates social coor-
dination. As mentioned earlier, Watson (1970)
classified 30 countries as either a contact culture
or a noncontact culture. Violations of the cultural
rules regarding touch are likely to be interpreted in
the same way as those of space, producing aversive
consequences.
Cultural Norms for Whole Body
Movements
As discussed in Chapter 15 of this handbook,
postures communicate attitudinal states and general
affect, as opposed to the specific emotions commu-
nicated by face and voice. There is surprisingly little
cross-cultural research on the production or inter-
pretation of the meaning of postures across cultures.
The studies that do exist suggest that people of
different cultures interpret postures according
to the same dimensions (i.e., positivity, status),
but they place different weights of importance on
specific aspects of these dimensions (Kudoh &
Matsumoto, 1985; Matsumoto & Kudoh, 1987). One
recent study examined whole body, triumphant dis-
plays of dominance that were produced by
Olympic athletes after winning in agonistic encoun-
ters (Hwang & Matsumoto, 2014). Although all
athletes produced some degree of these whole body
expressive behaviors when they won their final
matches, cultures moderated the amount of these
behaviors produced. Athletes from status-oriented
cultures tended to produce more of these whole body
dominance behaviors than did athletes from more
egalitarian cultures. Presumably status-oriented cul-
tures facilitate the expression of nonverbal behaviors
related to dominance more than will egalitarian
cultures, apropos of the norms regarding social inter-
action and social coordination in these cultures.
Gait refers to the pattern of movement of the
body when walking. A handful of studies have
examined cross-cultural differences in gait and
94
perceptions of it. Montepare and Zebrowitz (1993)
obtained judgments from Korean observers of 5- to
70-year-old Americans as they walked from one end
of a room to the other and back, and they compared
those judgments to those previously obtained from
American observers (Montepare & Zebrowitz-
McArthur, 1988). There was cross-cultural agree-
ment in perceptions of age, sex, strength, and hap-
piness, but there were cross-cultural differences on
perceptions of dominance. The authors suggested
that some reactions to gait information may be
universal, whereas other reactions may be more
influenced by culture.
There has also been some interesting research
in the speed with which individuals across cultures
typically move through their cities (Kirkcaldy,
Furnham, & Levine, 2001; Levine & Bartlett, 1984;
Levine, Lynch, Miyake, & Lucia, 1989; Levine &
Norenzayan, 1999). These studies have demon-
strated that pace is associated with punctuality,
coronary heart disease, and a variety of attitudinal
and personality traits. Such cultural differences open
the door to future studies examining how these
differences facilitate social coordination differently
in different cultures.
CONCLUSION: THE IMPLICATION OF THE
CULTURAL REGULATION OF NONVERBAL
COMMUNICATION TO INTERCULTURAL
COMMUNICATION
Cultures facilitate different context-specific norms
related to nonverbal communication and nonverbal
behaviors because they facilitate social coordina-
tion and reduce the potential for social chaos, all
of which enhance the possibility of the increased
social sharing of intentions. These functions help
groups to be efficient and, in turn, aid ultimately in
survival. Cultural differences in nonverbal commu-
nication, therefore, make sense when viewed from
the perspective of the culture in which the behaviors
are produced.
Cultural differences in nonverbal communica-
tion, however, are often difficult to experience
and to interpret in intercultural interactions, espe-
cially because nonverbal behaviors are the “silent
language” (Henley, 1977), and many people in

intercultural interactions focus on the words being
spoken and not the nonverbal behavior. Intercul-
tural communication is more likely to be marred by
uncertainty and ambiguity (Gudykunst & Nishida,
1984; Gudykunst, Yang, & Nishida, 1985; Hogg,
Sherman, Dierselhuis, Maitner, & Moffitt, 2007),
not only because of questions concerning the verbal
messages but also because of cultural differences in
the nonverbal messages that co-occur. These differ-
ences often lead to aversive reactions that increase
the potential for misunderstanding, miscommunica-
tion, and misattributions about intent or character,
which disrupts social coordination and increases
the potential for conflict. It is easier for people from
expressive cultures to judge those from reserved
cultures as being untrustworthy, inscrutable, sly,
deceptive, or shifty. At the same time, it is easier
for people from reserved cultures to judge those
from expressive cultures as arrogant, loud, rude,
immature, or vulgar. These negative reactions occur
unconsciously and automatically because they are
rooted in cultural filters for interpreting the appro-
priateness of behavior that are developed early on
through the process of enculturation within one’s
culture.
Many of these interpretations and attributions,
however, may be misguided because the cultural
filters with which one uses to interpret the non-
verbal communications of others may not be the
cultural framework within which the person’s
behavior is rooted. One of the goals of intercultural
interactions is to reduce the uncertainty inherent
in the situation (Gudykunst, Nishida, & Chua,
1986; Gudykunst et al., 1985). One way to do this
is to realize that cultural differences in nonverbal
communications exist, that those differences exist
for a reason that makes sense either now or in the
past for that culture in which those behaviors are
imbedded, that engaging with such differences may
produce negative emotional reactions, and that
these differences and reactions are a normal and
inevitable part of the communication process. By
creating these kinds of expectations, we can begin
to ensure that intercultural interactions are not an
obstacle but instead are a platform for staging the
development and exchange of ideas and the sharing
of goals in new and exciting ways not actualized by
The Cultural Bases of Nonverbal Communication
intracultural communication. Understanding the
cultural bases of nonverbal communication is an
essential aid in our efforts to do so in today’s multi-
cultural, pluralistic world.
References
Altman, I. (1975). The environment and social behavior:
Privacy, personal space, territory, crowding. Pacific
Grove, CA: Brooks Cole.
Argyle, M., & Cook, M. (1976). Gaze and mutual gaze.
New York, NY: Cambridge University Press.
Balter, M. (2010). Evolution of behavior: Did
working memory spark creative culture? Science,
328, 160–163. http://dx.doi.org/10.1126/
science.328.5975.160
Barry, H. (1980). Description and uses of the Human
Relations Area Files. In H. C. Triandis & J. W. Berry
(Eds.), Handbook of cross-cultural psychology: Vol. 2.
Methodology (pp. 445–478). Boston, MA: Allyn &
Bacon.
Baumeister, R. F. (2005). The cultural animal: Human
nature, meaning, and social life. http://dx.doi.
org/10.1093/acprof:oso/9780195167030.001.0001
Behrensmeyer, A. K. (2006). Climate change and human
evolution. Science, 311, 476–478. http://dx.doi.
org/10.1126/science.1116051
Beier, E. G., & Zautra, A. J. (1972). Identification of vocal
communication of emotions across cultures. Journal
of Consulting and Clinical Psychology, 39, 166. http://
dx.doi.org/10.1037/h0033170
Berry, J. W., Poortinga, Y. H., Segall, M. H., &
Dasen, P. R. (1992). Cross-cultural psychology:
Research and applications. New York, NY: Cambridge
University Press.
Biehl, M., Matsumoto, D., Ekman, P., Hearn, V.,
Heider, K., Kudoh, T., & Ton, V. (1997). Matsumoto
and Ekman’s Japanese and Caucasian Facial
Expressions of Emotion (JACFEE): Reliability
data and cross-national differences. Journal of
Nonverbal Behavior, 21, 3–21. http://dx.doi.
org/10.1023/A:1024902500935
Boas, F., Efron, D., & Foley, J. P. (1936). A comparative
investigation of gestural behavior patterns in
“racial” groups living under different as well as
similar environmental conditions. Psychological
Bulletin, 33, 760.
Bond, C. F., Jr., Omar, A., Mahmoud, A., &
Bonser, R. N. (1990). Lie detection across cultures.
Journal of Nonverbal Behavior, 14, 189–204. http://
dx.doi.org/10.1007/BF00996226
Bond, M. H., Leung, K., Au, A., Tong, K. K., Reimel de
Carrasquel, S., Murakami, F., . . . Lewis, J. R. (2004).
Culture-level dimensions of social axioms and their
95
Matsumoto and Hwang
correlates across 41 cultures. Journal of
Cross-Cultural Psychology, 35, 548–570. http://
dx.doi.org/10.1177/0022022104268388
Bouissac, P. (2006). Gesture in evolutionary perspective.
Gesture, 6, 189–204. http://dx.doi.org/10.1075/
gest.6.2.04bou
Boyer, P. (2000). Evolutionary psychology and
cultural transmission. American Behavioral
Scientist, 43, 987–1000. http://dx.doi.
org/10.1177/00027640021955711
Bruner, J. S., & Tagiuri, R. (1954). The perception of
people. In G. Lindzey (Ed.), Handbook of social
psychology (Vol. 2, pp. 634–654). Cambridge, MA:
Addison-Wesley.
Buss, D. M. (1988). The evolution of human intrasexual
competition: Tactics of mate attraction. Journal
of Personality and Social Psychology, 54, 616–628.
http://dx.doi.org/10.1037/0022-3514.54.4.616
Buss, D. M. (1991). Evolutionary personality psychology.
Annual Review of Psychology, 42, 459–491. http://
dx.doi.org/10.1146/annurev.psych.42.1.459
Buss, D. M. (2001). Human nature and culture:
An evolutionary psychological perspective.
Journal of Personality, 69, 955–978. http://dx.doi.
org/10.1111/1467-6494.696171
Capirci, O., & Volterra, V. (2008). Gesture and speech:
The emergence and development of a strong and
changing partnership. Gesture, 8, 22–44. http://
dx.doi.org/10.1075/gest.8.1.04cap
Cheung, F. M., van de Vijver, F. J. R., & Leong, F. T. L.
(2011). Toward a new approach to the study of
personality in culture. American Psychologist, 66,
593–603. http://dx.doi.org/10.1037/a0022389
Cosmides, L., & Tooby, J. (2000). Evolutionary
psychology and the emotions. In M. Lewis & J. M.
Haviland-Jones (Eds.), Handbook of emotions (2nd ed.,
pp. 91–115). New York, NY: Guilford Press.
DePaulo, B. M., Lindsay, J. J., Malone, B. E.,
Muhlenbruck, L., Charlton, K., & Cooper, H.
(2003). Cues to deception. Psychological Bulletin,
129, 74–118. http://dx.doi.org/10.1037/0033-
2909.129.1.74
Efron, D. (1941). Gesture and environment. Oxford,
England: King’s Crown Press.
Ekman, P. (1976). Movements with precise meanings.
Journal of Communication, 26, 14–26. http://dx.doi.
org/10.1111/j.1460-2466.1976.tb01898.x
Ekman, P., & Friesen, W. V. (1969). The repertoire of
nonverbal behavior: Categories, origins, usage, and
coding. Semiotica, 1, 49–98.
Ekman, P., Friesen, W. V., O’Sullivan, M., Chan, A.,
Diacoyanni-Tarlatzis, I., Heider, K., . . . Tzavaras, A.
(1987). Universals and cultural differences in the
96
judgments of facial expressions of emotion. Journal
of Personality and Social Psychology, 53, 712–717.
http://dx.doi.org/10.1037/0022-3514.53.4.712
Ekman, P., & O’Sullivan, M. (1988). The role of context
in interpreting facial expression: Comment on
Russell and Fehr (1987). Journal of Experimental
Psychology: General, 117, 86–88. http://dx.doi.
org/10.1037/0096-3445.117.1.86
Ekman, P., O’Sullivan, M., & Matsumoto, D. (1991).
Confusions about context in the judgment of facial
expression: A reply to “The contempt expression
and the relativity thesis.” Motivation and Emotion, 15,
169–176.
Elfenbein, H. A., & Ambady, N. (2002). On the
universality and cultural specificity of emotion
recognition: A meta-analysis. Psychological Bulletin,
128, 203–235. http://dx.doi.org/10.1037/0033-
2909.128.2.203
Exline, R. V., Jones, P., & Maciorowski, K. (1977,
August). Race, affiliative-conflict theory, and mutual
visual attention during conversation. Paper presented
at the American Psychological Association annual
meeting, San Francisco, CA.
Fehr, B. J., & Exline, R. V. (1987). Social visual
interactions: A conceptual and literature review.
In A. W. Siegman & S. Feldstein (Eds.), Nonverbal
behavior and communication (Vol. 2, pp. 225–326).
Hillsdale, NJ: Erlbaum.
Fernberger, S. W. (1928). False suggestions and the
Piderit model. American Journal of Psychology, 40,
562–568. http://dx.doi.org/10.2307/1414334
Fiske, A. P. (2000). Complementarity theory: Why
human social capacities evolved to require cultural
complements. Personality and Social Psychology
Review, 4, 76–94. http://dx.doi.org/10.1207/
S15327957PSPR0401_7
Forston, R. F., & Larson, C. U. (1968). The dynamics of
space: An experimental study in proxemic behavior
among Latin Americans and North Americans.
Journal of Communication, 18, 109–116. http://dx.doi.
org/10.1111/j.1460-2466.1968.tb00061.x
Freedman, N., & Hoffman, S. P. (1967). Kinetic behavior
in altered clinical states: Approach to objective
analysis of motor behavior during clinical interviews.
Perceptual and Motor Skills, 24, 527–539. http://
dx.doi.org/10.2466/pms.1967.24.2.527
Friesen, W. V. (1972). Cultural differences in facial
expressions in a social situation: An experimental test
of the concept of display rules (Unpublished doctoral
dissertation). University of California, San Francisco.
Friesen, W. V., Ekman, P., & Wallbott, H. (1979).
Measuring hand movements. Journal of Nonverbal
Behavior, 4, 97–112. http://dx.doi.org/10.1007/
BF01006354

Frijda, N. H. (1969). Recognition of emotion. In L.
Berkowitz (Ed.), Advances in experimental social
psychology (Vol. 4, pp. 167–224). New York, NY:
Academic Press.
Geertz, C. (1975). From the natives’ point of view: On the
nature of anthropological understanding. American
Scientist, 63, 47–53.
Gelfand, M. J., Raver, J. L., Nishii, L., Leslie, L. M., Lun,
J., Lim, B. C., . . . Yamaguchi, S. (2011). Differences
between tight and loose cultures: A 33-nation study.
Science, 332, 1100–1104. http://dx.doi.org/10.1126/
science.1197754
Global Deception Research Team. (2006). A world of
lies. Journal of Cross-Cultural Psychology, 37, 60–74.
http://dx.doi.org/10.1177/0022022105282295
Goffman, E. (1959). The presentation of self in everyday
life. Oxford, England: Doubleday.
Goldberg, H. D. (1951). The role of “cutting” in the
perception of the motion picture. Journal of Applied
Psychology, 35, 70–71. http://dx.doi.org/10.1037/
h0062192
Graham, J. A., & Argyle, M. (1975). A cross-cultural
study of the communication of extra-verbal
meaning by gestures. International Journal
of Psychology, 10, 57–67. http://dx.doi.
org/10.1080/00207597508247319
Gudykunst, W. B., & Nishida, T. (1984). Individual
and cultural influences on uncertainty reduction.
Communication Monographs, 51, 23–36. http://dx.doi.
org/10.1080/03637758409390181
Gudykunst, W. B., Nishida, T., & Chua, E. (1986).
Uncertainty reduction in Japanese–North American
dyads. Communication Research Reports, 3, 39–46.
Gudykunst, W. B., Yang, S.-M., & Nishida, T. (1985).
A cross-cultural test of uncertainty reduction theory:
Comparisons of acquaintances, friends, and dating
relationships in Japan, Korea, and the United States.
Human Communication Research, 11, 407–454. http://
dx.doi.org/10.1111/j.1468-2958.1985.tb00054.x
Guerin, B. (1986). Mere presence effects in humans:
A review. Journal of Experimental Social Psychology,
22, 38–77. http://dx.doi.org/10.1016/0022-1031
(86)90040-5
Hall, E. T. (1963). A system for the notation of proxemic
behaviors. American Anthropologist, 65, 1003–1026.
http://dx.doi.org/10.1525/aa.1963.65.5.02a00020
Hall, E. T. (1966). The hidden dimension. New York, NY:
Doubleday.
Hall, E. T. (1973). The silent language. New York, NY: Anchor.
Heine, S. J., & Hamamura, T. (2007). In search of
East Asian self-enhancement. Personality and
Social Psychology Review, 11, 4–27. http://dx.doi.
org/10.1177/1088868306294587
The Cultural Bases of Nonverbal Communication
Henley, N. M. (1977). Body politics: Power, sex, and
nonverbal communication. Englewood Cliffs, NJ:
Prentice Hall.
Hofstede, G. H. (2001). Culture’s consequences: Comparing
values, behaviors, institutions and organizations across
nations (2nd ed.). Thousand Oaks, CA: Sage.
Hogan, R. (1982). A socioanalytic theory of personality.
In M. Page (Ed.), Nebraska symposium on motivation
(Vol. 30, pp. 55–89). Lincoln: University of Nebraska
Press.
Hogg, M. A., Sherman, D. K., Dierselhuis, J.,
Maitner, A. T., & Moffitt, G. (2007). Uncertainty,
entitativity, and group identification. Journal of
Experimental Social Psychology, 43, 135–142.
http://dx.doi.org/10.1016/j.jesp.2005.12.008
Hwang, H. C., & Matsumoto, D. (2014). Cultural
differences in victory signals of triumph. Cross-
Cultural Research, 48, 177–191. http://dx.doi.
org/10.1177/1069397113510875
Jahoda, G. (1984). Do we need a concept of culture?
Journal of Cross-Cultural Psychology, 15, 139–151.
http://dx.doi.org/10.1177/0022002184015002003
Keltner, D., & Haidt, J. (1999). Social functions
of emotion at four levels of analysis. Cognition
and Emotion, 13, 505–521. http://dx.doi.
org/10.1080/026999399379168
Kendon, A. (1992). Some recent work from Italy on
quotable gestures (“emblems”). Journal of Linguistic
Anthropology, 2, 92–108. http://dx.doi.org/10.1525/
jlin.1992.2.1.92
Kendon, A. (1995). Gestures as illocutionary and
discourse structure markers in Southern Italian
conversation. Journal of Pragmatics, 23, 247–279.
http://dx.doi.org/10.1016/0378-2166(94)00037-F
Kinsbourne, M. (2006). Gestures as embodied cognition:
A neurodevelopmental interpretation. Gesture, 6,
205–214. http://dx.doi.org/10.1075/gest.6.2.05kin
Kirkcaldy, B., Furnham, A., & Levine, R. (2001).
Attitudinal and personality correlates of a
nation’s pace of life. Journal of Managerial
Psychology, 16, 20–34. http://dx.doi.
org/10.1108/02683940110366551
Kita, S. (2000). How representational gestures
help speaking. In D. McNeill (Ed.), Language
and gesture: Window into thought and action
(pp. 162–185). http://dx.doi.org/10.1017/
CBO9780511620850.011
Kita, S., & Ozyurek, A. (2003). What does cross-
linguistic variation in semantic coordination
of speech and gesture reveal? Evidence for an
interface representation of spatial thinking
and speaking. Journal of Memory and Language,
48, 16–32. http://dx.doi.org/10.1016/S0749-
596X(02)00505-3
97
Matsumoto and Hwang
Kita, S., Ozyurek, A., Allen, S., Brown, A., Furman, R., &
Ishizuka, T. (2007). Relations between syntactic
encoding and co-speech gestures: Implications for a
model of speech and gesture production. Language
and Cognitive Processes, 22, 1212–1236. http://dx.doi.
org/10.1080/01690960701461426
Knudsen, H. R., & Muzekari, L. H. (1983). The effects of
verbal statements of context on facial expressions of
emotion. Journal of Nonverbal Behavior, 7, 202–212.
http://dx.doi.org/10.1007/BF00986266
Kroeber, A. L., & Kluckhohn, C. (1963). Culture: A
critical review of concepts and definitions.
Cambridge, MA: Harvard University.
(Original work published 1952)
Kudoh, T., & Matsumoto, D. (1985). Cross-cultural
examination of the semantic dimensions of body
postures. Journal of Personality and Social Psychology,
48, 1440–1446. http://dx.doi.org/10.1037/0022-
3514.48.6.1440
LaFrance, M., & Mayo, C. (1976). Racial differences in
gaze behavior during conversations: Two systematic
observational studies. Journal of Personality and
Social Psychology, 33, 547–552. http://dx.doi.
org/10.1037/0022-3514.33.5.547
Leung, K., Bond, M. H., de Carrasquel, S. R., Muñoz,
C., Hernández, M., Murakami, F., . . . Singelis,
T. M. (2002). Social axioms: The search for
universal dimensions of general beliefs about
how the world functions. Journal of Cross-
Cultural Psychology, 33, 286–302. http://dx.doi.
org/10.1177/0022022102033003005
Levine, R. V., & Bartlett, K. (1984). Pace of life,
punctuality, and coronary heart disease
in six countries. Journal of Cross-Cultural
Psychology, 15, 233–255. http://dx.doi.
org/10.1177/0022002184015002009
Levine, R. V., Lynch, K., Miyake, K., & Lucia, M. (1989).
The Type A city: Coronary heart disease and the pace
of life. Journal of Behavioral Medicine, 12, 509–524.
http://dx.doi.org/10.1007/BF00844822
Levine, R. V., & Norenzayan, A. (1999). The
pace of life in 31 countries. Journal of Cross-
Cultural Psychology, 30, 178–205. http://dx.doi.
org/10.1177/0022022199030002003
Linton, R. (1936). The study of man: An introduction.
New York, NY: Appleton.
Little, K. B. (1968). Cultural variations in social
schemata. Journal of Personality and Social
Psychology, 10, 1–7. http://dx.doi.org/10.1037/
h0026381
Liu, J. H., Goldstein-Hawes, R., Hilton, D., Huang, L.-L.,
Gastardo-Conaco, C., Dresler-Hawke, E., . . .
Hidaka, Y. (2005). Social representations of events
and people in world history across 12 cultures.
98
Journal of Cross-Cultural Psychology, 36, 171–191.
http://dx.doi.org/10.1177/0022022104272900
Liu, J. H., Paez, D., Slawuta, P., Cabecinhas, R.,
Techio, E., Kokdemir, D., . . . Zlobina, A. (2009).
Representing world history in the 21st century:
The impact of 9/11, the Iraq War, and the nation-
state on dynamics of collective remembering.
Journal of Cross-Cultural Psychology, 40, 667–692.
http://dx.doi.org/10.1177/0022022109335557
Lonner, W., & Adamapolous, J. (2000). Culture and
psychology: Historical perspectives and theoretical
critique. In D. Matsumoto (Ed.), The handbook of
culture and psychology (pp. 11–34). New York, NY:
Oxford University Press.
Marwell, G., & Hage, J. (1970). The organization of
role-relationships: A systematic description.
American Sociological Review, 35, 884–900.
Masuda, T., Ellsworth, P. C., Mesquita, B., Leu, J.,
Tanida, S., & Van de Veerdonk, E. (2008). Placing
the face in context: Cultural differences in the
perception of facial emotion. Journal of Personality
and Social Psychology, 94, 365–381. http://dx.doi.
org/10.1037/0022-3514.94.3.365
Matsumoto, D. (1989). Cultural influences on
the perception of emotion. Journal of Cross-
Cultural Psychology, 20, 92–105. http://dx.doi.
org/10.1177/0022022189201006
Matsumoto, D. (1990). Cultural similarities and
differences in display rules. Motivation and
Emotion, 14, 195–214. http://dx.doi.org/10.1007/
BF00995569
Matsumoto, D. (1992). American–Japanese cultural
differences in the recognition of universal facial
expressions. Journal of Cross-Cultural Psychology, 23,
72–84. http://dx.doi.org/10.1177/0022022192231005
Matsumoto, D. (1993). Ethnic differences in affect
intensity, emotion judgments, display rule attitudes,
and self-reported emotional expression in an
American sample. Motivation and Emotion, 17,
107–123. http://dx.doi.org/10.1007/BF00995188
Matsumoto, D. (2001). Culture and emotion. In D.
Matsumoto (Ed.), The handbook of culture and
psychology (pp. 171–194). New York, NY: Oxford
University Press.
Matsumoto, D. (2012). The psychological dimensions of
context. Acta de Investigación Psicológica, 2, 611–622.
Matsumoto, D., Consolacion, T., Yamada, H., Suzuki,
R., Franklin, B., Paul, S., . . . Uchida, H. (2002).
American–Japanese cultural differences in judgments
of emotional expressions of different intensities.
Cognition and Emotion, 16, 721–747. http://dx.doi.
org/10.1080/02699930143000608
Matsumoto, D., & Ekman, P. (1989). American–Japanese
cultural differences in intensity ratings of facial

expressions of emotion. Motivation and Emotion, 13,
143–157. http://dx.doi.org/10.1007/BF00992959
Matsumoto, D., & Fletcher, D. (1996). Cultural
influences on disease. Journal of Gender, Culture, and
Health, 1, 71–82.
Matsumoto, D., & Hwang, H. (2010). Judging faces in
context. Social and Personality Psychology Compass,
3, 1–10.
Matsumoto, D., & Hwang, H. (2012). Culture
and emotion: The integration of biological
and cultural contributions. Journal of Cross-
Cultural Psychology, 43, 91–118. http://dx.doi.
org/10.1177/0022022111420147
Matsumoto, D., & Hwang, H. (2013a). Cultural
similarities and differences in emblematic gestures.
Journal of Nonverbal Behavior, 37, 1–27. http://dx.doi.
org/10.1007/s10919-012-0143-8
Matsumoto, D., & Hwang, H. (2013b). Ethnic differences
in paralinguistic features of conversation. Manuscript
submitted for publication.
Matsumoto, D., Hwang, H. S., & Yamada, H. (2012).
Cultural differences in the relative contributions of
face and context to judgments of emotion. Journal
of Cross-Cultural Psychology, 43, 198–218. http://
dx.doi.org/10.1177/0022022110387426
Matsumoto, D., Kasri, F., & Kooken, K. (1999).
American–Japanese cultural differences in judgments
of expression intensity and subjective experience.
Cognition and Emotion, 13, 201–218. http://dx.doi.
org/10.1080/026999399379339
Matsumoto, D., & Kishimoto, H. (1983). Developmental
characteristics in judgments of emotion from
nonverbal vocal cues. International Journal of
Intercultural Relations, 7, 415–424. http://dx.doi.
org/10.1016/0147-1767(83)90047-0
Matsumoto, D., & Kudoh, T. (1987). Cultural similarities
and differences in the semantic dimensions of body
postures. Journal of Nonverbal Behavior, 11, 166–179.
http://dx.doi.org/10.1007/BF00990235
Matsumoto, D., & Kupperbusch, C. (2001). Idiocentric
and allocentric differences in emotional expression,
experience, and the coherence between expression
and experience. Asian Journal of Social Psychology, 4,
113–131. http://dx.doi.org/10.1111/j.1467-839X.
2001.00080.x
Matsumoto, D., Takeuchi, S., Andayani, S., Kouznetsova,
N., & Krupp, D. (1998). The contribution of
individualism–collectivism to cross-national
differences in display rules. Asian Journal of
Social Psychology, 1, 147–165. http://dx.doi.
org/10.1111/1467-839X.00010
Matsumoto, D., Willingham, B., & Olide, A. (2009).
Sequential dynamics of culturally moderated facial
expressions of emotion. Psychological Science, 20,
The Cultural Bases of Nonverbal Communication
1269–1275. http://dx.doi.org/10.1111/j.1467-9280.
2009.02438.x
Matsumoto, D., Yoo, S. H., Fontaine, J. R. J.,
Alexandre, J., Altarriba, J., Anguas-Wong, A. M.,
. . . Zengaya, A. (2009). Hypocrisy or maturity:
Culture and context differentiation. European
Journal of Personality, 23, 251–264. http://dx.doi.
org/10.1002/per.716
Matsumoto, D., Yoo, S. H., Fontaine, J. R. J., Anguas-
Wong, A. M., Arriola, M., Ataca, B., . . . Grossi, E.
(2008). Mapping expressive differences around the
world: The relationship between emotional display
rules and individualism versus collectivism.
Journal of Cross-Cultural Psychology, 39, 55–74.
http://dx.doi.org/10.1177/0022022107311854
Matsumoto, D., Yoo, S. H., Hirayama, S., & Petrova, G.
(2005). Development and validation of a measure
of display rule knowledge: The Display Rule
Assessment Inventory. Emotion, 5, 23–40. http://
dx.doi.org/10.1037/1528-3542.5.1.23
McAuley, P. C., Bond, M. H., & Kashima, E. S.
(2002). Toward defining situations objectively:
A culture-level analysis of role dyads in Hong
Kong and Australia. Journal of Cross-Cultural
Psychology, 33, 363–379. http://dx.doi.
org/10.1177/00222102033004001
McCluskey, K. W., & Albas, D. C. (1981).
Perception of the emotional content of
speech by Canadian and Mexican children,
adolescents, and adults. International Journal
of Psychology, 16, 119–132. http://dx.doi.
org/10.1080/00207598108247409
McCrae, R. R., & Costa, P. T. (1999). A five-factor
theory of personality. In L. A. Pervin & O. John
(Eds.), Handbook of personality: Theory and
research (2nd ed., pp. 139–153). New York, NY:
Guilford Press.
McCrae, R. R., & Terracciano, A. (2005). Personality
profiles of cultures: Aggregate personality traits.
Journal of Personality and Social Psychology, 89,
407–425. http://dx.doi.org/10.1037/0022-3514.
89.3.407
McNeill, D. (2000). Analogic/analytic representations and
cross-linguistic differences in thinking for speaking.
Cognitive Linguistics, 11, 43–60.
Montepare, J. M., & Zebrowitz, L. A. (1993). A cross-
cultural comparison of impressions created by
age related variations in gait. Journal of Nonverbal
Behavior, 17, 55–68. http://dx.doi.org/10.1007/
BF00987008
Montepare, J. M., & Zebrowitz-McArthur, L. (1988).
Impressions of people created by age-related
qualities of their gaits. Journal of Personality and
Social Psychology, 55, 547–556. http://dx.doi.
org/10.1037/0022-3514.55.4.547
99
Matsumoto and Hwang
Mor, N., & Winquist, J. (2002). Self-focused
attention and negative affect: A meta-analysis.
Psychological Bulletin, 128, 638–662. http://dx.doi.
org/10.1037/0033-2909.128.4.638
Morris, D., Collett, P., Marsh, P., & O’Shaughnessy, M.
(1980). Gestures: Their origins and distribution. New
York, NY: Scarborough.
Murdock, G. P., Ford, C. S., & Hudson, A. E. (1971).
Outline of cultural materials. New Haven, CT: Human
Relations Area Files.
Murray, D. R., & Schaller, M. (2010). Historical
prevalence of infectious diseases within 230
geopolitical regions: A tool for investigating
origins of culture. Journal of Cross-Cultural
Psychology, 41, 99–108. http://dx.doi.
org/10.1177/0022022109349510
Nakamura, M., Buck, R., & Kenny, D. A. (1990). Relative
contributions of expressive behavior and contextual
information to the judgment of the emotional state of
another. Journal of Personality and Social Psychology,
59, 1032–1039. http://dx.doi.org/10.1037/0022-
3514.59.5.1032
Noesjirwan, J. (1977). Contrasting cultural patterns
on interpersonal closeness in doctors: Waiting
rooms in Sydney and Jakarta. Journal of Cross-
Cultural Psychology, 8, 357–368. http://dx.doi.
org/10.1177/002202217783008
Noesjirwan, J. (1978). A rule-based analysis
of cultural differences in social behavior:
Indonesia and Australia. International Journal
of Psychology, 13, 305–316. http://dx.doi.
org/10.1080/00207597808246634
Paez, D., Liu, J. H., Techio, E., Slawuta, P., Zlobina,
A., & Cabecinhas, R. (2008). “Remembering”
World War II and willingness to fight: Sociocultural
factors in the social representation of historical
warfare across 22 societies. Journal of Cross-
Cultural Psychology, 39, 373–380. http://dx.doi.
org/10.1177/0022022108316638
Pelto, P. J. (1968). The differences between “tight” and
“loose” societies. Trans-Action, 5, 37–40.
Pelto, P. J., & Pelto, G. H. (1975). Intra-cultural
diversity: Some theoretical issues. American
Ethnologist, 2, 1–18. http://dx.doi.org/10.1525/
ae.1975.2.1.02a00010
Pika, S., Nicoladis, E., & Marentette, P. F. (2006). A
cross-cultural study on the use of gestures: Evidence
for cross-linguistic transfer? Bilingualism: Language
and Cognition, 9, 319–327. http://dx.doi.org/10.1017/
S1366728906002665
Pika, S., Nicoladis, E., & Marentette, P. F. (2009). How
to order a beer: Cultural differences in the use
of conventional gestures for numbers. Journal of
Cross-Cultural Psychology, 40, 70–80. http://dx.doi.
org/10.1177/0022022108326197
100
Poortinga, Y. H. (1990). Towards a conceptualization of
culture for psychology? Cross-Cultural Psychology
Bulletin, 24, 2–10.
Premack, D. (2004). Is language the key to human
intelligence? Science, 303, 318–320. http://dx.doi.
org/10.1126/science.1093993
Rohner, R. P. (1984). Toward a conception of culture
for cross-cultural psychology. Journal of Cross-
Cultural Psychology, 15, 111–138. http://dx.doi.
org/10.1177/0022002184015002002
Russell, J. A., & Fehr, B. (1987). Relativity in the
perception of emotion in facial expressions. Journal
of Experimental Psychology: General, 116, 223–237.
http://dx.doi.org/10.1037/0096-3445.116.3.223
Sauter, D. A., & Eimer, M. (2010). Rapid detection
of emotion from human vocalizations. Journal of
Cognitive Neuroscience, 22, 474–481. http://dx.doi.
org/10.1162/jocn.2009.21215
Sauter, D. A., Eisner, F., Ekman, P., & Scott,
S. K. (2010). Cross-cultural recognition of
basic emotions through nonverbal emotional
vocalizations. Proceedings of the National Academy
of Sciences, USA, 107, 2408–2412. http://dx.doi.
org/10.1073/pnas.0908239106
Scheflen, A. E. (1972). Body language and the social order:
Communication as behavioral control. Englewood
Cliffs, NJ: Prentice Hall.
Scherer, K. R. (1986). Vocal affect expression:
A review and a model for future research.
Psychological Bulletin, 99, 143–165. http://dx.doi.
org/10.1037/0033-2909.99.2.143
Schwartz, S. H. (2004). Mapping and interpreting
cultural differences around the world. In H. Vinken,
J. Soeters, & P. Ester (Eds.), Comparing cultures:
Dimensions of culture in a comparative perspective
(pp. 43–73). Leiden, the Netherlands: Brill.
Schwartz, S. H., & Bardi, A. (2001). Value
hierarchies across cultures: Taking a
similarities perspective. Journal of Cross-
Cultural Psychology, 32, 268–290. http://dx.doi.
org/10.1177/0022022101032003002
Sedikides, C., Gaertner, L., & Vevea, J. L. (2005).
Pancultural self-enhancement reloaded: A meta-
analytic reply to Heine (2005). Journal of Personality
and Social Psychology, 89, 539–551. http://dx.doi.
org/10.1037/0022-3514.89.4.539
Sheldon, K. M. (2004). The psychology of optimal being:
An integrated, multi-level perspective. Mahwah, NJ:
Erlbaum.
Shteynberg, G., Gelfand, M. J., & Kim, K. (2009). Peering
into the “Magnum Mysterium” of culture: The
explanatory power of descriptive norms. Journal of
Cross-Cultural Psychology, 40, 46–69. http://dx.doi.
org/10.1177/0022022108326196

Shuter, R. (1976). Proxemics and tactility in Latin
America. Journal of Communication, 26, 46–52.
http://dx.doi.org/10.1111/j.1460-2466.1976.
tb01902.x
Simon-Thomas, E. R., Keltner, D. J., Sauter, D.,
Sinicropi-Yao, L., & Abramson, A. (2009). The voice
conveys specific emotions: Evidence from vocal
burst displays. Emotion, 9, 838–846. http://dx.doi.
org/10.1037/a0017810
Sussman, N. M., & Rosenfeld, H. M. (1978). Touch,
justification, and sex: Influences on the aversiveness
of spatial violations. Journal of Social Psychology, 106,
215–225. http://dx.doi.org/10.1080/00224545.1978.
9924173
Sussman, N. M., & Rosenfeld, H. M. (1982).
Influence of culture, language, and sex on
conversational distance. Journal of Personality
and Social Psychology, 42, 66–74. http://dx.doi.
org/10.1037/0022-3514.42.1.66
Tomasello, M. (1999). The cultural origins of human
cognition. Cambridge, MA: Harvard University Press.
Tomasello, M., Carpenter, M., Call, J., Behne, T., & Moll,
H. (2005). Understanding and sharing intentions:
The origins of cultural cognition. Behavioral
and Brain Sciences, 28, 675–691. http://dx.doi.
org/10.1017/S0140525X05000129
Tomasello, M., & Herrmann, E. (2010). Ape and human
cognition: What’s the difference? Current Directions
in Psychological Science, 19, 3–8. http://dx.doi.
org/10.1177/0963721409359300
The Cultural Bases of Nonverbal Communication
Tomasello, M., Kruger, A. C., & Ratner, H. H.
(1993). Cultural learning. Behavioral and Brain
Sciences, 16, 495–552. http://dx.doi.org/10.1017/
S0140525X0003123X
Tomkins, S. S. (1962). Affect, imagery, and consciousness:
Vol. 1. The positive affects. New York, NY: Springer.
Tomkins, S. S. (1963). Affect, imagery, and consciousness:
Vol. 2. The negative affects. New York, NY: Springer.
Triandis, H. C. (1972). The analysis of subjective culture.
New York, NY: Wiley.
Tylor, E. B. (1865). Researches into the early history of
mankind and development of civilization. London,
England: John Murray.
Van de Vliert, E. (2009). Climate, affluence, and culture.
New York, NY: Cambridge University Press.
Wang, Q. (2006). Culture and the development of self-
knowledge. Current Directions in Psychological
Science, 15, 182–187. http://dx.doi.org/10.1111/
j.1467-8721.2006.00432.x
Wang, Q., & Ross, M. (2007). Culture and memory. In S.
Kitayama & D. Cohen (Eds.), Handbook of cultural
psychology (pp. 645–667). New York, NY: Guilford
Press.
Watson, O. M. (1970). Proxemic behavior: A cross-cultural
study. The Hague, the Netherlands: Mouton.
Watson, O. M., & Graves, T. D. (1966). Quantitative
research in proxemic behavior. American
Anthropologist, 68, 971–985. http://dx.doi.
org/10.1525/aa.1966.68.4.02a00070
101
 Chapter 5
The Developmental Arc of
Nonverbal Communication:
Capacity and Consequence for
Human Social Bonds
Caroline F. Keating
The birth of a baby is intensely nonverbal. There
are oohs and ahs, groans and cries, laughter and
tears. Gazes are drawn to faces, and bodies to one
another. Earthy smells, powerful tactile sensations,
and unguarded, raw emotions are released. What
the body signals and receives overwhelms anything
cast in language. This is what makes nonverbal com-
munication so extraordinary for a species known for
linguistic capabilities. During the most meaningful
moments of our lives, words give way to nonverbal
expression, etching our most significant social inter-
changes into our social-emotional brain.
As extraordinary as nonverbal communication
can be, it is also ordinary. Its lifelong, work-a-day
function is to regulate social relationships (e.g.,
Burgoon, Buller, & Woodall, 1996; DePaulo, 1992;
Riggio & Feldman, 2005; Saarni & Weber, 1999;
see also Chapter 4, this handbook). By nonverbally
messaging feelings, intentions, and desires and by
scanning others for their signals, everyday social
coordination is enabled. Sensitivity to nonverbal
messages allows us to intuit when a person seeks to
end a social interaction (e.g., intention movements),
to sense whether they like us (e.g., immediacy,
mimicry), and to anticipate when they are a threat
(e.g., anger displays). Nonverbal communication,
in both its ordinary and extraordinary form, stitches
together the social fabric that envelops us at each
life stage.
Like other human competencies, the skillful
exchange of nonverbal signals arcs across the life
span, revealing a developmental time course for
the capacity to send (encode) and receive (decode)
http://dx.doi.org/10.1037/14669-005
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
messages. The aim of this chapter is to describe
ontogenetic patterns in nonverbal communication.
Which signaling capacities develop at what age, what
processes underlie them, and of what social conse-
quence are they? To what degree are encoding and
decoding processes an outcropping of phylogenetic
or sociocultural factors? How does expressivity and
receptivity to nonverbal messages change over time?
I begin this chapter by applying basic concep-
tualizations of developmental change to nonverbal
communication capacities. These processes help
explain the emergence and modulation of expres-
sive behavior; empirical illustrations help make
the case. Added to the mix is the construct of the
developmental niche (Super & Harkness, 2002),
which embeds the complex interactions between
exogenous and endogenous shapers of nonverbal
communication within the individuals’ physical and
social world.
Nonverbal communication is inexorably linked
to the development of the senses. Thus, the cen-
tral section of the chapter is organized by selected,
signaling modalities enabled by sensory develop-
ment. I start with two of the earliest developing
systems, touch and olfaction, and progress to those
enabled by vision, meaning in this case communi-
cation through face and body movement. Missing
but covered elsewhere is what the auditory sense
enables (vocalization and paralanguage) and what
motionless, morphological signaling systems con-
vey (see Chapters 10 and 12, this handbook). The
evidence trail for the three modalities reviewed here
is tracked across the life span. Classic discoveries,
103
Caroline F. Keating
theoretical models and mechanisms (where they
exist), and “trending” developmental research are
briefly reviewed. In essence, I have tried to hit con-
ceptual and empirical highlights and to identify
gaps in the developmental science of nonverbal
communication.
The focus of this chapter is on nonverbal com-
munication in a traditional sense, whereby sig-
nals produced by one individual create change,
intended or otherwise, in another (Smith, 1977).
That people gesture when no else one is around
(Pine, Gurney, & Fletcher, 2010) is a discussion
I omit. Likewise, the role of nonverbal behavior
in learning generally as well as the importance of
the information that nonverbal actions send to the
self (embodiment, to some; emotion regulation, to
many) are largely set aside. Relationships between
gesture and language development are not covered
here (see Chapter 12, this handbook). These admit-
tedly important omissions in the developmental
story of nonverbal behavior keep this chapter true
to my given mission of emphasizing interpersonal
communication.
GENERAL PROCESSES UNDERLYING THE
EARLY DEVELOPMENT OF NONVERBAL
COMMUNICATION
Conceptualizations meant to account for develop-
ment broadly are helpful in identifying patterns in
the ontogeny of nonverbal communication. Three
stand out: biological preparedness, involuntary to
voluntary control, and differentiation. Each is influ-
enced by aspects of the child’s physical and social
environment or developmental niche (Super &
Harkness, 1986, 2002).
Biological Preparedness
Individuals are endowed with capacities to express
some traits and to develop some behaviors more
readily than others as well as to learn some things
more easily than others; these are thought of as bio-
logically prepared (Figueredo, Hammond, & McKi-
ernan, 2006; Mineka & Ohman, 2002; Tooby &
Cosmides, 1992). The constraints and opportunities
they manifest have phyletic, morphological, genetic,
life-history, and environmental bases. Therefore,
104
for example, babies less than 5-minutes-old are
drawn to faces. With gaze and head-turns, they
follow face-like stimuli more than equally com-
plex, nonface-like arrays; learning is not required
(Goren, Sarty, & Wu, 1975). However, infants are
also prepared to learn preferences for certain faces;
these preferences emerge later. In the hours and
days following birth, infants gaze longest at their
mother’s face (Field, 2007), processing the gestalt
of its particulars (Turati, Macchi Cassia, Simion, &
Leo, 2006). As early as 48 hr, they suck hardest
on an apparatus rigged with a nipple when suck-
ing places their mother’s face in view instead of a
stranger’s, thereby revealing a readily learned prefer-
ence (Field, 2007). Visual learning is not required
for infants to smile in response to a familiar voice or
tickle; without the advantage of sight, the smiling of
congenitally blind children is indistinguishable from
that of sighted children (Eibl-Eibesfeldt, 1973). It is
a neat, nonverbal package in the service of develop-
ing attachments; babies come prepared to scan faces,
recognize and prefer particular ones, and smile
when they do.
Involuntary to Voluntary Control
A second, general theme in development is the ten-
dency for behavior to shift from involuntary to vol-
untary control. The diminution of neonatal reflexes
after the first month of life is a prime example.
For example, reflexive sucking and palmer grasps
become infant-directed and purposeful over the
early weeks of development, no longer the auto-
matic responses to stimulus events characteristic
of earlier days. Similarly, newborn smiles or gri-
maces are often the result of random jostling of the
nervous system or other endogenous events. This
may be unwelcome news for parents who attribute
their sleeping baby’s smile to sweet dreams; that
dreamy look more likely indicates the passing of gas.
Luckily—but more likely by an evolutionary design
enabled by the development of the frontal cortex
(Ruff & Rothbart, 1996)—smiling soon becomes
voluntary and directed toward familiar social targets
such as caregivers. By 3 or 4 months of age, infant
provocateurs can induce parents to dance with
lampshades on their heads if that is what it takes to
elicit infant social smiles.

Differentiation
Third, like other human capacities, nonverbal com-
munication is subject to differentiation. As for the
development of neural circuits and sensorimotor
behavior, responses become less diffuse, undifferen-
tiated, and nonspecific and more distinctive, refined,
organized, and directed with development. Though
facial musculature is considerably differentiated at
birth (Oster & Ekman, 1978), the ability to use it to
express a range of discrete emotions seems less so.
Age-related trends in the production and recogni-
tion of specific facial expressions of emotion reflect
differentiation in developmental process. The degree
to which differentiation is channeled by biological
preparedness versus cultural forces is hotly debated,
and arguments are often framed using divergent
conceptualizations (e.g., Barrett, 2011; Camras &
Shutter, 2010; Ekman, Levenson, & Friesen, 1983;
Fridlund & Russell, 2006; Matsumoto, Olide,
Schug, Willingham, & Callan, 2009; Oster, 2005;
Panksepp, 2007; Scherer, Clark-Polner, & Morti-
llaro, 2011; Shariff & Tracy, 2011). The scientific
debate is far from over.
Meanwhile, in the “real” world, facial expres-
sions present as amalgams of developmental con-
straints and opportunities. You could observe, as
Lewis, Sullivan, and Vasen (1987) did, the attempts
of toddlers, young children, and adults to pose six
facial expressions of emotion (happiness, surprise,
anger, sadness, fear, disgust) in response to the
request, “Can you make a ___ face?”. Lewis et al.
videotaped their observations, compared them
to objectively derived, muscle movement con-
figurations, and scored them for correctness. The
researchers found the expressions of 2-year-olds
barely distinguishable; the upper regions of their
faces were not expressive at all. Three-year-olds pro-
duced expressions of happiness and surprise best;
4- and 5-year-olds successfully portrayed a greater
range of expressions, but their attempts were often
partial—the lower part of the face carried most of
the expressive signal. Adult expressions were most
differentiated; they conveyed nearly the full array
of distinct emotions using both upper and lower
regions of the face, the exceptions being fear and
disgust (Lewis et al., 1987). Leaving aside the excep-
tions for the moment, it took until adulthood for the
The Developmental Arc of Nonverbal Communication
other four, nonrandom, biologically patterned facial
movements to sync with cultural ideals of discrete,
prototypical emotion labels and to make a match
with theoretically objective facial measurements.
Notably, it is unlikely anybody felt much of what
they expressed in front of the camera. Nevertheless,
in terms of communicating particular emotions via
the face, the differentiation prize goes to the adults.
The Developmental Niche
In combination and coordination, these three, onto-
genetic forces—preparedness, increasing voluntary
control, and differentiation—are augmented, syn-
ergized, or checked by the evocative powers of the
niche. Super and Harkness (1986, 2002) conceptual-
ized the developmental niche as a set of dynamic prop-
erties and forces that create unique environments for
a child that the child, in turn, impacts. These include
the physical and social settings, customs and prac-
tices of child rearing, the psychology of caretakers,
characteristics of the child (e.g., sex, temperament),
and local meaning systems. Thus, the dynamic niche
shapes individual expressive capacities and perfor-
mance and is, in turn, shaped by them.
Consider some examples of the niche’s ability to
bend and sway the development of nonverbal com-
munication. Zinacanteco neonates enter the world
in a smoke-filled hut, their faces covered to restrict
visual contact (Brazelton, 1972). In other cultures,
life starts with expressions of pain as a result of inoc-
ulation, circumcision, or scarification. Cranky Hausa
babies are immediately quieted at a mother’s breast;
consequently, they survive draught conditions better
than their less expressive cohorts. Soon after birth,
traditional Baganda newborns begin formal training
to encourage social smiling; these charming babies
smile earlier than most (Kilbride & Kilbride, 1975).
German mothers frequently interact face-to-face with
their babies; Cameroonian Nso mothers rarely do.
At 12 weeks of age, German infants smile more fre-
quently and imitate their mother’s actions more than
their Nso counterparts, perhaps as a consequence
(Wörmann, Holodynski, Kärtner, & Keller, 2012).
Once a year in Japan, babies endure the screams
of men who mean to start them crying—part of a
“sumo style” baby crying contest, in which the win-
ners are the “healthy” ones who are first to cry (see
105
Caroline F. Keating
Figure 5.1.  In many cities in Japan, an annual baby cry contest is held in the style of a sumo wrestling contest.
A referee shouts at the babies to make them cry. The baby who cries first is typically declared the winner.
Photograph by The Asahi Shinbun Collection/Getty Images. Reprinted with permission.
Figure 5.1). These are just early examples; the ebb
and flow of nonverbal communication ride the tides
of the niche throughout development.
DEVELOPMENTAL TRENDS IN
NONVERBAL COMMUNICATION ENABLED
BY THE MODALITIES OF TOUCH,
OLFACTION, AND VISION
Touch, olfaction, and vision are powerful channels
through which social information is conveyed (see
also Chapters 13, 14, and 15, this handbook). The
modalities humans rely on to support nonverbal
interchange likely follow a developmental time
course rooted in cross-species patterns. Schneider,
Call, and Liebal (2012) observed the early tactile
and visual expressive behaviors of four types of
apes over the first 20 months of life. They recorded
the onset of expressive behaviors, which modali-
ties were used and how often, and the context in
which they were deployed. All four species relied
on tactile and visual signals. The young of spe-
cies most closely related to humans—bonobos,
106
chimpanzees, and gorillas—revealed a different
pattern when compared with orangutans, a more
distantly related species. Infant orangutans took lon-
ger to develop gestural communication and signaled
predominantly about food. In African apes, tactile
signals decreased over time, whereas visual signaling
increased, thereby facilitating greater mobility and
independence (Schneider et al., 2012). Surprisingly,
there are few human studies comparable to this one,
where multiple modalities are followed across time
and life histories with the goal of revealing underly-
ing processes and functions.
Thus, we do not yet have a truly comprehensive
view of nonverbal communication development.
Researchers tend to specialize in studying one signal
channel or another, rarely examining the interplay
among modalities in shaping meaning (for excep-
tions, see Colonnesi, Zijlstra, van der Zande, &
Bögels, 2012; Zhou & Chen, 2009b). Though
sequence and characteristics of movement are essen-
tial to developing nonverbal meaning, the kinds of
signals captured for laboratory study are too often
static, taxodermic renditions—well controlled but

of questionable validity. Developmental studies
of human infant communication generally focus
on gesture in the context of language’s beginnings
rather than shifts among nonverbal modalities
(e.g., Suanda & Namy, 2013). Children’s nonverbal
behavior is often used only as a convenient indi-
cator of social status or emotionality (popularity,
prosociality, dominance, mind-reading, emotion
regulation); it is less often studied as a developing
social communication competence or skill set in and
of itself. Instead, researchers from different perspec-
tives have peeled off distinct layers from the non-
verbal communication competence whole. Some
investigate links between nonverbal communication
skill and social dominance via ethological traditions
(e.g., Blurton Jones, 1971; Brannigan & Humphries,
1972; Keating & Heltman, 1994). Others emphasize
the self-regulatory (self-communicative) functions
of nonverbal competencies (e.g., Saarni & Weber,
1999) or the application of nonverbal communication
skills in therapeutic contexts and/or in special popula-
tions. This leaves an admittedly piecemeal account of
what should be a developmental whole.
What do researchers know about the basic unfold-
ing of the nonverbal channels through which critical
aspects of human communication occur? Research
on communication via touch, olfaction, and vision is
reviewed here in sequence, though these modalities
typically operate in concert. Communication reliant
on vision—facial and body expressions—receives
most of the attention, as it does in the empirical lit-
erature (see Chapters 10, 12, and 15, this handbook).
My focus is on how each mode of communication
serves the development of social bonds.
Communication Through Touch
There is touch—skin contact—and then there is
communication through touch—sensory informa-
tion produced in response to skin contact. The two
are not the same. What registers in the sensory sys-
tem and beyond shapes the signals sent via touch
(see Hertenstein & Weiss, 2011), and they are
highly consequential.
Touch and early attachment.  Sensitivity to touch
is evident in the intrauterine environment, and it
is exquisitely developed by birth in most mam-
mals (Field, 2007; Montagu, 1971). Life depends
The Developmental Arc of Nonverbal Communication
on it; the rooting reflex, whereby newborns orient
toward stimulation around the mouth, guides neo-
nates toward nourishment. Maturation is spurred
by touch as well; Baganda babies’ twice-daily, herbal
massage is believed partially responsible for their
precocious, sensorimotor development (Kilbride &
Kilbride, 1975). In decades past, immediate, con-
tinuous skin-to-skin contact between mother and
neonate was believed to be the essential experience
for healthy, emotional attachments to form, though
presently, researchers are not so sure (Hertenstein,
Verkamp, Kerestes, & Holmes, 2006). Nevertheless,
skin-to-skin contact immediately after delivery has
been associated with better breastfeeding practices
and its concomitant, positive health and social-­
emotional outcomes (Field, 2007).
Contact comfort.  The kind of communication
that touch enables is deeply embedded in phylogeny.
Young mammals biological functioning depends on
tactile contact with their warm, furry mothers and lit-
termates; processes as basic as thermoregulation may
be at the root of mammalian attachment systems
(E. E. Nelson & Panksepp, 1998). In studies with
infant monkeys, Harlow (1959) showed that absent
of a real mother, infant monkeys experienced so-
called contact comfort, or a feeling of security, when
permitted to cling to inanimate, soft, terry-cloth sur-
rogates. The tactile seductiveness of soft things is evi-
dent in human children’s preferences for attachment
objects that are soft (e.g., blankets, stuffed animals)
and in sales of the “boyfriend pillow,” a shirted, torso-
shaped pillow with an “arm” to snuggle up against.
Kangaroo care.  Greater understanding of the
value of touch led development experts in Western
countries to encourage kangaroo care, a practice
whereby newborns are held and carried on the
chests of caregivers in a cloth pack, much the way
traditional societies use slings. Among the develop-
mental benefits of kangaroo care are enhanced com-
munication outcomes for parents and babies. In one
study, the interactions between mothers and infants
were observed at 37 weeks. Those assigned to the
kangaroo care group touched their infants more and
showed more positive emotion than those assigned
to traditional care. Kangaroo care infants were also
more alert and gazed more at their mother’s face. By
3 months, both kangaroo care mothers and fathers
107
Caroline F. Keating
were more sensitive to infants compared to tradi-
tional care parents (Feldman, Eidelman, Sirota, &
Weller, 2002). Touch is a powerful ally of early bonds.
Touch and relationships.  The psychological ben-
efits of touch in early development and later in life
are impressive (for reviews, see Feldman, 2011,
2012; Stack & Jean, 2011). At any age, touch by a
trusted agent has almost magical effects on the body,
calling forth neural responses that gate-away pain or
beat it to its target (in the brain) and rallying hor-
monal reinforcements that modulate cortisol levels
and stress (Field, 2007). Most toddlers in distress
seek body contact with caregivers and are comforted
by it; most children and adults deploy hugs, strokes,
and pats to comfort others. Nonhuman primates
reveal similar patterns (see Hertenstein et al., 2006,
for a review). In fact, Dunbar (2010) views nonhu-
man primate social grooming as the evolutionary
foundation for the effects of human touch on human
social bonds. The comforting effects of touch have
proven value in many contexts across the human life
span, from comforting infants during painful medi-
cal procedures (e.g., Feldman, Singer, & Zagoory,
2010), to the benefits of massage for women in
labor as well as for their newborns (Field, 2007),
to enhancing the attentiveness of autistic children
(Field, 2011), to quieting the nerves of individuals
deciding whether to take a financial risk (Levav &
Argo, 2010).
Touch conveys information about the me, the
not-me, and about relationships between the two.
Based on observations of touch between U.S. moth-
ers and infants, Ferber, Feldman, and Makhoul
(2008) categorized nine specific behaviors into three
basic touch categories: affectionate, stimulating, and
instrumental. The high rates of affectionate and stim-
ulating touch found during the first 6 months of life
diminished during the second, as reciprocal forms
of touch increased. Learning what it means to touch
and be touched affords the opportunity to be agent
as well as target in a relationship.
For example, tickling is a lot about relation-
ships; it is nearly impossible to tickle yourself. It is a
form of touch communication common to humans
cross-culturally, to nonhuman primates, and to
other species; even rats enjoy a good tickle once in
108
a while (Burgdorf & Panksepp, 2001; Hertenstein
et al., 2006). Tickling shares with play its provoca-
tive nature, inviting interaction with a degree of
unpredictability. Thus, the earliest facial responses
to tickling signal surprise as well as joy (Bennett,
Bendersky, & Lewis, 2005).
Provine (2004) argued that human tickling, as
an essentially social endeavor, serves to express the
special bond that develops between the (vulner-
able) tickled and a trusted, tickling agent (family
member, friend, lover). Babies are tickled worldwide
as a form of play and a sign of affection. Tickling
between family members and friends is common
during infancy and childhood. By adolescence,
Provine reported that tickling bouts between other-
sex individuals are seven times more likely than
same-sex tickling bouts. Tickling continues to be a
communication tactic inviting relationships, play,
and signaling affection throughout adulthood,
though at about 40 years of age, the self-reported
frequency of tickling bouts declines dramatically.
Tickling generally stimulates laughter and smiling
(Provine, 2004), the forms of which do not neces-
sarily signal purely positive affect. Reponses to
tickling share elements of pain as well as pleasure
(Harris & Alvarado, 2005) and may reflect as much
about the relationship between tickled and tickler as
about emotion per se.
Niche aspects of touch in relationships.  In many
societies, lots of tactile contact between young off-
spring and parents is the norm and for a reason.
Among traditional hunters and gatherers, contact
is facilitated by co-sleeping, breastfeeding, late
weaning, lengthy birth spacing, and other practices
shaped by life histories (Konner, 2010). These prac-
tices reinforce familial bonds and collective identi-
ties, key elements of the developmental niche of
many traditional societies.
Modern, cultural practices dramatically dimin-
ished body contact. Daytime observations of middle-
class families in Boston, Massachusetts, in the 1960s
and 1970s revealed that infants were held by parents
approximately 5% of the time; during the other 95%
of the time, infants were “held” by cribs, playpens,
swings, or other such devices. These practices likely
reinforce notions of independent self-construals.
In Congo, Aka infants are held by an adult nearly

all of their waking hours (Konner, 2010) as part
of a lifestyle compatible with interdependent
self-construals.
Nonverbal communication habits are shaped
by and shape the niche. Studies of Palestinian
and Israeli parents and their 5-month-old infants
revealed that Palestinian parents maintained con-
stant physical contact with offspring (and with one
another) but initiated little directed touching or
mutual gaze; Israeli parents displayed the reverse
pattern (Feldman, Masalha, & Alony, 2006; Feld-
man, Masalha, & Nadam, 2001). Palestinian babies
also maintained greater levels of calm, their moods
less labile than Israeli infants. These divergent habits
of touch communication likely impact developing
self-construals, as one embedded and interdepen-
dent in harmonious relationships with others
(characteristic of traditional Palestinian society) or
as one whose autonomy and independence from
others requires effort to ensure social coordination
(characteristic of Israeli society; Feldman, 2011).
Touch tunes the self to others in these ways and, in
turn, shapes societies.
Touch and social influence.  Like other forms of
nonverbal communication, touch is also a tool for
social influence and manipulation. Infants and
toddlers are fully capable of manipulating caregiv-
ers and others using touch. Nonnutritive sucking at
mother’s breast, clinging, pulling, patting, poking,
and guiding are used by the very young to capture
and direct the attention of others. Touch delivers
messages a preverbal child finds useful during con-
flict (Ingram, 2014); a hit, kick, shove, hair pull, or
bite conveys “No,” or “I want that,” or “It’s mine,”
or “I’m not going,” or “I don’t like you right now,”
or “Go away,” or all of the above. Though undiffer-
entiated in form and meaning, these tactile signals
stand a chance of achieving objectives by expressing
immediate wants and desires, the infant’s opening
hand. More subtle forms of touch as a means of gain-
ing advantage and compliance have been pretty well
studied in adults in the West (see Knapp, Hall, &
Horgan, 2014, for a comprehensive summary), but
comparable research on children, adolescents in the
West, or adults from other cultures is lacking.
The meaning of touch depends on who does it,
how, and in what context. Social expectations for
The Developmental Arc of Nonverbal Communication
relationships between individuals belonging to dif-
ferent genders and stations in life shape the mes-
sages of touch (J. A. Hall, 2011). Other things being
equal, individuals who initiate touch are typically
perceived as having higher status than those they
touch (J. A. Hall, Coats, & LeBeau, 2005). Touch
can signal various degrees of liking and intimacy in
adult relationships. Touching and being touched
within relationships feels good (has hedonic value)
and signals the quality of the social bond (Herten-
stein, 2011). For example, National Basketball Asso-
ciation players who touched one another frequently
during early season games improved most in terms
of individual and team performance over the sea-
son; touching reflected increased cooperative play
(Kraus, Huang, & Keltner, 2010).
Touch and the communication of affect.  Touch
transmits a slew of negative and positive feelings
(Hertenstein et al., 2006). Negative tactile stimula-
tion from mothers instructed to apply pressure to
their infant’s back signaled caution, slowing infant
exploration of objects (Hertenstein & Campos,
2001). Rough touch by mothers, recorded as they
fed their 3-month-olds, predicted infant behavioral
problems at 2 years of age (Weiss, Wilson, Seed, &
Paul, 2001). On the positive side, warm, body
contact coded from family photographs predicted
their kindergartener’s smile intensity in class photo-
graphs; these “thin slices” were suggestive of over-
all family and child affective style (Oveis, Gruber,
Keltner, Stamper, & Boyce, 2009).
Touch directed to the self or self-touch has
important affect-related, communicative conse-
quences for those who observe it. Repetitive self-
touching (self-manipulation) is associated with
anxiety-provoking situations in nonhuman primates
(Aureli & Whiten, 2003) and humans (Ekman &
Friesen, 1974; Montagu, 1971). It is also believed to
be self-protective (as when hands cover eyes, block-
ing off visual information) and expressive of the
degree to which engagement during interaction with
others is perceived as desirable (e.g., defensively
crossed arms or legs; Ekman & Friesen, 1969; cf.
Meadors & Murray, 2014).
Situations in which adults feel cognitively taxed
can elicit self-touch (Egloff & Schmukle, 2002).
109
Caroline F. Keating
These include social situations to which young
children may be exposed. At least as early as 1 year
of age, babies tune in to the nonverbal exchanges
of the adults in their world (Gräfenhain, Behne,
Carpenter, & Tomasello, 2009). Thus, observant
offspring are likely to perceive the emotional tone
of how adults feel about a social interaction by
monitoring the messages that self-touch transmits
(Castelli, De Dea, & Nesdale, 2008; Castelli, Zog-
maister, & Tomelleri, 2009). In this way, infants
and young children may absorb the social biases
transmitted by adult nonverbal behavior without a
mean word spoken.
Touch signals more than general emotional tone;
touch conveys discrete, differentiated emotions.
Discrete emotions are thought to be basic, universal,
neuro-affect programs that sync particular human
feelings to expression (Ekman, 1972; Izard, 1971;
Tomkins, 1962). Hertenstein, Holmes, McCullough,
and Keltner (2009) blindfolded one member of an
unacquainted, undergraduate dyad. The other dyad
member touched the blindfolded person “appro-
priately” to convey different emotions. Each blind-
folded participant selected the meaning of each
touch from a list of nine options; of these, anger,
fear, sadness, disgust, love, gratitude, and sympa-
thy were correctly identified at rates greater than
expected, or roughly 50%–70% of the time. Different
kinds of touch were associated with different emo-
tions; for example, sympathy was communicated by
patting and rubbing, and fear was communicated by
squeezing (Hertenstein et al., 2009). The data record
from other age groups is insufficient at the time of
this writing to understand how the differentiation
of emotional touch unfolds.
Summary of developmental trends in communica-
tion through touch.  The development of commu-
nication via touch can be described in terms of the
three basic, ontogenetic patterns identified earlier:
preparedness, increasing voluntary control, and dif-
ferentiation. Typical human neonates are naturally
drawn to soft, tactile stimuli and are psychologically
as well as physically comforted by it. Early, reflexive
responses to touch, such as rooting and grasping,
are replaced by self-guided forms used to communi-
cate with others and to manipulate them. Meanings
110
of touch and of being touched become differentiated
with development. What is communicated through
a singular touch can be pegged to a discrete emotion
label by adulthood. Even before that, the meaning
tone of touch comes to depend on who is doing the
touching and in what context.
The most extraordinary feature of touch as a
communication channel is how its physiological
functions have such powerful psychological effects.
Whereas early appetites for touch keep vulnerable
infants physically close to their protectors, what
develops in parallel is a sense of self and other.
In these ways, communication through touch sup-
ports social aspects of the niche, and vice versa.
Communication Through Olfaction
Olfaction, a phylogenetically ancient sense older
even than touch, comprises a range of chemosig-
nals that organize the experience of taste, but more
importantly for social bonds, provides a nonverbal
channel for crucial information about the identity
and state of conspecifics. Chemosignals support
early attachments and later relationships, and they
guide everyday interpersonal interactions. Simi-
lar to other forms of nonverbal communication,
olfactory signals operate at both conscious and
unconscious levels of awareness, meaning that their
power to influence human social bonds is likely
underappreciated.
Olfaction and early attachment.  The mammalian
olfactory sense is operational in the fetus and at
birth, providing a critical chemosensory communi-
cation channel that not only encourages feeding but
also enables individual recognition, a key component
of developing attachments. Humans are no excep-
tion; human neonates are biologically prepared to
use olfactory cues to guide them toward their moth-
er’s body and breast. Both hardwired and experien-
tial perceptual systems seem to be involved (Schaal
et al., 2009). Experiments with newborns reveal that
they orient toward their mother’s amniotic fluid and
that they distinguish and prefer the odor of their
mother’s milk over that of a stranger’s (Field, 2007).
Mothers, too, use this powerful nonverbal channel.
New mothers can distinguish their individual new-
born’s odors from those of other infants using scents

detected on their baby’s breath and from traces of
body odors transmitted to clothing and blankets
(Porter, Cernoch, & McLaughlin, 1983). These pat-
terns reinforce the view that olfaction conveys cru-
cial information between offspring and mother early
in life (Cheal, 1975; Stevenson, 2010).
Olfaction and relationships.  Olfaction’s role in
individual identification supports social bonds later
in life. When 5- to 8-year-olds sniffed the tee-shirts
of young children, they distinguished the odors of
their younger siblings from unfamiliar children at
above chance rates (Porter & Moore, 1981). For
the older siblings, the odiferous experience was not
all that pleasant. The fact that family members do
not smell good to one another has been proposed
as an incest avoidance mechanism (Weisfeld, Czilli,
Phillips, Gall, & Lichtman, 2003).
Friends’ odors are distinctive too. Nine-year-
olds can identify friends, and college students
can identify their roommates from olfactory cues
(e.g., Mallet & Schaal, 1998; Zhou & Chen, 2009b).
Moreover, adult friends perceive smells more simi-
larly than nonfriends; this may be associated with
the overlapping genetics between them (Christa-
kis & Fowler, 2014). One study found that adults
prefer the body odors of adults who share their
political beliefs; the researchers contend that such
olfactory sensitivities may serve as a means by which
to select a compatible mate (McDermott, Tingley, &
Hatemi, 2014). Cross-species relationships are also
supported by chemosignals. Owners of “man’s best
friend” can attest to the importance dogs place on
olfactory cues for identification. It turns out that
owners themselves can discriminate the odor of
their pet dog from that of other dogs (Wells &
Hepper, 2000).
Similar to other species, humans sense the state
of conspecifics through olfaction, though we may
not be aware of the cues that attract or repel us (Li,
Moallem, Paller, & Gottfried, 2007). Nonetheless,
these cues have measurable effects in intimate ways.
It took a while before science was ready to take on
this aspect of our animal nature. For example, the
involvement of olfactory signals in human reproduc-
tion was tentatively introduced by Rogel (1978).
At that time, research appeared suggesting that
The Developmental Arc of Nonverbal Communication
olfactory cues provided a mechanism by which men-
strual synchrony occurred; later accounts confirmed
it (K. Stern & McClintock, 1998). As for other spe-
cies, it turns out that olfactory cues serve as attrac-
tants in human mating (e.g., Gildersleeve, Haselton,
Larson, & Pillsworth, 2012; Rantala, Eriksson,
Vainikka, & Kortet, 2006).
Olfaction and the communication of
affect.  Olfaction also conveys information about
emotional condition. Individuals excrete chemosen-
sory “alarm” signals through sweat glands; others
detect their signal and can identify the state of fear.
The smell of fear has as powerful an effect on elicit-
ing fearful facial expressions as does audiovisual
fear cues (de Groot, Semin, & Smeets, 2014). In
combination with visual cues from faces, ambigu-
ous facial expressions can be made to look more
fearful when chemosensory fear cues are simulta-
neously presented (Zhou & Chen, 2009a). When
facial expressions mask true feelings, olfactory cues
may have increased value, at least when negative
states are aroused. However, some might be better
than others at detecting such cues. Zhou and Chen
(2009b) found commonalities between olfactory
and visual nonverbal sensitivities. Women who
were most accurate in identifying their roommate’s
body odor from tee-shirts also displayed superior
performance in identifying basic facial expressions
of emotion (Zhou & Chen, 2009b). This synthetic
work suggests individual differences in generalized,
nonverbal cue vigilance. However, what exactly is
communicated?
Olfaction has been called a sensory emotion
in that it elicits mood and emotion (Yeshurun &
Sobel, 2010). Olfaction shares with emotion neural
substrates that project to areas in the brain associ-
ated with emotion (LeDoux, 2007; Zhou & Chen,
2009b); a hedonic, experiential dimension (pleas-
ant/unpleasant); privileged connections to memory
(Yeshurun & Sobel, 2010; Zhou & Chen, 2009a);
and a female sensitivity advantage (Brand & Millot,
2001; Olsson, Barnard, & Turri, 2006). De Groot,
Smeets, Kaldewaij, Duijndam, and Semin (2012)
have demonstrated that specific facial and other
behavioral responses occur in response to fear and
disgust-related chemosignals, respectively. Is there
111
Caroline F. Keating
a differentiated, chemosensory system for discrete
emotions?
In their review of olfaction, Yeshurun and Sobel
(2010) wrote that “Humans are astonishingly good
at odor detection and discrimination” (p. 223)
and concluded that “Humans are astonishingly
bad at odor identification and naming” (p. 226).
These experts presented evidence supporting each
view—that humans are highly sensitive to tiny
amounts of odors and distinguish among them quite
well but cannot name them. Odors do not easily
sort themselves into neat, discrete, verbally acces-
sible categories. Making it easy by providing limited
choices of “basic” emotion labels helps. Using this
research strategy, adults linked specific odors to
happiness, disgust, and anxiety (Croy, Olgun, &
Joraschky, 2011). Using less constrained method-
ologies, however, most studies demonstrate very
limited, differentiated responding; perceivers gener-
ally agree on what constitutes good and bad smells,
and this hedonic continuum describes most human
reactions to odors (Yeshurun & Sobel, 2010).
Summary of developmental trends in communica-
tion through olfaction.  Olfaction plays more than
a supporting role in the performance of human non-
verbal communication. Developmentally, the func-
tions and effects of chemosensory cues parallel those
described for touch. Neonates come into the world
prepared to identify, remember, and prefer odors
linked to familiar caregivers. At the same time, the
odor cues released by offspring, relatives, friends,
and lovers confirm identities and signal states.
Voluntary control over body odor signals is increas-
ingly practiced at later ages, its particulars shaped by
the niche. Reactions to odor cues become differenti-
ated by social context and relationships. Olfactory
cues have hedonic value and, under certain proto-
cols, may elicit more discrete emotions in adults.
Communication Through Vision:
Developmental Trends in Facial
Expressions
Humans are social animals and visual ones. It is to
this sense that communication via the human face
owes its prominence. The structure and wiring
of faces evolved in part because their signal value
112
conferred fitness benefits (Keating, 2002). Humans
adapted to group living by becoming intuitive
face perceivers, detecting and expressing meaning
through dynamic expressions and static, physi-
ognomic traits. Here, I focus this developmental
review on dynamic facial signaling systems.
The dynamism of the face is a crucial commu-
nication tool. Emotional expressions, facial signals
and gestures, and the artful use of gaze to “connect”
with others and to direct others’ attention in both
subtle or dramatic ways are hallmarks of this com-
munication mode. It is rare when a person’s face
is not signaling something. Infants “know” this;
one event that upsets them mightily is when their
mother poses an expressionless face that moves
not at all. Thus, the so-called still-face paradigm
is deployed by researchers to alter infant social/
emotional state (e.g., Mesman, Linting, Joosen,
Bakermans-Kranenburg, & van Ijzendoorn, 2013).
Seeking and expressing information via the face
comes naturally to babies and expands during devel-
opment, thanks to the navigational forces of bio-
logical preparedness, increasing voluntary control,
differentiation, and the niche.
Sensation and facial expression.  Sensation opens
the floodgates of facial expression early in life. The
faces of young infants are biologically prepared to
move in response to sensory events—specifically,
to sudden, unexpected events; to pain; to taste; and
to meeting the gaze of others.
The startle facial expression.  The startle expres-
sion is a prime example of preparedness, with its
highly fixed onset and offset, exceptionally short
duration, and resistance to voluntary control even
when anticipated (Ekman, Friesen, & Simons,
1985). It has a choreographed pattern of facial mus-
cle movements, raising the brows (presumably for
better scanning), increasing blink rate, and pulling
the mouth sideways. Startle expressions in babies,
children, and adults share components and are
similarly triggered by sudden, unexpected, sensory
events (e.g., loud noises, unexpected sightings).
The startle is morphologically distinct from the
prototypical emotional display of surprise accord-
ing to Ekman and his colleagues, who consider the
startle facial response to be part of the startle reflex

in humans with little signal value (see, e.g., Ekman
et al., 1985). More could be done to probe this point
of view. In real-world situations, how do individuals
respond to others’ startles—or do they not respond?
Facial expressions of pain.  Expressions of pain
have obvious, communicative consequences, espe-
cially for assessing the condition of those whose
language may not be up to the task of capturing
internal experience. Researchers conducted a short-
term, longitudinal study of infant pain expressions
by videotaping infant facial responses during their
2-, 4-, 6-, and 12-month immunizations (inocula-
tions by injection; Kohut, Riddell, Flora, & Oster,
2012). Expressions were coded for particular facial
movements using an infant version of the Facial
Action Coding System (Baby FACS; Oster, 2006).
Components of adult pain expressions flashed across
infant faces in the brief instant following needle
insertion. However, infants did not express the kind
of discrete, negative emotional signals characteris-
tic of adults. Instead, they signaled general distress
and discomfort with their faces (Kohut et al., 2012).
In contrast, when Prkachin and colleagues applied
different coding schemes (Neonatal Facial Coding
System and Child Facial Coding System; see, e.g.,
Prkachin, 2009), they were able to identify facial
action units specifically associated with genuine
pain in neonates, children, younger adults, and the
elderly, including those with dementia. To Prkachin
and colleagues, these facial pain responses seemed
developmentally continuous (see Prkachin, 2009).
Measurement differences aside, there is more to
understand about the capacity of the face to express
pain, and its role in social relationships. Although
mothers—and for that matter, pediatricians—readily
detect general pain signals, they struggle to conjure
the specific nature of the pain. Adult judgments of
pain intensity from facial expression are quite vari-
able. Moreover, with increasing ability to voluntarily
control facial output, even pain expressions may
be faked. Perceivers have a hard time distinguish-
ing real from fake pain expressions. Computers do
it better; software “trained” to identify facial action
units (muscle contractions) associated with real
pain proved more reliable than human observers
in discerning genuine from feigned pain (Bartlett,
Littlewort, Frank, & Lee, 2014). Maybe that is
The Developmental Arc of Nonverbal Communication
because the human eye is used to searching not
only for signs of real pain but for expressions that
also comprise bids for sympathy and support—also
important (social) calls to answer. Moreover, pure
expressions of actual pain may be in short supply
in human circles, as is the case for other species.
Cloaking the full-blown extent and source of pain
and injury may disguise serious defects in newborns
dependent on parental investment and may preserve
self-presentational goals in vulnerable adults
(Keating, 2006).
Facial expressions in reaction to taste.  Expressions
triggered by sensations of taste also register with
observers in rather nonspecific ways. Babies’ reac-
tions to sweet, sour, and bitter tastes produce facial
expressions similar to those of adults (Ganchrow,
Steiner, & Daher, 1983). Sweet flavors elicit smiles,
lips pucker in response to bitterness, and noses
wrinkle when sour substances are tasted. Carefully
controlled studies showed that although untrained,
adult observers could detect from videotapes when
an infant sampled sweet versus sour or bitter
tastes, they could not distinguish facial reactions
to sour versus bitter stimuli (Rosenstein & Oster,
1988). Yet, a fine-grained analysis using Baby FACS
revealed distinctive facial muscle actions for these
negative hedonic responses, despite variability in
infant individual responses (Rosenstein & Oster,
1988). More recent research revealed that infant
responses to sour taste become more differentiated
between 4 and 12 months (Bennett et al., 2005).
Yet, as for pain, adult perceivers seem sensitive to
general, not specific, facial signals of distress in young
infants (Oster, 2005); the question that remains is why.
Responding to gaze.  Mutual gaze is a distal,
sensory event and an interesting example of how
pervasive the influence of the niche can be on seem-
ingly elemental nonverbal behaviors. From birth,
infants reveal a preference for mutual compared to
averted gaze (Farroni, Csibra, Simion, & Johnson,
2002). However, there is a lot to getting this basic
signal regulated “right” according to one’s culture.
Infants do not quite get it. In the first few months
of life, they exhibit captured attention, staring too
long into your eyes as if they know something and
await your confession. As caregivers tune infants to
synchronous, social engagement (Feldman, 2007),
113
Caroline F. Keating
infant fixations become less stuck on particulars of
the face. Still, developmental studies of Swedish and
Dutch infant and adult fixation patterns indicate
that gaze avoidance, at least in response to threaten-
ing faces, is not observed in 7-month-old infants
the way it is in adults, suggesting it may be learned
later in the context of the niche (Hunnius, de Wit,
Vrins, & von Hofsten, 2011; Rigato, Farroni, &
Johnson, 2010). In some cultures, children learn
that not meeting another’s gaze signals disinterest
and disrespect (Argyle & Cook, 1976; E. T. Hall,
1966). In others, active gaze avoidance is a hallmark
of respect for a higher status person (e.g., Schofield,
Parke, Castañeda, & Coltrane, 2008). Children and
adults (as well as nonhuman primates) use staring
as a threat gesture (Fehr & Exline, 1987); done dif-
ferently, mutual gaze signals attachment, connec-
tion, and—among adults in the West—romantic
interest (Moore, 2010). Giving and getting accurate
signals from gaze would seem to be important, and
so it is puzzling how long a cultural “leash” humans
seem to give it.
Emotion communication from facial expression. 
Facial expressions are vehicles for emotion commu-
nication. In fact, for some post-Darwinian theorists,
expression and emotion are one in the same. That
is, the activation of neuromuscular activity on the
face is as integral a part of an emotion as its internal
feeling state, and its particular physiological and
motivational signature (Ekman, 1972; Izard, 1971;
Tomkins, 1962). To these theorists, innate, evolved
capacities comprise discrete, “basic” emotions: hap-
piness, surprise, fear, sadness, anger, disgust, and
(for some) contempt (Ekman & Heider, 1988).
Izard (1971, 2007), whose differential emotions
theory was based on extensive work with infants,
included “interest” as a basic emotion. What makes
particular emotions basic according to Izard (1992)
is their “innate and unique neural, emotional, and
motivational substrates, including innate and univer-
sal expressive components” (p. 561; see also Izard,
2007). Other emotions are blended from these.
Facial expressions transmit to others but also to
self via feedback from muscle to brain (Tomkins,
1962). Each basic emotion is believed to have a
unique, autonomic response signature that appears
114
similar across the life span (Levenson, Carstensen,
Friesen, & Ekman, 1991). Thus, the proposition
that emotional expression and emotional experi-
ence are the conjoined twins of nonverbal behavior
has important implications for the development of
emotion regulation as well as for nonverbal commu-
nication (e.g., Campos, Frankel, & Camras, 2004;
Holodynski, 2004; Saarni & Weber, 1999). Here,
however, we stay focused on the interpersonal,
­communicative aspects of basic emotions theory.
Theories promoting basic, discrete, universal
emotions indexed by innate facial expression have
stimulated decades of research on the expressive
capacities and recognition abilities of infants, chil-
dren, and adults. Importantly, these theories gener-
ate potentially falsifiable predictions as to what to
expect from “top down” points of view. Toward that
end, the emotional signaling and receiving capacities
of infants and children can be compared to adult-
based standards or prototypes.
However, just as importantly, developmental
approaches have generated critical questions in the
pursuit of understanding emotion and its commu-
nicative functions. Are infant expressions associ-
ated with internal states such as those projected for
adults? Do infant expressions comprise elements of
the prototypical facial action units that eventually
metamorphose into full-fledged, differentiated adult
expressions? Can infants detect and discriminate dif-
ferent emotional expressions and the emotion states
they convey? Are all emotional expressions “equal,”
or do some attract more attention than others? To
what degree does experience modify the course of
biologically prepared, communicative signals? Do
the abilities to express and recognize emotional facial
expressions degrade over the life span? Finally, what
of the development of expressions not presumed
linked to emotion? Answers to these questions con-
tribute valuable, “bottom-up” challenges to higher
order assumptions of current models of emotion.
Research to date indicates that, within the constraints
of biological preparedness, emotion recognition and
expression journey from a relatively diffuse system
to one with greater refinement and specificity in both
action and meaning, a developmental change gener-
ally characterized as differentiation.

Recognizing and interpreting facial expressions
of emotion.  The detection and discrimination of
emotionally valenced facial expressions begins
strikingly early in life (see Leppänen & Nelson,
2006, for a review). Newborns are responsive to
changes in facial expressions, and they mimic their
configurations (Field, Woodson, Greenberg, &
Cohen, 1982). Discrimination of posed expres-
sions and static, facial expressions of happy, angry,
fearful, sad, and surprised has been reported in
3- to 4-month-olds (LaBarbera, Izard, Vietze, &
Parisi, 1976), with mothers facial poses being
especially provocative (Barrera & Maurer, 1981).
In these early months, infants are typically tuned
to caregiver’s emotional expressivity; the familiar
faces of their mothers (but not strangers) facili-
tate cross-modal matching of facial/vocal signs
of emotion (happy/sad vs. happy/angry) and so
do the faces of their fathers if they are frequent
caregivers (Montague & Walker-Andrews, 2001;
Walker-Andrews, Krogh-Jespersen, Mayhew, &
Coffield, 2011). In this case, familiarity breeds not
only liking but also a rudimentary understand-
ing of positively and negatively valenced emo-
tions. Interestingly, the reverse may also be true;
3-month-olds more readily learn a new face shown
with a smile than shown with a neutral expression
(Turati, Montirosso, Brenna, Ferrara, & Borgatti,
2011). Clearly, facial behavior has meaning remark-
ably early in life.
Generalized recognition of emotional expres-
sions.  Between 5 and 7 months of age, infants
generalize emotional expression discrimination
across dynamic and static expressions of unfamiliar
individuals (Bornstein & Arterberry, 2003; Caron,
Caron, & Myers, 1985; C. A. Nelson & Dolgin,
1985; Soken & Pick, 1999). Leppänen and Nelson
(2009, 2012) proposed that by this time in devel-
opment, exposure to positive and negative facial
expressions enlists the perceptual processes needed
to shape cognitions about emotion categories and
consequences. Evidence from event-related poten-
tial (ERP) work suggests that emotion categories
for happy versus sad expressions are formed by 7
months of age (Leppänen, Richmond, Vogel-Farley,
Moulson, & Nelson, 2009). Hemodynamic (blood
The Developmental Arc of Nonverbal Communication
flow) patterns also differ in response to happy and
angry facial expressions at about this age (Nakato,
Otsuka, Kanazawa, Yamaguchi, & Kakigi, 2011).
ERP evidence of discrimination between two
negative expressions, fear and anger, has also been
reported in 7-month-olds (Kobiella, Grossmann,
Reid, & Striano, 2008). As infant social worlds
expand, so do their capabilities to decipher facial
expressions needed to expand their repertoire of
social targets.
Differentiation in responses to emotional facial
expressions.  Very young infants respond differ-
ently (and appropriately) to facial expressions of
affect, suggesting that they connect expression to
general emotional tone. Observation of “en face”
interactions between mothers and infants is a win-
dow to the synchrony of emotional tone that devel-
ops by 10 weeks of age (Haviland & Lelwica, 1987).
With time and a niche favoring maternal expressiv-
ity (Bornstein, Arterberry, Mash, & Manian, 2011;
Field, 2007), 7-month-old infants recognize and
interpret positively and negatively valenced emo-
tionally expressive signals across familiar and unfa-
miliar faces, using facial cues to anticipate what will
happen next (Walker-Andrews, 1997). One longi-
tudinal study found that between 6 and 36 months,
children increasingly initiate gazing at their mother’s
faces whereas, the reverse is true of mothers (Farran
& Kasari, 1990).
These developments enable young infants to
employ faces as social referencing tools (McClure,
2000). Thus, infants are able to use their mother’s
facial expressions to interpret their world. Classic
studies of the visual cliff demonstrated the effec-
tiveness of maternal expressions to encourage or
discourage exploration of the ambiguous appear-
ance of a dangerous-looking “drop” (Sorce, Emde,
Campos, & Klinnert, 1985). Others have shown
that positive, maternal facial expressions cue infant
play and exploration, whereas posed sadness shuts it
down (Campos, Campos, & Barrett, 1989). During
immunizations, infants whose mothers displayed
fear instead of pain in anticipation of the procedure
displayed less pain after needle insertion (Horton &
Riddell, 2010). These nonverbal sensitivities eventu-
ally extend to others’ facial expressions. By 7 months
115
Caroline F. Keating

of age, infants look to stranger’s facial expressions to offsprings’ social-emotional signals. In adults gener-
check on apparent ambiguities in intentions toward ally, approach and avoidance tendencies are guided
them (Striano & Vaish, 2006). by expressions. Expressions of happiness and sad-
ness in infants (or in adults) stimulate motoric
Changing attunements to facial expressions
approach tendencies, indicating a readiness to
of emotion.  Attunements to different emotional
socially engage or help (Mizokawa, Minemoto,
expressions change over time. A host of different
Komiya, & Noguchi, 2013). Mothers’ brains, how-
methodologies measuring looking time, ERPs, heart
ever, are uniquely sensitive to their own baby’s
rate, and shifts in attention suggest that during the
smiles and cries (Doi & Shinohara, 2012).
first few months of life, infants seem drawn to posi-
Mothers may put a familial stamp on their chil-
tive expressions (e.g., Farroni, Menon, Rigato, &
dren’s early expression recognition abilities as well.
Johnson, 2007; LaBarbera et al., 1976; Vaish,
After all, mothers tailor their facial communication
Grossmann, & Woodworth, 2008) but become
like they do features of language, performing certain
tuned to fear expressions by the second half of
facial actions in a slow, exaggerated fashion (D. N.
the first year of life (e.g., Hoehl & Striano, 2010;
Stern, 1974). Chong, Werker, Russell, and Carroll
Leppänen & Nelson, 2009; Vaish et al., 2008).
(2003) identified three expressions commonly used
For example, using the overlap procedure and
by Canadian Chinese and English-speaking moth-
measuring scan paths, 5-month-old infants readily
ers when interacting with their infants in natural-
disengaged their attention away from fearful facial
istic settings—the fish face; mock surprise; and an
expression; by 7 months of age, they were reluctant
infant-directed, exaggerated smile. Exaggerated,
to do so (Peltola, Hietanen, Forssman, & Leppänen,
feigned-emotion-laden expressions such as these are
2013). Similarly, ERP studies indicated that whereas
attention-grabbing, making infant imitative actions
sensitivity to happy faces characterized 7-month-olds,
(and therefore social learning) more likely (Fuku-
12-month-olds displayed a more adult-like sensitiv-
yama & Myowa-Yamakoshi, 2013). At this point
ity to angry faces (Grossmann, Striano, & Friederici,
in development, caregivers likely fine-tune recog-
2007; Vaish et al., 2008). Moreover, infants, like
nition patterns as they do expressive productions
adults, scan threatening (angry and fearful) and
(Elfenbein & Ambady, 2002; Marsh, Elfenbein, &
nonthreatening (happy, sad, neutral) facial expres-
Ambady, 2003).
sions differently, revealing fewer, shorter fixations
The developing infant acts on the niche to shape
on the inner features of threatening faces, a pattern
what becomes meaningful in a face. Infant scan
described as vigilent (Hunnius et al., 2011). Young
paths in response to dynamic, talking faces reveal
children are also on alert for bad news. Like adults,
that whereas 3- to 4-month-olds attend equally
they detect angry faces embedded in neutral dis-
to mouth and eyes, 9-month-olds fixate mostly
tractors more quickly than happy ones (LoBue,
on the moving mouth, likely reflecting rudiments
Matthews, Harvey, & Thrasher, 2014). The tim-
of the preparedness to learn language (Wilcox,
ing of the shift toward sensitivity to fear cues syncs
Stubbs, Wheeler, & Alexander, 2013). The work
with dangers inherent in increased mobility and
of the niche in shaping facial emotion recognition
with an increase in a parent’s use of distal strategies
becomes increasingly apparent as children age.
(i.e., facial expressions) to influence infant behavior
Some researchers suggest that even the perceptual
(Braungart-Rieker, Hill-Soderlund, & Karrass, 2010;
structure of facial emotional expressions takes shape
Campos, Kermoian, & Zumbahlen, 1992;
over time (Gao, Maurer, & Nishimura, 2010). Oth-
Leppänen & Nelson, 2009; Peltola et al., 2013).
ers target cognitive processes. Widen and Russell
Niche elements in emotion recognition from (2010) found that as language skills develop, young
faces.  While infants begin to find similarities in U.S. children (3–10 years of age) more readily dif-
expressive features across individual faces, their ferentiate (label) basic emotions from emotion sto-
mothers start making distinctions, conforming to ries than from still, photographic images of facial
the cross-species pattern of specializing in their expressions depicting basic, prototypical emotions.

116

With the exception of surprise, even the youngest
children were more accurate when identifying pro-
totypical emotions and social emotions from stories
than from faces. This suggests the inculcation of a
conceptual emotion map partially fitted to expres-
sions rather than the other way around. In fact, adult
interpretations of facial expressions reveal “wiggle
room” in the degree to which specific emotional
expressions are exclusively pinned to single emotion
labels (Widen & Naab, 2012). Given a free response
paradigm, even adults have trouble coming up
with the predicted labels for some basic emotional
expressions (Elfenbein & Ambady, 2002).
Emotion recognition signals are also warped by
gender role expectations. When infants are believed
to be boys, their facial expressions are more likely
to be perceived by adults as expressing fear, anger,
and distress. When believed to be girls, infants are
perceived as expressing more joy (Haviland, 1977).
Similarly, adult perceivers are more sensitive to
expressions of anger in male faces and happiness
in female faces for social (as well as physiognomic)
reasons (Becker, Kenrick, Neuberg, Blackwell, &
Smith, 2007). Thus, basic emotion signals are
compromised by cues associated with gender.
Sex differences in emotion recognition.  Gender
and emotion recognition are linked in other ways.
A female advantage in recognizing emotion per-
sists across the life span. During infancy, child-
hood, and adulthood, meta-analytic results confirm
that females are more sensitive to expressions of
emotions than are males (J. A. Hall, 1984, 2006;
McClure, 2000). A study of 1,951 European 13- to
15-year-olds presented morphed faces expressing
blended emotions along four continua: anger/
sadness, anger/fear, happiness/fear, and happiness/
sadness (Lee et al., 2013). Accuracy and reaction
time to choose one of two labels (the end points of
each continuum) were recorded. Adolescent girls
displayed greater sensitivity to expressions; they
were more accurate and faster at choosing predomi-
nant emotions represented in the blends. Though
both sexes overidentified happiness, boys did so more.
Pubertal status had no predictive value for either sex
(Lee et al., 2013). Thus, past and present research
throws the weight of the evidence in the direction of
greater female sensitivity to emotional facial expressions.
The Developmental Arc of Nonverbal Communication
Several mechanisms have been proposed to
account for the sex difference, including the way
women and men look at and scan faces (e.g., J. K.
Hall, Hutton, & Morgan, 2010). Females generally
gaze more frequently and longer at faces than do
males (J. A. Hall, 1984), even as infants (Leeb &
Rejskind, 2004). At 3–4 and 9–10 months of age, as
well as during adulthood, females’ fixations target
internal compared to external facial features more
than do males, potentially leaving females more
sensitive to facial changes in expression (Rennels &
Cummings, 2013).
Why do females and males scan faces differently?
The universal emphasis on nurturance training for
girls (Barry, Josephson, Lauer, & Marshall, 1976)
may put a premium on their ability to detect signs
of distress and delight in faces. Some neuroscientists
suggest that this ability is acquired at a very primary,
automatic level in females (G. B. C. Hall, Witelson,
Szechtman, & Nahmias, 2004). Others researchers
argue that higher order, social norms and expecta-
tions drive females’ modest advantage on most tests
of nonverbal sensitivity (J. A. Hall, 2006). Develop-
mental work on how family contexts encourage and
discourage expressivity in girls and boys could be
revealing (e.g., Halberstadt, Crisp, & Eaton, 1999).
For example, in 1989, more nonverbal expressivity
was observed during discussions between daughters
and parents than between sons and parents (Nol-
ler & Callan, 1989). Is this still the case, where are
the exceptions, and what are the consequences for
social relationships?
Recognizing facial expressions of emotion later
in life.  There are some curious twists and turns
in the developmental research record of emotion
recognition capacities from school-age children
through adulthood. In general, studies of emotion
recognition, whereby various sets of prototypical
emotional expressions or morphed emotion-blends
are matched to emotion labels, reveal increased
accuracy and sensitivity up through late childhood
(e.g., Herba, Landau, Russell, Ecker, & Phillips, 2006;
Montirosso, Peverelli, Frigerio, Crespi, & Borgatti,
2010; K. M. Thomas et al., 2001; L. A. Thomas,
De Bellis, Graham, & LaBar, 2007). Theorists have
tried to tag this expanding skill set to maturation
in the brain’s emotion processing circuitry during
117
Caroline F. Keating
adolescence. However, complicating the issue is the
fact that trends in recognition performance differ
for different emotions. Some find increased sensitiv-
ity to happy and fearful expressions between 4 and
15 years of age but no change for sadness, anger, or
disgust (Herba et al., 2006). Others report children
and adolescents to be less sensitive to fear and anger
than adults (L. A. Thomas et al., 2007). There are
also reports of improvement in the recognition of
particular emotions for males only during adoles-
cence (Biele & Grabowska, 2006; Mancini, Agnoli,
Baldaro, Ricci Bitti, & Surcinelli, 2013). At present,
these odd blips on the emotion recognition radar
screen have no compelling explanation.
Some inconsistencies in emotion recognition
abilities correspond to tragedies that befall children
and come to characterize their niche. Children
with histories of abuse show abnormalities in their
responses to emotional displays by peers (e.g.,
Main & George, 1985). There are other, cruel ways
in which the niche surrounding an individual dis-
rupts emotion recognition. Former child soldiers in
Sierra Leone reveal deficits in accurately recognizing
prototypical expressions of sadness (Umiltà, Wood,
Loffredo, Ravera, & Gallese, 2013). When mislabel-
ing occurred, both they and the citizens of their war-
torn country most often mistook sad expressions
for anger, happiness expressions for anger, and fear
expressions for sadness (Umiltà et al., 2013). The
long-term effects of these emotion recognition dis-
turbances on children, adults, and (for that matter)
countries are unknown.
Existing laboratory research indicates a general
decline in performance on typical, static-emotion
recognition tasks among healthy, older adults,
although explanations of the apparent decline may
be compromised by the methodologies employed
(see Isaacowitz & Stanley, 2011, for a review). For
example, older adults judge emotion from video-
taped sequences of dyadic interactions better than
younger adults but perform worse on still images
of single emotions (Sze, Goodkind, Gyurak, & Lev-
enson, 2012). In general, though, cross-sectional
research reveals an arc in performance, whereby
very young and very old samples of individuals typi-
cally perform less well on emotion recognition tasks
than those in the middle-age ranges. Age-related
118
declines for some emotions are steeper than for
others. For example, differences between older
and younger adult recognition rates are greatest
for anger, fear, and sadness; are less for happiness
and surprise; and are slightly reverse direction for
disgust (Ruffman, Henry, Livingstone, & Phillips,
2008).
Explanations for these age-related patterns are
murky. Ruffman et al. (2008) cited evidence con-
sistent with neurological decline. Differences in
process—specifically, scan paths—are also impli-
cated. Older people look less at the upper half of
the face, and this may (Firestone, Turk-Browne, &
Ryan, 2007; Sullivan, Ruffman, & Hutton, 2007)
or may not (Isaacowitz & Stanley, 2011) account
for age declines in recognition ability. ERP research
suggests a slight delay in the early processing of
emotion discrimination from faces among 59- to
74-year-olds (Wieser, Mühlberger, Kenntner-Mabi-
ala, & Pauli, 2006). That said, emotional reactivity
in response to emotional facial expressions, mea-
sured physiologically and in the brain, show no age-
related difference (Wieser et al., 2006), even though
other types of emotion-related stimuli do (Neiss,
Leigland, Carlson, & Janowsky, 2009).
Developmental trends are hard to decipher from
cross-sectional data. Moreover, additional aspects of
the niche may be at play here. Age subsumes factors
known to influence facial signal decoding, includ-
ing status and familiarity with social stimuli. For
instance, facial wrinkles complicate facial signals,
making emotions and intentions harder to assess
and making mimicry less likely (Hess, Adams,
Simard, Stevenson, & Kleck, 2012; Malatesta,
Fiore, & Messina, 1987). In age-segregated niches,
older adults may be used to relying on cues other
than facial signals in their interactions with cohorts.
Producing facial expressions of emotion.  People’s
first expressions may not be entirely their own; they
may be shared. Faces are a platform for mimicry and
imitation—two pillars of social bonds. Beginning in
the first few hours of life, infants can imitate facial
expressions (e.g., mouth opening, tongue protru-
sions, and components of emotions) modeled by
adults (Field et al., 1982; Meltzoff & Moore, 1977).
This early matching of responses may not be reflex-
ive, but newborns in an alert state are certainly

prone (prepared) to do it. These early communica-
tive proclivities have important social consequences
in ways similar to touch. The synchrony inherent in
face-to-face exchanges that normally occur between
parents and newborns tunes developing brains to
social and emotional meaning, to self, and to rela-
tionships (Feldman, 2007).
Differentiation in emotional expression produc-
tion.  Currently debated is the degree to which the
emotional expressions produced by infants represent
precursors of differentiated, adult expressions in
morphology and meaning. Do infant expressions
in response, say, to frustrating or surprising events
have the same look and feel as do the expressions of
adults? Do infants display the same range of emo-
tions, both positive and negative, that adults do?
Does the niche partly shape these things? At the
heart of these issues for many is the degree to which
facial displays are isomorphic with particular emo-
tional states (Lewis & Michalson, 1983; Weinberg &
Tronick, 1994). An added challenge to these ques-
tions is the methodologies available to address them.
Infant facial responses to emotion-eliciting
events have been measured in two basic ways—first,
by using anatomically based coding systems specify-
ing the facial muscle movements linked to discrete
emotions (e.g., the maximally discriminative facial
action coding system [MAX; Izard, 1979], the Facial
Action Coding System [FACS; Ekman & Friesen,
1978], and Baby FACS [Oster, 2006]), and second,
through judgment studies in which adults match
infant expressions to predicted emotion labels
(judgment “accuracy”). In a comprehensive review
of these studies, Camras and Shutter (2010) noted
discrepancies both between coding protocols and
between judges, especially for the construal of nega-
tive emotions (anger, fear, and disgust). Specifically,
coding protocols for distinguishing between emo-
tions differed between manuals, and accuracy rates
for judges were described as “unimpressive” across
studies (Camras & Shutter, 2010). Issues with the
measurement of the dependent variable (emotion)
make it a slippery target.
The independent variable, emotion-eliciting
events, has issues too. Common to all developmen-
tal research is the vexing problem of equivalence
The Developmental Arc of Nonverbal Communication
across age groups; a lack of specificity and con-
sistent potency inherent in eliciting events across
age groups can muddy the findings. Indeed, across
a variety of studies, emotion situations proved to
be unreliable elicitors of infant facial expressions,
meaning that infant facial responses to events of
a given emotional tone (e.g., surprise, fear) often
failed to produce these expressions, or they pro-
duced expressions of a decidedly different nature
(Camras & Shutter, 2010). If theorists are right,
a given, discrete emotion should be expressed
exclusively in its—and only its—emotion-eliciting
context. However, what parent has not had the
experience that Camras (1992) described: her young
baby’s look of total surprise when presented with an
unsurprising, familiar toy (see also Camras, Lam-
brecht, & Michel, 1996)?
Researchers have had better luck finding
expected matches among older infants. For exam-
ple, Bennett et al. (2005) compared the emotional
expressions of 4- and 12-month-old babies follow-
ing different, emotion-eliciting events. Matches for
happiness, disgust, and anger increased with age
(and mismatches generally decreased), although fear
failed to show the same developmental trend.
Still, few developmentalists would argue that
infant emotional expressions are simple, “mini-me”
versions of adult expressions. Indeed, Oster (2005)
argued that infants manifest a nonverbal emotional
dialect all their own and proposed it be studied that
way. Cross-cultural work has shown that as late
as 11 months of age, babies who react behavior-
ally in ways indicative of feelings of anger/frustra-
tion (produced by having their arm restrained)
and fear (stimulated by presentation of a growling,
gorilla-head toy) do not display discrete, facial
responses of anger and fear exclusively in each
respective, emotion-eliciting situation. Rather, the
researchers found that whether infants were Euro-
pean, American, Japanese, or Chinese, their faces
expressed anger and fear interchangeably across
situations, signaling a sort of general negativity or
distress (Camras et al., 2007; Camras, Oster, Cam-
pos, & Bakemand, 2003). So far, it appears that
infant displays match generally positive or negative
expressions to generally positive or negative stimu-
lus events but that discrete emotional reactions,
119
Caroline F. Keating
especially negative ones, are hard to decipher using
the theories and tools at hand. Though infant facial
expressions may lack specificity, they are intensity-
graded, perhaps by signals as mundane as the degree
of eye widening and mouth opening (Messinger,
Fogel, & Dickson, 2001; Messinger, Mattson,
Mahoor, & Cohn, 2012). This means that caregivers
can react appropriately to the degree of delight or
distress at hand even if they are unable to identify a
specific cause.
Thus, infant production of emotional expressions
(and infant emotions) can be generally described
as less differentiated than those of adults (Bennett
et al., 2005; Bridges, 1932; Camras & Shutter, 2010;
Widen & Russell, 2003, 2008). It takes matura-
tion and tuning of neural circuitry for emotional
responding to get organized. In reacting to their
infant’s distress as signaled by facial expression,
parents may have little specificity to go on (Mesman,
Oster, & Camras, 2012) but ample opportunity to
shape infant behavior into culturally appropriate,
nonverbal dialects (Camras & Shutter, 2010; Scherer
et al., 2011; Widen & Russell, 2010). Cultural prac-
tices create different, nonverbal accents of emotional
expression (Marsh et al., 2003; Marsh, Elfenbein, &
Ambady, 2007) that may underlie ingroup advan-
tages in recognizing and producing expressive
signals (Elfenbein & Ambady, 2002, 2003; Scherer
et al., 2011).
Niche elements of facial emotional expression
production.  The niche operates on children’s pro-
duction of facial expressions in many ways. Family
environment shapes the degree of expressivity in
families (Halberstadt et al., 1999). For example,
Camras and colleagues (see, e.g., Camras, Bakeman,
Chen, Norris, & Cain, 2006) found that the overall
expressivity of 3-year-old Chinese girls adopted as
infants by European American families was more
similar to European American girls than Mainland
Chinese 3-year-olds when facial expressions in
response to emotion-eliciting photographs were
compared. The adopted girls’ facial behavior was
usually midrange between the two. However, for the
adopted and biological children raised by American
parents, the most potent predictor of positive emo-
tion (smiling) and of negative facial reactivity was
120
self-reported maternal strictness; strict, emotionally
controlling mothers had less expressive 3-year-olds
(Camras et al., 2006). These and similar findings
(Oveis et al., 2009), while correlational, suggest that
immediate family environments have pronounced
effects on expressivity.
More information is needed about how family
environments look from the infant’s point of
view. Some researchers have probed this question
by attaching a camera to infants’ heads (Sugden,
Mohamed-Ali, & Moulson, 2014; see Figure 5.2).
There is more potential here than first meets the
eye. While the head camera surveys what the social
environment has to offer, simultaneous observation
of infant gaze patterns could reveal the signals they
choose to view—and to avoid. The interplay could
capture how individual infants shape their nonver-
bal niche.
Sex differences in emotional expression produc-
tion.  Sex differences are evident in the production
of facial expressions linked to emotion. A recent
Figure 5.2.  The happy-face camera captures a baby’s
visual environment. The camera is located in the left
eye of the happy face. Whereas the eye-tracking appa-
ratus samples actual scanning, the head cam reveals
visual opportunities in the infant’s environment. From
“I Spy With My Little Eye: Typical, Daily Exposure
to Faces Documented From a First-Person Infant
Perspective,” by N. A. Sugden, M. I. Mohamed-Ali, and
M. C. Moulson, 2014, Developmental Psychobiology,
56, p. 252. Copyright 2014 by John Wiley & Sons, Inc.
Reprinted with permission.

meta-analysis found a small but persistent gender
difference in the West. Overall, girls expressed more
positive emotion and “internalizing” affect (sad-
ness, anxiety, and sympathy) than boys, whereas
boys showed more “externalizing” anger (Chaplin
& Aldao, 2013). Socialization may be at the root of
the difference. As children mature, they learn both
explicit and subtle display rules not only from
parents but from peers. In both sexes, this leads to
the dampening of emotional expressions conveying
vulnerability and anger (von Salisch, 2001). Girls
are generally encouraged to be more expressive than
boys, albeit in gender-role consistent ways (Chaplin
& Aldao, 2013). For example, a meta-analysis
revealed that females smile more and for different
reasons than do males (LaFrance, Hecht, & Paluck,
2003; see also Chapter 6, this handbook). Across
studies, smiling reflected more than positive affect.
Social signals from facial expressions.  Human
facial expressions can be viewed through the status-
signaling lens used by ethologists and behavioral
ecologists who study communication in nonhuman
social species. The basic premise shared by research-
ers from this perspective is that there is a lot less
emotion than meets the eye when facial muscular
movements are displayed. Facial movements relay
messages about social status and condition; that
is, they render social information about receptivity
(submissiveness, appeasement, affiliation) or threat
(a) (b)
Figure 5.3.  Depictions of facial gestures suspected of having phyletic roots: (A) shy smile (photograph by
Photolibrary Collection: Alphonse Pagano/Getty Images), (B) tongue show (photograph by Ashley Cole Siferd), and
(C) play faces (photograph by Riser Collection/Cultura: Patrick Wittmann/Getty Images). Reprinted with permission.
The Developmental Arc of Nonverbal Communication
(dominance, aggression, rejection) quite apart from
any read-out of the emotional state of the expresser
(Fridlund, 1997; Fridlund & Russell, 2006; Keating,
1985, 2002).
Smiling, for example, need not reflect the happi-
ness of the expresser to signal appeasement and to
invite approach (Volkmar & Siegel, 1982). Lowered
brows project dominance (Blurton Jones, 1971;
Keating & Bai, 1986) and ward off approach (Cam-
ras, 1977), regardless of whether the signaler is
feeling angry. The primate tongue show, whereby
the tongue is protruded slightly between closed
teeth, also inhibits approach (Smith, 1977). The
primate play face—a wide, open-mouth smile, often
displayed with head tilted backward and throat
exposed—indicates that rough-and-tumble play
is just that—play—with no threat intended (van
Hooff, 1972). The pout of frustrated offspring is
known to parents all over the world (Camras &
Shutter, 2010). What is interesting about these and
other facial status gestures is that they are similar in
form and function to nonhuman primate facial sig-
nals; in many cases, they are believed to be homolo-
gous (Keating et al., 1981; Lockard, Fahrenbruch,
Smith, & Morgan, 1977; Shariff & Tracy, 2011;
van Hooff, 1972). The panels of Figure 5.3 provide
illustrations of children’s shy (submissive) smiles,
tongue shows, and play faces.
Facial status gestures are robust signals that
convey intentions, regulate social interactions,
(c)
121
Caroline F. Keating
and serve impression management goals. In earlier
decades, they were extensively studied by human
ethologists interested in the role gestural commu-
nication played in the formation of status relation-
ships and hierarchies in children (e.g., Blurton
Jones, 1971; Brannigan & Humphries, 1972; Grant,
1969). Like status hierarchies (Mascaro & Csibra,
2012; Zitek & Tiedens, 2012), facial status gestures
are readily accessible. For example, Zivin (1977)
identified the “plus face,” an expression children
display in reaction to victory following agonistic
episodes and a reliable signal of their relative social
status. No smile, compared to slight smiles, made
adults appear to be the one who “tells other people
what to do” to children in the United States and to
men and women in 11 countries around the world
(Keating & Bai, 1986; Keating et al., 1981; cf. Hareli,
Shomrat, & Hess, 2009). Displays that include facial
cues for dominance and submissiveness—pride and
shame, respectively—are similarly perceived by chil-
dren and adults from disparate cultures (Tracy &
Matsumoto, 2008; Tracy & Robins, 2008; Tracy,
Shariff, Zhao, & Henrich, 2013; Widen & Russell,
2010; cf. Cole, Tamang, & Shrestha, 2006). Embar-
rassment, which also comprises submissive facial
cues, conveys appeasement (Keltner, 1995). The
spontaneous, facial productions of smiles, pride, and
embarrassment in blind people look pretty much
like those of sighted people (Cole, Jenkins, & Shott,
1989; Eibl-Eibesfeldt, 1973; Matsumoto & Willing-
ham, 2009). Thus, the social status signals conveyed
by faces reveal some of the same robustness that
facial expressions of emotions do.
Observations of children’s spontaneous, facial
expressions in naturalistic environments offer
alternatives to understanding the development of
facial expressions. If you try very hard, you can fit
preschoolers’ facial actions to incomplete displays
of prototypical emotions. Trying much less hard,
they are consistent with nonhuman primate status
signaling systems (e.g., Gaspar & Esteves, 2012;
Keating & Heltman, 1994; Ostrov & Keating, 2004).
In fact, presuming that a particular emotion lurks
beneath a given expression can be misleading. For
example, in a study of bids for help on the play-
ground, distraught children who looked sad received
less help from peers than victims who smiled at
122
their classmates, inviting approach. The nonverbal
exchanges of adolescents and their parents are a
study in approach/avoidance signaling. Kahlbaugh
and Haviland (1994) reported that during a family
interaction task, avoidance signaling through facial
expressions increased from pre- to late adolescence.
Parents, too, signaled more avoidance as their teen-
agers aged (Kahlbaugh & Haviland, 1994). Though
the specific expressions recorded by these experi-
menters were labeled as basic emotions, whether the
teens or parents actually felt any of them was not
determined. Data like these are consistent with a sta-
tus cues approach that emphasizes the importance
of communication over emotion.
Still, connections can be made between status
cues and emotion signaling. For example, Sacco and
Hugenberg (2009) proposed that the expressional
configurations of fear and anger may have evolved
by mimicking status-related physiognomic traits.
Emotion theorists have explored the status signaling
value of basic emotional expressions and other dis-
plays linked to emotional tone, especially credibil-
ity, pride, shame, and embarrassment (e.g., Hareli,
Harush, et al., 2009; Hareli & Hess, 2012; Keltner,
1995; Shariff & Tracy, 2011; Todorov, Baron, &
Oosterhof, 2008). Some developmental differences
have begun to emerge from this research. When
judging trustworthiness from faces, for example,
younger and older adults show different patterns of
association between credibility assessments and gra-
dations of happy/angry expressions (Éthier-Majcher,
Joubert, & Gosselin, 2013; cf. Todorov et al., 2008).
One question that could be asked about status
and emotion signaling systems is “Which came
first and serves more ‘basic’ functions?” Did status
(approach/avoidance) communication precede
emotion communication or vice versa? The point
is likely moot at this way station in evolutionary
history. Nonverbal communication is enabled by
the brain, brain systems evolve by accretion (Kon-
ner, 2010), and it is likely that what came before is
embedded in each assemblage of cues and meanings.
Communication Through Vision:
Developmental Trends in Body Expression
As for facial expression, the visual channel is the
transmission line for expression through body

movement. Developmental studies on the rec-
ognition and production of expressive body sig-
nals are spotty. Here, I review classic and recent
studies of the role of body gestures in social
interaction during infancy, childhood, and early
adulthood—specifically, pointing, nodding, and
body movements expressive of emotion. Though
there is considerable research on relationships
between gesture and language, that topic remains
outside the purview of this chapter (see Chapter 12,
this handbook).
Recognizing and producing body movement.  the prize (the infant or the experimenter), whereas
Body movement attracts attention from the earliest
moments of life. Newborns favor nodding, smil-
ing faces over stationary ones; looming bodies scare
them (Fantz, 1964). In the first few months of
life, infants begin to discriminate between motion
stimuli compatible with body movement and other
motion not natural for bodies. Fox and McDaniel
(1982) demonstrated this change in infants 2–6
months of age. Using point-light stimuli, 4- and
6-month-olds showed greatest sensitivity to body-
like motion. Relational movement is also attractive
to infants. The sequential actions of giving and
receiving objects in interaction with another person
are some of the earliest forms of play and may be the
embodiment of a human proclivity (preparedness)
for reciprocity. At the very least, “give and take”
games teach contingency—that personal power lies
in the coordination of action and reaction. Babies
seem to get the message. As early as 5 months of
age, babies prefer puppets whose movements facili-
tate rather than block the goal-directed actions of
another puppet, leading some to believe that this
kind of indirect reciprocity lies at the heart of a
“moral” human core (Hamlin, 2013).
Pointing gestures.  Pointing is also an early,
shared, communicative signal enabling coordinated
action in response to co-orientation toward a com-
mon focus. Pointing can be accomplished with body
parts (e.g., fingers, heads) and with gaze. Typical
infants follow the gaze direction of others to proxi-
mate targets between 3 and 6 months of age, and
to more distal targets by the end of the first year
(D’Entremont, Hains, & Muir, 1997). Most infants
respond to and deploy pointing with fingers at
The Developmental Arc of Nonverbal Communication
about 12 months of age, well before they speak, and
use it as a social signal to direct others’ attention
(Tomasello, Carpenter, Call, Behne, & Moll, 2005).
By 14 months of age, infants recognize that adult
pointing with fingers and gaze are purposeful signals
worth their attention. Thus, infants’ use of gaze cues
to determine the location of a hidden toy improves
with age up to 24 months (Behne, Carpenter, &
Tomasello, 2005). At 25 months of age, infants use
pointing to show an experimenter where a toy is
hidden. Infant chimpanzees do too, the difference
being that infants point regardless of who receives
chimpanzee point only under conditions where they
receive it (Bullinger, Zimmermann, Kaminski, &
Tomasello, 2011). This difference in capacity may
reflect human preparedness for empathy.
Head movements.  Social information is derived
from movement at the top of the body. Head nods
and shakes were studied over time as U.S. mothers
interacted with their 14- to 30-month-olds in a natu-
ral setting (Fusaro, Vallotton, & Harris, 2014). Nods
were generally confirmatory; mothers used them as
a sign of approval and to encourage early attempts at
language. Head shakes were less frequently observed
and were used to discourage toddler activities.
Generally speaking, head nods convey appease-
ment, not threat (Eibl-Eibesfeldt, 1972). They may
be a therapist’s best friend when interacting with
clients; nodding is part of the “immediacy” constel-
lation of behaviors that signals engagement and
encourages conversation (Mehrabian, 1971). When
in conversation, Western women both nod and are
nodded to more than men (Helweg-Larsen, Cun-
ningham, Carrico, & Pergram, 2004). Research
using avatar stimuli indicate that the dynamics of
the movement itself are suggestive of femininity
(Boker et al., 2011).
Emotion and body movements.  Body poses
and actions convey emotion. Characteristics of
gait—such as the way an adult swings their arm,
and the speed and length of their stride—convey
varying degrees of happiness, sadness, and
anger (Montepare, Goldstein, & Clausen, 1987;
Montepare & Zebrowitz, 1993). Adults recognize
basic, discrete emotions quite well from body
movement and poses (Aviezer, Trope, & Todorov,
123
Caroline F. Keating
2012; de Gelder, 2009; de Gelder et al., 2012;
Peelen, Atkinson, Andersson, & Vuilleumier,
2007), leading some to argue that much of the
time, emotion is best detected from bodies, espe-
cially when action rather than empathy is needed
(Aviezer et al., 2012; de Gelder, 2009). That may
be so, and researchers have begun to probe its
developmental time course.
Infant discrimination of body emotion cues has
been tested using stimuli from adult studies (Zieber,
Kangas, Hock, & Bhatt, 2014). In the first experi-
ment, 6.5-month-olds watched short video clips of
adult models (with masked faces) enacting “happy,”
“angry,” or “neutral” emotions; models enacted
each state in their own way. Pairs of different enact-
ments were shown side by side, and when infant
looking was recorded, a clear preference emerged
for happy versus neutral videos when stimuli were
shown upright (Zieber et al., 2014). In a second
experiment, 6.5-month-old infants “matched”
happy–angry videos to happy–angry vocal expres-
sions; they looked longer when the emotional
signals from body and voice were consistent.
The researchers interpreted these results as dem-
onstrating both discrimination (Experiment 1) and
recognition (Experiment 2) of emotion from body
movements (Zieber et al., 2014), though specific
emotion cues could not be identified.
Children’s abilities to decode expressive body
movements have been investigated through the
medium of dance (Boone & Cunningham, 1998).
Videotapes of a male and a female adult performing
four target emotions—happiness, sadness, anger,
and fear—were presented to 4-, 5-, and 8-year-olds
as well as to college-age adults. In this study, each
videotape was copied and altered by removing
selected, emotion cues to diminish the intensity of
each portrayal. Videos of different emotions and
different intensities were presented in pairs, and
the children and adults chose the one that best
matched each emotion (e.g., “Who’s happy?”) and
one that conveyed more emotion (“Who’s really
happy?”). Four-year-olds were above chance for
sadness, and 5-year-olds were above chance for hap-
piness, sadness, and fear. Eight-year-olds and adults
performed similarly and were above chance for all
four emotions. Sensitivity to intensity followed a
124
similar developmental pattern, suggesting that chil-
dren’s skill in differentiating emotion from dance
movements increases between 5 and 8 years of age
(Boone & Cunningham, 1998).
Children’s developing attunements to body
movement is evident during interactions with
peers; a walk through age-segregated, U.S. pre-
school classrooms during free play period pro-
vides a developmental “tour.” Basic, instrumental
acts of aggression—such as hitting and pushing,
a common form of communication among 1- and
2-year-olds—are phased out in the 3- and 4-year-
old classrooms (Ingram, 2014) as teachers’ cheery
voices encourage children to “Use your words!” to
solve disputes. Observation reveals that preschool-
ers largely use facial and body status gestures to get
their way (e.g., Keating & Heltman, 1994; Ostrov &
Keating, 2004). Common body dominance signals
include arms akimbo, chin thrusts, fist displays,
pointing, and erect posture. Common affiliative or
submissiveness body gestures include shrugging,
hunched shoulders, closed postures, open-palms,
bowed head, and feigned shoe-tying when under
attack (Camras, 1977; Keating & Heltman, 1994;
Zivin, 1977). Apart from shoe-tying, several of
these gestures are similar to the signals nonhuman
primates display when expressing dominance, sub-
missiveness, and reconciliation (De Waal, 1990;
Goodall, 1971). Recordings of children’s dominance
and submissiveness gestures can be used to deter-
mine status relationships and rankings, to predict
resource control in free play and task situations,
and to predict teacher and peer assessments of a
classmate’s social status (Keating & Heltman, 1994;
Ostrov & Keating, 2004). Artful displays of domi-
nance gestures are characteristic of preschool class-
room leaders, who are also best at managing their
body and face cues so as to appear truthful when tell-
ing a lie to an adult; the same holds for high school
and college students (Keating & Heltman, 1994;
Keating, Little, & Colligan, 2012). Nonverbal acting
skills belong within the constellation of social com-
petence (Cole, 1986; Feldman, Tomasian, & Coats,
1999; Halberstadt, Denham, & Dunsmore, 2001).
Posture is a potent cue for status among children
and adults. Weisfeld and Beresford (1982) studied
the postures of children engaged in a dodgeball

game on the playground. Erect postures signaled
successful play; slumped postures signaled fail-
ure. Expansive body postures are characteristic
of dominance displays and power in humans and
chimpanzees, and they parallel elements of human
victory displays following sporting events (Carney,
Cuddy, & Yap, 2010; Mitchell & Maple, 1985).
Hwang and Matsumoto (2014) analyzed the bodily
responses to victory and defeat in Olympic judo
competitions; some of the contestants were blind.
The researchers identified a constellation of behav-
iors displayed only by victors that included aspects
of body expansion, aggressive facial gestures, and
direct gaze (Hwang & Matsumoto, 2014). So far,
it appears that children and adults enact postures
and perform victory displays that may have phyletic
roots. The next question is how these signals influ-
ence social targets and future contests.
Summary of developmental trends in communica-
tion through face and body.  Sensory experience
in utero is richer in tactile and olfactory stimulation
than in visual stimulation, where varying amounts
of light likely account for the only excitement.
Considering how restricted visual experience is in
utero, and how novel and unrehearsed communi-
cation through visual means is to newborns, it is
remarkable how quickly babies become adept at
interpreting and producing face and body signals.
Biological preparedness serves them well from the
start, but the infant’s self-driven engagement happens
soon thereafter. Neonatal reflexes give up control
after a month, and developmental processes enabling
differentiation explode in the first year of life. The
lively debate over connections between visual signals
and underlying emotion has been stimulating in its
own way, driving researchers to place high value on
cross-cultural research and on other niche factors.
Though the human capacity for communication
through visual signals appears to improve and then
decline with age, limitations in both methodology
and explanation make this a tentative conclusion.
GENERAL CONCLUSIONS
Communication in social species is critical to survival.
At no time is this truer than during development,
The Developmental Arc of Nonverbal Communication
when offspring depend on the dedication of caregiv-
ers for their very lives. It is in these early days that
the capacities to form and maintain social bonds are
set loose, exercised, and expanded. It is then that
people’s phylogenetically oldest and (perhaps) wis-
est communication systems, nonverbal ones, take
center stage.
The three sensory platforms of the body reviewed
here—touch, olfaction, and vision—enable the kind
of information exchange that social bonds have
come to rely on. Each platform feathers the nest of
the developing organism by channeling information
about the social and physical environment. From
nonverbal tactile, olfactory, and visual cues, babies
detect whether they are with familiar others who
make them feel secure. These nonverbal signals of
identity (e.g., familiarity), affective state (e.g., emo-
tion, arousal), and social status (e.g., dominance,
threat, subordinate, affiliative) provide the basics
building blocks of social bonds. They differ in the
degree to which physical proximity between infant
and caregiver is required. In likely parallel to their
rates of use, touch and olfactory signaling systems
require a level of proximity that face and body visual
cues do not; the latter system grants increasingly
mobile offspring more independence. Elaborated
versions of these same nonverbal communication
capacities characterize peer bonds and pair bonds
later in life.
The developmental constructs used to organize
researchers’ thinking about the development of non-
verbal communication capacities—preparedness,
increasing voluntary control, and greater
differentiation—have limited reach. Although these
three forces would seem to propel nonverbal abili-
ties monotonically upward, friction from sources
embedded in the niche gradually slow their rise.
Physical constraints on sensory processes are only a
piece of the picture. Cultural constraints, expecta-
tions, habits, and display rules channel differentia-
tion and entrain voluntary action to standards (e.g.,
Matsumoto, Yoo, & Fontaine, 2008). You have only
to compare the expressivity of young children to
that of adults to reach this understanding.
Tracings of the developmental arc of nonverbal
communication are incomplete without longitudinal
data charting the varied experiences of individuals
125
Caroline F. Keating
around the globe. To construct the whole story,
effort will be needed to integrate the ontogeny of
nonverbal communication as a part of life history’s
puzzle (Hawley, 2014). What functions do signaling
systems serve, how are they integrated, and what
supports or deflects them? Without observation and
experimentation in actual social environments, this
will surely be a difficult task. What the researcher
sees in the lab is known, but what does the nonver-
bal communication environment look like to the
baby, the child, the adolescent, and the adult whose
job it is to negotiate it? What opportunities does it
provide, how do its moving components comple-
ment one another, and how is it shaped by the
individual?
So far, this story of the “arc” is only roughed
in. Research suggests that expressive and receptive
capacities build, peak, and decline with old age.
The general progression is set in motion within the
boundaries of evolved capacities sensitive to a con-
fluence of “niche” elements that organize the devel-
opmental environment (Super & Harkness, 2002).
Understanding those environments requires fine-
grained, longitudinal, cross-cultural studies to flesh
out the developmental processes responsible for
differentiation. For example, some researchers pro-
pose dynamic systems models, whereby expressions
of undifferentiated, positive or negative states of
arousal are “trained” to different, discrete emotions
and expressions through socialization and other
synergistic processes (Camras & Witherington,
2005; Fogel et al., 1992; see also Lewis & Granic,
2002). These models emphasize plasticity in the
development of nonverbal communication systems.
Plasticity notwithstanding, some signals are sim-
ply hard to miss no matter who or where or how old
you are. The fact that infants readily interpret canine
signals correctly reveals just how tuned to expressive
cues human brains have become (Flom, Whipple, &
Hyde, 2009). There is much that researchers do not
know about the “adaptive wisdom” of nonverbal
communication systems and how essential they are
to human nature and society (Campbell, 1975). As
technological interfaces expand and contract the uni-
verse of human communication, the fact that face-to-
face nonverbal communication likely got us to where
we are today suggests it has much to teach us.
126
References
Argyle, M., & Cook, M. (1976). Gaze and mutual gaze.
Cambridge, England: Cambridge University Press.
Aureli, F., & Whiten, A. (2003). Emotions and behavioral
flexibility. In D. Maestripieri (Ed.), Primate
psychology: The mind and behaviour of human and
nonhuman primates (pp. 289–323). Cambridge, MA:
Harvard University Press.
Aviezer, H., Trope, Y., & Todorov, A. (2012). Body
cues, not facial expressions, discriminate between
intense positive and negative emotions. Science, 338,
1225–1229. http://dx.doi.org/10.1126/science.
1224313
Barrera, M. E., & Maurer, D. (1981). The perception of
facial expressions by the three-month-old. Child
Development, 52, 203–206. http://dx.doi.org/
10.2307/1129231
Barrett, L. F. (2011). Was Darwin wrong about emotional
expressions? Current Directions in Psychological
Science, 20, 400–406. http://dx.doi.org/10.1177/
0963721411429125
Barry, H., Josephson, L., Lauer, E., & Marshall, C. (1976).
Traits inculcated in childhood: Cross-cultural codes
5. Ethnology, 15, 83–106.
Bartlett, M. S., Littlewort, G. C., Frank, M. G., & Lee, K.
(2014). Automatic decoding of facial movements
reveals deceptive pain expressions. Current Biology,
24, 738–743. http://dx.doi.org/10.1016/j.cub.2014.
02.009
Becker, D. V., Kenrick, D. T., Neuberg, S. L., Blackwell, K. C.,
& Smith, D. M. (2007). The confounded nature of
angry men and happy women. Journal of Personality
and Social Psychology, 92, 179–190. http://dx.doi.org/
10.1037/0022-3514.92.2.179
Behne, T., Carpenter, M., & Tomasello, M. (2005). One-
year-olds comprehend the communicative intentions
behind gestures in a hiding game. Developmental
Science, 8, 492–499. http://dx.doi.org/10.1111/
j.1467-7687.2005.00440.x
Bennett, D. S., Bendersky, M., & Lewis, M. (2005). Does
the organization of emotional expression change
over time? Facial expressivity from 4 to 12 months.
Infancy, 8, 167–187. http://dx.doi.org/10.1207/
s15327078in0802_4
Biele, C., & Grabowska, A. (2006). Sex differences in
perception of emotion intensity in dynamic and
static facial expressions. Experimental Brain Research,
171, 1–6. http://dx.doi.org/10.1007/s00221-005-
0254-0
Blurton Jones, N. G. (1971). Criteria for use in describing
facial expressions of children. Human Biology, 43,
365–413.
Boker, S. M., Cohn, J. F., Theobald, B. J., Matthews, I.,
Mangini, M., Spies, J. R., . . . Brick, T. R. (2011).

Something in the way we move: Motion dynamics,
not perceived sex, influence head movements in
conversation. Journal of Experimental Psychology:
Human Perception and Performance, 37, 874–891.
http://dx.doi.org/10.1037/a0021928
Boone, R. T., & Cunningham, J. G. (1998). Children’s
decoding of emotion in expressive body movement:
The development of cue attunement. Developmental
Psychology, 34, 1007–1016. http://dx.doi.org/10.1037/
0012-1649.34.5.1007
Bornstein, M. H., & Arterberry, M. E. (2003). Recognition,
discrimination and categorization of smiling by
5-month-old infants. Developmental Science, 6,
585–599. http://dx.doi.org/10.1111/1467-7687.00314
Bornstein, M. H., Arterberry, M. E., Mash, C., & Manian, N.
(2011). Discrimination of facial expression by 5-month-
old infants of nondepressed and clinically depressed
mothers. Infant Behavior and Development, 34,
100–106. http://dx.doi.org/10.1016/j.infbeh.2010.10.002
Brand, G., & Millot, J. L. (2001). Sex differences in human
olfaction: Between evidence and enigma. Quarterly
Journal of Experimental Psychology, Section B:
Comparative and Physiological Psychology, 54, 259–270.
http://dx.doi.org/10.1080/02724990143000045
Brannigan, C. R., & Humphries, D. A. (1972). Human
nonverbal behaviour, a means of communication.
In N. Blurton Jones (Ed.), Ethological studies of
child behaviour (pp. 37–64). Cambridge, England:
Cambridge University Press.
Braungart-Rieker, J. M., Hill-Soderlund, A. L., & Karrass,
J. (2010). Fear and anger reactivity trajectories from 4
to 16 months: The roles of temperament, regulation,
and maternal sensitivity. Developmental Psychology,
46, 791–804. http://dx.doi.org/10.1037/a0019673
Brazelton, T. B. (1972). Implications of infant
development among the Mayan Indians of Mexico.
Human Development, 15, 90–111. http://dx.doi.org/
10.1159/000271232
Bridges, K. (1932). Emotional development in early
infancy. Child Development, 3, 324–341. http://dx.
doi.org/10.2307/1125359
Bullinger, A. F., Zimmermann, F., Kaminski, J., &
Tomasello, M. (2011). Different social motives in the
gestural communication of chimpanzees and human
children. Developmental Science, 14, 58–68. http://
dx.doi.org/10.1111/j.1467-7687.2010.00952.x
Burgdorf, J., & Panksepp, J. (2001). Tickling induces
reward in adolescent rats. Physiology and Behavior,
72, 167–173. http://dx.doi.org/10.1016/S0031-
9384(00)00411-X
Burgoon, J. K., Buller, D. B., & Woodall, W. G. (1996).
Nonverbal communication (2nd ed.). New York, NY:
McGraw-Hill.
Campbell, D. T. (1975). On the conflicts between
biological and social evolution and between
The Developmental Arc of Nonverbal Communication
psychology and moral tradition. American Psychologist,
30, 1103–1126. http://dx.doi.org/10.1037/0003-066X.
30.12.1103
Campos, J. J., Campos, R. G., & Barrett, K. C. (1989).
Emergent themes in the study of emotional
development and emotion regulation. Developmental
Psychology, 25, 394–402. http://dx.doi.org/10.1037/
0012-1649.25.3.394
Campos, J. J., Frankel, C. B., & Camras, L. (2004). On the
nature of emotion regulation. Child Development, 75,
377–394. http://dx.doi.org/10.1111/j.1467-8624.
2004.00681.x
Campos, J. J., Kermoian, R., & Zumbahlen, M. R. (1992).
Socioemotional transformations in the family system
following infant crawling onset. In N. Eisenberg &
R. A. Fabes (Eds.), Emotion and its regulation in early
development (pp. 25–40). http://dx.doi.org/
10.1002/cd.23219925504
Camras, L. A. (1977). Facial expressions used by children
in a conflict situation. Child Development, 48,
1431–1435. http://dx.doi.org/10.2307/1128503
Camras, L. A. (1992). Expressive development and basic
emotions. Cognition and Emotion, 6, 269–283. http://
dx.doi.org/10.1080/02699939208411072
Camras, L. A., Bakeman, R., Chen, Y., Norris, K., &
Cain, T. R. (2006). Culture, ethnicity, and children’s
facial expressions: A study of European American,
Mainland Chinese, Chinese American, and adopted
Chinese girls. Emotion, 6, 103–114. http://dx.doi.org/
10.1037/1528-3542.6.1.103
Camras, L. A., Lambrecht, L., & Michel, G. F. (1996).
Infant “surprise” expressions as coordinative
motor structures. Journal of Nonverbal Behavior, 20,
183–195. http://dx.doi.org/10.1007/BF02281955
Camras, L. A., Oster, H., Bakeman, R., Meng, Z., Ujiie, T., &
Campos, J. J. (2007). Do infants show distinct negative
facial expressions for fear and anger? Emotional
expression in 11-month-old European American,
Chinese, and Japanese infants. Infancy, 11, 131–155.
http://dx.doi.org/10.1111/j.1532-7078.2007.tb00219.x
Camras, L. A., Oster, H., Campos, J. J., & Bakemand, R.
(2003). Emotional facial expressions in European-
American, Japanese, and Chinese infants. Annals of
the New York Academy of Sciences, 1000, 135–151.
http://dx.doi.org/10.1196/annals.1280.007
Camras, L. A., & Shutter, J. M. (2010). Emotional facial
expressions in infancy. Emotion Review, 2, 120–129.
http://dx.doi.org/10.1177/1754073909352529
Camras, L. A., & Witherington, D. C. (2005). Dynamical
systems approaches to emotional development.
Developmental Review, 25, 328–350. http://dx.doi.
org/10.1016/j.dr.2005.10.002
Carney, D. R., Cuddy, A. J., & Yap, A. J. (2010).
Power posing: Brief nonverbal displays affect
127
Caroline F. Keating
neuroendocrine levels and risk tolerance. Psycho­
logical Science, 21, 1363–1368. http://dx.doi.org/
10.1177/0956797610383437
Caron, R. F., Caron, A. J., & Myers, R. S. (1985). Do
infants see emotional expressions in static faces?
Child Development, 56, 1552–1560. http://dx.doi.
org/10.2307/1130474
Castelli, L., De Dea, C., & Nesdale, D. (2008). Learning
social attitudes: Children’s sensitivity to the verbal
and nonverbal behaviors of adult models during
interracial interactions. Personality and Social
Psychology Bulletin, 34, 1504–1513. http://dx.doi.
org/10.1177/0146167208322769
Castelli, L., Zogmaister, C., & Tomelleri, S. (2009). The
transmission of racial attitudes within the family.
Developmental Psychology, 45, 586–591. http://dx.doi.
org/10.1037/a0014619
Chaplin, T. M., & Aldao, A. (2013). Gender differences
in emotion expression in children: A meta-analytic
review. Psychological Bulletin, 139, 735–765. http://
dx.doi.org/10.1037/a0030737
Cheal, M. (1975). Social olfaction: A review of the ontogeny
of olfactory influences on vertebrate behavior.
Behavioral Biology, 15, 1–25. http://dx.doi.org/10.1016/
S0091-6773(75)92040-4
Chong, S. C. F., Werker, J. F., Russell, J. A., & Carroll, J. M.
(2003). Three facial expressions mothers direct
to their infants. Infant and Child Development, 12,
211–232. http://dx.doi.org/10.1002/icd.286
Christakis, N. A., & Fowler, J. H. (2014). Friendship
and natural selection. Proceedings of the National
Academy of Sciences, USA, 111(Suppl. 3),
10796–10801. http://dx.doi.org/10.1073/pnas.
1400825111
Cole, P. M. (1986). Children’s spontaneous control of
facial expression. Child Development, 57, 1309–1321.
http://dx.doi.org/10.2307/1130411
Cole, P. M., Jenkins, P. A., & Shott, C. T. (1989).
Spontaneous expressive control in blind and sighted
children. Child Development, 60, 683–688. http://
dx.doi.org/10.2307/1130733
Cole, P. M., Tamang, B. L., & Shrestha, S. (2006).
Cultural variations in the socialization of young
children’s anger and shame. Child Development,
77, 1237–1251. http://dx.doi.org/10.1111/j.1467-
8624.2006.00931.x
Colonnesi, C., Zijlstra, B. J. H., van der Zande, A., &
Bögels, S. M. (2012). Coordination of gaze, facial
expressions and vocalizations of early infant
communication with mother and father. Infant
Behavior and Development, 35, 523–532. http://
dx.doi.org/10.1016/j.infbeh.2012.02.004
Croy, I., Olgun, S., & Joraschky, P. (2011). Basic
emotions elicited by odors and pictures. Emotion, 11,
1331–1335. http://dx.doi.org/10.1037/a0024437
128
de Gelder, B. (2009). Why bodies? Twelve reasons
for including bodily expressions in affective
neuroscience. Philosophical Transactions of the Royal
Society B: Biological Sciences, 364, 3475–3484.
de Gelder, B., Van den Stock, J., Meeren, H. K., Sinke,
C. B., Kret, M. E., & Tamietto, M. (2010). Standing
up for the body: Recent progress in uncovering the
networks involved in the perception of bodies and
bodily expressions. Neuroscience and Biobehavioral
Reviews, 34, 513–527. http://dx.doi.org/10.1016/
j.neubiorev.2009.10.008
de Groot, J. H. B., Semin, G. R., & Smeets, M. A.
(2014). I can see, hear, and smell your fear:
Comparing olfactory and audiovisual media in fear
communication. Journal of Experimental Psychology:
General, 143, 825–834. http://dx.doi.org/10.1037/
a0033731
de Groot, J. H. B., Smeets, M. A. M., Kaldewaij, A.,
Duijndam, M. J. A., & Semin, G. R. (2012).
Chemosignals communicate human emotions.
Psychological Science, 23, 1417–1424. http://dx.doi.
org/10.1177/0956797612445317
D’Entremont, B., Hains, S. M. J., & Muir, D. W. (1997).
A demonstration of gaze following in 3- to 6-month-
olds. Infant Behavior and Development, 20, 569–572.
DePaulo, B. M. (1992). Nonverbal behavior and self-
presentation. Psychological Bulletin, 111, 203–243.
http://dx.doi.org/10.1037/0033-2909.111.2.203
De Waal, F. (1990). Peacemaking among primates.
Cambridge, MA: Harvard University Press.
Doi, H., & Shinohara, K. (2012). Event-related potentials
elicited in mothers by their own and unfamiliar
infants’ faces with crying and smiling expression.
Neuropsychologia, 50, 1297–1307. http://dx.doi.
org/10.1016/j.neuropsychologia.2012.02.013
Dunbar, R. I. M. (2010). The social role of touch in
humans and primates: Behavioural function and
neurobiological mechanisms. Neuroscience and
Biobehavioral Reviews, 34, 260–268. http://dx.doi.
org/10.1016/j.neubiorev.2008.07.001
Egloff, B., & Schmukle, S. C. (2002). Predictive validity
of an Implicit Association Test for assessing anxiety.
Journal of Personality and Social Psychology, 83,
1441–1455. http://dx.doi.org/10.1037/0022-
3514.83.6.1441
Eibl-Eibesfeldt, I. (1972). Similarities and differences
between cultures in expressive movements.
In R. Hinde (Ed.), Nonverbal communication
(pp. 297–312). New York, NY: Cambridge University
Press.
Eibl-Eibesfeldt, I. (1973). The expressive behavior of
the deaf-and-blind born. In M. von Cranach &
I. Vine (Eds.), Social communication and movement
(pp. 163–194). London, England: Academic Press.

Ekman, P. (1972). Universal and cultural differences in
facial expressions of emotion. In J. K. Coles (Ed.),
Nebraska Symposium on Motivation (pp. 207–283).
Lincoln: University of Nebraska Press.
Ekman, P., & Friesen, W. V. (1969). Nonverbal leakage
and clues to deception. Psychiatry, 32, 88–106.
Ekman, P., & Friesen, W. V. (1974). Detecting deception
from body or face. Journal of Personality and Social
Psychology, 29, 288–298. http://dx.doi.org/10.1037/
h0036006
Ekman, P., & Friesen, W. V. (1978). FACS: Facial Action
Coding System. Retrieved from http://www-2.cs.cmu.
edu/afs/cs/project/face/www/facs.htm
Ekman, P., Friesen, W. V., & Simons, R. C. (1985). Is the
startle reaction an emotion? Journal of Personality
and Social Psychology, 49, 1416–1426. http://dx.doi.
org/10.1037/0022-3514.49.5.1416
Ekman, P., & Heider, K. G. (1988). The universality of a
contempt expression: A replication. Motivation and
Emotion, 12, 303–308. http://dx.doi.org/10.1007/
BF00993116
Ekman, P., Levenson, R. W., & Friesen, W. (1983).
Autonomic nervous system activity distinguishes
among emotions. Science, 221, 1208–1210. http://
dx.doi.org/10.1126/science.6612338
Elfenbein, H. A., & Ambady, N. (2002). On the universality
and cultural specificity of emotion recognition: A meta-
analysis. Psychological Bulletin, 128, 203–235. http://
dx.doi.org/10.1037/0033-2909.128.2.203
Elfenbein, H. A., & Ambady, N. (2003). Universals and
cultural differences in recognizing emotions. Current
Directions in Psychological Science, 12, 159–164.
http://dx.doi.org/10.1111/1467-8721.01252
Éthier-Majcher, C., Joubert, S., & Gosselin, F. (2013).
Reverse correlating trustworthy faces in young and
older adults. Frontiers in Psychology, 4, 592. http://
dx.doi.org/10.3389/fpsyg.2013.00592
Fantz, R. L. (1964). Visual experience in infants:
Decreased attention to familiar patterns relative to
novel ones. Science, 146, 668–670. http://dx.doi.
org/10.1126/science.146.3644.668
Farran, D. C., & Kasari, C. (1990). A longitudinal
analysis of the development of synchrony in mutual
gaze in mother–child dyads. Journal of Applied
Developmental Psychology, 11, 419–430. http://dx.doi.
org/10.1016/0193-3973(90)90018-F
Farroni, T., Csibra, G., Simion, F., & Johnson, M. H.
(2002). Eye contact detection in humans from birth.
Proceedings of the National Academy of Sciences,
USA, 99, 9602–9605. http://dx.doi.org/10.1073/
pnas.152159999
Farroni, T., Menon, E., Rigato, S., & Johnson, M. H.
(2007). The perception of facial expressions in
The Developmental Arc of Nonverbal Communication
newborns. European Journal of Developmental
Psychology, 4, 2–13. http://dx.doi.org/10.1080/
17405620601046832
Fehr, B. J., & Exline, R. V. (1987). Social visual
interaction: A conceptual and literature review. In A.
W. Siegman & S. Feldstein (Eds.), Nonverbal behavior
and communication (2nd ed., pp. 225–325).
Hillsdale, NJ: Erlbaum.
Feldman, R. (2007). On the origins of background
emotions: From affect synchrony to symbolic
expression. Emotion, 7, 601–611. http://dx.doi.org/
10.1037/1528-3542.7.3.601
Feldman, R. (2011). Maternal touch and the developing
infant. In M. J. Hertenstein & S. J. Weiss (Eds.), The
handbook of touch (pp. 373–408). New York, NY:
Springer.
Feldman, R. (2012). Parent–infant synchrony: A bio-
behavioral model of mutual influences in the
formation of affiliative bonds. Monographs of the
Society for Research in Child Development, 77, 42–51.
http://dx.doi.org/10.1111/j.1540-5834.2011.00660.x
Feldman, R., Eidelman, A. I., Sirota, L., & Weller, A.
(2002). Comparison of skin-to-skin (kangaroo) and
traditional care: Parenting outcomes and preterm
infant development. Pediatrics, 110, 16–26. http://
dx.doi.org/10.1542/peds.110.1.16
Feldman, R., Masalha, S., & Alony, D. (2006).
Microregulatory patterns of family interactions:
Cultural pathways to toddlers’ self-regulation.
Journal of Family Psychology, 20, 614–623. http://
dx.doi.org/10.1037/0893-3200.20.4.614
Feldman, R., Masalha, S., & Nadam, R. (2001). Cultural
perspective on work and family: Dual-earner
Israeli-Jewish and Arab families at the transition
to parenthood. Journal of Family Psychology,
15, 492–509. http://dx.doi.org/10.1037/0893-
3200.15.3.492
Feldman, R., Singer, M., & Zagoory, O. (2010). Touch
attenuates infants’ physiological reactivity to stress.
Developmental Science, 13, 271–278. http://dx.doi.
org/10.1111/j.1467-7687.2009.00890.x
Feldman, R. S., Tomasian, J. C., & Coats, E. J. (1999).
Nonverbal deception abilities and adolescents’ social
competence: Adolescents with higher social skills are
better liars. Journal of Nonverbal Behavior, 23,
237–249. http://dx.doi.org/10.1023/A:1021369327584
Ferber, S. G., Feldman, R., & Makhoul, I. R. (2008).
The development of maternal touch across the first
year of life. Early Human Development, 84, 363–370.
http://dx.doi.org/10.1016/j.earlhumdev.2007.09.019
Field, T. M. (2007). The amazing newborn. Malden, MA:
Blackwell.
Field, T. M. (2011). Massage therapy: A review of recent
research In M. J. Hertenstein & S. J. Weiss (Eds.),
129
Caroline F. Keating
The handbook of touch: Neuroscience, behavioral, and
health perspectives (pp. 455–465). New York, NY:
Springer.
Field, T. M., Woodson, R., Greenberg, R., & Cohen, D.
(1982). Discrimination and imitation of facial
expression by neonates. Science, 218, 179–181.
http://dx.doi.org/10.1126/science.7123230
Figueredo, A. J., Hammond, K. R., & McKiernan, E. C.
(2006). A Brunswikian evolutionary developmental
theory of preparedness and plasticity. Intelligence,
34, 211–227. http://dx.doi.org/10.1016/
j.intell.2005.03.006
Firestone, A., Turk-Browne, N., & Ryan, J. (2007). Age-
related deficits in face recognition are related to
underlying changes in scanning behavior. Aging,
Neuropsychology, and Cognition, 14, 594–607. http://
dx.doi.org/10.1080/13825580600899717
Flom, R., Whipple, H., & Hyde, D. (2009). Infants’
intermodal perception of canine (Canis familairis)
facial expressions and vocalizations. Developmental
Psychology, 45, 1143–1151. http://dx.doi.org/
10.1037/a0015367
Fogel, A., Nwokah, E., Dedo, J. Y., Messinger, D.,
Dickson, K. L., Matusov, E., & Holt, S. A. (1992).
Social process theory of emotion: A dynamic systems
approach. Social Development, 1, 122–142. http://
dx.doi.org/10.1111/j.1467-9507.1992.tb00116.x
Fox, R., & McDaniel, C. (1982). The perception of
biological motion by human infants. Science, 218,
486–487. http://dx.doi.org/10.1126/science.7123249
Fridlund, A. J. (1997). The new ethology of human
facial expressions. In J. A. Russell & J. M.
Fernández-Dols (Eds.), The psychology of facial
expression (pp. 103–129). http://dx.doi.org/10.1017/
CBO9780511659911.007
Fridlund, A. J., & Russell, J. A. (2006). The functions
of facial expressions: What’s in a face? In V.
Manusov & M. L. Patterson (Eds.), The Sage
handbook of nonverbal communication (pp. 299–320).
http://dx.doi.org/10.4135/9781412976152.n16
Fukuyama, H., & Myowa-Yamakoshi, M. (2013).
Fourteen-month-old infants copy an action style
accompanied by social-emotional cues. Infant
Behavior and Development, 36, 609–617. http://
dx.doi.org/10.1016/j.infbeh.2013.06.005
Fusaro, M., Vallotton, C. D., & Harris, P. L. (2014).
Beside the point: Mothers’ head nodding and shaking
gestures during parent–child play. Infant Behavior
and Development, 37, 235–247. http://dx.doi.
org/10.1016/j.infbeh.2014.01.006
Ganchrow, J. R., Steiner, J. E., & Daher, M. (1983).
Neonatal facial expressions in response to different
qualities and intensities of gustatory stimuli. Infant
Behavior and Development, 6, 473–484. http://dx.doi.
org/10.1016/S0163-6383(83)90301-6
130
Gao, X., Maurer, D., & Nishimura, M. (2010). Similarities
and differences in the perceptual structure of
facial expressions of children and adults. Journal of
Experimental Child Psychology, 105, 98–115. http://
dx.doi.org/10.1016/j.jecp.2009.09.001
Gaspar, A., & Esteves, F. G. (2012). Preschooler’s faces in
spontaneous emotional contexts—How well do they
match adult facial expression prototypes? International
Journal of Behavioral Development, 36, 348–357.
http://dx.doi.org/10.1177/0165025412441762
Gildersleeve, K. A., Haselton, M. G., Larson, C. M., &
Pillsworth, E. G. (2012). Body odor attractiveness as
a cue of impending ovulation in women: Evidence
from a study using hormone-confirmed ovulation.
Hormones and Behavior, 61, 157–166. http://dx.doi.
org/10.1016/j.yhbeh.2011.11.005
Goodall, J. (1971). In the shadow of man. Boston, MA:
Houghton Mifflin.
Goren, C. C., Sarty, M., & Wu, P. Y. K. (1975). Visual
following and pattern discrimination of face-like
stimuli by newborn infants. Pediatrics, 56, 544–549.
Gräfenhain, A., Behne, T., Carpenter, M., & Tomasello, M.
(2009). One-year-olds’ understanding of nonverbal
gestures directed to a third person. Cognitive
Development, 24, 23–33. http://dx.doi.org/10.1016/
j.cogdev.2008.10.001
Grant, E. C. (1969). Human facial expression. Man, 4,
525–536. http://dx.doi.org/10.2307/2798193
Grossmann, T., Striano, T., & Friederici, A. D. (2007).
Developmental changes in infants’ processing
of happy and angry facial expressions: A
neurobehavioral study. Brain and Cognition, 64,
30–41. http://dx.doi.org/10.1016/j.bandc.2006.10.002
Halberstadt, A. G., Crisp, V. W., & Eaton, K. I. (1999).
Transmission of social norms regulating nonverbal
behavior. In P. Philippot, R. S. Feldman, &
E. J. Coats (Eds.), The social context of nonverbal
behavior (pp. 109–155). Cambridge, England:
Cambridge University Press.
Halberstadt, A. G., Denham, S. A., & Dunsmore, J. C.
(2001). Affective social competence. Social
Development, 10, 79–119. http://dx.doi.org/10.1111/
1467-9507.00150
Hall, E. T. (1966). The hidden dimension. New York, NY:
Doubleday.
Hall, G. B. C., Witelson, S. F., Szechtman, H., & Nahmias, C.
(2004). Sex differences in functional activation
patterns revealed by increased emotion processing
demands. NeuroReport, 15, 219–223. http://dx.doi.
org/10.1097/00001756-200402090-00001
Hall, J. A. (1984). Nonverbal sex differences:
Communication accuracy and expressive style.
Baltimore, MD: Johns Hopkins University Press.
Hall, J. A. (2006). How big are nonverbal sex differences?
The case of smiling and nonverbal sensitivity.

In K. Dindia & D. J. Canary (Eds.), Sex differences
and similarities in communication (2nd ed.,
pp. 59–81). Mahwah, NJ: Erlbaum.
Hall, J. A. (2011). Gender and status patterns in social
touch. In M. J. Hertenstein & S. J. Weiss (Eds.),
The handbook of touch: Neuroscience, behavioral, and
health perspectives (pp. 329–350). New York, NY:
Springer.
Hall, J. A., Coats, E. J., & LeBeau, L. S. (2005). Nonverbal
behavior and the vertical dimension of social
relations: A meta-analysis. Psychological Bulletin,
131, 898–924. http://dx.doi.org/10.1037/0033-
2909.131.6.898
Hall, J. K., Hutton, S. B., & Morgan, M. J. (2010). Sex
differences in scanning faces: Does attention to the
eyes explain female superiority in facial expression
recognition? Cognition and Emotion, 24, 629–637.
http://dx.doi.org/10.1080/02699930902906882
Hamlin, J. K. (2013). Moral judgment and action
in preverbal infants and toddlers evidence for
an innate moral core. Current Directions in
Psychological Science, 22, 186–193. http://dx.doi.
org/10.1177/0963721412470687
Hareli, S., Harush, R., Suleiman, R., Cossette, M.,
Bergeron, S., Lavoie, V., . . . Hess, U. (2009). When
scowling may be a good thing: The influence of anger
expressions on credibility. European Journal of Social
Psychology, 39, 631–638. http://dx.doi.org/10.1002/
ejsp.573
Hareli, S., & Hess, U. (2012). The social signal value
of emotions. Cognition and Emotion, 26, 385–389.
http://dx.doi.org/10.1080/02699931.2012.665029
Hareli, S., Shomrat, N., & Hess, U. (2009). Emotional
versus neutral expressions and perceptions of social
dominance and submissiveness. Emotion, 9, 378–384.
http://dx.doi.org/10.1037/a0015958
Harlow, H. F. (1959). Love in infant monkeys. Scientific
American, 200, 68–74. http://dx.doi.org/10.1038/
scientificamerican0659-68
Harris, C., & Alvarado, N. (2005). Facial expressions,
smile types, and self-report during humour, tickle,
and pain. Cognition and Emotion, 19, 655–669. http://
dx.doi.org/10.1080/02699930441000472
Haviland, J. M. (1977). Sex-related pragmatics in infants’
nonverbal communication. Journal of Communication,
27, 80–84. http://dx.doi.org/10.1111/j.1460-2466.
1977.tb01830.x
Haviland, J. M., & Lelwica, M. (1987). The induced affect
response: 10-week-old infants’ responses to three
emotion expressions. Developmental Psychology, 23,
97–104. http://dx.doi.org/10.1037/0012-1649.23.1.97
Hawley, P. H. (2014). Ontogeny and social dominance:
A developmental view of human power patterns.
Evolutionary Psychology, 12, 318–342.
The Developmental Arc of Nonverbal Communication
Helweg-Larsen, M., Cunningham, S. J., Carrico, A., &
Pergram, A. M. (2004). To nod or not to nod: An
observational study of nonverbal communication
and status in female and male college students.
Psychology of Women Quarterly, 28, 358–361. http://
dx.doi.org/10.1111/j.1471-6402.2004.00152.x
Herba, C. M., Landau, S., Russell, T., Ecker, C., &
Phillips, M. L. (2006). The development of emotion-
processing in children: Effects of age, emotion, and
intensity. Journal of Child Psychology and Psychiatry,
47, 1098–1106. http://dx.doi.org/10.1111/j.1469-
7610.2006.01652.x
Hertenstein, M. J. (2011). The communicative functions
of touch in adulthood. In M. J. Hertenstein & S. J.
Weiss (Eds.), The handbook of touch: Neuroscience,
behavioral, and health perspectives (pp. 299–328).
New York, NY: Springer.
Hertenstein, M. J., & Campos, J. J. (2001). Emotion
regulation via maternal touch. Infancy, 2, 549–566.
http://dx.doi.org/10.1207/S15327078IN0204_09
Hertenstein, M. J., Holmes, R., McCullough, M., &
Keltner, D. (2009). The communication of emotion
via touch. Emotion, 9, 566–573. http://dx.doi.org/
10.1037/a0016108
Hertenstein, M. J., Verkamp, J. M., Kerestes, A. M., &
Holmes, R. M. (2006). The communicative functions
of touch in humans, nonhuman primates, and rats:
A review and synthesis of the empirical research.
Genetic, Social, and General Psychology Monographs,
132, 5–94. http://dx.doi.org/10.3200/MONO.132.
1.5-94
Hertenstein, M. J., & Weiss, S. J. (Eds.). (2011). The
handbook of touch: Neuroscience, behavioral, and
health perspectives. New York, NY: Springer.
Hess, U., Adams, R. B., Jr., Simard, A., Stevenson, M. T., &
Kleck, R. E. (2012). Smiling and sad wrinkles: Age-
related changes in the face and the perception of
emotions and intentions. Journal of Experimental
Social Psychology, 48, 1377–1380. http://dx.doi.org/
10.1016/j.jesp.2012.05.018
Hoehl, S., & Striano, T. (2010). The development of
emotional face and eye gaze processing. Developmental
Science, 13, 813–825. http://dx.doi.org/
10.1111/j.1467-7687.2009.00944.x
Holodynski, M. (2004). The miniaturization of
expression in the development of emotional self-
regulation. Developmental Psychology, 40, 16–28.
http://dx.doi.org/10.1037/0012-1649.40.1.16
Horton, R. E., & Riddell, R. R. P. (2010). Mothers’ facial
expressions of pain and fear and infants’ pain response
during immunizations. Infant Mental Health Journal,
31, 397–411. http://dx.doi.org/10.1002/imhj.20262
Hunnius, S., de Wit, T. C. J., Vrins, S., & von Hofsten,
C. (2011). Facing threat: Infants’ and adults’ visual
131
Caroline F. Keating
scanning of faces with neutral, happy, sad, angry,
and fearful emotional expressions. Cognition and
Emotion, 25, 193–205. http://dx.doi.org/10.1080/
15298861003771189
Hwang, H. C., & Matsumoto, D. (2014). Dominance
threat display for victory and achievement in
competition context. Motivation and Emotion, 38,
206–214. http://dx.doi.org/10.1007/s11031-013-
9390-1
Ingram, G. P. (2014). From hitting to tattling to gossip:
An evolutionary rationale for the development of
indirect aggression. Evolutionary Psychology, 12,
343–363.
Isaacowitz, D. M., & Stanley, J. T. (2011). Bringing an
ecological perspective to the study of aging and
recognition of emotional facial expressions: Past,
current, and future methods. Journal of Nonverbal
Behavior, 35, 261–278. http://dx.doi.org/10.1007/
s10919-011-0113-6
Izard, C. E. (1971). The face of emotion. New York, NY:
Appleton-Century-Crofts.
Izard, C. E. (1992). Basic emotions, relations among
emotions, and emotion–cognition relations.
Psychological Review, 99, 561–565. http://dx.doi.
org/10.1037/0033-295X.99.3.561
Izard, C. E. (1979). The maximally discriminative facial
action coding system (MAX). Newark: University of
Delaware Instructional Resources Center.
Izard, C. E. (2007). Basic emotions, natural kinds,
emotion schemas, and a new paradigm. Perspectives
on Psychological Science, 2, 260–280. http://dx.doi.
org/10.1111/j.1745-6916.2007.00044.x
Kahlbaugh, P. E., & Haviland, J. M. (1994). Nonverbal
communication between parents and adolescents: A
study of approach and avoidance behaviors. Journal
of Nonverbal Behavior, 18, 91–113. http://dx.doi.org/
10.1007/BF02169080
Keating, C. F. (1985). Human dominance signals: The
primate in us. In S. L. Ellyson & J. F. Dovidio
(Eds.), Power, dominance, and nonverbal behavior
(pp. 89–108). http://dx.doi.org/10.1007/978-1-4612-
5106-4_5
Keating, C. F. (2002). Charismatic faces: Social status
cues put face appeal in context. In G. Rhodes & L. A.
Zebrowitz (Eds.), Facial attractiveness: Evolutionary,
cognitive, and social perspectives (pp. 153–192).
Westport, CT: Ablex Publishing.
Keating, C. F. (2006). Why and how the silent self
speaks volumes: Functional approaches to nonverbal
impression management. In V. L. Manusov & M.
L. Patterson (Eds.), The Sage handbook of nonverbal
communication (pp. 321–340). Thousand Oaks, CA:
Sage.
Keating, C. F., & Bai, D. L. (1986). Children’s
attributions of social dominance from facial cues.
132
Child Development, 57, 1269–1276. http://dx.doi.org/
10.2307/1130449
Keating, C. F., & Heltman, K. R. (1994). Dominance and
deception in children and adults: Are leaders the
best misleaders? Personality and Social Psychology
Bulletin, 20, 312–321. http://dx.doi.org/10.1177/
0146167294203009
Keating, C. F., Little, B. C., & Colligan, M. E. (2012,
May). Leadership emergence in teenage campers:
Arriving nicely, departing less nice. Paper presented
at the annual meeting of the Association for
Psychological Science, Chicago, IL.
Keating, C. F., Mazur, A., Segall, M. H., Cysneiros, P. G.,
Kilbride, J. E., Leahy, P., . . . Wirsing, R. (1981).
Culture and the perception of social dominance from
facial expression. Journal of Personality and Social
Psychology, 40, 615–626. http://dx.doi.org/10.1037/
0022-3514.40.4.615
Keltner, D. (1995). Signs of appeasement: Evidence for
the distinct displays of embarrassment, amusement,
and shame. Journal of Personality and Social
Psychology, 68, 441–454. http://dx.doi.org/10.1037/
0022-3514.68.3.441
Kilbride, J. E., & Kilbride, P. L. (1975). Sitting and
smiling behavior of Baganda infants: The influence
of culturally constituted experience. Journal of
Cross-Cultural Psychology, 6, 88–107. http://dx.doi.
org/10.1177/002202217500600103
Knapp, M. L., Hall, J. A., & Horgan, T. G. (2014).
Nonverbal communication in human interaction (8th
ed.). Boston, MA: Wadsworth/Cengage Learning.
Kobiella, A., Grossmann, T., Reid, V., & Striano, T.
(2008). The discrimination of angry and fearful facial
expressions in 7-month-old infants: An event-related
potential study. Cognition and Emotion, 22, 134–146.
http://dx.doi.org/10.1080/02699930701394256
Kohut, S. A., Riddell, R. P., Flora, D. B., & Oster, H.
(2012). A longitudinal analysis of the development
of infant facial expressions in response to acute pain:
Immediate and regulatory expressions. Pain, 153,
2458–2465. http://dx.doi.org/10.1016/j.pain.
2012.09.005
Konner, M. (2010). The evolution of childhood.
Cambridge, MA: Harvard University Press.
Kraus, M. W., Huang, C., & Keltner, D. (2010). Tactile
communication, cooperation, and performance: An
ethological study of the NBA. Emotion, 10, 745–749.
http://dx.doi.org/10.1037/a0019382
LaBarbera, J. D., Izard, C. E., Vietze, P., & Parisi, S. A.
(1976). Four- and six-month-old infants’ visual
responses to joy, anger, and neutral expressions.
Child Development, 47, 535–538. http://dx.doi.
org/10.2307/1128816
LaFrance, M., Hecht, M. A., & Paluck, E. L. (2003). The
contingent smile: A meta-analysis of sex differences

in smiling. Psychological Bulletin, 129, 305–334.
http://dx.doi.org/10.1037/0033-2909.129.2.305
LeDoux, J. (2007). The amygdala. Current Biology, 17,
868–874. http://dx.doi.org/10.1016/j.cub.2007.
08.005
Lee, N. C., Krabbendam, L., White, T. P., Meeter, M.,
Banaschewski, T., Barker, G. J., . . . Shergill, S. S.
(2013). Do you see what I see? Sex differences in the
discrimination of facial emotions during adolescence.
Emotion, 13, 1030–1040. http://dx.doi.org/10.1037/
a0033560
Leeb, R. T., & Rejskind, F. G. (2004). Here’s looking at
you, kid! A longitudinal study of perceived gender
differences in mutual gaze behavior in young infants.
Sex Roles, 50, 1–14. http://dx.doi.org/10.1023/
B:SERS.0000011068.42663.ce
Leppänen, J. M., & Nelson, C. A. (2006). The
development and neural bases of facial emotion and
recognition. In R. V. Kail (Ed.), Advances in child
development and behavior (Vol. 34, pp. 207–246).
http://dx.doi.org/10.1016/S0065-2407(06)80008-X
Leppänen, J. M., & Nelson, C. A. (2009). Tuning the
developing brain to social signals of emotions. Nature
Reviews Neuroscience, 10, 37–47. http://dx.doi.
org/10.1038/nrn2554
Leppänen, J. M., & Nelson, C. A. (2012). Early
development of fear processing. Current Directions
in Psychological Science, 21, 200–204. http://dx.doi.
org/10.1177/0963721411435841
Leppänen, J. M., Richmond, J., Vogel-Farley, V. K.,
Moulson, M. C., & Nelson, C. A. (2009). Categorical
representation of facial expressions in the infant
brain. Infancy, 14, 346–362. http://dx.doi.org/
10.1080/15250000902839393
Levav, J., & Argo, J. J. (2010). Physical contact and financial
risk taking. Psychological Science, 21, 804–810. http://
dx.doi.org/10.1177/0956797610369493
Levenson, R. W., Carstensen, L. L., Friesen, W. V., &
Ekman, P. (1991). Emotion, physiology, and
expression in old age. Psychology and Aging, 6,
28–35. http://dx.doi.org/10.1037/0882-7974.6.1.28
Lewis, M., & Michalson, L. (Eds.). (1983). Children’s
emotions and moods: Developmental theory and
measurement. New York, NY: Plenum.
Lewis, M., Sullivan, M. W., & Vasen, A. (1987). Making
faces: Age and emotion differences in the posing of
emotional expressions. Developmental Psychology, 23,
690–697. http://dx.doi.org/10.1037/0012-1649.
23.5.690
Lewis, M. D., & Granic, I. (Eds.). (2002). Emotion,
development, and self-organization: Dynamic systems
approaches to emotional development. Cambridge,
England: Cambridge University Press.
The Developmental Arc of Nonverbal Communication
Li, W., Moallem, I., Paller, K. A., & Gottfried, J. A.
(2007). Subliminal smells can guide social
preferences. Psychological Science, 18, 1044–1049.
http://dx.doi.org/10.1111/j.1467-9280.2007.02023.x
LoBue, V., Matthews, K., Harvey, T., & Thrasher, C.
(2014). Pick on someone your own size: The
detection of threatening facial expressions posed by
both child and adult models. Journal of Experimental
Child Psychology, 118, 134–142. http://dx.doi.
org/10.1016/j.jecp.2013.07.016
Lockard, J. S., Fahrenbruch, C. E., Smith, J. L., &
Morgan, C. J. (1977). Smiling and laughter: Different
phyletic origins? Bulletin of the Psychonomic
Society, 10, 183–186. http://dx.doi.org/10.3758/
BF03329317
Main, M., & George, C. (1985). Responses of abused
and disadvantaged toddlers to distress in agemates:
A study in the day care setting. Developmental
Psychology, 21, 407–412. http://dx.doi.org/10.1037/
0012-1649.21.3.407
Malatesta, C. Z., Fiore, M. J., & Messina, J. J. (1987).
Affect, personality, and facial expressive
characteristics of older people. Psychology and
Aging, 2, 64–69. http://dx.doi.org/10.1037/0882-
7974.2.1.64
Mallet, P., & Schaal, B. (1998). Rating and recognition
of peers’ personal odors by 9-year-old children: An
exploratory study. Journal of General Psychology, 125,
47–64. http://dx.doi.org/10.1080/00221309809595576
Mancini, G., Agnoli, S., Baldaro, B., Ricci Bitti, P. E., &
Surcinelli, P. (2013). Facial expressions of emotions:
Recognition accuracy and affective reactions during
late childhood. Journal of Psychology: Interdisciplinary
and Applied, 147, 599–617. http://dx.doi.org/10.1080/
00223980.2012.727891
Marsh, A. A., Elfenbein, H. A., & Ambady, N. (2003).
Nonverbal “accents”: Cultural differences in facial
expressions of emotion. Psychological Science,
14, 373–376. http://dx.doi.org/10.1111/1467-
9280.24461
Marsh, A. A., Elfenbein, H. A., & Ambady, N. (2007).
Separated by a common language: Nonverbal accents
and cultural stereotypes about Americans and
Australians. Journal of Cross-Cultural Psychology, 38,
284–301. http://dx.doi.org/10.1177/
0022022107300275
Mascaro, O., & Csibra, G. (2012). Representation of
stable social dominance relations by human infants.
Proceedings of the National Academy of Sciences,
USA, 109, 6862–6867. http://dx.doi.org/10.1073/
pnas.1113194109
Matsumoto, D., Olide, A., Schug, J., Willingham, B., &
Callan, M. (2009). Cross-cultural judgments of
spontaneous facial expressions of emotion. Journal
133
Caroline F. Keating
of Nonverbal Behavior, 33, 213–238. http://dx.doi.
org/10.1007/s10919-009-0071-4
Matsumoto, D., & Willingham, B. (2009). Spontaneous
facial expressions of emotion of congenitally
and noncongenitally blind individuals. Journal of
Personality and Social Psychology, 96, 1–10. http://
dx.doi.org/10.1037/a0014037
Matsumoto, D., Yoo, S. H., & Fontaine, J. (2008).
Mapping expressive differences around the world:
The relationship between emotional display rules
and individualism versus collectivism. Journal of
Cross-Cultural Psychology, 39, 55–74. http://dx.doi.
org/10.1177/0022022107311854
McClure, E. B. (2000). A meta-analytic review of sex
differences in facial expression processing and their
development in infants, children, and adolescents.
Psychological Bulletin, 126, 424–453. http://dx.doi.
org/10.1037/0033-2909.126.3.424
McDermott, R., Tingley, D., & Hatemi, P. K. (2014).
Assortative mating on ideology could operate
through olfactory cues. American Journal of Political
Science, 58, 997–1005. http://dx.doi.org/10.1111/
ajps.12133
Meadors, J. D., & Murray, C. B. (2014). Measuring
nonverbal bias through body language responses
to stereotypes. Journal of Nonverbal Behavior, 38,
209–229. http://dx.doi.org/10.1007/s10919-013-
0172-y
Mehrabian, A. (1971). Silent messages. Belmont, CA:
Wadsworth.
Meltzoff, A. N., & Moore, M. K. (1977). Imitation of
facial and manual gestures by human neonates.
Science, 198, 75–78. http://dx.doi.org/10.1126/
science.897687
Mesman, J., Linting, M., Joosen, K. J., Bakermans-
Kranenburg, M. J., & van Ijzendoorn, M. H. (2013).
Robust patterns and individual variations: Stability and
predictors of infant behavior in the still-face paradigm.
Infant Behavior and Development, 36, 587–598. http://
dx.doi.org/10.1016/j.infbeh.2013.06.004
Mesman, J., Oster, H., & Camras, L. (2012). Parental
sensitivity to infant distress: What do discrete
negative emotions have to do with it? Attachment and
Human Development, 14, 337–348. http://dx.doi.org/
10.1080/14616734.2012.691649
Messinger, D. S., Fogel, A., & Dickson, K. L. (2001).
All smiles are positive, but some smiles are more
positive than others. Developmental Psychology,
37, 642–653. http://dx.doi.org/10.1037/0012-
1649.37.5.642
Messinger, D. S., Mattson, W. I., Mahoor, M. H., &
Cohn, J. F. (2012). The eyes have it: Making positive
expressions more positive and negative expressions
more negative. Emotion, 12, 430–436. http://dx.doi.
org/10.1037/a0026498
134
Mineka, S., & Ohman, A. (2002). Phobias and preparedness:
The selective, automatic, and encapsulated nature of
fear. Biological Psychiatry, 52, 927–937. http://dx.doi.
org/10.1016/S0006-3223(02)01669-4
Mitchell, G., & Maple, T. L. (1985). Dominance in
nonhuman primates. In S. L. Ellyson & J. F.
Dovidio (Eds.), Power, dominance and nonverbal
communication (pp. 49–66). http://dx.doi.org/
10.1007/978-1-4612-5106-4_3
Mizokawa, A., Minemoto, K., Komiya, A., & Noguchi,
M. (2013). The effects of infants’ and adults’ facial
expression on approach and avoidance behavior.
Psychologia, 56, 33–44.
Montagu, A. (1971). Touching: The human significance of
the skin. New York, NY: Harper & Row.
Montague, D. P. F., & Walker-Andrews, A. S. (2001).
Peekaboo: A new look at infants’ perception of
emotion expressions. Developmental Psychology,
37, 826–838. http://dx.doi.org/10.1037/0012-
1649.37.6.826
Montepare, J. M., Goldstein, S. B., & Clausen, A. (1987).
The identification of emotions from gait information.
Journal of Nonverbal Behavior, 11, 33–42. http://
dx.doi.org/10.1007/BF00999605
Montepare, J. M., & Zebrowitz, L. A. (1993). A cross-
cultural comparison of impressions created by
age-related variations in gait. Journal of Nonverbal
Behavior, 17, 55–68. http://dx.doi.org/10.1007/
BF00987008
Montirosso, R., Peverelli, M., Frigerio, E., Crespi, M., &
Borgatti, R. (2010). The development of dynamic
facial expression recognition at different intensities
in 4- to 18-year-olds. Social Development, 19, 71–92.
http://dx.doi.org/10.1111/j.1467-9507.2008.00527.x
Moore, M. M. (2010). Human nonverbal courtship
behavior—A brief historical review. Journal of Sex
Research, 47, 171–180. http://dx.doi.org/10.1080/
00224490903402520
Nakato, E., Otsuka, Y., Kanazawa, S., Yamaguchi, M. K., &
Kakigi, R. (2011). Distinct differences in the
pattern of hemodynamic response to happy and
angry facial expressions in infants—A near-infrared
spectroscopic study. NeuroImage, 54, 1600–1606.
http://dx.doi.org/10.1016/j.neuroimage.2010.09.021
Neiss, M. B., Leigland, L. A., Carlson, N. E., & Janowsky,
J. S. (2009). Age differences in perception and
awareness of emotion. Neurobiology of Aging,
30, 1305–1313. http://dx.doi.org/10.1016/
j.neurobiolaging.2007.11.007
Nelson, C. A., & Dolgin, K. G. (1985). The generalized
discrimination of facial expressions by seven-month-
old infants. Child Development, 56, 58–61. http://
dx.doi.org/10.2307/1130173
Nelson, E. E., & Panksepp, J. (1998). Brain substrates of
infant–mother attachment: Contributions of opioids,

oxytocin, and norepinephrine. Neuroscience and
Biobehavioral Reviews, 22, 437–452. http://dx.doi.
org/10.1016/S0149-7634(97)00052-3
Noller, A., & Callan, V. (1989). Nonverbal behavior
in families with adolescents. Journal of Nonverbal
Behavior, 13, 47–64. http://dx.doi.org/10.1007/
BF01006473
Olsson, S. B., Barnard, J., & Turri, L. (2006). Olfaction
and identification of unrelated individuals:
Examination of the mysteries of human odor
recognition. Journal of Chemical Ecology, 32,
1635–1645. http://dx.doi.org/10.1007/s10886-006-
9098-8
Oster, H. (2005). The repertoire of infant facial expressions:
An ontogenetic perspective. In J. Nadel & D. Muir
(Eds.), Emotional development (pp. 261–292). Oxford,
England: Oxford University Press.
Oster, H. (2006). Baby FACS: Facial Action Coding
System for infants and young children (Unpublished
monograph and coding manual). New York
University, New York, NY.
Oster, H., & Ekman, P. (1978). Facial behavior in
child development. In A. Collins (Ed.), Minnesota
Symposium on Child Psychology (pp. 231–276).
Hillsdale, NJ: Erlbaum.
Ostrov, J. M., & Keating, C. F. (2004). Gender
differences in preschool aggression during free play
and structured interactions: An observational study.
Social Development, 13, 255–277. http://dx.doi.org/
10.1111/j.1467-9507.2004.000266.x
Oveis, C., Gruber, J., Keltner, D., Stamper, J. L., & Boyce,
W. T. (2009). Smile intensity and warm touch as
thin slices of child and family affective style. Emotion,
9, 544–548. http://dx.doi.org/10.1037/a0016300
Panksepp, J. (2007). Neurologizing the psychology of
affects: How appraisal-based constructivism and
basic emotion theory can coexist. Perspectives on
Psychological Science, 2, 281–296. http://dx.doi.org/
10.1111/j.1745-6916.2007.00045.x
Peelen, M. V., Atkinson, A. P., Andersson, F., &
Vuilleumier, P. (2007). Emotional modulation of
body-selective visual areas. Social Cognitive and
Affective Neuroscience, 2, 274–283. http://dx.doi.
org/10.1093/scan/nsm023
Peltola, M. J., Hietanen, J. K., Forssman, L., & Leppänen,
J. M. (2013). The emergence and stability of the
attentional bias to fearful faces in infancy. Infancy,
18, 905–926. http://dx.doi.org/10.1111/infa.12013
Pine, K. J., Gurney, D. J., & Fletcher, B. (2010). The
semantic specificity hypothesis: When gestures do
not depend upon the presence of a listener. Journal
of Nonverbal Behavior, 34, 169–178. http://dx.doi.
org/10.1007/s10919-010-0089-7
Porter, R. H., Cernoch, J. M., & McLaughlin, F. J. (1983).
Maternal recognition of neonates through olfactory
The Developmental Arc of Nonverbal Communication
cues. Physiology and Behavior, 30, 151–154. http://
dx.doi.org/10.1016/0031-9384(83)90051-3
Porter, R. H., & Moore, J. D. (1981). Human kin recognition
by olfactory cues. Physiology and Behavior, 27, 493–495.
http://dx.doi.org/10.1016/0031-9384(81)90337-1
Prkachin, K. M. (2009). Assessing pain by facial expression:
Facial expression as nexus. Pain Research and
Management, 14, 53–58.
Provine, R. R. (2004). Laughing, tickling, and the
evolution of speech and self. Current Directions in
Psychological Science, 13, 215–218. http://dx.doi.
org/10.1111/j.0963-7214.2004.00311.x
Rantala, M. J., Eriksson, C. J. P., Vainikka, A., & Kortet, R.
(2006). Male steroid hormones and female
preference for male body odor. Evolution and Human
Behavior, 27, 259–269. http://dx.doi.org/
10.1016/j.evolhumbehav.2005.11.002
Rennels, J. L., & Cummings, A. J. (2013). Sex differences
in facial scanning: Similarities and dissimilarities
between infants and adults. International Journal of
Behavioral Development, 37, 111–117. http://dx.doi.
org/10.1177/0165025412472411
Rigato, S., Farroni, T., & Johnson, M. H. (2010). The
shared signal hypothesis and neural responses to
expressions and gaze in infants and adults. Social
Cognitive and Affective Neuroscience, 5, 88–97. http://
dx.doi.org/10.1093/scan/nsp037
Riggio, E., & Feldman, R. S. (Eds.). (2005). Applications
of nonverbal communication. Mahwah, NJ: Erlbaum.
Rogel, M. J. (1978). A critical evaluation of the
possibility of higher primate reproductive and sexual
pheromones. Psychological Bulletin, 85, 810–830.
http://dx.doi.org/10.1037/0033-2909.85.4.810
Rosenstein, D., & Oster, H. (1988). Differential facial
responses to four basic tastes in newborns. Child
Development, 59, 1555–1568. http://dx.doi.org/
10.2307/1130670
Ruff, H. A., & Rothbart, M. K. (1996). Attention in early
development: Themes and variations. New York, NY:
Oxford University Press.
Ruffman, T., Henry, J. D., Livingstone, V., & Phillips, L. H.
(2008). A meta-analytic review of emotion recognition
and aging: Implications for neuropsychological
models of aging. Neuroscience and Biobehavioral
Reviews, 32, 863–881. http://dx.doi.org/10.1016/
j.neubiorev.2008.01.001
Saarni, C., & Weber, H. (1999). Emotional displays and
dissemblance in childhood: Implications for self-
presentation. In P. Philippot, R. S. Feldman, & E. J.
Coats (Eds.), The social context of nonverbal behavior
(pp. 71–108). Cambridge, England: Cambridge
University Press.
Sacco, D. F., & Hugenberg, K. (2009). The look of fear
and anger: Facial maturity modulates recognition
135
Caroline F. Keating
of fearful and angry expressions. Emotion, 9, 39–49.
http://dx.doi.org/10.1037/a0014081
Schaal, B., Coureaud, G., Doucet, S., Delaunay-El Allam,
M., Moncomble, A. S., Montigny, D., . . . Holley, A.
(2009). Mammary olfactory signalisation in females
and odor processing in neonates: Ways evolved by
rabbits and humans. Behavioural Brain Research,
200, 346–358. http://dx.doi.org/10.1016/
j.bbr.2009.02.008
Scherer, K. R., Clark-Polner, E., & Mortillaro, M. (2011).
In the eye of the beholder? Universality and cultural
specificity in the expression and perception of emotion.
International Journal of Psychology, 46, 401–435.
http://dx.doi.org/10.1080/00207594.2011.626049
Schneider, C., Call, J., & Liebal, K. (2012). Onset and
early use of gestural communication in nonhuman
great apes. American Journal of Primatology, 74,
102–113. http://dx.doi.org/10.1002/ajp.21011
Schofield, T. J., Parke, R. D., Castañeda, E. K., &
Coltrane, S. (2008). Patterns of gaze between parents
and children in European American and Mexican
American families. Journal of Nonverbal Behavior,
32, 171–186. http://dx.doi.org/10.1007/s10919-008-
0049-7
Shariff, F., & Tracy, J. L. (2011). Emotion expressions:
On signals, symbols, and spandrels—A response
to Barrett (2011). Current Directions in Psychological
Science, 20, 407–408. http://dx.doi.org/
10.1177/0963721411429126
Smith, W. J. (1977). The behavior of communicating:
An ethological approach. Cambridge, MA: Harvard
University Press.
Soken, N. H., & Pick, A. D. (1999). Infants’ perception of
dynamic affective expressions: Do infants distinguish
specific expressions? Child Development, 70,
1275–1282.
Sorce, J. F., Emde, R. N., Campos, J. J., & Klinnert,
M. D. (1985). Maternal emotional signaling:
Its effect on visual cliff behavior of 1-year-olds.
Developmental Psychology, 21, 195–200. http://dx.doi.
org/10.1037/0012-1649.21.1.195
Stack, D. M., & Jean, A. D. L. (2011). Communicating
through touch: Touching during parent–infant
interactions. In M. J. Hertenstein & S. J. Weiss
(Eds.), The handbook of touch: Neuroscience,
behavioral, and health perspectives (pp. 273–298).
New York, NY: Springer.
Stern, D. N. (1974). Mother and infant at play: The
dyadic interaction involving facial, vocal, and gaze
behaviors. In M. Lewis & L. A. Rosenblum (Eds.),
The effect of the infant on its caregiver (pp. 187–232).
Oxford, England: Wiley-Interscience.
Stern, K., & McClintock, M. K. (1998). Regulation of
ovulation by human pheromones. Nature, 392,
177–179. http://dx.doi.org/10.1038/32408
136
Stevenson, R. J. (2010). An initial evaluation of the
functions of human olfaction. Chemical Senses, 35,
3–20. http://dx.doi.org/10.1093/chemse/bjp083
Striano, T., & Vaish, A. (2006). Seven- to 9-month-old
infants use facial expressions to interpret others’
actions. British Journal of Developmental Psychology,
24, 753–760. http://dx.doi.org/10.1348/
026151005X70319
Suanda, S. H., & Namy, L. L. (2013). The organization
of words and symbolic gestures in 18-month-olds’
lexicons: Evidence from a disambiguation task.
Infancy, 18, 276–288. http://dx.doi.org/10.1111/
j.1532-7078.2012.00131.x
Sugden, N. A., Mohamed-Ali, M. I., & Moulson, M. C.
(2014). I spy with my little eye: Typical, daily
exposure to faces documented from a first-person
infant perspective. Developmental Psychobiology, 56,
249–261. http://dx.doi.org/10.1002/dev.21183
Sullivan, S., Ruffman, T., & Hutton, S. (2007). Age
differences in emotion recognition of skills and
the visual scanning of emotion faces. The Journals
of Gerontology, Series B: Psychological Sciences and
Social Sciences, 62, 53–60. http://dx.doi.org/10.1093/
geronb/62.1.P53
Super, C. M., & Harkness, S. (1986). The developmental
niche: A conceptualization at the interface of child
and culture. International Journal of Behavioral
Development, 9, 545–569.
Super, C. M., & Harkness, S. (2002). Culture structures
the environment for development. Human
Development, 45, 270–274. http://dx.doi.org/
10.1159/000064988
Sze, J. A., Goodkind, M. S., Gyurak, A., & Levenson, R. W.
(2012). Aging and emotion recognition: Not just a
losing matter. Psychology and Aging, 27, 940–950.
http://dx.doi.org/10.1037/a0029367
Thomas, K. M., Drevets, W. C., Whalen, P. J., Eccard, C. H.,
Dahl, R. E., Ryan, N. D., & Casey, B. J. (2001).
Amygdala response to facial expressions in children
and adults. Biological Psychiatry, 49, 309–316.
Thomas, L. A., De Bellis, M. D., Graham, R., & LaBar,
K. S. (2007). Development of emotional facial
recognition in late childhood and adolescence.
Developmental Science, 10, 547–558. http://dx.doi.
org/10.1111/j.1467-7687.2007.00614.x
Todorov, A., Baron, S. G., & Oosterhof, N. N.
(2008). Evaluating face trustworthiness: A model
based approach. Social Cognitive and Affective
Neuroscience, 3, 119–127. http://dx.doi.org/10.1093/
scan/nsn009
Tomasello, M., Carpenter, M., Call, J., Behne, T., & Moll,
H. (2005). Understanding and sharing intentions:
The origins of cultural cognition. Behavioral
and Brain Sciences, 28, 675–691. http://dx.doi.
org/10.1017/S0140525X05000129

Tomkins, S. S. (1962). Affect, imagery, and consciousness:
Vol. 1. The positive affects. New York, NY: Springer.
Tooby, J., & Cosmides, L. (1992). Psychological
foundations of culture. In J. H. Barkow, L. Cosmides, &
J. Tooby (Eds.), The adapted mind: Evolutionary
psychology and the generation of culture (pp. 19–136).
New York, NY: Oxford University Press.
Tracy, J. L., & Matsumoto, D. (2008). The spontaneous
expression of pride and shame: Evidence for
biologically innate nonverbal displays. Proceedings
of the National Academy of Sciences, USA, 105,
11655–11660. http://dx.doi.org/10.1073/pnas.
0802686105
Tracy, J. L., & Robins, R. W. (2008). The automaticity
of emotion recognition. Emotion, 8, 81–95. http://
dx.doi.org/10.1037/1528-3542.8.1.81
Tracy, J. L., Shariff, A. F., Zhao, W., & Henrich, J.
(2013). Cross-cultural evidence that the nonverbal
expression of pride is an automatic status signal.
Journal of Experimental Psychology: General, 142,
163–180. http://dx.doi.org/10.1037/a0028412
Turati, C., Macchi Cassia, V., Simion, F., & Leo, I.
(2006). Newborns’ face recognition: Role of inner
and outer facial features. Child Development,
77, 297–311. http://dx.doi.org/10.1111/j.1467-
8624.2006.00871.x
Turati, C., Montirosso, R., Brenna, V., Ferrara, V., &
Borgatti, R. (2011). A smile enhances 3-month-
olds’ recognition of an individual face. Infancy,
16, 306–317. http://dx.doi.org/10.1111/j.1532-
7078.2010.00047.x
Umiltà, M. A., Wood, R., Loffredo, F., Ravera, R., &
Gallese, V. (2013). Impact of civil war on emotion
recognition: The denial of sadness in Sierra Leone.
Frontiers in Psychology, 4, 523. http://dx.doi.
org/10.3389/fpsyg.2013.00523
Vaish, A., Grossmann, T., & Woodward, A. (2008). Not
all emotions are created equal: The negativity bias in
social-emotional development. Psychological Bulletin,
134, 383–403. http://dx.doi.org/10.1037/0033-
2909.134.3.383
van Hooff, J. A. (1972). A comparative approach to the
phylogeny of laughter and smiling. In R. A. Hinde
(Ed.), Nonverbal communication (pp. 209–238).
Cambridge, England: Cambridge University Press.
Volkmar, F. R., & Siegel, A. E. (1982). Responses to
consistent and discrepant social communications.
In R. S. Feldman (Ed.), Development of nonverbal
behavior in children (pp. 231–255). New York, NY:
Springer.
von Salisch, M. (2001). Children’s emotional
development: Challenges in their relationships to
parents, peers, and friends. International Journal of
Behavioral Development, 25, 310–319. http://dx.doi.
org/10.1080/01650250143000058
The Developmental Arc of Nonverbal Communication
Walker-Andrews, A. S. (1997). Infants’ perception of
expressive behaviors: Differentiation of multimodal
information. Psychological Bulletin, 121, 437–456.
http://dx.doi.org/10.1037/0033-2909.121.3.437
Walker-Andrews, A. S., Krogh-Jespersen, S., Mayhew,
E. M. Y., & Coffield, C. N. (2011). Young infants’
generalization of emotional expressions: Effects
of familiarity. Emotion, 11, 842–851. http://dx.doi.
org/10.1037/a0024435
Weinberg, M. K., & Tronick, E. Z. (1994). Beyond
the face: An empirical study of infant affective
configurations of facial, vocal, gestural, and
regulatory behaviors. Child Development, 65,
1503–1515. http://dx.doi.org/10.2307/1131514
Weisfeld, G. E., & Beresford, J. M. (1982). Erectness of
posture as an indicator of dominance or success in
humans. Motivation and Emotion, 6, 113–131. http://
dx.doi.org/10.1007/BF00992459
Weisfeld, G. E., Czilli, T., Phillips, K. A., Gall, J.
A., & Lichtman, C. M. (2003). Possible olfaction-
based mechanisms in human kin recognition and
inbreeding avoidance. Journal of Experimental Child
Psychology, 85, 279–295. http://dx.doi.org/10.1016/
S0022-0965(03)00061-4
Weiss, S. J., Wilson, P., Seed, M. S. J., & Paul, S. M.
(2001). Early tactile experience of low birth weight
children: Links to later mental health and social
adaptation. Infant and Child Development, 10, 93–115.
http://dx.doi.org/10.1002/icd.236
Wells, D. L., & Hepper, P. G. (2000). The discrimination
of dog odours by humans. Perception, 29, 111–115.
http://dx.doi.org/10.1068/p2938
Widen, S. C., & Naab, P. (2012). Can an anger face
also be scared? Malleability of facial expressions.
Emotion, 12, 919–925. http://dx.doi.org/10.1037/
a0026119
Widen, S. C., & Russell, J. A. (2003). A closer look
at preschoolers’ freely produced labels for facial
expressions. Developmental Psychology, 39, 114–128.
http://dx.doi.org/10.1037/0012-1649.
39.1.114
Widen, S. C., & Russell, J. A. (2008). Children acquire
emotion categories gradually. Cognitive Development,
23, 291–312. http://dx.doi.org/10.1016/
j.cogdev.2008.01.002
Widen, S. C., & Russell, J. A. (2010). Children’s scripts
for social emotions: Causes and consequences are
more central than are facial expressions. British
Journal of Developmental Psychology, 28, 565–581.
http://dx.doi.org/10.1348/026151009X457550d
Wieser, M. J., Mühlberger, A., Kenntner-Mabiala, R., &
Pauli, P. (2006). Is emotion processing affected by
advancing age? An event-related brain potential
study. Brain Research, 1096, 138–147. http://dx.doi.
org/10.1016/j.brainres.2006.04.028
137
Caroline F. Keating
Wilcox, T., Stubbs, J. A., Wheeler, L., & Alexander, G. M.
(2013). Infants’ scanning of dynamic faces during
the first year. Infant Behavior and Development,
36, 513–516. http://dx.doi.org/10.1016/
j.infbeh.2013.05.001
Wörmann, V., Holodynski, M., Kärtner, J., & Keller, H.
(2012). A cross-cultural comparison of the development
of the social smile: A longitudinal study of maternal
and infant imitation in 6- and 12-week-old infants.
Infant Behavior and Development, 35, 335–347. http://
dx.doi.org/10.1016/j.infbeh.2012.03.002
Yeshurun, Y., & Sobel, N. (2010). An odor is not worth
a thousand words: From multidimensional odors
to unidimensional odor objects. Annual Review of
Psychology, 61, 219–241. http://dx.doi.org/10.1146/
annurev.psych.60.110707.163639
Zhou, W., & Chen, D. (2009a). Fear-related chemosignals
modulate recognition of fear in ambiguous facial
expressions. Psychological Science, 20, 177–183.
http://dx.doi.org/10.1111/j.1467-9280.2009.02263.x
138
Zhou, W., & Chen, D. (2009b). Sociochemosensory
and emotional functions: Behavioral evidence for
shared mechanisms. Psychological Science, 20,
1118–1124. http://dx.doi.org/10.1111/j.1467-
9280.2009.02413.x
Zieber, N., Kangas, A., Hock, A., & Bhatt, R. S.
(2014). Infants’ perception of emotion from body
movements. Child Development, 85, 675–684. http://
dx.doi.org/10.1111/cdev.12134
Zitek, E. M., & Tiedens, L. Z. (2012). The fluency of
social hierarchy: The ease with which hierarchical
relationships are seen, remembered, learned, and
liked. Journal of Personality and Social Psychology,
102, 98–115. http://dx.doi.org/10.1037/
a0025345
Zivin, G. (1977). Facial gestures predict pre-
schoolers’ encounter outcomes. Social Science
Information, 16, 715–729. http://dx.doi.
org/10.1177/053901847701600604
 Chapter 6
Gender and Nonverbal
Behavior
Marianne LaFrance and Andrea C. Vial
It is widely believed that women and men are fun-
damentally different from each other. Indeed, the
belief that males and females possess different traits,
abilities, and inclinations pervades all age groups,
all time periods, and all cultures (Kite, Deaux, &
Haines, 2008). Such beliefs, better described as ste-
reotypes, have also been found to be highly resistant
to change (Dodge, Gilroy, & Fenzel, 1995; Heilman,
2001). Two dimensions, communality and agency,
capture a multitude of perceived differences (Bakan,
1966; Kite et al., 2008). Women are consistently
characterized as having a consistent predisposition
to be communal—to care for and attend to the well-
being of others. The typical woman is thought to
be kind, caring, sensitive, empathic, and emotional.
However, men are believed to be primarily agentic
and instrumental. The characteristic male is felt to
be independent, confident, decisive, aggressive, and
strong (Kite et al., 2008).
It is not surprising then that people believe that
women and men show distinctive patterns of non-
verbal behavior. For example, Briton and Hall (1995)
found that people think that women are more non-
verbally expressive and responsive than are men.
Women are also thought to be better at sending and
deciphering nonverbal messages. In contrast, males
are believed to be louder and more interruptive and
to show more restless body movements and dysflu-
ent vocal behaviors, such as inserting filled and
unfilled pauses while speaking. The issue here, as is
the case with stereotypes more generally, has to do
with the validity or accuracy of such beliefs. This
chapter addresses just that question and two related
http://dx.doi.org/10.1037/14669-006
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
ones—namely, what gender dimension best describes
differences that are examined, and if sex differences
are found, to what are they to be attributed?
There is more to gender beliefs than simple
assumptions such as the idea that women express
more positive emotion than men (Shields, 1987).
Not only are men and women believed to have
different repertoires of nonverbal behavior, some
nonverbal behaviors are understood a priori to be
feminine or masculine. Therefore, crying—which
is believed to be something that women do more
than men (Vingerhoets & Scheirs, 2000)—denotes
femininity in the crier (sometimes called effeminacy
if the crier happens to be male). This pregendering
of nonverbal behavior reinforces ideas about who
(men or women) should exhibit which behaviors,
and it impinges on what behaviors men and women
choose to display when motivated to avoid being
perceived as gender deviant. In fact, engaging in the
appropriate nonverbal gender repertoire (and avoid-
ing cross-gender behavior) is part of what some
scholars refer to as “doing gender” well (West &
Zimmerman, 1987).
DECONSTRUCTING GENDER
A substantial body of empirical work has addressed
whether and to what degree women and men differ
in their nonverbal behavior. Nonetheless, many stud-
ies have been primarily descriptive of sex differences
and only explanatory, if at all, after the fact. Although
some researchers have contributed sophisticated and
nuanced examinations of individual variation and
139
LaFrance and Vial
causal factors affecting gendered aspects of nonverbal
behavior, we believe this extensive literature would
benefit from first considering the relatively unex-
plored territory that becomes illuminated by decon-
structing what goes into gender in the first place.
By deconstructing or problematizing gender, we
mean expanding the typical binary category described
by biological sex (i.e., male and female) and focus-
ing instead on the multiple ways that gender can be
understood. For example, one line of work argues
that a key dimension known as sex-role identifica-
tion or psychological gender (Bem, 1977) reflects the
degree to which women and men identify with char-
acteristics that society typically assigns to females and
males. Regarding nonverbal behavior, the question
then converts to whether a person’s (male or female)
degree of identification with feminine and masculine
traits is reflected in his or her nonverbal behavior. An
example of the standard question is whether women
smile more than men. A sex-role identification ques-
tion asks, instead, whether people who
score high in femininity smile more than those who
score low in femininity and/or more than people
who score high in masculinity regardless of their sex.
A constrained sex-role identification (masculine
sex-typed or feminine sex-typed) theoretically moti-
vates men and women, respectively, to self-present
in gender-normative ways and to avoid behavior that
is considered more characteristic of the opposite
sex (Bem, 1977). Androgynous men and women, in
contrast, who identify equally with masculine and
feminine characteristics are theoretically more flex-
ible and less consistent in their nonverbal behavior
because they have larger repertoire from which to
draw, and thus they may freely engage in nonverbal
behavior that is seen as stereotypical of the other
sex if the situation calls for it (M. LaFrance &
Carmen, 1980). Thus, studies exploring the inter-
actions between biological sex and psychological
gender arguably offer an expanded framework for
understanding the relationship between gender and
nonverbal behavior. A sole focus on biological sex
is likely to yield findings that are more informative
of the ways gender norms impinge on the behavior
of men and women than they are informative about
the ways in which men and women choose to adopt
particular nonverbal behaviors.
140
Just as sex-role orientation (psychological
gender) appears to moderate sex differences in
nonverbal behavior, sexual orientation is also likely
to interact with biological and psychological gen-
der and to modulate gender differences in a variety
of nonverbal behaviors. To date, researchers have
shown surprisingly little interest in document-
ing these effects. Numerous common stereotypes
suggest that gay men and lesbian women pos-
sess nonverbal “markers” that distinguish them
from heterosexual men and women. Indeed, some
research has confirmed that gay men readily recog-
nize and utilize these cues to identify one another
(Carroll & Gilroy, 2002). It turns out that straight
people are also able to identify above chance which
men are gay and which are straight from photo-
graphs of their faces (Rule, Ambady, Adams, &
Macrae, 2008). Yet, whether the nonverbal behav-
iors of gay and lesbian individuals differ from those
of heterosexual males and females (and if so, in
what ways) remains essentially unexplored. Addi-
tionally, homophobic attitudes and vigilance on the
part of heterosexual individuals (particularly men)
to behave in hetero-normative ways may influence
the nonverbal behavior of heterosexual individu-
als in ways that would not necessarily be predicted
by biological sex alone. Thus, focusing exclusively
on the nonverbal behaviors of heterosexual men
and women will likely advance our knowledge of
the ways in which heterosexual scripts and hetero-
normative pressures modulate the nonverbal expres-
sions of men and women, rather than increasing our
understanding of the ways gender broadly construed
is manifest in the display of nonverbal cues.
Biological sex, psychological gender, and sexual
orientation not only interact in complex ways to
produce unique patterns of nonverbal behavior
but gender aspects of nonverbal behavior are also
exquisitely sensitive to social context. For example,
M. LaFrance and Carmen (1980) showed how the
gendered nature of a task (instrumental or expres-
sive) interacted with gender and sex-role orientation
to determine vocal nonverbal behavior. The next
section more fully discusses how context affects gen-
der aspects of nonverbal behavior. For the moment,
it is important to note one further way that emerges
when gender is deconstructed. Some contexts are

also gendered not just in the sense that they are
more likely to be occupied by females versus males
but because the behaviors expected there are pre-
sumed to have a more feminine or masculine qual-
ity to them such that whoever temporarily resides
in such spaces will more likely show the expected
behavior regardless of his or her sex.
CONSIDERING CONTEXT
As we will show, the presence of even small changes
in a given setting can magnify or minimize the
expression of nonverbal behavior differences in men
and women, such that gender differences in a given
domain (e.g., smiling) will be larger in certain con-
texts and smaller or even reversed in others. We have
mentioned that both men and women strive to “do
gender” well so as to avoid being perceived as gender-
deviant (e.g., West & Zimmerman, 1987). This sug-
gests that, to the extent that an individual feels that
his or her behavior is being monitored or judged,
he or she is likely to respond by behaving in gender-
normative ways. Alternatively, the absence of obser-
vation by others may lighten the pressure to behave
in accord with gendered expectations. In the first
instance, sex differences may be more manifestly
evident than in the latter instance. The key point is
that certain aspects of the situation (e.g., the presence
of observers) make gender norms and expectations
more salient and, thus, trigger more gender-normative
behavior. This has been shown for the behavior of smil-
ing. As will be discussed in more detail, women tend to
smile more than men, but this difference is even greater
when participants feel that they are being observed by
others (M. LaFrance, Hecht, & Paluck, 2003).
There are two additional contextual factors that
moderate the size of gender differences in nonver-
bal behavior. These have to do with the number of
people who are present as well as how many of each
sex are present (e.g., their sex composition). For
example, research has found that men have a higher
tendency than women to interrupt speakers and
that this difference is especially large in multiperson
compared to two-person encounters (Anderson &
Leaper, 1998). Additionally, the largest sex differ-
ences for interruptions occur in mixed- rather than
same-gender groups or dyads. Thus, these data
Gender and Nonverbal Behavior
show that when it comes to interruptive behavior,
a larger difference favoring males will be found for
mixed-sex groups; in these conditions, men would
be expected to interrupt the most, and women the
least. The difference would be attenuated, and a
much smaller effect would emerge in same-sex
interactions involving only two people.
However, gender composition has the opposite
effect on gaze behavior such that the largest gender
differences emerge in same- rather than mixed-gender
dyads (J. A. Hall, 1984; J. A. Hall & Gunnery, 2013;
Yee, Bailenson, Urbanek, Chang, & Merget, 2007).
Women tend to gaze at their interaction partners
more than men do, and they also tend to be looked
at more than men are. Both factors lead to the
highest levels of partner gazing in female–female
pairs. Similarly, some research suggests that touch
behavior is more acceptable and expected in
female–female dyads compared to male–male dyads
(Derlega, Catanzaro, & Lewis, 2001). Thus, it is
clear that an examination of gender differences in
nonverbal behavior that fails to consider the number
and gender of interaction partners will result in an
incomplete or incorrect picture of how much males
and females differ in their nonverbal behavior.
Finally, situational demands may override gender
norms to affect sex differences in nonverbal cues.
Various tasks and roles often prescribe particular
nonverbal behaviors of whoever is called upon to
engage in the salient activity. The effect of such situ-
ational demands is often to minimize sex differences
in accompanying nonverbal behavior. For example,
although women generally tend to smile more than
men, this difference is greatly reduced when both male
and female participants are engaged in caregiving activ-
ities (M. LaFrance et al., 2003). Likewise, although
women on average orient their bodies more face-on
with their interaction partners (J. A. Hall, 1984),
both men and women orient their bodies toward one
another (Fichten, Tagalakis, Judd, Wright, & Amsel,
1992) when flirtation is the operative dynamic. How-
ever, there are social contexts in which gender polar-
ization is assumed, and in that case, gender differences
in nonverbal behavior are probably accentuated.
Finally, in some contexts, behaviors can become
acceptable that are typically not regarded as such—
for example, women touch other women more than
141
LaFrance and Vial
men touch other men, and both men and women
consider male–male touch to be somewhat atypi-
cal and perhaps inappropriate (Derlega et al., 2001).
However, in a stereotypically masculine context,
such as contact sports, these norms are more lax;
male–male touch is greater, and, though not elimi-
nated, the size of the gender difference is substantially
reduced (Kneidinger, Maple, & Tross, 2001).
GENDER DIFFERENCES IN
NONVERBAL BEHAVIOR
We hope the preceding discussion demonstrates
reasons for regarding gender as a multidimensional
construct as well as the need to attend to context
so as to understand why elaboration is needed
regarding whether men and women differ in their
nonverbal behavior. In this section, we discuss sex
differences in specific behaviors, summarizing reli-
able findings from the existing literature. To do
this, we draw on several meta-analyses, but we also
cite and discuss individual studies that have taken
a more nuanced approach to the question of gender
and nonverbal behavior. Within each section,
we also provide a brief discussion of remaining
questions and avenues for future research.
We cover nonverbal behaviors that have
received the most empirical attention: encoding
and decoding accuracy, smiling, gazing, touching,
interpersonal distance, body orientation, gesture and
posture, and vocal nonverbal behavior (e.g., inter-
rupting). However, we also include sections discuss-
ing some nonverbal behaviors for which the study of
gender differences has been relatively minimal: gait,
blushing, and crying. Finally, we devote a section
to the discussion of gender differences in nonverbal
behavior in the case of heterosexual flirtation, as it
offers a social ritual where gender norms and expec-
tations impinge on the nonverbal behavior of men
and women as they interact with each other.
Encoding Accuracy
Women are more accurate than men in producing
and conveying nonverbal cues—that is, others are
more accurate in reading women’s nonverbal behav-
ior than they are at accurately reading men’s expres-
sive behavior. Presumably, this is the case because
142
women are more expressive in general
than men and/or the cues they send are more
easily read. Overall, this sex difference is substantial
(r = .25; 35 studies in J. A. Hall’s, 1984, meta-
analysis). Depending on how such a difference is
observed, whether researchers measure the spon-
taneous nonverbal behavior of men and women, or
whether men and women are asked to deliberately
convey particular emotions or affective sentiments,
women have an advantage over men in terms of the
ability to produce nonverbal behavior that others
can read as intended (J. A. Hall, 1984). This differ-
ence is stronger for facial expressions (e.g., smiling,
frowning) than for vocal cues (e.g., loudness, pitch;
J. A. Hall, 1984).
Meta-analyses show, however, that gender
differences in encoding accuracy vary in size
depending on other factors so that it is not always
the case that women are clearer senders than men.
For instance, sex differences in sending accuracy
increase with age, such that there are greater dif-
ferences among adult men and women than among
boys and girls (J. A. Hall, 1984), and research indi-
cates that this difference is mostly driven by a defi-
nite decrease in the accuracy of facial encoding by
boys after 4 years of age (Buck, 1977). In addition,
research on prepubescent children suggests that
greater social competence among girls is associated
with increased encoding ability, but the same rela-
tionship with social competence does not hold for
boys (Custrini & Feldman, 1989).
It is worth noting that gender differences in
encoding accuracy and overall expressivity do not
appear to derive from a difference in how much
women and men experience emotion. In other
words, the reasons why women exhibit an advan-
tage over men in the capacity and the tendency to
be more nonverbally expressive are greater than a
simple difference in how and how much men and
women experience emotions. Although cultural
stereotypes abound that women are more emo-
tional than men (Fischer, 1993; M. LaFrance &
Banaji, 1992; Shields, 1987; J. E. Williams & Best,
1990), empirical evidence for this gender differ-
ence in experienced emotion is inconsistent. In
fact, even though women have been observed to be
more spontaneously expressive than men, women

and men report experiencing the same emotions to
the same extent (Barrett, Robin, Pietromonaco, &
Eyssell, 1998; Kring & Gordon, 1998; Robinson,
Johnson, & Shields, 1998), and studies compar-
ing the physiological reactions of men and women
have found that, if anything, men’s physiological
responses to emotion-inducing events tend to be
stronger than women’s (Kring & Gordon, 1998).
Thus, although women and men seem to experience
the same emotions to the same degree, women are
consistently more nonverbally expressive (and more
readable) than are men.
The sex difference in encoding accuracy is likely
due to changes in both male and female behavior:
Men are more likely to suppress overt displays of
emotion where women do not, but in some con-
texts, women actually amplify their expressive
behaviors (M. LaFrance & Banaji, 1992; see also
Hochschild, 1983). In support for male suppression,
there is evidence that high expressivity by males is
often perceived as questionable, and even young
boys anticipate negative repercussions for openly
expressing emotion (Fuchs & Thelen, 1988). As to
whether femininity is associated with greater expres-
sivity, Zuckerman, DeFrank, Spiegel, and Larrance
(1982) found that more accurate encoders of inten-
tional cues (facial and vocal) were those who scored
higher on femininity and lower on masculinity.
Remaining Questions
The finding that encoding accuracy decreases sharply
in males during childhood suggests that socialization
plays a significant role affecting the inclination or
ability to display one’s feelings and intentions nonver-
bally. However, except for Zuckerman et al.’s (1982)
study noted earlier, little research to date has exam-
ined whether sex-role identification also affects encod-
ing accuracy. In short, the degree to which a person
identifies as feminine appears to predict encoding
accuracy in both men and women. Narus and Fischer
(1982) also found that androgynous males were more
emotionally expressive than “masculine” men.
As is the case with several nonverbal commu-
nication modalities, researchers exploring gender
differences in encoding accuracy have focused pri-
marily on heterosexual men and women, whereas
little is known as to whether sexual orientation
Gender and Nonverbal Behavior
might moderate these differences. For example,
the ability to accurately communicate nonverbally
may be higher in gay men compared to hetero-
sexual men and perhaps lower in lesbians than
heterosexual women.
Additionally, although observers can more accu-
rately identify emotional states from viewing female
than male faces (Wagner, MacDonald, & Manstead,
1986), this pattern does not generalize to all emo-
tions. For example, there is evidence that women
are more likely than men to suppress the expres-
sion of anger, presumably because anger is seen
to be incompatible with femininity or prescriptive
gender stereotypes (Heilman, 2001; Lerner, 1985;
Rudman, 1998). This suggests a possible interaction
between sex-role identification and situation on the
nonverbal expressions of men and women, such that
depending on the situation (e.g., a baby shower vs. a
competitive encounter), sex-typed (but not androg-
ynous) men and women would be expected to differ
the most in their nonverbal behavior.
Decoding Accuracy
Women are also more accurate than men in
correctly deciphering the nonverbal behaviors of
others, regardless of the gender of the target person
(Baron-Cohen, Wheelwright, Hill, Raste, & Plumb,
2001; Chan, Rogers, Parisotto, & Biesanz, 2011;
J. A. Hall, 1978, 1984; J. A. Hall & Matsumoto,
2004; Letzring, 2010; Rosenthal, Hall, DiMatteo,
Rogers, & Archer, 1979; Sasson et al., 2010;
Thomas & Fletcher, 2003; Vogt & Colvin, 2003;
but see Ickes, Gesn, & Graham, 2000, for contra-
dicting results). This sex difference has been amply
demonstrated in children and adolescents as well as
adults (Boyatzis, Chazan, & Ting, 1993; McClure,
2000; Székely et al., 2011). In fact, both meta-­
analyses (J. A. Hall, 1978, 1984; McClure, 2000)
and individual studies (L. M. Williams et al., 2009)
have found that although age does not significantly
moderate the effect of gender on decoding accuracy,
the size of the gender difference tends to be some-
what larger among adults (ranging between
r = .20 and r = .25 in J. A. Hall’s, 1984, meta-­
analysis) than among children and adolescents
(r = .18; 60 studies in McClure’s, 2000, meta-­
analysis). Cross-cultural research also indicates
143
LaFrance and Vial
that the country of the participant does not moderate
gender effect size for decoding accuracy (J. A. Hall,
1978; Izard, 1971; Merten, 2005; Scherer, Banse, &
Wallbott, 2001). Women’s higher decoding accuracy
also translates into their advantage at recalling other
people’s nonverbal behaviors, such as gazing, smil-
ing, and self-touching (J. A. Hall, Murphy, & Schmid
Mast, 2006). Although men tend to be generally less
successful than women at accurately decoding the
nonverbal cues of others (J. A. Hall, 1978, 1984), the
sexes part company even more so when the nonver-
bal cues in question are ambiguous (Farris, Treat,
Viken, & McFall, 2008). For example, in studies
assessing the perception of nonverbal behaviors that
may signal sexual interest, the data indicate that
males tend to perceive significantly more flirtatious-
ness, promiscuousness, and seductiveness than
female perceivers (ranging from r = .09 to r = .20
in the meta-analysis by B. H. La France, Henningsen,
Oates, & Shaw, 2009). Some have suggested that
these results are consistent with error management
theory, whereby from an evolutionary perspective, it
would be most advantageous for men to overestimate
women’s sexual interest because a false-negative
(i.e., missing cues of sexual interest) would lead to
a missed mating opportunity (B. H. La France et al.,
2009). This presumably causes men to have a lower
decisional threshold than women for labeling ambig-
uous behaviors as sexual (Haselton & Nettle, 2006).
The problem with this explanation, however, is
that men “see” more sexual interest in both males
and females (Shotland & Craig, 1988). Addition-
ally, Farris et al. (2008) found that male participants
mistook friendliness for flirtatiousness just as often
as they misread flirtatiousness as friendliness, pro-
viding evidence for men’s lower ability to accurately
read the nonverbal cues of others.
Finally, some research suggests that, regardless
of the gender of the perceiver, the accuracy with
which specific emotions are decoded may differ
greatly depending on the gender of the target and
the specific emotions in question. For example,
Plant, Hyde, Keltner, and Devine (2000) asked par-
ticipants to interpret photographs of adults show-
ing ambiguous anger/sadness expressions, and they
found that female targets were rated as sadder and
that male targets were rated as angrier, consistent
144
with gender stereotypes. Female targets’ poses were
rated as a mixture of anger and sadness even when
unambiguous expressions were presented (Plant
et al., 2000). Similarly, Hess, Adams, Grammer, and
Kleck (2009) found that androgynous faces were
more consistently and more quickly recognized
as male versus female when they displayed anger
versus happiness, respectively, which suggests that
gender stereotypes of emotion may greatly influence
observers’ accuracy when decoding ambiguous facial
expressions.
Remaining Questions
As is the case for encoding accuracy, sex-role iden-
tification, namely, the degree to which a person
self-identifies as feminine, appears to be a better
predictor of decoding accuracy than biological
sex—but, again, little research has seriously tested
this idea. One study is illustrative, however. Trom-
msdorff and John (1992) examined the communal
orientation and femininity of relationship partners
as they decoded each other’s emotions. They found
that decoding was better to the degree to which per-
ceivers had a feminine gender-role orientation.
Likewise, as is the case with other nonverbal
behaviors, researchers exploring gender differences
in decoding accuracy have focused primarily on het-
erosexual men and women, whereas little is known as
to whether sexual orientation might moderate these
differences. For example, the ability to accurately
communicate nonverbally may be higher in gay men
compared to heterosexual men. Similarly, some have
proposed that perceptual accuracy provides gay men
and lesbian women with self-protection from homo-
phobic violence, as it increases the likelihood that they
will identify other gay/lesbian individuals (Carroll &
Gilroy, 2002). Other research has shown that gays and
lesbians have higher accuracy than heterosexual men
and women when judging sexual orientation based on
nonverbal behavior and facial expression (Ambady,
Hallahan, & Conner, 1999). However, it is not known
whether this higher accuracy generalizes beyond the
detection of sexual orientation—it is possible that gay
men might have higher decoding accuracy than het-
erosexual men in general (and perhaps lesbian women
might have an advantage over heterosexual women as
well). These associations need to be empirically tested.

Although women’s advantage in accurately read-
ing nonverbal cues appears to be general, J. A. Hall
and Gunnery (2013) have questioned whether this
advantage holds across all attributions. Most stud-
ies on decoding accuracy typically ask participants
to draw inferences about a target’s emotional state,
which is a domain in which women are socialized to
have more interest than men and likely to develop
a higher level of expertise (Brody, 1999). Thus,
decoding accuracy is one area where contextualizing
gender might prove fruitful. For example, research
is needed to evaluate decoding accuracy in domains
in which men’s decoding accuracy might not differ
from women’s or, perhaps, might even be superior,
such as accurately detecting the intentions of a rival
from his nonverbal behavior during competitive
or combative interactions. Some evidence for such
an effect comes from research showing that men’s
recall accuracy for nonverbal behavior increases
in competitive versus noncompetitive contexts
(J. A. Hall & Schmid Mast, 2008). Interestingly,
research on flirting behavior indicates that both
males and females are better at decoding the sexual
interest of men than women (Place, Todd, Penke, &
Asendorpf, 2009). Thus, the contextual cues sur-
rounding nonverbal behaviors (and their distinct
self-relevance for men and women) likely impact the
size of the gender difference in decoding accuracy.
Smiling
Considerable research has examined gender differ-
ences in smiling in part inspired by the speculation
that women’s greater smiling reflects their low power
relative to men (Henley, 1977). For reviews of that
literature, see J. A. Hall, Carney, and Murphy (2002)
as well as M. LaFrance et al. (2003). Women are
found to generally smile more than men (r = .20 in
M. LaFrance et al.’s, 2003, meta-analysis of 418 stud-
ies). However, this effect increased or decreased in
response to a number of factors. For example, age
plays a key role, such that gender differences in smil-
ing tend to be absent among young children, largest
among adolescents, and smaller, though still present,
in adults, and all but disappearing after late middle
age. The meta-analysis by M. LaFrance et al. (2003)
found the largest gender effect (r = .28) among ado-
lescents who were 13–17 years of age; among older
Gender and Nonverbal Behavior
participants, the magnitude of the gender
difference decreased steadily, and it was lowest
(r = .06) among older adults who were 65 years
of age or older. Similarly, J. A. Hall’s (1984)
meta-analysis as well as more recent studies
(e.g., DeSantis, Mohan, & Steinhorst, 2005;
Dodd, Russell, & Jenkins, 1999; Else-Quest, Hyde,
Goldsmith, & Van Hulle, 2006; Wondergem &
Friedlmeier, 2012) have found no gender differences
in the social smiling of young children.
The social context in which smiling occurs
has also been found to substantially affect the size
of the sex difference (M. LaFrance et al., 2003).
Women smile more than men when the situation
involves social engagement, and this is particularly
evident when the context is marked by social ten-
sion. Accordingly, women’s tendency to smile more
than men is higher when they are being observed by
others, when they are instructed to get acquainted,
when they engage in self-disclosure, and when they
experience embarrassment (M. LaFrance et al.,
2003). These situations make communality more
salient, which heightens the expectation for more
feminine behavior. Of note, the size of the difference
is smaller or absent in situations where males and
females are engaged in the same task or occupy the
same social role.
Psychological gender has also been found to
affect the size of the sex difference in smiling. For
example, M. LaFrance and Carmen (1980) catego-
rized male and female participants according to
their sex-role orientation (i.e., feminine sex-typed,
masculine sex-typed, and androgynous; Bem, 1977),
and they observed their nonverbal behavior. In this
study, as expected, a main effect of biological gender
emerged, such that women smiled more than men.
However, androgynous men and women did not
differ in their smiling; rather, the effect was driven
by feminine women and masculine men, in which
the former smiled significantly more than the latter
(M. LaFrance & Carmen, 1980).
Although for many specific nonverbal behaviors
there is a dearth of cross-cultural examination of
gender differences, this is not the case with smiling.
Studies have examined the sex differences in smiling
across nations and within countries (M. LaFrance
et al., 2003). Moreover, the size of this difference varies
145
LaFrance and Vial
considerably, with the largest difference reported
with Canadian samples (r = .30), and the smallest
difference emerging in British nationals (r = .07).
Ethnicity also plays a role in U.S. samples, with Cau-
casians exhibiting larger differences (r = .22) than
African Americans (r = .13; M. LaFrance et al., 2003).
Remaining Questions
Whether sexual orientation might moderate the
effect of biological sex on smiling in a similar
way as sex-role orientation has shown to do
(M. LaFrance & Carmen, 1980) remains an empiri-
cal question. It is possible that gay men might smile
significantly more than heterosexual men, whereas
lesbian women might smile less than heterosexual
women. There certainly exist stereotypes about this,
yet to date no research has examined this proposition
empirically.
Gazing and Eye Contact
In infancy, girls gaze at social stimuli more than boys
(Connellan, Baron-Cohen, Wheelwright, Batki, &
Ahluwalia, 2000; Lutchmaya & Baron-Cohen, 2002),
a pattern that is evident as well with older children
and adults (J. A. Hall, 1984). The consistent finding
is that adult women gaze at their interaction part-
ners more than men do. In fact, the sex difference
in gazing is even larger among adults (r = .32 in
adult men and women; J. A. Hall, 1984) than it is
among infants and children, (r = .20 and r = .19,
respectively). It is worth noting that these patterns
are most pronounced when the measure of gazing
involves duration rather than the number of indi-
vidual looks at another person (J. A. Hall, 1984).
In fact, some investigators have reported the oppo-
site effect when the measure involves frequency of
glances. In short, men’s gaze patterns are such that
they look more frequently but for brief durations at
their interaction partners than women do (see, e.g.,
Bente, Donaghy, & Suwelack, 1998).
Gender differences in gazing are also sensitive
to sex composition of the interacting pair, such
that the largest sex difference in gazing favoring
females is observed when the comparison entails
contrasting female–female with male–male dyads
(r = .45; J. A. Hall, 1984; Yee et al., 2007). J. A.
Hall and Gunnery (2013) have suggested that this
146
is due to the contribution of both interactants. In
addition to women gazing at their dyadic partners
more than men, research shows that women are
gazed at by others more than men are (r = .31 based
on six studies in J. A. Hall’s, 1984, meta-analysis).
Thus, because women tend to gaze more, and
because individuals tend to gaze at women more,
female–female dyads exhibit more eye gazing
behavior than cross-sex or male–male dyads.
Although J. A. Hall and Gunnery (2013) have
suggested that men are somewhat uncomfortable
with eye-to-eye contact, they sometimes appear
very comfortable staring down others when they are
talking but look at others little when they are listen-
ing. Dovidio, Ellyson, Keating, Heltman, and Brown
(1988) have shown, for example, that men exhibit
a pattern called high-visual dominance: They gaze
more at their interaction partners while speaking
than while listening. This is especially true of men
in high-power positions. Women, in contrast, tend
to do the opposite: They gaze at their partners more
while they are listening than when they are speak-
ing. Again, we see the effect of context. Both sexes
sometimes show visual dominance when they
are in high-power roles (Ellyson, Dovidio,
& Brown, 1992).
As with smiling, femininity and masculinity
have been shown to moderate gender differences in
gazing. In the study described earlier, M. LaFrance
and Carmen (1980) looked at gazing behavior and
found that androgynous men and women did not
differ significantly from each other in gaze extent,
but androgynous males gazed more than masculine
males, and androgynous females gazed less than
feminine females.
Remaining Questions
It has been theorized that eye gaze is used in the
gay and lesbian community for purposes of iden-
tity recognition (Nicholas, 2004), which attests
to the possibility that gaze may add a function for
gay men and lesbians that is not utilized by het-
erosexual men and women. Indeed, one study has
demonstrated that eye-contact plays an important
role for lesbians and gay men in identifying one
another (Carroll & Gilroy, 2002). However, to date
no research has systematically evaluated whether

sexual orientation moderates gender differences in
duration and/or frequency of gazing behavior more
generally.
Finally, the degree of cross-cultural modera-
tion of gender differences in gazing behavior is
unknown. In East Asian cultures, for example, eye
contact is often construed as impolite, whereas
averted eye gaze is seen as respectful (Knapp & Hall,
2010). Whether stringent norms about gazing lead
to smaller (or even reversed) gender differences in
gazing behavior in Eastern versus Western cultures
is an empirical question.
Touch
Who touches whom, in what ways, how much,
and with what repercussions has been the subject
of empirical scrutiny for more than 40 years start-
ing with Henley’s (1977) proposal that high-status
people have greater license to touch a low-status
person than the reverse. She reported that men touch
women more than women initiate touch with men
and saw this asymmetry as reflecting status differ-
ences. J. A. Hall’s analyses have led her to conclude,
however, that the reverse pattern is more reliable.
Compared to men, J. A. Hall (1984, 2011) has
reported that women generally touch others more
than men do.
Subsequent efforts have attempted to specify
the factors that might explain these discrepant pat-
terns. Major, Schmidlin, and Williams (1990) posed
that several factors could potentially account for
the results. For example, age matters. For dyads
younger than 30 years of age, male-initiated touch
dominates, but the opposite (more female-initiated
touch) is observed for dyads older than 30 years
of age (J. A. Hall & Veccia, 1990; Willis & Briggs,
1992; Willis & Dodds, 1998).
The nature of the relationship also counts. Males
appear to initiate touch more than females when the
relationship is a nonintimate one and the setting is
public. Among married couples in contrast, wives
touch husbands more than the other way around
(Smith, Vogel, Madon, & Edwards, 2011). Consis-
tent with this, unmarried men are more comfortable
with touch than unmarried women, whereas the
reverse is true for married men and women (Hanzal,
Segrin, & Dorros, 2008).
Gender and Nonverbal Behavior
The sex composition of the relevant dyad also
impacts which person touches which other person.
Like gazing, male–male and female–female dyads
differ most in interpersonal touching, such that
female–female dyads exhibit the highest levels of
interpersonal touch and male–male dyads the lowest
(Kneidinger et al., 2001; Montemayor & Flannery,
1989; Stier & Hall, 1984). Women report feeling
more comfortable than men with same-sex touch
(Andersen & Leibowitz, 1978; Roese, Olson,
Borenstein, Martin, & Shores, 1992) but less
comfortable with touch from strangers (Heslin,
Nguyen, & Nguyen, 1983). Clearly, the meaning
of touch differs for males and females when the
encounter is a heterosexual one. The more females
perceive touch as sexual, the less they perceive it as
warm and friendly, whereas the more males perceive
touch as sexual, the more they perceive it as warm
and pleasant (Nguyen, Heslin, & Nguyen, 1975).
In a set of studies with college students, Roese
et al. (1992) examined attitudes toward sexual
minorities and same-sex touch, and they demon-
strated that self-reported homophobia and discom-
fort with same-sex touch were correlated among
male and female students. Moreover, the researchers
covertly observed and recorded frequency of touch
in same-sex dyads of students interacting in a cafete-
ria, and they later approached the dyads and asked
them to complete a scale on homophobic attitudes.
Homophobic attitudes were negatively correlated
with frequency of same-sex touch for all partici-
pants. Male participants, in particular, had stronger
homophobic attitudes than women, and they exhib-
ited lower frequencies of touch (Roese et al., 1992).
In another study, participants were asked to evaluate
touching versus nontouching line drawings showing
same- and cross-sex dyads, and the effect of partici-
pant sexual orientation on evaluations was examined
(Derlega et al., 2001). Heterosexual participants
(but not gay, lesbian, or bisexual men and women)
rated touch in male–male dyads as less appropriate
than touch among cross-sex or female–female dyads.
Heterosexual participants also tended to infer higher
levels of sexual involvement in touching versus non-
touching drawings depicting cross-sex or male–male
pairs compared to nonheterosexual participants
(but no effect emerged for female–female pairs).
147
LaFrance and Vial
Taken together, these findings suggest high
vigilance on the part of heterosexual men and
women to same-sex touch, particularly with regard
to male–male touch, likely stemming from nega-
tive attitudes toward homosexuality and sexual
minorities (Roese et al., 1992) as well as from an
increased tendency to perceive such touch as sex-
ual (Derlega et al., 2001). There is, however, the
noted exception to proscriptions against male–male
touch, specifically in settings involving competitive
sports. There, male–male touch is less inhibited and
is more likely to emerge compared to other settings
(although even in this context, male–male touch is
less frequent than female–female touch; Kneidinger
et al., 2001). One reason for this might be that the
unambiguous nature of the sports setting might
counter the tendency to perceive male–male touch
as sexual (Derlega et al., 2001), thus deeming it
more acceptable to heterosexual perceivers.
Gender differences in touch also depend on the
type and quality of touch examined. Research has
shown that men tend to touch more intimately and
for longer durations than women (McCormick &
Jones, 1989). Also, men touch women with the
hand more than women touch men with the hand,
but for nonhand touches, women touch more than
men (DiBiase & Gunnoe, 2004; J. A. Hall & Vec-
cia, 1990). Interestingly, type of touch seems to
interact with relationship status, such that effects
emerge for men and women who are not in a rela-
tionship; but for married couples, women touch
men more than the other way around, regardless
of the type of touch examined (i.e., expressive and
supportive touches; hand and nonhand touches;
Smith et al., 2011).
Research has also identified differences in the
accuracy or effectiveness with which men and
women use touch to communicate with others.
Among unacquainted participants, regardless of
the gender of their interaction partner, women are
more likely than men to successfully communicate
sympathy using touch, whereas men are more likely
than women to successfully use touch to commu-
nicate anger (Hertenstein & Keltner, 2011). This
research also found that happiness tended to be suc-
cessfully conveyed by touch in female–female dyads
only (Hertenstein & Keltner, 2011).
148
In addition to interpersonal touch, research has
examined gender differences in self-touch, which
some regard as indicating self-consciousness.
Compared to men, women touch themselves more
(r = .22; J. A. Hall, 1984; McCormick & Jones,
1989). Gender differences in self-touch have been
examined particularly in the context of cross-sex
flirting interactions. This research has typically
found that women tend to self-touch during the ini-
tial stages of flirting, before contact is initiated, more
so than men (Moore, 1995; Scheflen, 1965).
Remaining Questions
Cross-cultural research on gender differences in
touch has yet to receive the attention it deserves.
One study focusing on men and women’s attitudes
toward same-sex touch found that women were
more comfortable with this kind of touch in the
United States as well as in Malaysia, Spain, and
Chile (Willis & Rawdon, 1994). More than 50 years
ago, anthropologist Edward Hall (1959) proposed
that cultures varied in the degree to which they
were oriented toward physical contact or not. For
example, he noted that Southern European and
Middle Eastern peoples preferred close interpersonal
distances and more touching than people living in
more northern climes. Nonetheless, no data cur-
rently exist on the degree to which gender might
interact with these cultural patterns.
Many of the moderators that have shown to be
important for other nonverbal behaviors (e.g.,
smiling) have not been systematically explored with
respect to touch—such as the presence of observers.
Also, whether sex-role orientation might interact
with biological sex to determine touch behavior
is not known. It could be the case that qualita-
tive differences in the meaning of touch would
emerge, such that masculine men might use touch
most successfully and most often to communicate
anger or dominance (as has been shown recently;
Hertenstein & Keltner, 2011), whereas androgynous
men might use touch to express a wider variety of
emotions—such as sympathy.
Similarly, whereas some research has explored
how attitudes toward sexual minorities influence
attitudes toward same-sex touch, an understand-
ing of how sexual orientation interacts with actual

touch behavior is lacking. Because the existing
research suggests a high degree of vigilance on the
part of heterosexual men with respect to same-sex
touch, it might be expected that same-sex dyads
among gay men would touch significantly more
than male–male dyads among heterosexual men. For
women, it is less clear that sexual orientation would
interact with gender to influence touch behavior,
but this is an empirical question.
Interpersonal Distance, Body Orientation,
Gesture, and Posture
Compared to women, men tend to adopt larger
interpersonal distances with interaction partners
(r = .27; 17 studies in J. A. Hall’s, 1984, meta-
analysis). Not surprisingly, interpersonal distance
is at its maximum in male–male dyads compared to
female–female or mixed sex dyads. This has been
shown cross-culturally both in the United States and
Turkey (Ozdemir, 2008). Looked at from another
angle, J. A. Hall’s (1984) meta-analysis reported that
people tend to set larger interpersonal distances
when interacting with men than with women
(r = .43; nine studies). This was true of both adult
participants as well as children, although relatively
few studies have examined interpersonal distance in
children (J. A. Hall & Gunnery, 2013).
With respect to body orientation, female adults
as well as children tend to orient more directly
toward their interaction partners than males,
although this effect is smaller than other gender dif-
ferences in nonverbal communication (r = .15 and
r = .12, respectively; J. A. Hall, 1984). Moreover, in
the specific context of heterosexual flirtation, this
gender difference in body orientation disappears,
as both men and women orient their bodies toward
the person of interest (Fichten et al., 1992). Some
research has also found gender differences in body
synchrony or posture mirroring—the spontaneous
postural matching of interaction partners, which is
believed to convey interpersonal rapport (Scheflen,
1964). M. LaFrance and Ickes (1981) examined the
interaction between gender and sex-role orientation
on body mirroring. They found that in same-sex
dyads, feminine (i.e., sex-typed) females engaged
in significantly more body mirroring than same-
sex dyads of masculine males. However, among
Gender and Nonverbal Behavior
androgynous pairs, the effect was reversed with
male–male dyads showing more body mirroring
than female–female dyads.
With respect to small body movements, results
indicate that men tend to be reliably more fidgety
and restless than women (r = .34). However,
women engage in more head nodding when inter-
acting with others, a behavior sometimes referred
to as a back-channel response, such as uttering
“hmm” in reaction to a speaker’s statement.
Back-channel responses are used to convey that
one is actively listening to an interaction partner.
Women have also been found to use hand move-
ments and gestures while speaking more so than
men (r = .28; J. A. Hall, 1984).
Like body orientation, women have been found
to lean forward toward their interaction partner
more than men (r = .16; J. A. Hall, 1984; Helweg-
Larsen, Cunningham, Carrico, & Pergram, 2004).
Posture has also been described on an expansive-
compacted dimension, and here men tend to
adopt more relaxed postures (i.e., asymmetrically
arranged arms and legs; r = .33) as well as more
expansive body postures (i.e., limbs reaching far-
ther away from the body; r = .46; J. A. Hall, 1984),
whereas women, in contrast, typically maintain
more restricted postures with legs close together
and arms close to the torso. For example, obser-
vations of seated participants on an urban metro
revealed that men more often sat in an open pos-
ture with their legs apart and their arms away from
their sides while women sat in closed postures,
that is with upper legs against each other and
arms against the trunk (Vrugt & Luyerink, 2000).
This sex difference in posture expansiveness has
been linked with differences in dominance and
social power. Body openness in adults is positively
related to dominance (J. A. Hall, Coats, & LeBeau,
2005), and research on children reveals that one of
the key differences between dominant and
submissive individuals is body expansiveness
(Weisfeld & Beresford, 1982). With regard to
adults, research has also found that expansive
postures cause power-related feelings (Tiedens &
Fragale, 2003) and behavior as well as changes in
hormone levels normally associated with high rank
(Carney, Cuddy, & Yap, 2010).
149
LaFrance and Vial
Remaining Questions
As is the case with same-sex touch behavior, an
understanding of how sexual orientation moder-
ates gender differences in interpersonal distance
and orientation as well body movement and posture
is sorely lacking. Homophobic attitudes or greater
vigilance to potential threat by heterosexual men
may play a role in men’s preference for greater
interpersonal distance in same-sex dyads compared
to women. Future research might examine inter-
personal distance by varying dyadic composition
and sexual orientation. It is plausible that inter-
personal distance might be significantly reduced in
male–male dyads of gay compared to heterosexual
men. Similarly, research looking at hand gestur-
ing during speech and expressive body movements
might benefit from evaluating how sexual orienta-
tion might moderate gender differences. As with
smiling, there are stereotypes expecting gay men to
engage in more sociability, often expressed in more
hand movements while talking compared to hetero-
sexual men (Webbink, 1981), but whether this is
truly the case remains to be examined empirically.
As the findings by M. LaFrance and Ickes (1981)
suggest, femininity and masculinity might be bet-
ter predictors of some of the gender differences
discussed in this section than biological sex. For
example, maintaining a compressed, upright body
posture may be part of enacting femininity scripts,
and thus androgynous women might be expected to
display less restricted and more relaxed body pos-
tures compared to feminine-typed women. Similarly,
many of the behaviors shown to be more common
in women than men (such as maintaining closer
interpersonal distance, orienting the body toward
an interaction partner, head nodding, and forward
leaning) are behaviors denoting social sensibility—a
stereotypically communal characteristic. Thus, indi-
viduals with higher femininity scores might exhibit
these behaviors to a greater extent regardless of their
gender, and androgynous men and women might fall
somewhere in between sex-typed men and women.
More research is needed to address these possibilities.
Vocal Nonverbal Behavior
In general, men speak louder than women and
with more speech disturbances, such as filled
150
pauses—“ah,” “um,” or incomplete sentences
(J. A. Hall, 1984; Schmid Mast & Sczesny, 2010).
Research has also shown that men tend to be more
talkative than women (Leaper & Ayres, 2007),
although recent findings suggest that this tendency
might be moderated by men’s situational sense
of power (Brescoll, 2011). In other words, men’s
tendency to surpass women in talking time (i.e.,
volubility) tends to disappear when men are experi-
mentally induced into a low-power mindset.
As noted, women generally exceed men in the
use of back channels while listening to others
(J. A. Hall, 1984; Leaper & Robnett, 2011). In con-
trast, men are more likely than women to interrupt
others’ speech with the goal to take over the conver-
sation, sometimes referred to as intrusive interrup-
tions, rather than merely speaking at the same time as
another person (Anderson & Leaper, 1998). Intru-
sive interruptions are more likely to discourage the
original speaker from continuing. Thus, a relatively
small gender difference when considering inter-
ruptions in general (r = .08) becomes substantially
larger when considering intrusive interruptions spe-
cifically (r = .16; Anderson & Leaper, 1998). Simi-
lar to this, M. LaFrance and Carmen (1980) did not
find a significant difference in interruptions by men
and women when all types of interruptions were
combined into a single index. However, when the
researchers looked at interruptive statements (asser-
tive) and interruptive questions (responsive) sepa-
rately, clear gender differences emerged, with males
making significantly more interruptive statements
and women inserting more interruptive questions.
The number of interaction partners has been
shown to moderate the size of the gender difference
in the tendency to interrupt, such that men make
more intrusive interruptions than women, especially
in group settings (i.e., more than two interacting
partners; r = .30; Anderson & Leaper, 1998). The
difference is almost negligible for dyads (r = .06).
The same trend emerged when considering any kind
of interruption, though the difference tended to be
smaller (r = .13 in groups; r = .03 in dyads). In
Anderson and Leaper’s (1998) meta-analysis, dyadic
composition moderated the tendency for men to
interrupt more than women, such that the larg-
est difference emerged for intrusive interruptions

occurring in mixed-gender groups or dyads
(r = .30 in same-sex and r = .06 in mixed-sex
groups for intrusive interruptions, and r = .01 in
same-sex and r = .08 in mixed-sex groups or dyads
for overall interruptions; Anderson & Leaper, 1998).
Degree of familiarity between the interacting
partners has been shown to moderate the likelihood
that a gender difference in interruptive behavior will
emerge. For intrusive interruptions, Anderson and
Leaper (1998) found that gender effects were more
likely when conversing with strangers (r = .19)
rather than familiar persons (r = .09). Due to a
limited number of studies examining intrusive inter-
ruptions among familiar persons, Anderson and
Leaper were not able to examine different types of
relationships (e.g., friends vs. romantic partners).
For overall interruptions, however, they were able to
compare friends, romantic partners, and other types
of close relationships. They found the largest (albeit,
relatively small) gender difference among romantic
partners (r = .10). Interestingly, among friends, the
difference between men and women’s overall inter-
ruptive behavior had the opposite direction, with
women interrupting their friends more than men
did but only very slightly so (r = .07). Among other
close relationship partners as well as among strang-
ers, the pattern of gender differences for overall
interruptions mirrored that for intrusive interrup-
tions, although they were smaller in magnitude
(r = .06 and r = .08, respectively).
Finally, M. LaFrance and Carmen (1980) exam-
ined the interaction between psychological gender
orientation and biological gender on two kinds of
vocal behavior. Specifically, they looked at inter-
ruptions and filled pauses in both task-focused and
emotionally expressive contexts. In general, mascu-
line males emitted significantly more filled pauses
than androgynous males and feminine females.
Interruptive statements were significantly more
common among androgynous men and women com-
pared to sex-typed men and women (i.e., masculine
males and feminine females). Importantly, context
moderated these interactions, such that masculine
males and androgynous females emitted more inter-
ruptive statements in the task condition than in the
emotive condition, whereas sex-typed females made
few interruptive statements in both contexts, and
Gender and Nonverbal Behavior
androgynous males maintained a relatively high
level of interrupting in both contexts. Thus, this
research illuminates how biological gender, psycho-
logical gender, and gendered aspects of the specific
context interact to determine nonverbal behavior in
ways that would not be evident if all three factors
had been examined separately.
Remaining Questions
Whether the gender differences in vocal nonverbal
behavior summarized here would emerge cross-
culturally is for future research to determine. Polite-
ness rules, the importance assigned to hierarchy or
verticality within a specific culture, and level of gen-
der equality all may exacerbate or ameliorate men’s
tendency to talk more and interrupt more than
women. Similarly, whether these differences remain
somewhat stable throughout the life span or fluctu-
ate with age remains to be examined.
The impact of power and status on gender dif-
ferences in nonverbal behavior also needs to be
evaluated more thoroughly. To date, there is limited
experimental evidence that power increases men’s but
not women’s talking time (Brescoll, 2011). Whether
power and status may moderate other gender differ-
ences in vocal nonverbal behavior—such as the use of
back-channel responses, interruptive statements and
questions, and filled pauses, and so forth—is a more
open question. Men might curtail their interruptive
behavior and speech time when interacting with a
woman who holds greater power (i.e., she is per-
ceived to have more expertise in a specific domain).
For example, male patients interacting with female
physicians may not engage in the same type of vocal
nonverbal behavior that is generally found when
looking at other types of interaction contexts.
Other Nonverbal Domains
There are three remaining nonverbal domains that, for
whatever reason, have not been as central to the study
of nonverbal communication in general and gender
aspects in particular as the domains we have covered
thus far. The three domains are gait or global move-
ment style, blushing, and crying. Because few studies
have been designed to look at the ways these behav-
iors may differ in men and women, no meta-analyses
are available for us to draw from at this point.
151
LaFrance and Vial
Gait or walking style, however, differs in men
and women (Kerrigan, Todd, & Croce, 1998; Nigg,
Fisher, & Ronsky, 1994; Troje, 2002), and perceivers
can identify the gender of a walker with minimal cues
with above chance accuracy (Brooks et al., 2008; Pol-
lick, Kay, Heim, & Stringer, 2005). Swaying hips are
perceived to be more likely characteristic of walking
by females, whereas swaggering shoulders are per-
ceived to be more likely displayed by men, and it is
these perceptions that aid in the inference of the gen-
der of a walking target (Johnson & Tassinary, 2005).
Additionally, recent research suggests that the walk-
ing styles of gay men and lesbian women differ from
that of heterosexuals in degree of shoulder swagger
(a male-typical behavior) and hip sway (a female-
typical behavior; see Johnson, Gill, Reichman, &
Tassinary, 2007). Beyond replicating these basic
effects, more research is needed in this area to under-
stand how individual characteristics such as age and
race may moderate gender differences in walking
style. Moreover, in keeping with our theme of decon-
structing gender, we recommend that future research
examine whether gender differences in walking style
are moderated by psychological gender. For example,
do the walking styles of androgynous men and
androgynous women differ as much as those of
sex-typed men and sex-typed women? Likewise, do
different contexts inhibit or magnify the differences
between males and females in walking style?
Next, we turn our attention to research on
blushing. The blush is most commonly caused by
unwanted social attention (Leary, Britt, Cutlip, &
Templeton, 1992), and it generally emerges as a reac-
tion to situations that elicit “self-conscious” emo-
tions, such as embarrassment, guilt, and shame.
Women are thought to be more susceptible to blush-
ing than men, but experimental research has not
been consistent on this count (Drummond, 2013).
For example, Shearn, Spellman, Straley, Meirick,
and Stryker (1999) found no significant difference
in the blushing reactions of men and women in an
experiment in which they watched video clips of their
friends or strangers or themselves singing (the last sit-
uation frequently used to trigger blushing responses).
However, some self-report studies have found that
women report blushing more than men report doing
so (Bögels, Alberts, & de Jong, 1996; Neto, 1996)
152
There is also some evidence that females show
more “coyness blushing” in a courting context (von
Hooff, 2013), but again empirical verification has
been slight. Future research examining actual blush-
ing reactions as well as self-reported blushing propen-
sity across different life stages might reveal interesting
findings. For example, as is the case with smiling, it
is possible that gender differences in blushing might
vary with age, being slim in childhood, largest in ado-
lescence, and relatively less pronounced in adulthood.
In closing, we turn to crying, where research on
gender differences has been relatively more extensive
than that for either gait or blushing. To begin with,
there is abundant data attesting to the ubiquitous
stereotype of the tearful woman versus the stoic man
(Vingerhoets, 2013). Furthermore, studies focusing
on the relationship between biological gender and
actual weeping show that women cry more frequently
than men do (for reviews, see Bekker & Vingerhoets,
1999, 2001; Vingerhoets & Scheirs, 2000). This sex
difference is consistent across several cultures where
it has been studied, even though the magnitude of
the difference varies with the particular culture being
observed (Becht, Poortinga, & Vingerhoets, 2001).
With respect to babies and young children, how-
ever, the results are mixed as to whether a sex dif-
ference exists in the first years of life (Vingerhoets,
2013). In fact, some data suggest that boys show a
higher frequency of crying than girls and that it is
not until 8 years of age that girls show the pattern
of more crying than boys. Just why this divergence
happens has been the subject of considerable debate,
with some contending that boys are discouraged
from crying after childhood, and others arguing that
girls develop tearful crying because of its benefits
(Vingerhoets, 2013).
Gender, Nonverbal Behavior,
and Flirtation
Heterosexual courtship interactions in Western cul-
ture involve a complex set of nonverbal behaviors by
both sexes that are tightly and relationally scripted.
Both sexes use nonverbal cues to signal sexual inter-
est to potential romantic or sexual partners, and
both engage in decoding practices to try to read the
nonverbal cues that potentially signal the interest
of another person. The whole nonverbal repertoire

is used in flirtation–gaze, smile facial expression,
interpersonal distance, body orientation and pos-
ture, gestures, and touch. Depending on timing and
sequence, nonverbal cues can communicate aware-
ness, openness, and desire (or their opposites). As
such, the critical dimension in flirtation situations
is not so much how much a nonverbal behavior is
displayed but that it is displayed and shown in close
temporal proximity to other cues.
Although the stereotype of heterosexual flirta-
tion often suggests that the male is the one to ini-
tiate interaction, nonverbal researchers contend
that women perform the early nonverbal signaling
(Grammer, Kruck, Juette, & Fink, 2000). Females
have been characterized as “selectors” who attract
attention by displaying openness to interaction via
laughing, head tossing, grooming behaviors, self-
touching or caressing objects, and bodily keeping
time to music (Guéguen, 2008; Scheflen, 1965). In
particular, researchers have identified women’s coy
smiles (half a smile accompanied by either down-
ward facing eyes or darting eye contact) as especially
flirtatious (Moore, 1995). Men, in turn, are more
likely than women to actively approach a woman
in response to her nonverbal cues (Grammer et al.,
2000). In short, a successful flirtation is marked by
a coordinated and reciprocated sequence of each
party’s nonverbal behavior with that of the other.
The consensus then is that women are more
active in their use of nonverbal cues to communi-
cate romantic interest to men in the first stages of
flirting rather than the other way around (Moore,
2010). However, the displayed nonverbal behaviors
are more likely to be subtle so that, if necessary,
both parties can subsequently deny that that there
was ever any communication of interest. In fact
this female subtlety is so understated or ambigu-
ous at times that researchers find that both males
and females are more accurate at deciphering when
a man is being flirtatious than when a woman is
(Grammer et al., 2000; Place et al., 2009).
Fewer studies have focused on men’s nonverbal
behavior in a flirting context (J. A. Hall & Gun-
nery, 2013). Renninger, Wade, and Grammer (2004)
found that men who engage in brief, darting eye
contact, as well as moving among locations fre-
quently, touching other men (without being touched
Gender and Nonverbal Behavior
in return), and exhibiting expansive body postures,
were more likely to make contact with a flirting
female than men who engaged in fewer of these
nonverbal behaviors. Thus, whereas the nonverbal
behavior of women conveys interest in a subtle way
in a heterosexual situation, reflecting
the belief that women are more receptive than they
are active parties in a courtship, the nonverbal
behavior of men signals assertiveness, in accord with
scripts prescribing that men take a more active role
in courtship. However, behaviors by both sexes are
necessary for the interaction to be a successful one.
Once contact is established between flirting
partners, men and women tend to differ in the
way they use touch to communicate interest and to
escalate the interaction. Women continue to self-
touch more than men do, and they also touch
their partner in brief and casual ways that are
perceived to communicate playfulness and affection
(McCormick & Jones, 1989). Men, in contrast, tend
to touch more intimately and for longer durations,
and their touches are perceived as more strongly
sexual (McCormick & Jones, 1989).
Remaining Questions
Future research needs to examine how flirting
behavior is managed among gay and lesbian couples.
It is also likely that psychological gender (i.e., mas-
culinity/femininity) affects flirting behavior. More
feminine heterosexual women might employ the
subtlest nonverbal cues to signal interest to potential
partners, whereas more androgynous women might
take a relatively more direct approach. Similarly,
more feminine lesbian women might favor flirting
behaviors different from those preferred by more
androgynous lesbians. For gay men, it is possible
that flirting behavior might involve less subtle cues
and a more direct approach, especially in safe con-
texts where gay identification is assumed (e.g., gay
nightclubs). Clearly, the situation will dictate which
scripts are more likely.
CONCLUSIONS
Viewed through a gender lens, a review of the non-
verbal communication literature shows that gender
matters, although neither simply nor robustly.
153
LaFrance and Vial
The research literature on encoding and decod-
ing accuracy, smiling and gazing, touch and body
orientation, posture and gesture, gait, blushing,
and weeping shows that gender is moderately
implicated when predictions are made about the
frequency of various nonverbal behaviors. None-
theless, gender needs to be considered as a multi-
dimensional construct rather than a stand-in for
simple sex differences.
For one thing, psychological gender may be
more critical than biological gender. In other
words, nonverbal behaviors may often be telltale
indicators of femininity and masculinity rather
than manifestations of biological femaleness and
maleness. The critical issue may thus be the degree
to which any male or female personally subscribes
to societal definitions of masculinity and feminin-
ity. In some cases, psychological gender dovetails
with biological gender. Such would occur when a
biological male or biological female strongly iden-
tifies with the tenets of masculinity and feminin-
ity, respectively. The confluence of sex and gender
might be mistakenly taken as evidence of biological
gender effects when the pivotal factor is actually
consistency between psychological gender and
biological gender. To the degree to which individu-
als do not strongly identify with societal gender
norms, then we would expect less clear nonverbal
differentiation between the sexes in one or more
nonverbal behaviors.
Another aspect of the multidimensional nature
of gender is the recognition that demands to
behave in a gendered way are themselves variable.
Sometimes gender differences are manifestly
evident because the current situation induces
participants to respond with gender-differentiated
behavior. At other times, between-gender group
differences may be minimal. This does not mean
that the result nonverbal behavior is a random
fluctuation—now you see it, now you do not—but
rather that aspects of the situation make gender
salient or negligible or somewhere in between.
In the first case, we expect gender differences to
be magnified; in the second case, factors other
than gender affect the amount of observed non-
verbal behavior, and subject gender recedes as an
influential factor.
154
As a number of researchers have noted,
not only is the impact of gender highly vari-
able across situations but also that individuals
are themselves highly variable in terms of their
sensitivity to gender demands (Deaux & Major,
1987). This has the effect of producing substan-
tial within-sex variation in nonverbal display.
Depending on the circumstances, this means that
within-gender variation may exceed between-
gender variation, causing sex differences at the
group level to be minimal. At other times, gender
demands may be so salient that a substantial
proportion of women and men comply with what
they know to be the default patterns of gendered
nonverbal behavior.
As Table 6.1 reveals, gender seldom operates
alone in affecting the amount of observed non-
verbal behavior in women and men. Take smil-
ing for example. The data do show that women
smile more on average than men. However, the
data also indicate that the size of this effect cova-
ries with a number of factors. Age is one of these
moderators. Adolescents and young adults show
a clear gender pattern, with females out-smiling
males; however, among middle-age and older
adults, this sex difference all but disappears.
In similar fashion, the presence of social ten-
sion magnifies the gender difference in smiling.
Women smile more than men when the atmo-
sphere is tense, but that difference is significantly
reduced when the atmosphere is relaxed. In short,
individual differences and situational variations
affect the degree to which gender differences in
nonverbal behaviors are found.
One goal of the present review was to determine
whether women and men differ in their nonverbal
behavior. We looked for the presence and degree
of a gender difference across a range of nonverbal
behaviors and found a number of modest to moder-
ate effects. However, another goal of the present
review was to consider whether a series of variables
might help explain when such differences appear
and recede. Here, there was substantial evidence
that gender-marked nonverbal cues, far from
being fixed and stable, are malleable and flexible,
responsive to even small changes in the social and
psychological environment.
 Gender and Nonverbal Behavior

TABLE 6.1

Gender Differences in Nonverbal Behavior

Nonverbal behavior/domain Gender difference Moderator(s)

Encoding accuracy W>M Age; channel (facial vs. vocal)
Decoding accuracy W>M Cue ambiguity; flirting context; target gender; specific
emotion expressed
Smiling W>M Age; context (instrumental vs. expressive task; social
tension); psychological gender; country; race
Gaze (general) W<M Gender composition of dyad; psychological gender
Visual dominancea W<M Power mindset/status
Other-touch W>M Age; relationship type; gender composition of dyad;
type of touch (e.g., hand vs. nonhand)
Self-touch W>M
Interpersonal distance W<M Target gender
Orient body toward partner W>M Flirting context
Posture mirroring W>M Psychological gender; gender composition of dyad
Body restlessness, fidgeting W<M
Back-channel responsesb W>M
Hand gesturing W>M
Forward leaning W>M
Expansive body posturing W<M
Restricted body posturing W>M
Speech loudness W<M
Speech disturbances W<M
Volubility W<M Power mindset/status
Intrusive interruptions W<M Group versus dyadic context; gender composition of group
or dyad; relationship type; context (instrumental vs.
expressive task); psychological gender
Filled pauses W<M Context (instrumental vs. expressive task);
psychological gender
Hip sway W>M
Shoulder swagger W<M Sexual orientation
Blushing (general) W=M
“Coyness” blushing W>M
Crying frequency W>M Age

Note. W = women; M = men.

aGazing more at an interaction partner while speaking than while listening. bUsed to convey that one is actively listening

to an interaction partner, such as uttering “hmm” in reaction to a speaker’s statement.

References what, when, where, and how. Sex Roles,

Ambady, N., Hallahan, M., & Conner, B. (1999).
Accuracy of judgments of sexual orientation from
thin slices of behavior. Journal of Personality and
Social Psychology, 77, 538–547. http://dx.doi.
org/10.1037/0022-3514.77.3.538
Andersen, P. A., & Leibowitz, K. (1978). The development
and nature of the construct touch avoidance.
Environmental Psychology and Nonverbal Behavior, 3,
89–106. http://dx.doi.org/10.1007/BF01135607
Anderson, K. J., & Leaper, C. (1998). Meta-analyses of
gender effects on conversational interruption: Who,
39, 225–252. http://dx.doi.org/10.1023/
A:1018802521676
Bakan, D. (1966). The duality of human existence: An essay
on psychology and religion. Oxford, England: Rand
McNally.
Baron-Cohen, S., Wheelwright, S., Hill, J., Raste, Y., &
Plumb, I. (2001). The “Reading the Mind in the
Eyes” Test Revised Version: A study with normal
adults, and adults with Asperger syndrome or
high-functioning autism. Journal of Child Psychology
and Psychiatry, 42, 241–251. http://dx.doi.
org/10.1111/1469-7610.00715

155
LaFrance and Vial
Barrett, L. F., Robin, L., Pietromonaco, P. R., & Eyssell,
K. M. (1998). Are women the “more emotional”
sex? Evidence from emotional experiences in social
context. Cognition and Emotion, 12, 555–578.
http://dx.doi.org/10.1080/026999398379565
Becht, M. C., Poortinga, Y. H., & Vingerhoets, A. J. J. M.
(2001). Crying across countries. In A. J. J. M.
Vingerhoets & R. R. Cornelius (Eds.), Adult crying:
A biopsychosocial approach (pp. 135–158). Hove,
England: Brunner-Routledge.
Bekker, M. H. J., & Vingerhoets, A. J. J. M. (1999).
Adam’s tears: The relationship between crying,
biological sex, and gender. Psychology, Evolution, and
Gender, 1, 11–31.
Bekker, M. H. J., & Vingerhoets, A. J. J. M. (2001). Male
and female tears: Swallowing versus shedding?
The relationship between crying, biological sex,
and gender. In A. J. J. M. Vingerhoets & R. R.
Cornelius (Eds.), Adult crying: A biopsychosocial
approach (pp. 91–113). Hove, England: Brunner-
Routledge.
Bem, S. L. (1977). On the utility of alternative procedures
for assessing psychological androgyny. Journal of
Consulting and Clinical Psychology, 45, 196–205.
http://dx.doi.org/10.1037/0022-006X.45.2.196
Bente, G., Donaghy, W. C., & Suwelack, D. (1998).
Sex differences in body movement and visual
attention: An integrated analysis of movement
and gaze in mixed-sex dyads. Journal of
Nonverbal Behavior, 22, 31–58. http://dx.doi.
org/10.1023/A:1022900525673
Bögels, S. M., Alberts, M., & de Jong, P. J. (1996).
Self-consciousness, self-focused attention, blushing
propensity, and fear of blushing. Personality and
Individual Differences, 21, 573–581. http://dx.doi.
org/10.1016/0191-8869(96)00100-6
Boyatzis, C. J., Chazan, E., & Ting, C. Z. (1993).
Preschool children’s decoding of facial emotions.
Journal of Genetic Psychology, 154, 375–382.
http://dx.doi.org/10.1080/00221325.1993.10532190
Brescoll, V. L. (2011). Who takes the floor and why:
Gender, power, and volubility in organizations.
Administrative Science Quarterly, 56, 622–641.
http://dx.doi.org/10.1177/0001839212439994
Briton, N. J., & Hall, J. A. (1995). Beliefs about female
and male nonverbal communication. Sex Roles, 32,
79–90. http://dx.doi.org/10.1007/BF01544758
Brody, L. (1999). Gender, emotion, and the family.
Cambridge, MA: Harvard University Press.
Brooks, A., Schouten, B., Troje, N. F., Verfaillie, K.,
Blanke, O., & van der Zwan, R. (2008). Correlated
changes in perceptions of the gender and orientation
of ambiguous biological motion figures. Current
Biology, 18, R728–R729. http://dx.doi.org/10.1016/
j.cub.2008.06.054
156
Buck, R. (1977). Nonverbal communication of affect in
preschool in children: Relationships with personality
and skin conductance. Journal of Personality and
Social Psychology, 35, 225–236. http://dx.doi.
org/10.1037/0022-3514.35.4.225
Carney, D. R., Cuddy, A. J. C., & Yap, A. J. (2010).
Power posing: Brief nonverbal displays affect
neuroendocrine levels and risk tolerance.
Psychological Science, 21, 1363–1368. http://dx.doi.
org/10.1177/0956797610383437
Carroll, L., & Gilroy, P. J. (2002). Role of appearance
and nonverbal behaviors in the perception of
sexual orientation among lesbians and gay men.
Psychological Reports, 91, 115–122. http://dx.doi.
org/10.2466/pr0.2002.91.1.115
Chan, M., Rogers, K. H., Parisotto, K. L., & Biesanz, J. C.
(2011). Forming first impressions: The role of gender
and normative accuracy in personality perception.
Journal of Research in Personality, 45, 117–120. http://
dx.doi.org/10.1016/j.jrp.2010.11.001
Connellan, J., Baron-Cohen, S., Wheelwright, S., Batki,
A., & Ahluwalia, J. (2000). Sex differences in human
neonatal social perception. Infant Behavior and
Development, 23, 113–118. http://dx.doi.org/10.1016/
S0163-6383(00)00032-1
Custrini, R. J., & Feldman, R. S. (1989). Children’s
social competence and nonverbal encoding and
decoding of emotions. Journal of Clinical Child
Psychology, 18, 336–342. http://dx.doi.org/10.1207/
s15374424jccp1804_7
Deaux, K., & Major, B. (1987). Putting gender into
context: An interactive model of gender-related
behavior. Psychological Review, 94, 369–389.
http://dx.doi.org/10.1037/0033-295X.94.3.369
Derlega, V. J., Catanzaro, D., & Lewis, R. J. (2001).
Perceptions about tactile intimacy in same-sex and
opposite-sex pairs based on research participants’
sexual orientation. Psychology of Men & Masculinity,
2, 124–132. http://dx.doi.org/10.1037/1524-9220.
2.2.124
DeSantis, M., Mohan, P. J., & Steinhorst, R. K. (2005).
Smiling in photographs: Childhood similarities
between sexes become differences constant in
adulthood. Psychological Reports, 97, 651–665.
http://dx.doi.org/10.2466/pr0.97.2.651-665
DiBiase, R., & Gunnoe, J. (2004). Gender and culture
differences in touching behavior. Journal of Social
Psychology, 144, 49–62. http://dx.doi.org/10.3200/
SOCP.144.1.49-62
Dodd, D. K., Russell, B. L., & Jenkins, C. (1999). Smiling
in school yearbook photos: Gender differences from
kindergarten to adulthood. Psychological Record, 49,
543–554.
Dodge, K. A., Gilroy, F. D., & Fenzel, L. M. (1995).
Requisite management characteristics revisited:

Two decades later. Journal of Social Behavior and
Personality, 10, 253–264.
Dovidio, J. F., Ellyson, S. L., Keating, C. F., Heltman,
K., & Brown, C. E. (1988). The relationship of
social power to visual displays of dominance
between men and women. Journal of Personality
and Social Psychology, 54, 233–242. http://dx.doi.
org/10.1037/0022-3514.54.2.233
Drummond, P. D. (2013). Psychophysiology of the
blush. In W. R. Crozier & P. J. De Jong (Eds.),
The psychological significance of the blush
(pp. 15–38). Cambridge, England: Cambridge
University Press.
Ellyson, S. L., Dovidio, J. F., & Brown, C. E. (1992).
The look of power: Gender differences and
similarities in visual dominance behavior. In C.
Ridgeway (Ed.), Gender and interaction: The role of
microstructures in inequality (pp. 50–80). New York,
NY: Springer-Verlag.
Else-Quest, N. M., Hyde, J. S., Goldsmith, H. H., &
Van Hulle, C. A. (2006). Gender differences in
temperament: A meta-analysis. Psychological
Bulletin, 132, 33–72. http://dx.doi.org/10.1037/
0033-2909.132.1.33
Farris, C., Treat, T. A., Viken, R. J., & McFall, R. M.
(2008). Sexual coercion and the misperception of
sexual intent. Clinical Psychology Review, 28, 48–66.
http://dx.doi.org/10.1016/j.cpr.2007.03.002
Fichten, C. S., Tagalakis, V., Judd, D., Wright,
J., & Amsel, R. (1992). Verbal and nonverbal
communication cues in daily conversations and
dating. Journal of Social Psychology, 132, 751–769.
http://dx.doi.org/10.1080/00224545.1992.9712105
Fischer, A. H. (1993). Sex differences in
emotionality: Fact or stereotype? Feminism
and Psychology, 3, 303–318. http://dx.doi.
org/10.1177/0959353593033002
Fuchs, D., & Thelen, M. H. (1988). Children’s expected
interpersonal consequences of communicating their
affective state and reported likelihood of expression.
Child Development, 59, 1314–1322.
Grammer, K., Kruck, K., Juette, A., & Fink, B. (2000).
Non-verbal behavior as courtship signals: The role
of control and choice in selecting partners. Evolution
and Human Behavior, 21, 371–390. http://dx.doi.
org/10.1016/S1090-5138(00)00053-2
Guéguen, N. (2008). The effect of a woman’s smile
on a men’s courtship behavior. Social Behavior
and Personality, 36, 1233–1236. http://dx.doi.
org/10.2224/sbp.2008.36.9.1233
Hall, E. (1959). The silent language. Garden City, NY:
Doubleday.
Hall, J. A. (1978). Gender effects in decoding nonverbal
cues. Psychological Bulletin, 85, 845–857. http://
dx.doi.org/10.1037/0033-2909.85.4.845
Gender and Nonverbal Behavior
Hall, J. A. (1984). Nonverbal gender differences:
Communication accuracy and expressive style.
Baltimore, MD: The John Hopkins University Press.
Hall, J. A. (2011). Gender and status patterns in social
touch. In M. Hertenstein (Ed.), The handbook
of touch: Neuroscience, behavioral, and health
perspectives (pp. 329–350). New York, NY: Springer.
Hall, J. A., Carney, D. R., & Murphy, N. A. (2002).
Gender differences in smiling. In M. H. Abel (Ed.),
An empirical reflection on the smile (pp. 155–185).
New York, NY: Edwin Mellen Press.
Hall, J. A., Coats, E. J., & LeBeau, L. S. (2005). Nonverbal
behavior and the vertical dimension of social
relations: A meta-analysis. Psychological Bulletin,
131, 898–924. http://dx.doi.org/10.1037/
0033-2909.131.6.898
Hall, J. A., & Gunnery, S. D. (2013). Gender
differences in nonverbal communication. In
J. A. Hall & M. L. Knapp (Eds.), Nonverbal
communication (pp. 639–669). http://dx.doi.
org/10.1515/9783110238150.639
Hall, J. A., & Matsumoto, D. (2004). Gender differences
in judgments of multiple emotions from facial
expressions. Emotion, 4, 201–206. http://dx.doi.
org/10.1037/1528-3542.4.2.201
Hall, J. A., Murphy, N. A., & Schmid Mast, M. (2006).
Recall of nonverbal cues: Exploring a new definition
of interpersonal sensitivity. Journal of Nonverbal
Behavior, 30, 141–155. http://dx.doi.org/10.1007/
s10919-006-0013-3
Hall, J. A., & Schmid Mast, M. (2008). Are women always
more interpersonally sensitive than men? Impact
of goals and content domain. Personality and Social
Psychology Bulletin, 34, 144–155. http://dx.doi.
org/10.1177/0146167207309192
Hall, J. A., & Veccia, E. M. (1990). More “touching”
observations: New insights on men, women, and
interpersonal touch. Journal of Personality and
Social Psychology, 59, 1155–1162. http://dx.doi.
org/10.1037/0022-3514.59.6.1155
Hanzal, A., Segrin, C., & Dorros, S. M. (2008). The role
of marital status and age on men’s and women’s
reactions to touch from relational partner. Journal
of Nonverbal Behavior, 32, 21–35. http://dx.doi.
org/10.1007/s10919-007-0039-1
Haselton, M. G., & Nettle, D. (2006). The paranoid
optimist: An integrative evolutionary model of
cognitive biases. Personality and Social Psychology
Review, 10, 47–66. http://dx.doi.org/10.1207/
s15327957pspr1001_3
Heilman, M. E. (2001). Description and prescription:
How gender stereotypes prevent women’s ascent
up the organizational ladder. Journal of Social
Issues, 57, 657–674. http://dx.doi.org/10.1111/
0022-4537.00234
157
LaFrance and Vial
Helweg-Larsen, M., Cunningham, S. J., Carrico, A., &
Pergram, A. M. (2004). To nod or not to nod: An
observational study of nonverbal communication
and status in female and male college students.
Psychology of Women Quarterly, 28, 358–361. http://
dx.doi.org/10.1111/j.1471-6402.2004.00152.x
Henley, N. M. (1977). Body politics: Power, sex, and
nonverbal communication. Englewood Cliffs, NJ:
Prentice Hall.
Hertenstein, M. J., & Keltner, D. (2011). Gender and the
communication of emotion via touch. Sex Roles, 64,
70–80. http://dx.doi.org/10.1007/s11199-010-9842-y
Heslin, R., Nguyen, T. D., & Nguyen, M. L. (1983).
Meaning of touch: The case of touch from a stranger
or same sex person. Journal of Nonverbal Behavior, 7,
147–157. http://dx.doi.org/10.1007/BF00986945
Hess, U., Adams, R. B., Jr., Grammer, K., & Kleck, R.
E. (2009). Face gender and emotion expression:
Are angry women more like men? Journal of Vision,
9(12), 19. http://dx.doi.org/10.1167/9.12.19
Hochschild, A. (1983). The managed heart:
Commercialization of human feeling. Berkeley:
University of California Press.
Ickes, W., Gesn, P. R., & Graham, T. (2000). Gender
differences in empathic accuracy: Differential ability
or differential motivation? Personal Relationships, 7,
95–109. http://dx.doi.org/10.1111/j.1475-6811.2000.
tb00006.x
Izard, C. E. (1971). The face of emotion. New York, NY:
Appleton-Century-Crofts.
Johnson, K. L., Gill, S., Reichman, V., & Tassinary, L. G.
(2007). Swagger, sway, and sexuality: Judging sexual
orientation from body motion and morphology.
Journal of Personality and Social Psychology,
93, 321–334. http://dx.doi.org/10.1037/0022-
3514.93.3.321
Johnson, K. L., & Tassinary, L. G. (2005). Perceiving
sex directly and indirectly: Meaning in motion and
morphology. Psychological Science, 16, 890–897.
http://dx.doi.org/10.1111/j.1467-9280.2005.01633.x
Kerrigan, D. C., Todd, M. K., & Croce, U. D. (1998).
Gender differences in joint biomechanics during
walking: Normative study in young adults. American
Journal of Physical Medicine and Rehabilitation,
77, 2–7. http://dx.doi.org/10.1097/00002060-
199801000-00002
Kite, M. E., Deaux, K., & Haines, E. L. (2008). Gender
stereotypes. In F. L. Denmark & M. A. Paludi
(Eds.), Psychology of women: A handbook of issues
and theories (2nd ed., pp. 205–236). Westport, CT:
Praeger.
Knapp, M. L., & Hall, J. A. (2010). Nonverbal
communication in human interaction (7th ed.). Boston,
MA: Wadsworth.
158
Kneidinger, L. M., Maple, T. L., & Tross, S. A.
(2001). Touching behavior in sport: Functional
components, analysis of sex differences,
and ethological considerations. Journal of
Nonverbal Behavior, 25, 43–62. http://dx.doi.
org/10.1023/A:1006785107778
Kring, A. M., & Gordon, A. H. (1998). Sex differences in
emotion: Expression, experience, and physiology.
Journal of Personality and Social Psychology,
74, 686–703. http://dx.doi.org/10.1037/0022-
3514.74.3.686
La France, B. H., Henningsen, D. D., Oates, A., &
Shaw, C. M. (2009). Social-sexual interactions?
Meta-analyses of sex differences in perceptions of
flirtatiousness, seductiveness, and promiscuousness.
Communication Monographs, 76, 263–285.
http://dx.doi.org/10.1080/03637750903074701
LaFrance, M., & Banaji, M. (1992). Toward a
reconsideration of the gender–emotion relationship.
Emotion and Social Behavior, 14, 178–201.
LaFrance, M., & Carmen, B. (1980). The nonverbal
display of psychological androgyny. Journal of
Personality and Social Psychology, 38, 36–49.
http://dx.doi.org/10.1037/0022-3514.38.1.36
LaFrance, M., Hecht, M. A., & Paluck, E. L. (2003). The
contingent smile: A meta-analysis of sex differences
in smiling. Psychological Bulletin, 129, 305–334.
http://dx.doi.org/10.1037/0033-2909.129.2.305
LaFrance, M., & Ickes, W. (1981). Posture mirroring and
interactional involvement: Sex and sex typing effects.
Journal of Nonverbal Behavior, 5, 139–154. http://
dx.doi.org/10.1007/BF00986131
Leaper, C., & Ayres, M. M. (2007). A meta-
analytic review of gender variations in adults’
language use: Talkativeness, affiliative speech,
and assertive speech. Personality and Social
Psychology Review, 11, 328–363. http://dx.doi.
org/10.1177/1088868307302221
Leaper, C., & Robnett, R. D. (2011). Women are
more likely than men to use tentative language,
aren’t they? A meta-analysis testing for gender
differences and moderators. Psychology of
Women Quarterly, 35, 129–142. http://dx.doi.
org/10.1177/0361684310392728
Leary, M. R., Britt, T. W., Cutlip, W. D., II, & Templeton,
J. L. (1992). Social blushing. Psychological Bulletin,
112, 446–460. http://dx.doi.org/10.1037/0033-
2909.112.3.446
Lerner, H. G. (1985). The dance of anger. New York, NY:
Harper & Row.
Letzring, T. D. (2010). The effects of judge-target
gender and ethnicity similarity on the accuracy of
personality judgments. Social Psychology, 41, 42–51.
http://dx.doi.org/10.1027/1864-9335/a000007

Lutchmaya, S., & Baron-Cohen, S. (2002). Human
sex differences in social and non-social looking
preferences, at 12 months of age. Infant Behavior and
Development, 25, 319–325. http://dx.doi.org/10.1016/
S0163-6383(02)00095-4
Major, B., Schmidlin, A. M., & Williams, L. (1990).
Gender patterns in social touch: The impact
of setting and age. Journal of Personality and
Social Psychology. 58, 634–643. http://dx.doi.
org/10.1037/0022-3514.58.4.634
McClure, E. B. (2000). A meta-analytic review of sex
differences in facial expression processing and their
development in infants, children, and adolescents.
Psychological Bulletin, 126, 424–453. http://dx.doi.
org/10.1037/0033-2909.126.3.424
McCormick, N. B., & Jones, A. J. (1989). Gender
differences in nonverbal flirtation. Journal of Sex
Education and Therapy, 15, 271–282.
Merten, J. (2005). Culture, gender, and the recognition
of the basic emotions. Psychologia, 48, 306–316.
http://dx.doi.org/10.2117/psysoc.2005.306
Montemayor, R., & Flannery, D. J. (1989). A
naturalistic study of the involvement of children
and adolescents with their mothers and friends:
Developmental differences in expressive behavior.
Journal of Adolescent Research, 4, 3–14. http://dx.doi.
org/10.1177/074355488941001
Moore, M. M. (1995). Courtship signaling and
adolescents: “Girls just wanna have fun”? Journal
of Sex Research, 32, 319–328. http://dx.doi.
org/10.1080/00224499509551805
Moore, M. M. (2010). Human nonverbal courtship
behavior—A brief historical review. Journal
of Sex Research, 47, 171–180. http://dx.doi.
org/10.1080/00224490903402520
Narus, L. R., Jr., & Fischer, J. L. (1982). Strong but
not silent: A reexamination of expressivity in the
relationships of men. Sex Roles, 8, 159–168.
http://dx.doi.org/10.1007/BF00287920
Neto, F. (1996). Correlates of social blushing. Personality
and Individual Differences, 20, 365–373. http://dx.doi.
org/10.1016/0191-8869(95)00180-8
Nguyen, M. L., Heslin, R., & Nguyen, T. D. (1975).
The meaning of touch: Sex and marital status
differences. Representative Research in Social
Psychology, 7, 13–18.
Nicholas, C. L. (2004). Gaydar: Eye-gaze as identity
recognition among gay men and lesbians. Sexuality
and Culture, 8, 60–86. http://dx.doi.org/10.1007/
s12119-004-1006-1
Nigg, B. M., Fisher, V., & Ronsky, J. L. (1994). Gait
characteristics as a function of age and gender.
Gait and Posture, 2, 213–220. http://dx.doi.
org/10.1016/0966-6362(94)90106-6
Gender and Nonverbal Behavior
Ozdemir, A. (2008). Shopping malls: Measuring
interpersonal distance under changing conditions
across cultures. Field Methods, 20, 226–248.
http://dx.doi.org/10.1177/1525822X08316605
Place, S. S., Todd, P. M., Penke, L., & Asendorpf, J. B.
(2009). The ability to judge the romantic interest
of others. Psychological Science, 20, 22–26. http://
dx.doi.org/10.1111/j.1467-9280.2008.02248.x
Plant, E. A., Hyde, J. S., Keltner, D., & Devine, P. G.
(2000). The gender stereotyping of emotions.
Psychology of Women Quarterly, 24, 81–92. http://
dx.doi.org/10.1111/j.1471-6402.2000.tb01024.x
Pollick, F. E., Kay, J. W., Heim, K., & Stringer, R. (2005).
Gender recognition from point-light walkers. Journal
of Experimental Psychology: Human Perception
and Performance, 31, 1247–1265. http://dx.doi.
org/10.1037/0096-1523.31.6.1247
Renninger, L. A., Wade, T. J., & Grammer, K.
(2004). Getting that female glance: Patterns and
consequences of male nonverbal behavior in
courtship contexts. Evolution and Human
Behavior, 25, 416–431. http://dx.doi.org/10.1016/
j.evolhumbehav.2004.08.006
Robinson, M. D., Johnson, J. T., & Shields, S. A. (1998).
The gender heuristic and the database: Factors
affecting the perception of gender-related differences
in the experience and display of emotions. Basic and
Applied Social Psychology, 20, 206–219. http://dx.doi.
org/10.1207/s15324834basp2003_3
Roese, N. J., Olson, J. M., Borenstein, M. N., Martin,
A., & Shores, A. L. (1992). Same-sex touching
behavior: The moderating role of homophobic
attitudes. Journal of Nonverbal Behavior, 16, 249–259.
http://dx.doi.org/10.1007/BF01462005
Rosenthal, R., Hall, J. A., DiMatteo, M. R., Rogers, P.
L., & Archer, D. (1979). Sensitivity to nonverbal
communication: The PONS test. Baltimore, MD:
Johns Hopkins University Press.
Rudman, L. A. (1998). Self-promotion as a risk
factor for women: The costs and benefits of
counterstereotypical impression management.
Journal of Personality and Social Psychology,
74, 629–645. http://dx.doi.org/10.1037/0022-
3514.74.3.629
Rule, N. O., Ambady, N., Adams, R. B., Jr., & Macrae, C.
N. (2008). Accuracy and awareness in the perception
and categorization of male sexual orientation. Journal
of Personality and Social Psychology, 95, 1019–1028.
http://dx.doi.org/10.1037/a0013194
Sasson, N. J., Pinkham, A. E., Richard, J., Hughett, P.,
Gur, R. E., & Gur, R. C. (2010). Controlling for
response biases clarifies sex and age differences
in facial affect recognition. Journal of Nonverbal
Behavior, 34, 207–221. http://dx.doi.org/10.1007/
s10919-010-0092-z
159
LaFrance and Vial
Scheflen, A. E. (1964). The significance of posture in
communication systems. Psychiatry: Journal for the
Study of Interpersonal Processes, 27, 316–331.
Scheflen, A. E. (1965). Quasi-courtship behavior in
psychotherapy. Psychiatry: Journal for the Study of
Interpersonal Processes, 28, 245–257.
Scherer, K. R., Banse, R., & Wallbott, H. G. (2001).
Emotion inferences from vocal expression correlate
across languages and cultures. Journal of Cross-
Cultural Psychology, 32, 76–92. http://dx.doi.
org/10.1177/0022022101032001009
Schmid Mast, M., & Sczesny, S. (2010). Gender, power,
and nonverbal behavior. In J. C. Chrisler & D.
R. McCreary (Eds.), Handbook of gender research
in psychology (pp. 411–425). http://dx.doi.
org/10.1007/978-1-4419-1465-1_20
Shearn, D., Spellman, L., Straley, B., Meirick, J., &
Stryker, K. (1999). Empathic blushing in friends
and strangers. Motivation and Emotion, 23, 307–316.
http://dx.doi.org/10.1023/A:1021342910378
Shields, S. (1987). Women, men, and the dilemma of
emotion. In P. Shaver & C. Hendrick (Eds.),
Review of personality and social psychology: Vol. 7.
Sex and gender (pp. 229–251). Beverly Hills,
CA: Sage.
Shotland, R. L., & Craig, J. M. (1988). Can men and
women differentiate between friendly and sexually
interested behavior? Social Psychology Quarterly, 51,
66–73. http://dx.doi.org/10.2307/2786985
Smith, J. C. S., Vogel, D. L., Madon, S., & Edwards,
S. R. (2011). The power of touch: Nonverbal
communication within married dyads. Counseling
Psychologist, 39, 764–787. http://dx.doi.
org/10.1177/0011000010385849
Stier, D. S., & Hall, J. A. (1984). Gender differences in
touch: An empirical and theoretical review. Journal
of Personality and Social Psychology, 47, 440–459.
http://dx.doi.org/10.1037/0022-3514.47.2.440
Székely, E., Tiemeier, H., Arends, L. R., Jaddoe, V. W. V.,
Hofman, A., Verhulst, F. C., & Herba, C. M. (2011).
Recognition of facial expressions of emotions by
3-year-olds. Emotion, 11, 425–435. http://dx.doi.
org/10.1037/a0022587
Thomas, G., & Fletcher, G. J. O. (2003). Mind-reading
accuracy in intimate relationships: Assessing
the roles of the relationship, the target, and the
judge. Journal of Personality and Social Psychology,
85, 1079–1094. http://dx.doi.org/10.1037/0022-
3514.85.6.1079
Tiedens, L. Z., & Fragale, A. R. (2003). Power moves:
Complementarity in dominant and submissive
nonverbal behavior. Journal of Personality and
Social Psychology, 84, 558–568. http://dx.doi.
org/10.1037/0022-3514.84.3.558
160
Troje, N. F. (2002). Decomposing biological motion: A
framework for analysis and synthesis of human gait
patterns. Journal of Vision, 2(5), 2. http://dx.doi.
org/10.1167/2.5.2
Trommsdorff, G., & John, H. (1992). Decoding affective
communication in intimate relationships. European
Journal of Social Psychology, 22, 41–54. http://dx.doi.
org/10.1002/ejsp.2420220106
Vingerhoets, A. (2013). Why only humans weep:
Unravelling the mystery of tears. Oxford, England:
Oxford University Press.
Vingerhoets, A., & Scheirs, J. (2000). Gender
differences in crying: Empirical findings and
possible explanations. In A. H. Fisher (Ed.), Gender
and emotion: Social psychological perspectives
(pp. 143–165). http://dx.doi.org/10.1017/
CBO9780511628191.008
Vogt, D. S., & Colvin, C. R. (2003). Interpersonal
orientation and the accuracy of personality
judgments. Journal of Personality, 71, 267–295.
http://dx.doi.org/10.1111/1467-6494.7102005
von Hooff, J. A. R. A. M. (2013). Colors of the face: A
comparative glance. In W. R. Crpzzoer & P. J. De
Jong (Eds.), The psychological significance of the
blush (pp. 77–99). Cambridge, England: Cambridge
University Press.
Vrugt, A., & Luyerink, M. (2000). The contribution of
bodily posture to gender stereotypic impressions.
Social Behavior and Personality: An International
Journal, 28, 91–103. http://dx.doi.org/10.2224/
sbp.2000.28.1.91
Wagner, H. L., MacDonald, C. J., & Manstead, A. S.
(1986). Communication of individual emotions by
spontaneous facial expressions. Journal of Personality
and Social Psychology, 50, 737–743. http://dx.doi.
org/10.1037/0022-3514.50.4.737
Webbink, P. (1981). Nonverbal behavior and gay/lesbian
orientation. In C. Mayo & S. Henley (Eds.), Gender
and nonverbal behavior (pp. 253–259). http://dx.doi.
org/10.1007/978-1-4612-5953-4_12
Weisfeld, G. E., & Beresford, J. M. (1982). Erectness of
posture as an indicator of dominance or success in
humans. Motivation and Emotion, 6, 113–131.
http://dx.doi.org/10.1007/BF00992459
West, C., & Zimmerman, D. H. (1987). Doing gender.
Gender and Society, 1, 125–151. http://dx.doi.
org/10.1177/0891243287001002002
Williams, J. E., & Best, D. L. (1990). Measuring sex
stereotypes: A multination study (Rev. ed.). Thousand
Oaks, CA: Sage.
Williams, L. M., Mathersul, D., Palmer, D. M., Gur,
R. C., Gur, R. E., & Gordon, E. (2009). Explicit
identification and implicit recognition of facial
emotions: I. Age effects in males and females across

10 decades. Journal of Clinical and Experimental
Neuropsychology, 31, 257–277. http://dx.doi.
org/10.1080/13803390802255635
Willis, F. N., Jr., & Briggs, L. F. (1992). Relationship
and touch in public settings. Journal of Nonverbal
Behavior, 16, 55–63. http://dx.doi.org/10.1007/
BF00986879
Willis, F. N., Jr., & Dodds, R. A. (1998). Age,
relationship, and touch initiation. Journal of
Social Psychology, 138, 115–123. http://dx.doi.
org/10.1080/00224549809600359
Willis, F. N., Jr., & Rawdon, V. A. (1994). Gender and
national differences in attitudes toward same-gender
touch. Perceptual and Motor Skills, 78, 1027–1034.
http://dx.doi.org/10.2466/pms.1994.78.3.1027
Gender and Nonverbal Behavior
Wondergem, T. R., & Friedlmeier, M. (2012). Gender
and ethnic differences in smiling: A yearbook
photographs analysis from kindergarten through
12th grade. Sex Roles, 67, 403–411. http://dx.doi.
org/10.1007/s11199-012-0158-y
Yee, N., Bailenson, J. N., Urbanek, M., Chang, F., &
Merget, D. (2007). The unbearable likeness of being
digital: The persistence of nonverbal social norms in
online virtual environments. CyberPsychology and
Behavior, 10, 115–121. http://dx.doi.org/10.1089/
cpb.2006.9984
Zuckerman, M., DeFrank, R. S., Spiegel, N. H., &
Larrance, D. T. (1982). Masculinity–femininity and
encoding of nonverbal cues. Journal of Personality
and Social Psychology, 42, 548–556. http://dx.doi.
org/10.1037/0022-3514.42.3.548
161
 Chapter 7

Personality
Elysia R. Todd and David C. Funder

From the moment you meet another person, before
he or she has uttered a single word, you begin
to make inferences about his or her personality.
Decades of research on personality judgments made
at “zero-acquaintance” shows not only that these
assessments are made quickly but also with a certain
degree of accuracy (e.g., Borkenau & Liebler, 1992).
Once you start interacting with a person you will
continue to assess his or her personality, perhaps
revising your impression as you pick up on non-
verbal cues as well as information that is expressed
verbally, because behavior is not just what a person
does but how he or she does it. Expressive style and
patterns of nuance in behavior are major vehicles for
communicating personal identity. Yet, despite the
fundamental importance of nonverbal communica-
tion, research on its association with personality is
less than plethoric.
In the present chapter, we begin with a brief
survey of the historical foundations of research
connecting nonverbal behavior with personality,
and we consider some past controversies and meth-
odological issues that have taught these two fields
lessons that are not always heeded by contemporary
researchers. Then, after a short discussion of the
personality frameworks that might be best suited
for research on the relationship between personality
and nonverbal behavior, we summarize some of the
principal findings of this research.
In this chapter, we primarily focus on research
concerning the relationship between personality
 hough in raw quantity, researchers in nonverbal behavior continue to outpace their predecessors, there was a definite acceleration in the field in the
T
1
variables and two types of variables relevant to
nonverbal behavior. The first type concerns expres-
sions of nonverbal behavior. The second type of
variable concerns ability or skill in nonverbal
communication—specifically, sensitivity to non-
verbal behavior in others (decoding ability) and
the ability to exhibit intentional nonverbal behav-
iors (encoding ability; see also Chapter 23, this
handbook). Findings are organized in terms of
personality variables, and the overall question is to
what extent research has established relationships
between a given personality trait and nonverbal
behavior. Finally, we summarize our major conclu-
sions and offer some suggestions for the kind of
research needed for further progress in this area.
HISTORICAL CONTROVERSIES AND
METHODOLOGICAL ISSUES
Both personality and nonverbal behavior have long
been topics of widespread interest within psychol-
ogy, but the two topics have not often been con-
nected. Part of the reason for the lack of attention
to nonverbal correlates of personality is historical.
During the early heyday1 of nonverbal research,
personality psychology was undergoing (and only
slowly winning) a difficult battle for its very existence
known as the person–situation debate (Kenrick &
Funder, 1988). By the time the debate wound down
and the status of personality psychology had begun
to stabilize, many researchers in nonverbal behavior

1970s and early 1980s. This was likely in part due to the advancements of video-recording technology that allowed behaviors to be captured more
easily for later analysis.

http://dx.doi.org/10.1037/14669-007
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
163
Copyright © 2016 by the American Psychological Association. All rights reserved.
Todd and Funder
had moved on, either away from considering per-
sonality variables as relevant or from the field of
nonverbal behavior altogether.
Though today informed psychologists would
agree that personality traits are real and impor-
tant, there was a time, not so long ago, when many
accepted arguments that traits only exist in the eye
of the beholder or have relationships with behavior
that are too small to have any real impact. This atti-
tude seems to have carried over into the surround-
ing research in the 1970s and 1980s.
Brandt (1980), for instance, claimed the
following:
Respectable science holds . . . that there
is not any connection between the fea-
tures of the face and the character of the
person. . . . Any connection . . . would
bespeak some mystical system of cor-
respondences between the mind and the
body; scientists could only regard such a
system as absurd. (p. 91)
Such a statement was already out of date in 1980
considering that Allport and Vernon (1933) dem-
onstrated decades earlier how personality may be
expressed via observable cues. Yet, it was not until
the early 1990s that the flow of work resumed with
any confidence. Today, it is not uncommon to see
statements in the literature suggesting that it has
long been known that there is a solid connection
between personality and nonverbal behavior, but
such statements neglect the occasionally conflicting
eddies and flows of the two fields in parallel.
Personality and nonverbal variables can be
pieced together in several ways. Many researchers
on the personality side of the aisle have worked
hard to demonstrate that people can make accurate
judgments of strangers based on limited information
(e.g., Berry, 1991; Borkenau & Liebler, 1992; Gifford,
Ng, & Wilkinson, 1985; Watson, 1989), which
correlate well with self-reports of personality and
with ratings made by long-term acquaintances (e.g.,
Ambady & Rosenthal, 1992; Borkenau & Liebler,
1992; Funder & Sneed, 1993). This consistent find-
ing suggests that there must be cues in nonverbal
behavior that are indicative of personality (see also
Chapter 23, this handbook). However, the specific
164
behaviors and cues that might be related to person-
ality were less likely to be included in these studies.
In many cases, the question was “can accurate judg-
ments be made?” and not “how are accurate judg-
ments made?”
One of the discussions in the person–situation
debate regarded the consistency of behaviors across
situations, the primary outcome of which is the
idea that multiple instances and rank rather than
absolute measures of consistency may better reveal
relationships (Kenrick & Funder, 1988). This point
of view has led to a significant body of findings sug-
gesting that there are behavioral consistencies that
can be predicted by personality traits. Relatively
independently, questions about the consistency
and stability of patterns of nonverbal behaviors also
arose in several different subdomains of nonverbal
research (e.g., interpersonal synchrony; Bernieri,
Reznick, & Rosenthal, 1988). These discussions and
their outcomes can be helpful when compared and
applied in the context of the link between nonverbal
behavior and personality.
Another issue to consider when evaluating
relationships between personality and nonverbal
behavior is that the methodologies used to assess
nonverbal behavior are quite varied. These varia-
tions come in the form of differing scope, magnifi-
cation, and levels of analysis as well as in disparate
modalities of measurement. This methodological
diversity sometimes makes it difficult to compare
findings across studies as they relate to particular
personality traits.
For example, research on nonverbal commu-
nication has distinct camps of thought as to what
level of measurement is relevant. Some researchers
would argue that one should look at quantifiable
and isolatable individual behaviors, for example, the
proportion of times a person smiles in an interac-
tion. However, for all the appeal of the concept of
a clear-cut “body language,” some researchers have
professed difficulty in finding relationships between
microcoded behaviors and variables of interest
(Zuckerman, DePaulo, & Rosenthal, 1981). One
problem is born of limited sampling of situations.
It is not realistic to expect one behavior in one situ-
ation to correlate strongly with personality; such
relations are better detected from composites of

multiple behaviors or repeated measures of behav-
iors across multiple situations (Epstein, 1979)—yet,
relatively few studies assess participants’ behav-
ior more than once (Weisbuch, Slepian, Clarke,
Ambady, & Veenstra-VanderWeele, 2010).
Another issue is the optimal level of magnifica-
tion for assessing nonverbal behavior. For example,
early research on interpersonal mimicry (interac-
tion partners unintentionally copying each other’s
nonverbal behaviors) and its relationship to rapport
was fraught with difficulty. There were so many par-
ticular behaviors to code (e.g., head nods and hand
motions) that it was difficult to identify whether
mimicry was happening in high-rapport interac-
tions. Bernieri et al. (1988) eschewed painstaking
and expensive methods for microcoding specific
nonverbal behaviors that had achieved only spotty
success when tried by previous researchers. They
sought instead to demonstrate that raters’ gestalt
impressions of a pair of interactants’ nonverbal
synchrony would be more effectively correlated
with rapport, and, indeed, they found more success
with raters who were instructed to rate their overall
impression of nonverbal synchrony. Thus, findings
of weak or no relationships between a personality
variable and a set of specific and isolated nonverbal
behaviors might not necessarily imply that no such
relationship exists; it might be better illuminated
by assessing broader patterns of nonverbal behavior
instead.
Another instance of disparate modalities of mea-
surement arises in the literature on perceptions of
emotional expression and perception. Most studies
use only one type of stimulus, and participants are
typically either explicitly asked to pose an expres-
sion, or they are given a stimulus intended to elicit
emotional expressions more naturally (Zucker-
man, Larrance, Hall, DeFrank, & Rosenthal, 1979).
Methods sometimes shift for technological reasons
(e.g., still photos predate video clips and are even
now less costly to use and produce), and sometimes
these shifts alter the research question. For example,
it seems unwise to assume that a person’s ability
to intentionally express the emotion of happiness
(e.g., as demonstrated for a photograph) is psycho-
logically equivalent to how clearly his or her face
expresses happiness in motion (e.g., as captured by
Personality
video while viewing an emotion-eliciting stimuli). It
has long been known that separate brain systems are
involved in the posing of intentional and spontane-
ous facial expressions (Rinn, 1991). Despite this dis-
tinction, some reviews and meta-analyses lump such
findings into the same category.
A final problem that arises in attempting to
evaluate the literature is that many of the research
findings relating personality and nonverbal behav-
ior seem almost an afterthought, buried within
articles whose primary foci are on other topics.
Thus, although personality research and nonverbal
research are both active and thriving fields, there
is less intersection between them than might be
wished.
Many of the studies considered in this chapter
manifest at least one of the problems summarized
earlier. These issues are not insurmountable, and
they do not undermine the value of the research lit-
erature taken as a whole, but they are important to
keep in mind.
ORGANIZATIONAL FRAMEWORKS
Overall, one of the biggest issues with the litera-
ture on the relationship between personality and
nonverbal communication is its (dis)organization.
In an attempt to at least begin to ameliorate that
problem, we arrange the findings in this chapter by
personality variables. We are choosing to organize
this way instead of the other way around—in terms
of nonverbal behaviors—because whereas the list of
nonverbal behaviors is long and unstructured, some
strong contenders for models used to organize per-
sonality variables are available.
The most widely used model is the Big Five
framework, which organizes the domain of person-
ality in terms of the broad traits of Extraversion,
Neuroticism, Conscientiousness, Agreeableness,
and Openness to Experience. This framework has
become a near-consensual means of organizing the
personality literature, despite a range of slightly dif-
ferent perspectives on it, such as Goldberg’s (1990)
conception of the traits as broad domains of individ-
ual difference phenomena found in self-rating mea-
sures and McCrae and Costa’s (1996) Five-Factor
Model of personality, in which the traits are viewed
165
Todd and Funder

as basic mental structures. A large body of research INDIVIDUAL DIFFERENCES RELATED TO

relates Big Five traits to behaviors and important life NONVERBAL BEHAVIORS
outcomes (Ozer & Benet-Martínez, 2006), and it
Research on the association between personality
seems reasonable to expect that they are also related
and nonverbal behavior can mostly be organized in
to patterns of nonverbal behaviors. Therefore, the
terms of the Big Five framework, but, as was already
current chapter on personality correlates of nonver-
mentioned, this framework does not encompass all
bal behavior will be largely organized in terms of the
the important individual differences that have been
Big Five.
studied in relation to nonverbal behavior. Therefore,
This is not to suggest that there is not research
we begin with the Big Five, but we also include a
using personality variables that do not conform to
few other variables of interest.
the framework. Despite the near-ubiquity of the Big
Five, the broader definition of personality is not
Big Five Research
limited to trait-based theories. If the scope is broad-
Each of the Big Five is associated with a number of
ened to include all individual differences in psycho-
nonverbal behaviors. Many of the earliest studies on
logical functioning, then many more topics arise for
the expressions of traits were conducted before the
discussion, including gender, development, domi-
widespread acceptance of this framework. Because
nance, intelligence, relationship roles, and sexual
of this, a number of studies examine only one less-
orientation. For a review of the relationship of cul-
established trait or only parts of the framework.
ture to nonverbal communication, see Chapter 4,
Extraversion and Neuroticism have been established
this handbook. See Chapter 5, this handbook, for
and used for longer (e.g., Eysenck & Eysenck,
discussions of development. For topics on gender,
1968) and are most commonly included. However,
see Chapter 6, this handbook. We cede detailed
although they may share a label, these traits are not
reviews of these individual difference variables to
always measured comparably across studies. Defini-
our colleagues, but in the present chapter we survey
tions of constructs have evolved over time, as have
a few other individual difference variables outside
the measures used to assess them, so it is important
the Big Five.
to keep an eye on which particular measures are
It is also important to note the methodologi-
used in each study.
cal frameworks used in studying personality and
Although this part of the chapter is divided into
nonverbal behavior. In the research to be described
sections that consider each of the Big Five traits,
in this chapter, personality is measured in several
some studies are mentioned in more than one sec-
ways, the foremost of which are self-report, infor-
tion, as many studies measured more than one trait
mant or acquaintance report, and observed behav-
at a time. Table 7.1 provides a simplified overview
iors. Methods for assessing nonverbal behavior
of most of the relationships between personality and
include various forms of self-report, spontaneous
behavior that are considered in this chapter.
or elicited emotional responses, and intentional
posing or enacting of emotional states. The inter- Extraversion.  In general, extraverts appear out-
section of different methods of measurement going, talkative, and energetic. They are relatively
creates multiple perspectives on personality and likely to enjoy time spent with people and find less
nonverbal behavior. reward in time spent alone. Extraverts tend to be
This chapter’s summary of the literature is more assertive and experience positive emotions
in two major parts. In the first part, we consider more frequently. Those who are low on extraver-
personality traits that are related to spontane- sion tend to be more reserved and solitary (Hogan,
ous nonverbal behavior in response to particular Harkness, & Lubinski, 2000).
stimuli. In the second part, we consider the rela- Although extraversion is perhaps the best sup-
tionship between personality and the intentional ported of the Big Five in its relationship to non-
communication of affect or attitude via nonverbal verbal behavior, the findings concerning this trait
expression. are still not particularly well-organized. A few

166
 Personality

TABLE 7.1

Nonverbal Behavioral Correlates of Big Five Personality Traits

Nonverbal behavior
Personality trait Face Voice Body Overall
Extraversion
More nodsa More speakingb More Gesturesa,c Attractived
Friendly expressiond Loud voiced Less arm wrapa Refined and fashionable
appearanced
Self-assured expressiond Powerful voiced More left leg leana Positive expressivitye
Smilingd Pleasant voiced Less leg movementa
More facial affectf,g Clearer speechh Foot lift while walkingd
Greater visual dominancei Faster speechh More arm swingd
More eye contacti,j,k Rhythmic speechh Upright postureh
Neuroticism
Less friendly expressiond More conversational More self-touchb Unrefined appearanced
pausingb
Round faced Soft-voiced Fewer expressive gesturesb Poorly proportioned bodyd
Less pleasant voiced More negative expressivitye
Agreeableness
More nodsa More laughterl More body opennessc Attractived
Baby-facedd Fewer conversational Less gesturingc More warmthl
interruptionsl
Friendly faced Higher energy levell
Less visual dominancec More cheerfull
More visual attentionc More engaged in interactionl
Fewer negative facial More positive expressivitye
expressionsc
Conscientiousness
More direct eye-gazem Powerful voiced More foot lift while walkingd Attractived
More fluent speechl Head heightd Refined and fashionable
appearanced
Efficient movement d
Formal dressd
Less hand movement m
Less hostilityl,m
More engagement in
interactionl,m
Openness to Experience
More visual attentionc Low voiced Slow movementsd Dark garmentsd
Calm speechd Younger aged
Fewer verbalizationsc
Note. All relationships are phrased such that a correlation between the trait and behavior would be positive.
aGifford (1994). bCampbell and Rushton (1978). cBerry and Hansen (2000). dBorkenau and Liebler (1995). eGross and

John (1995). fGilbert and Reynolds (1984). gR. E. Riggio and Friedman (1986). hLippa (1998). iIizuka (1992). jMobbs
(1968). kKendon and Cook (1969). lFunder and Sneed (1993). mCarney, Jost, Gosling, and Potter (2008).

quantitative summaries of the literature have been
reported, though they are sometimes difficult to
interpret. La France, Heisel, and Beatty (2004)
conducted a meta-analysis of the literature on extra-
version and nonverbal behavior. They reported
that correlations between extraversion and various
nonverbal behaviors are somewhat inconsistent,
ranging from –.36 to .73. Their conclusion was that
the mean correlation coefficient between extraver-
sion and nonverbal behavior was .13. However, the
different behaviors measured in the studies that met
their inclusion criteria often were not connected

167
Todd and Funder
except for falling under the umbrella of “behavior”
and should probably not be considered together in
one average. This is particularly problematic given
that a negative correlation does not necessarily
reflect a failure to support a hypothesis that there
is a relationship between a behavior and extraver-
sion. For example, the variable “head-nodding” was
found to have a significant negative relationship to
extraversion. This finding should not be considered
counter to the hypothesis that extraversion has
nonverbal correlates. Furthermore, combining the
effect size of head-nodding’s relationship to extraver-
sion with the effect size for extraversion’s positive
relationship with voice volume implicitly assumes
that these different constructs can be compared on
the same scale. As the title of La France et al.’s (2004)
article suggests, a “profile” is perhaps a better approach
to considering the relationship between multiple
behaviors and a personality trait. Rather than testing
one overall effect size, it is probably better to focus on
individual relationships and their relative magnitudes
as well as how they function simultaneously.
Some of the specific behavioral correlates of
extraversion include those pertaining to gesture
and body, appearance, facial expressions, eye gaze,
and speech behaviors. Gifford (1994) unobtrusively
video-recorded unacquainted triads of same-sex
university students. Half of the 60 participants were
women. The triads were instructed to converse on
any subject they desired. Conversations each lasted
approximately 15 min. Gifford used Wiggins’s
(1979) Interpersonal Adjective Scales to assess self-
reported personality, including a gregarious-extra-
verted/aloof-introverted dimension. A number of
detailed nonverbal behaviors were coded by research
assistants using the Seated Kinesic Activity Notation
System (Version 4.1; Gifford, 1986). This coding
scheme involves quantifying frequencies, durations,
and percentage of times sampled for the different
behaviors. These nonverbal behaviors include mul-
tiple positions and actions involving the head, the
trunk of the body, the arms and hands, or the legs
and feet. Some examples include head-nodding and
leg-crossing. People with higher extraversion scores
exhibited more nods, more time spent gesturing,
less tendency to wrap arms across the body, more
left leg lean, and less leg movement (Gifford, 1994).
168
In another study using direct behavioral observa-
tion, Lippa (1998) assessed university students in a
public speaking context. They were video-recorded
as they delivered brief advertisement-style talks.
Their nonverbal behavior was rated by two inde-
pendent research assistants. These ratings included
specific cues, such as “inflected speech” and “high-
pitched voice,” as well as global ratings of expressive
style, including “jerky versus smooth” and “lethar-
gic versus animated.” Lippa measured personality
using the aggregated judgments of six additional
judges of a set of Big Five adjectives selected from
both McCrae and Costa (1996) and John (1990).
Extraverts demonstrated clearer enunciation of their
words and speech that was faster and more rhythmic
in cadence. Extraverts’ facial expressions were more
mobile, and their gestures were more energetic and
further from the body. Their posture was judged to
be more upright.
Berry and Hansen (2000) video-recorded
interactions among unacquainted pairs of women
undergraduates. They used what is known as the
Unstructured Paradigm, wherein participants are
unaware they are being recorded during a brief
period that follows the research assistant leaving the
room on a plausible pretense. In this case, during
the setup of a decoy camera, the research assistant
announced that he or she had brought the wrong
tape and needed to go retrieve the correct one. The
researchers measured participants’ Big Five person-
ality using the NEO Five-Factor Inventory (Costa &
McCrae, 1992). Consistent with other researchers’
findings, they found a positive relationship between
extraversion and gesturing.
Funder and Sneed (1993) gathered a slightly
different set of personality data from their under-
graduate participants. In this case, personality was
rated by informants who knew the participants well,
rather than the more common self-report methods.
Previously unacquainted pairs of opposite-sex par-
ticipants were video-recorded in 5-min unstructured
social interactions. Participants’ behaviors were
assessed by multiple trained raters using Riverside
Behavioral Q-sort (Funder, Furr, & Colvin, 2000).
Funder and Sneed found that extraversion was posi-
tively related to ratings of enthusiasm and energy
level, expressivity, physical animation, physical

contact and proximity with partner, and engagement
in the interaction. Extraversion was also associated
with speaking more and with a louder voice.
Borkenau and Liebler (1995) video-recorded
members of a community sample of German men
and women as they entered a room and then read a
weather report for the camera. A variety of nonver-
bal behaviors were coded by six research assistants.
These behaviors ranged from facial expressions to
stride length to judgments about appearance. Partic-
ipants’ self-reported personality was measured using
German translations (Borkenau, 1988) of Norman’s
(1963) Big Five scale. Borkenau and Liebler found
that participants who were higher in self-reported
extraversion were rated by observers as appearing
friendlier. Extraverts wore more self-assured expres-
sions and demonstrated a greater degree of smil-
ing. Extraversion was positively associated with a
participant’s attractiveness as well as the refinement
and fashionableness of his or her appearance. Extra-
version was also positively related to higher foot lift
while walking and more arm swing as well as pleas-
antness of voice and voice volume.
Iizuka (1992) had women participants inter-
viewed about their college life by a confederate,
surreptitiously video-recording them and then
later coding five nonverbal behaviors. The selected
behaviors were gazing at the interviewer, gazing at
the interviewer while listening, gazing at the inter-
viewer while speaking, gazing during subjects’ talk
time, and gazing during interviewers’ talk time.
Iizuka assessed personality using the Maudsley Per-
sonality Inventory (Eysenck, 1964) and found that
extraversion was positively correlated with gaze pro-
portion, average duration, duration during listening,
and frequency of looks.
Campbell and Rushton (1978) assessed a sample
of women occupational therapy students, who par-
ticipated in a dyadic interaction with a confeder-
ate during which they discussed their experiences
in their educational program. The participants’
extraversion was measured with the 16 Personality
Factor Questionnaire (Cattell, Saunders, & Stice,
1957) and the Eysenck Personality Questionnaire
(Eysenck & Eysenck, 1975). Fifteen measures of
nonverbal communication were coded from these
video-recorded interactions, ranging from frequency
Personality
of self-touch to speaking behaviors. Extraversion
was most strongly positively associated with amount
of speaking.
As can be seen in a later section of this chapter,
many of these behaviors overlap with nonverbal
behaviors relating to status or dominance. Given that
dominance is sometimes considered to be part of extra-
version (e.g., Costa & McCrae, 1992; DeYoung, Quilty, &
Peterson, 2007), these overlaps are unsurprising.
Expressivity is the aspect of nonverbal behavior
most frequently researched in relation to extraver-
sion. Most research assesses emotional expressivity
using either behavioral measures of encoding skill
or self-report instruments. Behavioral measures of
emotional expressivity and encoding most often
involve either posed or spontaneous expressions.
The self-report methods tend to measure either self-
perceptions of expressivity or self-reported behav-
iors related to expressivity.
Gilbert and Reynolds (1984) recruited men and
women from a community sample. Participants
completed personality measures, including the
Eysenck Personality Questionnaire (Eysenck &
Eysenck, 1975) and the Test of Attentional and
Interpersonal Style (TAIS; Nideffer, 1977). Later,
they were placed into trios to complete a mildly
competitive anagrams task while being surrepti-
tiously video-recorded. Movements of four different
body areas (face, head, fingers/hands, and body/
trunk) were coded using procedures developed
by the first author. The coded behaviors were
designed to reflect emotional responses, atten-
tion/task involvement, and bodily movement. The
facial-affect coding procedures were based on the
facial-affect cues described by Ekman and Friesen
(1976). Among other subscales, the TAIS differenti-
ates between negative and positive affect expression.
Participants with high scores on negative affect
expression reported a tendency to express anger and
negative feelings. Those scoring higher on positive
affect expression reported expressing feelings of
affection to others in both physical and verbal ways.
A total facial affect score was calculated by summing
scores for all the coded facial emotional expressiv-
ity, including “smiles,” “smiling while shaking head
no,” “shaking head yes,” and “negative facial emo-
tion.” Gilbert and Reynolds found extraversion to be
169
Todd and Funder
positively correlated with this measure of total facial
affect. Extraversion was also related to positive affect
expression as reported on the TAIS.
Gross and John (1995) also examined self-
reported expressivity. They developed the Berkeley
Expressivity Questionnaire to differentiate between
positive emotional expression and negative emo-
tional expression and gave it to a large number of
undergraduate university students. They measured
the personalities of their participants with the Big
Five Inventory (John, Donahue, & Kentle, 1991).
Their self-report measure of positive emotional
expressivity was positively related to extraversion,
and, of the Big Five, extraversion had the strongest
personality-to-expressivity relationship.
H. R. Riggio and Riggio (2002) conducted a more
focused meta-analysis on extraversion as related to
nonverbal emotional expressiveness. The studies in
their meta-analysis defined expressiveness in various
ways. Some measures focused on the expression of
emotions through facial expressions (e.g., R. E. Riggio,
1989). Other conceptualizations took broader
behaviors into account, such as social activity (e.g.,
R. E. Riggio & Friedman, 1986), and sometimes
conflated expressive ability with general tendencies
toward expressiveness. This is worth noting because
it is not too difficult to imagine a person who is very
expressive but frequently fails to express him- or
herself accurately. However, H. R. Riggio and Riggio
concluded that, overall, there is a significant positive
relationship between extraversion and expressivity
in general.
Expressivity, in its many different forms, seems
to be the behavior most indicative of extraversion.
Though the studies we surveyed used a wide range
of methods to operationalize expressivity, taken as a
whole they support one of the strongest personality-
to-nonverbal-behavior links in the literature. As for
the rest of the correlates of extraversion, the find-
ings are more varied but unsurprising. Behavioral
correlates—such as voice volume, upright posture,
and greater energy—seem generally congruent with
the concept of extraversion.
Neuroticism.  People who score high on neuroti-
cism experience more anxiety, anger, envy, guilt,
and depressed mood than those who score lower on
170
neuroticism. They are sensitive to stressors and tend
to interpret situations as frustrating and problematic
(Hogan et al., 2000).
In the study described earlier, Borkenau and
Liebler (1995) found that participants who were
higher in neuroticism as measured by self-report
were more likely to have an unrefined appearance,
poorly proportioned bodies, and rounder faces.
They tended to have less friendly expressions as
well as softer and less pleasant voices. Expanding
into more stylistic nonverbal behaviors, Campbell
and Rushton (1978), in the study described earlier,
found neuroticism to be associated with more self-
touch, more frequent conversational pausing, and
fewer expressive gestures.
Along similar lines, H. R. Riggio and Riggio’s
(2002) meta-analysis, described earlier, suggested
that neuroticism was negatively correlated with
behavioral measures of general expressivity, though
this effect did not hold for self-report measures. This
finding might in part be explained by an important
distinction between positive and negative expressiv-
ity. Gross and John (1995) differentiated between
positive emotional expression and negative emo-
tional expression. Their self-report measure of nega-
tive emotional expressivity was positively related to
neuroticism. Gilbert and Reynolds (1984), described
earlier, also found neuroticism to be positively
related to self-report measures of negative emotional
expressivity. They did not find neuroticism to be
correlated with their performance measure of nega-
tive facial expressivity during a brief competitive
activity. However, they found that neuroticism was
negatively correlated with a composite measure of
nonverbal agitation, which is perhaps not dissimilar
to overall negative expressivity.
In general, people who score high on neuroti-
cism measures appear to behave in ways that
are more tightly contained, hesitant, and self-
oriented. When they are expressive, it seems to
be in a negative manner. These behaviors are
almost the opposite of the behaviors seen for
extraversion.
Agreeableness.  People who score high on agree-
ableness are more likely to be described as kind,
sympathetic, cooperative, warm, and considerate

than people who are low on this dimension (Hogan
et al., 2000).
A few nonverbal behaviors have been identi-
fied as relating to a person’s agreeableness. Gifford
(1994), in the study described earlier, found that
agreeable individuals nodded more in dyadic inter-
actions. Remarkably, this was the only nonverbal
cue that was actually indicative of agreeableness,
despite the fact that perceivers used a great number
of other cues in their judgments of a target’s agree-
ableness. Borkenau and Liebler (1995) found that
participants who scored higher in agreeableness
were more likely to exhibit friendly facial expres-
sions. They were also rated as being more attractive
and having more of a baby-face.
Funder and Sneed (1993), described earlier,
found that agreeableness, as rated by knowledgeable
informants, was related to behaviors in the labora-
tory that showed more warmth, greater enthusiasm
and energy levels, more laughter, and more cheerful
behavior. Agreeable people displayed more engage-
ment with the interaction and interrupted their part-
ners less often.
Gross and John (1995) found that the self-
reported tendency toward positive expressivity
correlated positively with agreeableness, also
measured by self-report. Berry and Hansen (2000)
found that agreeableness positively associated with
observers’ and participants’ evaluations of interac-
tion quality. Agreeableness was also related to less
visual dominance, more visual attention, more
body openness, less gesturing, and fewer negative
facial expressions.
Out of all the Big Five traits, agreeableness seems
to have the most inconsistent relationship between
the nonverbal cues that are actually indicative of a
target’s agreeableness and those cues that are used by
judges attempting to rate a person’s agreeableness.
Both Gifford (1994) and Borkenau and Liebler (1995)
demonstrated this discrepancy clearly. However,
given that judges are able to accurately rate targets’
agreeableness even at minimal acquaintance, there are
likely some relevant nonverbal cues that lay judges
use but that researchers have not yet identified.
Conscientiousness.  Conscientiousness is defined
as being thorough, careful, or vigilant; it implies a
Personality
desire to do a task well. Conscientious people tend to
engage in self-discipline, tend to act dutifully, and are
efficient and organized as opposed to easy-going and
careless. When taken to an extreme, they may also be
workaholics, perfectionists, and compulsive in their
behavior. People who score low on conscientiousness
tend to be more laid back, less goal-oriented, and less
driven by success (Hogan et al., 2000).
Borkenau and Liebler (1995) found that
participants who were higher in self-reported con-
scientiousness were rated by observers as being
more attractive, refined and fashionable in appear-
ance, and more formally dressed. Conscientious
participants also exhibited greater foot lift while
walking, more powerful voices, as well as greater
head height, and they garnered higher ratings
regarding the efficiency of their movements.
Funder and Sneed (1993) found that consci-
entiousness, as rated by knowledgeable acquain-
tances of the participants, was positively related
to more fluent speech, more engagement with
the interaction, and less expression of hostile
behaviors.
Carney, Jost, Gosling, and Potter (2008), using
select nonverbal cues based on Funder and Sneed
(1993) and Borkenau and Liebler (1995), replicated
the associations between higher conscientiousness
and lower distractibility and hostility. Similarly,
they found that conscientious participants tended
to exhibit less hand movement and more direct eye
gaze with their interaction partners.
Fewer findings exist for conscientiousness than
for some of the other Big Five traits, but in general
it seems that conscientiousness relates to behaviors
that could be characterized as taking the interaction
seriously. Because conscientiousness is frequently
defined by tendencies that are not as directly con-
cerned with social interactions as are extraversion
and agreeableness, it is possible that these unstruc-
tured interactions are not ideally suited to capture
relationships with nonverbal behaviors. Perhaps in
more goal-oriented situations, conscientious people
would exhibit more distinct nonverbal cues.
Openness to experience.  The personality trait of
openness to experience consists of features such
as imagination, sensitivity to aesthetic features and
171
Todd and Funder
pursuits, attentiveness to inner feelings, pursuit
of variety and novelty, and intellectual curiosity.
People who score low on openness tend to be more
conventional or traditional in their behavior, prefer-
ring familiar routines to new interests (Hogan et al.,
2000).
Borkenau and Liebler (1995) found higher
openness to be positively associated with more
friendliness, more expressivity, less halting speech,
and more smiling. Carney et al. (2008) found simi-
lar relationships, reporting openness to be associ-
ated with more friendliness, more expressivity, and
more smiling. Similarly, Gross and John (1995)
found openness to be positively associated with
more friendliness and more expressivity, though
this finding was based on self-reported tendencies
rather than observations of behavior. Along those
same lines, Berry and Hansen (2000) found that
openness predicted observers’ evaluations of inter-
action quality such that the interactions appeared
to be more enjoyable when the participants scored
higher in openness. In their study, participant
openness was also related to more visual attention
and less verbalization.
On the other hand, Funder and Sneed (1993)
found that higher openness was positively related
to a greater disengagement from the interaction and
a greater propensity for discussing daydreams or
philosophy. This particular finding differs from oth-
ers in that the participants’ openness was rated by
knowledgeable informants.
Openness to experience has also been sug-
gested to significantly overlap with the dimension
of conservative–liberal (Feather, 1979; Levin &
Schalmo, 1974). A longitudinal study by Block and
Block (2006) revealed that many of the personal-
ity differences between liberals and conservatives
that appear in adulthood are already present when
children are in nursery school, long before they
define themselves in terms of political orientation.
Specifically, preschool children who later identified
themselves as liberal were perceived by their teach-
ers as self-reliant, energetic, emotionally expres-
sive, gregarious, and impulsive. By contrast, those
children who later identified as conservative were
seen as rigid, inhibited, indecisive, fearful, and
overcontrolled.
172
Some findings are conflicting. Expressivity has
examples going both directions. Because many
suggest that openness has two rather distinct
components (intellect and aesthetics), there may
be differences here caused by different measures
emphasizing different facets of the greater trait. As
openness to experience is generally considered to
be the least visible (Funder & Dobroth, 1987) and
perhaps least well-defined of the Big Five, it is not
surprising that there is less to say about the relation-
ships between it and nonverbal behavior.
Other Individual Differences
Although in this chapter we primarily focus on indi-
vidual differences in the context of Big Five traits,
many other major categories of individual differ-
ences can be considered, and we would be remiss in
not touching on a few of them. These stable individ-
ual differences include status and dominance, sexual
orientation, and intelligence.
Status and dominance.  Status and dominance
are not necessarily personality traits per se. They
are often influenced by life roles or achievement.
However, in some cases they are stable dispositions
(Gough, McClosky, & Meehl, 1951; Ray, 1981), and
their relationship to nonverbal communication can
be addressed.
Schwartz, Tesser, and Powell (1982) examined
how body position affected perceptions of domi-
nance. They used an artist’s rendition of a man and
a woman each in a standing position and in a seated
position. These images were also manipulated with
a pedestal to alter the apparent vertical position of
the depicted person. These drawings were paired
such that there was a figure on the left and one on
the right. Vertical elevation, gender of target, stand-
ing versus sitting, and whether the figure was in the
foreground of the image were manipulated to result
in 64 different combinations. Each undergraduate
participant viewed half of these, randomly sampled,
and marked which of the two figures was the more
dominant. Depictions of people who were physi-
cally higher in elevation and in standing posture
were perceived as being more dominant. Being in
the foreground was perceived as being more domi-
nant. Men were perceived as being slightly more

dominant than women, and figures on the right side
of the image were perceived as being slightly more
dominant. The authors were somewhat surprised
that the standing position was seen as more domi-
nant when across from a seated figure. However,
their standing figures were depicted with feet set
apart and with a hand in their pocket. The seated
figures had hands folded in their lap and had their
legs crossed, which have been identified as being
indicative of subordinate or submissive behavior
(Tracy & Robins, 2007).
Eye gaze, while being a very specific behavior,
does have a set of findings regarding its relationship
to status. Visual dominance is defined as the ratio
of looking while speaking to looking while listening
(Exline, Ellyson, & Long, 1975). Ellyson, Dovidio,
Corson, and Vinicur (1980) examined women’s gaze
behavior in situations where one member of the
dyad was assigned a high-status position. High-status
subjects demonstrated visual dominance ratios close
to 1 (equal time spent looking while speaking and
while listening), whereas low-status subjects exhib-
ited a much lower rate of visual dominance. Addi-
tionally, subjects who scored high on a trait measure
of dominance exhibited visual behavior in a manner
similar to subjects who had been assigned a high
status. This suggests that stable trait dominance and
more fluid role-based dominance both influence eye
gaze behaviors.
Dovidio, Ellyson, Keating, Heltman, and Brown
(1988) continued to explore this finding by adding
gender, power, and expertise as conditions. Men
and women who were assigned roles in which they
held high expertise or high reward power displayed
high visual dominance. This finding was the same
for people of either gender. Both men and women
in lower status conditions looked more while listen-
ing than while speaking, producing a relatively low
visual dominance ratio. When the roles assigned
were held constant, men displayed visual behavior
similar to their patterns in the higher status condi-
tions, whereas women exhibited visual behavior
similar to lower status conditions. More informa-
tion about gender and dominance behaviors can be
found in an excellent meta-analysis on the subject
by Hall, Halberstadt, and O’Brien (1997; see also
Chapter 6, this handbook).
Personality
Sexual orientation.  Sexual orientation is another
stable individual difference that can have an impact
in how people express themselves. This topic is fre-
quently difficult to navigate. As with discussions of
correlates of sex versus gender, correlates of sexual
orientation can be misused in proscriptive or deter-
ministic ways rather than descriptive. It is next to
impossible to separate the social and cultural iden-
tity of an orientation from its mental and biological
components. That is, the research in this area can-
not yet distinguish whether a nonverbal behavior
is a socially agreed-upon behavior to signal one’s
identity or is a result of the identity itself. Thus, it is
important to note that the research we discuss here
is descriptive.
Research does suggest that gay, lesbian, and
heterosexual sexual orientations can be accurately
identified even from very thin slices of behavior or
appearance. Ambady, Hallahan, and Conner (1999)
showed participants a series of eight still images or
silent video clips of gay, lesbian, and heterosexual
men and women. Participants made judgments of
the sexual orientation of the targets. Gay and les-
bian targets were correctly identified at levels above
chance, just from brief presentations of photos.
Judgments based on dynamic nonverbal behavior
(10-s and 1-s silent video segments) were more
accurate than on the still photos. Gay men and les-
bians were more accurate than heterosexual partici-
pants in judging still photographs and 1-s clips but
not in 10-s clips, where the accuracy was virtually
identical across judges.
Knöfler and Imhof (2007) examined posture
in people with different sexual orientations. They
recruited unacquainted same-sex dyads of young
adult men and women. Dyads were composed of
heterosexual participants, homosexual participants,
or one person of each orientation. A number of
nonverbal positions and behaviors were coded by
research assistants. Categories were defined draw-
ing from existing coding schemes (Cashdan, 1998;
Ekman & Friesen, 1969; Frey et al., 1989), which
combined in order to balance scope, detail, and
relevance. Codes for body posture were selected to
represent a feminine stereotype, a masculine stereo-
type, and a neutral pattern of behavior. Specifically,
feminine posture was characterized by arms being
173
Todd and Funder
closed and touching the torso; body leaning forward
and not touching the back of the chair; and legs
as closed, folded, and tucked closer to the body.
Masculine posture was characterized by an open
extension of the body, with arms away from the
torso, legs spread or extended, and torso lean-
ing back. The neutral posture was characterized
by being relaxed, with limbs and torso neither
extended nor retracted. Gender-neutral posture
occurred significantly longer in mixed-orientation
and homosexual dyads compared to heterosexual
dyads. The authors concluded that homosexual
individuals were no more likely to imitate a posture
more characteristic of the opposite sex—that is,
gay men did not present particularly feminine pos-
ture, and lesbian women did not display especially
masculine posture types. However, both lesbian
and gay individuals spent more time exhibiting a
gender-neutral posture.
Rule, Ambady, and Hallett (2009) narrowed the
scope of information, using only limited photos of
women’s faces that excluded information beyond
the narrow confines of facial features. Women’s
sexual orientation was identifiable above chance
levels, even when the image was presented as briefly
as 40 ms. One study found that accurate judgments
of male sexual orientation could be made using indi-
vidual facial features, even when judgments were
limited to photos of the target’s eyes (Rule, Ambady,
Adams, & Macrae, 2008).
Ding and Rule (2012) used similar methodolo-
gies but expanded the target pool to include bisex-
ual men and women in addition to straight, gay, and
lesbian orientations. Participants perceived bisexual
men to be significantly different from straight men
but not gay men. Similarly, whereas bisexual and
lesbian women were not rated differently, both
groups were distinguishable from straight female
targets. These findings suggest that a straight versus
nonstraight categorization scheme is used when
judging sexual orientation. Freeman, Johnson,
Ambady, and Rule (2010) investigated whether gen-
dered facial cues might be a driving factor in these
assessments. They altered the shape and texture of
computer-generated faces on masculine and femi-
nine dimensions. The more a face’s shape and tex-
ture characteristics were adjusted to look more like
174
the opposite sex, the more likely that face was to be
identified as gay or lesbian. These culture-typical
stereotypes about people who have nonheterosexual
orientations (Freeman et al., 2010; Hartman, 2013)
may explain some of the existing findings using
identifications of orientation from photos. Use of the
stereotype-consistent strategy increased accuracy on
the whole. Targets who did not match the stereotype
were reliably misidentified.
Most of the research to date has used still pho-
tographs as stimuli, leaving plenty of room for
advances in the study of dynamic cues. An example
is Nicholas’s (2004) qualitative field study of eye
gaze during social encounters. Nicholas sought to
understand gaze and its relationship to triggering or
reinforcing one person’s perception of another per-
son’s orientation. The direct and the broken stare
were two variations of gaze that Nicholas assessed
as important for signaling and identifying non-
heterosexual orientations. According to Nicholas,
these types of gazes can be accentuated by other
forms of nonverbal communication, such as pos-
ture, gestures, and smiles.
Currently, what little research exists on this
topic is limited to gay and lesbian sexual orienta-
tions with the very occasional inclusion of bisexu-
ality. Future research in this area would do well
to include the remaining established orientations
to sexual identity. Orientations rarely included
are those such as omnisexual and asexual orienta-
tions, which are, put more colloquially, the all or
nothing orientations, respectively. Additionally,
new research is needed that includes conceptions
of sexual orientation beyond those developed in
Western cultures (Kuru-Utumpala, 2013;
Tijsseling, 2011).
Intelligence.  Intelligence is an individual differ-
ence variable that has been shown to be associated
with important life outcomes. Despite this fact, to
our knowledge, only two studies have examined the
nonverbal correlates of intelligence.
Borkenau and Liebler (1995), described ear-
lier, assessed participants’ intelligence using
Subscales 1–9 of Horn’s (1983) Leistungsprüf-
system, a German intelligence test. People with
higher intelligence scores were less likely to dress

in a showy manner. They were more likely to be
rated as wearing self-assured facial expressions.
Their voices tended to be lower and were rated
as being more pleasant. They had fewer halts
in their speech and were rated as being easy to
understand.
According to Wartenburger et al. (2010), the
type of intelligence being considered is important
for understanding nonverbal correlates. Fluid
intelligence lends itself to selecting task-relevant
cognitive information quickly and efficiently. The
authors assessed a small sample of German young
men and late-adolescent boys. Fluid intelligence
was measured using Raven Advanced Progressive
Matrices (Heller, Kratzmeier, & Lengfelder, 1998;
Raven, 1958). Participants were given a geomet-
ric analogies task in which they were to identify
whether the relationship between a first pair of
shapes matched the relationship between a second
pair of shapes. Subsequently, they were asked to
describe their strategies for completing this analo-
gies test. This description was video-taped and
coded by the researchers using the ELAN annota-
tion software developed at the Max Planck Institute
for Psycholinguistics, Nijmegen, the Netherlands
(http://www.lat-mpi.eu/tools/elan). Gestures were
coded using the Neuropsychological Gesture Cod-
ing System (Lausberg & Slöetjes, 2009; Lausberg,
Zaidel, Cruz, & Ptito, 2007), specifically focusing
on movement gestures and representational ges-
tures. Movement gestures do not contain semantic,
iconic, or metaphoric information, whereas repre-
sentational gestures do. Men scoring high in fluid
intelligence used more movement and even more
representational gestures when explaining the
strategies they used during a geometric analogies
task.
These studies are limited to laboratory situations
and only include higher education students in their
samples, which would likely limit and skew the
range of intelligence scores. However, even at zero
acquaintance, intelligence can be accurately judged
to a certain degree (Borkenau, Mauer, Riemann,
Spinath, & Angleitner, 2004). Thus, there must be
some nonverbal indicators that are indicative of a
person’s intelligence. Clearly more research on this
topic is needed.
Personality
PERSONALITY CORRELATES OF
PERFORMANCE IN NONVERBAL
COMMUNICATION
Communication is much more than just words. A
significant portion of face-to-face communication
occurs on nonverbal channels (Mehrabian & Ferris,
1967). Even considering the proliferation of e-mail
as a means of communication, advances in video-
communication technology are growing apace, and
the inclusion of nonverbal communication ability
is of continued and growing importance. In this
section, we examine the relationships between per-
sonality and nonverbal communication skills and
abilities.
Nonverbal Decoding
The act of perceiving and interpreting nonverbal
behavior is discussed here as decoding. Previous
research on decoding has shown evidence of impor-
tant individual differences in sensitivity to the non-
verbal communications of others, and some studies
have associated these individual differences with
personality variables.
Neuroticism/anxiety.  Although being nonverbally
perceptive would seem like a beneficial ability to
have, being perceptive of nonverbal cues is not nec-
essarily always advantageous and may even come at
a cost. Some of the earlier work in this domain sug-
gested that elevated levels of trait anxiety are associ-
ated with an increased ability to accurately recognize
nonverbal emotional expressions. Cunningham
(1977) assessed undergraduate participants’ non-
verbal encoding and decoding abilities in a series of
induced and posed emotional expressions.
An equal number of men and women participants
were administered a stimulus designed to be either
elative (inducing positive mood) or depressive
(Velten, 1968). After the mood induction, partici-
pants completed two sessions where they read a
neutral paragraph aloud, expressing the assigned
emotion. In the first session, participants were
instructed to focus on using their face and voice
to express the assigned emotion and to focus on
body expression in the second session. Two weeks
later, participants returned to the lab to view fellow
participants’ videos and attempt to identify which
175
Todd and Funder
of the two emotions were being expressed. These
attempts were done on isolated channels; face, voice,
and body were decoded separately. Cunningham
assessed personality using Eysenck and Eysenck’s
(1968) Eysenck Personality Inventory (EPI). People
who scored higher on neuroticism were generally
more successful in decoding emotional expressions,
particularly in the face over voice and body. Women
were also better decoders, and although women
generally score higher on neuroticism (Barrett &
Eysenck, 1984), this did not drive the relationship.
Surcinelli, Codispoti, Montebarocci, Rossi, and
Baldaro (2006) presented Italian subjects with pho-
tos of faces. These faces were selected from Ekman
and Friesen’s (1976) photographs of emotional
facial expressions. Participants viewed each image
for 10 s and then identified which emotion the face
was expressing. Surcinelli et al. found that partici-
pants who scored higher in nonclinical trait anxiety
were more successful than their less anxious peers at
correctly detecting the emotional response of fear in
images they viewed for 10 s.
However, Cooper, Rowe, and Penton-Voak (2008)
were unable to replicate this finding when the presen-
tation time of stimuli was reduced to 4 s (or less if the
participants made their determination more quickly).
They used similar methods to Surcinelli et al. (2006).
Participants were presented with faces expressing
anger, sadness, happiness, surprise, disgust, fear, or
a neutral expression. These faces were selected from
Ekman and Friesen’s (1976) set of emotional facial
expression photographs. They gave U.S. university
students the State-Trait Anxiety Inventory (Spiel-
berger, Gorsuch, Lushene, Vagg, & Jacobs, 1983).
Nonclinical trait anxiety did not relate to differences in
emotion perception across any of the seven emotions.
Shifting findings in the other direction, Mat-
sumoto et al. (2000) measured individual differ-
ences in emotion recognition ability using Japanese
and Caucasian faces as stimuli. Faces were posed
expressing anger, contempt, disgust, fear, happiness,
sadness, and surprise. Though the authors reported
several different studies, only one study found a
negative relationship between neuroticism and per-
formance in emotion recognition. In that particular
study, personality was assessed using Eysenck and
Eysenck’s (1968) EPI.
176
Puccinelli and Tickle-Degnen (2004) examined
the relationship of personality traits to a nonverbal
construct (called leakage) in women university stu-
dents. Leakage refers to the finding that people are
better able to control their facial expressions than
their body’s nonverbal expressions (Babad, Bernieri, &
Rosenthal, 1989; Edelmann & Hampson, 1981;
Ekman & Friesen, 1974; Lippa, 1978; Zuckerman,
DePaulo, & Rosenthal, 1986). If a person’s underly-
ing state differs from the mood he or she is trying to
express, his or her nonverbal behavior in the body is
more likely to reflect his or her true state than is his
or her face. Puccinelli and Tickle-Degnen assessed
participants’ nonverbal decoding skill using the
Profile of Nonverbal Sensitivity (PONS; Rosenthal,
Hall, DiMatteo, Rogers, & Archer, 1979), which
allows researchers to gather separate assessments
of a person’s ability in perceiving face and body
cues. The PONS uses a number of 2-s clips of acted
scenarios that are designed to have two main dimen-
sions of tone. Each scenario has a valence option
(either positive or negative) and a dominance option
(either dominant or submissive). These clips are
edited so that each of the face, body, and two non-
verbal audio channels are shown in various combi-
nations. Participants are asked to choose between
two possible descriptions for the scene happening
in each clip. They also assessed subjects’ personali-
ties using the NEO Five-Factor Inventory (Costa &
McCrae, 1992). Randomly assigned dyads were
video-recorded while they discussed their daily lives
and aspirations for 10 min. After the interaction,
each person completed a survey that included a few
questions regarding the rapport he or she felt during
the interaction. They found that people who were
particularly capable at reading unintentional non-
verbal information—information that is “leaking”—
were more likely to make their interaction partners
uncomfortable. This effect was especially true when
the subject of this scrutiny was high in neuroticism.
This finding is perhaps in part due to a bias
in perceptions of emotional facial expressions.
Knyazev, Bocharov, Slobodskaya, and Ryabichenko
(2008) presented Russian adolescents and college
students with photographs of faces. These faces
were selected from Ekman and Friesen’s (1976) set
of emotional facial expression photographs. Knyazev

et al. used 10 photos each of angry, happy, and neu-
tral expressions. Participants rated each photo on a
scale ranging from –100 (very hostile) to +100 (very
friendly). These ratings were then correlated with a
number of personality variables. Trait anxiety was
measured using the State-Trait Anxiety Inventory
(Hanin, 1989; Spielberger, Gorsuch, & Lushene,
1970). Knyazev et al. used a Russian translation of
the International Personality Item Pool measure for
Big Five Factor Markers (Goldberg, 1992). People
who were higher in trait anxiety were more likely
to perceive all faces as being more hostile. Though
the finding was not statistically significant, they also
found a positive correlation between neuroticism
and hostility scores, which is consistent with other
findings summarized earlier.
Introversion/extroversion.  Another personality
dimension that has been found to relate to decod-
ing accuracy is introversion/extroversion. Akert and
Panter (1988) measured students’ personality with
the EPI (Eysenck & Eysenck, 1968). To measure
decoding accuracy, they used clips selected from
public television talk shows. These silent clips were
played for participants, who were then given multi-
ple-choice quizzes about what the people in the clip
were experiencing. They found that extraverts were
significantly more accurate in interpreting the mean-
ing of nonverbal communication than introverts.
In addition, extraverts were more confident that
they were accurate decoders than were introverts.
Similarly, in one of their studies, Matsumoto et al.
(2000), described earlier, found that extraverts were
more successful at correctly identifying emotional
facial expressions in Japanese and Caucasian faces.
Lieberman and Rosenthal (2001) expanded this
finding, suggesting that introverts are less able to
multitask and therefore are poorer at nonverbal
decoding only when it is not the only task they are
attempting to accomplish. They also assessed per-
sonality with the EPI (Eysenck & Eysenck, 1968).
Lieberman and Rosenthal assigned participants to
have a telephone conversation with a fellow par-
ticipant. Partners were assigned by the researchers
based on having similar introversion/extraversion
scores. After the interaction, partners each rated the
quality of the conversation. Specifically, they rated
Personality
how well they thought they were coming across to
their interaction partner. Some participants made
these ratings immediately after the interaction, thus
having to recall the conversation as well as make
their ratings simultaneously. Others were allowed
to listen to an audio-recording of the interaction so
they could listen to the conversation without having
to participate in it. Later, new judges listened to 30-s
content-filtered audio clips of those conversations.
Content filtering screens out the higher frequencies
on an audio recording that contain the information
used to understand words in human speech. The
result sounds something like how one might hear
a conversation through a wall. The tone of voice,
pitch, and prosody are preserved, but almost none of
the linguistic content can be identified. Ratings from
these judges were compared with ratings of conver-
sation participants to assess accuracy of decoding
the partner’s enjoyment of the interaction. Introverts
who were in the multitasking condition performed
poorly in comparison to the accuracy of extraverts
or introverts who were not multitasking. In a sec-
ond study, Lieberman and Rosenthal reinforced this
finding by using a similar procedure with two dif-
ferent multitasking activities. When introverts were
instructed to concentrate on making a good conver-
sation or to concentrate on assessing how well their
partners were enjoying the interaction, their later
assessments of the quality of the interaction were
less accurate than those of extraverts.
This literature is not entirely consistent. Cun-
ningham (1977), in the study described earlier,
found that extraverts tended to be less successful
decoders when it came to the body channel of non-
verbal behavior. There was no relationship between
extraversion and nonverbal decoding ability for
face or voice channels. It is perhaps important to
remember that Cunningham only used two emo-
tional states, elation and depression, and that these
expressions were intentional. However, it is interest-
ing that Cunningham included the differentiation
between nonverbal channels, whereas others did
not, which might be a route worth pursuing further
in better understanding the relationship between
extraversion and decoding ability.
Also worth mentioning is that Knyazev et al.
(2008), described earlier, found a pleasant bias in
177
Todd and Funder
the perceptions of extraverts. Perceivers of happy
faces who were higher in extraversion were more
likely to rate happy faces as more friendly.
Perhaps the way in which extraversion influences
decoding ability is through simple practice due to
extraverts’ heightened tendencies to socialize. On
the other hand, it could be argued that people who
are more successfully skilled in interpersonal inter-
actions will find them more rewarding, which may,
in turn, reinforce tendencies toward extraversion.
Other individual differences.  Though few find-
ings regarding the remainder of the Big Five traits
exist in the literature, there is one exception.
Matsumoto et al. (2000), described earlier, also
found positive correlations between two personality
traits, openness and conscientiousness, and perfor-
mance in decoding posed nonverbal expressions
of facial affect. This finding was replicated across
three of their studies and across both the Big Five
Inventory and NEO methods of self-report assess-
ment of the Big Five.
In addition to extraversion and anxiety find-
ings described earlier, Knyazev et al. (2008) found
decoding biases that were related to several other
traits. People who scored high on the following traits
were likely to perceive faces as being more hostile.
The Behavioral Approach System (Gray, 1970)
has been theorized to regulate motives to move
toward objects or outcomes. Anger, physical aggres-
sion, hostility, and verbal aggression all correlated
positively with higher than average perceptions of
hostility in faces. Intellect, agreeableness, and con-
scientiousness predisposed subjects to perceive faces
as more friendly.
Russell, Stokes, and Snyder (1987) assessed
nonverbal decoding skill in boys. Subjects were
recruited from local elementary schools and from
community mental health agencies serving chil-
dren with behavioral and emotional difficulties. To
measure their nonverbal decoding ability, Russell
et al. used a few different performance measures.
The Face and Body PONS (Rosenthal et al., 1979)
was used, which is a shorter version of the PONS
measure described earlier. Additional measures
included the Expression Grouping and Cartoon
Predictions subtests of the Four-Factor Test of
178
Social Intelligence (O’Sullivan & Guilford, 1976).
Expression Grouping is a 30-item multiple-choice
test in which subjects choose from four line draw-
ings depicting facial expressions, hand gestures, or
body postures the one drawing that best matches the
thought, feeling, or intention repeated in each of the
three stimulus drawings. The Expression Grouping
subtest assesses the subject’s ability to abstract com-
mon attributes from similar expressive behavior.
For the Cartoon Predictions subtest, subjects choose
from three line drawings the one that best depicts a
resolution to the emotional interpersonal situation
depicted by a stimulus line drawing. The Cartoon
Predictions subtest assesses the subject’s ability to
predict the social consequences of emotionally laden
interactions. Primary caregivers completed the 131-
item short form of the revised Personality Inventory
for Children (Lachar, Gdowski, & Snyder, 1982),
which was scored on its four factor scales: Undisci-
plined/Poor Self-Control (Factor 1), Social Incompe-
tence (Factor 2), Internalization/Somatic Symptoms
(Factor 3), and Cognitive Development (Factor 4).
Russell et al. found that children who were poorer at
behavioral inhibition were also poorer decoders of
nonverbal communication across the various perfor-
mance measures used.
Nowicki and Duke (1994) developed the Diag-
nostic Analysis of Nonverbal Accuracy Scale. In it,
participants are presented with a series of posed
expressions in face, voice, and body postures. The
emotions expressed include happiness, sadness,
anger, and fear. After a brief viewing of a still image
or an audio clip, participants must select which of the
four emotions are being displayed. Nowicki and Duke
examined children’s accuracy of emotion recognition
from posed face, voice, and body-posture stimuli.
Accuracy scores increased with age and were posi-
tively related to peer and teacher ratings of personal
and social adjustment but were not related to IQ.
The correlates of nonverbal decoding ability do
not compose a clear picture. Beneficial individual
difference variables, such as social adjustment and
extraversion, have a positive relationship to decod-
ing ability. On the other hand, neuroticism and
anxiety, which are less positive, also have a positive
relationship to decoding ability according to some
findings. Though much of the research here uses an

overall decoding ability in their studies, it is possible
that different subtypes of decoding ability would
lead to different relationships. A more nuanced
understanding of these relationships is needed.
Nonverbal Encoding
The act of displaying emotions or communicating
nonverbally is discussed here under the name encod-
ing. Con artists and actors alike make a living off
the fact that efficacy in encoding information is an
ability or skill that differs between persons and is
somewhat stable. However, nonverbal encoding has
a weak history of research. This is in part due to the
more extensive cost of assessing encoding abilities
with performance measures because it requires both
a participant to perform as well as multiple observ-
ers to rate the performance for each encoder.
H. R. Riggio and Riggio’s (2002) meta-analysis of
expressive behavior and its relationship to person-
ality demonstrated a relatively stable relationship
between extraversion and emotional expressiveness.
However, this meta-analysis combined several dif-
ferent types of measurements of expressiveness.
Earlier we discussed the findings relating extraver-
sion to stable tendencies toward expressivity. Here,
however, we are discussing the studies that included
a measure of quality of expressivity, though in
some cases these overlap. Of particular interest are
the studies they examined that used performance
methodologies. Though self-reports of expressiv-
ity had the strongest relationship to extraversion,
studies with posed and spontaneous performance
measures of expressivity also demonstrated a reli-
able relationship to neuroticism. Among the stud-
ies included in the meta-analysis, R. E. Riggio,
Widaman, and Friedman (1985); R. E. Riggio and
Friedman (1986); and Buck (1975) found that
extraverts were better at accurately encoding non-
verbal expression than introverts, and that people
who scored higher on neuroticism were gener-
ally less successful in encoding posed emotional
facial expressions. Cunningham (1977) found that
extraversion was positively related with emotion
encoding ability in both the posed and spontane-
ous conditions.
Tucker and Friedman (1993) video-recorded
participants in three situations. The situations
Personality
included engaging in natural social interaction,
describing a past emotional experience, and posing
various basic emotional expressions. Naive observ-
ers judged which emotion was being communicated.
Tucker and Friedman used the Eysenck Personality
Questionnaire (Eysenck & Eysenck, 1975) to assess
personality, and findings suggest that nonverbally
skilled, charismatic participants were higher in
extraversion and expressivity.
In an unpublished study involving a nonverbal
encoding measure, Todd and Bernieri (2014) moved
beyond posed facial expressions of emotions. They
developed a task that involved participants silently
acting out a variety of simple social scenarios chosen
to be relevant to everyday lives of university stu-
dents. These scenarios had three valence options for
the same scene: a positive, negative, or neutral man-
ner. Participants were assigned several different sce-
narios and valences and enacted them for observers.
A group of fellow participants were the judges who
attempted to identify which scenario and valence
the encoder had been assigned to portray. Whereas
encoding scores were positively correlated with
measures of empathy and other measures of social
skill, the correlations with personality variables in
the NEO Five-Factor Inventory (Costa & McCrae,
1992) and the California Adult Q-Sort (Block,
1961/1978; as modified for use by nonprofessionals
by Bem & Funder, 1978) were negligible, which
suggests that perhaps encoding skill beyond posed
facial expressions does not have a simple relation-
ship with personality.
There are very few studies measuring nonverbal
encoding ability, and even fewer that relate that
ability to measures of personality. More research is
clearly needed in this area before firm conclusions
can be offered.
CONCLUSIONS AND DIRECTIONS FOR
THE FUTURE
Though the picture painted is not always clear, there
seems to be little reason to doubt that nonverbal
behavior and personality have deep and meaning-
ful connections. Research thus far has not gener-
ally been organized from a personality perspective,
making integration of findings and applications to
179
Todd and Funder
other domains difficult. Future research needs to
be better organized. For now, we suggest that the
Big Five framework would be a good place to start.
Future studies of nonverbal behavior should rou-
tinely include personality measures, especially of
the Big Five, and report the correlations even when
the primary focus of the study is perhaps elsewhere.
This practice would, over time, allow consistent and
coherent relations between personality and nonver-
bal behavior to more clearly emerge.
A second suggestion is that future research more
purposefully focus on the intersection of personal-
ity and nonverbal behavior. As was seen throughout
this chapter, these relationships were often reported
as almost an aside to the main purpose of vari-
ous studies. This lack of emphasis on personality
is perhaps unsurprising considering the history of
the field and the fact that many of the researchers
in nonverbal communication were trained in social
psychology rather than personality. While review-
ing the literature, we often came across otherwise
excellent studies of nonverbal behavior that gave
individual differences only cursory attention or even
ignored them altogether. It is time that nonverbal
behavior and personality relationships got more of
the spotlight.
Third, researchers exploring the intersection
between personality and nonverbal behavior would
do well to heed the hard-won methodological gains
made in personality research over the past few
decades. These gains include aggregating behaviors
across situations, assessing behaviors at more gen-
eral levels of analysis, and using better standardized
measures.
Using aggregation or multiple measurements
is perhaps the biggest lesson learned from the
person–situation debate. Personality researchers have
established that it is unwise to focus on one behavior
or one situation when attempting to assess correlates
between traits and behavior. To establish whether
a nonverbal behavior relates to a personality trait,
multiple opportunities to assess the appearance of the
given behavior should be used rather than a single
instance. Behaviors, of course, change substantially
across situations, but relative or rank-order consis-
tency may reveal consistent patterns. For example,
people will vary considerably in their expressivity
180
depending on context, but highly expressive people
may consistently be the most expressive people in the
room, regardless of context.
Another hard-learned lesson is standardization.
Though advances have been made in nonverbal
measurement systems, assessment of nonverbal
behavior still has not settled the way personality
assessment has. It would be wise to move away from
the apparent grab-bag of variables that results in
each lab generating its own definitions and measures
of nonverbal behavior. As elsewhere in psychology,
most of the studies of the connection between per-
sonality and nonverbal behavior lack any form of
close replication. Many large gaps remain that make
it difficult to perform meta-analysis of similar meth-
odologies or findings. When looking at different
studies using different methods, the shifting rela-
tionships may look like instability rather than the
complex flux of many different variables intersect-
ing in a single moment. This is probably why in the
present chapter we were able to draw strong conclu-
sions only at the most general level (e.g., extraverts
are more expressive overall).
As any personality psychologist will tell you, one
size does not fit all, and the world tends to work
better when that fact is taken into consideration. It
stands to reason that in situations where different
personalities need to work together (e.g., military,
workplace, politics, anywhere people are), a good
understanding of how people’s personalities might
influence their nonverbal behaviors could be useful
for improving communication. There is still much
to be learned about the ways individual differences
influence nonverbal communication and styles, but
we hope that this chapter proves as a useful stepping
stone along the path of pioneers in this wide-open
domain.
References
Akert, R. M., & Panter, A. T. (1988). Extraversion and
the ability to decode nonverbal communication.
Personality and Individual Differences, 9, 965–972.
http://dx.doi.org/10.1016/0191-8869(88)90130-4
Allport, G. W., & Vernon, P. E. (1933). Studies in
expressive movement. http://dx.doi.org/
10.1037/11566-000
Ambady, N., Hallahan, M., & Conner, B. (1999).
Accuracy of judgments of sexual orientation from

thin slices of behavior. Journal of Personality and
Social Psychology, 77, 538–547. http://dx.doi.org/10.1037/
0022-3514.77.3.538
Ambady, N., & Rosenthal, R. (1992). Thin slices of
expressive behavior as predictors of interpersonal
consequences: A meta-analysis. Psychological
Bulletin, 111, 256–274. http://dx.doi.org/10.1037/
0033-2909.111.2.256
Babad, E., Bernieri, F., & Rosenthal, R. (1989).
Nonverbal communication and leakage in the
behavior of biased and unbiased teachers. Journal of
Personality and Social Psychology, 56, 89–94. http://
dx.doi.org/10.1037/0022-3514.56.1.89
Barrett, P., & Eysenck, S. B. G. (1984). The assessment
of personality factors across 25 countries. Personality
and Individual Differences, 5, 615–632. http://dx.doi.
org/10.1016/0191-8869(84)90110-7
Bem, D. J., & Funder, D. C. (1978). Predicting more of the
people more of the time: Assessing the personality of
situation. Psychological Review, 85, 485–501. http://
dx.doi.org/10.1037/0033-295X.85.6.485
Bernieri, F., Reznick, J. S., & Rosenthal, R. (1988).
Synchrony, pseudosynchrony, and dissynchrony:
Measuring the entrainment process in mother–infant
interactions. Journal of Personality and Social
Psychology, 54, 243–253. http://dx.doi.org/10.1037/
0022-3514.54.2.243
Berry, D. S. (1991). Accuracy in social perception:
Contributions of facial and vocal information. Journal
of Personality and Social Psychology, 61, 298–307.
http://dx.doi.org/10.1037/0022-3514.61.2.298
Berry, D. S., & Hansen, J. S. (2000). Personality, nonverbal
behavior, and interaction quality in female dyads.
Personality and Social Psychology Bulletin, 26, 278–292.
http://dx.doi.org/10.1177/0146167200265002
Block, J. (1978). The Q-sort method in personality assessment
and psychiatric research. Palo Alto, CA: Consulting
Psychologists Press. (Original work published 1961)
Block, J., & Block, J. H. (2006). Nursery school
personality and political orientation two decades
later. Journal of Research in Personality, 40, 734–749.
http://dx.doi.org/10.1016/j.jrp.2005.09.005
Borkenau, P. (1988). The multiple classification of acts
and the Big Five factors of personality. Journal of
Research in Personality, 22, 337–352. http://dx.doi.
org/10.1016/0092-6566(88)90034-7
Borkenau, P., & Liebler, A. (1992). Trait inferences:
Sources of validity at zero acquaintance. Journal
of Personality and Social Psychology, 62, 645–657.
http://dx.doi.org/10.1037/0022-3514.62.4.645
Borkenau, P., & Liebler, A. (1995). Observable attributes
as manifestations and cues of personality and
intelligence. Journal of Personality, 63, 1–25. http://
dx.doi.org/10.1111/j.1467-6494.1995.tb00799.x
Personality
Borkenau, P., Mauer, N., Riemann, R., Spinath, F. M., &
Angleitner, A. (2004). Thin slices of behavior as cues
of personality and intelligence. Journal of Personality
and Social Psychology, 86, 599–614. http://dx.doi.
org/10.1037/0022-3514.86.4.599
Brandt, A. (1980). Face reading: The persistence of
physiognomy. Psychology Today, 14, 90–96.
Buck, R. (1975). Nonverbal communication affect in
children. Journal of Personality and Social Psychology,
31, 644–653. http://dx.doi.org/10.1037/h0077071
Campbell, A., & Rushton, J. P. (1978). Bodily communication
and personality. British Journal of Social and Clinical
Psychology, 17, 31–36. http://dx.doi.org/10.1111/
j.2044-8260.1978.tb00893.x
Carney, D. R., Jost, J. T., Gosling, S. D., & Potter, J.
(2008). The secret lives of liberals and conservatives:
Personality profiles, interaction styles, and the things
they leave behind. Political Psychology, 29, 807–840.
http://dx.doi.org/10.1111/j.1467-9221.2008.00668.x
Cashdan, E. (1998). Smiles, speech, and body posture:
How women and men display sociometric status and
power. Journal of Nonverbal Behavior, 22, 209–228.
http://dx.doi.org/10.1023/A:1022967721884
Cattell, R. B., Saunders, D. R., & Stice, G. (1957).
Handbook for the 16 Personality Factor Questionnaire.
Champaign, IL: Institute for Personality and Ability
Testing.
Cooper, R. M., Rowe, A. C., & Penton-Voak, I. S.
(2008). The role of trait anxiety in the recognition
of emotional facial expressions. Journal of
Anxiety Disorders, 22, 1120–1127. http://dx.doi.
org/10.1016/j.janxdis.2007.11.010
Costa, P. T., Jr., & McCrae, R. R. (1992). Revised NEO
Personality Inventory (NEO-PI-R) and NEO Five-
Factor Inventory (NEO-FFI): Professional manual.
Odessa, FL: Psychological Assessment Resources.
Cunningham, M. R. (1977). Personality and the structure
of the nonverbal communication of emotion.
Journal of Personality, 45, 564–584. http://dx.doi.
org/10.1111/j.1467-6494.1977.tb00172.x
DeYoung, C. G., Quilty, L. C., & Peterson, J. B. (2007).
Between facets and domains: 10 aspects of the Big
Five. Journal of Personality and Social Psychology, 93,
880–896. http://dx.doi.org/10.1037/0022-3514.93.5.880
Ding, J. Y. C., & Rule, N. O. (2012). Gay, straight, or somewhere
in between: Accuracy and bias in the perception of bisexual
faces. Journal of Nonverbal Behavior, 36, 165–176. http://
dx.doi.org/10.1007/s10919-011-0129-y
Dovidio, J. F., Ellyson, S. L., Keating, C. F., Heltman,
K., & Brown, C. E. (1988). The relationship of
social power to visual displays of dominance
between men and women. Journal of Personality
and Social Psychology, 54, 233–242. http://dx.doi.
org/10.1037/0022-3514.54.2.233
181
Todd and Funder
Edelmann, R. J., & Hampson, S. E. (1981). The
recognition of embarrassment. Personality and
Social Psychology Bulletin, 7, 109–116. http://dx.doi.
org/10.1177/014616728171017
Ekman, P., & Friesen, W. V. (1969). The repertoire of
nonverbal behavior. Categories, origins, usage, and
coding. Semiotica, 1, 49–98.
Ekman, P., & Friesen, W. V. (1974). Detecting deception
from the body or face. Journal of Personality and
Social Psychology, 29, 288–298. http://dx.doi.
org/10.1037/h0036006
Ekman, P., & Friesen, W. V. (1976). Pictures of facial
affect. Palo Alto, CA: Consulting Psychologist Press.
Ellyson, S. L., Dovidio, J. F., Corson, R. L., & Vinicur,
D. L. (1980). Visual dominance behavior in female
dyads: Situational and personality factors. Social
Psychology Quarterly, 43, 328–336. http://dx.doi.
org/10.2307/3033735
Epstein, S. (1979). The stability of behavior: I. On
predicting most of the people much of the time. Journal
of Personality and Social Psychology, 37, 1097–1126.
http://dx.doi.org/10.1037/0022-3514.37.7.1097
Exline, R. V., Ellyson, S. L., & Long, B. (1975). Visual
behavior as an aspect of power role relationships. In
P. Pliner, L. Krames, & T. Alloway (Eds.), Nonverbal
communication of aggression (pp. 21–52). http://
dx.doi.org/10.1007/978-1-4684-2835-3_2
Eysenck, H. J. (1964). Manual of the Maudsley Personality
Inventory. London, England: University of London
Press.
Eysenck, H. J., & Eysenck, S. B. G. (1968). Manual for
the Eysenck Personality Inventory. San Diego, CA:
Educational and Industrial Testing Service.
Eysenck, H. J., & Eysenck, S. B. G. (1975). Manual of the
Eysenck Personality Questionnaire. San Diego, CA:
Educational and Industrial Testing Service.
Feather, N. T. (1979). Value correlates of conservatism. Journal
of Personality and Social Psychology, 37, 1617–1630.
http://dx.doi.org/10.1037/0022-3514.37.9.1617
Freeman, J. B., Johnson, K. L., Ambady, N., & Rule,
N. O. (2010). Sexual orientation perception
involves gendered facial cues. Personality and Social
Psychology Bulletin, 36, 1318–1331. http://dx.doi.
org/10.1177/0146167210378755
Frey, S., Bente, G., Fuchs, A., Preiswerk, G., Glatt, A., &
Imhof, P. (1989). Spontaneous motor activity in
healthy volunteers after single doses of haloperidol.
International Clinical Psychopharmacology, 4, 39–54.
http://dx.doi.org/10.1097/00004850-198901000-00005
Funder, D. C., & Dobroth, K. M. (1987). Differences
between traits: Properties associated with interjudge
agreement. Journal of Personality and Social
Psychology, 52, 409–418. http://dx.doi.org/10.1037/
0022-3514.52.2.409
182
Funder, D. C., Furr, R. M., & Colvin, R. (2000). The
Riverside Behavioral Q-sort: A tool for the description
of social behavior. Journal of Personality, 68, 451–489.
http://dx.doi.org/10.1111/1467-6494.00103
Funder, D. C., & Sneed, C. D. (1993). Behavioral
manifestations of personality: An ecological approach
to judgmental accuracy. Journal of Personality
and Social Psychology, 64, 479–490. http://dx.doi.
org/10.1037/0022-3514.64.3.479
Gifford, R. (1986). SKANS IV: Seated Kinesic Activity
Notation System (Version 4.1) [Computer software].
Victoria, British Columbia, Canada: University of
Victoria, Department of Psychology.
Gifford, R. (1994). A lens-mapping framework for understanding
the encoding and decoding of interpersonal dispositions
in nonverbal behavior. Journal of Personality and Social
Psychology, 66, 398–412. http://dx.doi.org/10.1037/
0022-3514.66.2.398
Gifford, R., Ng, C. F., & Wilkinson, M. (1985). Nonverbal
cues in the employment interview: Links between
applicant qualities and interviewer judgments. Journal
of Applied Psychology, 70, 729–736. http://dx.doi.
org/10.1037/0021-9010.70.4.729
Gilbert, D. G., & Reynolds, J. H. (1984). Type A
personality: Correlations with personality variables
and nonverbal emotional expressions during
interpersonal competition. Personality and Individual
Differences, 5, 27–34. http://dx.doi.org/10.1016/
0191-8869(84)90134-X
Goldberg, L. R. (1990). An alternative “description of
personality”: The Big-Five factor structure. Journal
of Personality and Social Psychology, 59, 1216–1229.
http://dx.doi.org/10.1037/0022-3514.59.6.1216
Goldberg, L. R. (1992). The development of markers
for the Big-Five factor structure. Psychological
Assessment, 4, 26–42. http://dx.doi.org/10.1037/
1040-3590.4.1.26
Gough, H. G., McClosky, H., & Meehl, P. E. (1951). A
personality scale for dominance. Journal of Abnormal
and Social Psychology, 46, 360–366. http://dx.doi.
org/10.1037/h0062542
Gray, J. A. (1970). The psychophysiological basis of
introversion–extraversion. Behaviour Research and
Therapy, 8, 249–266. http://dx.doi.org/10.1016/0005-
7967(70)90069-0
Gross, J. J., & John, O. P. (1995). Facets of emotional
expressivity: Three self-report factors and their correlates.
Personality and Individual Differences, 19, 555–568. http://
dx.doi.org/10.1016/0191-8869(95)00055-B
Hall, J. A., Halberstadt, A. G., & O’Brien, C. E. (1997).
“Subordination” and nonverbal sensitivity: A
study and synthesis of findings based on trait
measures. Sex Roles, 37, 295–317. http://dx.doi.
org/10.1023/A:1025608105284

Hanin, Y. L. (1989). Cross-cultural perspectives of
the individual differences diagnostic. Voprosy
Psychologii, 4, 118–125.
Hartman, J. E. (2013). Creating a bisexual display: Making
bisexuality visible. Journal of Bisexuality, 13, 39–62.
http://dx.doi.org/10.1080/15299716.2013.755727
Heller, K. A., Kratzmeier, H., & Lengfelder, A. (1998).
Matrices-test-manual: Vol. 2. A manual with German
norms for the standard progressive matrices of J. C.
Raven. Weinheim, Germany: Beltz-Test.
Hogan, R. T., Harkness, A. R., & Lubinski, D. (2000).
Personality and individual differences. Thousand
Oaks, CA: Sage.
Horn, W. (1983). Leistungsprüfsystem [Performance test
system]. Gottingen, Germany: Hogrefe.
Iizuka, Y. (1992). Extraversion, introversion, and visual
interaction. Perceptual and Motor Skills, 74, 43–50.
http://dx.doi.org/10.2466/pms.1992.74.1.43
John, O. P. (1990). The “Big Five” factor taxonomy:
Dimensions of personality in the natural language
and in questionnaires. In L. A. Pervin (Ed.),
Handbook of personality: Theory and research
(pp. 66–100). New York, NY: Guilford Press.
John, O. P., Donahue, E. M., & Kentle, R. L. (1991). The
Big Five Inventory: Versions 4a and 54 [Technical
report]. Berkeley: University of California, Institute
of Personality and Social Research.
Kendon, A., & Cook, M. (1969). The consistency
of gaze patterns in social interaction. British
Journal of Psychology, 60, 481–494. http://dx.doi.
org/10.1111/j.2044-8295.1969.tb01222.x
Kenrick, D. T., & Funder, D. C. (1988). Profiting from
controversy: Lessons from the person–situation
debate. American Psychologist, 43, 23–34. http://
dx.doi.org/10.1037/0003-066X.43.1.23
Knöfler, T., & Imhof, M. (2007). Does sexual orientation
have an impact on nonverbal behavior in interpersonal
communication? Journal of Nonverbal Behavior, 31,
189–204. http://dx.doi.org/10.1007/s10919-007-0032-8
Knyazev, G. G., Bocharov, A. V., Slobodskaya, H. R., &
Ryabichenko, T. I. (2008). Personality-linked
biases in perception of emotional facial expressions.
Personality and Individual Differences, 44, 1093–1104.
http://dx.doi.org/10.1016/j.paid.2007.11.001
Kuru-Utumpala, J. (2013). Butching it up: An analysis of
same-sex female masculinity in Sri Lanka. Culture,
Health, and Sexuality, 15(Suppl. 2), S153–S165.
http://dx.doi.org/10.1080/13691058.2013.807520
Lachar, D., Gdowski, C. L., & Snyder, D. K. (1982). Broad-
band dimensions of psychopathology: Factor scales
for the Personality Inventory for Children. Journal
of Consulting and Clinical Psychology, 50, 634–642.
http://dx.doi.org/10.1037/0022-006X.50.5.634
Personality
La France, B. H., Heisel, A. D., & Beatty, M. J. (2004).
Is there empirical evidence for a nonverbal profile
of extraversion? A meta-analysis and critique of the
literature. Communication Monographs, 71, 28–48.
http://dx.doi.org/10.1080/03634520410001693148
Lausberg, H., & Slöetjes, H. (2009). Coding gestural
behavior with the NEUROGES–ELAN system.
Behavior Research Methods, 41, 841–849. http://
dx.doi.org/10.3758/BRM.41.3.841
Lausberg, H., Zaidel, E., Cruz, R. F., & Ptito, A. (2007).
Speech-independent production of communicative
gestures: Evidence from patients with complete callosal
disconnection. Neuropsychologia, 45, 3092–3104. http://
dx.doi.org/10.1016/j.neuropsychologia.2007.05.010
Levin, B. H., & Schalmo, G. B. (1974). Self-rated
liberalism is correlated with sensation seeking.
Psychological Reports, 34, 298. http://dx.doi.
org/10.2466/pr0.1974.34.1.298
Lieberman, M. D., & Rosenthal, R. (2001). Why
introverts can’t always tell who likes them:
Multitasking and nonverbal decoding. Journal of
Personality and Social Psychology, 80, 294–310.
http://dx.doi.org/10.1037/0022-3514.80.2.294
Lippa, R. (1978). Expressive control, expressive consistency
and the correspondence between expressive behavior
and personality. Journal of Personality, 46, 438–461.
http://dx.doi.org/10.1111/j.1467-6494.1978.tb01011.x
Lippa, R. (1998). The nonverbal display and judgment
of extraversion, masculinity, femininity, and gender
diagnosticity: A lens model analysis. Journal of
Research in Personality, 32, 80–107. http://dx.doi.
org/10.1006/jrpe.1997.2189
Matsumoto, D., LeRoux, J., Wilson-Cohn, C., Raroque, J.,
Kooken, K., Ekman, P., . . . Goh, A. (2000). A
new test to measure emotion recognition ability:
Matsumoto and Ekman’s Japanese and Caucasian
Brief Affect Recognition Test (JACBERT). Journal
of Nonverbal Behavior, 24, 179–209. http://dx.doi.
org/10.1023/A:1006668120583
McCrae, R. R., & Costa, P. T. (1996). Toward a new
generation of personality theories: Theoretical
contexts for the Five-Factor Model. In J. S. Wiggins
(Ed.), The Five-Factor Model of personality
(pp. 51–87). New York, NY: Guilford Press.
Mehrabian, A., & Ferris, S. R. (1967). Inference of
attitudes from nonverbal communication in two
channels. Journal of Consulting Psychology, 31,
248–252. http://dx.doi.org/10.1037/h0024648
Mobbs, N. A. (1968). Eye contact in relation to social
introversion/extraversion. British Journal of Social
and Clinical Psychology, 7, 305–306. http://dx.doi.
org/10.1111/j.2044-8260.1968.tb00574.x
Nicholas, C. L. (2004). Gaydar: Eye-gaze as identity
recognition among gay men and lesbians. Sexuality
183
Todd and Funder
and Culture, 8, 60–86. http://dx.doi.org/10.1007/
s12119-004-1006-1
Nideffer, R. M. (1977). Comparison of self-report and
performance measures of attention—Second look.
Journal of Perceptual and Motor Skills, 45, 1291–1294.
Norman, W. T. (1963). Toward an adequate taxonomy
of personality attributes: Replicated factors structure
in peer nomination personality ratings. Journal of
Abnormal and Social Psychology, 66, 574–583. http://
dx.doi.org/10.1037/h0040291
Nowicki, S. J., Jr., & Duke, M. P. (1994). Individual
differences in the nonverbal communication of affect:
The Diagnostic Analysis of Nonverbal Accuracy
Scale. Journal of Nonverbal Behavior, 18, 9–35. http://
dx.doi.org/10.1007/BF02169077
O’Sullivan, M., & Guilford, J. P. (1976). Four-Factor Test
of Social Intelligence (Behavioral Cognition): Manual of
instructions and interpretations. Orange, CA: Sheridan
Psychological Services.
Ozer, D. J., & Benet-Martínez, V. (2006). Personality and
the prediction of consequential outcomes. Annual
Review of Psychology, 57, 401–421. http://dx.doi.
org/10.1146/annurev.psych.57.102904.190127
Puccinelli, N. M., & Tickle-Degnen, L. (2004). Knowing
too much about others: Moderators of the relationship
between eavesdropping and rapport in social
interaction. Journal of Nonverbal Behavior, 28, 223–243.
http://dx.doi.org/10.1007/s10919-004-4157-8
Raven, J. C. (1958). Advanced Progressive Matrices.
London, England: Lewis.
Ray, J. J. (1981). Authoritarianism, dominance and
assertiveness. Journal of Personality Assessment, 45,
390–397. http://dx.doi.org/10.1207/s15327752jpa4504_8
Riggio, H. R., & Riggio, R. E. (2002). Emotional
expressiveness, extraversion, and neuroticism:
A meta-analysis. Journal of Nonverbal Behavior, 26,
195–218. http://dx.doi.org/10.1023/A:1022117500440
Riggio, R. E. (1989). Manual for the Social Skills Inventory.
Palo Alto, CA: Consulting Psychologists Press.
Riggio, R. E., & Friedman, H. S. (1986). Impression
formation: The role of expressive behavior. Journal
of Personality and Social Psychology, 50, 421–427.
http://dx.doi.org/10.1037/0022-3514.50.2.421
Riggio, R. E., Widaman, K. F., & Friedman, H. S.
(1985). Actual and perceived emotional sending and
personality correlates. Journal of Nonverbal Behavior,
9, 69–83. http://dx.doi.org/10.1007/BF00987139
Rinn, W. E. (1991). Neuropsychology of facial
expression. In R. S. Feldman & B. Rime (Eds.),
Fundamentals of nonverbal behavior (pp. 3–30).
Cambridge, England: Cambridge University Press.
Rosenthal, R., Hall, J. A., DiMatteo, M. R., Rogers,
P. L., & Archer, D. (1979). Sensitivity to nonverbal
184
communication: The PONS Test. Baltimore, MD: Johns
Hopkins University Press.
Rule, N. O., Ambady, N., Adams, R. B., Jr., & Macrae, C. N.
(2008). Accuracy and awareness in the perception
and categorization of male sexual orientation. Journal
of Personality and Social Psychology, 95, 1019–1028.
http://dx.doi.org/10.1037/a0013194
Rule, N. O., Ambady, N., & Hallett, K. C. (2009). Female
sexual orientation is perceived accurately, rapidly, and
automatically from the face and its features. Journal of
Experimental Social Psychology, 45, 1245–1251. http://
dx.doi.org/10.1016/j.jesp.2009.07.010
Russell, R. L., Stokes, J. M., & Snyder, D. K. (1987).
Predicting sensitivity to nonverbal communication
from the Personality Inventory for Children. Journal
of Consulting and Clinical Psychology, 55, 439–440.
http://dx.doi.org/10.1037/0022-006X.55.3.439
Schwartz, B., Tesser, A., & Powell, E. (1982).
Dominance cues in nonverbal behavior. Social
Psychology Quarterly, 45, 114–120. http://dx.doi.
org/10.2307/3033934
Spielberger, C. D., Gorsuch, R. L., Lushene, R., Vagg,
P. R., & Jacobs, G. A. (1983). Manual for the State-
Trait Anxiety Inventory (Form Y). Palo Alto, CA:
Consulting Psychologists Press.
Spielberger, C. D., Gorsuch, R. L., & Lushene, R. E.
(1970). Manual for the State-Trait Anxiety Inventory.
Palo Alto, CA: Consulting Psychologists Press.
Surcinelli, P., Codispoti, M., Montebarocci, O.,
Rossi, N., & Baldaro, B. (2006). Facial emotion
recognition in trait anxiety. Journal of Anxiety
Disorders, 20, 110–117. http://dx.doi.org/10.1016/
j.janxdis.2004.11.010
Tijsseling, A. C. M. (2011). Review of sex in development:
Science, sexuality, and morality in global perspective.
Archives of Sexual Behavior, 40, 1079–1080. http://
dx.doi.org/10.1007/s10508-011-9800-y
Todd, E. R., & Bernieri, F. J. (2014). [Beaver
interpersonal sensitivity project: The nonverbal
acting task]. Unpublished raw data.
Tracy, J. L., & Robins, R. W. (2007). The prototypical
pride expression: Development of a nonverbal
behavior coding system. Emotion, 7, 789–801. http://
dx.doi.org/10.1037/1528-3542.7.4.789
Tucker, J. S., & Friedman, H. S. (1993). Sex differences
in nonverbal expressiveness: Emotional expression,
personality, and impressions. Journal of Nonverbal Behavior,
17, 103–117. http://dx.doi.org/10.1007/BF01001959
Velten, E., Jr. (1968). A laboratory task for the induction of
mood states. Behavioral Research and Therapy, 6, 473–482.
http://dx.doi.org/10.1016/0005-7967(68)90028-4
Wartenburger, I., Kuhn, E., Sassenberg, U., Foth, M.,
Franz, E. A., & van der Meer, E. (2010). On the

relationship between fluid intelligence, gesture
production, and brain structure. Intelligence, 38,
193–201. http://dx.doi.org/10.1016/
j.intell.2009.11.001
Watson, D. (1989). Strangers’ ratings of the five robust
personality factors: Evidence of a surprising
convergence with self-report. Journal of Personality
and Social Psychology, 57, 120–128. http://dx.doi.
org/10.1037/0022-3514.57.1.120
Weisbuch, M., Slepian, M. L., Clarke, A., Ambady,
N., & Veenstra-VanderWeele, J. (2010). Behavioral
stability across time and situations: Nonverbal versus
verbal consistency. Journal of Nonverbal Behavior, 34,
43–56. http://dx.doi.org/10.1007/s10919-009-0079-9
Wiggins, J. S. (1979). A psychological taxonomy of trait-
descriptive terms: The interpersonal domain. Journal
Personality
of Personality and Social Psychology, 37, 395–412.
http://dx.doi.org/10.1037/0022-3514.37.3.395
Zuckerman, M., DePaulo, B. M., & Rosenthal, R. (1981).
Verbal and nonverbal communication of deception.
Advances in Experimental Social Psychology, 14, 1–59.
http://dx.doi.org/10.1016/S0065-2601(08)60369-X
Zuckerman, M., DePaulo, B. M., & Rosenthal, R. (1986).
Humans as deceivers and lie detectors. In P. D.
Blanck, R. Buck, & R. Rosenthal (Eds.), Nonverbal
communication in the clinical context (pp. 13–35).
University Park: Pennsylvania State University.
Zuckerman, M., Larrance, D. T., Hall, J. A., DeFrank,
R. S., & Rosenthal, R. (1979). Posed and spontaneous
communication of emotion via facial and vocal cues.
Journal of Personality, 47, 712–733. http://dx.doi.
org/10.1111/j.1467-6494.1979.tb00217.x
185
 Part III

SOURCE OF MESSAGES
 Chapter 8
The Physical Environment and
Nonverbal Communication
Miles L. Patterson and Susanne Quadflieg
Nonverbal communication is a highly efficient
and pervasive means of interpersonal exchange.
Although it also shapes communication mediated
via television, smart phones, and the Internet, the
focus in this chapter lies specifically on nonverbal
communication in face-to-face interactions. The
occurrence and reception of nonverbal signals
during such encounters depend critically on the
physical environment in which they take place.
For example, the close approach of a stranger is
more likely to arouse discomfort when standing
in line at a cash machine than when attending a
crowded concert (see S. Li & Li, 2007). Although
it seems obvious that every in-person interaction
occurs in a particular physical setting, the impact of
the environment on nonverbal communication and
exchange has frequently been neglected in the sci-
entific psychological literature. The circumstances
surrounding this neglect merit some discussion in
setting the framework for a broader reflection on
the interplay between the physical environment and
nonverbal communication.
HISTORICAL CONTEXT
In their analysis of the role of social ecology in
psychological science, Oishi and Graham (2010)
highlighted the contrast between the recent lack
of attention to the environment and the consider-
able attention to the topic in the middle of the last
century. For example, Kurt Lewin’s (1939) field
theory approach recognized the role of internal,
dispositional characteristics and the pressure of
http://dx.doi.org/10.1037/14669-008
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
the environment in shaping behavioral outcomes.
Both Brunswik’s (1955) probabilistic functional-
ism and Gibson’s (1979) ecological perspective
emphasized sensitivity to critical stimulus charac-
teristics in the physical environment for adaptive
perception and action. In addition, the develop-
ment of ecological psychology highlighted the
study of human behavior in the natural environ-
ments of everyday life (Barker, 1968). In short,
ecological psychology stressed the importance
of external, or ecological, validity of research, in
contrast to internal validity, which is dominant in
most research paradigms today.
The period from the 1950s through the 1970s
was a time of intense interest specifically on the
environment and nonverbal communication. The
anthropologist Edward T. Hall (1959, 1966) sensi-
tized researchers to the role of space in social set-
tings and how cultural differences in physical and
social environments affected interactions. Sommer
(1959, 1962, 1969) published a number of studies
on distance and interaction and helped to popularize
the term “personal space.” Irv Altman’s (1975) clas-
sic book—The Environment and Social Behavior: Pri-
vacy, Personal Space, Territory, Crowding—provided
a comprehensive review of, and theoretical frame-
work for, the reciprocal relationship between the
environment and social behavior. In 1976, the jour-
nal Environmental Psychology and Nonverbal Behav-
ior was launched with the intention of publishing
work that avoided the dichotomy between the
environment and nonverbal behavior in the study
of human encounters (T. Lee, 1976). Ironically,
189
Patterson and Quadflieg
just 3 years later, the dichotomy was reinforced as
the journal became the Journal of Nonverbal Behav-
ior. Furthermore, 2 years later, in 1981, the first
issue of the Journal of Environmental Psychology
was published.
However, there were larger forces at work that
portended a declining interest in the environment.
In the 1980s, the acceleration of the cognitive revo-
lution in psychology shifted the focus away from
behavior to covert mental processes. This tectonic
change in theory and research was especially evi-
dent in social psychology, with a steady decline
in behavioral studies. For example, there was a
dramatic decrease in the number of articles measur-
ing or manipulating behavior in two of the field’s
most prestigious journals (Journal of Personality
and Social Psychology and Personality and Social
Psychology Bulletin) from the 1970s through the
early 2000s (Baumeister, Vohs, & Funder, 2007;
Patterson, 2008). Self-reported mental states and
judgments were the primary focus of research in
social psychology during that time, whereas the
environment–behavior link received little attention.
Due to this narrow focus, much of what researchers
know about environmental influences on nonverbal
communication comes from studies published
several decades ago.
Most recently, however, there have been signs
of increased interest in environmentally related
issues in psychology (Knapp, Hall, & Horgan,
2014; Oishi & Graham, 2010). This development
is partially related to a rediscovery of the idea that
the human mind coevolved with a physical body
whose sensations and actions inform thoughts and
feelings (Semin & Smith, 2008). Based on this idea,
numerous studies have begun to examine how
social-cognitive processes are intimately related to
an individual’s basic perceptual and motor experi-
ences in specific physical settings (cf. L. E. Williams,
Huang, & Bargh, 2009). Drawing from these con-
temporary insights, as well as from long-established
findings, the purpose of the present chapter is to
provide a broad review of the influence of the physi-
cal environment on social interaction and nonverbal
communication. We begin by establishing the utility
of nonverbal communication in everyday life before
addressing the link between nonverbal behavior and
190
environmental settings. We then discuss the impact
of specific environmental features on nonverbal
exchange and examine how prevalent social goals
modulate the postulated environment–behavior
link. Finally, we propose an ecological systems
model of nonverbal communication that incorpo-
rates the central role of the physical environment.
THE UTILITY OF NONVERBAL
COMMUNICATION
Face-to-face interactions occur in a variety of set-
tings. It is common to assume that these interac-
tions involve conversations, but, often, that is not
the case. Goffman (1963) used the term focused
interactions to identify interactions that are centered
(or focused) around conversations. In contrast,
unfocused interactions occur without any verbal
exchange. That is, people in close proximity to
one another interact more subtly through their
nonverbal behavior. For example, as people stand
in a checkout line at the grocery store or enter an
occupied elevator, they may have no intention of
having a conversation, but they still make nonverbal
adjustments to the close presence of others. That is,
individuals negotiate their position and relationship
to one another through their nonverbal behavior.
In exploring how physical settings modulate
processes of nonverbal communication, it is impor-
tant to understand the social utility of nonverbal
exchanges more broadly. Much of the early research
on the social functions of nonverbal communication
emphasized the role of affect as the primary determi-
nant of nonverbal adjustments. Although affect plays
a role, nonverbal communication serves a number
of different functions, often independent of how
people feel (Harrison, 1973; Patterson, 1982, 1983).
During human encounters, nonverbal communica-
tion provides socially relevant information to others,
regulates interaction, expresses intimacy, exercises
influence, and manages impressions (Patterson,
2011). In other words, as a basic means of interfac-
ing with the social world, nonverbal behavior does
much more than simply signaling feelings to others.
Perceivers frequently read the nonverbal affor-
dances presented in a target person’s appearance and
behavior and adjust their own actions accordingly

(Zebrowitz & Collins, 1997). For example, the
unique configuration of a toddler’s facial and bodily
appearance prompts adults to approach and nurture
them. Specifically, a young infant’s gaze from wide
eyes that are embedded in a disproportionately large
head attached to a short, chubby, and clumsily mov-
ing body rapidly invites perceivers to believe that
such an individual is particularly dependent, love-
able, and worthy of protection (Berry & McArthur,
1986; Montepare & Zebrowitz-McArthur, 1988;
Zebrowitz & Montepare, 1992). This functionality
of nonverbal signals is emphasized in the ecologi-
cal theory of social perception. The theory stresses
the interdependence of individuals sending and
those perceiving and acting upon nonverbal signals
(McArthur & Baron, 1983; Zebrowitz, 1997; see also
Chapters 9 and 10, this handbook).
Importantly, whereas verbal communication
tends to be characterized by partners taking turns
speaking and listening, the sending and receiving of
nonverbal signals typically operate simultaneously.
Consider the example of a job candidate who is try-
ing to make a good impression on an interviewer.
Achieving this goal requires not only an expressive
behavioral routine on the sending side but also sen-
sitivity in reading the reactions of the interviewer on
the receiving side. Increased gaze at the interviewer,
for instance, not only signals the interviewee’s
interest (sending) but also facilitates reading the
interviewer’s reactions to the interviewee (receiv-
ing). Similarly, in an office meeting, the executive’s
position at the head of a table not only provides him
or her a good visual perspective for monitoring the
reactions of his or her subordinates but also signals
his or her status toward them (Jackson, Engstrom, &
Hassenzahl, 2005). In other words, the same non-
verbal cues and behaviors can serve several different
functions at the same time, as simultaneous sending
and receiving processes complement one another.
It has been argued that such concurrent process-
ing is possible because mental strategies involved in
deciphering and producing social signals frequently
operate automatically and outside of awareness
(Choi, Gray, & Ambady, 2005; Patterson, 1995).
Controlled processing of nonverbal communication
may also occur, but it depends on one’s motivation
and the availability of adequate cognitive resources
The Physical Environment and Nonverbal Communication
(Gilbert, Pelham, & Krull, 1988). It is not unheard
of, for instance, that first dates are intentionally ana-
lyzed in retrospect according to whether a potential
prospect sought sufficient “accidental” bodily con-
tact and/or flashed meaningful enough glances.
For researchers studying nonverbal communica-
tion, it is often convenient to focus their attention
on one or two behaviors in the nonverbal system.
As a result, many studies focus on a few nonverbal
behaviors in isolation, whether it is the physical
distance people establish between themselves, the
duration and frequency of mutual gaze, or what type
of facial expressions they display while interact-
ing. In everyday life, however, the sending side of
an interaction is characterized by the initiation of
multiple behaviors in a relatively coordinated man-
ner, that is, by a pattern of nonverbal behavior. In
a similar fashion, perceivers do not usually register
isolated behaviors and then assemble them. Rather,
they react in a holistic fashion, quickly forming
judgments of others (Ambady, Bernieri, & Richeson,
2000; Patterson, 2011). One example of a pattern
metric is nonverbal involvement. The degree of non-
verbal involvement between two interaction part-
ners comprises a number of interactive behaviors,
including distance, gaze, facial expression, touch,
body orientation, and lean (Patterson, 1982, 2011).
Thus, when trying to understand the social utility
of nonverbal communication, focusing on a single
component provides an incomplete picture. In fact,
sometimes the inconsistency between components
in a particular behavioral display may be especially
telling for understanding its meaning, as in the case
of deception (Ekman, Friesen, & O’Sullivan, 1988;
Vrij, Semin, & Bull, 1996; for a discussion about
gesture–speech mismatches, see also Chapter 12,
this handbook).
Finally, several factors—such as biology, cul-
ture, development, gender, and personality—shape
an individual’s production and interpretation of
nonverbal behavior. These factors are discussed in
greater detail both in the second part of this hand-
book and elsewhere (J. A. Hall & Gunnery, 2013;
Manusov & Patterson, 2006; Patterson, 2011). Our
focus here lies merely in reminding the reader that
the utility of nonverbal communication in everyday
life also depends on the attributes of the specific
191
Patterson and Quadflieg
social agents encountering one another other in
particular physical settings.
THE POWER OF SETTINGS
To understand better the unfolding of nonverbal
communication in specific environments, it is impor-
tant to appreciate the role of behavior settings from
ecological psychology (Barker, 1968; Wicker, 1979).
In general, behavior settings are bounded geographi-
cal locations containing human and nonhuman ele-
ments operating together in facilitating events over a
limited period of time (Wicker, 1979). Examples of
behavior settings include a college math class, a foot-
ball game, a funeral service in a church, or an office
visit with a physician. In each case, there is a specific
physical environment limiting behavioral options
(Gibson, 1979), a program of relatively predictable
events, and a set of social norms structuring the
events (Coleman, 1990). In each setting, there are
also actors (e.g., an office visit) and, sometimes, an
audience that views the events (e.g., a football game).
Let us consider the college math class in further
detail. The setting’s physical environment most likely
comprises a well-lit, temperature-controlled room,
FIGURE 8.1.  Example of a college seminar room geared for undergraduate learning.
192
including chairs, desks, a whiteboard, and, perhaps,
a projector as well as a computer (see Figure 8.1).
Based on these environmental elements, the
execution of certain behavioral options (e.g., to
have a seat, to take notes, to give a presentation)
is supported, whereas other, learning-unrelated
activities are discouraged (e.g., to fall asleep, to play
ping pong or to sunbathe—to name just a few).
The setting-specific actors consist of a math profes-
sor and her students with both parties foreseeing a
rather predictable series of events: Whereas the pro-
fessor is expected to lecture on math, the students’
role includes listening carefully, asking questions,
or engaging in discussion. Importantly, these events
unfold within a set of social norms that structure the
setting-specific conduct. The professor, for instance,
is expected to present material in an accessible and
nonthreatening manner, whereas the students are
expected to act attentively and politely.
In contrast to an individual differences explana-
tion for behavior, ecological psychology assumes that
settings and their related characteristics are major
factors shaping human behavior (Drew, 1971).
Thus, to a considerable degree, similar behavioral
outcomes might be expected from different actors

as they come together in a specific environment.
Instead of behavior being driven by personality
traits, motives, or attitudes, behavior is substantially
constrained by the physical and social features of
behavior settings. Thus, the same person might be
an actor across various different settings, but his or
her behavior would differ significantly throughout.
If a person acted “football game” at a math class, for
instance, he or she might be quickly invited out of
the classroom.
The processes contributing to homogeneity of
(nonverbal) behavior within settings are complex.
First, as illustrated by the math class example, based
on their specific physical features, many settings
lead to significant behavior homogeneity by simulta-
neously fostering and discouraging certain types of
actions. Second, behavior homogeneity might addi-
tionally arise from shared social attitudes toward
certain settings. Visitors at the department of motor
vehicles waiting to register their car, for instance,
might share a similar impatience to complete the
bureaucratic hurdles. Third, people are rarely within
any particular setting by pure chance. Thus, some
settings not only invite certain types of behavior
and attitudes but also specific types of people. Con-
sider an exclusive country club, for instance, that
is not only very expensive to join but also highly
selective in admitting members. Finally, individuals
frequently select settings based on their own prefer-
ences. Although a local public library may admit
almost anyone, personal interests, attitudes, and
experiences are likely to affect individuals’ choices
to attend such a setting (Mehl & Pennebaker, 2003).
In other words, settings not only select individu-
als, but individuals also select settings (Motyl, Iyer,
Oishi, Trawalter, & Nosek, 2014; Snyder & Ickes,
1985; Wicker, 1979). Initial research suggests,
for instance, that individuals habitually seeking
intense experiences (so-called sensation seekers)
are attracted toward unpredictable and stimulating
environments (for a review, see Roberti, 2004), such
as settings characterized by arousing music (McNa-
mara & Ballard, 1999) and/or involving high-risk
sport activities (Wagner & Houlihan, 1994).
An important consequence of the complementary
selection by people and settings is that people in
any given setting are likely to have more in common
The Physical Environment and Nonverbal Communication
with one another than would people sampled from a
range of different settings. Thus, the greater similarity
among individuals within a particular setting, com-
pared to those across different settings, increases the
stability, uniformity, and predictability of nonverbal
(as well as verbal) communication patterns. In a pub-
lic library, for instance, a large proportion of intro-
verted and/or studious people who hold the same
setting-specific goal (i.e., reading) are likely to meet.
As a result, verbal and nonverbal exchanges (such as
mutual glances and smiles between strangers) might
be rather uncommon in such a setting. When attend-
ing a dance club, in contrast, systematic differences in
clientele (e.g., extraverts, drunks) and setting-specific
goals (e.g., wanting to have fun) make the exchange
of glances and smiles between strangers more likely.
In addition, settings can convey such powerful
expectations regarding what kind of people they
attract that they tempt observers to see and treat
others in light of their surroundings (Loersch &
Payne, 2011). The idea that environments can taint
the perception of its occupants, also referred to as
the ecological contamination hypothesis, was origi-
nally introduced by Werthman and Piliavin (1967).
Its validity is supported by contemporary research
on interracial encounters that suggests that race-
based responses toward others differ dramatically
depending on the settings in which they take place.
The activation of negative stereotypes toward
African Americans in the United States, for instance,
is enhanced when interracial encounters occur at a
graffiti-covered street corner rather than in front of a
church (Correll, Wittenbrink, Park, Judd, & Goyle,
2011; Wittenbrink, Judd, & Park, 2001). Unfortu-
nately, further systematic research dedicated toward
studying the role of specific behavioral settings
on unfocused interactions and unique patterns of
nonverbal behavior is rather limited. An important,
albeit broad, class of sites that has attracted repeated
empirical attention so far is that of public settings.
NONVERBAL BEHAVIOR IN
PUBLIC SETTINGS
Many nonverbal interactions between people occur
as they are walking through, or briefly occupying,
public spaces, such as streets, stores, malls, sporting
193
Patterson and Quadflieg
events, or restaurants. In such public settings,
people frequently feel ambivalent about the close
presence of strangers. Although they may want some
friendly “contact” with others, they are still wary of
the uncertainty presented by them (Eibl-Eibesfeldt,
1989). Some individuals might signal a benign
intention to strangers by flashing a brief smile
(Fridlund, 1994). Initiating a brief glance and smile
signals attention to others and can open the door
for a response (Ellsworth & Langer, 1976; Laidlaw,
Foulsham, Kuhn, & Kingstone, 2011). In contrast,
people who are uncomfortable with the close pres-
ence of strangers may compensate by averting their
gaze and quickly escaping the communal space
(Patterson, 1982). Goffman (1963) used the term
civil inattention to identify a common pattern of
mixed-motive behavior when strangers approach
one another. For example, as pedestrians approach
one another on the sidewalk, they might initiate
a brief glance outside a distance of 2.5–3 m. This
glance simply serves to recognize the presence of
the other person. However, as they approach more
closely, they look away to respect the other per-
son’s privacy. Goffman offered the analogy of driv-
ing at night and dimming one’s bright lights as an
approaching car gets close.
The results of three studies on gazing behavior
among elevator riders supported Goffman’s (1963)
prediction of civil inattention in unfocused interac-
tions (Zuckerman, Miserandino, & Bernieri, 1983).
Specifically, a single glance toward another passen-
ger, followed by gaze avoidance was the most com-
mon and most positively rated pattern of elevator
behavior.
Another series of studies, however, this time
on pedestrians passing one another on the side-
walk, found that individuals did not lower their
heads and avert gaze as they closely approached
and passed one another (Cary, 1978). To examine
this observed, yet unexpected, pattern of nonverbal
exchange at greater depth, several subsequent exper-
iments focused on microinteractions between pedes-
trians (e.g., Patterson & Tubbs, 2005; Patterson,
Webb, & Schwartz, 2002; Wesselmann, Cardoso,
Slater, & Williams, 2012). In samples from an urban
Midwest area in the United States, almost half of the
observed pedestrians glanced at passing confederates
194
within the range of 2.5–3 m where Goffman (1963)
predicted gaze avoidance. Furthermore, as confeder-
ates initiated greater involvement toward approach-
ing pedestrians (i.e., from avoid, to look, to look and
smile), glances, smiles, nods, and greetings toward
the confederates also increased (Patterson & Tubbs,
2005; Patterson et al., 2002). Importantly, the viola-
tion of mutual looking and acknowledgment norms
can induce feelings of social exclusion (Wesselmann
et al., 2012; K. D. Williams, Shore, & Grahe, 1998;
Wirth, Sacco, Hugenberg, & Williams, 2010). For
instance, when confederates were instructed to look
at passing pedestrians without making eye contact,
that is, “as though looking through air,” the affected
individuals subsequently reported stronger feelings
of disconnection from others than individuals who
received either a gaze or a gaze and smile from the
confederate (Wesselmann et al., 2012).
A lack of mutual nonverbal acknowledgment,
however, is likely to be more counternormative
in some public settings than others (e.g., in small
towns relative to large cities; Korte, 1980; Newman &
McCauley, 1977) and may also depend on cultural
contexts (Kleinke, 1986). For example, differences
in unfocused interactions of pedestrians were
found between Japan and the United States
(Patterson et al., 2007). Although there were no
significant differences in the glancing rates of
Japanese and American pedestrians toward a
confederate, there were large differences in other
nonverbal reactions. Specifically, Americans
were much more likely to smile, nod, or greet the
approaching confederate. Only 2% of the Japanese
smiled back at the confederates, whereas 25% of
the Americans did. These contrasting cultural
differences may be the result of Americans and
Japanese having different norms and expectations
for relating to strangers during brief microinterac-
tions. Specifically, Japanese pedestrians, compared
to Americans, may feel less obligated to recipro-
cate greetings to outgroup members, and avoiding
passing strangers is a simple tactic to “save face”
in these ambiguous brief encounters (Lebra, 1976;
Triandis, Bontempo, Villareal, Asai, & Lucca,
1988). Thus, cultural differences fundamentally
shape the course of nonverbal exchange in
pedestrian settings.

Note that many unfocused interactions also
involve seated encounters, such as when strang-
ers meet in waiting areas, meet on park benches,
meet in libraries, or use public transportation.
A number of studies in the 1960s and 1970s
investigated behavioral adjustments to spatial
invasions while people were seated in pub-
lic spaces. In these experiments, confederates
approached solitary individuals and sat at vary-
ing distances from them. Observers recorded
how long individuals remained in the setting
(Felipe & Sommer, 1966) and/or the kinds of
behavioral adjustments they made in response to
the intrusion (Fisher & Byrne, 1975; Patterson,
Mullens, & Romano, 1971). In general, closer
approaches (sometimes as close as 15–30 cm),
precipitated more rapid departure from the set-
ting (e.g., Felipe & Sommer, 1966) and/or more
subtle behavioral compensation, such as leaning
or turning away from the confederate (Patterson
et al., 1971; see Figure 8.2).
When options of rapid departure or behavioral
compensation are limited, such as in some seating
arrangements on public transport, the close pres-
ence of others can precipitate stress, reflected in
enhanced cortisol levels and increased negative
mood (Evans & Wener, 2007). Under such circum-
stances, many people seem to prefer standing over
FIGURE 8.2.  Illustration of two unfocused interactions on a public park bench. Note how the spatial invasion by
the man as portrayed in Image B compared to Image A elicits a behavioral compensation, including leaning and
turning away by the woman.
The Physical Environment and Nonverbal Communication
bearing the close presence of strangers (e.g., when
faced with the option of taking a middle seat in a
three-seat arrangement; McGeehan, 2005).
THE POWER OF ENVIRONMENTAL
FEATURES
In addition to the broad effects of behavioral settings
on unfocused interactions, considerable research
examines how the presence or absence of specific
environmental features influences nonverbal behav-
ior. Environmental features are typically thought
of as physical elements that can be shared by dif-
ferent behavioral settings. The lighting conditions
of a doctor’s and a manager’s office, for instance,
may be identical, despite the fact that both settings
facilitate rather different social events. Although
one can think of numerous physical elements of
social relevance (cf. Moos, Harris, & Schonborn,
1969; Rohles, 1967; Vielhauer, 1965), this section
focuses specifically on factors that have attracted
repeated empirical scrutiny. They include lighting,
color, temperature, noise, and smell conditions;
however, they also include the presence of certain
architectural elements, objects, and personal belong-
ings, including mobile devices. Given that these ele-
ments rarely occur in isolation, behavioral settings
frequently share some but not all physical elements.
195
Patterson and Quadflieg
As a result, many behavioral settings are character-
ized by a unique combination of physical elements.
Think, for example, of the systematic changes in
smell, sound, temperature, lighting, spaciousness,
and mobile device use in a doctor’s office compared
to a football stadium. Despite such differences in
patterns of environmental features, existing studies
have largely examined the social consequences of a
single environmental variable at a time.
Whereas some of the existing work investigated
the impact of environmental features on well-
circumscribed aspects of nonverbal behavior (i.e.,
the interpretation of facial expressions, interper-
sonal distance, mutual gaze, gesture use), other
work has focused on consequences for broader
social mindsets, such as interpersonal attraction,
trust, or aggression. The latter scientific approach
assumes that certain mindsets trigger specific pat-
terns of nonverbal behavior (e.g., interpersonal
attraction should heighten nonverbal involvement).
Thus, it is worth considering both sets of studies in
concert, especially as both frequently adopt a zero
acquaintance paradigm (cf. Ambady, Hallahan, &
Rosenthal, 1995; Norman & Goldberg, 1966). In
other words, they investigate the impact of envi-
ronmental features on the evaluation of or response
toward unfamiliar others without the occurrence of
any verbal exchange. In combination, these studies
FIGURE 8.3.  Images illustrating the power of environmental features on behavioral settings. Note how dim lighting
as portrayed in Image B compared to bright lighting in Image A fosters an atmosphere of intimacy and impoverishes
a perceiver’s ability to encode highly detailed visual information.
196
offer important insights regarding the environmen-
tal modulation of nonverbal communication.
Lighting
Throughout history, humans have aimed to increase
control over their environmental lighting conditions
(Bowers, 1998). Due to numerous inventions and
technological advances, most localized lighting con-
ditions are now adjustable to people’s preferences
(see Figure 8.3).
Early studies on the effects of lighting on human
encounters revealed that dimmer lighting can foster
an atmosphere of intimacy (Carr & Dabbs, 1974;
Lecomte, Bernstein, & Dumont, 1981). In circum-
stances where such an atmosphere seems inappro-
priate (e.g., when interacting with strangers), people
tend to compensate for dim lighting conditions by
keeping a greater physical distance from others
(L. Adams & Zuckerman, 1991) and/or reducing the
amount of mutual eye gaze (Carr & Dabbs, 1974).
In many societies around the globe, however,
dim lights are specifically utilized in settings to
decrease social inhibition, such as in bars, at parties,
and in the bedroom (see Page & Moss, 1976). In
line with their usage, experimental studies provide
evidence that dimmer lighting can loosen social
inhibition and elicit feelings of anonymity (Hirsh,
Galinsky, & Zhong, 2011; Page & Moss, 1976).

Dim lighting, for instance, enhances the likelihood
of antisocial behavior, such as aggression or dishon-
esty toward strangers (Page & Moss, 1976; Zhong,
Bohns, & Gino, 2010). Alternatively, it also fosters
prosocial behavior that requires overcoming social
reservations, such as mutual touch (Gergen,
Gergen, & Barton, 1973), the readiness to disclose
personal information (Miwa & Hanyu, 2006), and
the willingness to collaborate (Baron, Rea, &
Daniels, 1992; Steidle, Hanke, & Werth, 2013).
In turn, simply thinking of a deed from one’s
past that involved social disinhibition can influence
experiences of darkness. In one study, individuals
instructed to recall a previously committed antisocial
deed judged an experimental room to be darker than
those instructed to recall a prosocial deed (Banerjee,
Chatterjee, & Sinha, 2012). Although a perceptual
explanation for this finding has been ruled out (i.e.,
the room does not actually look darker but is expe-
rienced as such; see Firestone & Scholl, 2014), such
data suggest an interesting bidirectional link between
aspects of social disinhibition and experiences of
darkness that requires further investigation.
Studies on the effects of anonymity more gener-
ally indicate that individuals who feel anonymous
rely more strongly on salient contextual cues and
display an increased conformity to situation-specific
norms (for a meta-analysis, see Postmes & Spears,
1998). In other words, the normative power of
behavioral settings as described in the previous
section of this chapter may be enhanced under con-
ditions of dim lighting. Finally, dim lighting condi-
tions also impoverish a perceiver’s ability to encode
highly detailed visual information about others
(Zhong & House, 2012). Perceptual difficulties that
obscure the detection and identification of individu-
als significantly influence the course and outcome of
the impression formation process. Under conditions
of stimulus degradation and impoverished viewing,
in particular, the perception of others as random
instances of social categories rather than as unique
individuals is increased (Cloutier & Macrae, 2007;
Cloutier, Mason, & Macrae, 2005; Macrae, Quinn,
Mason, & Quadflieg, 2005).
Perceiving others in terms of their sex, age,
and race, for instance, can encourage the use of
stereotypes and prejudice. As a result, dim lighting
The Physical Environment and Nonverbal Communication
conditions might strengthen well-known social
biases, such as interpreting identical nonverbal
behavior (i.e., an ambiguous shove) more positively
when initiated by an ingroup rather than an outgroup
member (Duncan, 1976). Indeed, initial evidence
suggests that under conditions of ambient darkness,
negative stereotypes toward outgroup members are
more easily triggered, in particular for individuals
who chronically worry about the world being a
dangerous place (Schaller, Park, & Mueller, 2003).
Although available research has pointed to some
interesting effects of lighting on nonverbal commu-
nication, several important questions have remained
unanswered. For instance, do the observed effects
differ as a function of artificial versus natural light?
Do circadian rhythms play a role in the effects of
lighting on nonverbal behavior? To what extent
does prolonged exposure and adaptation to particu-
lar lighting conditions, compared to the short-
term exposure in laboratory experiments, modify
nonverbal behavior?
Temperature
Another prominent feature of any physical envi-
ronment is temperature. In the 1970s, when social
psychologists began examining the effects of
temperature on social encounters, they typically
conceptualized temperature as a thermal stressor.
In particular, they studied the effects of hot tem-
peratures (>30 °C [86 °F]) on interpersonal attrac-
tion and aggression. These early studies suggested
that heat reduces people’s attraction to one another
(Griffitt, 1970; Griffitt & Veitch, 1971; but for
counterevidence, see Bell, Garnand, & Heath, 1984)
and fosters interpersonal aggression (but for an
ongoing debate, see Bell, 2005; Bushman, Wang, &
Anderson, 2005).
In contrast to these early studies, contemporary
researchers interested in the impact of temperature
on human encounters largely study nonextreme
temperature effects (i.e., the effects of cold tempera-
tures ranging from 15 to 19 °C [59 to 66 °F], and
the effects of warm temperatures ranging from 21 to
25 °C [70 to 77 °F]). Importantly, the dimension of
warmth (rather than mere heat) is relevant not only
when it comes to assessing the physical world but
also during psychological evaluations. Decades of
197
Patterson and Quadflieg

social psychological research suggest that the
assessment of warmth plays a pivotal role when
forming impressions of individuals, couples, and
social groups (Asch, 1946; Fiske, Cuddy, & Glick,
2007; Simpson, Rholes, & Phillips, 1996). By estab-
lishing whether a person is warm or cold, perceivers
frequently also infer whether the person is generous,
sociable, and popular (Asch, 1946).
The application of warm and cold as physical
and psychological referents across many cultures
may originate from universal early childhood expe-
riences. As caregivers provide infants with physi-
cal warmth (i.e., as they hold and cuddle an infant
and thus expose it to experiencing the warmth of a
human body) and psychological intimacy (i.e., as
they engage the child in a loving and trusting rela-
tionship), powerful associations between physical
and psychological warmth are forged (cf. Beckes &
Coan, 2011; IJzerman & Semin, 2010; L. E.
Williams & Bargh, 2008). These associations are
frequently reflected in linguistic expressions—such
as when we speak about a “cold-hearted” person or
a “lukewarm” date. By adulthood, the link between
physical and psychological warmth seems to foster
functional interchangeability. Put differently, adults’
experiences of physical warmth and coldness can
impact their feelings toward and thoughts about
others. Briefly holding a cup of hot, rather than cold,
coffee, for instance, tempts individuals to rate a
stranger as warmer and friendlier (L. E. Williams &
Bargh, 2008). Similarly, holding a warm, rather than
a cold, cup (IJzerman & Semin, 2009) or working
in a warm, rather than a cold, room (Steinmetz &
Mussweiler, 2011) increases perceived similarity
with a comparison person. Unfortunately, to date,
the impact of physical experiences of different
temperatures on psychological evaluations has
largely been shown with regard to hypothetical
others. In other words, it remains uncertain to what
extent subtle physical temperature effects suffice to
bias social impression during real-world encounters
(see Figure 8.4).

FIGURE 8.4.  Image illustrating the power of temperature cues on social

encounters. Will the exchange between these two strangers differ depending on
whether the woman’s cup contains a hot or a cold beverage (cf. L. E. Williams &
Bargh, 2008)?

198

Based on existing work, however, temperature-
based increases in perceived social warmth and
similarity are expected to enhance psychological
attraction and to foster verbal and nonverbal har-
mony as well as the probability of friendship
(Byrne, 1971). In support of this claim, participants
asked to briefly touch a warm therapeutic tempera-
ture pack show more trust toward a stranger in a
subsequent nonverbal online trust game (Kang,
Williams, Clark, Gray, & Bargh, 2011) and report
less loneliness (Bargh & Shalev, 2012) than par-
ticipants asked to touch a cold pack. There is even
evidence that merely imagining a specific tempera-
ture-related experience from a first-person perspec-
tive suffices to produce changes in social evaluation.
Thus, participants who simply imagined holding a
cold cup while waiting for a friend rated a hypothet-
ical person as less warm than participants asked
to imagine holding a warm cup or no cup at all
(Macrae, Raj, Best, Christian, & Miles, 2013).
It also appears that the causal link between tem-
perature experiences and social connectedness may
be bidirectional. For example, participants asked to
stand close to two strangers produced higher esti-
mates of room temperature than participants asked
to stand farther apart from the others, even though
no actual change in temperature was measurable
across these two conditions (IJzerman & Semin,
2010). Similarly, participants asked to recall a social
inclusion experience gave higher estimates of room
temperature than participants recalling an exclusion
experience (Zhong & Leonardelli, 2008). In addition,
participants excluded during a nonverbal ball-tossing
game reported a greater subsequent desire for warm
food and drink than participants who were included
(Zhong & Leonardelli, 2008). Notably, the latter
experiment suggests that experiencing social cold-
ness may elicit attempts to create physical warmth
experiences (such as longing for and drinking a hot
beverage) to compensate for the negative experience.
Further evidence for compensatory mechanisms
between physical and psychological warmth was
found in subsequent studies. In a scenario study
that required participants to adopt the mindset of a
service employee, participants asked to work in cold
rooms showed warmer behavior toward customers
(i.e., they reported a higher willingness to help and
The Physical Environment and Nonverbal Communication
were willing to give higher discounts) than partici-
pants asked to work in a warm room, possibly due
to a higher motivation to establish warm relation-
ships (Kolb, Gockel, & Werth, 2012). Along similar
lines, participants asked to recall a social exclusion
experience and then hold a cold or a warm tempera-
ture pack differed in subsequent reports of their
need to affiliate. Participants in the exclusion-
memory-followed-by-warm-pack condition reported
a lower need to affiliate than participants in the
exclusion-memory-followed-by-cold-pack condition
and also than participants in an additional control
condition (neutral-memory-followed-by-warm-pack;
Bargh & Shalev, 2012). The authors suggested that
the experience of physical warmth substituted for
the social needs elicited by the exclusion memory.
This compensation is particularly fascinating, given
that people seem to have no explicit insight into the
functional relationship between temperature- and
community-based feelings of warmness and cold-
ness (Bargh & Shalev, 2012). Future research will
need to determine through which exact mechanism
assessments of physical and psychological warmth
may be linked. Whereas the observed overlap may
arise from mere semantic associations (i.e., holding
a warm cup may trigger the concept of warmth and
its associated psychological meanings), the idea of
basic shared experiential states (i.e., holding a warm
cup might elicit physiological changes similar to
those triggered by the experience of interpersonal
warmth) offers an alternative explanation.
Odor
It is hardly surprising that a stranger’s perfume or
body odor can influence whether people find a person
attractive and/or are inclined to approach the indi-
vidual (e.g., Guéguen, 2001; Hold & Schleidt, 1977;
Roberts et al., 2011; see also Chapter 21, this hand-
book). Further evidence suggests, however, that even
person-unrelated, environmental scents influence pat-
terns of nonverbal communication between strangers.
For example, reading emotional expressions from
faces can be facilitated by concurrent incidental olfac-
tory cues. Specifically, facial expressions of disgust are
more rapidly and accurately detected when accompa-
nied by an unpleasant (but also by a pleasant) smell
relative to a neutral smell (Seubert et al., 2010).
199
Patterson and Quadflieg
Furthermore, in the presence of pleasant odors
(such as the smell of pastries), strangers are more
likely to offer each other help or to accept another’s
help than when surrounded by a neutral smell
(Baron, 1997; Baron & Thomley, 1994; Guéguen,
2012). Additionally, pleasant smells indicative of
cleanness, such as the smell of citrus-scented room
cleaner, promote reciprocity in strangers (as mea-
sured in a nonverbal financial trust game) and chari-
table intentions to donate money and to volunteer
(Liljenquist, Zhong, & Galinsky, 2010). Intrigu-
ingly, although effects of pleasant scents increasing
reciprocity and positive intentions are consistent
with the well-established effect of positive affect
increasing nonverbal reciprocity (Patterson, 1976),
existing research suggests that the effects of positive
odors unfold even if participants do not report dif-
ferences in mood across different scent conditions
(cf. Study 2, Liljenquist et al., 2010).
Although favorable environmental smells enhance
sociability in nonverbal exchanges, negative smells
frequently interfere with prosocial intentions (but
see Bonini et al., 2011). For example, in the presence
of negative, compared to positive or neutral, environ-
mental smells, perceivers find strangers less attractive
(Demattè, Österbauer, & Spence, 2007). Further-
more, exposure to unpleasant fishy smells enhances
suspicion toward others. That is, native English
speakers (frequently exposed to the expression that
something “smells fishy”) who were seated in an area
sprayed with fish oil were less trusting of an unfamil-
iar partner than comparable others who were seated
in an area sprayed with either a nonfishy negative or
a neutral smell (S. W. S. Lee & Schwarz, 2012).
The same study revealed that socially awkward and
suspicious behavior (such as a confederate hiding a
piece of paper from a participant) enhanced a per-
son’s ability to correctly label fishy smells, suggesting
that interpersonal suspicion and fishy smells mutu-
ally facilitate each other’s cognitive accessibility.
Additional evidence suggests that disgusting
smells can induce feelings of antipathy toward
social outgroups (such as gay men) that engage in
behaviors considered by some as immoral or impure
(E. Cunningham, Forestell, & Dickter, 2013; Inbar,
Pizarro, & Bloom, 2012). Finally, obnoxious smells
can induce negative affect and can enhance a person’s
200
motivation to escape from a social situation (Asmus &
Bell, 1999). Thus, odor-induced emotional and/or
cognitive states (such as comfort, disgust, or suspicion)
can become powerful determinants of social perception
and behavior (cf. Aglioti & Pazzaglia, 2011).
Future research will have to clarify the extent
to which incidental olfactory effects are sufficiently
strong to influence traditional markers of nonverbal
exchange, such as interpersonal distance preferences,
gaze patterns, and facial expressions. Similarly unre-
solved is the role of gender differences on olfactory
effects. Women’s sense of smell is known to vary
through their menstrual cycle (e.g., Gangestad &
Thornhill, 1998; Hummel, Gollisch, Wildt, &
Kobal, 1991; Rikowski & Grammer, 1999) and
tends to be more sensitive than men’s sense of smell
(for a review, see Doty & Cameron, 2009). Do such
smell sensitivity differences systematically affect
the impact of environmental olfactory cues on
the nonverbal behavior (see also Chapter 14, this
handbook) of women and men?
Sounds
In the late 1960s and early 1970s, social psychologists
began paying close attention to the effects of environ-
mental sound levels on interpersonal behavior (for a
review, see Jones & Chapman, 1984). Originally, this
work aimed to elucidate the impact of urban stressors
on social interactions. Its primary focus lay in exam-
ining how different types and levels of background
sounds influenced people’s behavior toward others,
especially their willingness to help strangers (cf.
Milgram, 1970). Several early studies indicated that
people’s willingness to assist others decreases as noise
intensity increases (e.g., Boles & Hayward, 1978;
Page, 1977). One favored explanation for this phe-
nomenon was that a “desire to escape” the situation
may motivate decreased helping, but data collected by
Mathews and Canon (1975) challenged this account.
Instead, further evidence suggested that noise can
result in cognitive overload that substantially inter-
feres with people’s capacity to process social cues
(e.g., Cohen & Lezak, 1977; Moser, 1988) and thus
reduces their ability to notice when help is needed.
Additional research on the social consequences
of environmental sounds revealed that noise
enhanc­­es aggression, especially in angered individuals
 The Physical Environment and Nonverbal Communication

(e.g., Donnerstein & Wilson, 1976; Konecni, 1975).
For example, participants given the opportunity to
shock a confederate who had previously upset them
administered more shocks when they were exposed
to loud tones in the experiment room (Konecni,
1975). Noise did not increase aggression, however,
in individuals not previously angered by the
confederate, suggesting that loud noise itself is not
sufficient to induce interpersonal aggression.
Circumstantial evidence further suggests that
noisy environments affect the extent to which
humans rely on nonverbal communication
(Jones, Chapman, & Auburn, 1981). The impor-
tance of nonverbal signals increases especially when
noise interferes with verbal exchange. For instance,
people seem to enhance their use of gestures, pos-
ture, and facial expression when communicating
in noisy environments (Kryter, 1970). External
noise also encourages seated, as well as standing,
individuals to seek closer interpersonal distance
(Lloyd, Coates, Knopp, Oram, & Rowbotham, 2009;
Mathews, Canon, & Alexander, 1974), apparently to
compensate for speech masking. The quality of noise,
however, might determine whether interpersonal
distance preferences during noisy encounters shrink
or expand. For example, when external sounds are
experienced as negative and threatening, individuals
may choose to increase, rather than decrease, inter-
personal distance from unfamiliar others (Tajadura-
Jiménez, Pantelidou, Rebacz, Västfjäll, & Tsakiris,
2011). Other factors that are likely to moderate the
impact of environmental sounds on nonverbal com-
munication include perceived noise predictability
and controllability as well as personality differences
(Jones & Chapman, 1984; Weinstein, 1976; for a
recent meta-analysis, see Szalma & Hancock, 2011).
For example, introverted individuals seem more eas-
ily affected by environmental noise than extraverts
(Davies & Hockey, 1966; Elliott, 1971).
Architectural Elements
Both permanent (e.g., walls, columns, partitions)
and/or movable (furniture or plants) architectural
elements are frequently utilized in everyday life to
regulate social interactions (E. T. Hall, 1966; Manzo,
2005; Mehta, 2009; Robson, 2008). Collectively
termed as anchors, they are design features that
can facilitate and support, but also interfere with,
interpersonal exchange (Robson, 2008; Sommer &
Ross, 1958; A. G. White, 1953). In public spaces,
for example, seating options are important features
allowing people to stop and engage in prolonged
social encounters (Mehta, 2009; Whyte, 1980;
see Figure 8.5).

FIGURE 8.5.  Images illustrating the power of architectural elements on social encounters. Note how the absence (Image
A) and presence (Image B) of seating opportunities affect people’s occupancy of public spaces. From “Look Closely
and You Will See, Listen Carefully and You Will Hear: Urban Design and Social Interaction on Streets,” by V. Mehta,
2009, Journal of Urban Design, 14, pp. 46, 49. Copyright 2009 by the Taylor & Francis Group. Reprinted with permission.

201
Patterson and Quadflieg
In contrast, architectural elements, such as
walls or barriers, can limit spatial access to a person
and provide temporary screening from the sight,
sound, smell, and/or touch of others. Obviously,
such screening has important consequences for the
exchange of nonverbal, as well as verbal, signals
between people. For example, blocking the visibility
of a partner with a barrier prevents gaze exchange
and interferes with both information transfer and
turn taking (Boyle, Anderson, & Newlands, 1994).
However, even the presence of architectural
elements that do not directly separate individu-
als affects patterns of nonverbal behavior. Spatial
proximity toward strangers, for instance, produces
more discomfort in a space confined by walls (i.e., a
vacant room) than in an unconfined space (i.e., an
empty soccer field; Cochran, Hale, & Hissam, 1984).
This effect, however, may differ depending on a per-
son’s cultural background. An early observational
study, conducted in the United States, revealed that
European Americans kept a similar distance toward
a partner in both indoor and outdoor interactions,
whereas Latin Americans clustered more closely
outdoors than indoors, and African Americans were
closer indoors than outdoors (Baxter, 1970). Thus,
the impact of architectural elements on nonverbal
behavior is likely affected by people’s cultural
background (see also E. T. Hall, 1966).
Further research suggests that encounters with
others considered a social threat (such as members
of a social outgroup) progress differently depending
on whether they occur in confined or nonconfined
spaces. Although interracial encounters in a confined
space, such as a small booth, enhance the accessibil-
ity of aggression-related concepts (such as attack or
punch), similar encounters elicit heightened accessibil-
ity of withdrawal concepts (such as flee and run) when
occurring in an open field (Cesario, Plaks, Hagiwara,
Navarrete, & Higgins, 2010). Thus, architectural ele-
ments play an important role in determining a place’s
sense of confinement (Stamps, 2010) and in high-
lighting different behavioral options in a given setting
(Baum & Valins, 1979; Macintyre & Homel, 1997).
Objects and Personal Belongings
Whether it is a dorm room covered in laundry, a
gym displaying boxing posters, or a hotel reception
202
decorated with flowers, many environmental set-
tings display various kinds of small-scale objects
and/or personal belongings serving as powerful
social signals (Campbell, 1979; Gosling, Craik,
Martin, & Pryor, 2005a, 2005b; Wollin &
Montagne, 1981). According to Buss (1987),
occupants influence the belongings and objects in
their environments in three major ways. First, they
may select certain objects that they like to display
(e.g., a doctor putting his certificates on the office
wall). Second, they may also manipulate these items
in a personal manner (e.g., a student decorating his
or her wardrobe with stickers). Finally, they may
unintentionally evocate (i.e., provoke, elicit) changes
in the objects of their environments (e.g., a teacher
staining his or her books with coffee due to clumsi-
ness). Based on these three processes, environmen-
tal objects might signal (a) an occupant’s attempt
to influence how he or she feels/thinks in a specific
setting, (b) an occupant’s effort to broadcast infor-
mation about him- or herself or an entity associated
with him- or herself (e.g., a company), or (c) an
occupant’s inadvertent traces of items utilized as he
or she accomplished a certain task (cf. Gosling,
Gaddis, & Vazire, 2008; Webb, Campbell,
Schwartz, & Sechrest, 1966).
Perceivers frequently rely on environmental
traces, such as objects and personal belongings,
to speculate about the social attributes of a set-
ting’s occupants. Specifically, they tend to infer the
behavior or the disposition that created the physical
evidence (Harris & Sachau, 2005). A room covered
in laundry, for instance, could either signal that a
person had been too busy to clean the room or that
a person is messy. Perceivers also rely on the brands
of environmental objects to inform their opinion
(Aaker, 1997). Though different observers tend
to form rather consensual impressions based on
environmental objects, the accuracy of their social
judgments typically depends on the type of infer-
ence drawn. For example, observers tend to be more
accurate when judging occupants’ conscientiousness
than agreeableness and when the setting to
be judged is a private rather than a public space
(Gosling, Ko, Mannarelli, & Morris, 2002).
However, how can the presence or absence of
objects and personal belongings impact face-to-face

interactions? First, individuals utilize objects and
personal belongings to influence patterns of non-
verbal exchange. A study conducted at a library,
for example, provided evidence of systematic gen-
der differences in responding to spatial intrusions.
Although females were more affected by an immedi-
ately adjacent intruder, males were more affected by
an intruder directly opposite them (Fisher & Byrne,
1975). The same study also revealed that, as a result,
females were more likely to put their books and per-
sonal items adjacent to them, whereas males were
more likely to put their books and materials directly
opposite them. Thus, both genders manipulated
environmental objects in a manner to protect their
particularly vulnerable personal space.
Second, and maybe less obvious, certain types of
objects and personal belongings can convey rather
strong nonverbal messages to specific groups of
people. The presence of stereotypical masculine
objects (e.g., football posters and video game boxes)
compared to gender neutral items (e.g., nature
poster, water bottles) in a computer science class-
room, for instance, can signal women that they do
not belong—neither in the classroom nor in the dis-
cipline (Cheryan, Meltzoff, & Kim, 2011; Cheryan,
Plaut, Davies, & Steele, 2009). Along similar lines,
the lack of racial diversity cues in U.S. health care set-
tings (e.g., a lack of a racially diverse health providers
and/or a lack of decorative artwork that signals the
appreciation of racial diversity) may prompt minority
patients to disengage from medical treatment
(Burgess, Warren, Phelan, Dovidio, & van Ryn, 2010).
Thus, environmental objects can delineate physical
environments based on pivotal social categories and
turn into gatekeepers that make some members of
society shy away from attending certain settings.
Third, individuals who try to enter, or are
required to enter, unreceptive environments are
likely to change their nonverbal communication
due to their psychological conundrum. Specifically,
environment-related increases in anxiety and vigi-
lance (Spencer, Steele, & Quinn, 1999), physiologi-
cal arousal (Blascovich, Spencer, Quinn, & Steele,
2001), and negative thinking (Cadinu, Maass,
Rosabianca, & Kiesner, 2005) may decrease a per-
son’s ability to send unambiguous nonverbal (and
verbal) signals and to interpret adequately such
The Physical Environment and Nonverbal Communication
signals from others (Burgess et al., 2010; Trawalter,
Richeson, & Shelton, 2009). In addition, affected
individuals may engage in nonverbal behavior that
minimizes their degree of social involvement in
these settings. Ethnic and religious minorities, for
instance, were found across several U.S. colleges to
select seating options peripheral to a classroom’s
central zone of participation (R. S. Adams & Biddle,
1970; Haber, 1982).
Color
Though the meaning and consequences of color
cues are popular topics in folk psychology, reliable
empirical insights into the affective and behavioral
consequences of color in one’s surroundings are
sparse (Elliot & Maier, 2007). Scientific investiga-
tions generally suggest that the meaning of certain
colors is either determined by evolutionary mecha-
nisms (e.g., red is associated with avoidance; Elliot,
Maier, Moller, Friedman, & Meinhardt, 2007) or
through culture-specific learning (e.g., pink signal-
ing femininity in the West; S. J. Cunningham &
Macrae, 2011). Despite considering both paths of
meaning acquisition, relatively little is known about
how color affects psychological functioning.
Numerous studies suggest that the presence
of specific colors in people’s environments may
influence their mood, arousal, physical strength,
creativity, and performance (Küller, Mikellides, &
Janssens, 2009; Stone & English, 1998; Valdez &
Mehrabian, 1994). These results, however, are fre-
quently inconsistent and contradictory. For exam-
ple, pink fosters tranquility (Pellegrini, Schauss, &
Miller, 1981; Profusek & Rainey, 1987; Schauss,
1985; but see Gilliam & Unruh, 1988), blue pro-
motes negative mood (e.g., Hamid & Newport,
1989; but see Stone, 2001), green facilitates creativ-
ity (Lichtenfeld, Elliot, Maier, & Pekrun, 2012), and
red impairs performance (Elliot et al., 2007; Stone,
2001). Additional work suggests, however, that
color effects frequently depend on the simultaneous
presence of contrasting colors (e.g., the positivity of
white only emerges in the presence of black; Lakens,
Semin, & Foroni, 2012; Meier, Robinson, & Clore,
2004). Alternatively, color effects may be culture
dependent (Ishii, Miyamoto, Rule, & Toriyama,
2014; Lakens, 2011).
203
Patterson and Quadflieg
Research elucidating the impact of colors on
nonverbal communication is particularly scarce.
Initial work suggests that the color of personal
belongings, such as clothing, can impact person
impressions. Individuals wearing black, for instance,
are perceived as particularly aggressive, at least in
sport contexts (Frank & Gilovich, 1988). In addi-
tion, women dressed in red (rather than green or
blue) seem to elicit heightened attraction. Specifi-
cally, male perceivers sit closer to them and ask
them more intimate questions (Niesta Kayser,
Elliot, & Feltman, 2010). In a similar way, women
perceive men dressed in red as more attractive and
as sexually more desirable (Elliot et al., 2010). It is
less certain, however, to what extent environmental
colors, such as wall, furniture, and/or object colors,
can influence patterns of nonverbal communication
between strangers. Initial data suggest that merely
standing in front of a red background (rather than
dressing in red) may suffice to trigger attraction
in others (Elliot et al., 2010; Schwarz & Singer,
2013). In addition, using red objects in competitive
interactions (e.g., red rather than blue chips in a
poker game) are associated with enhanced impres-
sions of a competitor’s dominance and perceiver’s
withdrawal (Ten Velden, Baas, Shalvi, Preenen, &
De Dreu, 2012). Future work is required to deter-
mine whether additional color effects on nonverbal
behavior exist, and if so, under what conditions they
may occur.
Mobile Technology
The prevalence of digital communication technol-
ogy in modern life introduces a new and different
kind of environmental feature profoundly affecting
face-to-face encounters (Bugeja, 2005). The impact
of digital communication on nondigital social inter-
actions regularly attracts debates, especially outside
academic circles (see, e.g., Bilton, 2013). Only a
few systematic studies have begun to focus on the
effects of mobile devices on interaction patterns in
public spaces. For example, Patterson, Lammers,
and Tubbs (2014) found that when a confederate
passing on the sidewalk initiated a look, smile, and
greeting, pedestrians using mobile devices smiled
less than controls did. This effect was present
even when the mobile-device users glanced at the
204
confederate. That is, the lower frequency of smiles
from mobile-device users was not the product of
not noticing the confederate while they focused
on their mobile devices. Usually, the simple and
rapid perception of others’ behavior is sufficient to
trigger reciprocity or behavioral mimicry without
conscious awareness, especially with a behavior
as simple as a smile (Chartrand & Lakin, 2013).
Nevertheless, the cognitive demand from mobile
device use seems to disrupt the otherwise automatic
perception–behavior link, leading to fewer smiles.
Additional studies have examined how listen-
ing to music through headphones impacts nonver-
bal exchanges between strangers. Although some
research suggests that wearers of headphones keep
a larger interpersonal distance from others (Lloyd
et al., 2009), other work indicates that such effects
may be music dependent. Specifically, individu-
als wearing headphones and listening to positively
rated instrumental music tolerated closer interper-
sonal distances than those listening to negatively
rated instrumental music (Tajadura-Jiménez et al.,
2011). Most important, initial insights on the non-
verbal consequences of mobile device use indicate
that electronic gadgets can turn into powerful
means for privacy regulation in crowded environ-
ments (e.g., on public transport in urban settings).
For example, the discomfort precipitated by the
close presence and/or looks of unfamiliar others
in a densely packed train or bus might be reduced
by choosing to listen to one’s preferred music via
headphones. Alternatively, focusing on one’s mobile
device to text friends, updating one’s Facebook
status, or playing distracting games may aid in pri-
vacy regulation. However, such devices may reduce
sensitivity to nearby others and adversely affect the
quality of face-to-face interactions (Bilton, 2013;
Bugeja, 2005).
THE POWER OF SOCIAL GOALS
The discussion thus far has highlighted both the
broad effects of settings on nonverbal commu-
nication and the more specific environmental-
feature effects. To appreciate more clearly these
environment–behavior links, the role of social goals
also requires attention. Social goals determine the

sending and receiving of nonverbal signals and,
thus, shape the dynamic relationship between
the environment and nonverbal communication.
Although social goals may be elicited in a deliber-
ate manner (i.e., an individual decides to act on a
certain need), they can also be activated without
conscious awareness (Bargh, 1997). Finally, social
goals vary in their specificity. The two metagoals of
efficiency and appropriateness, for example, struc-
ture many interpersonal encounters across various
environmental settings (Berger, 1997). Additional
goals concerning issues of belonging, understand-
ing, controlling, and enhancing the self modulate
the environment–behavior link in a more setting-
specific manner (Fiske & Taylor, 2013). Finally,
privacy goals have attracted repeated empirical
attention and change frequently across settings and
time (Altman, 1976). To understand the conse-
quences of these diverse sets of social goals on the
environment–behavior link in more detail, in the
following sections, we discuss their influence on
nonverbal behavior.
The Regulation of Efficiency and
Appropriateness
Social psychologists frequently refer to humans as
“cognitive misers” (cf. Fiske & Taylor, 1984). That
is, perceivers generally prefer to minimize mental
effort when judging and/or responding to others,
devoting cognitive resources reluctantly and only
when clearly necessary to do so (Bargh & Chartrand,
1999; Macrae, Milne, & Bodenhausen, 1994).
Practically, this means that the need for efficiency
can bias individuals to respond rather rapidly and
heuristically, both in their social judgments of oth-
ers and in initiating their own nonverbal behavior.
In particular, in social situations that are novel or
ambiguous, a high need for efficiency can strengthen
environment–behavior links. Under such circum-
stances, environmental cues may prompt quick or
habitual judgments and behavior without conscious
awareness or cognitive scrutiny (Bargh, Gollwitzer, &
Oettingen, 2010). A series of studies showed, for
instance, that exposure to objects common to the
domain of business (e.g., boardroom tables and
briefcases) increased the cognitive accessibility of
competition and enhanced people’s inclination to
The Physical Environment and Nonverbal Communication
perceive socially ambiguous behavior of others as
competitive and to act in a competitive manner
themselves (Kay, Wheeler, Bargh, & Ross, 2004).
In contrast to such automatic environment-
induced effects, the metagoal of appropriateness
enhances people’s deliberate attempts to comply
with environment-based social norms. Appropri-
ateness inclines people to abide by social norms
and customs in interactions to avoid calling undue
negative attention to themselves (Berger, 1997).
Obviously, the need to be appropriate is more
salient in some environmental settings than others
(e.g., at a graduation ceremony compared to at a
college party)—a difference occasionally captured
by referring to settings as either formal or informal
(Knapp et al., 2014). In addition, the availability
of environmental cues that signal what is con-
sidered appropriate social behavior varies across
settings. Upon entering a library, for instance,
perceivers may decipher quickly that approach-
ing a stranger for a chat violates setting norms.
That is, a unique combination of environmental
cues—such as shelves filled with books, isolated
tables, and silence—promotes solitary behavior.
However, some environments—such as malls,
parks, or airports—provide less obvious cues
regarding appropriateness. As a result, different
users may have rather divergent ideas about which
nonverbal behaviors are suitable in these settings,
especially with regard to displays of interpersonal
intimacy (e.g., Heslin & Boss, 1980; R. White &
Sutton, 2001).
The Regulation of Belonging,
Understanding, Controlling, and
Self-Enhancement
Decades of social psychological research suggests
that humans frequently structure social interactions
to fulfill a seemingly universal set of social needs.
For example, people generally form and maintain
close bonds with others (Baumeister & Leary,
1995). They also strive to establish a socially shared
understanding of events and engage in information
seeking when mutual understanding fails (Kelley,
1972; Sinclair, Lowery, Hardin, & Colangelo, 2005).
Furthermore, people attempt to control the contin-
gencies between their actions and the outcomes of
205
Patterson and Quadflieg
these actions, in particular in social relationships
that are characterized by interdependency. Finally,
they aim to think of themselves in a positive light,
especially when their sense of self-worth has been
questioned or threatened (Beauregard & Dunning,
1998; Krueger, 1998; Sedikides, 1993).
Importantly, these social goals fundamentally
shape the sending and receiving of nonverbal sig-
nals. A high need to affiliate, for instance, enhances
individuals’ sensitivity to nonverbal cues, such as
vocal tone and facial expression (Bernstein, Sacco,
Brown, Young, & Claypool, 2010; Pickett, Gardner, &
Knowles, 2004), and increases mimicry of partners’
nonverbal behavior (Lakin & Chartrand, 2003;
Lumsden, Miles, Richardson, Smith, & Macrae,
2012). Furthermore, feelings of being in control can
foster social approach behavior (Smith & Bargh,
2008). Of course, individuals can strive for multiple
goals in a particular setting, for example, simultane-
ously wanting to be liked by a partner but also try-
ing to influence the partner to a particular course of
action. Different settings and different environmental
features affect the frequency and strength of social
goals and their corresponding nonverbal behavior.
Cold rooms, for instance, seem to enhance people’s
need to belong (Kolb et al., 2012), whereas fishy
smells make people question the degree of mutual
understanding (S. W. S. Lee & Schwarz, 2012).
In a similar way, work settings typically enhance
the saliency of social control and self-enhancement
goals, whereas leisure settings reinforce the needs of
belonging and mutual understanding.
Beyond settings triggering social goals, however,
individuals frequently bring social goals of varying
strength and saliency to environmental settings.
On the one hand, the extent to which a social goal
is preactivated in a given situation increases an
individual’s sensitivity to goal-relevant stimuli in
the environment (Custers & Aarts, 2003). On the
other hand, the strength of a preactivated social goal
may modulate whether nonverbal behaviors serving
the activated goal are susceptible to environmental
influences. For example, a strong need to establish
a close bond with another individual might trigger
intimate nonverbal behavior, such as close per-
sonal distance, occasional touch, and eye contact,
independent of incidental lighting or temperature
206
conditions. Nevertheless, when people select set-
tings in the pursuit of specific social goals, they are
also “selecting” the physical features that promote
those social goals. For example, a temporarily high
need to self-enhance might encourage an athletic
person to go to the gym where the facilities and
equipment facilitate favorable social comparisons
with others. The same environments, however,
might deliver rather different social incentives to
different people. As previously discussed, certain
settings may signal some people (e.g., racial minor-
ity members) that their social goals will not be met
(Burgess et al., 2010; Cheryan et al., 2009).
The Regulation of Privacy
Another pervasive, and often unconscious, metagoal
across many diverse settings is the regulation of pri-
vacy. It is common to think of privacy as isolation or
separation from others. This is one aspect of privacy,
but Altman (1976) presented a different and more
complex view of the concept. Altman proposed that
privacy is a dialectic process, driven by opposing
motives of being both open and closed to contact
with others. That is, privacy may be viewed as con-
tinuum, ranging from low privacy at one end to high
privacy at the other end. The strengths of these two
motives can change with different circumstances over
time. Because privacy regulation is such a pivotal
social goal affecting environment–behavior links, the
relationship between the physical environment and
privacy regulation is discussed in more detail.
Although privacy is a pervasive issue in
all kinds of settings, private spaces—such as
homes—frequently support its regulation. In pub-
lic spaces, however, individuals often have to rely
more on their behavioral adjustments to control
the desired amount of contact with others. Many of
these encounters are brief and subtle exchanges. In
such encounters, people use their verbal and non-
verbal behavior, along with features of the physi-
cal environment, to achieve their preferred level of
privacy. For example, if someone wanted greater
privacy to finish a report, she might retreat to her
office and close the door until she was done. After
completing the report, her desired level of privacy
may be very low. Then, because she is looking for
social contact, she might leave her office for the

break room, approach others, and engage in a lively
conversation. In both circumstances, this person has
a desired level of privacy, acts on the environment,
and manages her behavior to achieve the desired pri-
vacy. In fact, privacy management in organizations
is an important factor that affects productivity and
satisfaction of employees (Kupritz, 2000).
Of course, achieving a desired level of low
or high privacy may not be as easy as the earlier
example suggests. According to Altman (1976), pri-
vacy regulation is a dynamic process. When people
initially are unsuccessful in achieving their desired
level of privacy, they may have to make additional
modifications to achieve their preferred level of pri-
vacy. Sometimes even changes to their environment
and adjustments in their nonverbal behavior may
not suffice to achieve the desired privacy. As the
difference between the desired level and achieved
levels of privacy increases, the intensity of negative
affect also increases. Specifically, large deficits in
desired high privacy are experienced as crowding,
whereas large deficits in desired low privacy are
experienced as loneliness (Altman, 1976).
Altman’s (1976) model incorporates the comple-
mentary dynamics of environmental features and
nonverbal behavior in regulating privacy. Consistent
with ecological psychology, people not only choose
particular settings for maximizing desired privacy
but they can also manipulate features of their chosen
environment to meet their privacy needs. In some
social settings, however, people have little control
over their immediate physical environment (see
Evans & Wener, 2007) and are more dependent on
their own nonverbal behavior to achieve their pre-
ferred levels of privacy. For example, subway trains
may be filled to capacity with passengers standing in
the aisles. In such a setting, solitary passengers typi-
cally try to avoid making eye contact with strang-
ers, looking down at the floor or up at the overhead
advertisements. They may also turn their bodies to
minimize accidental contact with others.
In contrast, when individuals are in their own
territories, they have greater control in managing
privacy. In general, the term territory refers to a geo-
graphically limited area that is owned or controlled
by an individual or group (Bell, Greene, Fisher, &
Baum, 2001). In turn, territoriality refers to the
The Physical Environment and Nonverbal Communication
behaviors and cognitions related to managing and/or
defending territories. Some locations—such as one’s
home, bedroom, or office—are considered primary ter-
ritories because they clearly belong to particular indi-
viduals and are central to their daily activities (Altman,
1975). In contrast, more temporary territories, such as
a table in a restaurant or a picnic site in the park, may
be used by a particular individual or group for a lim-
ited period time. Sometimes, the boundaries of primary
territories are very explicit with “no trespassing” signs,
fences and walls, or simply closed doors. In temporary
territories, territorial claims may take a more subtle
form of nonverbal communication. That is, when
people are temporarily absent from a location, physi-
cal markers constitute a claim on the territory and can
prevent intrusions. For example, Becker (1973) found
that library tables with more markers—and especially
personal ones, such as a coat or a notebook—were
more resistant to intrusion from newcomers than were
tables with fewer personal markers. Territories are also
important in organizing interaction and communica-
tion (Bell et al., 2001). When individuals are in their
own territories, their nonverbal behavior is typically
more relaxed and they have greater control in an inter-
action than when they are visiting in others’ territories.
Finally, the number of people in a setting affects
privacy regulation. In this regard, it is important to
distinguish between density and crowding. Whereas
density refers to the number of individuals per unit
area (i.e., a physical state), crowding refers to the
negative affect or feeling (i.e., a psychological state)
resulting from high density (Stokols, 1972). There are
several different explanations for the effects of density
on the experience of crowding, including high den-
sity creating overstimulation (Milgram, 1970), inter-
ference with other activities (Schopler & Stockdale,
1977), or reducing behavioral freedom (Sundstrom,
1978). Our focus here is on a specific approach that
emphasizes the role of nonverbal communication.
First, as density increases within a limited area, the
arousal levels of individuals also tend to increase
(Worchel & Teddie, 1976). However, arousal does
not inevitably lead to feelings of being crowded. What
is important is how that arousal is interpreted or
labeled (Patterson, 1976). For example, comparably
high levels of density in an airport security line and
at a lively party both increase arousal but result in
207
Patterson and Quadflieg
very different feelings. That is, the situational cues in
the airport security line are likely to be negative and
precipitate a feeling of crowding, whereas the situ-
ational cues at the lively party are likely to be posi-
tive and precipitate a feeling of excitement. In other
words, high density in social settings necessarily
leads to close interpersonal distances and increased
arousal. However, the labeling of the arousal varies as
a function of the circumstances, sometimes produc-
ing a feeling of crowding and other times a feeling of
excitement and fun (Worchel & Teddie, 1976).
A study by Schaeffer and Patterson (1980) found,
however, that feelings of crowding can occur even
in the absence of close spatial arrangements. In this
experiment, low versus high density, low versus high
levels of gaze from confederates in the group, and pos-
itive versus negative expectancies about other group
members were manipulated. High density did not
affect ratings of crowding and confinement, but there
was an interesting gaze by expectancy interaction on
the ratings. Specifically, recipients of high gaze in the
negative expectancy condition precipitated increased
crowding and confinement ratings compared to those
in the contrasting conditions. Consequently, the psy-
chological experience of crowding may not be simply
the product of the close physical proximity of others
but may be more generally a product of increased non-
verbal involvement (e.g., gaze, forward lean, or even
touch) from nearby others (Schaeffer & Patterson,
1980). That is, close spacing among people is only
one element of a larger effect of increased nonverbal
involvement across several behaviors that may occur
in high density environments. When the increased
nonverbal involvement from surrounding others is
experienced as crowding, individuals often make com-
pensatory nonverbal adjustments to reduce the close
presence of others (Patterson, 1976). For example, if
they cannot easily leave the setting, they might mini-
mize gaze toward others and turn their bodies away
from the closest others (Patterson, 1973).
AN ECOLOGICAL SYSTEMS MODEL OF
NONVERBAL COMMUNICATION
It is clear from our discussion that understanding
the give-and-take of nonverbal communication
requires attention to interaction settings. That is,
208
there is a dynamic and reciprocal relationship
between characteristics of individuals and their
social goals on the one hand and the features of
environmental settings on the other (Barker, 1968;
Wicker, 1979). In this section, we present a brief
overview of all the relevant components introduced
in our chapter and how they relate to each other to
form an ecological systems model of nonverbal
communication (see Figure 8.6).
As outlined at the beginning of our chapter,
nonverbal communication serves a wide variety of
social functions, including transmitting information,
regulating interaction, expressing intimacy, exercis-
ing influence, and managing impressions. Typically,
these functions serve an individual’s specific social
goals. For example, one person might express more
intimate nonverbal behavior—such as standing
closer, smiling, and increasing his/her gaze toward a
partner—when striving to establish social connect-
edness. Of course, some types of nonverbal commu-
nication might themselves alter salient social goals.
For example, a spatial intrusion from a stranger
might activate privacy protection efforts, such as a
harsh glare or turning away from the intruder. Thus,
the relationship between social goals and nonverbal
communication is a bidirectional one.
In addition, on the left side of the figure are the
person-specific dispositional factors shaping social
goals and affecting nonverbal communication.
These factors include people’s biological and cul-
tural makeup, development, gender, and personal-
ity (as discussed in detail in the second part of this
handbook). These factors determine habitual
patterns of nonverbal communication as well as the
extent to which specific social goals are generally
salient and relevant to a person. However, these
factors also influence the types of behavioral settings
and environmental conditions people habitually
choose (Wicker, 1979).
Once individuals enter particular settings, they
encounter setting-specific events and behavioral
options. Typically, these events and options differ
systematically, including whether they are public or
private settings. In both cases, however, the setting’s
environmental features and behavioral goals tend
to complement one another, a condition known
as synomorphy (Wicker, 1979). For example, close

FIGURE 8.6.  An ecological systems model of nonverbal communication.
seating options in a bar encourage social interac-
tion, whereas isolated tables in a library encourage
solitary reading or studying. Thus, setting attributes
and environmental features can support or discour-
age certain types of social goals and patterns of
nonverbal communication. Design and arrangement
features set physical limits on how people interact.
An executive’s large desk, for instance, keeps visi-
tors seated in a directly opposing chair at a consider-
able distance, and typically leads to a more formal
interaction. Thus, the visitor might sit more upright,
with a less relaxed posture, and be more constrained
in expressiveness. In contrast, if the executive
directed the visitor to the comfortable sofas around
a coffee table, a more casual interaction is possible.
In such a case, the visitor would be physically closer
to the executive and might act in a more relaxed
and expressive manner. The social consequences
of physical features typically become tangible over
time in the form of setting-specific social norms.
Of course, social goals, both conscious and
unconscious, can also drive the selection of par-
ticular settings and environmental features. That is,
individuals select particular settings to attain their
The Physical Environment and Nonverbal Communication
goals. In addition, individuals can act on their envi-
ronments and change features to facilitate particular
goals. For example, moveable chairs might get rear-
ranged, lighting and temperature changed, and the
sonic environment manipulated, all in the service of
promoting specific nonverbal exchanges and achiev-
ing particular goals (Cook, 1970; Riess & Rosenfeld,
1980). In a classroom setting, for instance, educa-
tors might seat pupils who do not get along closer to
each other, as such an arrangement promotes liking
among 10- to 12-year-old children (van den Berg,
Segers, & Cillessen, 2012). Most important, the
relationship between the physical environment and
social behavior is a reciprocal one.
In concert, the combination of self- and setting-
selection processes in the service of particular goals
and the reciprocal influence between goals and
setting features contribute to greater predictability
and stability in nonverbal communication. Thus,
goal activation in a particular setting precipitates
patterns of nonverbal communication to achieve
the social goal. In turn, the outcomes from such
exchanges determine subsequent processes. That
is, the processes in the model are iterative, with
209
Patterson and Quadflieg
feedback from outcomes affecting other elements
and generating new sequences. In the short term,
outcomes in the form of goal achievement affect
subsequent goal preferences, setting choices, and
patterns of nonverbal communication. For example,
the failure to achieve a particular goal in a specific
setting might lead to intensifying the goal pursuit,
either in the original setting or in a new one. Alter-
natively, failure to achieve a certain goal might lead
an individual to abandon the original goal for a dif-
ferent one. The likelihood of either course is at least
partly determined by the accessibility of goal-serving
settings and environmental features. Over time (i.e.,
in the longer term), outcomes can also modify per-
sonality predispositions, developmental changes,
and cultural adaptation. These changes can affect
later iterations of the dynamics in the model, includ-
ing the give-and-take in the nonverbal exchange
between partners.
CONCLUSION
This chapter provided a broad review of the dynamic
relationships between the physical environment
and nonverbal communication. The effects of the
physical environment on nonverbal communication
are not only pervasive but also subtle, as they often
unfold automatically and outside of individuals’
conscious awareness. However, individuals are not
simply passive respondents to random environmen-
tal features. People also choose particular environ-
ments to meet their needs and, consequently, shape
their own patterns of nonverbal communication.
Although early, as well as contemporary, research
demonstrated the interdependence between nonver-
bal behavior and physical settings, frequently both
components are still studied as isolated processes.
As a result, a long-established criticism that the
environment is typically separated from nonverbal
communication in the study of social encounters
(T. Lee, 1976) still remains relevant today. Address-
ing this goal is particularly important in promoting
the ecological validity and translational aspects of
nonverbal communication research and, more gen-
erally, social–psychological research (Baumeister
et al., 2007; Patterson, 2008). The ecological systems
model discussed in this chapter provides one way of
210
representing these complex relationships between
individuals and their environments in an integrated
framework. Ultimately, only by understanding how
nonverbal communication and environmental set-
tings act in concert can physical settings be con-
structed and manipulated to serve diverse human
needs (Griffin, Mauritzen, & Kasmar, 1969; J. Li &
Robertson, 2011; Murphy & Walton, 2013).
References
Aaker, J. L. (1997). Dimensions of brand personality.
Journal of Marketing Research, 34, 347–356.
http://dx.doi.org/10.2307/3151897
Adams, L., & Zuckerman, D. (1991). The effect of
lighting conditions on personal space requirements.
Journal of General Psychology, 118, 335–340. http://
dx.doi.org/10.1080/00221309.1991.9917794
Adams, R. S., & Biddle, B. (1970). Realities of teaching:
Explorations with video tape. New York, NY: Holt,
Rinehart, & Winston.
Aglioti, S. M., & Pazzaglia, M. (2011). Sounds and scents
in (social) action. Trends in Cognitive Sciences, 15,
47–55. http://dx.doi.org/10.1016/j.tics.2010.12.003
Altman, I. (1975). The environment and social behavior:
Privacy, personal space, territory, crowding. Monterey,
CA: Brooks/Cole.
Altman, I. (1976). Privacy: A conceptual analysis.
Environment and Behavior, 8, 7–29. http://dx.doi.
org/10.1177/001391657600800102
Ambady, N., Bernieri, F., & Richeson, J. (2000). Towards
a histology of social behavior: Judgmental accuracy
from thin slices of behavior. In M. P. Zanna
(Ed.), Advances in experimental social psychology
(pp. 201–272). San Diego, CA: Academic Press.
Ambady, N., Hallahan, M., & Rosenthal, R. (1995).
On judging and being judged accurately in zero-
acquaintance situations. Journal of Personality
and Social Psychology, 69, 518–529. http://dx.doi.
org/10.1037/0022-3514.69.3.518
Asch, S. E. (1946). Forming impressions of personality.
Journal of Abnormal and Social Psychology, 41,
258–290. http://dx.doi.org/10.1037/h0055756
Asmus, C. L., & Bell, P. A. (1999). Effects of
environmental odor and coping style on negative
affect, anger, arousal, and escape. Journal of Applied
Social Psychology, 29, 245–260. http://dx.doi.
org/10.1111/j.1559-1816.1999.tb01384.x
Banerjee, P., Chatterjee, P., & Sinha, J. (2012).
Is it light or dark? Recalling moral behavior
changes perception of brightness. Psychological
Science, 23, 407–409. http://dx.doi.
org/10.1177/0956797611432497

Bargh, J. A. (1997). The automaticity of everyday life.
In R. S. Wyer (Ed.), Advances in social cognition
(pp. 1–61). Mahwah, NJ: Erlbaum.
Bargh, J. A., & Chartrand, T. L. (1999). The unbearable
automaticity of being. American Psychologist, 54,
462–479. http://dx.doi.org/10.1037/0003-066X.
54.7.462
Bargh, J. A., Gollwitzer, P. M., & Oettingen, G. (2010).
Motivation. In S. T. Fiske, D. T. Gilbert, & G.
Lindzey (Eds.), Handbook of social psychology
(5th ed., pp. 268–316). New York, NY: Wiley.
Bargh, J. A., & Shalev, I. (2012). The substitutability of
physical and social warmth in daily life. Emotion, 12,
154–162. http://dx.doi.org/10.1037/a0023527
Barker, R. G. (1968). Ecological psychology: Concepts
and methods for studying the environment of human
behavior. Stanford, CA: Stanford University Press.
Baron, R. A. (1997). The sweet smell of . . . helping:
Effects of pleasant ambient fragrances on prosocial
behavior in shopping malls. Personality and Social
Psychology Bulletin, 23, 498–503. http://dx.doi.
org/10.1177/0146167297235005
Baron, R. A., Rea, M. S., & Daniels, S. G. (1992).
Effects of indoor lighting (illuminance and spectral
distribution) on the performance of cognitive tasks
and interpersonal behaviors: The potential mediating
role of positive affect. Motivation and Emotion, 16,
1–33. http://dx.doi.org/10.1007/BF00996485
Baron, R. A., & Thomley, J. (1994). A whiff of reality:
Positive affect as a potential mediator of the effects of
pleasant fragrances on task performance and helping.
Environment and Behavior, 26, 766–784. http://
dx.doi.org/10.1177/0013916594266003
Baum, A., & Valins, S. (1979). Architectural mediation
of residential density and control: Crowding
and regulation of social conduct. Advances in
Experimental Social Psychology, 12, 131–175. http://
dx.doi.org/10.1016/S0065-2601(08)60261-0
Baumeister, R. F., & Leary, M. R. (1995). The
need to belong: Desire for interpersonal
attachments as a fundamental human motivation.
Psychological Bulletin, 117, 497–529. http://dx.doi.
org/10.1037/0033-2909.117.3.497
Baumeister, R. F., Vohs, K. D., & Funder, D. C. (2007).
Psychology as the science of self-reports and finger
movements. Whatever happened to actual behavior?
Perspectives on Psychological Science, 2, 396–403.
http://dx.doi.org/10.1111/j.1745-6916.2007.00051.x
Baxter, J. C. (1970). Interpersonal spacing in natural
settings. Sociometry, 33, 444–456. http://dx.doi.
org/10.2307/2786318
Beauregard, K. S., & Dunning, D. (1998). Turning up
the contrast: Self-enhancement motives prompt
egocentric contrast effects in social judgments.
The Physical Environment and Nonverbal Communication
Journal of Personality and Social Psychology, 74,
606–621. http://dx.doi.org/10.1037/0022-3514.
74.3.606
Becker, F. D. (1973). Study of spatial markers. Journal
of Personality and Social Psychology, 26, 439–445.
http://dx.doi.org/10.1037/h0034442
Beckes, L., & Coan, J. A. (2011). Social baseline theory:
The role of social proximity in emotion and economy
of action. Social and Personality Psychology Compass,
5, 976–988. http://dx.doi.org/10.1111/j.1751-9004.
2011.00400.x
Bell, P. A. (2005). Reanalysis and perspective in the
heat–aggression debate. Journal of Personality
and Social Psychology, 89, 71–73. http://dx.doi.
org/10.1037/0022-3514.89.1.71
Bell, P. A., Garnand, D. B., & Heath, D. (1984). Effects
of ambient temperature and seating arrangement on
personal and environmental evaluations. Journal of
General Psychology, 110, 197–200. http://dx.doi.org/
10.1080/00221309.1984.9709963
Bell, P. A., Greene, T. C., Fisher, J. D., & Baum, A. (2001).
Environmental psychology (5th ed.). Fort Worth, TX:
Harcourt College.
Berger, C. B. (1997). Planning strategic interaction.
Mahwah, NJ: Erlbaum.
Bernstein, M. J., Sacco, D. F., Brown, C. M., Young, S.
G., & Claypool, H. M. (2010). A preference for
genuine smiles following social exclusion. Journal of
Experimental Social Psychology, 46, 196–199. http://
dx.doi.org/10.1016/j.jesp.2009.08.010
Berry, D. S., & McArthur, L. Z. (1986). Perceiving
character in faces: The impact of age-related
craniofacial changes on social perception.
Psychological Bulletin, 100, 3–18. http://dx.doi.
org/10.1037/0033-2909.100.1.3
Bilton, N. (2013, September 1). Disruptions: More
connected, yet more alone. New York Times. Retrieved
from http://bits.blogs.nytimes.com/2013/09/01/
disruptions-more-connected-yet-more-alone
Blascovich, J., Spencer, S. J., Quinn, D., & Steele, C.
(2001). African Americans and high blood pressure:
The role of stereotype threat. Psychological Science,
12, 225–229. http://dx.doi.org/10.1111/1467-9280.
00340
Boles, W. E., & Hayward, S. C. (1978). The effects of
urban noise and sidewalk density upon pedestrian
cooperation and tempo. Journal of Social Psychology,
104, 29–35. http://dx.doi.org/10.1080/00224545.
1978.9924035
Bonini, N., Hadjichristidis, C., Mazzocco, K., Demattè,
M. L., Zampini, M., Sbarbati, A., & Magon, S. (2011).
Pecunia olet: The role of incidental disgust in the
ultimatum game. Emotion, 11, 965–969. http://dx.doi.
org/10.1037/a0022820
211
Patterson and Quadflieg

Bowers, B. (1998). Lengthening the day: A history of
lighting technology. Oxford, England: Oxford
University Press.
Boyle, E. A., Anderson, A. H., & Newlands, A. (1994).
The effects of visibility on dialogue and performance
in a cooperative problem solving task. Language and
Speech, 37, 1–20.
Brunswik, E. (1955). Representative design and
probabilistic theory in a functional psychology.
Psychological Review, 62, 193–217. http://dx.doi.
org/10.1037/h0047470
Bugeja, M. (2005). Interpersonal divide: The search for
community in a technological age. New York, NY:
Oxford University Press.
Burgess, D. J., Warren, J., Phelan, S., Dovidio, J., &
van Ryn, M. (2010). Stereotype threat and health
disparities: What medical educators and future
physicians need to know. Journal of General Internal
Medicine, 25, 169–177. http://dx.doi.org/10.1007/
s11606-009-1221-4
Bushman, B. J., Wang, M. C., & Anderson, C. A. (2005).
Is the curve relating temperature to aggression
linear or curvilinear? Assaults and temperature
in Minneapolis reexamined. Journal of Personality
and Social Psychology, 89, 62–66. http://dx.doi.
org/10.1037/0022-3514.89.1.62
Buss, D. M. (1987). Selection, evocation, and
manipulation. Journal of Personality and Social
Psychology, 53, 1214–1221. http://dx.doi.org/
10.1037/0022-3514.53.6.1214
Byrne, D. (1971). The attraction paradigm. New York, NY:
Academic Press.
Cadinu, M., Maass, A., Rosabianca, A., & Kiesner,
J. (2005). Why do women underperform under
stereotype threat? Evidence for the role of negative
thinking. Psychological Science, 16, 572–578. http://
dx.doi.org/10.1111/j.0956-7976.2005.01577.x
Campbell, D. E. (1979). Interior office design and visitor
response. Journal of Applied Psychology, 64, 648–653.
http://dx.doi.org/10.1037/0021-9010.64.6.648
Carr, S. J., & Dabbs, J. M., Jr. (1974). The effects of
lighting, distance and intimacy of topic on verbal and
visual behavior. Sociometry, 37, 592–600.
http://dx.doi.org/10.2307/2786430
Cary, M. S. (1978). Does civil inattention exist in
pedestrian passing? Journal of Personality and Social
Psychology, 36, 1185–1193. http://dx.doi.org/
10.1037/0022-3514.36.11.1185
Cesario, J., Plaks, J. E., Hagiwara, N., Navarrete, C. D., &
Higgins, E. T. (2010). The ecology of automaticity:
How situational contingencies shape action
semantics and social behavior. Psychological Science,
21, 1311–1317. http://dx.doi.org/10.1177/
0956797610378685
Chartrand, T. L., & Lakin, J. L. (2013). The antecedents
and consequences of human behavioral mimicry.
Annual Review of Psychology, 64, 285–308. http://
dx.doi.org/10.1146/annurev-psych-113011-143754
Cheryan, S., Meltzoff, A. N., & Kim, S. (2011).
Classrooms matter: The design of virtual classrooms
influences gender disparities in computer science
classes. Computers and Education, 57, 1825–1835.
http://dx.doi.org/10.1016/j.compedu.2011.02.004
Cheryan, S., Plaut, V. C., Davies, P. G., & Steele, C. M.
(2009). Ambient belonging: How stereotypical cues
impact gender participation in computer science.
Journal of Personality and Social Psychology, 97,
1045–1060. http://dx.doi.org/10.1037/a0016239
Choi, Y. S., Gray, H. M., & Ambady, N. (2005).
The glimpsed world: Unintended communication
and unintended perception. In R. R. Hassin, J. S.
Uleman, & J. A. Bargh (Eds.), The new unconscious
(pp. 309–333). New York, NY: Oxford University
Press.
Cloutier, J., & Macrae, C. N. (2007). Who or what
are you? Facial orientation and person construal.
European Journal of Social Psychology, 37,
1298–1309. http://dx.doi.org/10.1002/ejsp.433
Cloutier, J., Mason, M. F., & Macrae, C. N. (2005).
The perceptual determinants of person construal:
Reopening the social-cognitive toolbox. Journal
of Personality and Social Psychology, 88, 885–894.
http://dx.doi.org/10.1037/0022-3514.88.6.885
Cochran, C. D., Hale, W. D., & Hissam, C. P. (1984).
Personal space requirements in indoor vs. outdoor
locations. Journal of Psychology: Interdisciplinary and
Applied, 117, 121–123. http://dx.doi.org/10.1080/
00223980.1984.9923667
Cohen, S., & Lezak, A. (1977). Noise and inattentiveness
to social cues. Environment and Behavior, 9, 559–572.
http://dx.doi.org/10.1177/001391657794007
Coleman, J. (1990). Foundations of social thought.
Cambridge, MA: Belknap.
Cook, M. (1970). Experiments on orientation and
proxemics. Human Relations, 23, 61–76. http://
dx.doi.org/10.1177/001872677002300107
Correll, J., Wittenbrink, B., Park, B., Judd, C. M., &
Goyle, A. (2011). Dangerous enough: Moderating
racial bias with contextual threat cues. Journal of
Experimental Social Psychology, 47, 184–189. http://
dx.doi.org/10.1016/j.jesp.2010.08.017
Cunningham, E., Forestell, C. A., & Dickter, D. (2013).
Induced disgust affects implicit and explicit
responses toward gay men and lesbians. European
Journal of Social Psychology, 43, 362–369. http://
dx.doi.org/10.1002/ejsp.1945
Cunningham, S. J., & Macrae, C. N. (2011). The colour
of gender stereotyping. British Journal of Psychology,

212

102, 598–614. http://dx.doi.org/10.1111/j.2044-
8295.2011.02023.x
Custers, R., & Aarts, H. (2003). On the role of processing
goals in evaluative judgments of environments:
Effects on memory–judgment relations. Journal of
Environmental Psychology, 23, 289–299. http://dx.doi.
org/10.1016/S0272-4944(02)00076-2
Davies, D. R., & Hockey, G. R. J. (1966). The effects
of noise and doubling the signal frequency
on individual differences in visual vigilance
performance. British Journal of Psychology, 57,
381–389. http://dx.doi.org/10.1111/j.2044-
8295.1966.tb01039.x
Demattè, M. L., Österbauer, R., & Spence, C. (2007).
Olfactory cues modulate facial attractiveness.
Chemical Senses, 32, 603–610. http://dx.doi.
org/10.1093/chemse/bjm030
Donnerstein, E., & Wilson, D. W. (1976). Effects of noise
and perceived control on ongoing and subsequent
aggressive behavior. Journal of Personality and
Social Psychology, 34, 774–781. http://dx.doi.
org/10.1037/0022-3514.34.5.774
Doty, R. L., & Cameron, E. L. (2009). Sex differences
and reproductive hormone influences on human
odor perception. Physiology and Behavior,
97, 213–228. http://dx.doi.org/10.1016/j.
physbeh.2009.02.032
Drew, C. J. (1971). Research on the psychological–
behavioral effects of the physical environment.
Review of Educational Research, 41, 447–465.
Duncan, B. L. (1976). Differential social perception and
attribution of intergroup violence: Testing the lower
limits of stereotyping of Blacks. Journal of Personality
and Social Psychology, 34, 590–598. http://dx.doi.
org/10.1037/0022-3514.34.4.590
Eibl-Eibesfeldt, I. (1989). Human ethology. New York,
NY: Aldine de Gruyter.
Ekman, P., Friesen, W. V., & O’Sullivan, M. (1988).
Smiles when lying. Journal of Personality and Social
Psychology, 54, 414–420. http://dx.doi.org/10.1037/
0022-3514.54.3.414
Elliot, A. J., & Maier, M. A. (2007). Color and
psychological functioning. Current Directions in
Psychological Science, 16, 250–254. http://dx.doi.
org/10.1111/j.1467-8721.2007.00514.x
Elliot, A. J., Maier, M. A., Moller, A. C., Friedman, R., &
Meinhardt, J. (2007). Color and psychological
functioning: The effect of red on performance
attainment. Journal of Experimental Psychology:
General, 136, 154–168. http://dx.doi.org/10.1037/
0096-3445.136.1.154
Elliot, A. J., Niesta Kayser, D., Greitemeyer, T.,
Lichtenfeld, S., Gramzow, R. H., Maier, M. A., &
Liu, H. (2010). Red, rank, and romance in women
The Physical Environment and Nonverbal Communication
viewing men. Journal of Experimental Psychology:
General, 139, 399–417. http://dx.doi.org/10.1037/
a0019689
Elliott, C. D. (1971). Noise tolerance and extraversion in
children. British Journal of Psychology, 62, 375–380.
http://dx.doi.org/10.1111/j.2044-8295.1971.
tb02048.x
Ellsworth, P. C., & Langer, E. J. (1976). Staring and
approach: An interpretation of the stare as a
nonspecific activator. Journal of Personality and
Social Psychology, 33, 117–122. http://dx.doi.
org/10.1037/0022-3514.33.1.117
Evans, G. W., & Wener, R. E. (2007). Crowding and
personal space invasion on the train: Please don’t
make me sit in the middle. Journal of Environmental
Psychology, 27, 90–94. http://dx.doi.org/10.1016/
j.jenvp.2006.10.002
Felipe, N., & Sommer, R. (1966). Invasion of personal
space. Social Problems, 14, 206–214. http://dx.doi.
org/10.2307/798618
Firestone, C., & Scholl, B. J. (2014). “Top-down”
effects where none should be found: The El
Greco fallacy in perception research. Psychological
Science, 25, 38–46. http://dx.doi.org/10.1177/
0956797613485092
Fisher, J. D., & Byrne, D. (1975). Too close for comfort:
Sex differences in response to invasions of personal
space. Journal of Personality and Social Psychology,
32, 15–21. http://dx.doi.org/10.1037/h0076837
Fiske, S. T., Cuddy, A. J. C., & Glick, P. (2007).
Universal dimensions of social cognition: Warmth
and competence. Trends in Cognitive Sciences, 11,
77–83. http://dx.doi.org/10.1016/j.tics.2006.11.005
Fiske, S. T., & Taylor, S. E. (1984). Social cognition.
Reading, MA: Addison-Wesley.
Fiske, S. T., & Taylor, S. E. (2013). Social cognition: From
brains to culture. Thousand Oaks, CA: Sage.
Frank, M. G., & Gilovich, T. (1988). The dark side
of self- and social perception: Black uniforms
and aggression in professional sports. Journal of
Personality and Social Psychology, 54, 74–85. http://
dx.doi.org/10.1037/0022-3514.54.1.74
Fridlund, A. J. (1994). Human facial expression: An
evolutionary view. San Diego, CA: Academic Press.
Gangestad, S. W., & Thornhill, R. (1998). Menstrual
cycle variation in women’s preferences for the scent
of symmetrical men. Proceedings of the Royal Society
B: Biological Sciences, 265, 927–933. http://dx.doi.
org/10.1098/rspb.1998.0380
Gergen, K. J., Gergen, M. M., & Barton, W. H. (1973).
Deviance in the dark. Psychology Today, 7, 129–130.
Gibson, J. J. (1979). The ecological approach to visual
perception. Boston, MA: Houghton Mifflin.
213
Patterson and Quadflieg

Gilbert, D. T., Pelham, B. W., & Krull, D. S. (1988). Haber, G. M. (1982). Spatial relations between dominants
On cognitive busyness: When person perceivers and marginals. Social Psychology Quarterly, 45,
meet persons perceived. Journal of Personality 219–228. http://dx.doi.org/10.2307/3033917
and Social Psychology, 54, 733–740. http://dx.doi.
Hall, E. T. (1959). The silent language. Garden City, NY:
org/10.1037/0022-3514.54.5.733
Doubleday.
Gilliam, J. E., & Unruh, D. (1988). The effects of Baker-
Hall, E. T. (1966). The hidden dimension. Garden City,
Miller pink on biological, physical and cognitive
NY: Doubleday.
behavior. Journal of Orthomolecular Medicine, 3,
202–206. Hall, J. A., & Gunnery, S. D. (2013). Gender differences
in nonverbal communication. In J. A. Hall &
Goffman, E. (1963). Behavior in public places. New York,
M. L. Knapp (Eds.), Nonverbal communication
NY: Free Press.
(pp. 639–669). http://dx.doi.org/10.1515/
Gosling, S. D., Craik, K. H., Martin, N. R., & Pryor, M. R. 9783110238150.639
(2005a). Material attributes of personal living
Hamid, P. N., & Newport, A. G. (1989). Effect of
spaces. Home Cultures, 2, 51–88. http://dx.doi.
colour on physical strength and mood in children.
org/10.2752/174063105778053436
Perceptual and Motor Skills, 69, 179–185. http://
Gosling, S. D., Craik, K. H., Martin, N. R., & Pryor, dx.doi.org/10.2466/pms.1989.69.1.179
M. R. (2005b). The Personal Living Space Cue
Harris, P. B., & Sachau, D. (2005). Is cleanliness next to
Inventory: An analysis and evaluation. Environment
godliness? The role of housekeeping in impression
and Behavior, 37, 683–705. http://dx.doi.
formation. Environment and Behavior, 37, 81–101.
org/10.1177/0013916504274011
http://dx.doi.org/10.1177/0013916504266803
Gosling, S. D., Gaddis, S., & Vazire, S. (2008). First
Harrison, R. P. (1973). Nonverbal communication. In I. S.
impressions based on the environments we create
Pool, W. Schramm, N. Maccoby, F. Fry, E. Parker, &
and inhabit. In N. Ambady & J. J. Skowronski (Eds.),
J. L. Fern (Eds.), Handbook of communication
First impressions (pp. 334–356). New York, NY:
(pp. 93–115). Chicago, IL: Rand McNally.
Guilford Press.
Heslin, R., & Boss, D. (1980). Nonverbal intimacy in
Gosling, S. D., Ko, S. J., Mannarelli, T., & Morris, M. E.
airport arrival and departure. Personality and Social
(2002). A room with a cue: Personality judgments
Psychology Bulletin, 6, 248–252. http://dx.doi.
based on offices and bedrooms. Journal of Personality
org/10.1177/014616728062010
and Social Psychology, 82, 379–398. http://dx.doi.
org/10.1037/0022-3514.82.3.379 Hirsh, J. B., Galinsky, A. D., & Zhong, C.-B. (2011).
Drunk, powerful, and in the dark: How general
Griffin, W. V., Mauritzen, J. H., & Kasmar, J. V. (1969).
processes of disinhibition produce both
The psychological aspects of the architectural
prosocial and antisocial behavior. Perspectives on
environment: A review. American Journal of
Psychological Science, 6, 415–427. http://dx.doi.
Psychiatry, 125, 1057–1062. http://dx.doi.org/
org/10.1177/1745691611416992
10.1176/ajp.125.8.1057
Hold, B., & Schleidt, M. (1977). The importance of
Griffitt, W. (1970). Environmental effects on
human odour in non-verbal communication.
interpersonal affective behavior: Ambient effective
Zeitschrift für Tierpsychologie, 43, 225–238. http://
temperature and attraction. Journal of Personality
dx.doi.org/10.1111/j.1439-0310.1977.tb00072.x
and Social Psychology, 15, 240–244. http://dx.doi.
org/10.1037/h0029432 Hummel, T., Gollisch, R., Wildt, G., & Kobal, G.
(1991). Changes in olfactory perception during the
Griffitt, W., & Veitch, R. (1971). Hot and crowded:
menstrual cycle. Experientia, 47, 712–715. http://
Influences of population density and temperature
dx.doi.org/10.1007/BF01958823
on interpersonal affective behavior. Journal of
Personality and Social Psychology, 17, 92–98. http:// IJzerman, H., & Semin, G. R. (2009). The thermometer
dx.doi.org/10.1037/h0030458 of social relations: Mapping social proximity on
temperature. Psychological Science, 20, 1214–1220.
Guéguen, N. (2001). Effect of a perfume on prosocial
http://dx.doi.org/10.1111/j.1467-9280.2009.02434.x
behavior of pedestrians. Psychological Reports,
88, 1046–1048. http://dx.doi.org/10.2466/ IJzerman, H., & Semin, G. R. (2010). Temperature
pr0.2001.88.3c.1046 perceptions as a ground for social proximity. Journal
of Experimental Social Psychology, 46, 867–873.
Guéguen, N. (2012). The sweet smell of . . . implicit
http://dx.doi.org/10.1016/j.jesp.2010.07.015
helping: Effects of pleasant ambient fragrance on
spontaneous help in shopping malls. Journal of Social Inbar, Y., Pizarro, D. A., & Bloom, P. (2012). Disgusting
Psychology, 152, 397–400. http://dx.doi.org/10.1080/ smells cause decreased liking of gay men. Emotion,
00224545.2011.630434 12, 23–27. http://dx.doi.org/10.1037/a0023984

214
 The Physical Environment and Nonverbal Communication

Ishii, K., Miyamoto, Y., Rule, N. O., & Toriyama, R. Krueger, J. I. (1998). Enhancement bias in descriptions
(2014). Physical objects as vehicles of cultural of self and others. Personality and Social
transmission: Maintaining harmony and uniqueness Psychology Bulletin, 24, 505–516. http://dx.doi.
through colored geometric patterns. Personality and org/10.1177/0146167298245006
Social Psychology Bulletin, 40, 175–188. http://dx.doi.
Kryter, K. D. (1970). The effects of noise on man. London,
org/10.1177/0146167213508151
England: Academic Press.
Jackson, D., Engstrom, E., & Hassenzahl, D. M. (2005).
Küller, R., Mikellides, B., & Janssens, J. (2009). Color,
Effects of sex and seating arrangement on selection
arousal, and performance—A comparison of three
of leader. Perceptual and Motor Skills, 100, 815–818.
experiments. Color Research and Application, 34,
http://dx.doi.org/10.2466/PMS.100.3.815-818
141–152. http://dx.doi.org/10.1002/col.20476
Jones, D. M., & Chapman, A. J. (1984). Noise and society.
Kupritz, V. M. (2000). Privacy management at work:
Chichester, England: Wiley.
A conceptual model. Journal of Architectural and
Jones, D. M., Chapman, A. J., & Auburn, T. C. (1981). Planning Research, 17, 47–63.
Noise in the environment: A social perspective.
Laidlaw, K. E., Foulsham, T., Kuhn, G., & Kingstone, A.
Journal of Environmental Psychology, 1, 43–59. http://
(2011). Potential social interactions are important to
dx.doi.org/10.1016/S0272-4944(81)80017-5
social attention. Proceedings of the National Academy
Kang, Y., Williams, L. E., Clark, M. S., Gray, J. R., & of Sciences, USA, 108, 5548–5553. http://dx.doi.
Bargh, J. A. (2011). Physical temperature effects on org/10.1073/pnas.1017022108
trust behavior: The role of insula. Social Cognitive
Lakens, D. (2011). Orange as a perceptual representation
and Affective Neuroscience, 6, 507–515. http://dx.doi.
of the Dutch nation: Effects on perceived national
org/10.1093/scan/nsq077
identification and color evaluation. European Journal
Kay, A. C., Wheeler, S. C., Bargh, J. A., & Ross, L. (2004). of Social Psychology, 41, 924–929. http://dx.doi.
Material priming: The influence of mundane physical org/10.1002/ejsp.848
objects on situational construal and competitive
Lakens, D., Semin, G. R., & Foroni, F. (2012). But for the
behavioral choice. Organizational Behavior and
bad, there would not be good: Grounding valence
Human Decision Processes, 95, 83–96. http://dx.doi.
in brightness through shared relational structures.
org/10.1016/j.obhdp.2004.06.003
Journal of Experimental Psychology: General, 141,
Kelley, H. H. (1972). Attribution in social interaction. 584–594. http://dx.doi.org/10.1037/a0026468
In E. E. Jones, D. E. Kanouse, H. H. Kelley, R. E.
Lakin, J. L., & Chartrand, T. L. (2003). Using
Nisbett, S. Valins, & B. Weiner (Eds.), Attribution:
nonconscious behavioral mimicry to create affiliation
Perceiving the causes of behavior (pp. 1–26).
and rapport. Psychological Science, 14, 334–339.
Morristown, NJ: General Learning Press.
http://dx.doi.org/10.1111/1467-9280.14481
Kleinke, C. L. (1986). Gaze and eye contact: A research
Lebra, T. S. (1976). The Japanese pattern of behavior.
review. Psychological Bulletin, 100, 78–100. http://
Honolulu: University of Hawaii Press.
dx.doi.org/10.1037/0033-2909.100.1.78
Lecomte, C., Bernstein, B. L., & Dumont, F. (1981).
Knapp, M. L., Hall, J. A., & Horgan, T. G. (2014). The
Counseling interactions as a function of spatial-
effects of the environment on human communication.
environmental conditions. Journal of Counseling
In M. L. Knapp, J. A. Hall, & T. G. Horgan (Eds.),
Psychology, 28, 536–539. http://dx.doi.org/10.1037/
Nonverbal communication in human interaction
0022-0167.28.6.536
(8th ed., pp. 89–122). Boston, MA: Wadsworth.
Lee, S. W. S., & Schwarz, N. (2012). Bidirectionality,
Kolb, P., Gockel, C., & Werth, L. (2012). The effects
mediation, and moderation of metaphorical effects:
of temperature on service employees’ customer
The embodiment of social suspicion and fishy smells.
orientation: An experimental approach. Ergonomics,
Journal of Personality and Social Psychology, 103,
55, 621–635. http://dx.doi.org/10.1080/00140139.
737–749. http://dx.doi.org/10.1037/a0029708
2012.659763
Lee, T. (1976). Psychology and the environment. London,
Konecni, V. J. (1975). The mediation of aggressive
England: Methuen.
behavior: Arousal level versus anger and cognitive
labeling. Journal of Personality and Social Psychology, Lewin, K. (1939). Field theory and experiment in social
32, 706–712. http://dx.doi.org/10.1037/0022-3514. psychology: Concepts and methods. American
32.4.706 Journal of Sociology, 44, 868–896. http://dx.doi.
org/10.1086/218177
Korte, C. (1980). Urban–nonurban differences in social
behavior and social psychological models of urban Li, J., & Robertson, T. (2011). Physical space and
impact. Journal of Social Issues, 36, 29–51. http:// information space: Studies of collaboration in
dx.doi.org/10.1111/j.1540-4560.1980.tb02034.x distributed multi-disciplinary medical team

215
Patterson and Quadflieg
meetings. Behaviour and Information Technology,
30, 443–454. http://dx.doi.org/10.1080/
0144929X.2011.577194
Li, S., & Li, Y.-M. (2007). How far is far enough?
A measure of information privacy in terms of
interpersonal distance. Environment and Behavior,
39, 317–331. http://dx.doi.org/10.1177/
0013916506290956
Lichtenfeld, S., Elliot, A. J., Maier, M. A., & Pekrun,
R. (2012). Fertile green: Green facilitates
creative performance. Personality and Social
Psychology Bulletin, 38, 784–797. http://dx.doi.
org/10.1177/0146167212436611
Liljenquist, K. A., Zhong, C.-B., & Galinsky, A. D.
(2010). The smell of virtue: Clean scents promote
reciprocity and charity. Psychological Science,
21, 381–383. http://dx.doi.org/10.1177/
0956797610361426
Lloyd, D. M., Coates, A., Knopp, J., Oram, S., &
Rowbotham, S. (2009). Don’t stand so close to me:
The effect of auditory input on interpersonal space.
Perception, 38, 617–620. http://dx.doi.org/10.1068/
p6317
Loersch, C., & Payne, B. K. (2011). The situated
inference model: An integrative account of the effects
of primes on perception, behavior, and motivation.
Perspectives on Psychological Science, 6, 234–252.
http://dx.doi.org/10.1177/1745691611406921
Lumsden, J., Miles, L. K., Richardson, M. J., Smith, C.
A., & Macrae, C. N. (2012). Who syncs? Social
motives and interpersonal coordination. Journal of
Experimental Social Psychology, 48, 746–751. http://
dx.doi.org/10.1016/j.jesp.2011.12.007
Macintyre, S., & Homel, R. (1997). Danger on the dance
floor: A study of interior design, crowding, and
aggression in nightclubs. In R. Homel (Ed.), Policing
for prevention: Reducing crime, public intoxication,
and injury (pp. 92–113). Monsey, NY: Criminal
Justice Press.
Macrae, C. N., Milne, A. B., & Bodenhausen, G. V.
(1994). Stereotypes as energy-saving devices: A peek
inside the cognitive toolbox. Journal of Personality
and Social Psychology, 66, 37–47. http://dx.doi.
org/10.1037/0022-3514.66.1.37
Macrae, C. N., Quinn, K. A., Mason, M. F., & Quadflieg,
S. (2005). Understanding others: The face and
person construal. Journal of Personality and
Social Psychology, 89, 686–695. http://dx.doi.
org/10.1037/0022-3514.89.5.686
Macrae, C. N., Raj, R. S., Best, S. B., Christian, B. M., &
Miles, L. K. (2013). Imagined sensory experience
can shape person perception: It’s a matter of
visual perspective. Journal of Experimental Social
Psychology, 49, 595–598. http://dx.doi.org/10.1016/
j.jesp.2012.10.002
216
Manusov, V., & Patterson, M. L. (2006). The Sage
handbook of nonverbal communication. Thousand
Oaks, CA: Sage.
Manzo, J. (2005). Social control and the management
of “personal” space in shopping malls.
Space and Culture, 8, 83–97. http://dx.doi.
org/10.1177/1206331204265991
Mathews, K. E., Jr., & Canon, L. K. (1975). Environment
noise level as a determinant of helping behavior.
Journal of Personality and Social Psychology, 32,
571–577. http://dx.doi.org/10.1037/0022-3514.
32.4.571
Mathews, K. E., Jr., Canon, L. K., & Alexander, K.
(1974). The influence of level of empathy and
ambient noise on the body buffer zone. Personality
and Social Psychology Bulletin, 1, 367–369. http://
dx.doi.org/10.1177/0146167274001001124
McArthur, L. Z., & Baron, R. M. (1983). Toward an
ecological theory of social perception. Psychological
Review, 90, 215–238. http://dx.doi.org/10.1037/0033-
295X.90.3.215
McGeehan, P. (2005, May 31). For train riders, middle
seat isn’t the center of attention. New York Times,
pp. Al, C11.
McNamara, L., & Ballard, M. E. (1999). Resting arousal,
sensation seeking, and music preference. Genetic,
Social, and General Psychology Monographs, 125,
229–251.
Mehl, M. R., & Pennebaker, J. W. (2003). The sounds
of social life: A psychometric analysis of students’
daily social environments and natural conversations.
Journal of Personality and Social Psychology,
84, 857–870. http://dx.doi.org/10.1037/0022-
3514.84.4.857
Mehta, V. (2009). Look closely and you will see, listen
carefully and you will hear: Urban design and
social interaction on streets. Journal of Urban Design,
14, 29–64. http://dx.doi.org/10.1080/
13574800802452658
Meier, B. P., Robinson, M. D., & Clore, G. L. (2004).
Why good guys wear white: Automatic inferences
about stimulus valence based on brightness.
Psychological Science, 15, 82–87. http://dx.doi.
org/10.1111/j.0963-7214.2004.01502002.x
Milgram, S. (1970). The experience of living in cities.
Science, 167, 1461–1468. http://dx.doi.org/10.1126/
science.167.3924.1461
Miwa, Y., & Hanyu, K. (2006). The effects of interior
design on communication and impressions of a
counselor in a counseling room. Environment and
Behavior, 38, 484–502. http://dx.doi.org/10.1177/
0013916505280084
Montepare, J. M., & Zebrowitz-McArthur, L. (1988).
Impressions of people created by age-related

qualities of their gaits. Journal of Personality and
Social Psychology, 55, 547–556. http://dx.doi.
org/10.1037/0022-3514.55.4.547
Moos, R. H., Harris, R., & Schonborn, K. (1969).
Psychiatric patients and staff reaction to their
physical environment. Journal of Clinical Psychology,
25, 322–324. http://dx.doi.org/10.1002/1097-
4679(196907)25:3<322::AID-JCLP2270250332>
3.0.CO;2-C
Moser, G. (1988). Urban stress and helping behavior:
Effects of environmental overload and noise on
behavior. Journal of Environmental Psychology,
8, 287–298. http://dx.doi.org/10.1016/S0272-
4944(88)80035-5
Motyl, M., Iyer, R., Oishi, S., Trawalter, S., & Nosek, B.
A. (2014). How ideological migration geographically
segregates groups. Journal of Experimental Social
Psychology, 51, 1–14. http://dx.doi.org/10.1016/
j.jesp.2013.10.010
Murphy, M. C., & Walton, G. M. (2013). From
prejudiced people to prejudiced places: A social-
contextual approach to prejudice. In C. Stangor & C.
Crandall (Eds.), Frontiers in social psychology series:
Stereotyping and prejudice (pp. 181–204). New York,
NY: Psychology Press.
Newman, J., & McCauley, C. (1977). Eye contact
with strangers in city, suburb, and small town.
Environment and Behavior, 9, 547–558. http://dx.doi.
org/10.1177/001391657794006
Niesta Kayser, D., Elliot, A. J., & Feltman, R. (2010).
Red and romantic behavior in men viewing women.
European Journal of Social Psychology, 40, 901–908.
http://dx.doi.org/10.1002/ejsp.757
Norman, W. T., & Goldberg, L. R. (1966). Raters, ratees,
and randomness in personality structure. Journal of
Personality and Social Psychology, 4, 681–691. http://
dx.doi.org/10.1037/h0024002
Oishi, S., & Graham, J. (2010). Social ecology: Lost and
found in psychological science. Perspectives on
Psychological Science, 5, 356–377. http://dx.doi.org/
10.1177/1745691610374588
Page, R. A. (1977). Noise and helping behavior.
Environment and Behavior, 9, 311–334. http://dx.doi.
org/10.1177/001391657700900302
Page, R. A., & Moss, M. K. (1976). Environmental
influences on aggression: The effects of darkness
and proximity of victim. Journal of Applied Social
Psychology, 6, 126–133. http://dx.doi.org/10.1111/
j.1559-1816.1976.tb01318.x
Patterson, M. L. (1973). Compensation in nonverbal
immediacy behaviors: A review. Sociometry, 36,
237–252. http://dx.doi.org/10.2307/2786569
Patterson, M. L. (1976). An arousal model of
interpersonal intimacy. Psychological Review,
The Physical Environment and Nonverbal Communication
83, 235–245. http://dx.doi.org/10.1037/0033-
295X.83.3.235
Patterson, M. L. (1982). A sequential functional model
of nonverbal exchange. Psychological Review, 89,
231–249. http://dx.doi.org/10.1037/0033-295X.
89.3.231
Patterson, M. L. (1983). Theoretical approaches to
nonverbal exchange: A brief historical perspective.
Academic Psychology Bulletin, 5, 375–388.
Patterson, M. L. (1995). A parallel process model of
nonverbal communication. Journal of Nonverbal
Behavior, 19, 3–29. http://dx.doi.org/10.1007/
BF02173410
Patterson, M. L. (2008). Back to social behavior:
Mining the mundane. Basic and Applied Social
Psychology, 30, 93–101. http://dx.doi.org/10.1080/
01973530802208816
Patterson, M. L. (2011). More than words: The power of
nonverbal communication. Barcelona, Spain: Editorial
Aresta.
Patterson, M. L., Iizuka, Y., Tubbs, M. E., Ansel,
J., Tsutsumi, M., & Anson, J. (2007). Passing
encounters East and West: Comparing Japanese and
American pedestrian interactions. Journal of
Nonverbal Behavior, 31, 155–166. http://dx.doi.
org/10.1007/s10919-007-0028-4
Patterson, M. L., Lammers, V. M., & Tubbs, M. E. (2014).
Busy signal: Effects of mobile device usage on
pedestrian encounters. Journal of Nonverbal Behavior,
38, 313–324. http://dx.doi.org/10.1007/s10919-014-
0182-4
Patterson, M. L., Mullens, S., & Romano, J. (1971).
Compensatory reactions to spatial intrusion.
Sociometry, 34, 114–121. http://dx.doi.org/10.2307/
2786354
Patterson, M. L., & Tubbs, M. E. (2005). Through a glass
darkly: Effects of smiling and visibility on recognition
and avoidance in passing encounters. Western
Journal of Communication, 69, 219–231. http://dx.doi.
org/10.1080/10570310500202389
Patterson, M. L., Webb, A., & Schwartz, W. (2002).
Passing encounters: Patterns of recognition and
avoidance in pedestrians. Basic and Applied Social
Psychology, 24, 57–66. http://dx.doi.org/10.1207/
S15324834BASP2401_5
Pellegrini, R. J., Schauss, A. G., & Miller, M. E. (1981). Room
color and aggression in a criminal detention holding
cell: A test of the “tranquilizing pink” hypothesis.
Journal of Orthomolecular Psychiatry, 10, 174–181.
Pickett, C. L., Gardner, W. L., & Knowles, M. (2004).
Getting a cue: The need to belong and enhanced
sensitivity to social cues. Personality and Social
Psychology Bulletin, 30, 1095–1107. http://dx.doi.org/
10.1177/0146167203262085
217
Patterson and Quadflieg
Postmes, T., & Spears, R. (1998). Deindividuation
and antinormative behavior: A meta-analysis.
Psychological Bulletin, 123, 238–259. http://dx.doi.
org/10.1037/0033-2909.123.3.238
Profusek, P. J., & Rainey, D. W. (1987). Effects of Baker-
Miller pink and red on state anxiety, grip strength,
and motor precision. Perceptual and Motor Skills, 65,
941–942. http://dx.doi.org/10.2466/pms.1987.65.3.941
Riess, M., & Rosenfeld, P. (1980). Seating preferences
as nonverbal communication: A self-presentational
analysis. Journal of Applied Communication
Research, 8, 22–30. http://dx.doi.org/10.1080/
00909888009360267
Rikowski, A., & Grammer, K. (1999). Human body
odour, symmetry and attractiveness. Proceedings of
the Royal Society B: Biological Sciences, 266, 869–874.
http://dx.doi.org/10.1098/rspb.1999.0717
Roberti, J. W. (2004). A review of behavioral and
biological correlates of sensation seeking. Journal of
Research in Personality, 38, 256–279. http://dx.doi.
org/10.1016/S0092-6566(03)00067-9
Roberts, S. C., Kralevich, A., Ferdenzi, C., Saxton, T. K.,
Jones, B. C., DeBruine, L. M., . . . Havlicek, J. (2011).
Body odor quality predicts behavioral attractiveness in
humans. Archives of Sexual Behavior, 40, 1111–1117.
http://dx.doi.org/10.1007/s10508-011-9803-8
Robson, S. K. A. (2008). Scenes from a restaurant:
Privacy regulation in stressful situations. Journal of
Environmental Psychology, 28, 373–378. http://dx.doi.
org/10.1016/j.jenvp.2008.03.001
Rohles, F. H. (1967). Environmental psychology: A
bucket of worms. Psychology Today, 1, 54–62.
Schaeffer, G. H., & Patterson, M. L. (1980). Intimacy,
arousal, and small group crowding. Journal of
Personality and Social Psychology, 38, 283–290.
http://dx.doi.org/10.1037/0022-3514.38.2.283
Schaller, M., Park, J. H., & Mueller, A. (2003). Fear of the
dark: Interactive effects of beliefs about danger and
ambient darkness on ethnic stereotypes. Personality
and Social Psychology Bulletin, 29, 637–649. http://
dx.doi.org/10.1177/0146167203029005008
Schauss, A. G. (1985). The physiological effect of color
on the suppression of human aggression: Research
on Baker-Miller pink. International Journal of
Biosocial Research, 7, 55–64.
Schopler, J., & Stockdale, J. (1977). An interference
analysis of crowding. Environmental Psychology and
Nonverbal Behavior, 1, 81–88. http://dx.doi.org/
10.1007/BF01145457
Schwarz, S., & Singer, M. (2013). Romantic red revisited:
Red enhances men’s attraction to young, but not
menopausal women. Journal of Experimental Social
Psychology, 49, 161–164. http://dx.doi.org/10.1016/
j.jesp.2012.08.004
218
Sedikides, C. (1993). Assessment, enhancement, and
verification determinants of the self-evaluation
process. Journal of Personality and Social Psychology,
65, 317–338. http://dx.doi.org/10.1037/0022-
3514.65.2.317
Semin, G. R., & Smith, E. R. (Eds.). (2008).
Embodied grounding. http://dx.doi.org/10.1017/
CBO9780511805837
Seubert, J., Kellermann, T., Loughead, J., Boers, F., Brensinger,
C., Schneider, F., & Habel, U. (2010). Processing of
disgusted faces is facilitated by odor primes: A functional
MRI study. NeuroImage, 53, 746–756. http://dx.doi.
org/10.1016/j.neuroimage.2010.07.012
Simpson, J. A., Rholes, W. S., & Phillips, D. (1996).
Conflict in close relationships: An attachment
perspective. Journal of Personality and Social
Psychology, 71, 899–914. http://dx.doi.org/10.1037/
0022-3514.71.5.899
Sinclair, S., Lowery, B. S., Hardin, C. D., & Colangelo,
A. (2005). Social tuning of automatic racial
attitudes: The role of affiliative motivation. Journal
of Personality and Social Psychology, 89, 583–592.
http://dx.doi.org/10.1037/0022-3514.89.4.583
Smith, P. K., & Bargh, J. A. (2008). Nonconscious
effects of power on basic approach and avoidance
tendencies. Social Cognition, 26, 1–24. http://dx.doi.
org/10.1521/soco.2008.26.1.1
Snyder, M., & Ickes, W. (1985). Personality and social
behavior. In G. Lindzey & E. Aronson (Eds.),
Handbook of social psychology (3rd ed., pp. 883–947).
New York, NY: Random House.
Sommer, R. (1959). Studies in personal space. Sociometry,
22, 247–260. http://dx.doi.org/10.2307/2785668
Sommer, R. (1962). The distance for comfortable conver­
sation: A further study. Sociometry, 25, 111–116. http://
dx.doi.org/10.2307/2786041
Sommer, R. (1969). Personal space: The behavioral basis of
design. Englewood Cliffs, NJ: Prentice Hall.
Sommer, R., & Ross, H. (1958). Social interaction in
a geriatric ward. International Journal of Social
Psychiatry, 4, 128–133. http://dx.doi.org/10.1177/
002076405800400207
Spencer, S. J., Steele, C. M., & Quinn, D. M. (1999).
Stereotype threat and women’s math performance.
Journal of Experimental Social Psychology, 35, 4–28.
http://dx.doi.org/10.1006/jesp.1998.1373
Stamps, A. E. (2010). Effects of permeability on perceived
enclosure and spaciousness. Environment and
Behavior, 42, 864–886. http://dx.doi.org/10.1177/
0013916509337287
Steidle, A., Hanke, E.-V., & Werth, L. (2013). In the
dark we cooperate: The situated nature of embodied
procedures. Social Cognition, 31, 275–300. http://
dx.doi.org/10.1521/soco.2013.31.2.275

Steinmetz, J., & Mussweiler, T. (2011). Breaking the
ice: How physical warmth shapes social comparison
consequences. Journal of Experimental Social
Psychology, 47, 1025–1028. http://dx.doi.org/
10.1016/j.jesp.2011.03.022
Stokols, D. (1972). On the distinction between density
and crowding: Some implications for future research.
Psychological Review, 79, 275–277. http://dx.doi.
org/10.1037/h0032706
Stone, N. J. (2001). Designing effective study
environments. Journal of Environmental Psychology, 21,
179–190. http://dx.doi.org/10.1006/jevp.2000.0193
Stone, N. J., & English, A. J. (1998). Task type, posters,
and workspace color on mood, satisfaction,
and performance. Journal of Environmental
Psychology, 18, 175–185. http://dx.doi.org/10.1006/
jevp.1998.0084
Sundstrom, E. (1978). Crowding as a sequential process:
Review of research on the effects of population
density of humans. In A. Baum & Y. M. Epstein
(Eds.), Human responses to crowding (pp. 31–116).
Hillsdale, NJ: Erlbaum.
Szalma, J. L., & Hancock, P. A. (2011). Noise effects
on human performance: A meta-analytic synthesis.
Psychological Bulletin, 137, 682–707. http://dx.doi.
org/10.1037/a0023987
Tajadura-Jiménez, A., Pantelidou, G., Rebacz, P.,
Västfjäll, D., & Tsakiris, M. (2011). I-space: The
effects of emotional valence and source of music on
interpersonal distance. PLoS ONE, 6, e26083. http://
dx.doi.org/10.1371/journal.pone.0026083
Ten Velden, F. S., Baas, M., Shalvi, S., Preenen, P. T. Y., &
De Dreu, C. K. W. (2012). In competitive interaction
displays of red increase actors’ competitive approach
and perceivers’ withdrawal. Journal of Experimental
Social Psychology, 48, 1205–1208. http://dx.doi.
org/10.1016/j.jesp.2012.04.004
Trawalter, S., Richeson, J. A., & Shelton, J. N. (2009).
Predicting behavior during interracial interactions:
A stress and coping approach. Personality and Social
Psychology Review, 13, 243–268. http://dx.doi.
org/10.1177/1088868309345850
Triandis, H. C., Bontempo, R., Villareal, M. J., Asai, M., &
Lucca, N. (1988). Individualism and collectivism:
Cross-cultural perspectives on self-ingroup
relationships. Journal of Personality and Social
Psychology, 54, 323–338. http://dx.doi.org/10.1037/
0022-3514.54.2.323
Valdez, P., & Mehrabian, A. (1994). Effects of color
on emotions. Journal of Experimental Psychology:
General, 123, 394–409. http://dx.doi.org/10.1037/
0096-3445.123.4.394
van den Berg, Y. H. M., Segers, E., & Cillessen, A. H. N.
(2012). Changing peer perceptions and victimization
The Physical Environment and Nonverbal Communication
through classroom arrangements: A field experiment.
Journal of Abnormal Child Psychology, 40, 403–412.
http://dx.doi.org/10.1007/s10802-011-9567-6
Vielhauer, J. (1965). The development of a semantic
scale for the description of the physical environment
(Unpublished doctoral dissertation). Louisiana State
University, Baton Rouge.
Vrij, A., Semin, G. R., & Bull, R. (1996). Insight into
behavior displayed during deception. Human
Communication Research, 22, 544–562. http://dx.doi.
org/10.1111/j.1468-2958.1996.tb00378.x
Wagner, A. M., & Houlihan, D. (1994). Sensation seeking
trait anxiety in hang-glider pilots and golfers.
Personality and Individual Differences, 16, 975–977.
http://dx.doi.org/10.1016/0191-8869(94)90240-2
Webb, E., Campbell, D., Schwartz, R., & Sechrest, L.
(1966). Unobtrusive measures: Nonreactive research in
the social sciences. Chicago, IL: Rand McNally.
Weinstein, N. D. (1976). Human evaluations of
environmental noise. In K. Craik & E. H. Zube
(Eds.), Perceiving environmental quality
(pp. 229–252). http://dx.doi.org/10.1007/978-1-
4684-2865-0_12
Werthman, C., & Piliavin, I. (1967). Gang members
and the police. In D. Bordua (Ed.), The police: Six
sociological essays (pp. 56–98). New York, NY:
Wiley.
Wesselmann, E. D., Cardoso, F. D., Slater, S., & Williams,
K. D. (2012). To be looked at as though air: Civil
attention matters. Psychological Science, 23, 166–168.
http://dx.doi.org/10.1177/0956797611427921
White, A. G. (1953). The patient sits down: A clinical
note. Psychosomatic Medicine, 15, 256–257. http://
dx.doi.org/10.1097/00006842-195305000-00005
White, R., & Sutton, A. (2001). Social planning for mall
redevelopment: An Australian case-study. Local
Environment, 6, 65–80. http://dx.doi.org/10.1080/
13549830120024251
Whyte, W. H. (1980). The social life of small urban spaces.
Washington, DC: Conservation Foundation.
Wicker, A. W. (1979). An introduction to ecological
psychology. Monterey, CA: Brooks/Cole.
Williams, K. D., Shore, W. J., & Grahe, J. E. (1998).
The silent treatment: Perceptions of its behaviors and
associated feelings. Group Processes and Intergroup
Relations, 1, 117–141. http://dx.doi.org/10.1177/
1368430298012002
Williams, L. E., & Bargh, J. A. (2008). Experiencing
physical warmth promotes interpersonal warmth.
Science, 322, 606–607. http://dx.doi.org/10.1126/
science.1162548
Williams, L. E., Huang, J. Y., & Bargh, J. A. (2009).
The scaffolded mind: Higher mental processes
219
Patterson and Quadflieg
are grounded in early experience of the physical
world. European Journal of Social Psychology, 39,
1257–1267. http://dx.doi.org/10.1002/ejsp.665
Wirth, J. H., Sacco, D. F., Hugenberg, K., & Williams,
K. D. (2010). Eye gaze as relational evaluation:
Averted eye gaze leads to feelings of ostracism
and relational devaluation. Personality and Social
Psychology Bulletin, 36, 869–882. http://dx.doi.
org/10.1177/0146167210370032
Wittenbrink, B., Judd, C. M., & Park, B. (2001).
Spontaneous prejudice in context: Variability
in automatically activated attitudes. Journal of
Personality and Social Psychology, 81, 815–827.
http://dx.doi.org/10.1037/0022-3514.81.5.815
Wollin, D. D., & Montagne, M. (1981). College
classroom environment: Effects of sterility versus
amiability on student and teacher performance.
Environment and Behavior, 13, 707–716. http://
dx.doi.org/10.1177/0013916581136004
Worchel, S., & Teddie, C. (1976). The experience of
crowding: A two-factor theory. Journal of Personality
and Social Psychology, 34, 30–40. http://dx.doi.
org/10.1037/0022-3514.34.1.30
Zebrowitz, L. A. (1997). Reading faces: Window to the
soul? Boulder, CO: Westview Press.
220
Zebrowitz, L. A., & Collins, M. A. (1997). Accurate social
perception at zero acquaintance: The affordances
of a Gibsonian approach. Personality and Social
Psychology Review, 1, 204–223. http://dx.doi.
org/10.1207/s15327957pspr0103_2
Zebrowitz, L. A., & Montepare, J. M. (1992). Impressions
of babyfaced individuals across the life span.
Developmental Psychology, 28, 1143–1152. http://
dx.doi.org/10.1037/0012-1649.28.6.1143
Zhong, C.-B., Bohns, V. K., & Gino, F. (2010). Good lamps
are the best police: Darkness increases dishonesty
and self-interested behavior. Psychological Science, 21,
311–314. http://dx.doi.org/10.1177/0956797609360754
Zhong, C.-B., & House, J. (2012). Hawthorne revisited:
Organizational implications of the physical work
environment. Research in Organizational Behavior, 32,
3–22. http://dx.doi.org/10.1016/j.riob.2012.10.004
Zhong, C.-B., & Leonardelli, G. J. (2008). Cold and
lonely: Does social exclusion literally feel cold?
Psychological Science, 19, 838–842. http://dx.doi.
org/10.1111/j.1467-9280.2008.02165.x
Zuckerman, M., Miserandino, M., & Bernieri, F. J. (1983).
Civil inattention exists—In elevators. Personality and
Social Psychology Bulletin, 9, 578–586. http://dx.doi.
org/10.1177/0146167283094007
 Chapter 9
Appearance and Physiognomy
Daniel E. Re and Nicholas O. Rule
The face may be the richest source of nonverbal
human communication. Facial appearance purveys
reliable information about a diverse array of human
characteristics, from one’s health and quality as a
potential mating partner to complex cognitive states
and traits, such as one’s emotions, attitudes, and
propensity for particular behaviors. The face plays
such an enormous role in social interaction that the
human brain contains regions in the visual system
that may specialize in processing facial stimuli
(Hadjikhani, Kveraga, Naik, & Ahlfors, 2009;
Kanwisher, Chun, McDermott, & Ledden, 1996;
Sergent, Ohta, & MacDonald, 1992). Indeed,
humans selectively attend to faces over other types
of stimuli (Ro, Russell, & Lavie, 2001; Theeuwes &
Van der Stigchel, 2006), and even infants prefer
face-like stimuli over other visual patterns as early
as the first few minutes after birth (Goren, Sarty, &
Wu, 1975).
Faces have an immense impact on human life, and
so it may come as no surprise that an entire field of sci-
entific research is dedicated to uncovering the variety
of effects that facial appearance has upon social func-
tioning. In the following chapter, we discuss the cur-
rent state of several fields of face research. We examine
the multifaceted nature of facial attractiveness, includ-
ing contemporary theories on the facial characteristics
that influence perceptions of beauty. We then visit the
ancient topic of physiognomy—the art of evaluating
personality from facial appearance—and detail the
resurgence of this field in the studies of contemporary
researchers. In the following section, we analyze the
validity of face research using two-dimensional (2D)
http://dx.doi.org/10.1037/14669-009
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
images and compare results to recent studies using
video stimuli and three-dimensional (3D) face image
software. Finally, we discuss some new directions in
face research at the helm of taking physiognomy to
places never previously considered (for discussion
of research on facial morphology and expressions,
see Chapter 10, this handbook).
FACIAL ATTRACTIVENESS
Physical attractiveness, or beauty, influences almost
every aspect of human social life. Although one
may like to think that attractiveness holds no
advantages here in the 21st century, this is not the
case. In fact, it can be alarming just how far the
effects of attractiveness pervade society. It may
not be surprising that attractive people are more
strongly desired as sexual partners or spouses
(Buss & Barnes, 1986). What is surprising, how-
ever, is that attractive people are viewed more
favorably elsewhere as well—attractive students
are viewed as more competent by teachers (Ritts,
Patterson, & Tubbs, 1992), attractive people are
more likely to be hired for jobs (Toledano, 2013),
attractive employees are given better performance
reviews and earn higher incomes (Mobius &
Rosenblat, 2006; Toledano, 2013), attractive political
leaders gain more electoral votes (Banducci, Karp,
Thrasher, & Rallings, 2008; Berggren, Jordahl, &
Poutvaara, 2010; King & Leigh, 2009), and so on.
In fact, attractive people are viewed more positively,
in general, a phenomenon known as the attractiveness
halo effect (Dion, Berscheid, & Walster, 1972).
221
Re and Rule
Given the effects of attractiveness in society,
it is no surprise that facial attractiveness has been
the focus of a wealth of empirical research. Studies
on facial attractiveness continue unabated to this
day, with state-of-the-art face-processing technol-
ogy providing researchers with a closer look at
facial features and characteristics that affect how
attractive a person looks. Certain aspects of facial
attractiveness are specific to a particular geographic
region or culture. For example, it is not uncom-
mon to see deliberate facial disfigurements in par-
ticular cultures—for example, teeth blackening
as a practice among some peoples in Borneo, the
traditional insertion of large lip plates among the
Sara people of Central Africa, or even the facial
piercings that frequently adorn the faces of North
American youths. The effects of these external adorn-
ments on social perception can be significant and
may vary by temporal, geographic, or cultural norms
(Cash, Dawson, Davis, Bowen, & Galumbeck, 1989;
­Mulhern, Fieldman, Hussey, Lévêque, & Pineau,
2003; S­ ecord, 1958).
However, there are many natural facial character-
istics that have documented effects on perceptions
of attractiveness across cultures. In this chapter, we
focus on these aspects of attractiveness. Given their
universal nature, it is perhaps natural to think that
these elements provide some insight to a person’s
potential quality as a mating partner. In the follow-
ing section, we discuss research on several facial
characteristics that influence attractiveness, and we
evaluate theories of why these features may affect
perceptions of mate quality.
Skin Condition
It may seem obvious that skin condition should
have a great effect on how attractive a face appears.
After all, the skin covers the surface of the face, and
it would be very difficult to make attractiveness
judgments based on face shape and structure while
disregarding the appearance of the skin. Surpris-
ingly, though, skin condition has not received as
much empirical attention as other aspects of facial
attractiveness, such as averageness, symmetry, and
dimorphism (discussed later in this section).
Skin condition influences perceptions of age and
health (Barber, 1995), whereas skin abnormalities
222
(such as lesions and growths) may indicate poor
health or a compromised genetic disposition
(Symons, 1995).
A homogeneous distribution of skin color
across the face enhances attractiveness such that
faces with evenly distributed coloration appear
younger and healthier than those with more varia-
tion in their coloring (Fink, Grammer, & Matts,
2006; Matts, Fink, Grammer, & Burquest, 2007).
In fact, increasing color homogeneity can make a
face appear 5 years younger (Fink & Matts, 2008),
and applying the difference in skin color between
groups of younger and older adults to a single face
alters perceived age accordingly (a process known
as transforming that is often done with computer
software in face perception studies; similar trans-
formation techniques can manipulate any specified
face characteristic and are discussed throughout
this chapter). Skin texture also impacts attractive-
ness such that people with greater homogeneity
in skin texture are perceived to be better look-
ing (Fink, Grammer, & Thornhill, 2001). When
researchers increase skin color ­homogeneity along
with skin topography homogeneity (e.g., the
evening out of wrinkles), age perceptions can
decrease as much as 15 years (Fink & Matts,
2008). These studies suggest that the distribu-
tion of skin color and texture alone affect face
perception—homogeneous skin color and texture
across a face make a person appear younger,
healthier, and more attractive.
Previous studies have examined the role of skin
color distribution on attractiveness. The specific
coloration that affects attractiveness, however, is a
topic of recent burgeoning interest. So far, research-
ers have identified three main color components of
attractiveness: redness, yellowness, and lightness
(Stephen, Law Smith, Stirrat, & Perrett, 2009). Skin
color and reflectance can be measured electroni-
cally using calibrated spectrophotometers under
standardized lighting conditions, recording separate
values for redness, yellowness, and lightness. Skin
redness is indicative of oxygenated blood levels that
increase with respiratory health (Armstrong &
Welsman, 2001). High physical fitness increases
skin redness (Johnson, 1998), as does estrogen in
women (Thornton et al., 2006). There is evidence

that testosterone increases skin redness in some non-
human primate species as well (in rhesus macaques:
L. Rhodes et al., 1997; in male mandrills: Setchell &
Dixson, 2001). Thus, high levels of redness in the
skin may indicate high mate quality, as suggested by
the handicap hypothesis (Folstad & Karter, 1992)
discussed in more depth later. In contrast, high
levels of deoxygenated blood give the skin a bluish
color associated with cardiac and respiratory illness
(Ponsonby, Dwyer, & Couper, 1997). Skin redness
therefore acts as an indicator of health and mate
quality. When given the chance to transform face
redness in a realistic fashion (simulating blood oxy-
genation levels) to optimize perceived health, par-
ticipants in one study increased oxygenated blood
levels for 98% of all faces (Stephen, Law Smith,
et al., 2009). It should be noted that participants
increased redness to a realistic point, as increasing
redness to too great a degree decreases perceived
health. Furthermore, participants increased redness
more for faces that were lower in starting redness.
Preferences for skin redness are not simply a pref-
erence for the color red (such as seen in clothing
preferences; Beall & Tracy, 2013; Elliot & Niesta,
2008; Elliot et al., 2010) but, rather, seem to reflect
coloration indicative of high cardiovascular fitness
(Re, Whitehead, Xiao, & Perrett, 2011; Stephen,
Law Smith, et al., 2009).
Skin yellowness is altered by carotenoid con-
sumption and melanin (Edwards & Duntley,
1939; Stamatas, Zmudzka, Kollias, & Beer, 2004).
Carotenoid pigmentation affects the dermis, thus
skin yellowness directly reflects carotenoid level.
Carotenoids are found in fruits and vegetables,
and they are used to help resist free radical dam-
age that occurs when fighting disease (Alaluf et al.,
2001). For example, those afflicted with HIV or
malaria have low carotenoid levels (Friis et al.,
2001). Carotenoid levels are depleted in fight-
ing and averting illness, thus skin yellowness is
an indicator of current immunological capacity.
As with redness, participants in face perception
studies increase yellowness to make a face appear
healthier and more attractive (Stephen, Coetzee,
Law Smith, & Perrett, 2009). One recent study
examined how real facial skin coloration is altered
with changes in carotenoid consumption over a
Appearance and Physiognomy
6-week period, revealing that an increase in carot-
enoids equivalent to an extra 3.3 portions of fruits
and vegetables per day was enough to make par-
ticipants reliably more attractive (Whitehead, Re,
Xiao, Ozakinci, & Perrett, 2012). The effects of
skin yellowness and carotenoid consumption on
facial attractiveness have been replicated in groups
with different basal skin tone levels (Caucasians,
Asians, and Africans) and across cultures (Coetzee,
Perrett, & Stephen, 2009; Whitehead, Coetzee,
Ozakinci, & Perrett, 2012).
The third skin color component that affects
attractiveness is lightness (pale vs. tanned skin).
Skin lightness is primarily affected by melanin, with
higher melanin pigmentation making skin darker
(but also making skin yellower). Melanin protects
the body from ultraviolet (UV) radiation by filtering
UV rays, reducing the probability of developing skin
cancer and sunburn (Robins, 1991). Melanin also
helps prevent pregnancy defects in women (Omaye,
1993). High melanin levels incur a health cost, how-
ever, as melanin’s UV-filtering properties inhibit Vita-
min D synthesis, which could lead to bone-related
deformities (Jablonski & Chaplin, 2000). In general,
women have lighter skin than men (Edwards &
Duntley, 1939; van den Berghe & Frost, 1986), per-
haps because high levels of Vitamin D are required
during pregnancy for increased calcium absorption
and bone development (Jablonski & Chaplin, 2000).
Stephen, Coetzee, et al. (2009) found that, given
the opportunity, participants increased skin light-
ness (simulating lower melanin levels) to optimize
perceptions of health. In accordance with natural
color dimorphism, participants lightened female
faces more than male faces. Preferences for light skin
in women have been found across cultures (Coetzee
et al., 2012; van den Berghe & Frost, 1986) and may
reflect the need for increased Vitamin D synthesis
(Jablonski & Chaplin, 2000).
As described earlier, skin condition has a great
impact on attractiveness, and it is easy to understand
why. Skin condition is indicative of current cues to
actual physical health, such as cardiovascular ­fitness
and carotenoid level. However, unlike other elements
of attractiveness that we visit later (averageness,
symmetry, dimorphism), skin condition is strongly
related to perceived health (Jones, Little, Burt, &
223
Re and Rule
Perrett, 2004; Stephen, Coetzee, et al., 2009). Fur-
thermore, unlike face structure, skin condition is
plastic and reflects current health, not just genetic
stability or hormonal status during embryonic
development or puberty. All of these factors make
studies on skin condition an exciting aspect of face
perception research. In fact, studies on skin condi-
tion are a good example of face research with direct
application to real life. Recent findings on how
cardiovascular fitness and fruit and vegetable con-
sumption affect attractiveness provide added incen-
tives for people to exercise and practice a healthy
diet—lifestyle changes that could increase individu-
als’ health and have an impact on national health
care costs (Whitehead, Ozakinci, & Perrett, 2012).
Averageness
Galton (1878) first noted that superimposing
images of individual faces created a composite face
that was more attractive than any of the component
faces. More than 100 hundred years later, digital
face-­processing technology has allowed for this
hypothesis to be tested experimentally. Langlois and
Roggman (1990) were among the first to examine
whether average faces were attractive. They pro-
duced averages of groups of 8, 16, or 32 individual
faces and found that the resultant faces were indeed
more attractive than their constituents. They
hypothesized that facial composites were similar
to a mental template of a face, or more “facelike,”
without any of the irregularities that can be found
in an individual face. If this theory were true, it
would mean that a face could not get more attrac-
tive than the most average configuration for a
population. Over time, the idea that average facial
configurations are the most attractive came to
be known as the averageness hypothesis (Perrett,
May, & Yoshikawa, 1994; Valentine, Darling, &
Donnelly, 2004).
Technological problems can arise when averag-
ing faces with computer software, as averaged skin
complexions are smoother (and therefore found to
be more attractive for the reasons discussed earlier;
Benson & Perrett, 1992) and because averaged faces
will not show the effects of any asymmetries present
in an individual face (Alley & Cunningham, 1991).
However, averaged faces are more attractive even
224
when skin texture is held constant (Little & Han-
cock, 2002), and line-drawings tracing the features
of averaged faces (and thus not depicting skin tex-
ture at all) are more attractive than line-drawings of
nonaverage faces (G. Rhodes & Tremewan, 1996).
Furthermore, increasing averageness boosts attrac-
tiveness even when symmetry is digitally controlled
(Jones, DeBruine, & Little, 2007; G. Rhodes, Sum-
ich, & Byatt, 1999), and average face configurations
are more attractive than nonaverage faces even when
only side views (which do not allow for evaluations
of symmetry) are observed (Valentine et al., 2004).
These findings suggest that the effect of averageness
on attractiveness cannot be explained by skin tex-
ture or symmetry alone—indicating that averageness
itself does contribute independently to facial beauty
(see Figure 9.1).
It is important to note that, although average
faces are attractive, the most attractive faces are not
average (DeBruine, Jones, Unger, Little, & Feinberg,
2007). If an average face configuration was most
attractive, the average of an entire population should
be more attractive than the average of any smaller
subset from the same population. A face averaged
from a particular population, however, can be made
more attractive by transforming that face toward
the configuration of the most attractive subset of
the same population (DeBruine et al., 2007). This is
true even if the same transformation makes the face
appear less average-looking. On some level, this is
intuitive: The most attractive face should not just
be mathematically average but, rather, should have
some features that catch the eye and distinguish it
from other faces in one’s environment. Facial char-
acteristics that are perceived as most attractive when
not in an average configuration, such as skin color,
adiposity, and masculinity, are discussed elsewhere
in this chapter.
Despite the fact that an average face is not neces-
sarily the most attractive face possible, a wealth of
empirical evidence supports the fact that facial aver-
ageness is certainly attractive. Why should this be
the case? It is possible that facial averageness is a cue
to a healthy genetic composition. Abnormalities in
the face decrease averageness; they are the opposite
of average. Increased sexual dimorphism (discussed
later) may raise attractiveness beyond an average
 Appearance and Physiognomy

FIGURE 9.1.  Example of an averaged face (right) created by aggregating four individual
female faces (left). Averaged faces tend to be more attractive than their constituent faces, but
averaged faces are not necessarily the most attractive that a face can be.

face configuration, possibly acting as a cue to high
immunocompetence (i.e., the ability to respond nor-
mally to pathogens). However, nonaverage features
that are not due to dimorphic growth may be a cue
to poor genetic quality. Even minor abnormalities
may indicate instability in embryo development
(Hoyme, 1994). Face abnormalities can indicate
poor health (Symons, 1995), thus facial averageness
may represent a lack of health problems.
Despite this theoretical link, however, few stud-
ies have examined the link between facial average-
ness and real measures of health. Ratings of facial
distinctiveness (the opposite of attractiveness)
­negatively correlate with perceptions of health
(G. Rhodes et al., 2001). Furthermore, ratings of the
distinctiveness of 17-year-olds’ faces were found to
correlate with measures of poor childhood health in
men and of poor current health in women, though
these correlations were moderate in size (r = –.28
and –.25, respectively; G. Rhodes et al., 2001).
Thus, although the relationship between facial
averageness and health is somewhat unclear, there
is little question that averageness does influence
attractiveness. Future studies should be conducted
to elucidate the proximate cause for the relationship
between averageness and attractiveness.
Symmetry
In human anatomical terms, symmetry refers to simi-
larity of features across the median sagittal plane
of the body. Asymmetry can be considered either
directional asymmetry or fluctuating asymmetry.
Directional asymmetry is an evolved natural asym-
metry (Thornhill & Møller, 1997), an example of
which is the human heart, which lies on the left side
of the body. Fluctuating asymmetry refers to asym-
metry in a characteristic that is symmetric across a
population but varies across individuals (Palmer &
Strobeck, 1986; Thornhill & Gangestad, 1996).
The term fluctuating is slightly misleading, as ana-
tomical symmetry does not fluctuate within an indi-
vidual (barring some external force such as surgery
or traumatic injury); rather, the degree of symmetry
of a particular characteristic may fluctuate across a
population. For example, eye size for the left and
right eyes is symmetric across a population, and any
asymmetry in this trait in an individual would repre-
sent fluctuating asymmetry. Fluctuating asymmetry

225
Re and Rule
is thought to reflect developmental stability and
is thus an indicator of mate quality (Gangestad &
Simpson, 2000; Møller & Thornhill, 1998). In this
chapter, symmetry refers exclusively to fluctuating
asymmetry.
Early studies on facial symmetry found prefer-
ences for asymmetrical faces (Kowner, 1996;
Langlois, Roggman, & Musselman, 1994; Samuels,
Butterworth, Roberts, Graupner, & Hole, 1994).
These studies, however, produced their symmetrical
stimuli by taking half of a face (divided by a vertical
midline) and mirroring it on the other side, the
product of which is called a chimera. Chimeras
often appear abnormal, as distortions present on
one side of the face are mirrored on the other. Later
studies in facial symmetry used more technologi-
cally advanced computer manipulations to avoid
confounds present in using mirror-image chimeras.
Perrett et al. (1999) produced symmetrical stimuli
by digitally marking face structure landmarks and
by distorting the image surface. By altering the posi-
tions of two paired points on opposite sides of the
vertical midline of a face, they were able to alter
facial symmetry without creating the odd appear-
ance found in chimeras. Thus, they found that more
structurally symmetrical faces were perceived to
be more attractive than faces with lower structural
symmetry; furthermore, structurally symmetrical
faces have been found to be preferred in both natu-
ral and digitally manipulated faces (Gangestad,
Merriman, & Emery Thompson, 2010; Little, Jones,
Burt, & Perrett, 2007; Penton-Voak et al., 2001), a
preference that is retained across cultures (Little,
Apicella, & Marlowe, 2007).
Face symmetry has been found to positively cor-
relate with perceived health (Jones, Little, Feinberg,
et al., 2004; G. Rhodes et al., 2001; although see
Gangestad et al., 2010). Women’s preferences for
symmetry correlate with their own perceptions of
attractiveness (Little, Burt, Penton-Voak, & Perrett,
2001) and are higher during the fertile phase of
their menstrual cycle when judged in a short-term
relationship context but not in a long-term context
(Little & Jones, 2012). These results demonstrate
that facial symmetry may be indicative of heritable
benefits to offspring (i.e., good genes) that are more
appealing to women when they are more likely to
226
conceive (and thus benefit from the mate quality of
their sexual partner).
The studies mentioned earlier suggest that sym-
metry is indicative of health and genetic quality.
Much like averageness, facial symmetry may serve as
a cue to developmental stability. However, whereas
facial asymmetries may be associated with chro-
mosomal disorders (Hoyme, 1994), self-reported
respiratory illnesses (Thornhill & Gangestad, 2006),
and measures of oxidative stress (Gangestad et al.,
2010), several studies have failed to find relation-
ships between symmetry and health (Hönekopp,
Rudolph, Beier, Liebert, & Muller, 2007; G. Rhodes
et al., 2001; Tomkinson & Olds, 2000). Other
studies have found correlations between symmetry
and self-reported health, but the relationships have
been weak or have failed to replicate (Hume &
Montgomerie, 2001; Shackelford & Larsen, 1997).
It is odd that symmetry should have such an effect on
attractiveness yet few discernible links to health. It
is possible that symmetry had a much stronger rela-
tionship with health in more ancient times and that
modern medicine has weakened that relationship
(G. Rhodes, 2006). However, more research needs
to be conducted to determine any possible connec-
tion (past or present) between symmetry and health.
Whatever that connection may be, it is clear that
symmetry does have an influence on facial attrac-
tiveness, preferences for partners, and perceptions
of health—with more symmetrical faces appearing
more attractive, more preferred, and belonging to
healthier people.
Facial Adiposity
Adiposity (defined by Coetzee et al., 2009, as “the
perception of weight in the face,” p. 1701) is among
the most recent face characteristics discovered to
have a profound effect on attractiveness. Although
body weight has a well-documented relationship
with body attractiveness, facial cues to body weight
were not empirically analyzed until recently. Coetzee
et al. (2009) first examined facial adiposity as an
accurate cue to body weight. They found that weight
judgments made from face images alone (or per-
ceived facial adiposity) were an accurate predictor
of actual body mass index (BMI; a measure of body
weight scaled for height). Later studies quantified

measureable dimensions associated with perceived
facial adiposity and found that perimeter-to-area
ratio, width-to-height ratio, and cheek-to-jaw ratio all
independently predicted perceived adiposity for both
female and male faces (Coetzee, Chen, Perrett, &
Stephen, 2010). Furthermore, these ratios showed
similar correlations with actual BMI, with the excep-
tion of perimeter-to-area ratio in female faces. These
results both established facial dimensions that influ-
enced perceived facial adiposity and revealed that
these dimensions were correlated with actual BMI.
Early research found that facial adiposity reflects
actual BMI, and other studies revealed that it has
effects on perceived attractiveness and health. One
study revealed that ratings of facial adiposity pre-
dicted both perceived attractiveness and perceived
health in a quadratic pattern (i.e., faces rated as
average weight were perceived as healthier and
more attractive than those rated as underweight or
overweight; Coetzee et al., 2009). These results dem-
onstrate that facial adiposity is not only an accurate
predictor of actual BMI but that it also alters the per-
ception of attractiveness and health as well as predicts
actual health measures. More recent studies have
given participants the chance to alter facial adiposity
to maximize the perceived health and attractiveness
of given faces. Facial adiposity transforms of indi-
vidual faces can be made using prototypes (averaged
faces used as template-based reference points for the
transformation of individual faces). Because the BMIs
of the individuals used to create the prototypes are
known, and because the average BMI for the proto-
types can be computed, the resulting BMI of an indi-
vidual face after transformation can be also calculated
(see Figure 9.2). One study of British participants
found that women manipulated female faces to repre-
sent a BMI of 19.76 kg/m2 to maximize attractiveness;
yet, they chose a BMI of 20.84 kg/m2 to maximize
perceived health (Coetzee, Re, Perrett, Tiddeman, &
Xiao, 2011). These BMI values were significantly
different, indicating that women show a disparity
in what they believe to be most attractive and most
healthy in terms of facial adiposity (at least in Brit-
ain). Male participants did not show this disparity,
choosing BMI values of 20.01 kg/m2 to maximize
attractiveness and 19.63 kg/m2 to maximize health.
This study indicates that women have different ideas
Appearance and Physiognomy
about what looks healthy and what looks attractive
in other women, a disparity possibly caused by the
unhealthily thin body ideals portrayed in female-
targeted media (Coetzee et al., 2011). Preferences for
low facial adiposity were also found among a sample
of South African women (Coetzee et al., 2012), dem-
onstrating cross-cultural consistency in adiposity
preferences. As Coetzee et al. (2012) noted, however,
South African university students are considered
among the “African elite,” and so preferences for low
adiposity may not extend across all of Africa. Indeed,
the optimally attractive body weight is reported to be
higher for rural South Africans than in Western coun-
tries such as the United Kingdom; however, South
Africans’ preferences for body weight were found
to converge with typical Western preferences after
immigrating to the United Kingdom (Tovée, Swami,
Furnham, & Mangalparsad, 2006).
Much like skin condition, adiposity may serve
as a criterion for mate choice through its reflection
of actual current health. Ratings of facial adiposity
correlate linearly with the self-reported frequency
of respiratory infections and antibiotic use (Coetzee
et al., 2009) as well as physical ailments, such as
a running and congested nose, diarrhea, nausea,
and headaches (Tinlin et al., 2013). The relation-
ship between adiposity and health also appears to
remain constant across time: One study showed
that ratings of adiposity from high school yearbook
photographs predicted individuals’ future weight
and measures of health problems later in life as
well as higher mortality rates (Reither, Hauser, &
Swallen, 2009). Furthermore, facial adiposity may
be associated with mental health problems, as
higher adiposity was found to correlate with self-
reported stress, anxiety, depression, and negative
mood in women (Tinlin et al., 2013). New studies
have found that facial adiposity is negatively cor-
related with immunocompetence (as measured
by levels of Hepatitis B antibody produced in
response to a Hepatitis B vaccination; Rantala et al.,
2013) and that adiposity mediates the relationship
between attractiveness and immunocompetence.
These studies indicate that adiposity serves as a
cue to attractiveness via its strong relationship
with health and immunocompetence—a heritable
genetic trait that can be passed to offspring.
227
Re and Rule

FIGURE 9.2.  Female (upper panels) and male (lower panels) faces transformed in
facial adiposity to represent a body mass index (BMI) of 16 kg/m2 (left panels) and
26 kg/m2 (right panels). Facial adiposity has a strong influence on attractiveness and
correlates with measures of immunocompetence.

Sexual Dimorphism dimorphism affects perceptions of men’s and wom-

The role of sexually dimorphic face shape on attrac-
tiveness has been one of the most interesting and
complex relationships studied in face perception
research. Sexual dimorphism refers to the mascu-
linity or femininity of a face. Preferences for facial
dimorphism have been investigated in dozens of
studies, sometimes with discrepant results. As facial
en’s faces differently, we discuss each sex separately.
Femininity in women’s faces.  Facial feminin-
ity is characterized by large eyes, full lips, a small
and pointed chin, and high cheek bones (see Figure
9.3). Estrogen, the primary female sex hormone, is
largely responsible for the development of feminine

228
 Appearance and Physiognomy

FIGURE 9.3.  Example of a masculinity transform of a male (upper row) and female (lower row)
face, including the original face (center column) and the same face manipulated 50% toward an aver-
age female face shape (–50%; left column) and 50% toward an average male face shape (+50%; right
column). Facial masculinity has been found to influence attractiveness and perceived dominance.

facial features by inhibiting the masculinizing effects
of testosterone, as described later. Law Smith et al.
(2006) found that facial femininity correlated with
levels of circulating estrogen for women in the late-
follicular phase of their menstrual cycle. Estrogen
has been linked to higher success rates in conceiving
(Lipson & Ellison, 1996); thus, facial femininity may
indicate higher reproductive potential in women.
The relationship between women’s facial femi-
ninity and facial attractiveness is clear: Femininity,
as rated by men and women, positively correlates
with attractiveness. Perrett et al. (1998) allowed
participants to masculinize or feminize female faces
by taking the mathematical difference between the
average male and female face shape for a population
and applying some percentage of that difference to
a face. Perrett et al. (1998) found that participants
increased femininity by 24.2% in Caucasian and
10.2% in Japanese female faces to optimize attractive-
ness. Since then, several studies have demonstrated
that a facial structure that is more feminine than
average is attractive in female faces (Cunningham,
1986; Johnston, Hagel, Franklin, Fink, & Grammer,
2001; Koehler, Simmons, Rhodes, & Peters, 2004;
Lee et al., 2014; G. Rhodes, Hickford, & Jeffery,
2000; see Jones, 2014, for review).

229
Re and Rule
Masculinity in men’s faces.  Testosterone, the pri-
mary male sex hormone, has been theorized to have
immunosuppressive effects (see Muehlenbein &
Bribiescas, 2005, for review), though this relation-
ship has been called into question (M. L. Roberts,
Buchanan, & Evans, 2004). Hamilton and Zuk
(1982) proposed that characteristics that indicate
high resistance to infection, and thus high genetic
quality, should be attractive to the opposite sex. In
the case of sexual dimorphism, indicators of high lev-
els of testosterone would “handicap” a man (Zahavi,
1975), and thus only the men with the highest
genotypic quality can develop these features. This
handicap hypothesis (Folstad & Karter, 1992) states
that displays of high testosterone in men indirectly
indicates genotypic quality by signaling the ability
to overcome the hormone’s immunosuppressive
effects. Indeed, indicators of high testosterone are
attractive to females across a great number of mam-
mal species (Andersson, 1994). Furthermore, males
with high testosterone often develop traits useful
for male–male competition, which plays a significant
role in mating success for many mammalian species
(Andersson, 1994) and is thought to have been
a crucial element to human reproductive success
throughout human history (Puts, 2010).
Masculinity in male faces is characterized by
several traits, including a prominent brow ridge, a
squared jaw, and high cheek bones (Fink &
Penton-Voak, 2002; Miller & Todd, 1998; see
Figure 9.3). Men with high testosterone levels (as
measured from saliva samples) are rated as more
masculine (Penton-Voak & Chen, 2004), and,
behaviorally, men with high testosterone smile less
than those with low testosterone (Dabbs, 1997).
Likewise, men with higher facial masculinity show
relatively higher increases in testosterone after com-
petition (Pound, Penton-Voak, & Surridge, 2009).
Furthermore, perceptions of masculinity drawn
from men’s faces correlate with actual grip strength
(Fink, Neave, & Seydel, 2007). Findings like these
indicate that masculinity is clearly a distinguishable
facial characteristic. Unlike the clear association
between femininity and attractiveness in women’s
faces, however, the relationship between masculin-
ity and attractiveness in men’s faces is relatively less
straightforward (see DeBruine, 2014, for review).
230
Though the handicap hypothesis suggests that
women should prefer men with higher testosterone
levels, empirical data on the issue have shown dispar-
ities. Perrett et al. (1998) found that women prefer
feminized men’s faces in both Japanese and Cauca-
sian samples. These results were replicated in differ-
ent populations (Little & Hancock, 2002; G. Rhodes
et al., 2000). Several others have found preferences
for masculinity, however (Cunningham, Barbee, &
Pike, 1990; Grammer & Thornhill, 1994; Scheib,
Gangestad, & Thornhill, 1999), and yet other stud-
ies have found no general preference for measures of
facial masculinity or femininity in men’s faces at all
(Koehler et al., 2004; Lee et al., 2014; Swaddle &
Reierson, 2002; Thornhill & Gangestad, 2006).
The high amount of variation in women’s prefer-
ences for men’s facial masculinity may arise from the
differential benefits attributed to masculine and fem-
inine faces. Whereas a masculine face structure may
be a cue to heritable genetic benefits, masculinity
is also linked to undesirable personality traits and
aggressive behavior, both perceptually and in real
life (this topic is revisited later). Furthermore, men
with high testosterone may be more able to attract
mates, and they have been found to show lower levels
of self-reported monogamous relationship com-
mitment in the real world (Booth & Dabbs, 1993;
Burnham et al., 2003; Gray, Kahlenberg, Barrett,
Lipson, & Ellison, 2002). Feminine facial character-
istics in men correlate with perceptions of positive
personality traits, such as higher parental invest-
ment and faithfulness (Boothroyd, Jones, Burt, &
Perrett, 2007; Perrett et al., 1998). Thus, men’s
facial masculinity and femininity are associated with
very different personality attributions and real-world
behavior. These differential benefits may explain
some of the variation in women’s preferences for
men’s facial dimorphism (a notion described as
trade-off theory; Gangestad & Simpson, 2000).
Masculinity preferences may also vary by con-
text. Women seem to prefer masculine male faces
when “good genes” (as exhibited by a resistance to
the immunosuppressive effects of testosterone) may
take priority over desirable personality traits. For
example, women prefer more masculine faces in a
short-term (primarily sexual) relationship context
than a long-term relationship context (when they

are more likely to prioritize personality traits such
as commitment and paternal interest; Conway,
Jones, DeBruine, & Little, 2010; Little, Burriss, Jones,
DeBruine, & Caldwell, 2008; Little, Cohen, Jones, &
Belsky, 2007; Little & Jones, 2012). Masculinity
preferences have been found when women are in the
most fertile period of their menstrual cycle, when
chances of conceiving are highest (Gildersleeve
et al., 2013; Gildersleeve, Haselton, & Fales, 2014;
Jones et al., 2008; Little & Jones, 2012; Penton-
Voak et al., 1999; though see Harris, 2011, 2013),
and they are reduced before puberty and after
menopause (Little et al., 2010). Likewise, prefer-
ences for feminine male faces are higher during the
nonfertile phase of women’s menstrual cycle—when
hormone levels simulate pregnancy and when male
relationship investment is of greater importance to
women (Jones et al., 2008). Women who perceive
themselves to be attractive (and thus more likely
to retain a high-quality mate) demonstrate greater
preferences for facial masculinity (Little et al., 2001;
Little & Mannion, 2006).
Masculinity preferences are also affected by culture.
Preferences for facial masculinity are higher in cultures
where paternal investment is traditionally low, such as
rural Jamaica (Penton-Voak, Jacobson, & Trivers,
2004), suggesting that women select mates for
genetic benefits when male commitment is unlikely.
Masculinity preferences have also been shown to cor-
relate with indices of women’s sensitivity to pathogen-
related disgust (DeBruine, Jones, Tybur, Lieberman, &
Griskevicius, 2010; as measured by the three-domain
disgust scale: Tybur, Lieberman, & Griskevicius,
2009) and to increase in areas with less access to med-
ical care (Penton-Voak et al., 2004), perhaps due to
the immunocompetence benefits associated with mas-
culinized features. In line with these findings, recent
studies have reported that masculinity preferences
are higher in areas of low national health (DeBruine,
Jones, Crawford, Welling, & Little, 2010) and after
exposure to images of pathogen contagion, such as
an image of a bodily fluid on a white cloth (Little,
DeBruine, & Jones, 2011). More recent work has
also demonstrated that women’s preference for men’s
facial masculinity is higher after exposure to images
of male–male competition and after seeing images of
items of high monetary value, such as expensive cars
Appearance and Physiognomy
and watches (Little, DeBruine, & Jones, 2013). These
results indicate that masculinity preferences can be
enhanced by awareness of male–male aggression and
wealth (when wealth is in abundance, women prefer
cues to good genes over relationship investment).
These results indicate that masculinity preferences
do not just vary between individuals but can be influ-
enced by visual exposure to cues suggesting different
environmental and social contexts.
With masculinity preferences being so highly vari-
able, one must wonder what masculinity indicates in
terms of mate quality. As mentioned, masculinity in
men’s faces is thought to affect attractiveness because
it reflects heritable immunocompetence. Several stud-
ies have found strong correlations between perceived
health and attractiveness (Henderson & Anglin, 2003;
Jones et al., 2001; Kalick, Zebrowitz, Langlois, &
Johnson, 1998; Krupp, DeBruine, & Jones, 2011), but
relatively few empirical studies have assessed the rela-
tionship between facial masculinity and actual health.
G. Rhodes, Chan, Zebrowitz, and Simmons (2003)
found a weak relationship between men’s perceived
facial masculinity and health scores in adolescence
(based on medical examinations and health histories),
although the faces perceived as masculine were not
perceived as attractive—despite the fact they were
perceived as healthy. Thornhill and Gangestad (2006)
used an objective measure of facial masculinity (based
on principal components analysis of face shape) and
found that men’s facial masculinity had a negative
correlation with the self-reported frequency and dura-
tion of respiratory illnesses but showed no correlation
with stomach illnesses. They also found that rated
attractiveness did not correlate with health measures
in their sample. More recently, Gangestad et al. (2010)
found that women’s ratings of men’s facial masculin-
ity were positively related to attractiveness and were
inversely related to measures of oxidative stress.
Although preferences for facial masculinity are
thought to reflect predilections for indicators of
heritable immunocompetence based on displays of
testosterone-related development, recent studies
have questioned whether face preferences are based
on testosterone at all. Moore, Al Dujaili, et al. (2011)
used facial images of people with high and low levels
of both testosterone and cortisol, a stress-related
hormone that also suppresses the immune system.
231
Re and Rule
They discovered that testosterone levels had no
influence on rated attractiveness; however, faces
of people with low cortisol levels were rated as
more attractive than those with high cortisol levels.
Moore, Cornwell, et al. (2011) extended this study
by finding that the effects of cortisol on attractive-
ness were reduced when testosterone levels were
high. They argued that testosterone affects facial
attractiveness indirectly by moderating the effects of
cortisol on attractiveness. Furthermore, one study
found that preferences for low cortisol were ubiq-
uitous across 13 countries, whereas preferences for
high testosterone were only present in countries
with high human development index scores (a scale
of societal development in which higher scores indi-
cate a lower standard of living), such as Cameroon
and Namibia (Moore et al., 2013). Collectively,
these results indicate that preferences for cortisol
cues in the face may be more consistent than those
for testosterone, which vary across different envi-
ronments and cultures. Moreover, another recent
study found that facial adiposity, not masculinity,
mediated the relationship between facial attractive-
ness and immunocompetence (Rantala et al., 2013).
These findings suggest that plastic, flexible cues
such as adiposity may be better indicators of immu-
nocompetence than masculinity, a static cue that is
invariable after puberty.
It is possible that men’s facial attractiveness is
more closely tied to plastic cues to current health,
such as skin color and adiposity, than by a stable fea-
ture such as masculinity. Facial masculinity may con-
ceivably play a larger role in intrasexual competition
between men, and preferences for masculinity may be
due to women desiring competitive men rather than
apparent immunocompetence (Puts, 2010; Scott,
Clark, Boothroyd, & Penton-Voak, 2013). However,
although masculinity may be clearly associated with
perceptions of dominance, it has ties with health
(see G. Rhodes et al., 2003; Thornhill & Gangestad,
2006). It is possible that modern medicine affects the
masculinity–health relationship in ways that were not
possible throughout the majority of human history
(Little, 2013). It is also important to remember that
preferences for masculinity are strongest during times
when women would prioritize genetic benefits in a
partner (i.e., peak fertility, short-term relationships,
232
circumstances of low paternal investment), sug-
gesting that masculinity has some relationship with
heritable genetic traits. Although the relative effects
of masculinity on intrasexual competition and inter-
sexual selection are not entirely clear, it is important
to point out that the effects of a particular cue in one
domain do not preclude its effects in another. Given
the empirical evidence, it is likely that facial mas-
culinity is both a cue to physical dominance and to
mate quality.
Whereas facial femininity has a clear correla-
tion with perceived attractiveness in women’s faces,
the influence of facial masculinity on men’s faces
is less clear. Masculinity preferences are affected
by social context and can change even within an
individual. Despite a volume of studies, the role of
facial masculinity in attractiveness is not understood
comprehensively. That means that although facial
masculinity is one of the most studied aspects of
attractiveness, there is still a great deal of research
to be done, which should produce new and exciting
findings to reconcile apparent disparities in the field.
PERSONALITY ATTRIBUTIONS
FROM FACES
In the previous section, we described why certain
facial characteristics may be attractive; for instance,
face coloration indicates cardiovascular health
and carotenoid level, and facial adiposity relates to
immune function. These features are representative
of health and are, in at least some cases, reliable cues
to mate quality. If facial cues can signify underlying
health and mating condition, this leaves one to won-
der whether they can also reflect cognitive and emo-
tional traits such as personality. In this section, we
examine the evidence that faces can signal particular
personality and behavioral characteristics.
Physiognomy: A Study Gone By?
The idea that one’s traits can be detected from one’s
face is known as physiognomy, a notion that stems
back to ancient times. Among the first prominent
thinkers to contemplate the relationship between
face structure and personality was ­Aristotle, who is
credited with writing the Greek treatise Physiognom-
ica, in which he draws assessments of personality

from similarities between one’s body structure and
animals. For example, a lion has large extremities
and is brave; therefore, a man with large extremi-
ties is also brave (Evans, 1969). The theories behind
physiognomy developed through the Middle Ages
and were later even taught at universities as a skill
that only select practitioners could master. Physiog-
nomists were later outlawed by King Henry VII of
England as charlatans and criminals. Despite this,
the practice continued, and in 1772 the Swiss pastor
Johann C. Lavater published a series of influential
articles titled Essays on Physiognomy in which he
stated that imperfections in the face reflected imper-
fections as a person.
Modern Physiognomy
Although it is obvious that early concepts of physi-
ognomy were misguided (e.g., facial blemishes do
not necessarily signal a terrible person, and look-
ing like a certain type of animal is unlikely to indi-
cate behavioral similarities; though see Zebrowitz
et al., 2011), the idea of facial appearance reflecting
personality traits has not vanished. Indeed, recent
studies suggest that facial characteristics influence
perceptions of personality and may, in fact, be reli-
able indicators of actual personality driven behavior.
Early scientific studies on physiognomy found
very little (if any) evidence for a relationship
between facial features and personality traits
(Cleeton & Knight, 1924). Similar results were
found throughout most of the 1900s (Alley, 1988).
However, more recent studies have brought a differ-
ent line of thought to physiognomy. Whereas early
studies examined relationships between isolated
facial features and personality (e.g., eye size and
impulsiveness; Cleeton & Knight, 1924), modern
studies have examined how overall face shape and
the configuration of features within a face relate
to measures of behavior. This shift has a scientific
basis: Research in face processing has established
that faces are processed largely “holistically” rather
than in a feature-by-feature fashion (Tanaka &
Farah, 1993). Given that faces are processed as a
whole, it follows that perceived social characteristics
are also dependent on whole face stimuli.
Since the theoretical shift toward studying social
judgments as a product of whole faces (and not
Appearance and Physiognomy
individual features), dozens of studies have reported
reliable personality judgments of target faces across
observers. Most of these studies have focused on
perceptions of the Big Five personality traits—
extraversion, agreeableness, conscientiousness,
emotional stability/neuroticism, and openness to
experience (Costa & McCrae, 1985; McCrae &
Costa, 1987). Remarkably, several experiments have
found that, when shown an image of a person’s face,
independent observers will make similar judgments
of these traits (see Albright et al., 1997; Berry &
McArthur, 1985; McArthur & Apatow, 1984; Secord,
Dukes, & Bevan, 1954). That is to say that separate
people agree (or show “consensus”) about how a
person may behave based solely on images of his or
her face. Judgments of personality reach high con-
sensus even among 3-year-old children (Cogsdill,
Todorov, Spelke, & Banaji, 2014).
That there is any consistency in judgments of
personality from faces may be surprising enough;
however, perhaps more interesting is that these
judgments may accurately reflect real personality.
Many studies have now shown that perceptions of
personality drawn from single face images, short
videotapes, or very brief personal encounters align
with the target’s self-reported personality (Ambady,
Hallahan, & Rosenthal, 1995; Berry, 1990; Bond,
Berry, & Omar, 1994; Borkenau & Liebler, 1992a,
1992b, 1993a, 1993b). Accurate personality judg-
ments can even be made from computer-averaged
faces. Penton-Voak, Pound, Little, and Perrett
(2006) collected facial photographs of 294 people
and asked them to complete a self-report of measures
of the Big Five personality traits. They found that
observers made accurate judgments of extraversion
(r = .24), emotional stability (r = .18), and open-
ness (r = .22) for male faces, but only of extraver-
sion (r = .25) for female faces. In a separate study,
they averaged the faces of the 15 people (per sex)
who rated themselves highest, and the 15 people
who rated themselves lowest, for each personal-
ity trait (see Figure 9.4). Penton-Voak et al. found
that the composite made from the people who
self-reported a high degree of a particular trait was
rated higher for that trait than was the compos-
ite made from the people who self-reported a low
degree of the same trait for each of agreeableness,
233
Re and Rule

extraversion, and emotional stability (though the last faces. Although the correlations are small, they are
of these was only found for men’s faces). This study significant—suggesting that there is indeed some
suggests that personality can be perceived from “kernel of truth” to judgments of personality made
digitally synthesized faces, and not just individual from faces.

High Agreeableness High Conscientiousness High Extraversion High Emotional Stability High Openness

Low Agreeableness Low Conscientiousness Low Extraversion Low Emotional Stability Low Openness

High Agreeableness High Conscientiousness High Extraversion High Emotional Stability High Openness

Low Agreeableness Low Conscientiousness Low Extraversion Low Emotional Stability Low Openness

FIGURE 9.4.  Composites averaged from faces of men (top two rows) and women (bottom two rows) rated high and
low for each of the Big Five personality traits (agreeableness, conscientiousness, extraversion, emotional stability,
and openness). Studies indicate that personality can be reliably estimated from faces. From “Personality Judgments
From Natural and Composite Facial Images: More Evidence for a ‘Kernel of Truth’ in Social Perception,” by I. S.
Penton-Voak, N. Pound, A. C. Little, and D. I. Perrett, 2006, Social Cognition, 24, pp. 622–623. Copyright 2006 by
Guilford Press. Adapted with permission.

234

How Do Faces Reflect Personality?
Now that researchers have established that faces con-
tain some element that reflects personality, one must
question how this relationship manifests. How could
an external part of a physical being be in accordance
with what is, in essence, a product of the mind?
Zebrowitz and Collins (1997) suggested four primary
causes, which we briefly describe in the next section.
Because personality is a catch-all term for the cogni-
tive, emotional, attitudinal, and behavioral character-
istics that a person displays, it would be an enormous
undertaking to cover every personality trait (and per-
haps redundant, as many seemingly distinct personal-
ity traits are actually highly correlated; Penton-Voak
et al., 2006; Zebrowitz & Collins, 1997). Oosterhof
and Todorov (2008) used computer modeling tech-
nology to isolate two main components of social
judgments from faces—whether we should approach
or avoid a person (trustworthiness, often based on
emotional expression) and whether the person is
capable of inflicting harm upon us (dominance).
Given this, and for the purposes of brevity, we restrict
our analysis of the causation behind the personality/
facial appearance relationship to one social judgment
that has an enormous impact on human social per-
ception and interaction: perceived dominance.
Physical dominance has had a great impact on
human social interaction throughout history (Puts,
2010). Traumatic injuries found in ancient skel-
etons (such as skull fractures) suggest that physical
conflict was highly prevalent in humanity’s ancestral
environment, likely leading to a large proportion
of mortalities and possibly shaping human social
behavior (Bowles, 2009; Walker, 2001). Gaulin and
Sailer (1984) calculated that the force of a blow in
primates (e.g., a punch in humans) increases as a
cubic function of mass, whereas the ability to resist
a blow increases parabolically with the cross-sectional
width of bone. Larger, more physically dominant pri-
mates are therefore able to inflict disproportionately
more damage than smaller conspecifics. In humans,
size and strength correlate with the frequency of
physical aggression and confrontation (Archer &
Thanzami, 2007; Felson, 1996; Tremblay et al., 1998).
Given the role that aggression and physical con-
frontation have played in shaping human history, it
would have benefited early humans to quickly and
Appearance and Physiognomy
accurately process a potential rival’s physical size
and strength. Faces are especially salient in forming
social judgments of others, perhaps more so than
other body domains (Currie & Little, 2009; Mueser,
Grau, Sussman, & Rosen, 1984; Peters, Rhodes, &
Simmons, 2007), thus it is conceivable that faces
could convey accurate information about physical
dominance. Consistent with this, many studies have
demonstrated that dominance is a trait that can be
perceived from human faces. Ratings of physical dom-
inance have been found to correlate with handgrip
strength (Fink et al., 2007; Windhager, Schaefer, &
Fink, 2011), mid-arm circumference (including the
bicep, a muscle highly correlated with physical
strength; Undurraga et al., 2010), and shoulder
width (Windhager et al., 2011). Sell et al. (2009)
discovered that naive participants could accurately
assess men’s upper body strength (as determined by
weight-lifting measures such as arm curls, abdomi-
nal crunches, chest presses, and super long pulls)
from face images alone.
Consistent with its relationship to physical domi-
nance, facial appearance can also be indicative of per-
ceived and actual dominant behavior. Masculinizing
human faces increases perceived dominance in male
and female faces, whereas making a face more femi-
nine decreases perceived dominance (Boothroyd et al.,
2007; DeBruine et al., 2006; Jones et al., 2010; Main,
Jones, DeBruine, & Little, 2009; Perrett et al., 1998;
see Figure 9.3). People with masculine and dominant-
looking faces are perceived as more aggressive and
threatening (Oosterhof & Todorov, 2008), and men
with facial characteristics associated with dominance
are more likely to be perceived as untrustworthy
(Stirrat & Perrett, 2010). Men with dominant-looking
faces have higher social status (Mueller & Mazur,
1996), and people are more likely to follow the gaze
of dominant-looking individuals (Jones et al., 2010;
Ohlsen, van Zoest, & van Vugt, 2013). Perceptions of
dominance also drive the expectation of displays of
anger and contempt (Hess, Adams, & Kleck, 2005),
and men with dominant-looking faces act less coop-
eratively and less trustworthy in real life (Stirrat &
­Perrett, 2010). Dominance is such an important trait
in humans that reliable perceptions of dominance
can be made from faces with neutral expressions in
as little as 39 ms (Carré, McCormick, & Mondloch,
235
Re and Rule

2009; see also Rule, Adams, Freeman, & Ambady,
2012), and even children as young as 3 years of age
make reliable judgments when asked to rate faces on
a strong/not strong scale (a scale of dominance revised
for children; Cogsdill et al., 2014).
Facial Appearance and Personality—Four
Possible Causal Relationships
There are four possible mechanisms through which
facial appearance and personality typically converge.
Two of these mechanisms theorize a shared underly-
ing state that affects both appearance and person-
ality: a common biological cause and a common
environmental cause. The other two mechanisms
dictate directional causation: that facial appearance
shapes personality and that personality shapes facial
appearance. In the sections that follow, we describe
these four mechanisms in detail, using facial domi-
nance as an illustrative example.
Common biological cause.  The studies mentioned
earlier suggest that perception of dominance and
masculinity correlate with actual measures of forceful
behavior. But how do dominant people get dominant-
looking faces? The answer may lie in the underlying
hormones responsible for both brain development
and physical appearance. For example, testosterone is
important in prenatal brain development (Chowen-
Breed, Steiner, & Clifton, 1989) and is responsible for
masculinization in puberty (August, Grumbach, &
Kaplan, 1972). Testosterone can manifest in personal-
ity traits through enhanced aggression—testosterone
levels correlate with physical and verbal aggression
as well as aggressive responses to provocation and
threat (Archer, 1991; Mattsson, Schalling, Olweus,
Löw, & Svensson, 1980; Mazur & Booth, 1998;
Olweus, Mattsson, Schalling, & Löw, 1980, 1988).
Testosterone affects facial development as well: The
faces of men with high testosterone levels are per-
ceived as more masculine (Penton-Voak & Chen,
2004; Roney, Hanson, Durante, & Maestripieri, 2006)
and more physically dominant (Moore, Al Dujaili,
et al., 2011; Swaddle & Reierson, 2002) than are
the faces of men with lower testosterone levels. One
study found that perceived facial masculinity and
dominance correlated with an indicator of prenatal
testosterone level (2D:4D finger ratio; Neave, Laing,
Fink, & Manning, 2003; see also Chapter 3, this
handbook). In addition, men with masculine facial
structures exhibit greater surges in circulating tes-
tosterone in response to winning competitions than
men with less masculine faces (Pound et al., 2009).
These studies indicate that perceptions of masculin-
ity and dominance correspond with actual measures
of testosterone, a hormone associated with domi-
nant behavior (Archer, 1991; Mattsson et al., 1980;
Mazur & Booth, 1998). Using these examples, it is
possible to conclude that the relationship between
personality and facial appearance may be mediated by
a common hormonal mechanism affecting both.
Common environmental cause.  The biological
basis for the appearance/personality relationship
is compelling but is only one of several possible
explanations for why appearance may reflect per-
sonality. Although internal physiology and endocri-
nology may alter personality and appearance, it is
also possible that these characteristics are similarly
affected by the external environment. That is, an
individual’s own circumstances may shape his or
her behavior and physical development. Let’s exam-
ine this possibility from the perspective of domi-
nance. Testosterone spikes at puberty, which affects
both facial appearance and personality. Studies
have shown that an unstable environment during
childhood—for instance, growing up in a dangerous
neighborhood, parental conflict, or divorce—leads
to the early onset of puberty (Belsky, Steinberg, &
Draper, 1991; Wierson, Long, & Forehand, 1993).
An influx of testosterone would lead to masculine,
dominant facial features at a younger age. At the
same time, environmental instability may lead to
the development of an aggressive, anxious, and con-
frontational personality. Thus, an individual’s own
circumstances could create concordant physical and
behavioral characteristics independent of biologi-
cal underpinnings. Like the previous example of
the influence of hormones, facial appearance and
personality could develop separately, yet converge to
show congruence across domains.
Facial appearance shapes personality.  The bio-
logical and environmental explanations both illus-
trate how personality and appearance could develop
simultaneously. It is also possible that there are direct

236

causal relationships between the two. For example,
facial appearance itself may influence personality.
Using dominance as a model once again, it is con-
ceivable that some people are born or develop faces
that purvey the appearance of dominance by chance.
The appearance may be initially unrelated to any
actual personality disposition. If, however, other
people react to a dominant-looking person in a fear-
ful or submissive way, and this reaction is consistent
throughout a person’s life, it is possible that the
person may learn to act aggressively and to force his
or her will on others, thus developing a dominant
personality over time (i.e., a self-fulfilling prophecy
effect; Zebrowitz & Collins, 1997).
Personality shapes facial appearance.  Just as
appearance could dictate personality, the opposite
could also occur. It is possible that a person’s face
could be molded by his or her mindset. Let’s take our
example of dominance once more. If a person has a
dominant disposition and regularly acts with aggres-
sion and initiates confrontation, he or she would be
prone to frequent facial expressions of anger. One
characteristic of a face that drives perceptions of
dominance is the furrowing of the brow, or lowering
the eyebrows closer to the eyes, which could lead to
interbrow wrinkles and hypertrophy of the corruga-
tor supercilii muscles (Keating, 1985; Keating et al.,
1981; Zebrowitz & Lee, 1999). Similarly, frequent
jaw clenching among individuals with dominant
personalities could enhance the mandible muscles,
increasing the appearance of a strong jaw and poten-
tially expediting the early development of apparent
frown lines. In a man, a dominant mindset may
instigate the growth of a beard, a feature of appear-
ance that increases perceived dominance (Neave &
Shields, 2008). Furthermore, aggressive behavior
may lead an individual into physical confrontation,
which may leave visible scars that could also give the
impression of dominance. Thus, a person may have
an inherently dominant personality and subsequently
develop a dominant appearance. Similar relationships
could be formed with other personalities as well
(imagine a happy and approachable person smiling
to the point of developing permanent laugh lines,
etc.). These examples illustrate how one’s frame of
mind could alter his or her external appearance (i.e.,
Appearance and Physiognomy
a Dorian Gray effect; Zebrowitz & Collins, 1997; see
also Malatesta, Fiore, & Messina, 1987).
In all likelihood, it is not one of these theories but,
rather, a combination of all that accounts for the rela-
tionship between facial appearance and p ­ ersonality.
Biological underpinnings could account for similari-
ties between internal and external characteristics,
but it is just as possible that environmental causes
shape both. Likewise, facial cues may encourage
personality through the actions of third parties, and
an individual’s natural disposition could account for
variation in facial appearance. It is conceivable, and
indeed likely, that appearance and personality interact
in a feedback loop—the way one looks could affect
the way one is treated, which, in turn, affects the
chemical processes associated with behavior, which
itself could influence the way one looks. Although
early concepts of physiognomy were rife with unjus-
tified theories and claims, we can conclude that more
recent studies show that the relationship between
facial appearance and personality is not pseudosci-
ence. Indeed, the documented association between
personality and appearance indicates that maybe the
physiognomists had the right idea the whole time but
that their theories just needed a bit of guidance.
Who Can Read Faces, and What Faces
Can They Read?
Many studies have shown consensus about per-
sonality judgments across participants; however, it
seems that some people are better at it than others.
Ambady et al. (1995) found that people who receive
high scores on the Profile of Nonverbal Sensitivity
scale (a measure of one’s ability to recognize the
communication of feelings and attitudes; Rosenthal,
Hall, DiMatteo, Rogers, & Archer, 1979) were more
likely to accurately perceive extraversion and posi-
tive affect in target faces (as measured by self-report
scales) than people with lower scores. Interestingly,
however, people with high sociability scores were
relatively less accurate at judging personality from
faces. Conversely, people show an enhanced ability
to accurately perceive the personality traits of socia-
ble people than less sociable people from faces alone.
Other studies have found that people who are more
confident, are more sociable, and have higher self-
esteem are perceived more accurately than people
237
Re and Rule
who are shy and introverted (Borkenau & Liebler,
1992b). Ambady et al. (1995) qualified these find-
ings by noting that such faces are only more accu-
rately perceived for some characteristics, such as
extraversion and agreeableness, and not necessarily
all personality traits. Penton-Voak et al. (2006) also
found that certain traits were easier to read in male
faces than female faces, whereas others have noted
that women are better at reading personality from
faces than men overall (Ambady et al., 1995).
Recent studies have found that the perception of
facial dominance is affected by individual differences
as well. As dominance may inform which individu-
als to approach and which to avoid (Oosterhof &
Todorov, 2008), it is perhaps unsurprising that people
of large physical stature are less sensitive to facial cues
of dominance than others. For example, tall men and
women are less likely to perceive changes in domi-
nance associated with the masculinization of faces and
voices than are shorter men and women (Watkins,
Jones, & DeBruine, 2010; Watkins, Quist, Smith,
Debruine, & Jones, 2012). Likewise, men who score
higher on the Dominance subscale of the Interna-
tional Personality Item Pool (Goldberg, 1999) are less
likely to perceive masculinized men’s faces as more
dominant (Watkins et al., 2010). Similar results have
been shown for women with high dominance scores
viewing masculinized women’s faces (Watkins et al.,
2012). These studies suggest that just as some people
are better at reading positive personality traits from
faces, others are more sensitive to cues of dominance.
Caveat: Be Wary of Overgeneralization
This section has thus far focused on accurate judg-
ments of personality that can be made from facial
appearance alone. It would be imprudent, however,
to suggest that faces give away all of the information
about a person’s personality. In fact, it is far from the
truth. The studies mentioned earlier have found reli-
able relationships between appearance and personal-
ity; however, these findings usually represent a “kernel
of truth”—that is, there is some predictive validity
of appearance to personality, but only some. Indeed,
reported correlations between personality judgments
are usually minor to moderate (with the most striking
correlation coefficients reaching 0.2–0.4, and most
falling below this range; Zebrowitz & Collins, 1997).
238
Facial appearance can convey an element of an
individual’s personality, but these cues can also lead
to overgeneralization, often with deleterious impli-
cations. For example, attractive people may enjoy
the benefits of a “halo effect” in which their beauty
increases the likelihood of being perceived in a posi-
tive way (Dion et al., 1972), including sociability,
intelligence, and warmth. Although it is possible
that attractive people may be slightly more sociable
than average (likely due to the confidence invoked
by being attractive), the halo effect is essentially
an overgeneralization of positive attributes. Like-
wise, natural resting faces may resemble emotional
expressions simply by chance, which can lead to
overgeneralization of emotional states; for example,
male faces in a neutral state look angrier than female
faces (Adams, Nelson, Soto, Hess, & Kleck, 2012;
Zebrowitz, Kikuchi, & Fellous, 2010), and Caucasian
faces resemble expressions of anger more than Black
or Korean faces (Zebrowitz et al., 2010). In turn,
this overgeneralization can, and indeed does, have
unintended consequences in society. For instance,
attractive students are perceived as more intelligent
and are judged to have higher potential by teach-
ers (Ritts et al., 1992), and attractive people face less
stringent prison sentences than unattractive people
who commit the same crimes (Stewart, 1980). Simi-
larly, resemblance to emotional expressions in the
resting faces of Black or Korean targets contribute to
racial stereotypes among Caucasian perceivers in the
United States (Zebrowitz et al., 2010).
As mentioned earlier, King Henry VII outlawed
physiognomy during his reign, believing the study
to be the practice of charlatans. It is true that the
notions of physiognomy understood in King Henry’s
time were incorrect, and scientific studies in the early
20th century failed to find relationships between
individual facial features and personality traits.
Despite this, more contemporary research suggests
that face judgments of personality can reflect, at least
in small part, aspects of an individual’s personality.
Although it is important not to overgeneralize these
judgments, it does seem that people can decipher
some elements of behavior from appearance. Faces of
people with particular personalities are easier to read,
and some people are better at reading faces than oth-
ers, but one thing is clear—there is some accuracy in

judgments of personality from faces after all. Aristotle
would be proud.
DYNAMIC FACIAL CUES
In most of this chapter, we have focused on research
involving stationary images of faces. These stud-
ies usually present a face with a neutral expression
and ask observers to rate the stimuli for some social
judgment, such as attractiveness or dominance. It is
important to note, however, that faces are viewed in
dynamic motion in day-to-day life. This raises the
question of how much ecological validity studies
using static images of neutral faces actually have. Are
there differences between ratings of stationary images
and the perception of faces in the real world? How do
dynamic and malleable facial cues affect social judg-
ments beyond what can be captured in a still photo-
graph? In this section, we review how dynamic and
expressive facial cues affect social perception, and
compare studies across dynamic and static stimuli.
Face Perception Using Static Versus
Dynamic Stimuli
It is obvious that perceptual differences may arise
between static images of faces and video clips show-
ing dynamic movement, including head motion or
facial expression, of the same face. Static images cap-
ture a face at one point in time and from a particular
perspective (usually straight-on in face research stud-
ies). Such images cannot show an entire face, and
they cannot capture the range of emotional expres-
sion of which a face is capable. In theory, video clips
can present everything that a static photograph
can—they retain all the useful qualities of a still pho-
tograph but also allow observers to view a face in its
natural dynamic state, capturing expressive reactions
to interactions with others. Indeed, people are better
at recognizing faces of famous people from dynamic
video sequences than from individual still frames,
even if there are enough frames to present the face
from the same perspectives as the video (Lander,
Christie, & Bruce, 1999), suggesting that the fluid
whole is greater than the sum of its static parts.
Face recognition may be easier from dynamic face
stimuli than still images, but does perception of facial
cues vary between the two? A handful of studies have
Appearance and Physiognomy
attempted to examine whether social judgments of
static 2D facial photographs correlate with those from
videotapes of the same faces. Rubenstein (2005) took
10-s video clips of 48 women’s faces while they read
a passage with a neutral expression. Male observ-
ers rated attractiveness for both the video and still
frames extracted from the clip. Attractiveness ratings
for the dynamic and static image groups were very
similar when averaged across faces, indicating that
one stimulus type was not inherently more attractive
than the other. Surprisingly, however, attractiveness
ratings of clips and videos of the same face did not
significantly correlate (r = .19, p = .26). That is, faces
that were found attractive in videos were not neces-
sarily found attractive in still images. Furthermore,
although observers rated the images as emotionally
neutral, emotion ratings of the video clips showed
high variation, and this variation correlated with
attractiveness ratings. Thus, attractiveness ratings of
dynamic face stimuli may be affected by perceived
emotion, which may not be reflected in still images.
It is also possible that still images do not afford depth
information the way that dynamic stimuli might,
which could affect perceptions of attractiveness. Other
studies have since failed to find correlations in attrac-
tiveness ratings between static and dynamic stimuli
for male faces (Lander, 2008; Penton-Voak & Chang,
2008), though both of these studies found a relation-
ship for female faces.
The aforementioned studies failed to find signifi-
cant and generalizable correlations between percep-
tions of static and dynamic stimuli of faces. These
studies could have been affected by methodological
issues. Early studies used between-subjects designs,
such that different participants would rate the static
and dynamic versions of the same face. This method
prevents any confounds present in rating the same
face twice, yet it is possible that individual differ-
ences in face preferences could account for variance
in perceived attractiveness. S. C. Roberts et al. (2009)
conducted a comprehensive study to investigate
whether there were discrepancies in attractiveness
ratings between static and dynamic stimuli using
both within- and between-subjects designs. They
found strong and significant correlations for static
and dynamic stimuli across all levels of target and
observer sex (all rs ≥ .73). Nevertheless, correlations
239
Re and Rule
were stronger in the within-subjects condition (when
the same people rated both the static and dynamic
stimuli of the same face) than in the between-­
subjects condition (when different people rated static
and dynamic stimuli of the same face). The discrep-
ancy between ratings of static and dynamic stimuli
was also greatest for women rating male faces. These
findings may explain why previous studies found
nonsignificant correlations between stimulus types
using a between-subjects design with male faces.
More recent studies have manipulated faces in
video clips, similar to techniques used in static images.
One study manipulated faces within video clips and
found that feminized female faces were more attrac-
tive than masculinized female faces, but no preference
was found for feminized versus masculinized male
faces (Morrison, Clark, Tiddeman, & Penton-Voak,
2010). These results replicated across faces acting in
a social manner (the filmed participant was asked to
respond to a question as though to encourage interac-
tion with the asker) or antisocial manner (discourag-
ing interaction). It is important to note, however, that
still images extracted from the videos also failed to
produce masculinity preferences in male faces. Fur-
ther work used similar transforms to examine whether
individual differences may account for masculinity
preferences in video clips of male faces, finding that
women’s own attractiveness predicted preferences
for men’s facial masculinity (O’Connor et al., 2012).
This replicates results from static stimuli demonstrat-
ing that attractive women prefer more masculine men
FIGURE 9.5.  Three-dimensional (3D) scanned images of a female face, shown from center view
and left and right half profiles (three-quarter views); 3D stimuli are advantageous in face percep-
tion studies, as they present faces from various angles and perspectives.
240
(Penton-Voak et al., 2003) and may partly explain
the lack of masculinity preferences found in previous
studies where observers’ attractiveness was not rated
(Morrison et al., 2010). Other studies have found
high agreement between static and dynamic stimuli in
terms of attractiveness (Kościński, 2013; G. Rhodes
et al., 2011), and they have also found that ratings of
averageness, symmetry, and masculinity all correlate
with attractiveness in video clips of men’s faces, much
as they do for static face images (G. Rhodes et al.,
2011), whose faces are perceived as attractive from
video clips also report more sexual partners, similar to
findings in static stimuli (G. Rhodes, Simmons, &
Peters, 2005). Taken together, the results of studies
using dynamic face stimuli suggest that social judg-
ments made from static images correlate with those
made from video clips.
Faces in 2D Versus 3D
The vast majority of face perception studies have used
static, 2D images. Yet, recent advances in computing
and imaging technology have allowed for the wide-
scale use of high-quality 3D stimuli in face research
studies. These faces can be “rotated” on a screen from
side to side to show a face from all angles (see Figure
9.5). Therefore, 3D face processing allows for stimuli
that are more ecologically valid. Some software pro-
grams allow for manipulations of 3D faces in much the
same way as 2D faces. Thus, researches have begun to
use transforms in masculinity and facial adiposity in
studies of 3D stimuli as well (Re, Coetzee, et al., 2011).

Studies using 3D faces have become more preva-
lent in recent years, allowing for comparisons to
results found with 2D faces. In recent research on
face preferences, Coetzee et al. (2011) and Re, Coe-
tzee, et al. (2011) found that participants manipu-
lated adiposity in women’s faces to reflect BMI levels
of 19–20 kg/m2. These results align with similar
findings in 2D faces (Re & Perrett, 2014) and BMI
preferences for overall body physique (Tovée &
Cornelissen, 2001; Tovée, Maisey, Emery, & Cornelis-
sen, 1999; Tovée, Reinhardt, Emery, & Cornelissen,
1998). Such findings show some cross-validation of
results between 3D faces and 2D faces and bodies.
Comparable findings using 2D and 3D stimuli are
reassuring but not conclusive. One study has now
directly examined correlations in social perception
using 2D and 3D stimuli of the same faces. Tigue,
Pisanski, O’Connor, Fraccaro, and Feinberg (2012)
collected 2D and 3D images of 39 women. The faces
were presented to 31 men, with the 3D faces rotating
180° from side to side. They found a strong correla-
tion between ratings of attractiveness for 2D and 3D
versions of the same face (r = .71), indicating that
2D and 3D stimuli supply similar cues to attractive-
ness. Interestingly, however, the 3D stimuli were
rated as significantly more attractive than the 2D
stimuli, on average. Tigue et al. interpreted this find-
ing as an indication that the extra visual information
available in 3D stimuli may increase the perception
of attractiveness (though these results do not seem
to replicate when comparing static images to video
clips). Therefore, 3D face stimuli may not only pro-
duce similar social judgments as 2D stimuli but may
also enhance such perceptions, suggesting that 3D
stimuli may increase ecological validity and thereby
provide evidence to promote the use of 3D faces
in future studies. Further studies could be done to
assess how social judgments made from 3D stimuli
relate to those made from video stimuli—the two
forms of dynamic face presentation.
NEW DIRECTIONS IN CONTEMPORARY
PHYSIOGNOMY
Early concepts of physiognomy, or the study of
personality judgments from faces, were somewhat
misguided, yet more recent theories and methods
Appearance and Physiognomy
find relationships between personality characteris-
tics and facial appearance. The personality dimen-
sions examined in most of these studies have been
fairly basic. Traits such as those discussed in previ-
ous sections, including the Big Five and dominance,
can be measured as continuous variables and can be
applied to any individual (Costa & McCrae, 1985;
Watkins et al., 2010).
Face perception researchers have established
reliable relationships between appearance and basic
personality traits. However, can appearance convey
information about more complex aspects of per-
sonhood? Can facial cues inform perceivers of an
individual’s ability to achieve success? What about
distinct social preferences? Given the existence of
facial cues to personality, it is perhaps not much of
an extension to think that appearance could relay
information about personal preferences and disposi-
tions. Indeed, recent empirical studies have started
to uncover relationships between faces and facets of
identity never before theorized.
Detecting Success From Faces
The Physiognomica discussed how human personality
could be revealed by features shared with particular
animals. Humans with facial characteristics resem-
bling those of a lion, for example, were thought to
be strong and courageous, capable of dominating
rivals (Evans, 1969). Personality may not actually
correlate with animalistic features (though it is still
possible that resembling a particular animal may
affect perceptions of personality; Zebrowitz et al.,
2011), but recent studies suggest that faces may
reveal an individual’s ability to succeed. Several
studies over the past 10 years have examined cor-
relations between attributions of faces and objective
measures of workplace success. Much of this work
has focused on the faces of business leaders, as their
success can be objectively measured through their
companies’ financial performance. Interestingly, sev-
eral studies have demonstrated that business leaders’
success is correlated with perceptions of dominance
and power drawn from images of their faces (Rule &
Ambady, 2008b, 2009). Other studies have found
similar relationships between perceived power and
success among the faces of managing partners of law
firms (Rule & Ambady, 2011), indicating that facial
241
Re and Rule
correlates of success extend across different types
of businesses. Further studies have found that facial
dimensions associated with dominance and aggres-
sion (facial width-to-height ratio; Carré et al., 2009)
also correlate with success in business leaders’ faces
(Wong, Ormiston, & Haselhuhn, 2011). Similarly,
although measures of leadership success are less
defined in politics than they are in business, several
studies have demonstrated that judgments of domi-
nance and competence from political candidates’
faces correlate with electoral success (see Olivola &
Todorov, 2010, for review). Conversely, facial char-
acteristics that make a person appear youthful and
friendly (sometimes referred to as babyfacedness) also
make one appear weak and incompetent (Zebrowitz-
McArthur & Apatow, 1984). In turn, babyfacedness
reduces the likelihood of attaining leadership posi-
tions in politics (Zebrowitz & Montepare, 2005),
and such features are not beneficial to leaders in the
business world (Rule & Ambady, 2008b), though
these patterns may be moderated by race (Livings-
ton & Pearce, 2009) and culture (Rule, Ambady,
et al., 2010). Thus, whereas animalistic features may
not correspond with matching personality attributes,
judgments of power and babyfacedness do predict
success in terms of business and political leadership.
Judging Sexual Orientation From Faces
One study in the field of nonverbal behavior discov-
ered that both heterosexual and homosexual partici-
pants could make judgments of sexual orientation at
higher-than-chance accuracy when presented with
short, silent video clips of individuals (Ambady,
Hallahan, & Conner, 1999). In addition, recent stud-
ies have found that sexual orientation can be judged
more accurately than chance using still photographs
of faces. Rule and Ambady (2008a) reported that
men and women were able to categorize sexual
orientation significantly better than chance when
viewing male faces for 50 ms or longer. Interestingly,
the accuracy of categorization was the same at 50 ms
as it was when participants were able to view faces
for an unrestricted amount of time, suggesting that
sexual orientation is perceived relatively rapidly and
efficiently. Sexual orientation can also be determined
from women’s faces (Rule, Ambady, & Hallett, 2009),
even when cropped tightly to only show the eyes.
242
Although perceptions of sexual orientation are accu-
rate at above-chance levels at nearly unconscious pre-
sentation times (40 ms), accuracy is attenuated when
participants are instructed to think carefully and to
deliberate about their judgments, providing more evi-
dence that the categorization of sexual orientation is
processed automatically (Rule et al., 2009).
Further research has focused on social factors
that affect the perception of sexual orientation. One
study found that American, Spanish, and Japanese
participants were able to accurately categorize sexual
orientation from male faces from all three countries,
demonstrating consistency across cultures (Rule,
Ishii, Ambady, Rosen, & Hallett, 2011). American
perceivers were better at judging sexual orientation
than were Spanish and Japanese perceivers, though
all groups of perceivers were significantly more accu-
rate than chance. Furthermore, Japanese participants
were more likely to categorize someone as straight,
but Spanish perceivers were equally likely to rate a
straight man as gay and a gay man as straight, pos-
sibly reflecting the disparate levels of tolerance for
homosexuality in those countries. Further research
has demonstrated that accuracy in judgments of sex-
ual orientation is consistent across Asian, Black, and
Caucasian perceivers and faces as well (Rule, 2011),
and that straight individuals who have more experi-
ence interacting with gay men are better at catego-
rizing male sexual orientation (Brambilla, Riva, &
Rule, 2013; see also Rule et al., 2015), suggesting
that differences in categorization are based on cul-
tural standards and experience rather than ethnicity.
One further study found that women’s interest
in mating alters the accuracy of judgments of sexual
orientation from men’s faces (Rule, Rosen, Sle-
pian, & Ambady, 2011). Women’s sensitivity to male
sexual orientation varies by menstrual cycle phase,
such that accuracy increases closer to peak ovula-
tion. Women’s sensitivity to sexual orientation in
other women’s faces does not vary across the cycle,
however. Women also are better at categorizing
male sexual orientation when primed to think about
romance and mate choice, regardless of menstrual
cycle phase. These results suggest that automatic
processing of sexual orientation may function to
efficiently select members of one’s gender of interest
that may be eligible for further romantic interaction.

The aforementioned studies indicate that sexual
orientation can be judged with above-chance accu-
racy for both men and women. Recent studies
have begun to examine the physical characteristics
responsible for categorization of sexual orientation
from the face. Inverting sexually dimorphic cues
in the face of a particular sex (i.e., masculinizing a
female face or feminizing a male face in either shape
or texture) increases the likelihood of categoriz-
ing that face as homosexual (Freeman, Johnson,
Ambady, & Rule, 2010). One quantitative analysis
of physical features found that faces of homosexuals
were less symmetrical than faces of heterosexuals,
and that perceptions of sexual preference varied
by face symmetry (Hughes & Bremme, 2011). It is
important to note that although variables extraneous
to the face, such as hairstyle, influence the accu-
racy of judgments of sexual orientation, accurate
judgments can be made when faces are presented
without hair and clothing cues, and even when only
isolated facial features such as eyes or mouths are
displayed (Rule, Ambady, Adams, & Macrae, 2008;
Tskhay, Feriozzo, & Rule, 2013).
Accuracy in Judging Social Group
Membership From Faces
At first glance, it may be surprising that workplace
success and sexual orientation can be correctly cat-
egorized by facial cues alone. However, leadership
success correlates with judgments of dominance
from faces (Rule & Ambady, 2008a), which relates
to actual measures of dominant behavior (Watkins,
2011). Furthermore, some research suggests that
sexual orientation may be influenced by prenatal hor-
mone exposure (Lippa, 2003; Meyer-Bahlburg et al.,
1995), which has also been hypothesized to affect
facial development (Neave et al., 2003). It would
seem, however, that a direct hormonal relationship
between physiology and personality is not required
to purvey accurate social judgments from faces.
For example, studies have found that people can
discern political party affiliation with above-chance
accuracy (e.g., Jahoda, 1954). Using facial stimuli
of American politicians (cropped tightly around the
outer edge of the hair), Rule and Ambady (2010)
found that faces of Republicans were viewed as
more ­powerful (a variable formed from ratings of
Appearance and Physiognomy
dominance and maturity) and that this impression of
power was found to mediate the relationship between
perceived and actual party affiliation. These results
extended to college students who professed alle-
giance to either the Democrat or Republican student
organizations at their university. Further research
found that political attitudes could be judged cross-
culturally, with German and Swiss perceivers cor-
rectly judging left- or right-wing affiliation from the
faces of Swiss and German participants, respectively
(Samochowiec, Wanke, & Fiedler, 2010). Ratings
of dominance again predicted both perceptual and
actual political affiliation, with more dominant-­
looking people tending to have right-wing affilia-
tions (Samochowiec et al., 2010).
Political affiliation is not the only ideologi-
cal group membership that can be perceived from
facial appearance. The mid-20th century saw a pre-
ponderance of research on whether people could
be accurately classified as Jewish from appearance
alone. Jewish people suffered extreme prejudice
and discrimination during World War II, and folk
belief suggested that Jews could be distinguished
from non-Jews by physical appearance (though in
actuality, Nazis had training films on how to identify
Jews based on stereotypes, and Jewish people were
forced to wear clothing indicative of their culture
so that they could be identified in Nazi Germany;
Allport, 1954). A string of empirical studies investi-
gated whether Jews could, in fact, be distinguished
by appearance (Allport & Kramer, 1946; Elliott &
Wittenberg, 1955; Lindzey & Rogolsky, 1950;
Pulos & Spilka, 1961). Two recent meta-analytic
reviews of this research found that people were
able to accurately categorize Jews and non-Jews
from still facial photographs alone (Andrzejewski,
Hall, & Salib, 2009; Rice & Mullen, 2003), and one
study found that similar effects still apply in modern
times (Andrzejewski et al., 2009). The link between
sociocultural group membership and facial appear-
ance was also found for members of the Church of
Jesus Christ of Latter-Day Saints (commonly known
as Mormons). Some Mormons believe they can
distinguish people of their own faith from others.
Interestingly, studies revealed that both Mormon
and non-Mormon perceivers could correctly cat-
egorize faces of Mormons and non-Mormons with
243
Re and Rule
above-chance accuracy (Rule, Garrett, & Ambady,
2010b). Further studies showed that this distinction
may be due to perceptions of health, with Mormons
having healthier looking skin (Rule, Garrett, &
Ambady, 2010a). The latter finding is particularly
interesting, as Mormons are actually healthier than
non-Mormons in the United States (Enstrom &
Breslow, 2008), again suggesting that accuracy in
contemporary physiognomy may stem from real rela-
tionships between physiology and personal lifestyle.
The studies discussed in this section all point to
the same surprising finding: Aspects of workplace
ability and personal lifestyle with no clearly definable
physical characteristics can be discerned from facial
appearance with above-chance accuracy. It should be
pointed out that “above chance” does not necessarily
implicate large effect sizes. For example, categoriz-
ing sexual orientation with 65% accuracy (Tskhay &
Rule, 2013) could still be significantly above chance
but does not represent comprehensive and near-
universal accuracy, as is found in perception of more
obvious categories such as race (correctly categorized
for 99% of faces; Remedios, Chasteen, Rule, & Plaks,
2011). A recent meta-analysis reviewed the literature
on categorization of sexual orientation, political affili-
ation, and religious group membership (i.e., Jewish
and Mormon targets and controls). The mean effect
size for 92 studies using still photographs of faces was
statistically significant (r = .26, 95% CI [.21, .32];
Tskhay & Rule, 2013). A file-drawer analysis (Rosen-
thal, 1979) showed that an estimated 20,000 studies
with an averaged null effect would be needed to make
the current literature barely significant (p = .05).
Furthermore, symmetry analysis of a funnel graph
found no evidence of publication bias (Tskhay &
Rule, 2013). Thus, it is clear that the perception of
ambiguous group membership is a robust and repli-
cable effect, with evidence of accuracy in categorizing
sexual, political, and religious preferences.
Studies done on the perception of leadership suc-
cess and ambiguous group membership have pro-
duced startling results, indicating that propensity for
success and personal preferences may be imparted
through facial traits. Clearly, it would be imprudent to
suggest that one could discern an individual’s ability
to succeed or his or her personal and social disposi-
tion with 100% accuracy, or that a significant level of
244
accuracy would apply to all groups, preferences, or
domains. Despite this, the average effect size in the
meta-analysis of accuracy in categorizing perceptu-
ally ambiguous groups (r = .26) was significantly
larger than recent estimates of the average effect size
of studies across social and personality psychology
in general (r = .21; Richard, Bond, & Stokes-Zoota,
2003). The studies on perception of ambiguous group
membership discussed here may therefore give rise
to a new line of face research examining how social
characteristics previously thought unrelated to facial
appearance might be reflected in the face. Such “kernel
of truth” studies have implications for multiple aspects
of human social life, and they provide a new avenue
for research in contemporary physiognomy.
CONCLUSION
The face is an exceptional source of nonverbal informa-
tion. The research discussed here demonstrates how
facial appearance provides insight into a myriad of per-
sonal characteristics, from health status and mate qual-
ity to personality traits. Advancements in technology
have allowed face researchers to render faces in three
dimensions to enhance the ecological validity of their
studies while simultaneously establishing the legiti-
macy of the 2D face images used in traditional studies.
New research suggests that faces offer information on
personal traits previously considered to be unrelated
to external appearance. There are currently hundreds
of scientists dedicated to revealing the many ways in
which faces affect human social interaction. However,
despite the great many discoveries already made, new
and exciting contributions to the face research literature
continue unabated. In the last section of this chap-
ter, we described experiments that have discovered
new and unique relationships in face perception
research—most of which have been conducted in the
last few years. To this end, do not be surprised if the
contents of this chapter need to be updated to include
many new and exciting findings in the years to come.
References
Adams, R. B., Jr., Nelson, A. J., Soto, J. A., Hess, U., &
Kleck, R. E. (2012). Emotion in the neutral face: A
mechanism for impression formation? Cognition and
Emotion, 26, 431–441. http://dx.doi.org/10.1080/
02699931.2012.666502

Alaluf, S., Heath, A., Carter, N., Atkins, D., Mahalingam, H.,
Barrett, K., . . . Smit, N. (2001). Variation in
melanin content and composition in Type V and
VI photoexposed and photoprotected human
skin: The dominant role of DHI. Pigment Cell
Research, 14, 337–347. http://dx.doi.org/10.1034/
j.1600-0749.2001.140505.x
Albright, L., Malloy, T. E., Dong, Q., Kenny, D. A., Fang,
X., Winquist, L., & Yu, D. (1997). Cross-cultural
consensus in personality judgments. Journal of
Personality and Social Psychology, 72, 558–569.
http://dx.doi.org/10.1037/0022-3514.72.3.558
Alley, T. R. (1988). Social and applied aspects of face
perception: An introduction. Hillsdale, NJ: Erlbaum.
Alley, T. R., & Cunningham, M. R. (1991). Averaged
faces are attractive, but very attractive faces are not
average. Psychological Science, 2, 123–125. http://
dx.doi.org/10.1111/j.1467-9280.1991.tb00113.x
Allport, G. W. (1954). The nature of prejudice. Reading,
MA: Addison-Wesley.
Allport, G. W., & Kramer, B. M. (1946). Some roots of
prejudice. Journal of Psychology, 22, 9–39. http://
dx.doi.org/10.1080/00223980.1946.9917293
Ambady, N., Hallahan, M., & Conner, B. (1999).
Accuracy of judgments of sexual orientation from
thin slices of behavior. Journal of Personality and
Social Psychology, 77, 538–547. http://dx.doi.
org/10.1037/0022-3514.77.3.538
Ambady, N., Hallahan, M., & Rosenthal, R. (1995).
On judging and being judged accurately in zero-
acquaintance situations. Journal of Personality
and Social Psychology, 69, 518–529. http://dx.doi.
org/10.1037/0022-3514.69.3.518
Andersson, M. (1994). Sexual selection. Princeton, NJ:
Princeton University Press.
Andrzejewski, S. A., Hall, J. A., & Salib, E. R. (2009).
Anti-Semitism and identification of Jewish group
membership from photographs. Journal of Nonverbal
Behavior, 33, 47–58. http://dx.doi.org/10.1007/
s10919-008-0060-z
Archer, J. (1991). The influence of testosterone on
human aggression. British Journal of Psychology, 82,
1–28. http://dx.doi.org/10.1111/j.2044-8295.1991.
tb02379.x
Archer, J., & Thanzami, V. (2007). The relation between
physical aggression, size and strength, among
a sample of young Indian men. Personality and
Individual Differences, 43, 627–633. http://dx.doi.
org/10.1016/j.paid.2007.01.005
Armstrong, N., & Welsman, J. R. (2001). Peak oxygen
uptake in relation to growth and maturation in 11-
to 17-year-old humans. European Journal of Applied
Physiology, 85, 546–551. http://dx.doi.org/10.1007/
s004210100485
Appearance and Physiognomy
August, G. P., Grumbach, M. M., & Kaplan, S. L. (1972).
Hormonal changes in puberty: III. Correlation of
plasma testosterone, LH, FSH, testicular size, and
bone age with male pubertal development. Journal of
Clinical Endocrinology and Metabolism, 34, 319–326.
http://dx.doi.org/10.1210/jcem-34-2-319
Banducci, S. A., Karp, J. A., Thrasher, M., & Rallings, C.
(2008). Ballot photographs as cues in low-information
elections. Political Psychology, 29, 903–917. http://
dx.doi.org/10.1111/j.1467-9221.2008.00672.x
Barber, N. (1995). The evolutionary psychology of
physical attractiveness: Sexual selection and human
morphology. Ethology and Sociobiology, 16, 395–424.
http://dx.doi.org/10.1016/0162-3095(95)00068-2
Beall, A. T., & Tracy, J. L. (2013). Women are more
likely to wear red or pink at peak fertility.
Psychological Science, 24, 1837–1841. http://dx.doi.
org/10.1177/0956797613476045
Belsky, J., Steinberg, L., & Draper, P. (1991). Childhood
experience, interpersonal development, and
reproductive strategy: An evolutionary theory of
socialization. Child Development, 62, 647–670. http://
dx.doi.org/10.2307/1131166
Benson, P., & Perrett, D. (1992). Face to face with the
perfect image. New Scientist, 133, 32–35.
Berggren, N., Jordahl, H., & Poutvaara, P. (2010). The
looks of a winner: Beauty and electoral success.
Journal of Public Economics, 94, 8–15. http://dx.doi.
org/10.1016/j.jpubeco.2009.11.002
Berry, D. S. (1990). Taking people at face value:
Evidence for the kernel of truth hypothesis. Social
Cognition, 8, 343–361. http://dx.doi.org/10.1521/
soco.1990.8.4.343
Berry, D. S., & McArthur, L. Z. (1985). Some components
and consequences of a babyface. Journal of
Personality and Social Psychology, 48, 312–323.
http://dx.doi.org/10.1037/0022-3514.48.2.312
Bond, C. F., Jr., Berry, D. S., & Omar, A. (1994). The
kernel of truth in judgments of deceptiveness. Basic
and Applied Social Psychology, 15, 523–534. http://
dx.doi.org/10.1207/s15324834basp1504_8
Booth, A., & Dabbs, J. M. (1993). Testosterone and men’s
marriages. Social Forces, 72, 463–477. http://dx.doi.
org/10.1093/sf/72.2.463
Boothroyd, L. G., Jones, B. C., Burt, D. M., & Perrett,
D. I. (2007). Partner characteristics associated with
masculinity, health and maturity in male faces.
Personality and Individual Differences, 43, 1161–1173.
http://dx.doi.org/10.1016/j.paid.2007.03.008
Borkenau, P., & Liebler, A. (1992a). The cross-modal
consistency of personality: Inferring strangers traits
from visual or acoustic information. Journal of
Research in Personality, 26, 183–204. http://dx.doi.
org/10.1016/0092-6566(92)90053-7
245
Re and Rule
Borkenau, P., & Liebler, A. (1992b). Trait inferences:
Sources of validity at zero acquaintance. Journal
of Personality and Social Psychology, 62, 645–657.
http://dx.doi.org/10.1037/0022-3514.62.4.645
Borkenau, P., & Liebler, A. (1993a). Consensus and
self–other agreement for trait inferences from minimal
information. Journal of Personality, 61, 477–496.
http://dx.doi.org/10.1111/j.1467-6494.1993.tb00779.x
Borkenau, P., & Liebler, A. (1993b). Convergence of
stranger ratings of personality and intelligence
with self-ratings, partner ratings, and measured
intelligence. Journal of Personality and Social
Psychology, 65, 546–553. http://dx.doi.
org/10.1037/0022-3514.65.3.546
Bowles, S. (2009). Did warfare among ancestral
hunter–gatherers affect the evolution of human
social behaviors? Science, 324, 1293–1298. http://
dx.doi.org/10.1126/science.1168112
Brambilla, M., Riva, P., & Rule, N. O. (2013). Familiarity
increases the accuracy of categorizing male sexual
orientation. Personality and Individual Differences, 55,
193–195. http://dx.doi.org/10.1016/
j.paid.2013.02.023
Burnham, T. C., Chapman, J. F., Gray, P. B., McIntyre,
M. H., Lipson, S. F., & Ellison, P. T. (2003). Men
in committed, romantic relationships have lower
testosterone. Hormones and Behavior, 44, 119–122.
http://dx.doi.org/10.1016/S0018-506X(03)00125-9
Buss, D. M., & Barnes, M. (1986). Preferences in
human mate selection. Journal of Personality and
Social Psychology, 50, 559–570. http://dx.doi.
org/10.1037/0022-3514.50.3.559
Carré, J. M., McCormick, C. M., & Mondloch, C. J.
(2009). Facial structure is a reliable cue of aggressive
behavior. Psychological Science, 20, 1194–1198.
http://dx.doi.org/10.1111/j.1467-9280.2009.02423.x
Cash, T. F., Dawson, K., Davis, P., Bowen, M., &
Galumbeck, C. (1989). Effects of cosmetics use
on the physical attractiveness and body image
of American college women. Journal of Social
Psychology, 129, 349–355. http://dx.doi.org/10.1080/
00224545.1989.9712051
Chowen-Breed, J. A., Steiner, R. A., & Clifton, D. K.
(1989). Sexual dimorphism and testosterone-
dependent regulation of somatostatin gene
expression in the periventricular nucleus of the rat
brain. Endocrinology, 125, 357–362. http://dx.doi.
org/10.1210/endo-125-1-357
Cleeton, G., & Knight, F. B. (1924). Validity of character
judgments based on external criteria. Journal of
Applied Psychology, 8, 215–231. http://dx.doi.
org/10.1037/h0072525
Coetzee, V., Chen, J., Perrett, D. I., & Stephen, I. D.
(2010). Deciphering faces: Quantifiable visual cues
246
to weight. Perception, 39, 51–61. http://dx.doi.
org/10.1068/p6560
Coetzee, V., Faerber, S. J., Greeff, J. M., Lefevre, C. E., Re,
D. E., & Perrett, D. I. (2012). African perceptions of
female attractiveness. PLoS ONE, 7, e48116. http://
dx.doi.org/10.1371/journal.pone.0048116
Coetzee, V., Perrett, D. I., & Stephen, I. D. (2009).
Facial adiposity: A cue to health? Perception, 38,
1700–1711. http://dx.doi.org/10.1068/p6423
Coetzee, V., Re, D., Perrett, D. I., Tiddeman, B. P., &
Xiao, D. (2011). Judging the health and attractiveness
of female faces: Is the most attractive level of facial
adiposity also considered the healthiest? Body
Image, 8, 190–193. http://dx.doi.org/10.1016/
j.bodyim.2010.12.003
Cogsdill, E. J., Todorov, A. T., Spelke, E. S., & Banaji, M. R.
(2014). Inferring character from faces: A developmental
study. Psychological Science, 25, 1132–1139. http://
dx.doi.org/10.1177/0956797614523297
Conway, C. A., Jones, B. C., DeBruine, L. M., & Little, A. C.
(2010). Sexual dimorphism of male face shape,
partnership status and the temporal context of
relationship sought modulate women’s preferences for
direct gaze. British Journal of Psychology, 101, 109–121.
http://dx.doi.org/10.1348/000712609X436435
Costa, P. T., & McCrae, R. R. (1985). The NEO
Personality Inventory: Manual, Form S, and Form R.
Lutz, FL: Psychological Assessment Resources.
Cunningham, M. R. (1986). Measuring the physical
in physical attractiveness: Quasi-experiments on
the sociobiology of female facial beauty. Journal
of Personality and Social Psychology, 50, 925–935.
http://dx.doi.org/10.1037/0022-3514.50.5.925
Cunningham, M. R., Barbee, A. P., & Pike, C. L. (1990).
What do women want? Facialmetric assessment
of multiple motives in the perception of male
facial physical attractiveness. Journal of Personality
and Social Psychology, 59, 61–72. http://dx.doi.
org/10.1037/0022-3514.59.1.61
Currie, T. E., & Little, A. C. (2009). The relative
importance of the face and body in judgments of
human physical attractiveness. Evolution and Human
Behavior, 30, 409–416. http://dx.doi.org/10.1016/
j.evolhumbehav.2009.06.005
Dabbs, J. M., Jr. (1997). Testosterone, smiling, and facial
appearance. Journal of Nonverbal Behavior, 21, 45–55.
http://dx.doi.org/10.1023/A:1024947801843
DeBruine, L. M. (2014). Women’s preferences for male
facial features. In V. A. Weekes-Shackelford &
T. K. Shackelford (Eds.), Evolutionary perspectives on
human sexual psychology and behavior (pp. 261–275).
http://dx.doi.org/10.1007/978-1-4939-0314-6_14
DeBruine, L. M., Jones, B. C., Crawford, J. R., Welling,
L. L. M., & Little, A. C. (2010). The health of a nation

predicts their mate preferences: Cross-cultural
variation in women’s preferences for masculinized
male faces. Proceedings of the Royal Society B:
Biological Sciences, 277, 2405–2410. http://dx.doi.
org/10.1098/rspb.2009.2184
DeBruine, L. M., Jones, B. C., Little, A. C., Boothroyd,
L. G., Perrett, D. I., Penton-Voak, I. S., . . . Tiddeman,
B. P. (2006). Correlated preferences for facial
masculinity and ideal or actual partner’s masculinity.
Proceedings of the Royal Society B: Biological
Sciences, 273, 1355–1360. http://dx.doi.org/
10.1098/rspb.2005.3445
DeBruine, L. M., Jones, B. C., Tybur, J. M., Lieberman,
D., & Griskevicius, V. (2010). Women’s preferences
for masculinity in male faces are predicted by
pathogen disgust, but not by moral or sexual disgust.
Evolution and Human Behavior, 31, 69–74. http://
dx.doi.org/10.1016/j.evolhumbehav.2009.09.003
DeBruine, L. M., Jones, B. C., Unger, L., Little, A. C., &
Feinberg, D. R. (2007). Dissociating averageness
and attractiveness: Attractive faces are not always
average. Journal of Experimental Psychology: Human
Perception and Performance, 33, 1420–1430. http://
dx.doi.org/10.1037/0096-1523.33.6.1420
Dion, K., Berscheid, E., & Walster, E. (1972). What is
beautiful is good. Journal of Personality and Social
Psychology, 24, 285–290. http://dx.doi.org/10.1037/
h0033731
Edwards, E. A., & Duntley, S. Q. (1939). The pigments
and color of living human skin. American Journal
of Anatomy, 65, 1–33. http://dx.doi.org/10.1002/
aja.1000650102
Elliot, A. J., & Niesta, D. (2008). Romantic red: Red
enhances men’s attraction to women. Journal of
Personality and Social Psychology, 95, 1150–1164.
http://dx.doi.org/10.1037/0022-3514.95.5.1150
Elliot, A. J., Niesta Kayser, D., Greitemeyer, T., Lichtenfeld,
S., Gramzow, R. H., Maier, M. A., & Liu, H. (2010).
Red, rank, and romance in women viewing men.
Journal of Experimental Psychology: General, 139,
399–417. http://dx.doi.org/10.1037/a0019689
Elliott, D. N., & Wittenberg, B. H. (1955). Accuracy of
identification of Jewish and non-Jewish photographs.
Journal of Abnormal and Social Psychology, 51,
339–341. http://dx.doi.org/10.1037/h0043566
Enstrom, J. E., & Breslow, L. (2008). Lifestyle and
reduced mortality among active California Mormons,
1980–2004. Preventive Medicine, 46, 133–136. http://
dx.doi.org/10.1016/j.ypmed.2007.07.030
Evans, E. C. (1969). Physiognomics in the ancient world.
Transactions of the American Philosophical Society, 59,
1–101. http://dx.doi.org/10.2307/1006011
Felson, R. B. (1996). Big people hit little people: Sex
differences in physical power and interpersonal
Appearance and Physiognomy
violence. Criminology, 34, 433–452. http://dx.doi.
org/10.1111/j.1745-9125.1996.tb01214.x
Fink, B., Grammer, K., & Matts, P. J. (2006). Visible
skin color distribution plays a role in the perception
of age, attractiveness, and health in female faces.
Evolution and Human Behavior, 27, 433–442. http://
dx.doi.org/10.1016/j.evolhumbehav.2006.08.007
Fink, B., Grammer, K., & Thornhill, R. (2001).
Human (Homo sapiens) facial attractiveness in
relation to skin texture and color. Journal of
Comparative Psychology, 115, 92–99. http://dx.doi.
org/10.1037/0735-7036.115.1.92
Fink, B., & Matts, P. J. (2008). The effects of skin colour
distribution and topography cues on the perception
of female facial age and health. Journal of the European
Academy of Dermatology and Venereology, 22, 493–498.
http://dx.doi.org/10.1111/j.1468-3083.2007.02512.x
Fink, B., Neave, N., & Seydel, H. (2007). Male facial
appearance signals physical strength to women.
American Journal of Human Biology, 19, 82–87.
http://dx.doi.org/10.1002/ajhb.20583
Fink, B., & Penton-Voak, I. (2002). Evolutionary
psychology of facial attractiveness. Current Directions
in Psychological Science, 11, 154–158. http://dx.doi.
org/10.1111/1467-8721.00190
Folstad, I., & Karter, A. J. (1992). Parasites, bright males, and
the immunocompetence handicap. American Naturalist,
139, 603–622. http://dx.doi.org/10.1086/285346
Freeman, J. B., Johnson, K. L., Ambady, N., & Rule, N. O.
(2010). Sexual orientation perception involves
gendered facial cues. Personality and Social
Psychology Bulletin, 36, 1318–1331. http://dx.doi.
org/10.1177/0146167210378755
Friis, H., Gomo, E., Koestel, P., Ndhlovu, P., Nyazema,
N., Krarup, H., & Michaelsen, K. F. (2001). HIV and
other predictors of serum beta-carotene and retinol
in pregnancy: A cross-sectional study in Zimbabwe.
American Journal of Clinical Nutrition, 73, 1058–1065.
Galton, F. (1878). Composite portraits. Nature, 18,
97–100. http://dx.doi.org/10.1038/018097a0
Gangestad, S. W., Merriman, L. A., & Emery
Thompson, M. (2010). Men’s oxidative stress,
fluctuating asymmetry and physical attractiveness.
Animal Behaviour, 80, 1005–1013. http://dx.doi.
org/10.1016/j.anbehav.2010.09.003
Gangestad, S. W., & Simpson, J. A. (2000). The evolution
of human mating: Trade-offs and strategic pluralism.
Behavioral and Brain Sciences, 23, 573–587. http://
dx.doi.org/10.1017/S0140525X0000337X
Gaulin, S. J. C., & Sailer, L. D. (1984). Sexual
dimorphism in weight among the primates: The
relative impact of allometry and sexual selection.
International Journal of Primatology, 5, 515–535.
http://dx.doi.org/10.1007/BF02692284
247
Re and Rule
Gildersleeve, K., DeBruine, L., Haselton, M. G., Frederick,
D. A., Penton-Voak, I. S., Jones, B. C., & Perrett, D. I.
(2013). Shifts in women’s mate preferences across
the ovulatory cycle: A critique of Harris (2011) and
Harris (2012). Sex Roles, 69, 516–524. http://dx.doi.
org/10.1007/s11199-013-0273-4
Gildersleeve, K., Haselton, M. G., & Fales, M. R. (2014).
Do women’s mate preferences change across the
ovulatory cycle? A meta-analytic review. Psychological
Bulletin, 140, 1205–1259. http://dx.doi.org/10.1037/
a0035438
Goldberg, L. R. (1999). A broad-bandwidth, public domain,
personality inventory measuring the lower-level
facets of several five-factor models. In I. Mervielde, I.
Deary, F. De Fruyt, & F. Ostendorf (Eds.), Personality
psychology in Europe (Vol. 7, pp. 7–28). Tilburg, the
Netherlands: Tilburg University Press.
Goren, C. C., Sarty, M., & Wu, P. Y. K. (1975). Visual
following and pattern discrimination of face-like
stimuli by newborn infants. Pediatrics, 56, 544–549.
Grammer, K., & Thornhill, R. (1994). Human (Homo
sapiens) facial attractiveness and sexual selection:
The role of symmetry and averageness. Journal of
Comparative Psychology, 108, 233–242. http://dx.doi.
org/10.1037/0735-7036.108.3.233
Gray, P. B., Kahlenberg, S. M., Barrett, E. S., Lipson, S. F., &
Ellison, P. T. (2002). Marriage and fatherhood
are associated with lower testosterone in males.
Evolution and Human Behavior, 23, 193–201. http://
dx.doi.org/10.1016/S1090-5138(01)00101-5
Hadjikhani, N., Kveraga, K., Naik, P., & Ahlfors, S. P.
(2009). Early (M170) activation of face-specific
cortex by face-like objects. NeuroReport, 20, 403–407.
http://dx.doi.org/10.1097/WNR.0b013e328325a8e1
Hamilton, W. D., & Zuk, M. (1982). Heritable true fitness
and bright birds: A role for parasites? Science, 218,
384–387. http://dx.doi.org/10.1126/science.7123238
Harris, C. R. (2011). Menstrual cycle and facial
preferences reconsidered. Sex Roles, 64, 669–681.
http://dx.doi.org/10.1007/s11199-010-9772-8
Harris, C. R. (2013). Shifts in masculinity preferences
across the menstrual cycle: Still not there. Sex Roles, 69,
507–515. http://dx.doi.org/10.1007/s11199-012-0229-0
Henderson, J. J. A., & Anglin, J. M. (2003). Facial
attractiveness predicts longevity. Evolution and
Human Behavior, 24, 351–356. http://dx.doi.
org/10.1016/S1090-5138(03)00036-9
Hess, U., Adams, R., Jr., & Kleck, R. (2005). Who
may frown and who should smile? Dominance,
affiliation, and the display of happiness and anger.
Cognition and Emotion, 19, 515–536. http://dx.doi.
org/10.1080/02699930441000364
Hönekopp, J., Rudolph, U., Beier, L., Liebert, A., &
Muller, C. (2007). Physical attractiveness of face
248
and body as indicators of physical fitness in men.
Evolution and Human Behavior, 28, 106–111. http://
dx.doi.org/10.1016/j.evolhumbehav.2006.09.001
Hoyme, H. E. (1994). Minor anomalies: Diagnostic
clues to aberrant human morphogenesis. In T. A.
Markow (Ed.), Developmental instability: Its origins
and evolutionary implications (pp. 309–317). http://
dx.doi.org/10.1007/978-94-011-0830-0_24
Hughes, S. M., & Bremme, R. (2011). The effects of facial
symmetry and sexually-dimorphic facial proportions
on assessments of sexual orientation. Journal of
Social, Evolutionary, and Cultural Psychology, 5,
214–230. http://dx.doi.org/10.1037/h0099261
Hume, D. K., & Montgomerie, R. (2001). Facial
attractiveness signals different aspects of “quality”
in women and men. Evolution and Human Behavior,
22, 93–112. http://dx.doi.org/10.1016/S1090-
5138(00)00065-9
Jablonski, N. G., & Chaplin, G. (2000). The evolution
of human skin coloration. Journal of Human
Evolution, 39, 57–106. http://dx.doi.org/10.1006/
jhev.2000.0403
Jahoda, G. (1954). Political attitudes and judgments
of other people. Journal of Abnormal and Social
Psychology, 49, 330–334. http://dx.doi.org/10.1037/
h0062563
Johnson, J. M. (1998). Physical training and the
control of skin blood flow. Medicine and Science
in Sports and Exercise, 30, 382–386. http://dx.doi.
org/10.1097/00005768-199803000-00007
Johnston, V. S., Hagel, R., Franklin, M., Fink, B., &
Grammer, K. (2001). Male facial attractiveness:
Evidence for hormone-mediated adaptive design.
Evolution and Human Behavior, 22, 251–267. http://
dx.doi.org/10.1016/S1090-5138(01)00066-6
Jones, B. C. (2014). Agreement and individual
differences in men’s preferences for women’s facial
characteristics. In V. A. Weekes-Shackelford & T. K.
Shackelford (Eds.), Evolutionary perspectives on
human sexual psychology and behavior (pp. 87–102).
New York, NY: Springer.
Jones, B. C., DeBruine, L. M., & Little, A. C. (2007).
The role of symmetry in attraction to average faces.
Perception and Psychophysics, 69, 1273–1277. http://
dx.doi.org/10.3758/BF03192944
Jones, B. C., DeBruine, L. M., Main, J. C., Little, A. C.,
Welling, L. L. M., Feinberg, D. R., & Tiddeman, B. P.
(2010). Facial cues of dominance modulate the short-
term gaze-cuing effect in human observers. Proceedings
of the Royal Society B: Biological Sciences, 277,
617–624. http://dx.doi.org/10.1098/rspb.2009.1575
Jones, B. C., DeBruine, L. M., Perrett, D. I., Little, A. C.,
Feinberg, D. R., & Law Smith, M. J. (2008). Effects
of menstrual cycle phase on face preferences.

Archives of Sexual Behavior, 37, 78–84. http://dx.doi.
org/10.1007/s10508-007-9268-y
Jones, B. C., Little, A. C., Burt, D. M., & Perrett, D. I. (2004).
When facial attractiveness is only skin deep. Perception,
33, 569–576. http://dx.doi.org/10.1068/p3463
Jones, B. C., Little, A. C., Feinberg, D. R., Penton-Voak,
I. S., Tiddeman, B. P., & Perrett, D. I. (2004). The
relationship between shape symmetry and perceived
skin condition in male facial attractiveness. Evolution
and Human Behavior, 25, 24–30. http://dx.doi.
org/10.1016/S1090-5138(03)00080-1
Jones, B. C., Little, A. C., Penton-Voak, I. S., Tiddeman,
B. P., Burt, D. M., & Perrett, D. I. (2001). Facial
symmetry and judgements of apparent health:
Support for a “good genes” explanation of the
attractiveness–symmetry relationship. Evolution
and Human Behavior, 22, 417–429. http://dx.doi.
org/10.1016/S1090-5138(01)00083-6
Kalick, S. M., Zebrowitz, L. A., Langlois, J. H., & Johnson,
R. M. (1998). Does human facial attractiveness
honestly advertise health? Longitudinal data on an
evolutionary question. Psychological Science, 9, 8–13.
http://dx.doi.org/10.1111/1467-9280.00002
Kanwisher, N., Chun, M. M., McDermott, J., & Ledden,
P. J. (1996). Functional imaging of human visual
recognition. Cognitive Brain Research, 5, 55–67.
Keating, C. F. (1985). Gender and the physiognomy of
dominance and attractiveness. Social Psychology
Quarterly, 48, 61–70. http://dx.doi.org/10.2307/3033782
Keating, C. F., Mazur, A., Segall, M. H., Cysneiros, P.
G., Kilbride, J. E., Leahy, P., . . . Wirsing, R. (1981).
Culture and the perception of social dominance
from facial expression. Journal of Personality and
Social Psychology, 40, 615–626. http://dx.doi.
org/10.1037/0022-3514.40.4.615
King, A., & Leigh, A. (2009). Beautiful politicians.
Kyklos, 62, 579–593. http://dx.doi.org/10.1111/
j.1467-6435.2009.00452.x
Koehler, N., Simmons, L. W., Rhodes, G., & Peters, M.
(2004). The relationship between sexual dimorphism
in human faces and fluctuating asymmetry. Proceedings
of the Royal Society B: Biological Sciences, 271(Suppl. 4),
S233–S236. http://dx.doi.org/10.1098/rsbl.2003.0146
Kościński, K. (2013). Perception of facial attractiveness
from static and dynamic stimuli. Perception, 42,
163–175. http://dx.doi.org/10.1068/p7378
Kowner, R. (1996). Facial asymmetry and attractiveness
judgment in developmental perspective. Journal
of Experimental Psychology: Human Perception
and Performance, 22, 662–675. http://dx.doi.
org/10.1037/0096-1523.22.3.662
Krupp, D. B., DeBruine, L. M., & Jones, B. C. (2011).
Apparent health encourages reciprocity. Evolution
Appearance and Physiognomy
and Human Behavior, 32, 198–203. http://dx.doi.
org/10.1016/j.evolhumbehav.2010.10.001
Lander, K. (2008). Relating visual and vocal attractiveness
for moving and static faces. Animal Behaviour,
75, 817–822. http://dx.doi.org/10.1016/
j.anbehav.2007.07.001
Lander, K., Christie, F., & Bruce, V. (1999). The role
of movement in the recognition of famous faces.
Memory and Cognition, 27, 974–985. http://dx.doi.
org/10.3758/BF03201228
Langlois, J. H., & Roggman, L. A. (1990). Attractive faces
are only average. Psychological Science, 1, 115–121.
http://dx.doi.org/10.1111/j.1467-9280.1990.tb00079.x
Langlois, J. H., Roggman, L. A., & Musselman, L. (1994).
What is average and what is not average about
attractive faces. Psychological Science, 5, 214–220.
http://dx.doi.org/10.1111/j.1467-9280.1994.tb00503.x
Lavater, J. C. (1772). Essays on physiognomy. Retrieved
from https://archive.org/details/04851455.5902.
emory.edu
Law Smith, M. J., Perrett, D. I., Jones, B. C., Cornwell,
R. E., Moore, F. R., Feinberg, D. R., . . . Hillier, S. G.
(2006). Facial appearance is a cue to oestrogen
levels in women. Proceedings of the Royal Society
B: Biological Sciences, 273, 135–140. http://dx.doi.
org/10.1098/rspb.2005.3296
Lee, A. J., Mitchem, D. G., Wright, M. J., Martin, N. G.,
Keller, M. C., & Zietsch, B. P. (2014). Genetic
factors that increase male facial masculinity
decrease facial attractiveness of female relatives.
Psychological Science, 25, 476–484. http://dx.doi.
org/10.1177/0956797613510724
Lindzey, G., & Rogolsky, S. (1950). Prejudice and
identification of minority group membership. Journal
of Abnormal and Social Psychology, 45, 37–53. http://
dx.doi.org/10.1037/h0055637
Lippa, R. A. (2003). Are 2D:4D finger-length ratios related to
sexual orientation? Yes for men, no for women. Journal
of Personality and Social Psychology, 85, 179–188. http://
dx.doi.org/10.1037/0022-3514.85.1.179
Lipson, S. F., & Ellison, P. T. (1996). Comparison
of salivary steroid profiles in naturally occurring
conception and non-conception cycles. Human
Reproduction, 11, 2090–2096. http://dx.doi.
org/10.1093/oxfordjournals.humrep.a019055
Little, A. C. (2013). Multiple motives in women’s
preferences for masculine male faces: Comment on
Scott et al. Behavioral Ecology, 24, 590–591. http://
dx.doi.org/10.1093/beheco/ars102
Little, A. C., Apicella, C. L., & Marlowe, F. W. (2007).
Preferences for symmetry in human faces in two
cultures: Data from the UK and the Hadza, an
isolated group of hunter-gatherers. Proceedings of the
249
Re and Rule
Royal Society B: Biological Sciences, 274, 3113–3117.
http://dx.doi.org/10.1098/rspb.2007.0895
Little, A. C., Burriss, R. P., Jones, B. C., DeBruine, L. M., &
Caldwell, C. A. (2008). Social influence in
human face preference: Men and women are
influenced more for long-term than short-term
attractiveness decisions. Evolution and Human
Behavior, 29, 140–146. http://dx.doi.org/10.1016/
j.evolhumbehav.2007.11.007
Little, A. C., Burt, D. M., Penton-Voak, I. S., & Perrett,
D. I. (2001). Self-perceived attractiveness influences
human female preferences for sexual dimorphism
and symmetry in male faces. Proceedings of the Royal
Society B: Biological Sciences, 268, 39–44. http://
dx.doi.org/10.1098/rspb.2000.1327
Little, A. C., Cohen, D. L., Jones, B. C., & Belsky, J. (2007).
Human preferences for facial masculinity change
with relationship type and environmental harshness.
Behavioral Ecology and Sociobiology, 61, 967–973.
http://dx.doi.org/10.1007/s00265-006-0325-7
Little, A. C., DeBruine, L. M., & Jones, B. C. (2011).
Exposure to visual cues of pathogen contagion
changes preferences for masculinity and symmetry
in opposite-sex faces. Proceedings of the Royal Society
B: Biological Sciences, 278, 2032–2039. http://dx.doi.
org/10.1098/rspb.2010.1925
Little, A. C., DeBruine, L. M., & Jones, B. C. (2013).
Environment contingent preferences: Exposure to
visual cues of direct male–male competition and
wealth increase women’s preferences for masculinity
in male faces. Evolution and Human Behavior, 34,
193–200. http://dx.doi.org/10.1016/
j.evolhumbehav.2012.11.008
Little, A. C., & Hancock, P. J. B. (2002). The role of
masculinity and distinctiveness in judgments
of human male facial attractiveness. British
Journal of Psychology, 93, 451–464. http://dx.doi.
org/10.1348/000712602761381349
Little, A. C., & Jones, B. C. (2012). Variation in facial
masculinity and symmetry preferences across
the menstrual cycle is moderated by relationship
context. Psychoneuroendocrinology, 37, 999–1008.
http://dx.doi.org/10.1016/j.psyneuen.2011.11.007
Little, A. C., Jones, B. C., Burt, D. M., & Perrett, D. I.
(2007). Preferences for symmetry in faces change
across the menstrual cycle. Biological Psychology,
76, 209–216. http://dx.doi.org/10.1016/
j.biopsycho.2007.08.003
Little, A. C., & Mannion, H. (2006). Viewing attractive or
unattractive same-sex individuals changes self-rated
attractiveness and face preferences in women. Animal
Behaviour, 72, 981–987. http://dx.doi.org/10.1016/
j.anbehav.2006.01.026
Little, A. C., Saxton, T. K., Roberts, S. C., Jones, B. C.,
Debruine, L. M., Vukovic, J., . . . Chenore, T.
250
(2010). Women’s preferences for masculinity in
male faces are highest during reproductive age range
and lower around puberty and post-menopause.
Psychoneuroendocrinology, 35, 912–920. http://
dx.doi.org/10.1016/j.psyneuen.2009.12.006
Livingston, R. W., & Pearce, N. A. (2009). The teddy-
bear effect: Does having a baby face benefit Black
chief executive officers? Psychological Science,
20, 1229–1236. http://dx.doi.org/10.1111/
j.1467-9280.2009.02431.x
Main, J. C., Jones, B. C., DeBruine, L. M., & Little, A. C.
(2009). Integrating gaze direction and sexual
dimorphism of face shape when perceiving the
dominance of others. Perception, 38, 1275–1283.
http://dx.doi.org/10.1068/p6347
Malatesta, C. Z., Fiore, M. J., & Messina, J. J.
(1987). Affect, personality, and facial expressive
characteristics of older people. Psychology and
Aging, 2, 64–69. http://dx.doi.org/10.1037/0882-
7974.2.1.64
Matts, P. J., Fink, B., Grammer, K., & Burquest, M. (2007).
Color homogeneity and visual perception of age,
health, and attractiveness of female facial skin.
Journal of the American Academy of Dermatology,
57, 977–984. http://dx.doi.org/10.1016/
j.jaad.2007.07.040
Mattsson, A., Schalling, D., Olweus, D., Löw, H., &
Svensson, J. (1980). Plasma testosterone, aggressive
behavior, and personality dimensions in young male
delinquents. Journal of the American Academy of Child
Psychiatry, 19, 476–490. http://dx.doi.org/10.1016/
S0002-7138(09)61065-7
Mazur, A., & Booth, A. (1998). Testosterone and dominance
in men. Behavioral and Brain Sciences, 21, 353–363.
http://dx.doi.org/10.1017/S0140525X98001228
McCrae, R. R., & Costa, P. T., Jr. (1987). Validation of the
five-factor model of personality across instruments
and observers. Journal of Personality and Social
Psychology, 52, 81–90. http://dx.doi.org/10.1037/
0022-3514.52.1.81
Meyer-Bahlburg, H. F. L., Ehrhardt, A. A., Rosen, L. R.,
Gruen, R. S., Veridiano, N. P., Vann, F. H., &
Neuwalder, H. F. (1995). Prenatal estrogens and
the development of homosexual orientation.
Developmental Psychology, 31, 12–21. http://dx.doi.
org/10.1037/0012-1649.31.1.12
Miller, G. F., & Todd, P. M. (1998). Mate choice turns
cognitive. Trends in Cognitive Sciences, 2, 190–198.
http://dx.doi.org/10.1016/S1364-6613(98)01169-3
Mobius, M. M., & Rosenblat, T. S. (2006). Why beauty
matters. American Economic Review, 96, 222–235.
http://dx.doi.org/10.1257/000282806776157515
Møller, A. P., & Thornhill, R. (1998). Bilateral
symmetry and sexual selection: A meta-analysis.

American Naturalist, 151, 174–192. http://dx.doi.
org/10.1086/286110
Moore, F. R., Al Dujaili, E. A. S., Cornwell, R. E., Law
Smith, M. J., Lawson, J. F., Sharp, M., & Perrett,
D. I. (2011). Cues to sex- and stress-hormones in
the human male face: Functions of glucocorticoids
in the immunocompetence handicap hypothesis.
Hormones and Behavior, 60, 269–274. http://dx.doi.
org/10.1016/j.yhbeh.2011.05.010
Moore, F. R., Coetzee, V., Contreras-Garduño, J.,
Debruine, L. M., Kleisner, K., Krams, I., . . . Suzuki,
T. N. (2013). Cross-cultural variation in women’s
preferences for cues to sex- and stress-hormones in
the male face. Biology Letters, 9, 20130050. http://
dx.doi.org/10.1098/rsbl.2013.0050
Moore, F. R., Cornwell, R. E., Law Smith, M. J., Al Dujaili,
E. A. S., Sharp, M., & Perrett, D. I. (2011). Evidence
for the stress-linked immunocompetence handicap
hypothesis in human male faces. Proceedings of the
Royal Society B: Biological Sciences, 278, 774–780.
http://dx.doi.org/10.1098/rspb.2010.1678
Morrison, E. R., Clark, A. P., Tiddeman, B. P., & Penton-
Voak, I. S. (2010). Manipulating shape cues in dynamic
human faces: Sexual dimorphism is preferred in female
but not male faces. Ethology, 116, 1234–1243. http://
dx.doi.org/10.1111/j.1439-0310.2010.01839.x
Muehlenbein, M. P., & Bribiescas, R. G. (2005).
Testosterone-mediated immune functions and male
life histories. American Journal of Human Biology, 17,
527–558. http://dx.doi.org/10.1002/ajhb.20419
Mueller, U., & Mazur, A. (1996). Facial dominance of
West Point cadets as a predictor of later military
rank. Social Forces, 74, 823–850. http://dx.doi.
org/10.1093/sf/74.3.823
Mueser, K. T., Grau, B. W., Sussman, S., & Rosen, A. J.
(1984). You’re only as pretty as you feel: Facial
expression as a determinant of physical attractiveness.
Journal of Personality and Social Psychology, 46,
469–478. http://dx.doi.org/10.1037/0022-3514.46.2.469
Mulhern, R., Fieldman, G., Hussey, T., Lévêque, J. L., &
Pineau, P. (2003). Do cosmetics enhance female
Caucasian facial attractiveness? International Journal
of Cosmetic Science, 25, 199–205. http://dx.doi.
org/10.1046/j.1467-2494.2003.00188.x
Neave, N., Laing, S., Fink, B., & Manning, J. T. (2003).
Second to fourth digit ratio, testosterone and
perceived male dominance. Proceedings of the Royal
Society B: Biological Sciences, 270, 2167–2172.
http://dx.doi.org/10.1098/rspb.2003.2502
Neave, N., & Shields, K. (2008). The effects of facial hair
manipulation on female perceptions of attractiveness,
masculinity, and dominance in male faces.
Personality and Individual Differences, 45, 373–377.
http://dx.doi.org/10.1016/j.paid.2008.05.007
Appearance and Physiognomy
O’Connor, J. J. M., Feinberg, D. R., Fraccaro, P. J., Borak,
D. J., Tigue, C. C., Re, D. E., . . . Tiddeman, B.
(2012). Female preferences for male vocal and facial
masculinity in videos. Ethology, 118, 321–330. http://
dx.doi.org/10.1111/j.1439-0310.2011.02013.x
Ohlsen, G., van Zoest, W., & van Vugt, M. (2013).
Gender and facial dominance in gaze cuing:
Emotional context matters in the eyes that we follow.
PLoS ONE, 8, e59471. http://dx.doi.org/10.1371/
journal.pone.0059471
Olivola, C. Y., & Todorov, A. (2010). Elected in 100
milliseconds: Appearance-based trait inferences and
voting. Journal of Nonverbal Behavior, 34, 83–110.
http://dx.doi.org/10.1007/s10919-009-0082-1
Olweus, D., Mattsson, A., Schalling, D., & Löw, H.
(1980). Testosterone, aggression, physical, and
personality dimensions in normal adolescent males.
Psychosomatic Medicine, 42, 253–269. http://dx.doi.
org/10.1097/00006842-198003000-00003
Olweus, D., Mattsson, A., Schalling, D., & Löw, H.
(1988). Circulating testosterone levels and
aggression in adolescent males: A causal analysis.
Psychosomatic Medicine, 50, 261–272. http://dx.doi.
org/10.1097/00006842-198805000-00004
Omaye, S. T. (1993). Nutrient deficiencies and pregnancy
outcome. In R. P. Sharma (Ed.), Dietary factors and
birth defects (pp. 12–41). San Francisco, CA: Pacific
Division, AAAS.
Oosterhof, N. N., & Todorov, A. (2008). The functional
basis of face evaluation. Proceedings of the National
Academy of Sciences, USA, 105, 11087–11092.
http://dx.doi.org/10.1073/pnas.0805664105
Palmer, A. R., & Strobeck, C. (1986). Fluctuating
asymmetry: Measurement, analysis, patterns. Annual
Review of Ecology and Systematics, 17, 391–421. http://
dx.doi.org/10.1146/annurev.es.17.110186.002135
Penton-Voak, I. S., & Chang, H. Y. (2008). Attractiveness
judgements of individuals vary across emotional
expression and movement conditions. Journal of
Evolutionary Psychology, 6, 89–100. http://dx.doi.
org/10.1556/JEP.2008.1011
Penton-Voak, I. S., & Chen, J. Y. (2004). High salivary
testosterone is linked to masculine male facial
appearance in humans. Evolution and Human
Behavior, 25, 229–241. http://dx.doi.org/10.1016/
j.evolhumbehav.2004.04.003
Penton-Voak, I. S., Jacobson, A., & Trivers, R. (2004).
Populational differences in attractiveness judgements
of male and female faces: Comparing British and
Jamaican samples. Evolution and Human Behavior,
25, 355–370. http://dx.doi.org/10.1016/
j.evolhumbehav.2004.06.002
Penton-Voak, I. S., Jones, B. C., Little, A. C., Baker, S.,
Tiddeman, B., Burt, D. M., & Perrett, D. I. (2001).
251
Re and Rule
Symmetry, sexual dimorphism in facial proportions
and male facial attractiveness. Proceedings of the
Royal Society B: Biological Sciences, 268, 1617–1623.
http://dx.doi.org/10.1098/rspb.2001.1703
Penton-Voak, I. S., Little, A. C., Jones, B. C., Burt, D. M.,
Tiddeman, B. P., & Perrett, D. I. (2003). Female
condition influences preferences for sexual dimorphism
in faces of male humans (Homo sapiens). Journal of
Comparative Psychology, 117, 264–271. http://dx.doi.
org/10.1037/0735-7036.117.3.264
Penton-Voak, I. S., Perrett, D. I., Castles, D. L., Kobayashi,
T., Burt, D. M., Murray, L. K., & Minamisawa, R.
(1999). Menstrual cycle alters face preference. Nature,
399, 741–742. http://dx.doi.org/10.1038/21557
Penton-Voak, I. S., Pound, N., Little, A. C., & Perrett,
D. I. (2006). Personality judgments from natural
and composite facial images: More evidence for
a “kernel of truth” in social perception. Social
Cognition, 24, 607–640. http://dx.doi.org/10.1521/
soco.2006.24.5.607
Perrett, D. I., Burt, D. M., Penton-Voak, I. S., Lee, K. J.,
Rowland, D. A., & Edwards, R. (1999). Symmetry
and human facial attractiveness. Evolution and
Human Behavior, 20, 295–307. http://dx.doi.
org/10.1016/S1090-5138(99)00014-8
Perrett, D. I., Lee, K. J., Penton-Voak, I., Rowland, D.,
Yoshikawa, S., Burt, D. M., . . . Akamatsu, S.
(1998). Effects of sexual dimorphism on facial
attractiveness. Nature, 394, 884–887. http://dx.doi.
org/10.1038/29772
Perrett, D. I., May, K. A., & Yoshikawa, S. (1994). Facial
shape and judgements of female attractiveness. Nature,
368, 239–242. http://dx.doi.org/10.1038/368239a0
Peters, M., Rhodes, G., & Simmons, L. W. (2007).
Contributions of the face and body to overall
attractiveness. Animal Behaviour, 73, 937–942.
http://dx.doi.org/10.1016/j.anbehav.2006.07.012
Ponsonby, A. L., Dwyer, T., & Couper, D. (1997).
Sleeping position, infant apnea, and cyanosis: A
population-based study. Pediatrics, 99, e3. http://
dx.doi.org/10.1542/peds.99.1.e3
Pound, N., Penton-Voak, I. S., & Surridge, A. K. (2009).
Testosterone responses to competition in men are
related to facial masculinity. Proceedings of the Royal
Society B: Biological Sciences, 276, 153–159.
http://dx.doi.org/10.1098/rspb.2008.0990
Pulos, L., & Spilka, B. (1961). Perceptual selectivity,
memory, and anti-Semitism. Journal of Abnormal
and Social Psychology, 62, 690–692. http://dx.doi.
org/10.1037/h0048348
Puts, D. A. (2010). Beauty and the beast: Mechanisms of
sexual selection in humans. Evolution and Human
Behavior, 31, 157–175. http://dx.doi.org/10.1016/
j.evolhumbehav.2010.02.005
252
Rantala, M. J., Coetzee, V., Moore, F. R., Skrinda, I.,
Kecko, S., Krama, T., . . . Krams, I. (2013). Adiposity,
compared with masculinity, serves as a more
valid cue to immunocompetence in human mate
choice. Proceedings of the Royal Society B: Biological
Sciences, 280, 20122495. http://dx.doi.org/10.1098/
rspb.2012.2495
Re, D. E., Coetzee, V., Xiao, D. K., Buls, D., Tiddeman,
B. P., Boothroyd, L. G., & Perrett, D. I. (2011).
Viewing heavy bodies enhances preferences for facial
adiposity. Journal of Evolutionary Psychology, 9,
295–308. http://dx.doi.org/10.1556/JEP.9.2011.4.2
Re, D. E., & Perrett, D. I. (2014). The effects of facial
adiposity on attractiveness and perceived leadership
ability. Quarterly Journal of Experimental Psychology,
67, 676–686. http://dx.doi.org/10.1080/17470218.
2013.825635
Re, D. E., Whitehead, R. D., Xiao, D., & Perrett, D.
I. (2011). Oxygenated-blood colour change
thresholds for perceived facial redness, health, and
attractiveness. PLoS ONE, 6, e17859. http://dx.doi.
org/10.1371/journal.pone.0017859
Reither, E. N., Hauser, R. M., & Swallen, K. C. (2009).
Predicting adult health and mortality from adolescent
facial characteristics in yearbook photographs.
Demography, 46, 27–41. http://dx.doi.org/10.1353/
dem.0.0037
Remedios, J. D., Chasteen, A. L., Rule, N. O., & Plaks,
J. E. (2011). Impressions at the intersection of
ambiguous and obvious social categories: Does
gay plus Black = likable? Journal of Experimental
Social Psychology, 47, 1312–1315. http://dx.doi.
org/10.1016/j.jesp.2011.05.015
Rhodes, G. (2006). The evolutionary psychology of
facial beauty. Annual Review of Psychology, 57,
199–226. http://dx.doi.org/10.1146/annurev.psych.
57.102904.190208
Rhodes, G., Chan, J., Zebrowitz, L. A., & Simmons, L. W.
(2003). Does sexual dimorphism in human faces
signal health? Proceedings of the Royal Society B:
Biological Sciences, 270(Suppl. 1), S93–S95.
http://dx.doi.org/10.1098/rsbl.2003.0023
Rhodes, G., Hickford, C., & Jeffery, L. (2000). Sex-
typicality and attractiveness: Are supermale
and superfemale faces super-attractive? British
Journal of Psychology, 91, 125–140. http://dx.doi.
org/10.1348/000712600161718
Rhodes, G., Lie, H. C., Thevaraja, N., Taylor, L., Iredell,
N., Curran, C., . . . Simmons, L. W. (2011). Facial
attractiveness ratings from video-clips and static
images tell the same story. PLoS ONE, 6, e26653.
http://dx.doi.org/10.1371/journal.pone.0026653
Rhodes, G., Simmons, L. W., & Peters, M. (2005).
Attractiveness and sexual behavior: Does
attractiveness enhance mating success? Evolution

and Human Behavior, 26, 186–201. http://dx.doi.
org/10.1016/j.evolhumbehav.2004.08.014
Rhodes, G., Sumich, A., & Byatt, G. (1999). Are average
facial configurations attractive only because of their
symmetry? Psychological Science, 10, 52–58. http://
dx.doi.org/10.1111/1467-9280.00106
Rhodes, G., & Tremewan, T. (1996). Averageness,
exaggeration, and facial attractiveness.
Psychological Science, 7, 105–110. http://dx.doi.
org/10.1111/j.1467-9280.1996.tb00338.x
Rhodes, G., Zebrowitz, L. A., Clark, A., Kalick, S. M.,
Hightower, A., & McKay, R. (2001). Do facial
averageness and symmetry signal health? Evolution
and Human Behavior, 22, 31–46. http://dx.doi.
org/10.1016/S1090-5138(00)00060-X
Rhodes, L., Argersinger, M. E., Gantert, L. T., Friscino,
B. H., Hom, G., Pikounis, B., . . . Rhodes, W. L.
(1997). Effects of administration of testosterone,
dihydrotestosterone, oestrogen and fadrozole, an
aromatase inhibitor, on sex skin colour in intact male
rhesus macaques. Journal of Reproduction and Fertility,
111, 51–57. http://dx.doi.org/10.1530/jrf.0.1110051
Rice, D. R., & Mullen, B. (2003). Isaac, Ishmael, and
Janus: Past and future lessons regarding the ethnic
categorization of faces. Applied Cognitive Psychology,
17, 1129–1147. http://dx.doi.org/10.1002/acp.990
Richard, F. D., Bond, C. F., Jr., & Stokes-Zoota, J. J.
(2003). One hundred years of social psychology
quantitatively described. Review of General
Psychology, 7, 331–363. http://dx.doi.org/10.1037/
1089-2680.7.4.331
Ritts, V., Patterson, M. L., & Tubbs, M. E. (1992).
Expectations, impressions, and judgments of
physically attractive students: A review. Review of
Educational Research, 62, 413–426. http://dx.doi.
org/10.3102/00346543062004413
Ro, T., Russell, C., & Lavie, N. (2001). Changing faces:
A detection advantage in the flicker paradigm.
Psychological Science, 12, 94–99. http://dx.doi.
org/10.1111/1467-9280.00317
Roberts, M. L., Buchanan, K. L., & Evans, M. R. (2004).
Testing the immunocompetence handicap hypothesis: A
review of the evidence. Animal Behaviour, 68, 227–239.
http://dx.doi.org/10.1016/j.anbehav.2004.05.001
Roberts, S. C., Saxton, T. K., Murray, A. K., Burriss, R. P.,
Rowland, H. M., & Little, A. C. (2009). Static and
dynamic facial images cue similar attractiveness
judgements. Ethology, 115, 588–595. http://dx.doi.
org/10.1111/j.1439-0310.2009.01640.x
Robins, A. H. (1991). Biological perspectives on
human pigmentation. http://dx.doi.org/10.1017/
CBO9780511600463
Roney, J. R., Hanson, K. N., Durante, K. M., & Maestripieri,
D. (2006). Reading men’s faces: Women’s mate
Appearance and Physiognomy
attractiveness judgments track men’s testosterone and
interest in infants. Proceedings of the Royal Society B:
Biological Sciences, 273, 2169–2175.
Rosenthal, R. (1979). The file drawer problem and tolerance
for null results. Psychological Bulletin, 86, 638–641.
http://dx.doi.org/10.1037/0033-2909.86.3.638
Rosenthal, R., Hall, J. A., DiMatteo, M. R., Rogers, P. L., &
Archer, D. (1979). Sensitivity to nonverbal
communication: The PONS test. Baltimore, MD:
Johns Hopkins University Press.
Rubenstein, A. J. (2005). Variation in perceived
attractiveness: Differences between dynamic and
static faces. Psychological Science, 16, 759–762.
http://dx.doi.org/10.1111/j.1467-9280.2005.01610.x
Rule, N. O. (2011). The influence of target and perceiver
race in the categorisation of male sexual orientation.
Perception, 40, 830–839. http://dx.doi.org/10.1068/
p7001
Rule, N. O., Adams, R. B., Jr., Freeman, J. B., & Ambady,
N. (2012). Perceptions of dominance following
glimpses of faces and bodies. Perception, 41,
687–706. http://dx.doi.org/10.1068/p7023
Rule, N. O., & Ambady, N. (2008a). Brief exposures:
Male sexual orientation is accurately perceived
at 50 ms. Journal of Experimental Social Psychology,
44, 1100–1105. http://dx.doi.org/10.1016/
j.jesp.2007.12.001
Rule, N. O., & Ambady, N. (2008b). The face of success:
Inferences from chief executive officers’ appearance
predict company profits. Psychological Science,
19, 109–111. http://dx.doi.org/10.1111/
j.1467-9280.2008.02054.x
Rule, N. O., & Ambady, N. (2009). She’s got the look:
Inferences from female chief executive officers’ faces
predict their success. Sex Roles, 61, 644–652. http://
dx.doi.org/10.1007/s11199-009-9658-9
Rule, N. O., & Ambady, N. (2011). Face and fortune:
Inferences of personality from managing partners’
faces predict their law firms’ financial success.
Leadership Quarterly, 22, 690–696. http://dx.doi.
org/10.1016/j.leaqua.2011.05.009
Rule, N. O., Ambady, N., Adams, R. B., Jr., & Macrae, C. N.
(2008). Accuracy and awareness in the perception
and categorization of male sexual orientation. Journal
of Personality and Social Psychology, 95, 1019–1028.
http://dx.doi.org/10.1037/a0013194
Rule, N. O., Ambady, N., Adams, R. B., Jr., Ozono, H.,
Nakashima, S., Yoshikawa, S., & Watabe, M. (2010).
Polling the face: Prediction and consensus across
cultures. Journal of Personality and Social Psychology,
98, 1–15. http://dx.doi.org/10.1037/a0017673
Rule, N. O., Ambady, N., & Hallett, K. C. (2009).
Female sexual orientation is perceived accurately,
rapidly, and automatically from the face and its
253
Re and Rule
features. Journal of Experimental Social Psychology,
45, 1245–1251. http://dx.doi.org/10.1016/
j.jesp.2009.07.010
Rule, N. O., Garrett, J. V., & Ambady, N. (2010a). On
the perception of religious group membership
from faces. PLoS ONE, 5, e14241. http://dx.doi.
org/10.1371/journal.pone.0014241
Rule, N. O., Garrett, J. V., & Ambady, N. (2010b). Places
and faces: Geographic environment influences the
ingroup memory advantage. Journal of Personality
and Social Psychology, 98, 343–355. http://dx.doi.
org/10.1037/a0018589
Rule, N. O., Ishii, K., Ambady, N., Rosen, K. S., &
Hallett, K. C. (2011). Found in translation: Cross-
cultural consensus in the accurate categorization
of male sexual orientation. Personality and Social
Psychology Bulletin, 37, 1499–1507. http://dx.doi.
org/10.1177/0146167211415630
Rule, N. O., Rosen, K. S., Slepian, M. L., & Ambady, N.
(2011). Mating interest improves women’s
accuracy in judging male sexual orientation.
Psychological Science, 22, 881–886. http://dx.doi.
org/10.1177/0956797611412394
Rule, N. O., Tskhay, K. O., Brambilla, M., Riva, P.,
Andrzejewski, S. A., & Krendl, A. C. (2015). The
relationship between anti-gay prejudice and the
categorization of sexual orientation. Personality
and Individual Differences, 77, 74–80. http://dx.doi.
org/10.1016/j.paid.2014.12.038
Samochowiec, J., Wanke, M., & Fiedler, K. (2010).
Political ideology at face value. Social Psychological
and Personality Science, 1, 206–213. http://dx.doi.
org/10.1177/1948550610372145
Samuels, C. A., Butterworth, G., Roberts, T., Graupner, L., &
Hole, G. (1994). Facial aesthetics: Babies prefer
attractiveness to symmetry. Perception, 23, 823–831.
http://dx.doi.org/10.1068/p230823
Scheib, J. E., Gangestad, S. W., & Thornhill, R. (1999).
Facial attractiveness, symmetry and cues of good
genes. Proceedings of the Royal Society B: Biological
Sciences, 266, 1913–1917. http://dx.doi.org/10.1098/
rspb.1999.0866
Scott, I. M. L., Clark, A. P., Boothroyd, L. G., & Penton-
Voak, I. S. (2013). Do men’s faces really signal
heritable immunocompetence? Behavioral Ecology,
24, 579–589. http://dx.doi.org/10.1093/beheco/ars092
Secord, P. F. (1958). Facial features and inferential
processes on interpersonal perception. In R. Tagivri &
L. Petrullo (Eds.), Person perception and interpersonal
behavior (pp. 300–316). Stanford, CA: Stanford
University Press.
Secord, P. F., Dukes, W. F., & Bevan, W. (1954).
Personalities in faces: I. An experiment in social
perceiving. Genetic Psychology Monographs, 49,
231–279.
254
Sell, A., Cosmides, L., Tooby, J., Sznycer, D., von Rueden,
C., & Gurven, M. (2009). Human adaptations for
the visual assessment of strength and fighting ability
from the body and face. Proceedings of the Royal
Society B: Biological Sciences, 276, 575–584. http://
dx.doi.org/10.1098/rspb.2008.1177
Sergent, J., Ohta, S., & MacDonald, B. (1992). Functional
neuroanatomy of face and object processing: A
positron emission tomography study. Brain: A
Journal of Neurology, 115, 15–36. http://dx.doi.
org/10.1093/brain/115.1.15
Setchell, J. M., & Dixson, A. F. (2001). Circannual
changes in the secondary sexual adornments
of semifree-ranging male and female mandrills
(Mandrillus sphinx). American Journal of Primatology,
53, 109–121. http://dx.doi.org/10.1002/1098-
2345(200103)53:3<109::AID-AJP2>3.0.CO;2-I
Shackelford, T. K., & Larsen, R. J. (1997). Facial
asymmetry as an indicator of psychological,
emotional, and physiological distress. Journal of
Personality and Social Psychology, 72, 456–466.
http://dx.doi.org/10.1037/0022-3514.72.2.456
Stamatas, G. N., Zmudzka, B. Z., Kollias, N., & Beer, J. Z.
(2004). Non-invasive measurements of skin
pigmentation in situ. Pigment Cell Research, 17,
618–626. http://dx.doi.org/10.1111/
j.1600-0749.2004.00204.x
Stephen, I. D., Coetzee, V., Law Smith, M., & Perrett, D. I.
(2009). Skin blood perfusion and oxygenation colour
affect perceived human health. PLoS ONE, 4, e5083.
http://dx.doi.org/10.1371/journal.pone.0005083
Stephen, I. D., Law Smith, M. J., Stirrat, M. R., & Perrett,
D. I. (2009). Facial skin coloration affects perceived
health of human faces. International Journal of
Primatology, 30, 845–857. http://dx.doi.org/10.1007/
s10764-009-9380-z
Stewart, J. E. (1980). Defendant’s attractiveness as a factor in
the outcome of criminal trials: An observational study.
Journal of Applied Social Psychology, 10, 348–361. http://
dx.doi.org/10.1111/j.1559-1816.1980.tb00715.x
Stirrat, M., & Perrett, D. I. (2010). Valid facial cues
to cooperation and trust: Male facial width and
trustworthiness. Psychological Science, 21, 349–354.
http://dx.doi.org/10.1177/0956797610362647
Swaddle, J. P., & Reierson, G. W. (2002). Testosterone
increases perceived dominance but not attractiveness
in human males. Proceedings of the Royal Society B:
Biological Sciences, 269, 2285–2289. http://dx.doi.
org/10.1098/rspb.2002.2165
Symons, D. (1995). Beauty is in the adaptations of the
beholder: The evolutionary psychology of human
female sexual attractiveness. In P. R. Abramson &
S. D. Pinkerton (Eds.), Sexual nature, sexual culture
(pp. 80–120). Chicago, IL: University of Chicago
Press.

Tanaka, J. W., & Farah, M. J. (1993). Parts and
wholes in face recognition. Quarterly Journal
of Experimental Psychology, Section A: Human
Experimental Psychology, 46, 225–245. http://dx.doi.
org/10.1080/14640749308401045
Theeuwes, J., & Van der Stigchel, S. (2006). Faces
capture attention: Evidence from inhibition of
return. Visual Cognition, 13, 657–665. http://dx.doi.
org/10.1080/13506280500410949
Thornhill, R., & Gangestad, S. W. (1996). The evolution
of human sexuality. Trends in Ecology and Evolution,
11, 98–102. http://dx.doi.org/10.1016/0169-
5347(96)81051-2
Thornhill, R., & Gangestad, S. W. (2006). Facial
sexual dimorphism, developmental stability,
and susceptibility to disease in men and women.
Evolution and Human Behavior, 27, 131–144. http://
dx.doi.org/10.1016/j.evolhumbehav.2005.06.001
Thornhill, R., & Møller, A. P. (1997). Developmental
stability, disease and medicine. Biological Reviews
of the Cambridge Philosophical Society, 72, 497–548.
http://dx.doi.org/10.1017/S0006323197005082
Thornton, M. J., Nelson, L. D., Taylor, A. H., Birch, M. P.,
Laing, I., & Messenger, A. G. (2006). The modulation
of aromatase and estrogen receptor alpha in cultured
human dermal papilla cells by dexamethasone: A
novel mechanism for selective action of estrogen
via estrogen receptor beta? Journal of Investigative
Dermatology, 126, 2010–2018. http://dx.doi.
org/10.1038/sj.jid.5700344
Tigue, C. C., Pisanski, K., O’Connor, J. J. M., Fraccaro,
P. J., & Feinberg, D. R. (2012). Men’s judgments of
women’s facial attractiveness from two- and three-
dimensional images are similar. Journal of Vision, 12,
3. http://dx.doi.org/10.1167/12.12.3
Tinlin, R. M., Watkins, C. D., Welling, L. L. M., DeBruine,
L. M., Al-Dujaili, E. A. S., & Jones, B. C. (2013).
Perceived facial adiposity conveys information
about women’s health. British Journal of Psychology,
104, 235–248. http://dx.doi.org/10.1111/
j.2044-8295.2012.02117.x
Toledano, E. (2013). The looking-glass ceiling:
Appearance-based discrimination in the workplace.
Cardozo Journal of Law and Gender, 19, 683–714.
Tomkinson, G. R., & Olds, T. S. (2000). Physiological
correlates of bilateral symmetry in humans.
International Journal of Sports Medicine, 21, 545–550.
http://dx.doi.org/10.1055/s-2000-8479
Tovée, M. J., & Cornelissen, P. L. (2001). Female and male
perceptions of female physical attractiveness in front-
view and profile. British Journal of Psychology, 92,
391–402. http://dx.doi.org/10.1348/000712601162257
Tovée, M. J., Maisey, D. S., Emery, J. L., & Cornelissen, P. L.
(1999). Visual cues to female physical attractiveness.
Appearance and Physiognomy
Proceedings of the Royal Society B: Biological Sciences,
266, 211–218. http://dx.doi.org/10.1098/rspb.1999.0624
Tovée, M. J., Reinhardt, S., Emery, J. L., & Cornelissen,
P. L. (1998). Optimum body-mass index and
maximum sexual attractiveness. Lancet, 352, 548.
http://dx.doi.org/10.1016/S0140-6736(05)79257-6
Tovée, M. J., Swami, V., Furnham, A., & Mangalparsad, R.
(2006). Changing perceptions of attractiveness as
observers are exposed to a different culture. Evolution
and Human Behavior, 27, 443–456. http://dx.doi.
org/10.1016/j.evolhumbehav.2006.05.004
Tremblay, R. E., Schaal, B., Boulerice, B., Arseneault, L.,
Soussignan, R. G., Paquette, D., & Laurent, D.
(1998). Testosterone, physical aggression, dominance,
and physical development in early adolescence.
International Journal of Behavioral Development, 22,
753–777. http://dx.doi.org/10.1080/016502598384153
Tskhay, K. O., Feriozzo, M. M., & Rule, N. O. (2013).
Facial features influence the categorization of female
sexual orientation. Perception, 42, 1090–1094.
http://dx.doi.org/10.1068/p7575
Tskhay, K. O., & Rule, N. O. (2013). Accuracy in
categorizing perceptually ambiguous groups: A
review and meta-analysis. Personality and Social
Psychology Review, 17, 72–86. http://dx.doi.
org/10.1177/1088868312461308
Tybur, J. M., Lieberman, D., & Griskevicius, V. (2009).
Microbes, mating, and morality: Individual
differences in three functional domains of disgust.
Journal of Personality and Social Psychology, 97,
103–122. http://dx.doi.org/10.1037/a0015474
Undurraga, E. A., Eisenberg, D. T. A., Magvanjav, O.,
Wang, R., Leonard, W. R., McDade, T. W., . . . Godoy,
R. A. (2010). Human’s cognitive ability to assess facial
cues from photographs: A study of sexual selection
in the Bolivian Amazon. PLoS ONE, 5, e11027. http://
dx.doi.org/10.1371/journal.pone.0011027
Valentine, T., Darling, S., & Donnelly, M. (2004). Why
are average faces attractive? The effect of view and
averageness on the attractiveness of female faces.
Psychonomic Bulletin and Review, 11, 482–487.
http://dx.doi.org/10.3758/BF03196599
van den Berghe, P. L., & Frost, P. (1986). Skin color
preference, sexual dimorphism and sexual selection:
A case of gene culture coevolution. Ethnic and Racial
Studies, 9, 87–113. http://dx.doi.org/10.1080/
01419870.1986.9993516
Walker, P. L. (2001). A bioarchaeological perspective
on the history of violence. Annual Review
of Anthropology, 30, 573–596. http://dx.doi.
org/10.1146/annurev.anthro.30.1.573
Watkins, C. (2011). Dominant themes. Psychologist, 24,
550–551.
Watkins, C. D., Jones, B. C., & DeBruine, L. M. (2010).
Individual differences in dominance perception:
255
Re and Rule
Dominant men are less sensitive to facial cues
of male dominance. Personality and Individual
Differences, 49, 967–971. http://dx.doi.org/10.1016/
j.paid.2010.08.006
Watkins, C. D., Quist, M. C., Smith, F. G., Debruine,
L. M., & Jones, B. C. (2012). Individual differences
in women’s perceptions of other women’s
dominance. European Journal of Personality, 26,
79–86. http://dx.doi.org/10.1002/per.837
Whitehead, R. D., Coetzee, V., Ozakinci, G., & Perrett,
D. I. (2012). Cross-cultural effects of fruit and
vegetable consumption on skin color. American
Journal of Public Health, 102, 212–213. http://dx.doi.
org/10.2105/AJPH.2011.300495
Whitehead, R. D., Ozakinci, G., & Perrett, D. I. (2012).
Attractive skin coloration: Harnessing sexual
selection to improve diet and health. Evolutionary
Psychology, 10, 842–854.
Whitehead, R. D., Re, D., Xiao, D., Ozakinci, G., & Perrett,
D. I. (2012). You are what you eat: Within-subject
increases in fruit and vegetable consumption confer
beneficial skin-color changes. PLoS ONE, 7, e32988.
http://dx.doi.org/10.1371/journal.pone.0032988
Wierson, M., Long, P. J., & Forehand, R. L. (1993).
Toward a new understanding of early menarche:
The role of environmental stress in pubertal timing.
Adolescence, 28, 913–924.
Windhager, S., Schaefer, K., & Fink, B. (2011).
Geometric morphometrics of male facial shape
in relation to physical strength and perceived
attractiveness, dominance, and masculinity.
American Journal of Human Biology, 23, 805–814.
http://dx.doi.org/10.1002/ajhb.21219
Wong, E. M., Ormiston, M. E., & Haselhuhn, M. P.
(2011). A face only an investor could love: CEOs’
256
facial structure predicts their firms’ financial
performance. Psychological Science, 22, 1478–1483.
http://dx.doi.org/10.1177/0956797611418838
Zahavi, A. (1975). Mate selection—A selection for a
handicap. Journal of Theoretical Biology, 53, 205–214.
http://dx.doi.org/10.1016/0022-5193(75)90111-3
Zebrowitz, L., & Lee, S. Y. (1999). Appearance,
stereotype-incongruent behavior, and
social relationships. Personality and Social
Psychology Bulletin, 25, 570–585. http://dx.doi.
org/10.1177/0146167299025005003
Zebrowitz, L. A., & Collins, M. A. (1997). Accurate social
perception at zero acquaintance: The affordances
of a Gibsonian approach. Personality and Social
Psychology Review, 1, 204–223. http://dx.doi.
org/10.1207/s15327957pspr0103_2
Zebrowitz, L. A., Kikuchi, M., & Fellous, J. M. (2010).
Facial resemblance to emotions: Group differences,
impression effects, and race stereotypes. Journal
of Personality and Social Psychology, 98, 175–189.
http://dx.doi.org/10.1037/a0017990
Zebrowitz, L. A., & Montepare, J. M. (2005). Psychology:
Appearance does matter. Science, 308, 1565–1566.
http://dx.doi.org/10.1126/science.1114170
Zebrowitz, L. A., Wadlinger, H. A., Luevano, V. X.,
White, B. M., Xing, C., & Zhang, Y. (2011). Animal
analogies in first impressions of faces. Social
Cognition, 29, 486–496. http://dx.doi.org/10.1521/
soco.2011.29.4.486
Zebrowitz-McArthur, L., & Apatow, K. (1984).
Impressions of baby-faced adults. Social
Cognition, 2, 315–342. http://dx.doi.org/10.1521/
soco.1984.2.4.315
 Chapter 10
Facial Expressions
Hyisung C. Hwang and David Matsumoto
Of the various channels of nonverbal behaviors, the
face is one of the most complex signal systems in
the body, containing four sources of information
(Ekman, 1972; Ekman & Friesen, 1978). Static signs
refer to facial physiognomy, which include the bony
structure of the skull; features including the size,
shape, and location of the eyes, brows, nose, and
mouth; and skin pigmentation. Slow signs include
bags, sags, and pouches; wrinkles; blotches; and
facial and scalp hair. Artificial signs include glasses,
cosmetics, alteration of hair, and so forth. Finally,
rapid signs include appearance changes to the face
due to contractions of the underlying facial muscles
that move skin and change the shape of features.
Each of these is an important source of information.
This chapter is concerned with facial
expressions—the rapid signals of the face. (For a
review of the effects of the static, slow, and artificial
signs of the face, see Chapter 9, this handbook.)
As we discuss later, people’s faces contain many
muscles that can produce literally thousands of
different types of expressions. Moreover, muscle
contractions of the face are under the neural control
of two different areas of the brain, one controlling
voluntary movements and the other involuntary
reactions. These two facets make research on the
face complicated and challenging, yet exciting and
informative.
Within the area of facial expressions, the bulk of
the research has focused on emotional expressions.
Studies on facial expressions of emotion have their
roots in Darwin’s (1872/1998) seminal work that
suggested that emotions are biologically innate and
http://dx.doi.org/10.1037/14669-010
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
evolutionarily adaptive and that facial expressions
evolved as a universal skill in all humans.
We review evidence supporting this view later in
this chapter, demonstrating not only the universality
but also the biological innateness of facial expres-
sions of emotion, including evidence from studies of
humans across cultures, blind individuals, twins and
families, infants, and nonhuman primates.
We also review evidence concerning the coherence
between facial expressions and other components of
the emotion response package, the mechanics and
functions of facial expressions of emotion, and the
influence of culture on facial expressions of emo-
tion. We also present evidence about what else facial
expressions signal, including signs of other affective
states, individual differences, mental and physical
health, deception, cognitive states, and conversation
markers.
One very broad area of research related to facial
expressions of emotion concerns perceptions and
judgments of them. This area includes research
examining presentation effects on judgments, the
neural substrates underlying judgments, categorical
versus dimensional perception of facial expres-
sions of emotion, gender differences, and differ-
ences among groups of individuals. Unfortunately,
space restrictions prevent us from reviewing
the totality of the evidence in this area of research
(but for related discussions, see Chapters 4
and 9, this handbook). Instead, we review only that
evidence concerning judgments of facial expres-
sions of emotion that is directly related to questions
concerning the universality of facial expressions of
257
Hwang and Matsumoto
emotions. Space restrictions also prevent us from
reviewing the major debates in this area (but see
Matsumoto & Hwang, 2013a, for a review of those
debates). We begin our review with a primer on the
facial anatomy to ground readers with the structure
of the facial muscles that produce expressions.
FACIAL ANATOMY PRIMER
Facial expressions are produced by muscle contrac-
tions in the face that move skin and change the
shape of features, producing the appearance changes
we know as expressions. These muscle movements
are known as the morphology of the face
(as opposed to the structure of the face and skull,
which is known as physiognomy), and expressions
are also referred to as morphological changes. There
are 20 anatomically independent muscles in the
face, all of which are innervated by a single nerve,
the VIIth cranial nerve, which originates in the brain
stem. There are additional muscles in the head that
control movements of the jaw and neck area; these
are controlled by a different nerve (the Vth cranial
nerve). The facial muscles are striated muscles and
are some of the only muscles in the body that are
attached on one side to bone and other side to skin.
The facial muscles, therefore, are somewhat special-
ized for communication.
Although there are structurally 20 indepen-
dent muscles in the face, they function in different
ways. For example, the muscle that runs across
the forehead (frontalis) is a single muscle (Gray &
Goss, 1966), but it can produce two independent
movements—one in the inner corners of the brows
and one in the outer corners. The muscles that bring
the brows down and together are actually composed
of three muscles (the corrugator muscle group),
even though in adults they only function together.
For this reason, facial measurement systems, such as
the Facial Action Coding System (FACS; Ekman &
Friesen, 1978; see also Chapter 17, this handbook),
identify more than 40 independent facial actions
according to function, not structure. This allows
for the production of literally thousands of different
expressions.
There are individual differences in the presence
or absence of some facial muscles. For example,
258
zygomatic minor is not present in all humans.
However, cadaver studies have demonstrated that,
despite individual differences in the existence of
the total number of facial muscles, there is uniform
occurrence in the muscles that are required to pro-
duce the universal facial expressions of emotion.
Furthermore, the same facial musculature that
exists in adult humans exists in newborn infants
and is fully functional for most individuals at birth
(Ekman & Oster, 1979).
The VIIth cranial nerve and, thus, facial expres-
sions are under the neural control of two distinct
areas of the brain, which roughly coincide with a
voluntarily versus involuntary distinction
(Matsumoto & Lee, 1993). The pyramidal tract
drives voluntary facial actions and originates in the
cortical motor strip, whereas the extrapyramidal
tract drives involuntary emotional expressions
and originates in the subcortical areas of the brain
(see Figure 10.1). The subcortical areas of the brain
are those areas that humans share with their ani-
mal relatives; these areas are concerned with body
functions related to fighting, fleeing, feeding, and
reproducing. The cortical motor strip is the area
of the brain that is unique to humans and controls
FIGURE 10.1.  Neuroanatomical roots of emotional
expressions. From http://commons.wikimedia.org/wiki/
Category:Medical_illustrations_by_Patrick_Lynch#
mediaviewer/File:Head_facial_nerve_branches.jpg,
Creative Commons Attribution 2.5 License 2006.
Copyright 2006 by Patrick J. Lynch. Reprinted with
permission.

voluntary movement. The part of people’s bodies
that receives the largest degree of innervation from
this area—and, thus, that is under the greatest
degree of voluntary control—is the hands. The next
largest part is people’s faces, and within the face, the
lower face is more highly represented (Rinn, 1984).
This makes sense because people learn to use the
many muscles that they have in their lower faces
for eating, speech articulation, and controlling their
emotions and words.
Therefore, when emotions are elicited, spontane-
ously produced expressions are under the neural
control of the subcortical areas of the brain. When
individuals attempt to control their expressions,
however, those impulses are likely to emanate from
the cortical motor strip.
FACIAL EXPRESSIONS OF EMOTION
The Evolutionary Origins of the Study of
Facial Expressions of Emotion
Early research on facial expressions was rooted in
Darwin’s (1872/1998) evolutionary theory. Accord-
ing to Dewsbury (2009), Darwin made three main
contributions to the understanding of evolution.
First, Darwin offered a viable mechanism for evolu-
tionary change and natural selection. He observed,
for example, that individual differences in heritable
appearances and populations are capable of increas-
ing exponentially. Second, he claimed that evolu-
tionary change is totally explicable by natural causes
without any inherent purpose or predetermined
direction. This theory of naturalism would affect
many developing disciplines. Third, he accumulated
massive amounts of supporting evidence across
phylogenetic lines (e.g., data on beetles, barnacles,
and people from different cultures).
One of Darwin’s (1872/1998) interesting ideas
concerned the continuity between human and
nonhuman animals. Humans are different from
all other species in many respects. However, we
all share a common origin and core fundamental
properties. This does not mean that there are no
differences, only that the issues of similarities and
differences are open to study using similar meth-
ods with different species. Many of Darwin’s works
were about instinctive behavior ranging from the
Facial Expressions
supernatural and theological to the natural. Darwin
facilitated the study of individual differences within
species because such differences were essential for
selection to work.
Darwin (1872/1998) believed that emotions and
their expressions are functionally adaptive and
biologically innate (see also Chapter 3, this hand-
book). In particular, Darwin claimed in his principle
of serviceable associated habits that facial expres-
sions are the residual actions of more complete
behavioral responses. For example, Darwin sug-
gested that humans express anger by furrowing the
brow and tightening the lips with teeth displayed
because these actions are part of an attack response.
Facial expressions occur in combination with other
bodily responses—vocal, postural, gestural, and
skeletal muscle movements—and are elements of a
coordinated response involving multiple response
systems.
Darwin (1872/1998) also believed that all
humans, regardless of race or culture, possessed the
ability to express emotions on faces in similar ways.
Darwin wrote The Expression of the Emotions in Man
and Animals to refute the claims of Sir Charles Bell,
the leading facial anatomist of his time and a teacher
of Darwin’s, about how God designed humans with
unique facial muscles to express uniquely human
emotions. Relying on advances in photography and
anatomy (e.g., the work of Duchenne de Boulogne,
1862/1990), Darwin engaged in a detailed study of
the muscle actions involved in emotion and con-
cluded that the muscle actions are universal and
that their antecedents can be seen in the expressive
behaviors of nonhuman primates and other mam-
mals. This analysis set the stage for the development
of coding systems used in the identification of facial
expression (see Chapter 17, this handbook), and
these have been central to the empirical literatures.
In addition to the principle of serviceable asso-
ciated habits, Darwin (1872/1998) suggested an
inhibition hypothesis in that people are unable to
perfectly simulate facial expressions in the absence
of the genuine emotion, and they are unable to com-
pletely suppress their true expressions when feeling
strong emotions, resulting in emotional leakage on
the face. This leakage is a result of the direct link
between affective regions of the brain and the facial
259
Hwang and Matsumoto
muscles (see further discussion on this issue in the
Facial Expressions Are Reliable Leakage Clues to
Deception section).
Although Darwin (1872/1998) did have data, his
ideas and work were criticized by others, notably
anthropologists (Birdwhistell, 1970), who reported
immense variations in expressive behavior across
cultures and insisted that facial expressions are
culture specific, learned like spoken languages that
are different across cultures or countries. In the first
half of the 20th century, there were only seven stud-
ies that attempted to test the universality or culture
specificity of facial expression, but these studies
were inconclusive (see review in Ekman, Friesen, &
Ellsworth, 1972). Thus, an influential review of the
literature (Bruner & Tagiuri, 1954) concluded that
facial expressions were not universal but culturally
learned.
The First Studies to Document the
Universality of Facial Expressions of
Emotion
It was not until almost a century after Darwin that
the first systematic evidence for the universality of
facial expressions of emotion appeared. In the mid-
1960s, Tomkins (1962, 1963) resurrected interest in
the study of emotions and faces with the publication
of his landmark volumes entitled Affect, Imagery,
and Consciousness. Tomkins and McCarter (1964)
conducted the first study demonstrating that facial
expressions were reliably judged to be associated
with certain emotional states, and later studies
showed consistent and similar findings (Ekman,
Sorenson, & Friesen, 1969; Izard, 1971).
Those initial findings were criticized, however,
because the evidence for universality (i.e., the high
levels of cross-cultural agreements in judgments)
might have occurred because of influences of
mass media (e.g., TV) and shared visual input
(e.g., Hollywood movies, magazines). To address
these potential limitations, Ekman and colleagues
conducted two studies with two visually isolated,
preliterate tribes in the highlands of New Guinea
(Ekman & Friesen, 1971; Ekman et al., 1969).
In the first study, the tribespeople could reliably
recognize facial expressions of emotion (anger,
disgust, fear, happiness, surprise, sadness) posed by
260
Westerners; in the second study, films of the tribes-
people expressing emotions were shown to Ameri-
cans who had never seen New Guineans before, and
the Americans were able to recognize the expres-
sions of the New Guineans. Thus, the ability to
recognize facial expressions of emotion in the earlier
studies did not occur because of learning through
mass media or other shared visual input, as the
New Guineans had no exposure to the outside world
at that time.
One of the most important original findings
related to the universality of facial expressions of
emotions came from Friesen’s (1972) cross-cultural
study of expressions that occurred spontaneously in
reaction to emotion-eliciting films. In that study,
25 American and Japanese participants watched
neutral and highly stressful films, respectively
(first episode: body mutilation; second episode:
sinus surgery) while their faces were recorded.
­Coding of the facial behaviors that occurred indi-
cated that the American and Japanese participants
displayed emotions (e.g., disgust, fear, sadness,
anger) similarly when they were alone and that
these expressions corresponded to the facial expres-
sions portrayed in the stimuli used in the previous
judgment studies. This study provided the first
evidence that facial expressions of emotion were
universally produced when emotions were sponta-
neously elicited.
Subsequent Research Documenting the
Universality of Facial Expressions of
Emotion
Since the pioneering studies described earlier that
­provided the first systematic evidence for the uni-
versality of facial expressions of emotion, many
studies examining judgments of the same facial
expressions of emotion have replicated the find-
ing of universal recognition of emotion in the face
(Matsumoto, 2001). This evidence includes cross-
cultural judgments of spontaneous expressions
(Matsumoto, Olide, Schug, Willingham, & Callan,
2009). For example, Biehl et al. (1997) used stimuli
that validly and reliably portrayed emotional expres-
sions (the Japanese and Caucasian Facial Expres-
sions of Emotion; Matsumoto & Ekman, 1988) and
demonstrated that the seven emotions were reliably

identified in Hungary, Japan, Poland, Sumatra, the
United States, and Vietnam. In addition, a meta-­
analysis of 168 data sets examining judgments of
those emotions in the face and other nonverbal behav-
iors indicated universal emotion recognition well
above chance levels (Elfenbein & Ambady, 2002).
Even when low intensity expressions are judged,
there is strong agreement across cultures in judgment
(Ekman et al., 1987; Matsumoto et al., 2002).
The original universality studies provided
­evidence for the cross-cultural expression and
­recognition of six emotions—anger, disgust, fear,
happiness, sadness, and surprise. Later, contempt
was also identified as a universally recognized
expression (Ekman & Heider, 1988; Matsumoto,
1992, 2005). The evidence for the universality of
the contempt expression was documented in a study
involving 10 cultures (Ekman & Friesen, 1986) and
was later replicated in additional cultures, including
Japan; Vietnam; Poland; Hungary; and the
Minangkabau in Sumatra, Indonesia (Ekman &
Friesen, 1986; Ekman & Heider, 1988; Matsumoto,
2005; Matsumoto & Ekman, 2004). Today, there
is strong evidence for the universal r­ ecognition of
seven facial expressions of emotion.
Not only are the seven universal facial expres-
sions panculturally recognized (Elfenbein &
Ambady, 2002; Matsumoto, 2001) but cultures
are similar in other aspects of emotion judgment
as well. For example, there is pancultural similar-
ity in judgments of relative intensity among faces;
that is, when comparing expressions, people of dif-
ferent countries agree on which is more strongly
expressed (Ekman et al., 1987; Matsumoto &
Ekman, 1989). There is also cross-cultural agree-
ment in the association between perceived expres-
sion intensity and inferences about subjective
experiences (Matsumoto, Kasri, & Kooken, 1999)
and in the secondary emotions portrayed in an
expression (Biehl et al., 1997; Ekman et al., 1987;
Matsumoto & Ekman, 1989).
Besides judgment studies, there have been more
than 75 studies that have demonstrated that the
facial expressions of anger, contempt, disgust,
fear, joy, sadness, and surprise are universally
­produced when emotions are elicited spontane-
ously (see review in Matsumoto, Keltner, Shiota,
Facial Expressions
O’Sullivan, & Frank, 2008). For example, a recent
study examined the expressions of 84 judo athletes
from 35 countries at the 2004 Athens Olympic
Games (Matsumoto & Willingham, 2006). The
spontaneous facial expressions of winners and
­losers that were first observed at the completion of
their final medal match were consistent with the
universal expressions. Winners displayed Duch-
enne/genuine smiles, whereas losers displayed sad-
ness, disgust, anger, and other negative emotions.
Duchenne smiles are smiles that involve not only
the smiling muscle (zygomatic major), which raises
the lip corners, but also the muscles surrounding
the eyes (orbicularlis oculi), which raise the cheeks,
thin the eyes, and narrow the eye cover fold; many
studies have shown that only these types of smiles
are correlated with experienced positive emo-
tion (see Ekman, Davidson, & Friesen, 1990, for a
review). Although there were many other cross-cul-
tural studies of facial expressions of emotion, this
Olympic study was encouraging for several reasons.
First, the results of the study were based on sponta-
neous, not posed, expressions. Second, the results
were produced by individuals from many differ-
ent cultures of the world. Third, the findings were
impressive given that previous research findings
documenting the universality of facial expressions
of emotions had been mainly tested in laboratory,
not field, conditions (albeit in different laboratories
using different methodologies with participants
from many different cultures around the world,
but all converging on the same pattern of results).
Thus, today there is robust e­ vidence for the univer-
sality of seven facial expressions of emotions (see
Figure 10.2).
Sources of the Universal Facial
Expressions of Emotion
Merely documenting the universality of emotional
expression in many cultures around the world does
not address questions concerning the source of the
universality. Facial expressions of emotion may be
universal because of at least two reasons. First, emo-
tional expressions may be a biologically innate skill
that all humans are born with. Or second, they may
be a skill that is learned in the same way all around
the world in different cultures through culture
261
Hwang and Matsumoto
Surprise
Happy Sadness
Contempt
FIGURE 10.2.  The seven universal expressions of emotion. Copyright by David Matsumoto. Reprinted with permission.
constant learning. Demonstrating cross-cultural
agreement in either the production or judgment
of expressions does not address which of the two
sources may produce the agreement; other meth-
odologies are necessary to do so. Here, we briefly
review some of the representative studies in these
areas, all of which point to a biologically innate
source of universality.
Studies of blind individuals.  One of the critical
challenges to the notion of the biological innateness
of emotion is that humans can easily (or entirely)
learn and imitate emotional expressions from oth-
ers. Blind individuals who are limited in observing
and imitating others’ behaviors compared to sighted
people are an exceptional group in which to explore
the pure effect of biologically wired systems on the
universality of emotions. This is especially true for
studies involving congenitally blind individuals
because they are expected to be have limited social
learning about how to produce sophisticated facial
muscle movements of each emotion because they
could not visually learn them from birth.
262
Fear
Anger
Disgust
Many similarities between blind and sighted
individuals in their spontaneous facial expressions
of emotion have been reported, including stud-
ies of congenitally blind individuals. For example,
researchers have measured the spontaneous facial
behaviors when emotions were aroused in blind
children (Cole, Jenkins, & Shott, 1989) and adults
of many different cultures (Galati, Miceli, & Sini,
2001; Galati, Sini, Schmidt, & Tinti, 2003), and
they have reported similarities in facial expressions
between blind and sighted individuals. One of the
early studies in this area was that of Thompson
(1941), who examined whether congenitally blind
children differentially produce emotions on their
faces compared to sighted children. The results
indicated a substantial similarity between the two
groups, especially for expressions of emotions such
as smiling and laughter or anger. Fulcher (1942)
conducted a similar study with a group of 50 con-
genitally blind individuals and a group of 118
sighted people. The findings were consistent with
those of previous studies. People in the two groups
similarly posed the expressions of joy, sadness,

anger, and fear; no gender variation was found.
The same finding was obtained in a study examining
whether congenitally blind participants and sighted
participants could pose expressions of fear, anger,
surprise, disgust, and humor (Rinn, 1991).
Galati et al. (2003) examined 10 congenitally
blind and 10 sighted children ranging in age from
7 years, 8 months to 11 years in Italy and reported
that the children expressed similar facial expressions
of emotions in social interactions that elicited
emotions such as fear, disgust, sadness, surprise,
anger, or joy in their classroom. Although the blind
children used certain facial movements producing
more mouth opening, eye closing, or head move-
ments, the sighted children frequently masked their
negative emotions more than blind children.
The studies reviewed earlier, however, did
not directly compare cultural variations among
the expressions of blind individuals, and most of
the studies tested posed, not spontaneous, facial
expressions in experimental contexts rather than
naturally, emotionally aroused contexts. These gaps
were addressed by a later study of the spontaneous
expressions of blind athletes at the 2004 Athens
Paralympic Games. Matsumoto and Willingham
(2009) examined spontaneous facial expressions of
emotions of congenitally and noncongenitally blind
judo athletes at these Paralympic Games, which was
a highly intense and emotional event for any ath-
lete. The blind athletes, who came from 23 cultures,
produced the same facial configurations of emotion
documented in sighted athletes in the same
emotionally evocative situations occurring in the
2004 Athens Olympic Games (see Matsumoto &
Willingham, 2006). The study also found high
concordance between the blind and sighted athletes
in their expressions. Winners displayed all types of
smiles, especially Duchenne smiles, more frequently
than the defeated athletes, who displayed more
disgust, sadness, and combined negative emotions.
When receiving the medal, all athletes smiled;
however, winners of the last match (gold and bronze
medalists) displayed Duchenne smiles more
frequently than did the defeated (silver medalists),
who displayed more non-Duchenne smiles.
Collectively, evidence for the similarity of facial
expressions of emotions between blind and sighted
Facial Expressions
individuals is compelling, and it strongly suggests
that facial expressions of emotion are biologically
innate. Studies of congenitally blind individuals
have been especially important because it is impos-
sible for congenitally blind individuals to produce
by imitation the complicated facial expressions
involved in complex muscle combinations fired
spontaneously in less than a second when they
experience an emotion; they would not have these
automatic reactions unless they were born with the
capability of experiencing and expressing the
emotions in the same way as sighted individuals.
We believe that these studies provide convincing
evidence for a biologically based, emotion-expression
linkage that is universal.
Evidence from twin and family studies.  Another
source of evidence for the biological origins of
emotion-expression linkages comes from studies of
twins and family relatives. Facial behaviors of blind
individuals are more concordant with their kin than
with strangers (Peleg et al., 2006); in this study’s
facial movement analysis during an individual
­interview, the correlations between movements of
21 congenitally blind subjects with those of their
30 relatives, especially in relation to such expres-
sions as sadness or anger, were significantly more
similar to each other than with nonfamily members.
The results provided evidence for a unique family
facial expression signature.
Moreover, some facial expressions in response
to emotionally provocative stimuli were more
­concordant among monozygotic twin pairs than
dizygotic twins when age and gender were con-
trolled for (Kendler et al., 2008). The monozygotic
twins showed significant correlations in expres-
sions of happiness, surprise, and anger. Considering
that the twins had minimal contact with each other
before their tests and were reared apart, the results
supported genetic effects on the facial expressions
of certain emotions. These studies supported the
consistent conclusion concerning the universality
of facial expressions of emotion and suggested that
facial expressions of emotions are hereditable and
genetic.
Evidence from the developmental literature. 
More evidence for the biological basis of facial
263
Hwang and Matsumoto
expressions of emotion comes from the develop-
mental literature. Infants have a rich and varied
repertoire of facial expressions, including those that
signal not only emotional states but also interest
and attention (Oster, 2005). There is widespread
consensus that smiling, distaste (the infant pre-
cursor of adult disgust), and crying (the univer-
sal signal of sadness/distress) occur in neonates
(Oster, 2005). A later study (Dondi et al., 2007)
focused on neonatal smiling observed among 21
newborns and reported that babies showed dis-
cretely non-Duchenne and Duchenne smiles. There
is some controversy as to when differentiated and
discrete negative emotions occur. Some authors
suggest that discrete negative emotions exist from
birth or shortly t­ hereafter and emerge according
to a maturational timetable (Izard, 1991; Izard &
Malatesta, 1987; Tronick, 1989). Others suggest that
infants, at least within the first year of life, display
relatively undifferentiated or modulated negative
expressions, which ultimately transform into more
­differentiated, discrete expressions later (Camras,
Oster, Campos, & Bakemand, 2003; Oster, 2005).
Discrete expressions of anger and sadness have been
reported in the early part of the second year of life
(Hyson & Izard, 1985; Shiller, Izard, & Hembree,
1986). Regardless, by the time of preschool, children
display discrete expressions of the other emotions
as well (Casey, 1993). Not only facial expressions
of emotions but also limb gestures and whole body
movements among infants (2–23 months) and
young children (2–5 years) are distinguishable and
communicative (Sherida, 1977).
As previously mentioned, early observational
studies concluded that congenitally blind children
and adults showed the same facial expressions of
emotion as sighted individuals, both spontaneously
(Dumas, 1932; Eibl-Eibesfeldt, 1973; Freedman,
1964; Goodenough, 1932; Thompson, 1941) and
posed (Fulcher, 1942). This work suggested that
facial expressions of emotion are good candidates
for being considered genetically encoded, universal
social behaviors that are present in the first stages
of neonate maturation, prior to extensive cultural
input. It is difficult to conceive of how facial expres-
sions of emotion are produced if the children did
not have the biological capability to do so in the first
264
place. Although it is still a mystery as to whether
and when babies can completely obtain and develop
all emotions that adults experience and express,
many findings indicate that there is a genetically
connected baseline of human emotions from birth.
Relatedly, language development parallels emo-
tional expression development. Human children
are not born speaking a language, but they make
sounds; these sounds eventually congeal into words
and unfold over time in a fairly regular fashion
into ­language. This language development process
is uncontroversially agreed upon to be an innate
­capability (e.g., Chomsky, 1965).
Evidence from nonhuman primates.  The facial
expressions considered to be universal among
humans also have been observed in nonhuman
primates (de Waal, 2003; see also Chapter 16,
this handbook). For years, ethologists (Chevalier-
Skolnikoff, 1973; Geen, 1992; Hauser, 1993;
Snowdon, 2003; Van Hoof, 1972) have noted the
morphological similarities between human expres-
sions of emotion and nonhuman primate expres-
sions displayed in similar contexts. Van Hoof
(1972) described the evolution of the smile and
laugh along two different evolutionary tracts across
early ­mammals, monkeys, apes, chimpanzees, and
humans. Redican (1982) suggested that nonhu-
man primates and humans both showed grimaces
and open-mouth grimaces as a threat display and
that nonhuman primates displayed a play face that
was similar to the happy face of humans. He also
­suggested that the nonhuman pout served a similar
function to the human sad face.
Ueno, Ueno, and Tomonaga (2004) demon-
strated that infant chimpanzee expressions were
more similar to human facial expressions than
to that of rhesus macaques. However, even some
of the smaller apes, such as siamangs (Symphal-
angus ­syndactylus), noted for their limited facial
­expressions repertoire, have distinguishable facial
expressions accompanying sexuality, agonistic
behavior, grooming, and play (Liebal, Pika, &
Tomasello, 2004). de Waal (2002) suggested that for
some states a species less closely related to humans
than chimpanzees, the bonobos, may have more
emotions in common with humans.

The most recent research has gone beyond dem-
onstrating equivalence in morphological descrip-
tions of expressions to identifying the exact facial
musculature used in producing the expressions
being described. Indeed, the strongest support
for the biological bases of facial expression would
be demonstration of homology in facial expres-
sion between humans and related species, such as
­chimpanzees (Pan troglodytes). This would suggest
that as humans evolved during the 120 million years
of primate evolution, facial muscles and expressions
similarly evolved, presumably to serve similar social
functions. Thus, one of the first questions to address
concerns the similarities in the facial musculature
between humans and nonhuman primates. ­Burrows,
Waller, Parr, and Bonar (2006) have reported that
the forehead musculature of chimps is less well
developed than that of humans. (They speculated
that the greater hairiness of chimps makes eyebrow
movements less visible, hence less communicative.)
However, many other facial muscles and expres-
sions have homologues and analogues comparable
to those descriptions in the human FACS (Ekman &
Friesen, 1978).
Based on the work described earlier, a chim-
panzee version of FACS (ChimpFACS) was devel-
oped that allows for identification of the specific
facial muscle movements chimpanzees use in
producing facial expressions. Chimpanzees have
a functional facial musculature that, although not
as differentiated as that of humans, includes the
same muscles that are used in human emotional
expressions (Bard, 2003; Burrows et al., 2006).
The chimpanzee facial musculature produces
many of the similar appearance changes as does
the human musculature, according to a compari-
son of the human and chimpanzee versions of the
FACS (Vick, Waller, Parr, Smith Pasqualini, &
Bard, 2007). Chimpanzees as well as rhesus
macaques can categorize by selecting images of
facial expressions of emotion much as humans do
(Parr, Waller, Burrows, Gothard, & Vick, 2010;
Parr, Waller, & Heintz, 2008; Waller, Lembeck,
Kuchenbuch, Burrows, & Liebal, 2012). The most
recent work in this area has also produced a FACS
that can be used with orangutans and gibbons
(Waller et al., 2012).
Facial Expressions
Furthermore, chimpanzees display a complex,
flexible facial expression repertoire with many phys-
ical and functional similarities to humans
(Parr et al., 2008). Consistently, chimpanzees pro-
duce distinct laughs depending on context and
interactants, similar to human beings (Davila-Ross,
Allcock, Thomas, & Bard, 2011). A recent study
by Rosati and Hare (2013) documented that chim-
panzees had human-like patterns of choice and
decision-making processes that are involved in
cognition, emotion, and motivation; thus, it seems
reasonable to suspect they would affect expression
similarly as well.
Summary.  The evidence reported in this
­section points to the notion that the emotions
that are u­ niversally expressed and recognized are
­dominantly biologically innate and hardwired. These
findings have led to research that has suggested that
the universally expressed and recognized emotions
belong to a specific class of emotions that has cer-
tain specific and unique characteristics. This class
of emotions is known as basic emotions (Ekman,
1999). One of the core principles of basic emotions
is that each of the components of an ­emotional
response—facial expressions, voice, physiology, and
so forth—should be related to each other in a coher-
ent and organized manner. In the next section, we
discuss the evidence for this coherence with facial
expressions of emotion.
The Coherence Between Facial
Expressions and Other Emotion
Responses
Darwin (1872/1998) suggested, in his principle
of serviceable habits, that facial expressions are
the residual actions of more complete behavioral
responses. By implication, facial expressions should
be coordinated with other reactions when emotions
are triggered, such as subjective experience and
autonomic or neuroendocrine responses, because
this coordination should enable individuals to
respond quickly and adaptively. Researchers refer to
this possibility in terms of emotion packages, emo-
tion response system coherence, or response covari-
ation (Bonanno & Keltner, 2004; Ekman, 1992a,
1992b; Lazarus, 1991; Levenson, 1994). Here, we
265
Hwang and Matsumoto
review a broad swath of studies that indicate that
facial expressions of emotion covary systematically
with subjective experience, appraisal processes,
physiological response, and specific actions.
Links with subjective experience.  There is ample
evidence that indicates that the universal expres-
sions are reliable signals of an internal feeling state
when there is no reason to modify or manage them.
Several studies have reported correlations between
the universal emotional expressions and self-reports
of the experience of discrete emotions (see Table 10.1).
These studies involved a variety of different types
of emotion elicitors, researchers, and experimental
designs. Perhaps the most salient aspect of these
findings is that linkage between discrete facial
expressions of emotion and self-reports of the
same emotional states is stronger in within-subject
designs that involve precise, second-to-second
­measurement of both expression and experi-
ence (e.g., Rosenberg & Ekman, 1994). In Mauss,
Levenson, McCarter, Wilhelm, and Gross’s (2005)
study, cross-lagged, within-individual correlations
between facial behavior and experience intensity for
amusing and sadness-eliciting films were rs = .73
and .74, respectively. When correlations were
­corrected for disattenuation, they were even higher,
rs = .89 and .97, respectively.
Moreover, important nonfindings in these stud-
ies need to be considered. In Bonanno and Keltner’s
(1997) study, for instance, whereas anger, con-
tempt, and sadness were positively correlated with
grief, fear and disgust were not. In Ekman, Friesen,
and Ancoli’s (1980) study, expressions of disgust
were positively correlated with self-report of disgust
but were negatively correlated with self-reports
of anger and sadness. In Harris and Alvarado’s
(2005) study, Duchenne smiles were correlated
with self-reports of happiness and amusement but
not with anxious, angry, or embarrassed. These
findings indicated that the discrete facial expres-
sions do not coincide with valence dimensions of
pleasantness–unpleasantness (e.g., Russell & Fehr,
1987) but with discrete subjective experiences.
Several lines of research provide convergent
evidence. Matsumoto and Kupperbusch (2001)
reported significant correlations between judged
266
expressions and self-reported experience. Spon-
taneous laughter and smiling were found to have
some distinct experiential correlates (Keltner &
Bonanno, 1997). The intensity of laughter or
­smiling correlated with self-reports of the funni-
ness of the humorous stimuli (McGhee, 1977; Ruch,
1995). Matsumoto et al. (1999) demonstrated high,
positive correlations between ratings of perceived
expression intensity and inferences about subjective
experiences across cultures. Matsumoto, Nezlek,
and Koopmann (2007) reported that the relation-
ship between self-reported emotion intensity and
expressive behavioral reactions was consistent
across approximately 3,000 participants in 27 coun-
tries and was not moderated by culture.
Several methodological issues are noteworthy
to mention, one involving studies of smiles. Early
research did not differentiate among different types
of smiles, only correlating activity of zygomatic
major with self-reports of positive emotional expe-
rience (Ekman et al., 1980). Since then, however,
several studies that involved fine-grained analyses
of the smiles that occurred when people actually
felt enjoyment have demonstrated that smiles that
involve the orbicularis oculi (the muscle surround-
ing the eyes)—the Duchenne smile—are reliably
related to positive subjective experiences of joy and
enjoyment (Ekman et al., 1990; Ekman & Friesen,
1982; Ekman, Friesen, & O’Sullivan, 1988; Frank &
Ekman, 1993; Frank, Ekman, & Friesen, 1993;
­Keltner & Bonanno, 1997; Matsumoto & Willingham,
2006). Smiles without activity of the orbicularis
oculi are not associated with positive subjective
experience. Individuals with Duchenne smiles
are also perceived as being more pleasant, genu-
ine, ­outgoing, expressive, sociable, calm, natural,
relaxed, honest, trustworthy, and likeable compared
to individuals expressing non-Duchenne smiles of
similar intensity (Frank et al., 1993). Thus, stud-
ies examining smiles need to make this distinction
in facial measurement. The Duchenne smile is the
­universal signal of enjoyment and is reliably cor-
related with subjective experience (and specific
autonomic and brain activity signatures; see Ekman
et al., 1990; Levenson, Ekman, & Friesen, 1990).
Duchenne smiles have been correlated with the
experience of positive emotion in young and old
 Facial Expressions

TABLE 10.1

Studies Reporting Significant Correlations Between Spontaneous Facial Expressions of Emotion and
Self-Reports of Specific Emotions

Citationa Facial expressions measured Self-reports obtained Correlation

Bonanno & Keltner (1997) Duchenne laughing Grief (concurrent) −.39*
Anger .36*
Contempt .31*
Sadness .34*
Keltner & Bonanno (1997) Duchenne smiles Distress −.49**
Fear −.31*
Enjoyment .35*
Duchenne laughing Distress −.36*
Enjoyment .34*
Bonanno & Keltner (2004) Duchenne smiles Distress −.44*
Duchenne laughing Anger −.51**
Ekman et al. (1980) Smiling (frequency) Happiness .60**
Smiling (duration) Happiness .35*
Smiling (intensity) Happiness .34*
Disgust (frequency) Disgust .37*
Anger −.35*
Sadness −.46**
Disgust (duration) Disgust .55**
Fear .46*
Pain .41*
Ekman et al. (1990) Duchenne smiles Amusement .70*
Happiness .59*
Excitement .39***
Interest .40***
Anger −.38***
Sadness −.44***
Harris & Alvarado (2005) Duchenne smiles Happiness .19† (humor condition)
Amusement .28**
Mauss et al. (2005) Duchenne smiles Amusement .73***
Sadness Sadness .74***
Rosenberg & Ekman (1994)b Disgust and fear Disgust and fear .71 (rat film)
.90 (amputation film)
.83 (amputation film)
Ruch (1994) Duchenne smiles Positive affectivity .33* (experimental group)
.52* (control group)
.78* (rank order of cell means)
Verbal enjoyment .28*
Ruch (1995) Duchenne smiles Funniness .55 (between-subjects design, aggregate data)
.96 (within-subjects design, aggregate data)
.61 (between-subjects design, raw data)
.71 (within-subjects design, raw data)
.63 (across all stimuli and subjects design)
aGosselin et al. (1995) obtained self-reports but did not report correlations between the ratings and facial expressions.
bThe statistics reported for this study are the probability of co-occurrence between the rating of specific emotion categories
and the corresponding facial expression.
* p < .05. ** p < .01. *** p < .001. †p < .10.

267
Hwang and Matsumoto
adults (Frank et al., 1993; Hess, Banse, & Kappas,
1995; Keltner & Bonanno, 1997; M. C. Smith, 1995)
and have distinguished nonharassed and harassed
job applicants (Woodzicka & LaFrance, 2001).
A second methodological note concerns the tim-
ing of the assessments of self-reported emotions and
expressions. Because facial expressions of emotion
are transient, studies that assess global self-reports
of emotion after an event are much less likely to
demonstrate associations between expressions and
self-reports. Rosenberg and Ekman (1994) showed
that facial expressions of emotion correlated signifi-
cantly with self-reported emotions when both were
assessed at specific points in time.
Links with distinct appraisals.  If facial expres-
sions are part of a coherent package, then they
should be related to the appraisals that elicit emo-
tions in the first place. Two theoretical perspec-
tives provide evidence in support of this linkage.
The first is based on the notion of the existence
of innate affect programs that store the full-face
configurations of the expressions (Ekman, 2003).
This theory suggests that individuals are born with
the capacity to produce the emotional expressions
when emotions are elicited, and when specific
themes associated with the appraisal of a potentially
emotion-eliciting event occur, they produce an
emotional reaction (including facial expressions).
Some evidence supports this view. For example,
bereaved adults’ facial expressions of anger and
sadness while discussing their deceased spouse
co-occurred with distinct appraisal themes (jus-
tice and loss) coded from participants’ spontane-
ous narratives that were contemporaneous with
the facial expressions (Bonanno & Keltner, 1997).
Posing facial expressions of anger was related to
the appraisal that others were responsible for social
events, whereas posing facial sadness was associ-
ated with the appraisal that the same events were
due to situational causes (Keltner, Ellsworth, &
Edwards, 1993). Optimistic appraisals are associ-
ated with discrete expressions of fear, anger, and
disgust, and they mediated the relationship between
facial expression and physiological response (Lerner,
Gonzalez, Dahl, Hariri, & Taylor, 2005). Individuals
imagining situations high in perceived obstacles to
268
goals are more likely to show furrowed eyebrows
(C. A. Smith, 1989; C. A. Smith & Scott, 1989).
Furthermore, expressions of anger, contempt,
and disgust are reliably associated with appraisals
related to moral violations of autonomy,
community, and divinity (Rozin, Lowery, Imada, &
Haidt, 1999).
A second theory suggests that facial expres-
sions occur because they reflect the outcome of the
appraisal process of emotion elicitation (Scherer &
Ellgring, 2007). This theory suggests that differ-
ent components of the appraisal process evaluating
emotion elicitors unfold across time in sequence,
and each influences expressive behaviors and other
physiological reactions. That is, appraisals link
specific muscle actions with specific evaluative
steps in an appraisal process and would predict
that the facial components would be produced as a
result of their corresponding evaluative step in that
process. Raising the brows and lifting the upper
eyelid, for instance, increases the visual field, and
appraisal theories would predict that this facial
component is linked to an evaluative step associ-
ated with novel orientation. Some recent research
has begun to provide evidence for this view
(Scherer & Ellgring, 2007).
Links with distinct physiological responses. 
Facial expressions of emotion are coordinated with
physiology. When emotions are aroused and facial
expressions are used as markers of those emotions,
discrete physiological signatures occur in both the
autonomic nervous system and the brain (Davidson,
2003; Ekman et al., 1990; Ekman, Levenson, &
Friesen, 1983; Levenson & Ekman, 2002; Levenson
et al., 1990; Levenson, Ekman, Heider, & Friesen,
1992; Mauss et al., 2005; Tsai & Levenson, 1997).
Table 10.2, adapted from Levenson (2003), summa-
rizes the major findings to date. Emotional expres-
sions are linked with physiological changes that help
prepare individuals to act by initiating and perhaps
maintaining appropriate whole-body activity. These
patterns exist in people from cultures as widely
divergent as the United States and the Minangkabau
of West Sumatra, Indonesia.
Lerner et al. (2005) demonstrated that the
­discrete facial expressions of fear, anger, and
 Facial Expressions

TABLE 10.2

Changes in Appearance and Autonomic Nervous System (ANS) Activity Associated With Facial
Expressions of Emotion

Emotion Type of change Change ANS mediation

Anger Coloration Reddening Vasodilation, increased contractibility
Moisture and secretions Foaming Salivary glands
Protrusions Piloerection Muscle fibers at base of hair follicles
Blood vessels bulging Vasodilation
Eye appearance Constriction Pupils
Bulging Eyelid muscles
Disgust Moisture and secretions Salivating, drooling Salivary glands
Fear Coloration Blanching Vasoconstriction
Moisture and secretions Sweating, clamminess Sweat glands
Protrusions Piloerection Muscle fibers at base of hair follicles
Eye appearance Dilation Pupils
Bulging Eyelid muscles
Happiness Eye appearance Twinkling Lacrimal glands plus contraction of orbicularlis oculi
Sadness Moisture and secretions Tearing, crying Lacrimal glands
Embarrassment Coloration Blushing Vasodilation

Note. From “Blood, Sweat, and Fears,” by R. W. Levenson, 2003, Annals of the New York Academy of Sciences, 1000, p. 356.
Copyright 2003 by Wiley. Adapted with permission.

disgust were reliably linked not only to cardio-
vascular responses but to neuroendocrine activ-
ity as well. Participants were exposed to three
different types of stressors during which they
were videotaped, and their cardiovascular and
hypothalamic–pituitary–adrenocortical
(cortisol) responses were measured. Fear expres-
sions were associated with elevated cardiovas-
cular and cortisol levels; anger and disgust were
­associated with reduced responses. Matsumoto
et al. (2007) reported moderately sized correlations
between self-reported expressive behavior and three
types of physiological sensations (ergotropic, tro-
photropic, and felt temperature) in approximately
3,000 respondents from 27 countries. They also
reported consistent correlations between verbal and
nonverbal expressions as well as between emotion
intensity and physiological sensations, all of which
suggest coherence in an underlying neurophysi-
ological reality.
To be sure, research on the coherence of facial
expression and physiology has not always been
consistent. Some studies (Brown & Schwartz, 1980;
Cacioppo, Martzke, Petty, & Tassinary, 1988) dem-
onstrated only low correlations between expression
and physiological response, and some demon-
strated even no relationships (Buck, 1977; Mauss,
Wilhelm, & Gross, 2004). We believe these nega-
tive findings are the result of several methodologi-
cal factors: (a) defining emotion by the attempt to
manipulate it instead of the independent confirma-
tion of its elicitation, (b) the type of emotion elic-
ited, (c) the nature of the measures of emotional
responding used, (d) the temporal resolution of the
measurement (Mauss et al., 2005), (e) the fact that
the laboratory may not be the optimal context in
which to elicit physiological responses associated
with the behaviors that emotions enable people to
perform when dealing with real-life situations, and
(f) the difference between within- and between-
subjects designs. Evidence for the linkage between
facial expressions of emotion and physiological
responses comes from studies that address these
methodological issues well.
Links with subsequent behaviors.  Another
source of evidence supporting the link between
expression and emotional response comes from
studies that demonstrate a link between facial
expressions of emotion and subsequent behaviors.

269
Hwang and Matsumoto
Because emotion is an important basis of motivation
(Tomkins, 1962, 1963), and because facial expres-
sions are reliable signals of emotion, it comes as no
surprise that they can signal behavioral intent. In
the first study to demonstrate this effect, Ekman,
Liebert, et al. (1972) examined the relationship
between facial expressions of emotion produced
by children as they watched television and their
subsequent hurtful behaviors and aggressive play.
Children were videotaped as they watched either a
violent scene from a movie or competitive sports.
Afterward, they were placed in a situation where
they could either help another child or hurt them,
and then they engaged in a free play period. Boys
who smiled during the violent scenes engaged in
more hurtful behavior and aggressive play afterward;
boys who showed sadness during the violent scenes
engaged in more helpful behavior and less aggres-
sive play when the video was finished.
In a subsequent study, Matsumoto, Haan,
Yabrove, Theodorou, and Carney (1986) video-
taped the facial behaviors of preschool dyads as they
engaged in a Prisoner’s Dilemma game. The chil-
dren displayed rich and varied emotional responses,
and these were reliably linked to the actions of the
game. Cooperative behaviors elicited decreased neg-
ative emotion, whereas children who were defected
against expressed more non-Duchenne smiles and
positive/negative blends. Most importantly, the
expressions that occurred after an action predicted
the subsequent action. Children who expressed
Duchenne smiles after their partner cooperated were
more likely to also cooperate; those who expressed
anything else were more likely to defect. Defections
that were followed by non-Duchenne smiling were
more likely to lead to subsequent, repeated defec-
tions; when defections were followed by Duchenne
smiling, however, the subsequent act was likely to
be cooperative.
Keltner, Moffitt, and Stouthamer-Loeber (1995),
in their study of adolescent boys, examined relations
between facial expressions of emotion observed in
a 2-min interaction and teacher reports of social
behavior. Facial displays of anger observed in an
interactive IQ testing context correlated significantly
with teacher ratings of delinquent and aggressive
behavior at school; facial displays of fear correlated
270
negatively with these behaviors and correlated posi-
tively with withdrawal-related behaviors.
More recently, Matsumoto and Hwang (2013b)
demonstrated that specific variants of the uni-
versal facial expression of anger were reliably
associated with two types of acts of imminent
aggression—premeditated assault and loss of
impulse control assault. Collectively, these studies
provide evidence that specific facial expressions of
emotion are linked to specific, subsequent behav-
iors. Of course facial expressions of emotion do not
always link to behavior. Emotion prepares the body
for action; however, whether people actually engage
in action is dependent on many factors, such as
the context within which the emotion has occurred,
the target of the emotion, the perceived consequences
of one’s actions, previous experiences, and so forth
(Baumeister, Vohs, DeWall, & Zhang, 2007).
The Mechanics of Facial Expressions
of Emotion
Spontaneously produced facial expressions of emo-
tion have specific timing and laterality character-
istics (Ekman & Friesen, 1982; Ekman, Hager, &
Friesen, 1981; Hess & Kleck, 1990). That is, when
emotions are elicited spontaneously, the facial
muscles respond with relatively smooth onsets and
offsets, are innervated at approximately the same
time, and are symmetrical on both sides of the face
(the exception to the laterality feature would be
expressions of contempt, which occur unilaterally).
Voluntarily posed expressions, however, produce
expressions in which different muscles are activated
at different times and apex at different speeds and
are more asymmetrical (Hager & Ekman, 1985).
Recent research continues to provide evidence
for differences between voluntarily posed and spon-
taneously produced facial expressions. For example,
onset and offset speed, amplitude of movement, and
offset duration for zygomatic major is greater when
smiles are voluntarily posed (Schmidt, Ambadar,
Cohn, & Reed, 2006; Schmidt, Bhattacharya, &
Denlinger, 2009). Different morphology and
timing characteristics appear to be able to distin-
guish between smiles of amusement, politeness, and
embarrassment/nervousness (Ambadar, Cohn, &
Reed, 2009).

Even when posing expressions, some facial
­ uscles are more difficult to voluntarily produce
m people’s evolutionary history).
than others (Gosselin, Perron, & Beaupré, 2010).
Interestingly, those that are more difficult to pro-
duce voluntarily are in fact more reliable signals of
emotional states (Mehu, Mortillaro, Bänziger, &
Scherer, 2012). These differences occur because
neural representations to the facial muscles differ,
as described earlier. There is more representation
to the lower face, and most of this is contralaterally
represented in the brain; there is less representation
to the upper face, and relatively more of this repre-
sentation is bilateral (Matsumoto & Lee, 1993).
The Functions of Facial Expressions
of Emotion
Intrapersonal effects.  As previously described,
facial expressions co-occur with many other compo-
nents that are part of an emotion response package.
However, facial expressions have functions in their
own right. One of these functions is intrapersonal,
and Darwin (1872/1998) called these serviceable
habits. For example, the raising of the brows and
upper eyelid in expressions of fear and surprise
allows for a widening of the visual field, allowing for
individuals to see more objects in the environment.
It also allows for faster eye movements during object
localization, and an increase in nasal volume and air
velocity during inspiration (Susskind et al., 2008).
The eye widening of fear or surprise also enhances
stimulus detection, whereas eye narrowing enhances
stimulus discrimination (Lee, Mirza, Flanagan, &
Anderson, 2014). The dropping or opening of the
jaw also allows for increased oxygen intake, priming
the body for action. The wrinkling of the nose that
occurs in expressions of disgust helps to close off
nasal volume, allowing for a restriction of the intake
of objects nasally. This makes sense if disgust is an
emotion in response to contaminated objects; the
closing off of the nasal passages makes the passage
of contaminated objects into the body more difficult.
The raising of the inner corners of the brows and
muscle around the eyes in sadness stimulates the
lacrimal glands, which produce tears. The lowering
of the brows and glaring of the eyes in anger aids in
focusing on a target, whereas the baring of the teeth
Facial Expressions
prepares for attack responses (biting, a vestige of
Facial expressions also augment subjective emo-
tional experience. This effect is known as the facial
feedback hypothesis. Although different versions
of this hypothesis exist (Camras, Holland, & Pat-
terson, 1993; McIntosh, 1996), numerous studies
have demonstrated that facial expressions augment
contemporaneous subjective emotional experi-
ence (Hess, Kappas, McHugo, Lanzetta, & Kleck,
1992; Lanzetta, Biernat, & Kleck, 1982; Larsen,
Kasimatis, & Frey, 1992; McCanne & Anderson,
1987; Rutledge & Hupka, 1985; Soussignan, 2002;
Zuckerman, Klorman, Larrance, & Spiegel, 1981).
The effect occurs for both positive and negative
emotions (Larsen et al., 1992), and it influences
not only subjective experience but also autonomic
responses (Hess et al., 1992; Laird, 1974; Lanzetta
et al., 1982; Vaughan & Lanzetta, 1981; Zucker-
man et al., 1981). Facial feedback effects are most
powerful when the facial configurations correspond
to the universal emotional expressions (Soussignan,
2002), and numerous reviews of this literature over
the years have consistently reported such augmen-
tation effects (Camras et al., 1993; Laird, 1984;
­Matsumoto, 1987; McIntosh, 1996; Winton, 1986).
In some cases, researchers have suggested that facial
expressions can initiate emotional experience
(Levenson et al., 1990; McIntosh, 1996).
Interpersonal effects.  Facial expressions have
interpersonal effects because they evoke responses
from others, signal the nature of interpersonal
relationships, and provide incentives for desired
social behavior (Keltner, 2003). These interper-
sonal effects of facial expressions of emotion allow
individuals to solve social problems. For example,
because facial expressions of emotion are universal
social signals, they contain meaning not only about
the expressor’s psychological state but also about
that person’s intent and subsequent behavior. This
information affects what the perceiver is likely to
do. People observing fearful faces, for instance, are
more likely to produce approach-related behav-
iors, whereas people who observe angry faces are
more likely to produce avoidance-related behaviors
(Marsh, Ambady, & Kleck, 2005). This likely occurs
271
Hwang and Matsumoto
because angry faces increase the effectiveness and
enhance the credibility of threats (Reed, DeScioli, &
Pinker, 2014), suggesting that angry faces evolved
to enhance cues of strength (Sell, Cosmides, &
Tooby, 2014). Even subliminal presentation of
smiles produces increases in how much beverage
people pour and consume and how much they are
willing to pay for it; presentation of angry faces
decreases these behaviors (Winkielman, Berridge, &
Wilbarger, 2005). Emotional displays evoke spe-
cific, complementary emotional responses from
observers; for example, anger evokes fear in others
(Dimberg & Ohman, 1996; Esteves, Dimberg, &
Ohman, 1994), whereas distress evokes sympathy
and aid (Eisenberg et al., 1989).
Emotional expressions also provide information
about the nature of the relationships among inter-
actants. Some of the most important and provoca-
tive set of findings in this area come from studies
involving married couples (Gottman & Levenson,
1992; Gottman, Levenson, & Woodin, 2001). In
this research, married couples visited a laboratory
after having not seen each other for 24 hr, and then
engaged in intimate conversations about daily events
or issues of conflict. Discrete expressions of con-
tempt, especially by the men, and disgust, especially
by the women, predicted later marital dissatisfaction
and even divorce.
Facial expressions of emotion are also important
regulators of social interaction. In the developmen-
tal literature, this concept has been investigated
under the concept of social referencing (Klin-
nert, Campos, & Sorce, 1983); that is, the process
whereby infants seek out information from others to
clarify a situation and then use that information to
act. To date, the strongest demonstration of social
referencing comes from work on the visual cliff (a
solid table with a checkboard design, where half
way the checkboard design changes to clear glass
so it looks like a cliff, but still feels solid). In the
first study to investigate this concept, Sorce, Emde,
Campos, and Klinnert (1985) placed mothers on the
far end of the “cliff” from the infant. Mothers first
smiled to the infants and placed a toy on top of the
safety glass to attract them; infants invariably began
crawling to their mothers. When the infants were
in the center of the table, however, the mother then
272
posed an expression of fear, sadness, anger, inter-
est, or joy. The results were clearly different for the
different faces: No infant crossed the table when
the mother showed fear; only 6% crossed when the
mother posed anger; 33% crossed when the mother
posed sadness; and approximately 75% of the infants
crossed when the mother posed joy or interest.
Other studies provide similar support for facial
expressions as regulators of social interaction. In
one study (Bradshaw, 1986), experimenters posed
facial expressions of neutral, anger, or fear toward
babies as they moved toward an object and mea-
sured the amount of inhibition the babies showed
in touching the object. The results for 10- and
15-month olds were the same: Anger produced
the greatest inhibition, followed by disgust, with
neutral the least. This study was later replicated
(Hertenstein & Campos, 2004) using joy and dis-
gust expressions, altering the method so that the
infants were not allowed to touch the toy (compared
with a distractor object) until 1 hr after exposure to
the expression. At 14 months of age, significantly
more infants touched the toy when they saw joyful
expressions, but fewer touched the toy when the
infants saw disgust.
Sociocultural effects.  Because one of the major
functions of culture is to maintain social order
to ensure group efficiency and thus survival, cul-
tures create worldviews, rules, guidelines, and
norms concerning emotions and their expressions
because emotions have important intra- and inter-
personal functions, as described earlier, and are
important motivators of behavior (see also Chapter
4, this handbook). Norms concerning the regula-
tion of emotional expressions in all cultures serve
the purpose of maintaining social order. Cultural
worldviews and norms help individuals manage
and modify their expressions by facilitating cultur-
ally appropriate emotion elicitation in the first place
and by managing expressions once emotions are
triggered. By doing so, culturally moderated expres-
sions allow individuals to engage in socially appro-
priate behaviors, as defined by their cultures, and
thus reduce social complexity and increase social
order, avoiding social chaos and allowing humans
to live relatively harmonious and constructive lives

in groups. If cultural worldviews and norms about
emotions and expressions did not exist, people
would just run amok having all kinds of emotional
experiences, expressing their emotions and then
behaving in all sorts of unpredictable and potentially
harmful ways. If that were the case, it would be very
difficult for groups and societies to function effec-
tively, and even for humans to survive as a species,
if emotions were not regulated in culturally defined
ways for the common, social good. Thus, facial
expressions of emotion also have important socio-
cultural effects and play a critical role in the suc-
cessful functioning of any society and culture. In the
next section, we explore in more detail the influence
of culture on facial expressions of emotion.
The Influence of Culture
Despite the existence of universal facial expressions
of emotion, people around the world use the uni-
versal expressions differently. The first evidence for
cultural differences in expression was in Friesen’s
(1972) study. As mentioned earlier, in the first
condition of this study, Americans and Japanese
viewed the stressful films alone and produced the
same expressions. In a later condition, they viewed
the films in the presence of an older, male experi-
menter. Here, cultural differences emerged; whereas
the Americans continued to express their negative
emotions, the Japanese tended to mask the expres-
sion of those very same negative emotions they dis-
played earlier when the experimenter was observing
them with smiles. Later evidence demonstrated that
Japanese control not only the display of negative
feelings but also feelings of happiness more than
Americans do (Matsumoto, Takeuchi, Andayani,
Kouznetsova, & Krupp, 1998). Ekman and Friesen
(1969) coined the term cultural display rules to
account for cultural differences in facial expressions
of emotion. These are rules learned early in child-
hood that help individuals manage and modify their
emotional expressions depending on social circum-
stances. In the first condition of Friesen’s (1972)
experiment, there was no reason for display rules to
modify expressions because the participants were
alone and because their display rules were inopera-
tive; in the later condition, display rules dictated
that the Japanese mask their negative emotions in
Facial Expressions
the presence of the experimenter (Ekman, 1972; Fri-
esen, 1972).
There are a number of ways by which universal
expressions can be managed (Ekman & Friesen,
1969, 1975). Individuals can express emotions as
they feel them with no modification. However, indi-
viduals can also amplify (exaggerate) or deamplify
(minimize) their expressions; for instance, feelings
of sadness may be intensified (amplification) at
funerals or minimized (deamplification) at wed-
dings. People can mask or conceal their emotions
by expressing something other than what they feel,
as when nurses or physicians hide their emotions
when speaking with patients with terminal illness,
or when employees in service industries (e.g., flight
attendants) interact with customers. Individu-
als may also learn to neutralize their expressions,
expressing nothing, such as when playing poker
(poker face), or to qualify their feelings by express-
ing emotions in combination, such as when feel-
ings of sadness are mixed with a smile, with the
smile commenting on the sadness, saying “I’ll be
OK.” People can also simulate emotions, expressing
them even when they are not felt. These behavioral
responses have been found to occur when sponta-
neous expressive behaviors have been studied, even
in preschoolers (Cole, 1986). (For more discussion
about cultural display rules, see Chapter 4, this
handbook.)
As studies documenting cultural differences in
expression peppered the literature (Argyle, Hen-
derson, Bond, Iizuka, & Contarello, 1986; Edel-
mann et al., 1987; Gudykunst & Nishida, 1984;
Gudykunst & Ting-Toomey, 1988; Matsumoto &
Kupperbusch, 2001; Noesjirwan, 1978; Szarota, 2010;
Waxer, 1985), a consensus emerged that when emo-
tions are aroused, the displays are either universal or
culture specific, depending on context. A recent study
(Matsumoto, Willingham, & Olide, 2009), however,
showed that emotional displays can be both for the
same person in the same context, if displays are exam-
ined in sequence across time. In this study, changes
in Olympic athletes’ expressions after their initial
reactions were classified into one of several regulation
strategies, and the relationships between these expres-
sive styles and cultural variables (such as Hofstede’s,
2001, cultural dimensions; i.e., country-level scores
273
Hwang and Matsumoto
on the Individualism, Power Distance, Uncertainty
Avoidance, Masculinity, and Long-Term Orienta-
tion dimensions) and country demographics (such
as population density and affluence) were examined.
Although the athletes’ initial reactions were universal,
their subsequent expressions were culturally regulated
and were associated with population density, afflu-
ence, and individualism. Athletes from urban, indi-
vidualistic cultures expressed their emotions more;
athletes from less urban, more collectivistic cultures
masked their emotions more.
The average length of time from an initial,
­universal emotional expression to a culturally
­moderated modification was less than 1 s. This
suggests that when emotions are aroused, they trig-
ger an automatic set of reactions mediated by an
archaic, core emotion system in the brain that is
part of people’s evolutionary history. When that
occurs, individuals also process the cultural mean-
ing of those triggered emotions in that context, and
culturally moderated norms for expressive regula-
tion ­subsequently kick in. That the average length
of time from the immediate, biologically based
response to the culturally moderated one was less
than 1 s suggests a very quick cognitive processing
of cultural display rules in context.
WHAT ELSE DO FACIAL EXPRESSIONS
SIGNAL?
As we mentioned at the beginning of this chapter,
emotion is just one of the messages that facial
expressions can signal (and it is the signal that has
received the most research attention). The face,
however, is an extremely complex signal system
and is used to convey many other mental states as
well. In this final section, we review a broad range
of studies that exemplify the many types of signals
facial expressions produce. One of the biggest
mistakes that can be made when understanding
facial expressions is to overinterpret many differ-
ent types of facial behavior as emotional expres-
sions. This problem is compounded by the fact
that despite the large number of muscles people
have in the face, sometimes the same muscles
used in signaling emotions are also used in signal-
ing other states.
274
Facial Expressions Signal Other Emotions
and Affective States
Research has provided initial evidence that the face
signals emotions and affective states other than
the seven universal, biologically based expressions
described earlier. For example, there is some evi-
dence for the existence of an expression of pain
(LeResche, 1982; Prkachin & Craig, 1995), anxiety
(Perkins, Inchley-Mort, Pickering, Corr, & Bur-
gess, 2012), and sexual excitement (Fernández-
Dols, ­Carrera, & Crivelli, 2011). These recent,
interesting findings suggest that humans have dif-
ferent types of emotions, some of which may be
signaled in the face.
Even among positive emotions, early research
reported the existence of a single expression of a
positive emotion (happiness or joy). Recent studies
have provided evidence that different types of posi-
tive emotional states can be differentiated by specific
morphological characteristics of smiling. Thus, there
is evidence for unique signals of embarrassment,
amusement, shyness, sensory pleasure, elation, con-
tentment, and relief (Ambadar et al., 2009; Campos,
Shiota, Keltner, Gonzaga, & Goetz, 2013; Keltner,
1995; Keltner & Buswell, 1997; Krumhuber &
Scherer, 2011; Ricci-Bitti, Caterina, & Garotti,
1996). Smiles are also a part of displays of other
emotions, such as pride (Tracy & Robins, 2004) and
triumph (Hwang & Matsumoto, 2014a, 2014b;
Matsumoto & Hwang, 2012).
Emotional expressions are also signals of other
affective states and intentions. For example, spon-
taneously occurring smiles have also been linked to
trust, cooperation, and prosocial behavior, although
recent research suggests that overall expressiv-
ity, not limited to smiles, may be such a marker
(Boone & Buck, 2003; Schug, Matsumoto, Horita,
Yamagishi, & Bonnet, 2010).
Facial Expressions Signal Broad
Individual Differences
Facial expressions signal individual differences,
which by their very nature reflect broad tendencies
to think and act in a specific fashion. For example,
Bonanno and Keltner (1997) examined bereaved
adults’ facial expressions as they talked about their
recently deceased spouse. Facial expressions of

anger predicted increased grief severity 14 and
25 months postloss; laughing and smiling, however,
predicted reduced grief over time. Importantly,
facial expressions predicted long-term adjustment
independent of initial levels of grief and individual
differences in the tendency to report high levels of
distress. These findings suggested that even brief
facial expressions predict broad patterns of adapta-
tion to important life events.
Harker and Keltner (2001) coded women’s
­college yearbook photos and showed that Duchenne
smiling was positively correlated with multiple
­measures of personality (i.e., affiliation, warmth,
competence), personal well-being, and marital
satisfaction at various times over the next 30 years.
Abe and Izard (1999) measured discrete facial
expressions of emotion in 18-month-old infants
during episodes of the strange situation procedure,
and they correlated these with maternal ratings of
the five-factor model of personality when the chil-
dren were 3.5 years of age. Negative expressions
strongly correlated with Neuroticism and were
inversely related to Agreeableness and Conscien-
tiousness; full-face positive expressions were posi-
tively correlated with Extraversion and Openness
to Experience. These findings suggested that facial
expressions of emotions systematically covary with
the coherent patterns of thought, feeling, and action
captured in personality measures.
Facial Expressions Covary With Measures
of Mental and Physical Health
Because facial expressions are part of a set of
coherent responses to the environment, they are
related to measures of mental and physical health,
which capture maladaptive patterns of thought,
action, and feeling. Anger displays are related to
the incidence of ischemia in patients with coro-
nary artery disease (Rosenberg et al., 2001). The
oblique eyebrows and pressed lips of sympathy
correlate with reduced heart rate, whereas winces
of pain are related to elevated heart rate (Eisen-
berg et al., 1989). Facial expressions differentiate
between genuine pain, masked pain, and faked pain
(Craig, Hyde, & P ­ atrick, 1991; Prkachin, 1992),
and between healthy individuals and psychiatric
patients (Ellgring, 1986; Steimer-Krause, Krause, &
Facial Expressions
Wagner, 1990), schizophrenic and psychosomatic
patients (Steimer-Krause et al., 1990), schizo-
phrenic and depressed patients (Berenbaum & Olt-
manns, 1992; Ekman, Matsumoto, & Friesen, 1997;
Ellgring, 1986), s­ uicidal and nonsuicidal depressed
patients (Heller & Haynal, 1994), and patients
with major versus minor depression (Ekman et al.,
1997). Facial expressions of disgust (and glaring)
differentiate between ­individuals with Type A or B
personalities, and facial expressions of contempt,
anger, and disgust all correlated with various
speech indices of hostility, anger, competitive-
ness, and despondency (Chesney, Ekman, Friesen,
Black, & Hecker, 1990). Duchenne smiles differ-
entiate whether depressed patients are improving
as a result of treatment (Ekman et al., 1997), and
patients with right hemisphere damage are impaired
in the production of facial expressions of emotion,
particularly for positive emotion (Barod, Koff, Perl-
man Lorch, & Nicholas, 1986). Because of their
neural substrates, in fact, facial expressions were
often used as diagnostic markers of brain injury in
the past before the advent of functional magnetic
resonance imaging, computed tomography, or
positron emission tomography scans (Howieson &
Lezak, 2010).
Pathologies also affect the ability to pose and
­recognize emotional expressions. Abused chil-
dren, for instance, have difficulties both posing and
­recognizing facial expressions (Camras et al., 1988;
Pollak, Cicchetti, Hornung, & Reed, 2000; P ­ ollak &
Sinha, 2002). Severely autistic children have broad
and pervasive deficits in recognizing emotions (Hob-
son, 1986; Ozonoff, Pennington, & Rogers, 1990).
Children with high-functioning autism or Asperger’s
syndrome have emotion-specific deficits, generally
being able to recognize happiness, sadness, fear, and
anger (Capps, Yirmiya, & Sigman, 1992) but not
facial signals of embarrassment or shame (Heerey,
Keltner, & Capps, 2003). Individuals with high trait
anxiety recognize fear faces better than those without
such anxiety (Surcinelli, Codispoti, M ­ ontebarocci,
Rossi, & Baldaro, 2006), and ­individuals who are
depressed are generally worse at recognizing all
facial emotions (Persad & Polivy, 1993). Substance-
dependent individuals with a history of alcohol
abuse are also generally worse at recognizing facial
275
Hwang and Matsumoto
emotions (Foisy et al., 2005). Furthermore, patients
with myotonic dystrophy Type 1 have difficulty
recognizing angry, disgusted, and fearful faces (Win-
blad, Hellström, Lindberg, & Hansen, 2006).
Facial Expressions Are Reliable Leakage
Clues to Deception
The study of facial expressions as possible reliable
signs of deception is based in large part on the study
of micro-momentary facial expressions of emotion
known as microexpressions. These are extremely
quick facial expressions of emotion that are gener-
ally unobservable to the untrained person. The idea
that microexpressions exist has its roots in Darwin’s
(1872/1998) inhibition hypothesis that suggested
that facial actions that cannot be controlled
voluntarily may be produced involuntarily even if
the individual is trying to control his or her expres-
sions. The neuroanatomical bases of emotional
expressions discussed earlier in this chapter suggest
how this occurs. Facial expressions are under the
neural control of two distinct areas of the brain, one
controlling voluntary movements and one control-
ling spontaneously occurring, involuntary move-
ments. When individuals are in intensely emotional
situations but need to control their expressions, they
likely activate both systems, which engage in a neu-
ral “tug of war” over control of the face, allowing for
the quick, fleeting leakage of microexpressions.
Haggard and Isaacs (1966) were the first to
empirically verify the existence of microexpres-
sions in their studies of clinical interviews. After
Haggard and Isaacs’s discovery, Ekman and Friesen
(1969, 1974) studied them when examining films of
­psychiatric patients who had lied during a clinical
interview, and who had concealed either hallucina-
tions or plans to commit suicide. Ekman (1985)
later elaborated on them, speculating that microex-
pressions are reliable signs of deception.
Despite their widespread popularity as indicators
of deception, for years, however, the actual valid-
ity evidence for facial expressions of emotion, and
specifically microexpressions, to be valid indicators
of veracity and lying was sparse. Fortunately, recent
studies have begun to address this gap, and there is
growing evidence that when individuals lie, espe-
cially about their emotions (e.g., saying that one is
276
feeling pleasant when in fact one is outraged), signs
of the underlying, truly felt emotion appear in the
face despite attempts to conceal (Ekman et al., 1988;
Porter, ten Brinke, & Wallace, 2012). Attempts to
control one’s facial expressions of emotion, and
failing to do so, have also been linked to lying
(­Hurley & Frank, 2011).
Facial Expressions Signal Cognitive and
Physical States
People use their faces to signal when they are
thinking or confused, often furrowing their brows
or pursing their lips when concentrating, are
­perplexed, or when conducting mental searches.
Facial expressions also convey emblematic ­gestures;
these are movements that symbolically give v­ erbal
meaning that can be conveyed by words (see Chap-
ter 12, this handbook). Although emblematic
gestures are mainly displayed via hand movements,
some emblems are displayed in the face, such as the
disbelieving, doubtful look produced by raising the
upper lip and pushing the lower lip up, or skepti-
cism shown by a raised outer portion of a single
eyebrow. The eyebrow flash (the raising of the eye-
brows, often with a head nod up) is used as a greet-
ing in many cultures. Facial emblems include mock
astonishment, affirmation and negation, and skep-
ticism (Eibl-Eibesfeldt, 1972). Because emblems
are largely culture specific, we suspect that facial
emblems are also culture specific.
Facial expressions also signal cognitive control
over one’s emotions and words. These expressions
are produced by pushing the lower lip up, pressing
the lips together, and/or tightening the lip corners.
This same expression is often used when individuals
exert physical effort. In this case, the facial muscles
are used to intake large amounts of oxygen and to
hold it to facilitate the exertion.
Facial Expressions Play an Important Role
in Conversation
Facial expressions play several important roles
in conversation, and the amount of expressions
that occur is mediated by the degree of interac-
tion ­(Chovil, 1991). Many of the facial muscles in
the lower face are used for speech articulation, and
because different languages require the production

of different sounds, different languages may be asso-
ciated with unique expressions that are produced to
facilitate sound production.
Humans also use facial expressions to illus-
trate speech, just as they use their hands to do so
(see also Chapter 12, this handbook). The brows
in particular are used as such conversational sig-
nals, being raised or lowered when individuals
emphasize, punctuate, underline, or animate their
speech, or even raise questions (Birdwhistell, 1970;
Eibl-Eibesfeldt, 1972). These brow movements can
occur with or without speech, and they are often
associated with changes in vocal features. Individu-
als often raise their brows when being inquisitive or
when they raise their voices, and they lower their
brows when they lower their voices.
Facial expressions are also used by listeners in a
conversation as part of back-channel communica-
tion, that is, to provide feedback to the speaker that
the listener is hearing what is being said. The brows
are used to signal both agreement responses with
what is being said as well as calls for information
(Dittmann, 1972). The brows are also especially
used as signs of conversation regulation, moderat-
ing turn-taking in an interaction (Dittmann, 1972;
Kendon & Ferber, 1973). It is no wonder that brows
play important roles in social interaction related to
language and thought, as the existence of brows is
uniquely human and probably coevolved with the
capacity for speech.
CONCLUSION AND FUTURE DIRECTIONS
Darwin (1872/1998) originally suggested that
­emotions and their expressions had evolved across
species and were evolutionarily adaptive, biologi-
cally innate, and universal. Darwin’s idea has been
fully examined in numerous studies, as discussed
earlier. We started by reviewing the original and
subsequent evidence for the universal expression
and recognition of facial expressions of emotion.
We then reviewed evidence concerning the source
of the universality of facial expressions on emotions,
examining research findings on blind individuals,
twins, infants and children, as well as nonhuman
primates. Those reports provided a solid and consis-
tent conclusion that universal facial expressions of
Facial Expressions
emotions are biologically innate. This characteristic
of facial expressions of emotion has led to a greater
understanding of the class of emotions known as
basic emotions, which we then described.
At the same time, culture plays an important role
in moderating how the universal facial expressions
of emotion are used, via the cultural modulation of
what to become emotional about in the first place
and the regulation of expressive behaviors when
emotions are elicited. The interaction between
culture and facial expressions does not mean that
culture absolutely defines and creates facial expres-
sions and emotions but rather that culture is asso-
ciated with how one might socially modify facial
expressions when desirable or needed. It is crucial
to understand that the cultural variations in how to
display facial expressions are not interpreted as cul-
tural control over immediate behavioral reactions on
the face, as these are likely very difficult to control.
Instead, culture is an essential guideline for people
to socially modify their more voluntarily based facial
behaviors, which occur after immediate behavioral
reactions, to smooth their social interactions for
social coordination. Culture more heavily influences
learned expressions, such as those associated with
speech illustration or emblematic information.
There is still much to be learned, even about the
universal facial expressions of emotion. Future stud-
ies based on dynamic approaches to the concepts of
emotions stated earlier may be desirable if research-
ers can clarify which aspects of emotions they
examine in which contexts (for a related discussion,
see Chapter 9, this handbook). Domain and con-
text specificity in the field’s understanding of facial
expressions of emotion and their initial classifica-
tion will lead to clearer findings with less confusion
or unreasonable contradictions in the interpreta-
tion of results. Researchers need to more clearly
define emotions and the specific emotion domains
in which they are interested to approach conduct-
ing research and understanding their fi ­ ndings with
greater specificity and sophistication. For example,
research based on evolutionary theory, as stated
earlier, may focus on spontaneous expressions on
the face that immediately occur right after emotional
events because facial expressions are part of the evo-
lutionarily derived, biologically innate package of
277
Hwang and Matsumoto
emotion components. Research based on self-reports
of emotions that are based on memory or recall,
however, represents a different domain of emotion,
measuring emotions after immediate r­ eactions.
Thus, comparing findings from the two different
approaches, and in particular giving them equal
weight, may not make sense. Researchers should
fairly evaluate studies considering the domain of
emotion being examined.
Also, evolutionary theory does not neglect the
effects of context. In contrast, evolutionists believe
that universal behavioral reactions and cultural
­specific management of those reactions occur in
specific contexts. Yet, very few studies have system-
atically attempted to directly examine the effects
of context. To disentangle the unidentified and
seemingly contradicting layers produced by differ-
ent c­ ontexts, examining how culture specifically
interacts with universal behavioral reactions across a
wide variety of targeted contexts is an on-going task
in future research on facial expressions of emotions.
More studies concerning spontaneous facial
expressions of emotion in real life situations should
be considered for future research. Although research
in laboratory-based, experimental environments
is understandable and has a scientifically unique
meaning, conducting studies in real-life contexts
provides solid and practical results that can enhance
the applicability of research findings to real life.
This is especially true of the facial expressions of
basic emotions, which are rooted in spontaneous
behavioral reactions that usually occur in natural
settings rather than in experimental ones.
Future research will continue to explore the
wonder and power of facial expressions beyond
the universal facial expressions of emotion. Excit-
ing new work will continue to expand research-
ers’ knowledge concerning the role of the face in
other emotional and affective reactions. As with the
research on the universal, biologically based expres-
sions, future research should examine the facial
signs of pain, anxiety, sexual excitement, various
positive emotions, pride, triumph, trust, and cooper-
ation across cultures and correlates with other men-
tal and physiological processes. Future research will
need to examine and develop a taxonomy of facial
emblems and illustrators across cultures as well as
278
to investigate the role of the face in interaction and
groups. Despite the wealth of knowledge the field
has amassed in the last few decades of research, we
have only just begun to unravel the mystery and
complexity of the face.
References
Abe, J. A. A., & Izard, C. E. (1999). A longitudinal study
of emotion expression and personality relations in
early development. Journal of Personality and Social
Psychology, 77, 566–577. http://dx.doi.org/
10.1037/0022-3514.77.3.566
Ambadar, Z., Cohn, J. F., & Reed, L. I. (2009). All smiles
are not created equal: Morphology and timing of
smiles perceived as amused, polite, and embarrassed/
nervous. Journal of Nonverbal Behavior, 33, 17–34.
http://dx.doi.org/10.1007/s10919-008-0059-5
Argyle, M., Henderson, M., Bond, M., Iizuka, Y., &
Contarello, A. (1986). Cross-cultural variations
in relationship rules. International Journal
of Psychology, 21, 287–315. http://dx.doi.
org/10.1080/00207598608247591
Bard, K. A. (2003). Development of emotional
expressions in chimpanzees (Pan troglodytes).
Annals of the New York Academy of Sciences, 1000,
88–90. http://dx.doi.org/10.1196/annals.1280.018
Barod, J. C., Koff, E., Perlman Lorch, M., & Nicholas, M.
(1986). The expression and perception of
facial emotion in brain-damaged patients.
Neuropsychologia, 24, 169–180. http://dx.doi.
org/10.1016/0028-3932(86)90050-3
Baumeister, R. F., Vohs, K. D., DeWall, C. N., &
Zhang, L. (2007). How emotion shapes behavior:
Feedback, anticipation, and reflection, rather
than direct causation. Personality and Social
Psychology Review, 11, 167–203. http://dx.doi.
org/10.1177/1088868307301033
Berenbaum, H., & Oltmanns, T. F. (1992). Emotional
experience and expression in schizophrenia and
depression. Journal of Abnormal Psychology, 101,
37–44. http://dx.doi.org/10.1037/0021-843X.101.1.37
Biehl, M., Matsumoto, D., Ekman, P., Hearn, V., Heider, K.,
Kudoh, T., & Ton, V. (1997). Matsumoto
and Ekman’s Japanese and Caucasian Facial
Expressions of Emotion (JACFEE): Reliability
data and cross-national differences. Journal of
Nonverbal Behavior, 21, 3–21. http://dx.doi.
org/10.1023/A:1024902500935
Birdwhistell, R. L. (1970). Kinesics and context: Essays on
body motion communications. Philadelphia: University
of Pennsylvania Press.
Bonanno, G. A., & Keltner, D. (1997). Facial expressions
of emotion and the course of conjugal bereavement.

Journal of Abnormal Psychology, 106, 126–137.
http://dx.doi.org/10.1037/0021-843X.106.1.126
Bonanno, G. A., & Keltner, D. (2004). The coherence
of emotion systems: Comparing “on-line” measures
of appraisal and facial expressions, and self-report.
Cognition and Emotion, 18, 431–444. http://dx.doi.
org/10.1080/02699930341000149
Boone, R. T., & Buck, R. (2003). Emotional expressivity
and trustworthiness: The role of nonverbal
behavior in the evolution of cooperation. Journal
of Nonverbal Behavior, 27, 163–182. http://dx.doi.
org/10.1023/A:1025341931128
Bradshaw, D. (1986). Immediate and prolonged
effectiveness of negative emotion expressions in
inhibiting infants’ actions (Unpublished doctoral
dissertation). University of California, Berkeley.
Brown, S. L., & Schwartz, G. E. (1980). Relationships
between facial electromyography and
subjective experience during affective imagery.
Biological Psychology, 11, 49–62. http://dx.doi.
org/10.1016/0301-0511(80)90026-5
Bruner, J. S., & Tagiuri, R. (1954). The perception of
people. In G. Lindzey (Ed.), Handbook of social
psychology (Vol. 2, pp. 634–654). Reading, MA:
Addison-Wesley.
Buck, R. (1977). Nonverbal communication of affect in
preschool in children: Relationships with personality
and skin conductance. Journal of Personality and
Social Psychology, 35, 225–236. http://dx.doi.
org/10.1037/0022-3514.35.4.225
Burrows, A. M., Waller, B. M., Parr, L. A., & Bonar, C. J.
(2006). Muscles of facial expression in the
chimpanzee (Pan troglodytes): Descriptive,
comparative and phylogenetic contexts. Journal of
Anatomy, 208, 153–167. http://dx.doi.org/10.1111/
j.1469-7580.2006.00523.x
Cacioppo, J. T., Martzke, J. S., Petty, R. E., & Tassinary, L. G.
(1988). Specific forms of facial EMG response
index emotions during an interview: From Darwin
to the continuous flow hypothesis affect-laden
information processing. Journal of Personality
and Social Psychology, 54, 592–604. http://dx.doi.
org/10.1037/0022-3514.54.4.592
Campos, B., Shiota, M. N., Keltner, D., Gonzaga, G. C., &
Goetz, J. L. (2013). What is shared, what is different?
Core relational themes and expressive displays of eight
positive emotions. Cognition and Emotion, 27, 37–52.
http://dx.doi.org/10.1080/02699931.2012.683852
Camras, L. A., Holland, E. A., & Patterson, M. J. (1993).
Facial expression. In M. Lewis & J. M. Haviland
(Eds.), Handbook of emotions (pp. 199–208).
New York, NY: Guilford Press.
Camras, L. A., Oster, H., Campos, J. J., & Bakemand, R.
(2003). Emotional facial expressions in European-
American, Japanese, and Chinese infants. Annals of
Facial Expressions
the New York Academy of Sciences, 1000, 135–151.
http://dx.doi.org/10.1196/annals.1280.007
Camras, L. A., Ribordy, S., Hill, J., Martino, S., Spaccarelli, S.,
& Stefani, R. (1988). Recognition and posing of
emotional expressions by abused children and their
mothers. Developmental Psychology, 24, 776–781.
http://dx.doi.org/10.1037/0012-1649.24.6.776
Capps, L., Yirmiya, N., & Sigman, M. (1992).
Understanding of simple and complex emotions in
non-retarded children with autism. Journal of Child
Psychology and Psychiatry, 33, 1169–1182. http://
dx.doi.org/10.1111/j.1469-7610.1992.tb00936.x
Casey, R. J. (1993). Children’s emotional experience:
Relations among expression, self-report, and
understanding. Developmental Psychology,
29, 119–129. http://dx.doi.org/10.1037/0012-
1649.29.1.119
Chesney, M. A., Ekman, P., Friesen, W. V., Black, G. W., &
Hecker, M. H. (1990). Type A behavior pattern:
Facial behavior and speech components.
Psychosomatic Medicine, 52, 307–319. http://dx.doi.
org/10.1097/00006842-199005000-00005
Chevalier-Skolnikoff, S. (1973). Facial expression of
emotion in nonhuman primates. In P. Ekman (Ed.),
Darwin and facial expression (pp. 11–89). New York,
NY: Academic Press.
Chomsky, N. (1965). Aspects of the theory of syntax.
Cambridge, MA: MIT Press.
Chovil, N. (1991). Social determinants of facial displays.
Journal of Nonverbal Behavior, 15, 141–154.
http://dx.doi.org/10.1007/BF01672216
Cole, P. M. (1986). Children’s spontaneous control of
facial expression. Child Development, 57, 1309–1321.
http://dx.doi.org/10.2307/1130411
Cole, P. M., Jenkins, P. A., & Shott, C. T. (1989).
Spontaneous expressive control in blind and sighted
children. Child Development, 60, 683–688. http://
dx.doi.org/10.2307/1130733
Craig, K. D., Hyde, S. A., & Patrick, C. J. (1991).
Genuine, suppressed and faked facial behavior
during exacerbation of chronic low back pain.
Pain, 46, 161–171. http://dx.doi.org/10.1016/0304-
3959(91)90071-5
Darwin, C. (1998). The expression of the emotions in man
and animals. New York, NY: Oxford University Press.
(Original work published 1872)
Davidson, R. J. (2003). Parsing the subcomponents of
emotion and disorders of emotion: Perspectives from
affective neuroscience. In R. J. Davidson,
K. R. Scherer, & H. H. Goldsmith (Eds.), Handbook
of affective sciences (pp. 8–24). New York, NY:
Oxford University Press.
Davila-Ross, M., Allcock, B., Thomas, C., & Bard, K.
(2011). Aping expressions? Chimpanzees produce
279
Hwang and Matsumoto
distinct laugh types when responding to laughter
of others. Emotion, 11, 1013–1020. http://dx.doi.
org/10.1037/a0022594
de Waal, F. B. M. (2002). Apes from Venus: Bonobos and
human social evolution. In F. B. M. de Waal (Ed.),
Tree of origin: What primate behavior can tell us about
human social evolution (pp. 39–68). Cambridge, MA:
Harvard University Press.
de Waal, F. B. M. (2003). Darwin’s legacy and the study
of primate visual communication. Annals of the New
York Academy of Sciences, 1000, 7–31. http://dx.doi.
org/10.1196/annals.1280.003
Dewsbury, D. A. (2009). Charles Darwin and psychology
at the bicentennial and sesquicentennial: an
introduction. American Psychologist, 64, 67–74.
http://dx.doi.org/10.1037/a0013205
Dimberg, U., & Ohman, A. (1996). Behold the wrath:
Psychophysiological responses to facial stimuli.
Motivation and Emotion, 20, 149–182. http://dx.doi.
org/10.1007/BF02253869
Dittmann, A. T. (1972). Developmental factors
in conversational behavior. Journal of
Communication, 22, 404–423. http://dx.doi.
org/10.1111/j.1460-2466.1972.tb00165.x
Dondi, M., Messinger, D., Colle, M., Tabasso, A., Simion, F.,
Barba, B. D., & Fogel, A. (2007). A new perspective on
neonatal smiling: Differences between the judgments
of expert coders and naive observers. Infancy, 12,
235–255. http://dx.doi.org/10.1111/j.1532-7078.2007.
tb00242.x
Duchenne de Boulogne, G. B. (1990). The mechanism of
human facial expression. http://dx.doi.org/10.1017/
CBO9780511752841 (Original work published 1862)
Dumas, F. (1932). La mimique des aveugles [Facial
expression of the blind]. Bulletin de l’Académie de
Médecine, 107, 607–610.
Edelmann, R. J., Asendorpf, J., Contarello, A., Georgas, J.,
Villanueva, C., & Zammuner, V. (1987). Self-
reported verbal and non-verbal strategies for coping
with embarrassment in five European cultures.
Social Science Information, 26, 869–883. http://dx.doi.
org/10.1177/053901887026004008
Eibl-Eibesfeldt, I. (1972). Similarities and differences
between cultures in expressive movements.
In R. A. Hinde (Ed.), Nonverbal communication
(pp. 297–312). Cambridge, England: Cambridge
University Press.
Eibl-Eibesfeldt, I. (1973). The expressive behavior of the
dead-and-blind born. In M. von Cranach & I. Vine
(Eds.), Social communication and movement (pp. 163–
194). London, England: Academic Press.
Eisenberg, N., Fabes, R. A., Miller, P. A., Fultz, J., Shell, R.,
Mathy, R. M., & Reno, R. R. (1989). Relation
of sympathy and personal distress to prosocial
280
behavior: A multimethod study. Journal of Personality
and Social Psychology, 57, 55–66. http://dx.doi.
org/10.1037/0022-3514.57.1.55
Ekman, P. (1972). Universal and cultural differences
in facial expression of emotion. In J. R. Cole (Ed.),
Nebraska Symposium on Motivation, 1971 (pp.
207–283). Lincoln: Nebraska University Press.
Ekman, P. (1985). Telling lies: Clues to deceit in the
marketplace, politics, and marriage. New York, NY:
Norton.
Ekman, P. (1992a). Are there basic emotions? Psychological
Review, 99, 550–553. http://dx.doi.org/10.1037/0033-
295X.99.3.550
Ekman, P. (1992b). Facial expressions of emotion: New
findings, new questions. Psychological Science, 3,
34–38. http://dx.doi.org/10.1111/j.1467-9280.1992.
tb00253.x
Ekman, P. (1999). Basic emotions. In T. D. T. Power (Ed.),
The handbook of cognition and emotion (pp. 45–60).
Sussex, England: Wiley.
Ekman, P. (2003). Emotions revealed. New York, NY:
Times Books.
Ekman, P., Davidson, R. J., & Friesen, W. V. (1990).
The Duchenne smile: Emotional expression and
brain physiology: II. Journal of Personality and
Social Psychology, 58, 342–353. http://dx.doi.
org/10.1037/0022-3514.58.2.342
Ekman, P., & Friesen, W. V. (1969). The repertoire of
nonverbal behavior: Categories, origins, usage, and
coding. Semiotica, 1, 49–98.
Ekman, P., & Friesen, W. V. (1971). Constants
across cultures in the face and emotion. Journal
of Personality and Social Psychology, 17, 124–129.
http://dx.doi.org/10.1037/h0030377
Ekman, P., & Friesen, W. V. (1974). Nonverbal behavior
and psychopathology. In R. J. Friedman & M. Katz
(Eds.), The psychology of depression: Contemporary
theory and research (pp. 3–31). Washington, DC:
Winston and Sons.
Ekman, P., & Friesen, W. V. (1975). Unmasking the face:
A guide to recognizing the emotions from facial cues.
Englewood Cliffs, NJ: Prentice Hall.
Ekman, P., & Friesen, W. V. (1978). Facial Action
Coding System: Investigator’s guide. Palo Alto, CA:
Consulting Psychologists Press.
Ekman, P., & Friesen, W. V. (1982). Felt, false, and
miserable smiles. Journal of Nonverbal Behavior, 6,
238–252. http://dx.doi.org/10.1007/BF00987191
Ekman, P., & Friesen, W. V. (1986). A new pan-cultural
facial expression of emotion. Motivation and Emotion,
10, 159–168. http://dx.doi.org/10.1007/BF00992253
Ekman, P., Friesen, W. V., & Ancoli, S. (1980). Facial
signs of emotional experience. Journal of Personality

and Social Psychology, 39, 1125–1134. http://dx.doi.
org/10.1037/h0077722
Ekman, P., Friesen, W. V., & Ellsworth, P. (1972).
Emotion in the human face: Guidelines for research
and an integration of findings. New York, NY:
Pergamon Press.
Ekman, P., Friesen, W. V., & O’Sullivan, M. (1988).
Smiles when lying. Journal of Personality and
Social Psychology, 54, 414–420. http://dx.doi.
org/10.1037/0022-3514.54.3.414
Ekman, P., Friesen, W. V., O’Sullivan, M., Chan, A.,
Diacoyanni-Tarlatzis, I., Heider, K., . . . Tzavaras, A.
(1987). Universals and cultural differences in the
judgments of facial expressions of emotion. Journal
of Personality and Social Psychology, 53, 712–717.
http://dx.doi.org/10.1037/0022-3514.53.4.712
Ekman, P., Hager, J. C., & Friesen, W. V. (1981). The
symmetry of emotional and deliberate facial actions.
Psychophysiology, 18, 101–106. http://dx.doi.
org/10.1111/j.1469-8986.1981.tb02919.x
Ekman, P., & Heider, K. G. (1988). The universality of a
contempt expression: A replication. Motivation and
Emotion, 12, 303–308. http://dx.doi.org/10.1007/
BF00993116
Ekman, P., Levenson, R. W., & Friesen, W. V. (1983).
Autonomic nervous system activity distinguishes
among emotions. Science, 221, 1208–1210. http://
dx.doi.org/10.1126/science.6612338
Ekman, P., Liebert, R. M., Friesen, W. V., Harrison, R.,
Zlatchin, C., Malmstrom, E. J., & Baron, R. A.
(1972). Facial expressions of emotion while watching
televised violence as predictors of subsequent
aggression. In G. A. Comstock, E. A. Rubinstein, &
J. P. Murray (Eds.), Television and social behavior:
A technical report to the Surgeon General’s Scientific
Advisory Committee on Television and Social
Behavior (Vol. 1, pp. 22–58). Washington, DC: U.S.
Government Printing Office.
Ekman, P., Matsumoto, D., & Friesen, W. (1997). Facial
expression in affective disorders. In P. Ekman &
E. L. Rosenberg (Eds.), What the face reveals: Basic
and applied studies of spontaneous expression using the
Facial Action Coding System (FACS) (pp. 331–341).
New York, NY: Oxford University Press.
Ekman, P., & Oster, H. (1979). Facial expressions
of emotion. Annual Review of Psychology, 30,
527–554. http://dx.doi.org/10.1146/annurev.
ps.30.020179.002523
Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969).
Pan-cultural elements in facial displays of emotion.
Science, 164, 86–88. http://dx.doi.org/10.1126/
science.164.3875.86
Elfenbein, H. A., & Ambady, N. (2002). On the
universality and cultural specificity of emotion
recognition: A meta-analysis. Psychological Bulletin,
Facial Expressions
128, 203–235. http://dx.doi.org/10.1037/0033-
2909.128.2.203
Ellgring, H. (1986). Nonverbal expression of
psychological states in psychiatric patients. European
Archives of Psychiatry and Neurological Sciences, 236,
31–34. http://dx.doi.org/10.1007/BF00641055
Esteves, F., Dimberg, U., & Ohman, A. (1994).
Automatically elicited fear: Conditioned skin
conductance responses to masked facial expressions.
Cognition and Emotion, 8, 393–413. http://dx.doi.
org/10.1080/02699939408408949
Fernández-Dols, J.-M., Carrera, P., & Crivelli, C.
(2011). Facial behavior while experiencing sexual
excitement. Journal of Nonverbal Behavior, 35, 63–71.
http://dx.doi.org/10.1007/s10919-010-0097-7
Foisy, M.-L., Philippot, P., Verbanck, P., Pelc, I., van
der Straten, G., & Kornreich, C. (2005). Emotional
facial expression decoding impairment in persons
dependent on multiple substances: Impact of a
history of alcohol dependence. Journal of Studies on
Alcohol, 66, 673–681.
Frank, M. G., & Ekman, P. (1993). Not all smiles are
created equal: The differences between enjoyment
and nonenjoyment smiles. Humor: International
Journal of Humor Research, 6, 9–26. http://dx.doi.
org/10.1515/humr.1993.6.1.9
Frank, M. G., Ekman, P., & Friesen, W. V. (1993).
Behavioral markers and recognizability of the
smile of enjoyment. Journal of Personality and
Social Psychology, 64, 83–93. http://dx.doi.
org/10.1037/0022-3514.64.1.83
Freedman, D. G. (1964). Smiling in blind infants and
the issue of innate vs. acquired. Journal of Child
Psychology and Psychiatry, 5, 171–184. http://dx.doi.
org/10.1111/j.1469-7610.1964.tb02139.x
Friesen, W. V. (1972). Cultural differences in facial
expressions in a social situation: An experimental test
of the concept of display rules (Unpublished doctoral
dissertation). University of California, San Francisco.
Fulcher, J. S. (1942). “Voluntary” facial expression in
blind and seeing children. Archives of Psychology,
272, 5–49.
Galati, D., Miceli, R., & Sini, B. (2001). Judging
and coding facial expression of emotions in
congenitally blind children. International Journal of
Behavioral Development, 25, 268–278. http://dx.doi.
org/10.1080/01650250042000393
Galati, D., Sini, B., Schmidt, S., & Tinti, C. (2003).
Spontaneous facial expressions in congenitally blind
and sighted children aged 8–11. Journal of Visual
Impairment and Blindness, 97, 418–428.
Geen, T. R. (1992). Facial expressions in socially
isolated nonhuman primates: Open and closed
programs for expressive behavior. Journal of
281
Hwang and Matsumoto
Research in Personality, 26, 273–280. http://dx.doi.
org/10.1016/0092-6566(92)90044-5
Goodenough, F. L. (1932). Expression of emotions in
a blind-deaf child. Journal of Abnormal and Social
Psychology, 27, 328–333. http://dx.doi.org/10.1037/
h0076099
Gosselin, P., Kirouac, G., & Doré, F. Y. (1995).
Components and recognition of facial expression in
the communication of emotion by actors. Journal of
Personality and Social Psychology, 68, 83–96. http://
dx.doi.org/10.1037/0022-3514.68.1.83
Gosselin, P., Perron, M., & Beaupré, M. (2010). The
voluntary control of facial action units in adults.
Emotion, 10, 266–271. http://dx.doi.org/10.1037/
a0017748
Gottman, J. M., & Levenson, R. W. (1992). Marital
processes predictive of later dissolution: Behavior,
physiology, and health. Journal of Personality and
Social Psychology, 63, 221–233. http://dx.doi.
org/10.1037/0022-3514.63.2.221
Gottman, J. M., Levenson, R. W., & Woodin, E. (2001).
Facial expressions during marital conflict. Journal
of Family Communication, 1, 37–57. http://dx.doi.
org/10.1207/S15327698JFC0101_06
Gray, H., & Goss, C. M. (1966). Anatomy of the human
body (28th ed.). Philadelphia, PA: Lea & Febiger.
Gudykunst, W. B., & Nishida, T. (1984). Individual
and cultural influences on uncertainty reduction.
Communication Monographs, 51, 23–36. http://dx.doi.
org/10.1080/03637758409390181
Gudykunst, W. B., & Ting-Toomey, S. (1988).
Culture and affective communication. American
Behavioral Scientist, 31, 384–400. http://dx.doi.
org/10.1177/000276488031003009
Hager, J. C., & Ekman, P. (1985). The
asymmetry of facial actions is inconsistent
with models of hemispheric specialization.
Psychophysiology, 22, 307–318. http://dx.doi.
org/10.1111/j.1469-8986.1985.tb01605.x
Haggard, E. A., & Isaacs, K. S. (1966). Micro-momentary
facial expressions as indicators of ego mechanisms in
psychotherapy. In L. A. Gottschalk & A. H. Auerbach
(Eds.), Methods of research in psychotherapy
(pp. 154–165). http://dx.doi.org/10.1007/978-1-
4684-6045-2_14
Harker, L., & Keltner, D. (2001). Expressions of positive
emotion in women’s college yearbook pictures
and their relationship to personality and life
outcomes across adulthood. Journal of Personality
and Social Psychology, 80, 112–124. http://dx.doi.
org/10.1037/0022-3514.80.1.112
Harris, C. R., & Alvarado, N. (2005). Facial expressions,
smile types, and self-report during humour, tickle,
and pain. Cognition and Emotion, 19, 655–669. http://
dx.doi.org/10.1080/02699930441000472
282
Hauser, M. D. (1993). Right hemisphere dominance for
the production of facial expression in monkeys.
Science, 261, 475–477. http://dx.doi.org/10.1126/
science.8332914
Heerey, E. A., Keltner, D., & Capps, L. M. (2003).
Making sense of self-conscious emotion: Linking
theory of mind and emotion in children with autism.
Emotion, 3, 394–400. http://dx.doi.org/10.1037/1528-
3542.3.4.394
Heller, M., & Haynal, V. (1994). Depression and suicide
faces. Cahiers Psychiatriques Genevois, 16, 107–117.
Hertenstein, M. J., & Campos, J. J. (2004). The retention
effects of an adult’s emotional displays on infant
behavior. Child Development, 75, 595–613. http://
dx.doi.org/10.1111/j.1467-8624.2004.00695.x
Hess, U., Banse, R., & Kappas, A. (1995). The intensity
of facial expression is determined by underlying
affective states and social situations. Journal of
Personality and Social Psychology, 69, 280–288.
http://dx.doi.org/10.1037/0022-3514.69.2.280
Hess, U., Kappas, A., McHugo, G. J., Lanzetta, J. T., &
Kleck, R. E. (1992). The facilitative effect of facial
expression on the self-generation of emotion.
International Journal of Psychophysiology, 12,
251–265. http://dx.doi.org/10.1016/0167-
8760(92)90064-I
Hess, U., & Kleck, R. E. (1990). Differentiating emotion
elicited and deliberate emotional facial expressions.
European Journal of Social Psychology, 20, 369–385.
http://dx.doi.org/10.1002/ejsp.2420200502
Hobson, R. P. (1986). The autistic child’s appraisal of
expressions of emotion: A further study. Journal of
Child Psychology and Psychiatry, 27, 671–680. http://
dx.doi.org/10.1111/j.1469-7610.1986.tb00191.x
Hofstede, G. (2001). Culture’s consequences: Comparing
values, behaviors, institutions, and organizations across
nations (2nd ed.). Thousand Oaks, CA: Sage.
Howieson, D., & Lezak, M. (2010). The
neuropsychological evaluation. In S. C. Yudofsky &
R. E. Hales (Eds.), Essentials of neuropsychiatry
and behavioral neurosciences (2nd ed., pp. 29–54).
Arlington, VA: American Psychiatric Publishing.
Hurley, C. M., & Frank, M. G. (2011). Executing facial
control during deception situations. Journal of
Nonverbal Behavior, 35, 119–131. http://dx.doi.
org/10.1007/s10919-010-0102-1
Hwang, H. C., & Matsumoto, D. (2014a). Cultural
differences in victory signals of triumph. Cross-
Cultural Research, 48, 177–191. http://dx.doi.
org/10.1177/1069397113510875
Hwang, H. C., & Matsumoto, D. (2014b). Dominance
threat display for victory and achievement in
competition context. Motivation and Emotion,
38, 206–214. http://dx.doi.org/10.1007/s11031-013-
9390-1

Hyson, M. C., & Izard, C. E. (1985). Continuities
and changes in emotion expressions during brief
separation at 13 and 18 months. Developmental
Psychology, 21, 1165–1170. http://dx.doi.
org/10.1037/0012-1649.21.6.1165
Izard, C. E. (1971). The face of emotion. East Norwalk,
CT: Appleton-Century-Crofts.
Izard, C. E. (1991). The psychology of emotions. http://
dx.doi.org/10.1007/978-1-4899-0615-1
Izard, C. E., & Malatesta, C. Z. (1987). Perspectives on
emotional development: I. Differential emotions theory
of early emotional development. In J. S. Osofsky &
J. Doniger (Eds.), Handbook of infant development
(2nd ed., pp. 494–554). New York, NY: Wiley.
Keltner, D. (1995). The signs of appeasement: Evidence
for the distinct displays of embarrassment,
amusement, and shame. Journal of Personality
and Social Psychology, 68, 441–454. http://dx.doi.
org/10.1037/0022-3514.68.3.441
Keltner, D. (2003). Expression and the course
of life: Studies of emotion, personality, and
psychopathology from a social-functional
perspective. Annals of the New York Academy of
Sciences, 1000, 222–243. http://dx.doi.org/10.1196/
annals.1280.011
Keltner, D., & Bonanno, G. A. (1997). A study of laughter
and dissociation: Distinct correlates of laughter and
smiling during bereavement. Journal of Personality
and Social Psychology, 73, 687–702. http://dx.doi.
org/10.1037/0022-3514.73.4.687
Keltner, D., & Buswell, B. N. (1997). Embarrassment:
Its distinct form and appeasement functions.
Psychological Bulletin, 122, 250–270. http://dx.doi.
org/10.1037/0033-2909.122.3.250
Keltner, D., Ellsworth, P. C., & Edwards, K. (1993).
Beyond simple pessimism: Effects of sadness and
anger on social perception. Journal of Personality
and Social Psychology, 64, 740–752. http://dx.doi.
org/10.1037/0022-3514.64.5.740
Keltner, D., Moffitt, T. E., & Stouthamer-Loeber, M.
(1995). Facial expressions of emotion and
psychopathology in adolescent boys. Journal of
Abnormal Psychology, 104, 644–652. http://dx.doi.
org/10.1037/0021-843X.104.4.644
Kendler, K. S., Halberstadt, L. J., Butera, F., Myers, J.,
Bouchard, T., & Ekman, P. (2008). The similiarity
of facial expressions in response to emotion-
inducing films in reared-apart twins. Psychological
Medicine, 38, 1475–1483. http://dx.doi.org/10.1017/
S0033291707001535
Kendon, A., & Ferber, A. (1973). A description of some
human greetings. In R. P. Michael & J. H. Crook
(Eds.), Comparative behaviour and ecology of primates
(pp. 591–668). London, England: Academic Press.
Facial Expressions
Klinnert, M. D., Campos, J. J., & Sorce, J. F. (1983).
Emotions as behavior regulators: Social referencing
in infancy. In R. Plutchik & H. Kellerman (Eds.),
Emotion: Theory, research, and experience (pp.
57–86). http://dx.doi.org/10.1016/B978-0-12-
558702-0.50009-1
Krumhuber, E. G., & Scherer, K. R. (2011). Affect bursts:
Dynamic patterns of facial expression. Emotion, 11,
825–841. http://dx.doi.org/10.1037/a0023856
Laird, J. D. (1974). Self-attribution of emotion: The
effects of expressive behavior on the quality of
emotional experience. Journal of Personality and
Social Psychology, 29, 475–486. http://dx.doi.
org/10.1037/h0036125
Laird, J. D. (1984). The real role of facial response in
the experience of emotion: A reply to Tourangeau
and Ellsworth, and others. Journal of Personality
and Social Psychology, 47, 909–917. http://dx.doi.
org/10.1037/0022-3514.47.4.909
Lanzetta, J. T., Biernat, J. J., & Kleck, R. E. (1982).
Self-focused attention, facial behavior, autonomic
arousal and the experience of emotion. Motivation
and Emotion, 6, 49–63. http://dx.doi.org/10.1007/
BF00992137
Larsen, R. J., Kasimatis, M., & Frey, K. (1992).
Facilitating the furrowed brow: An unobtrusive
test of the facial feedback hypothesis applied to
unpleasant affect. Cognition and Emotion, 6, 321–338.
http://dx.doi.org/10.1080/02699939208409689
Lazarus, R. (1991). Emotion and adaptation. New York,
NY: Oxford University Press.
Lee, D. H., Mirza, R., Flanagan, J. G., & Anderson, A. K.
(2014). Optical origins of opposing facial expression
actions. Psychological Science, 25, 745–752. http://
dx.doi.org/10.1177/0956797613514451
LeResche, L. (1982). Facial expression in pain: A study of
candid photographs. Journal of Nonverbal Behavior, 7,
46–56. http://dx.doi.org/10.1007/BF01001777
Lerner, J. S., Gonzalez, R. M., Dahl, R. E., Hariri, A. R., &
Taylor, S. E. (2005). Facial expressions of emotion
reveal neuroendocrine and cardiovascular stress
responses. Biological Psychiatry, 58, 743–750. http://
dx.doi.org/10.1016/j.biopsych.2005.08.011
Levenson, R. W. (1994). Human emotions: A functional
view. In P. Ekman & R. J. Davidson (Eds.), The
nature of emotion: Fundamental questions (pp. 17–42).
New York, NY: Oxford University Press.
Levenson, R. W. (2003). Blood, sweat, and fears. Annals
of the New York Academy of Sciences, 1000, 348–366.
http://dx.doi.org/10.1196/annals.1280.016
Levenson, R. W., & Ekman, P. (2002). Difficulty does
not account for emotion-specific heart rate changes
in the directed facial action task. Psychophysiology,
39, 397–405. http://dx.doi.org/10.1017/
S0048577201393150
283
Hwang and Matsumoto
Levenson, R. W., Ekman, P., & Friesen, W. V. (1990).
Voluntary facial action generates emotion-specific
autonomic nervous system activity. Psychophysiology,
27, 363–384. http://dx.doi.org/10.1111/
j.1469-8986.1990.tb02330.x
Levenson, R. W., Ekman, P., Heider, K., & Friesen, W. V.
(1992). Emotion and autonomic nervous system
activity in the Minangkabau of West Sumatra. Journal
of Personality and Social Psychology, 62, 972–988.
http://dx.doi.org/10.1037/0022-3514.62.6.972
Liebal, K., Pika, S., & Tomasello, M. (2004). Social
communication in siamangs (Symphalangus
syndactylus): Use of gestures and facial expressions.
Primates, 45, 41–57. http://dx.doi.org/10.1007/
s10329-003-0063-7
Marsh, A. A., Ambady, N., & Kleck, R. E. (2005). The
effects of fear and anger facial expressions on
approach- and avoidance-related behaviors. Emotion,
5, 119–124. http://dx.doi.org/10.1037/1528-
3542.5.1.119
Matsumoto, D. (1987). The role of facial response in
the experience of emotion: More methodological
problems and a meta-analysis. Journal of Personality
and Social Psychology, 52, 769–774. http://dx.doi.
org/10.1037/0022-3514.52.4.769
Matsumoto, D. (1992). American–Japanese cultural
differences in the recognition of universal facial
expressions. Journal of Cross-Cultural Psychology, 23,
72–84. http://dx.doi.org/10.1177/0022022192231005
Matsumoto, D. (2001). Culture and emotion. In D.
Matsumoto (Ed.), The handbook of culture and
psychology (pp. 171–194). New York, NY: Oxford
University Press.
Matsumoto, D. (2005). Scalar ratings of contempt
expressions. Journal of Nonverbal Behavior, 29, 91–104.
http://dx.doi.org/10.1007/s10919-005-2742-0
Matsumoto, D., Consolacion, T., Yamada, H., Suzuki, R.,
Franklin, B., Paul, S., . . . Uchida, H. (2002).
American–Japanese cultural differences in judgments
of emotional expressions of different intensities.
Cognition and Emotion, 16, 721–747. http://dx.doi.
org/10.1080/02699930143000608
Matsumoto, D., & Ekman, P. (1988). Japanese and
Caucasian Facial Expressions of Emotion and Neutral
Faces (JACFEE and JACNeuF). Retrieved from http://
www.davidmatsumoto.com/research.php
Matsumoto, D., & Ekman, P. (1989). American–Japanese
cultural differences in intensity ratings of facial
expressions of emotion. Motivation and Emotion, 13,
143–157. http://dx.doi.org/10.1007/BF00992959
Matsumoto, D., & Ekman, P. (2004). The relationship
among expressions, labels, and descriptions
of contempt. Journal of Personality and Social
Psychology, 87, 529–540. http://dx.doi.org/10.1037/
0022-3514.87.4.529
284
Matsumoto, D., Haan, N., Yabrove, G., Theodorou, P., &
Carney, C. C. (1986). Preschoolers’ moral actions
and emotions in prisoner’s dilemma. Developmental
Psychology, 22, 663–670. http://dx.doi.
org/10.1037/0012-1649.22.5.663
Matsumoto, D., & Hwang, H. S. (2012). Evidence
for a nonverbal expression of triumph. Evolution
and Human Behavior, 33, 520–529. http://dx.doi.
org/10.1016/j.evolhumbehav.2012.01.005
Matsumoto, D., & Hwang, H. C. (2013a). Culture and
nonverbal communication. In J. Hall &
M. Knapp (Eds.), Handbook of nonverbal
communication (pp. 697–728). http://dx.doi.
org/10.1515/9783110238150.697
Matsumoto, D., & Hwang, H. C. (2013b). Differences
between individuals with and without experience
with assault in identifying facial signs of imminent
aggression. In K. Glass, R. Colbaugh, A. Sanfillippo,
C. Corley, J. Li, L. Khan, A. Wynne, L. Coote,
W. Mao, D. Zeng, & A. Yaghooby (Eds.), 2013 IEEE
International Conference on Intelligence and Security
Informatics (pp. 176–178). Seattle, WA: IEEE.
Matsumoto, D., Kasri, F., & Kooken, K. (1999).
American–Japanese cultural differences in judgments
of expression intensity and subjective experience.
Cognition and Emotion, 13, 201–218. http://dx.doi.
org/10.1080/026999399379339
Matsumoto, D., Keltner, D., Shiota, M. N., O’Sullivan, M., &
Frank, M. G. (2008). What’s in a face? Facial
expressions as signals of discrete emotions. In
M. Lewis, J. M. Haviland, & L. F. Barrett (Eds.),
Handbook of emotions (3rd ed., pp. 211–234). New
York, NY: Guilford Press.
Matsumoto, D., & Kupperbusch, C. (2001). Idiocentric
and allocentric differences in emotional expression,
experience, and the coherence between expression
and experience. Asian Journal of Social Psychology,
4, 113–131. http://dx.doi.org/10.1111/j.1467-
839X.2001.00080.x
Matsumoto, D., & Lee, M. (1993). Consciousness,
volition, and the neuropsychology of facial
expressions of emotion. Consciousness and
Cognition, 2, 237–254. http://dx.doi.org/10.1006/
ccog.1993.1022
Matsumoto, D., Nezlek, J. B., & Koopmann, B. (2007).
Evidence for universality in phenomenological
emotion response system coherence. Emotion, 7,
57–67. http://dx.doi.org/10.1037/1528-3542.7.1.57
Matsumoto, D., Olide, A., Schug, J., Willingham, B., &
Callan, M. (2009). Cross-cultural judgments of
spontaneous facial expressions of emotion. Journal
of Nonverbal Behavior, 33, 213–238. http://dx.doi.
org/10.1007/s10919-009-0071-4
Matsumoto, D., Takeuchi, S., Andayani, S., Kouznetsova, N., &
Krupp, D. (1998). The contribution of individualism

vs. collectivism to cross-national differences in
display rules. Asian Journal of Social Psychology,
1, 147–165. http://dx.doi.org/10.1111/1467-
839X.00010
Matsumoto, D., & Willingham, B. (2006). The thrill
of victory and the agony of defeat: Spontaneous
expressions of medal winners of the 2004 Athens
Olympic Games. Journal of Personality and
Social Psychology, 91, 568–581. http://dx.doi.
org/10.1037/0022-3514.91.3.568
Matsumoto, D., & Willingham, B. (2009). Spontaneous
facial expressions of emotion of congenitally
and noncongenitally blind individuals. Journal of
Personality and Social Psychology, 96, 1–10.
http://dx.doi.org/10.1037/a0014037
Matsumoto, D., Willingham, B., & Olide, A. (2009).
Sequential dynamics of culturally moderated facial
expressions of emotion. Psychological Science,
20, 1269–1275. http://dx.doi.org/10.1111/j.1467-
9280.2009.02438.x
Mauss, I. B., Levenson, R. W., McCarter, L., Wilhelm, F. H., &
Gross, J. J. (2005). The tie that binds? Coherence
among emotion experience, behavior, and
physiology. Emotion, 5, 175–190. http://dx.doi.
org/10.1037/1528-3542.5.2.175
Mauss, I. B., Wilhelm, F. L., & Gross, J. J. (2004). Is there
less to social anxiety than meets the eye? Emotion
experience, expression, and bodily responding.
Cognition and Emotion, 18, 631–642. http://dx.doi.
org/10.1080/02699930341000112
McCanne, T. R., & Anderson, J. A. (1987). Emotional
responding following experimental manipulation
of facial electromyographic activity. Journal of
Personality and Social Psychology, 52, 759–768.
http://dx.doi.org/10.1037/0022-3514.52.4.759
McGhee, P. E. (1977). Children’s humour: A review of
current research trends. In A. J. Chapman & H. C. Foot
(Eds.), It’s a funny thing, humour (pp. 199–209). http://
dx.doi.org/10.1016/B978-0-08-021376-7.50041-3
McIntosh, D. N. (1996). Facial feedback hypotheses:
Evidence, implications, and directions. Motivation
and Emotion, 20, 121–147. http://dx.doi.org/10.1007/
BF02253868
Mehu, M., Mortillaro, M., Bänziger, T., & Scherer, K. R.
(2012). Reliable facial muscle activation enhances
recognizability and credibility of emotional
expression. Emotion, 12, 701–715. http://dx.doi.
org/10.1037/a0026717
Noesjirwan, J. (1978). A rule-based analysis of
cultural differences in social behaviour:
Indonesia and Australia. International Journal
of Psychology, 13, 305–316. http://dx.doi.
org/10.1080/00207597808246634
Oster, H. (2005). The repertoire of infant facial
expressions: An ontogenetic perspective.
Facial Expressions
In J. Nadel & D. Muir (Eds.), Emotional development
(pp. 261–292). New York, NY: Oxford University
Press.
Ozonoff, S., Pennington, B. F., & Rogers, S. J. (1990).
Are there emotion perception deficits in young
autistic children? Journal of Child Psychology
and Psychiatry, 31, 343–361. http://dx.doi.
org/10.1111/j.1469-7610.1990.tb01574.x
Parr, L. A., Waller, B. M., Burrows, A. M., Gothard, K. M., &
Vick, S. J. (2010). MaqFACS: A muscle-based facial
movement coding system for the rhesus macaque.
American Journal of Physical Anthropology, 143,
625–630. http://dx.doi.org/10.1002/ajpa.21401
Parr, L. A., Waller, B. M., & Heintz, M. (2008). Facial
expression categorization by chimpanzees using
standardized stimuli. Emotion, 8, 216–231. http://
dx.doi.org/10.1037/1528-3542.8.2.216
Peleg, G., Katzir, G., Peleg, O., Kamara, M., Brodsky, L.,
Hel-Or, H., . . . Nevo, E. (2006). Hereditary
family signature of facial expression. Proceedings
of the National Academy of Sciences, USA, 103,
15921–15926. http://dx.doi.org/10.1073/
pnas.0607551103
Perkins, A. M., Inchley-Mort, S. L., Pickering, A. D.,
Corr, P. J., & Burgess, A. P. (2012). A facial
expression for anxiety. Journal of Personality and
Social Psychology, 102, 910–924. http://dx.doi.
org/10.1037/a0026825
Persad, S. M., & Polivy, J. (1993). Differences between
depressed and nondepressed individuals in the
recognition of and response to facial emotional cues.
Journal of Abnormal Psychology, 102, 358–368.
http://dx.doi.org/10.1037/0021-843X.102.3.358
Pollak, S. D., Cicchetti, D., Hornung, K., & Reed, A.
(2000). Recognizing emotion in faces:
Developmental effects of child abuse and neglect.
Developmental Psychology, 36, 679–688. http://dx.doi.
org/10.1037/0012-1649.36.5.679
Pollak, S. D., & Sinha, P. (2002). Effects of early experience
on children’s recognition of facial displays of emotion.
Developmental Psychology, 38, 784–791. http://dx.doi.
org/10.1037/0012-1649.38.5.784
Porter, S., ten Brinke, L., & Wallace, B. (2012). Secrets
and lies: Involuntary leakage in deceptive facial
expressions as a function of emotional intensity.
Journal of Nonverbal Behavior, 36, 23–37. http://
dx.doi.org/10.1007/s10919-011-0120-7
Prkachin, K. M. (1992). The consistency of facial
expressions of pain: A comparison across modalities.
Pain, 51, 297–306. http://dx.doi.org/10.1016/0304-
3959(92)90213-U
Prkachin, K. M., & Craig, K. D. (1995). Expressing pain:
The communication and interpretation of facial pain
signals. Journal of Nonverbal Behavior, 19, 191–205.
http://dx.doi.org/10.1007/BF02173080
285
Hwang and Matsumoto
Redican, W. K. (1982). An evolutionary perspective on
human facial displays. In P. Ekman (Ed.), Emotion in
the human face (Vol. 2, pp. 212–280). New York, NY:
Cambridge University Press.
Reed, L. I., DeScioli, P., & Pinker, S. A. (2014). The
commitment function of angry facial expressions.
Psychological Science, 25, 1511–1517. http://dx.doi.
org/10.1177/0956797614531027
Ricci-Bitti, P., Caterina, R., & Garotti, P. (1996).
Different behavioral markers in different smiles.
In N. H. Frijda (Ed.), Proceedings of the Eighth
Conference of the International Society for Research on
Emotions (pp. 297–301). Storrs, CT: International
Society for Research on Emotions.
Rinn, W. E. (1984). The neuropsychology of facial
expression: A review of the neurological and
psychological mechanisms for producing facial
expressions. Psychological Bulletin, 95, 52–77.
http://dx.doi.org/10.1037/0033-2909.95.1.52
Rinn, W. E. (1991). Neuropsychology of facial
expression. In R. Feldman & B. Rime (Eds.),
Fundamentals of nonverbal behavior (pp. 3–70).
New York, NY: Cambridge University Press.
Rosati, A. G., & Hare, B. (2013). Chimpanzees and bonobos
exhibit emotional responses to decision outcomes.
PLoS ONE, 8, e63058. http://dx.doi.org/10.1371/
journal.pone.0063058
Rosenberg, E. L., & Ekman, P. (1994). Coherence
between expressive and experiential systems in
emotion. Cognition and Emotion, 8, 201–229.
http://dx.doi.org/10.1080/02699939408408938
Rosenberg, E. L., Ekman, P., Jiang, W., Babyak, M.,
Coleman, R. E., Hanson, M., . . . Blumenthal, J. A.
(2001). Linkages between facial expressions of anger
and transient myocardial ischemia in men with
coronary artery disease. Emotion, 1, 107–115.
http://dx.doi.org/10.1037/1528-3542.1.2.107
Rozin, P., Lowery, L., Imada, S., & Haidt, J. (1999).
The CAD triad hypothesis: A mapping between
three moral emotions (contempt, anger, disgust)
and three moral codes (community, autonomy,
divinity). Journal of Personality and Social Psychology,
76, 574–586. http://dx.doi.org/10.1037/0022-
3514.76.4.574
Ruch, W. (1994). Extroversion, alcohol and enjoyment.
Personality and Individual Differences, 16, 89–102.
http://dx.doi.org/10.1016/0191-8869(94)90113-9
Ruch, W. (1995). Will the real relationship between
facial expression and affective experience
please stand up: The case of exhilaration.
Cognition and Emotion, 9, 33–58. http://dx.doi.
org/10.1080/02699939508408964
Russell, J. A., & Fehr, B. (1987). Relativity in the
perception of emotion in facial expressions. Journal
286
of Experimental Psychology: General, 116, 223–237.
http://dx.doi.org/10.1037/0096-3445.116.3.223
Rutledge, L. L., & Hupka, R. B. (1985). The facial
feedback hypothesis: Methodological concerns and
new supporting evidence. Motivation and Emotion, 9,
219–240. http://dx.doi.org/10.1007/BF00991829
Scherer, K. R., & Ellgring, H. (2007). Are facial
expressions of emotion produced by categorical
affect programs or dynamically driven by appraisal?
Emotion, 7, 113–130. http://dx.doi.org/10.1037/1528-
3542.7.1.113
Schmidt, K. L., Ambadar, Z., Cohn, J. F., & Reed, L. I.
(2006). Movement differences between deliberate
and spontaneous facial expressions: Zygomaticus
major action in smiling. Journal of Nonverbal
Behavior, 30, 37–52. http://dx.doi.org/10.1007/
s10919-005-0003-x
Schmidt, K. L., Bhattacharya, S., & Denlinger, R. (2009).
Comparison of deliberate and spontaneous facial
movement in smiles and eyebrow raises. Journal
of Nonverbal Behavior, 33, 35–45. http://dx.doi.
org/10.1007/s10919-008-0058-6
Schug, J., Matsumoto, D., Horita, Y., Yamagishi, T., &
Bonnet, K. (2010). Emotional expressivity as
a signal of cooperation. Evolution and Human
Behavior, 31, 87–94. http://dx.doi.org/10.1016/
j.evolhumbehav.2009.09.006
Sell, A., Cosmides, L., & Tooby, J. (2014). The human
anger face evolved to enhance cues of strength.
Evolution and Human Behavior, 35, 425–429. http://
dx.doi.org/10.1016/j.evolhumbehav.2014.05.008
Sherida, M. D. (1977). Observations on the development
of spontaneous tele-kinesic communication in
babies and young children. Child: Care, Health
and Development, 3, 189–199. http://dx.doi.
org/10.1111/j.1365-2214.1977.tb00040.x
Shiller, V. M., Izard, C. E., & Hembree, E. A. (1986).
Patterns of emotion expression during separation
in the strange-situation procedure. Developmental
Psychology, 22, 378–382. http://dx.doi.
org/10.1037/0012-1649.22.3.378
Smith, C. A. (1989). Dimensions of appraisal and
physiological response in emotion. Journal of
Personality and Social Psychology, 56, 339–353.
http://dx.doi.org/10.1037/0022-3514.56.3.339
Smith, C. A., & Scott, H. S. (1989). A componential
approach to the meaning of facial expression. In
J. A. Russell & J. M. Fernández-Dols (Eds.), The
psychology of facial expression (229–254). Cambridge,
England: Cambridge University Press.
Smith, M. C. (1995). Facial expression in mild dementia of
the Alzheimer type. Behavioural Neurology, 8, 149–156.
Snowdon, C. T. (2003). Expression of emotion in
nonhuman animals. In R. J. Davidson, K. Scherer, &

H. H. Goldsmith (Eds.), Handbook of affective sciences
(pp. 457–480). New York, NY: Oxford University Press.
Sorce, J. F., Emde, J. J., Campos, J. J., & Klinnert, M. D.
(1985). Maternal emotional signaling: Its effect
on the visual cliff behavior of 1-year-olds.
Developmental Psychology, 21, 195–200. http://dx.doi.
org/10.1037/0012-1649.21.1.195
Soussignan, R. (2002). Duchenne smile, emotional
experience, and autonomic reactivity: A test of the
facial feedback hypothesis. Emotion, 2, 52–74. http://
dx.doi.org/10.1037/1528-3542.2.1.52
Steimer-Krause, E., Krause, R., & Wagner, G. (1990).
Interaction regulations used by schizophrenic and
psychosomatic patients: Studies on facial behavior in
dyadic interactions. Psychiatry, 53, 209–228.
Surcinelli, P., Codispoti, M., Montebarocci, O.,
Rossi, N., & Baldaro, B. (2006). Facial emotion
recognition in trait anxiety. Journal of Anxiety
Disorders, 20, 110–117. http://dx.doi.org/10.1016/
j.janxdis.2004.11.010
Susskind, J. M., Lee, D. H., Cusi, A., Feiman, R.,
Grabski, W., & Anderson, A. K. (2008). Expressing
fear enhances sensory acquisition. Nature Neuroscience,
11, 843–850. http://dx.doi.org/10.1038/nn.2138
Szarota, P. (2010). The mystery of the European smile:
A comparison based on individual photographs
provided by Internet users. Journal of Nonverbal
Behavior, 34, 249–256. http://dx.doi.org/10.1007/
s10919-010-0093-y
Thompson, J. (1941). Development of facial expression
of emotion in blind and seeing children. Archives de
Psychologie, 37, 1–47.
Tomkins, S. S. (1962). Affect, imagery, and consciousness:
Vol. 1. The positive affects. New York, NY: Springer.
Tomkins, S. S. (1963). Affect, imagery, and consciousness:
Vol. 2. The negative affects. New York, NY: Springer.
Tomkins, S. S., & McCarter, R. (1964). What and where
are the primary affects? Some evidence for a theory.
Perceptual and Motor Skills, 18, 119–158. http://
dx.doi.org/10.2466/pms.1964.18.1.119
Tracy, J. L., & Robins, R. W. (2004). Show your pride:
Evidence for a discrete emotion expression.
Psychological Science, 15, 194–197. http://dx.doi.
org/10.1111/j.0956-7976.2004.01503008.x
Tronick, E. Z. (1989). Emotions and emotional
communication in infants. American Psychologist, 44,
112–119. http://dx.doi.org/10.1037/0003-066X.44.2.112
Tsai, J. L., & Levenson, R. W. (1997). Cultural
influences of emotional responding: Chinese
American and European American dating couples
during interpersonal conflict. Journal of Cross-
Cultural Psychology, 28, 600–625. http://dx.doi.
org/10.1177/0022022197285006
Facial Expressions
Ueno, A., Ueno, Y., & Tomonaga, M. (2004). Facial
responses to four basic tastes in newborn rhesus
macaques (Macaca mulatta) and chimpanzees (Pan
troglodytes). Behavioural Brain Research, 154, 261–271.
http://dx.doi.org/10.1016/j.bbr.2004.02.014
Van Hoof, J. A. R. A. M. (1972). A comparative approach to
the phylogeny of laughing and smiling. In R. A. Hinde
(Ed.), Nonverbal communication (pp. 209–241).
Cambridge, England: Cambridge University Press.
Vaughan, K. B., & Lanzetta, J. T. (1981). The effect of
modification of expressive displays on vicarious
emotional arousal. Journal of Experimental
Social Psychology, 17, 16–30. http://dx.doi.
org/10.1016/0022-1031(81)90003-2
Vick, S. J., Waller, B. M., Parr, L. A., Smith Pasqualini, M. C.,
& Bard, K. A. (2007). A cross-species comparison
of facial morphology and movement in humans and
chimpanzees using the Facial Action Coding System
(FACS). Journal of Nonverbal Behavior, 31, 1–20.
http://dx.doi.org/10.1007/s10919-006-0017-z
Waller, B. M., Lembeck, M., Kuchenbuch, P.,
Burrows, A. M., & Liebal, K. (2012). GibbonFACS:
A muscle based coding system for the hylobatids.
International Journal of Primatology, 33, 809–821.
http://dx.doi.org/10.1007/s10764-012-9611-6
Waxer, P. H. (1985). Video ethology: Television as a data
base for cross-cultural studies in nonverbal displays.
Journal of Nonverbal Behavior, 9, 111–120. http://
dx.doi.org/10.1007/BF00987142
Winblad, S., Hellström, P., Lindberg, C., & Hansen, S.
(2006). Facial emotion recognition in myotonic
dystrophy Type 1 correlates with CTG repeat
expansion. Journal of Neurology, Neurosurgery, and
Psychiatry, 77, 219–223. http://dx.doi.org/10.1136/
jnnp.2005.070763
Winkielman, P., Berridge, K. C., & Wilbarger, J. L.
(2005). Unconscious affective reactions to masked
happy versus angry faces influence consumption
behavior and judgments of value. Personality and
Social Psychology Bulletin, 31, 121–135. http://dx.doi.
org/10.1177/0146167204271309
Winton, W. M. (1986). The role of facial response in
self-reports of emotion: A critique of laird. Journal
of Personality and Social Psychology, 50, 808–812.
http://dx.doi.org/10.1037/0022-3514.50.4.808
Woodzicka, J., & LaFrance, M. (2001). Real versus
imagined gender harassment. Journal of Social Issues,
57, 15–30. http://dx.doi.org/10.1111/0022-4537.00199
Zuckerman, M., Klorman, R., Larrance, D. T., &
Spiegel, N. H. (1981). Facial, autonomic, and
subjective components of emotion: The facial
feedback hypothesis versus externalizer–internalizer
distinction. Journal of Personality and Social
Psychology, 41, 929–944. http://dx.doi.
org/10.1037/0022-3514.41.5.929
287
 Chapter 11
The Voice: From Identity
to Interactions
Sophie Scott and Carolyn McGettigan
The human voice is a ubiquitous and a complex
phenomenon. Voices are used for conversational
speech, where the vocal expression conveys a
great deal of linguistic and nonlinguistic informa-
tion. There is also considerable modification of
the talker’s voice, and of the listener’s perceptual
processing, during interactions. In this chapter, we
address the different kinds of information expressed
in the voice and the various factors, including those
derived from cognitive neuroscience, that lead to
variations in how people speak. We note at the out-
set that indexical aspects of someone’s voice, which
refer to talker-specific properties of the voice, can-
not be completely distinguished from the properties
of his or her speech. For example, some factors,
such as the accent with which he or she speaks, will
affect both indexical and linguistic aspects of his or
her spoken voice.
ANATOMY, EVOLUTION, AND THE
MECHANICS OF VOICE
The Role of the Larynx
Most human vocalizations are made with voicing—a
term that is confusingly similar to the wider concept
of “the voice” but that refers to the sound made
in the larynx. Voicing is produced when the vocal
folds, within the larynx, are held together, and very
fine movements of the intercostal muscles are used
to produce a constant air pressure underneath the
vocal folds (called subglottal pressure). This pressure
forces air through the vocal folds, which are pushed
apart and then snap back into position repeatedly
http://dx.doi.org/10.1037/14669-011
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
due to Bernoulli forces. The rapid cycle of opening
and snapping shut of the vocal folds leads to a buzz-
ing sound. The pitch of a talker’s voice is associated
with the rate of vibration of the vocal folds (reflect-
ing the subglottal pressure) and the position of the
larynx—higher pitched voices are associated with
higher larynx positions, and vice versa.
Voicing adds two different kinds of informa-
tion to speech. First, it is relevant phonetically—in
many languages, the presence or absence of voicing
codes for linguistic information (e.g., “sue” is a dif-
ferent word from “zoo,” and in phonetic terms the
first phoneme is unvoiced in sue and is voiced in
zoo—try saying “sssss” or “zzzzz” with your fingers
on your windpipe at the front of your throat; you
should feel a vibration during the voiced “z” sounds).
Second, voicing gives the voice its pitch and rhythm
and, thus, contributes strongly to the acoustic prop-
erties of the voice that are discussed in this chapter.
Some of this variation is relevant linguistically; for
example, pitch is a cue to linguistic stress. Pitch is
also used at the sentence level to change the mode of
an utterance—for example, the difference between
a statement and a question is often associated with
the presence (or nonpresence) of a rising intonation
through a spoken sentence. Pitch also contributes
paralinguistically to accent, contributes to the emo-
tional and contextual modification of spoken lan-
guage, and serves as a cue to speaker identity.
Anatomy of Speech
According to the source/filter theory of speech
production (Fant, 1960), the sound produced at
289
Scott and McGettigan
FIGURE 11.1.  Structural magnetic resonance imaging
(MRI) image of a female British English talker, producing
the vowel “e” in “me.” Gross anatomical structures are
indicated on the image.
the larynx (the source) is then shaped by the move-
ments of the articulators—the tongue, soft palate,
lips, and jaw (the filter; see Figure 11.1). These are
moved into distinct configurations that alter the
spectral resonance characteristics of the vocal tract.
Different movements and positions of the articula-
tors, in combination with the larynx, give speech
sounds their distinct perceptual characteristics. The
most basic linguistic units in speech are phonemes,
which are each characterized by distinct movements
and configurations of the articulators. Speech con-
sists of sequences of these, arranged in syllables, and
the dynamics of articulating speech means that the
realization of individual phonemes is affected by
the position of the phoneme within a syllable—in
English, /p/ sounds at the start of a word are highly
aspirated, whereas those not in word onset posi-
tions are not (try saying “port” vs. “sport”). Pho-
nemes are also affected by the surrounding speech
sounds—the sound /s/ is produced with different
mouth shapes for the words “see,” “sue,” and “saw,”
as the upcoming vowel sound is anticipated (see
also Bailey & Summerfield, 1980). This is called
coarticulation, and it means that speech cannot be
290
thought of as comprising a sequence of indepen-
dent sounds. In the sequences /apa/ and /aba/, the
phonemes /p/ and /b/ are distinguished phonetically
in terms of the absence or presence of voicing, but
because of coarticulation, there are multiple separa-
ble acoustic differences between /apa/ and /aba/—for
example, the vowel duration before the /b/ is longer
than before the unvoiced /p/ (Lisker, 1977). The
perception of speech, therefore, does not rely on one
single acoustic feature. In terms of the talker and his
or her voice, these supralaryngeal cues contribute
to speaker identity and accent, and they are highly
modified by context and time as pronunciations
change.
Evolution of the Voice
The anatomical structures that people use for
speech and vocalizations have all been modified in
the course of evolution to enable people to speak.
Humans can only breathe for speech (by permit-
ting the constant subglottal pressure described
earlier) because they walk upright, and they do not
need to use their ribcages to support their weight
(MacLarnon & Hewitt, 1999). People’s larynxes
are not anchored to the root of the tongue at the
top of the trachea as they are in other primates and
human infants (shown in Figure 11.1). Instead, it
moves early in people’s lives to a lower position in
the trachea (P. Lieberman, 1968; Negus, 1949). This
lack of anchoring means that the human tongue can
make a much wider range of movements than those
of other primates, and in speech sounds the position
of the tongue—for example, at the back or the front
of the oral cavity—is a critical feature. This lowering
of the larynx also gives people a larger, longer pipe
with which to make sounds, and this longer vocal
tract allows a wider spectral range so that people can
produce lower, richer sounds. Note in Figure 11.1
that the distance between the larynx and the lips
is longer than if the larynx was fixed to the back of
the tongue. Furthermore, humans have relatively
flat faces, so their tongues are shorter and rounder
than those of other primates (D. E. Lieberman,
2011). Behind the alveolar ridge, people’s hard pal-
ates are domed, unlike the flat hard palates of other
mammals (Verhaegen & Munro, 2004). This gives
humans the ability to vary tongue height, which is

another dimension within the oral cavity for tongue
movements and the corresponding influences on the
spectral properties of the sounds produced.
Features of Individuals That Affect the
Voice
It is worth emphasizing at the outset that ­people’s
voices—the sounds they make when they
vocalize—are complex sounds that happen when
they actively do something. According to Nolan
(2012), “speech is both language and action”
(p. 263). In other words, people only have a voice if
they produce some vocal behavior—from speaking
or coughing, laughing or singing. If someone does
not speak, no one will know what his or her voice
sounds like. This means that there is no such thing
as a passive voice: When we discuss voices, we are
always discussing a signal that exists because some-
one has produced a sound. This introduces some
stark differences from studies of the face, which
­statically can transmit information about people
(e.g., when they are asleep, or from photos) in addi-
tion to the ways that the face moves expressively
(see Chapter 10, this handbook). When we work
with voices, we are forced to consider the dynamic
properties of the signal, as all sounds only have
structure as those sounds evolve over time. Strik-
ingly, one of the key properties of human voices is
how much they change, both between ­individuals
and within a single individual (Karpf, 2007;
Scott, 2008).
It is also worth emphasizing that the anatomy and
articulations that allow human speech also give rise
to the human voice. Other than saying our names
when we call someone on the phone, we do not pro-
duce extra sounds to convey vocal identity (in the
way that we can do to convey emotion; e.g., with
laughter or screaming), and of course saying our
names involves speaking. When we listen to a talker,
therefore, we are decoding his or her linguistic signal
at the same time as we are perceiving information
about his or her mood, identity, health, age, and so
forth all from the same signal, that is, from his or her
voice (Belin, Fecteau, & Bédard, 2004). In terms of
the sounds of spoken language, it can be difficult to
fully distinguish cues to identity, for example, from
those to linguistic meaning (Nolan, 2012).
The Voice
Anatomical Differences Between Talkers
Some vocal properties arise directly because of the
anatomy of the vocal tract. A critical factor is body
size, as the sounds an organism makes will vary in
direct proportion to its physical size. Studies have
shown that listeners are good at predicting body size
from simply hearing the voice (Patterson, Smith,
van Dinther, & Walters, 2008). Thus, children are
smaller than adults, and their smaller larynxes,
shorter vocal folds, and shorter vocal tracts mean
that they produce higher pitched vocalizations with
a smaller spectral range than adults. In adulthood,
the pitches of voices drop, and the spectral ranges
broaden as body size increases, and there is also
some sexual dimorphism. In men, the larynx under-
goes a secondary descent in adolescence: The larynx
also increases in size, and adult men typically have
longer, thicker vocal folds than adult women. This
means that male and female humans of equivalent
body size will still commonly differ in their voices,
with the male voice being typically lower in pitch
and having a broader spectral range than the female
voice (Fitch & Giedd, 1999).
Pharyngeal and Hard Palate Morphology
There is some variation in the shape of the vocal
tract: Some people have more or less “domed”
hard palates, for example (Hiki & Itoh, 1986), and
this can affect how they produce particular speech
sounds. Speakers with an artificially flattened palate
(through use of a prosthesis) change how they
produce /r/ sounds in American English, for
­example, from a retroflex (with the tip of the tongue
curled back against the roof of the mouth) to a
“bunched” production (with the tongue shorted
and the back raised; Tiede, Gracco, Shiller, Espy-
Wilson, & Boyce, 2005). Measurements using mag-
netic resonance imaging to make detailed dynamic
anatomical images of the articulators (lips, tongue,
soft palate, and jaw) moving have revealed further
complexity, such that the shape and angle of the
alveolar ridge, as well as the back pharyngeal wall,
affect articulations. The data showed that talkers
adapted their articulations to produce consistent
speech sounds despite these anatomical d ­ ifferences
(Lammert, Proctor, & Narayanan, 2013). The
authors noted that these features may make fruitful
291
Scott and McGettigan
areas of study for the relationship between anatomi-
cal morphology and the acoustics of speech, and this
has importance for voice research.
Other Sources of Difference Between
Speakers
Talkers can vary in the ways that they hold their
vocal folds when speaking: In “modal” voice, the
vocal folds are held in a way which leads to a saw-
toothed waveform—this is considered a “norma-
tive” manner of voicing (see Figure 11.2). However,
other manners of voicing are also relatively com-
mon. “Creaky” voice (also known as “glottal fry”)
involves the vocal folds being held under a differ-
ent tension than in modal voice, such that they
vibrate differently—the saw-toothed waveform
looks rounder (see Figure 11.2; Ladefoged, 1975).
This gives creaky voice a distinctly different har-
monic profile from that of modal voice, and it is
perceptually quite distinct. Creaky voice is relatively
common in the United States, and it frequently con-
tributes to what non-U.S. listeners think constitutes
an American accent. “Breathy” voice occurs when
the vocal folds are not completely closed during
their vibration: This allows air to escape the lungs
along with the acoustic effects of voicing, and it con-
fers a breathy quality to speech. As this is a relatively
inefficient way of speaking, this can be a tiring way
FIGURE 11.2.  Laryngograph traces for modal voice
(upper panel) and creaky voice (lower panel). Both are
produced by a native British English voice artist who is
speaking in a British English accent in the upper panel
and who is speaking in an American English accent in
the lower panel.
292
of speaking if someone has to talk continuously, for
example, when delivering a lecture.
Although their manner of voicing is not the only
contribution to the sound of someone’s voice, it is a
characteristic that can contribute to the perceptual
characteristics of his or her voice and the ways that
they can be recognized from his or her voice. Other
speaker specific voice contributions can come from
the more idiosyncratic ways the articulators are
used—some speakers have continuously nasalized
voices, for example, or speak with a lot of tension
in the jaw.
Voices and Faces
Does a person’s face reflect his or her voice? The
variation of palatal and pharyngeal anatomy
described earlier may be expressed in aspects of the
face and head morphology (though this has yet to
be determined empirically). A recent study showed
that participants are better than chance at matching
unfamiliar voices to faces in an experimental setting
(Mavica & Barenholtz, 2013), though the effects
are not large. The effects appeared to be driven by
some item effects—three of the 64 stimuli were
very frequently accurately matched. It would be
interesting to know more about the morphology of
the vocal tract in these individuals, as it may sug-
gest some shared factors between the anatomy of
the face and the articulators. This study needs to be
set against the findings of other studies that have
not seen such a relationship for static facial images
(Kamachi, Hill, Lander, & Vatikiotis-Bateson, 2003;
Lachs & Pisoni, 2004).
Acoustic Cues to Vocal Identity
A multidimensional scaling study of short vowels
showed that listeners use two main acoustic fac-
tors to create a “voice space”: mainly, the proper-
ties of the fundamental frequency and spectral
shape of the voice—cues that roughly correspond
to laryngeal- and supralaryngeal-based properties
(Baumann & Belin, 2010). Do these features cor-
respond to the ways that humans recognize talkers
from their voice alone?
Given an excellent recording of a relatively
lengthy portion of speech (1 s and longer) from
a small set of familiar talkers, the recognition

accuracy of vocal identity is at ceiling (Pardo &
Remez, 2006). However, if any of these factors are
altered—the recording is short, there is a lot of
noise, or the talker is unfamiliar—then the recogni-
tion of vocal identity becomes much harder, and
accuracy is lowered. Experimentally, the effects of
familiarity with the spoken language and with the
talkers in the to-be-recognized set are critical
(Hollien, Majewski, & Doherty, 1982). Many of the
sources of vocal change that were discussed in the
start of this chapter contribute to this variability,
along with the fact that talkers frequently speak dif-
ferent words in novel sequences, and speech is a
motor act that is produced in a flexible and plastic
fashion (Nolan, 2012). Software that aims to rec-
ognize talkers for security reasons still uses fixed
questions to determine talker identity. Yet, the lack
of the use of voice recognizers for security systems
speaks to the limitations of a search for invariant
voice characteristics, and the problems of false rejec-
tions with voice recognizers means that humans
must stay involved in such security processes
(Pardo & Remez, 2006).
Many different acoustic properties have been
studied to try and identify the acoustic param-
eters that are critical to the recognition of a
talker, including the fundamental profile, voicing
variation, spectral shape, jitter (microvariations
in the vibrations of the vocal folds), and shimmer
(microvariations in loudness). These features can
be highly variable: Bricker and Pruzansky (1966)
found that the voices that were confusable on one
vowel were not necessarily confused on another
vowel. Although roles for formant frequencies
(Baumann & Belin, 2010; Murry & Singh, 1980),
hoarseness (Murry & Singh, 1980; Singh & Murry,
1978), vowel duration (Murry & Singh, 1980;
Singh & Murry, 1978), and shimmer (Kreiman,
Gerratt, Precoda, & Berke, 1992) have all been
identified, the fundamental frequency of a voice has
remained the most constant finding in talker identi-
fication across all studies (Creel & Bregman, 2011).
Other approaches systematically transform
voice recordings to specifically alter certain acous-
tic parameters and to determine the effects of this
on the recognition of talker identity. Van Lancker,
Kreiman, and Emmorey (1985a, 1985b) tested
The Voice
the identification of famous people’s voices under
altered duration and temporal reversal (vs. nonre-
versed presentation): Different voices were found to
be recognizable under different acoustic conditions.
It has even been shown that the sex of a talker and
his or her identity can be correctly determined in
sine wave speech—a very reduced signal comprising
sine wave analogues of the first and second formants
in speech—but not from fundamental frequency
(Fellowes, Remez, & Rubin, 1997; Remez,
Fellowes, & Rubin, 1997).
Contextual and Interactive Factors
Although certain acoustic parameters may be par-
ticularly important for certain voices, and although
listeners may also vary individually in the weight
that they give certain cues (Kreiman, 1997; Kreiman
et al., 1992), the recognition of voice identity does
not rest on one set of acoustic factors. This is strik-
ingly like the picture in speech perception, where
no one acoustic property is critical for the percep-
tion of speech, and if one cue is taken away, listen-
ers will use others (e.g., Shannon, Zeng, Kamath,
Wygonski, & Ekelid, 1995). This perhaps should
not be a surprise: As outlined in the introduction,
people normally determine talker information from
the speech the talker produces, and people do so in
a plastic, flexible manner.
In interactions, speaker identity cues and speech
information often interact. People understand
speech in background noise better if they are famil-
iar with the speaker, which indicates that the acous-
tic/phonetic cues to identity and language are not
completely distinct (Nygaard, Sommers, & Pisoni,
1994). Listeners will adapt to talker-specific varia-
tion in specific phonemes, and they appear to do this
to help improve speech comprehension (Eisner &
McQueen, 2005, 2006). Eisner and McQueen (2005,
2006) modified a talker’s /s/ or /sh/ sounds across
conditions such that the /s/ was acoustically more
like /sh/, or vice versa. Listeners listened passively
to a story read by one talker whose voice was modi-
fied in one of these two ways. Afterward, the listen-
ers performed a categorization task on the /s/–/sh/
continuum, where they showed a shift in their per-
ception of the continuum that specifically reflected
the modified speech they had been listening to.
293
Scott and McGettigan
Critically, listeners only showed this shift when
they were presented with the same talker’s voice
for the categorization task, and they did not remap
their phonemic boundary for a new speaker. Thus,
people adapt to the ways that novel talkers produce
phonemes, but this adaptation is talker specific. As
people are continually confronted with other people
who speak the same language as them, but whose
voices are widely different, this adaptation and inte-
gration of speaker specific information are part of
normal speech perception.
More generally, the cues that can be used to
identify speakers in one language can be phonologi-
cally contrastive in another (Gordon & Ladefoged,
2001). Creaky voice and breathy voice are both
talker cues in British English, but in the Mexican
language of Jalapa Mazatec, the voicing of vowels
produced (e.g., produced with modal or creaky
voice) affects their phonetic identities (Kirk, Lade-
foged, & Ladefoged, 1993). In this language, there-
fore, these voicing cues cannot be used for talker
identity purposes.
Learning to tell different talkers apart may entail
using language specific cues that will vary based on
one’s individual experience—Perrachione, Del Tufo,
and Gabrieli (2011) showed that English speak-
ers were better at distinguishing between talkers of
English than they were at telling Chinese talkers
apart. Notably in this same study, English speaking
listeners with dyslexia were poor at distinguishing
between speakers in either language, suggesting
that their perceptual problems are not limited to
the kinds of information that relate specifically to
speech perception.
CULTURAL, PERSONAL, AND LIFE-SPAN
FACTORS THAT AFFECT THE VOICE
Geographical Origins
In many linguistic communities, there are geo-
graphical influences on how talkers produce the
same linguistic items. These variations reflect the
historical influences on these different populations
of speakers, including migration. They operate at
the level of semantics (in the United Kingdom, the
word for a large soft piece of furniture on which
several people can sit varies from couch, to settee, to
294
sofa) and syntax (in the north of England it is com-
mon for people to say “I were” rather than “I was”).
These variations also affect the ways people speak,
from the phonemes that they produce through to
the pitch and posture of the voice (for examples
of B­ ritish English, see http://sounds.bl.uk/accents-
and-dialects/survey-of-english-dialects). Thus, the
vowel in “bath” is said with varying durations in the
United Kingdom, from a short “a” in many northern
locations to a longer “ar” in many southern places
(e.g., Evans & Iverson, 2004). The melody and
rhythm of accents can vary sharply—the sentence
level intonation of a North-Eastern English
(e.g., Geordie) accent is very different from the sen-
tence level intonation of North-Western English
accents. Rhythm and timing can vary across accents
of a language as well: Spoken English is typically
­produced with what is known as stress timing—where
stressed syllables are produced with emphasis, are
longer, and are louder, whereas unstressed syllables
are quieter, and the vowels are more reduced. This
gives spoken English a very characteristic rhythm.
This rhythm can vary across accents of English—for
example, Jamaican and Hispanic speakers of Ameri-
can English produce speech that is less strongly stress
timed, with a greater weight given to all syllables
(i.e., more like a syllabled timed language such as
Spanish or French; Thomas & Carter, 2006).
Lifetime Changes in the Voice—Aging
and Cultural Variation
Some vocal characteristics are altered as people age,
with male voices often becoming higher in pitch as
the vocal folds atrophy, and female voices deepening
in pitch as peri-menopausal hormonal changes influ-
ence and “masculinize” the female larynx
(Linville, 2004). These changes are much more
­individually variable than the influences from body
size and the effects of adolescence on male voices, and
they can interact with other factors—for example,
weight gain (e.g., Gunter & Manning, 1982).
Other sources of variation in voices are the
geographical factors that affect speech and voice.
These change with geography and over time,
­typically driven by influences from younger people
(Tagliamonte & Denis, 2010). Nowadays, for
example, one never meets the Londoners who

produce “w” instead of “v” that Dickens described
(e.g., saying “warmints” instead of “varmints”).
London accents are noticeably different among older
Londoners and younger Londoners, as younger
Londoners have accents more strongly influenced
by some of the more recent multicultural changes in
London. In other words, the speech of younger
Londoners is incorporating aspects of the accents
of people who may well be speaking English as a
second language (Cheshire, Kerswill, Fox, &
Torgersen, 2011). This is a normal process of lin-
guistic change (Labov, 1963), which can also occur
via the medium of television: A recent study found a
positive relationship between changes in Glaswegian
accents among adolescents and their engagement
with the BBC soap opera Eastenders (which features
modern London Cockney accents; Stuart-Smith,
Pryce, Timmins, & Gunter, 2013).
Furthermore, accents can change within a
speaker over his or her lifetime. A study of the
British Queen’s speech from recordings of her
voice over her adult life shows how much her
accent has changed since the earliest recordings
in the 1950s (Harrington, Palethorpe, & Watson,
2000). For example, the Queen has changed how
she produces the vowel in “food” from a position
of the tongue toward the front of the mouth to a
more “back” position, mirroring a change in the
average pronunciation for U.K. speakers from the
1950s to the 1980s. Queen Elizabeth II literally no
longer speaks “the Queen’s English,” at least as it
was spoken in the 1950s. This pattern of change
indicates that older adults are not necessarily
immune to the changes in speaking styles that are
originating in younger populations.
Voice and Social Roles
The act of talking occurs within a social context
that also strongly affects the voice, along with
perceived affiliative and aspirational factors of the
interaction—the way people speak with someone
will often reflect their relationship with him or her
and their emotional response to him or her (Karpf,
2007). For example, children have higher pitched
voices than adults, and there is no sexual dimorphic
influence on the voices prior to adolescence. Despite
this, male and female children of the same size often
The Voice
are found to speak with different spectral charac-
teristics to their formant structure, which has been
identified as the boys striving to speak with a more
adult male voice than the girls (Cartei, Cowels,
Banerjee, & Reby, 2014; Perry, Ohde, & Ashmead,
2001). Boys may be aligning their voices toward
the lower formant frequencies of the males in their
­environments, and/or adults may unconsciously
encourage the differentiation.
These social factors also affect adult voices.
Women in the West (e.g., Europe, United States)
started to speak with lower pitches over the last
40 years or so: This has been suggested to follow the
increased profile of women in the workplace, often
in shared environments with men (Karpf, 2007).
Because the voice is a relatively plastic phenom-
ena, it may be a gender-linked cue that women can
­modulate, perhaps with greater relative ease than
other factors. In other cultures, women can be found
to speak with much higher pitched voices—for
example, in Japan (Loveday, 1981; van Bezooijen,
1995). Similarly, the men in Japan speak with lower
pitches than men in the West, indicating that both
men and women are influenced by cultural expecta-
tions into using their voices in certain ways.
Personality and the Voice
As the voice is an action, it is strongly affected by
dispositional traits that affect how people act. This
means that listeners can be good at determining
­personal characteristics and traits from vocal stim-
uli. Personality traits such as extraversion (Scherer,
1978) and neuroticism can be predicted from the
voice (Hu, Wang, Short, & Fu, 2012). The percep-
tion of dominance is associated with certain vocal
characteristics such as lower pitch (Anderson &
Klofstad, 2012), though it is unclear whether these
reflect personality traits of cultural expectations
about what leaders should sound like (currently,
it appears that we consider that they should sound
like men). The relationship between the voice and
­personality traits reflects the tight relationship
between the voice and the actions that lead to speak-
ing. This may not simply mean that people’s per-
sonalities are expressed and hence affect their voice:
One study found a relationship between introver-
sion and speech dysfluencies, suggesting that speech
295
Scott and McGettigan

difficulties may affect personality (Gawda, 2007).
Furthermore, there is growing evidence for listener
differences in the perception of vocal character-
istics. For example, the specific preferences that
women have in prospective partners affect how
women perceive male voices (Vukovic et al., 2011).
Thus, women who desire short-term relationships
prefer male voices with low pitches, even though
women rate low-pitched male voices as low in
trustworthiness.
BIOLOGICAL EVENTS THAT AFFECT
THE VOICE
Clinical and Pharmacological Changes
to the Voice
Clinical factors can affect the voice. These can be
transient and direct—for example, a viral inflam-
mation of the vocal folds will immediately influ-
ence a talker’s voice. These influences can also be
indirect—a case of colonic perforation (following an
endoscopic investigation) that initially presented as a
vocal change (Kirkpatrick, Koo, Zalev, Burnstein, &
Warren, 1999) due to peri-pharyngeal emphysema
instead transpired to be due to interstitial tracking of
gas that had escaped from the tear.
Long-term illnesses can also have vocal conse-
quences. Disorders of the vocal folds (e.g., spasmodic
dysphonia) will immediately and continuously affect
the voice (Gregory, Chandran, Lurie, & Sataloff,
2012). Such patients’ voices are often described as
sounding tight, “broken,” or whispery. Parkinson’s
disease can be associated with hypokinetic dysar-
thria, which can affect the voice as well as speech
the speech of the Captain of the Exxon Valdez, who
was considered to have been drinking around the
time of the accident, revealed this pattern (Tanford,
Pisoni, & Johnson, 1991). However, variation in
how individual voices are affected by alcohol (some
participants showing no effects with alcohol intake
or even negative effects) means that the voice is at
best a problematic measure for consistently deter-
mining the extent of alcohol intoxication.
Emotion and the Voice
Emotional speech.  Affective states influence the
sound of the voice—Figure 11.3 shows the fun-
damental frequency of a talker’s voice changing
as she starts to laugh while speaking: The altered
control of her intercostal muscles (Kohler, 2008)
as the laughter increases causes her to lose control
of the constant subglottal pressure necessary for
speech (MacLarnon & Hewitt, 1999). This can
lead to a change in the pitch of the speech, which
starts first to vary more widely and then increases
rapidly. Many of these changes are sufficient to over-
whelm talker specific cues in speech—anecdotally,
talkers can become unrecognizable with grief, for
example—and in extreme circumstances can prevent
people from speaking altogether.
Darwin (1872) was the first to specifically sug-
gest that the physiological changes associated with
0
500
Pitch (Hz)

production, and the voice can be breathy and harsh

in tone, with range of pitch variation (e.g., Skodda,
Grönheit, Mancinelli, & Schlegel, 2013).
Chemicals can also change the voice. For
­example, alcohol intoxication affects speech pro-
duction, with a consistent relationship for both
men and women in the height of the pitch of the
voice (­ measured as fundamental frequency) and an
increase in speech dysfluencies with the amount
of alcohol ingested (Hollien, DeJong, M ­ artin,
Schwartz, & Liljegren, 2001). This raises the
­possibilities of using voice measures as an index of
alcohol intake, for example, a detailed analysis of
75
0 8.053
Time (s)
FIGURE 11.3.  Voice pitch estimation for a British
English speaker who is reading the news on live radio
and who starts to laugh. The changes in pitch profile
and the increases in pitch indicate the progression of
the laughter, which starts to dominate the control of the
intercostal muscles and articulators.

296

emotional states are made directly manifest in the
voice. Work by Klaus R. Scherer has developed this
into a model of vocal emotions, which incorporates
these physiological changes along with other cues
to context and culture (e.g., display rules; Ekman,
1972), which describe the ways that certain emotions
are not polite or appropriate to display in certain
cultures, which will then modulate the expressed
emotions (Scherer, 1986).
A detailed study of ways that emotion affects
the voice, using nonsense sentences as stimuli,
identified a set of specific acoustic measures that
could be used as predictors of judges’ ratings of
the emotional speech and to automatically classify
the sentences into different emotional categories
(Banse & Scherer, 1996). Within this range of
acoustic factors, the study identified different acous-
tic cues that were associated with specific categories
of emotional speech. For example, higher ratings
for “panic fear” and “despair” were associated with
higher mean pitch, and higher ratings for boredom,
pride, and contempt were associated with lower
mean pitch. The broad band spectral characteristics
of the speech were also important: Higher ratings
for “hot anger” were associated with larger differ-
ences between lower (0–2000 Hz) and midrange
(2000–5000 Hz) spectral energy, and higher ratings
for “elation” were associated with higher propor-
tions of energy more than 1000 Hz. Lower levels of
intensity (associated with perceptual loudness) were
an index of the ratings for “shame” and “sadness.”
Other studies have shown a role for pitch, spectral
profile, and loudness as indexes of emotional states
in speech (Juslin & Laukka, 2001), with a particular
emphasis on pitch as a critical cue in the expression
and recognition of emotion in speech (Bänziger &
Scherer, 2005). It has been hard to quantify the
acoustic correlates of the perceptual properties of
emotional speech (Scherer, 1986). For example,
angry voices have a lot of tension in them
(Murray & Arnott, 1993), reflecting how the jaw is
held: This is hard to express as a simple acoustic
factor. Most studies have therefore looked at a sub-
set of possible acoustic measures, often related to
pitch (Juslin & Laukka, 2003).
In contrast to the idea that specific emotional
states can be inferred from speech, it has been argued
The Voice
that emotional speech provides an indication of “non-
specific arousal” associated with ongoing affective
states (Bachorowski, 1999). A series of studies have
found support for this, and significant and complex
acoustic correlates of affective dimensions—such
as activation, valence, intensity, and potency—have
also been determined in emotionally inflected speech
(Bachorowski & Braaten, 1994; Bachorowski &
Owren, 1995; Laukka, Juslin, & Bresin, 2005).
Nonverbal emotional expressions.  Emotional
speech is not the only way that talkers demonstrate
their affective state in their voices: In Figure 11.3,
the talker’s voice is changing as she starts to laugh,
but when she starts to laugh completely, she stops
speaking altogether, though she continues to make
a vocal noise. This change can be distinguished
somewhat by differentiating between voluntary
and involuntary vocalizations. Speech, even if it
is emotionally inflected, is controlled by a lateral
motor cortical system, which is associated with the
planning and control of breathing, voicing, and
articulation necessary during speech: This system
appears to be specific to humans. In contrast, non-
verbal emotional expressions—such as screams and
laughs—and highly automatic, highly emotionally
laden terms—such as swear words—are associ-
ated with an evolutionarily older system involving
midline structures, including the anterior cingu-
late (Jürgens, 2002). These two systems can be
dissociated in cases of expressive aphasia, when
people can be (possibly temporarily) robbed of
the ability to speak voluntarily but can still laugh,
cry, and so forth in appropriate circumstances.
These nonverbal emotional expressions have been
found to complement emotional facial expressions
(Scott et al., 1997)—for example, the “basic” emo-
tions of fear, anger, sadness, surprise, and disgust
are recognized universally from the face (Ekman,
Sorenson, & Friesen, 1969) and the voice (Sauter,
Eisner, Ekman, & Scott, 2010). Work in nonverbal
vocal expressions of emotion has also indicated that,
as predicted by Ekman (1992), the voice is an effec-
tive way of expressing positive emotions (Sauter &
Scott, 2007), such a laughter and relief.
Nonverbal vocalizations have been described as
being more like animal vocalizations than like speech
297
Scott and McGettigan
(Scott, Sauter, & McGettigan, 2009), for two general
reasons. First, as described earlier, the spontaneous
expression of such nonverbal sounds is associated
with an evolutionary older neural system, which we
share with other mammals (Jürgens, 2002). Second,
these nonverbal expressions are characterized and
distinguished by amplitude, pitch, and broad spectral
features. They do not contain the spectral structure
and complexity of speech, which reflects the supra-
laryngeal movements that underlie speech sounds.
Scherer (1994) has termed these sounds affect bursts,
but we prefer to use the term nonverbal emotional
expressions, as the term “burst” has general connota-
tions of brevity, which does not characterize all such
sounds. Empirically, for example, perceptual ratings
for “surprise” are higher for shorter sounds, whereas
ratings for “contentment” are higher for longer
sounds (Sauter, Eisner, Calder, & Scott, 2010).
VOICES, ADAPTATIONS, AND
INTERACTIONS
Voice and Acoustic Context
Voices are strongly and often unconsciously affected
by the places and contexts in which we speak. In the
Lombard effect, people alter how they speak depend-
ing on the ambient noise levels around them—talkers
will increase the loudness of their voice in a way
that correlates with the loudness of the surrounding
sounds (Lombard, 1911). If the content of the back-
ground noise is varied, studies have shown that talk-
ers will vary the spectral characteristics of their voice
to exploit “gaps” in the noise spectrum (Cooke &
Lu, 2010). This suggests that vocal responses to
background sounds are complex responses to the
problems of making the voice understood (or more
understandable) to listeners in a difficult environ-
ment. In different contexts, people will change their
voice if they are speaking to someone who is wearing
a hearing aid or if they are speaking to a young child.
The voice is thus also affected by the communicative
situation in which a talker finds him- or herself.
Conversational Interaction—Voice
Alignment and Unconscious Change
Several studies have shown that voice change
is a critical feature of conversational alignment
298
(Garrod & Pickering, 2004). This alignment
describes how talkers will start to breathe together
and speak at the same rate in conversational speech
(McFarland, 2001; Pardo, 2006). Talkers will also
start to use the same postures (postural congru-
ence) and will converge on similar pitches with
their voices; moreover, if there is a greater rapport
between the talkers, they will also start to use simi-
lar words and syntactic structures and converge on
the pronunciations of specific words (Chartrand &
Bargh, 1999). This means that normal conversa-
tional speech can entail a considerable amount of
speech and voice change, depending on who one is
talking to and how much one likes them—the more
a person likes someone, the more he or she will mir-
ror back that person’s behavior, voice, and language
(e.g., Farley, Hughes, & LaFayette, 2013). Voice
change can thus be seen as an important part of
normal affiliative social behavior.
Conscious Changes in the Voice
On occasion, talkers will overtly try and change
their voices, often with the aim of affecting how they
are perceived by listeners. Margaret Thatcher was
famously coached to lower her voice when she was
seeking higher office in the U.K. government (Karpf,
2007), although the reasoning behind this coaching
may have reflected somewhat sexist assumptions
about the perception of female voices (Cameron,
2008). The effect was that Thatcher deliberately and
consciously adopted a vocal range that differed from
her “normal” range when speaking in any public
setting. In retrospect, the overall effect nowadays
sounds somewhat mannered, though that may
reflect other aspects of people’s assessment of her
voice and her time in office.
In contrast to perceptually effortful and not
always effective attempts to change the voice, there
are individuals who excel at this task. Whole
disciplines of acting and voice artistry are based
on individuals’ abilities to deliberately alter their
voices, and politicians (male and female) still try
to modulate their voices to sound more palatable
to the electorate (with greater and lesser degrees of
success). Such expertise is beyond the remit of this
chapter, though we would note that such expert
change can be very dramatic. We have worked with
 The Voice

a professional impressionist who plays a character
on a children’s TV program—although we know
that it is him, and we know his voice and its range,
we get no sense of recognition when his voice is so
altered.
Nonprofessionals can consciously change their
vocal sounds with a greater ability than we com-
monly believe. Strikingly, it has been shown that
people untrained and uncoached in voice change
can still alter their voices to the point where they
can achieve typical male and female target voices
(Cartei et al., 2014) when instructed to. This delib-
erate control of voice change may underlie talk-
ers’ abilities to disguise themselves on the phone,
though such attempts will be more successful
when talkers speak to someone who does not know
their “normal” voice. Criminal cases in which
people need to be identified from their voice can
be adversely affected by this tendency for people
to change their normal speaking voice when moti-
vated to conceal their identity, and by differences
between how people sound during police interviews
compared to other recordings (Bahr & Pass, 1996).
The early hopes of a “voice print,” which might
be as individual as a fingerprint, have proved to be
­elusive, and proving a talker’s identity objectively
can be hard to establish in court (Nolan, 1991).
Because most people have had the experience of
­recognizing a familiar individual from his or her
voice, they can be mistaken in believing that they
are better at recognizing unfamiliar voices, even
though the data show that people are poor at recog-
nizing unfamiliar voices (Nolan, 2001).
Voices on the Brain?
Neuropsychological studies have shown some dif-
ferentiation in the kinds of organic brain damage that
can affect voice perception and speech perception.
Difficulties comprehending spoken language are
associated with left lateralized damage to the dorso-
lateral temporal lobe (Wernicke, 1874/1974; see
also Rauschecker & Scott, 2009), specifically the left
superior temporal gyrus. In contrast, patients with
phonagnosia—a problem with recognizing talkers
from their voices—typically have lesions in the right
temporal lobe (Van Lancker, Kreiman, & Cummings,
1989). A proviso here is that perception of talker
information may not be consistently investigated in
aphasic patients—a recent study did show impaired
accent recognition in individuals with receptive apha-
sia, for example (Newton, Burns, & Bruce, 2013).
Recent developments in functional neuroimag-
ing have largely supported this left–right distinc-
tion, with sensitivity to intelligible speech in the
left temporal lobe, regardless of what sort of voice
appears to be speaking (Scott, Blank, Rosen, &
Wise, 2000). In contrast, the right temporal lobe is
sensitive to speaker differences (Belin & Zatorre,
2003), indexical information about talker identity
(Scott et al., 2000), and even to basic properties of
pitch variation in the voice (Kyong et al., 2014; see
Figure 11.4). A voice area has been described in the

Left hemisphere Right hemisphere

Primary auditory
cortex

Superior
temporal gyrus

Superior
temporal sulcus

FIGURE 11.4.  Left and right lateral aspects of the human cortex, with regions particularly sensitive to speech showing
in the left superior temporal gyrus, and with regions particularly sensitive to pitch variation, talker change, and talker
identity showing in the right superior temporal gyrus.

299
Scott and McGettigan
right temporal lobe (the right temporal lobe voice
area [TVA]), which encompasses many of these pro-
cesses (e.g., Belin, Zatorre, Lafaille, Ahad, & Pike,
2000; Bestelmeyer, Belin, & Grosbras, 2011). Inter-
estingly, a recent study from our group, in which
we asked functional magnetic resonance imaging
participants to speak in different accents and vocal
impressions, found that the TVA was s­ pecifically
activated when people spoke (or attempted to
speak) like named individuals (e.g., Sean Connery;
McGettigan et al., 2013). This indicates that (for
nonprofessionals at least) the TVA is recruited to
support deliberate voice change, when the talker has
a particular vocal identity target in mind. This, in
turn, suggests that the TVA is not driven solely to
acoustic factors when it responds to voices, and it
may represent abstract information about individual
voices.
A problem with all univariate functional imag-
ing techniques is that they are aimed at showing
differences—and the sorts of studies here are empha-
sizing different ways that speech and talker informa-
tion are represented and processed in the human
brain. Multivariate approaches to speech and talker
information have indicated a more complex inter-
hemispheric relationship between processing speech
and talker information (Formisano, De Martino,
Bonte, & Goebel, 2008)—as we might predict from
the behavioral data—and further studies will shine
light on, for example, the ways that talker-specific
adaptations take place in normal speech perception.
THE FUTURE
The voice is an acoustic manifestation of the self
in a social context (Sidtis & Kreiman, 2012), and
researchers need to appreciate this complexity and
plasticity when they think about it. No one acous-
tic cue underlies the perception of speech, and it
is becoming clear that the same is true for many
aspects of voice processing, particularly issues
around talker characteristics and identity. Future
studies will embrace this complexity and allow
researchers to address, for example, the ways that
the talker cues dynamically vary with language as
well as how people use talker information to help
them understand speech. The opportunities to
300
explore the roles of individual variations in the mor-
phology of the vocal tract on speech and voice—and
how these might relate to the face—will continue to
develop as magnetic resonance imaging techniques
improve (e.g., Lammert et al., 2013).
A recent study showed that people, when
unknowingly presented with their own voice among
the recordings of strangers and asked to make
aesthetic judgments about the pleasantness of the
voices, tend to rate their own voice as more pleas-
ant than the voices of others (Hughes & Harrison,
2013). This can be interpreted as showing a narcis-
sistic relationship with one’s own voice—a finding
at odds with the studies that typically find that peo-
ple dislike the sound of their own voice when they
are aware that they are listening to a recording of it.
However, if we recall that the voice is an active vehi-
cle for many different kinds of influences on us as
social animals, a way that we continuously display
our affiliations and aspirations, then perhaps we like
the sound of our own voice most of all—as once
removed from our knowledge that it is us speaking,
we can appreciate the aspirations and affiliations as
chiming very precisely with our own. When we talk
about human voices, it is impossible to meaning-
fully strip them of the factors that influence how we
would like the world to hear us when we speak.
References
Anderson, R. C., & Klofstad, C. A. (2012). Preference
for leaders with masculine voices holds in the case
of feminine leadership roles. PLoS ONE, 7, e51216.
http://dx.doi.org/10.1371/journal.pone.0051216
Bachorowski, J. A. (1999). Vocal expression and
perception of emotion. Current Directions in
Psychological Science, 8, 53–57. http://dx.doi.
org/10.1111/1467-8721.00013
Bachorowski, J. A., & Braaten, E. B. (1994). Emotional
intensity: Measurement and theoretical implications.
Personality and Individual Differences, 17, 191–199.
http://dx.doi.org/10.1016/0191-8869(94)90025-6
Bachorowski, J. A., & Owren, M. J. (1995). Vocal
expression of emotion: Acoustic properties of speech
are associated with emotional intensity and context.
Psychological Science, 6, 219–224. http://dx.doi.org/
10.1111/j.1467-9280.1995.tb00596.x
Bahr, R. H., & Pass, K. J. (1996). The influence of style-
shifting on voice identification. International Journal
of Speech Language and the Law, 3, 25–38.

Bailey, P. J., & Summerfield, Q. (1980). Information in
speech: Observations on the perception of [s]-stop
clusters. Journal of Experimental Psychology: Human
Perception and Performance, 6, 536–563. http://
dx.doi.org/10.1037/0096-1523.6.3.536
Banse, R., & Scherer, K. R. (1996). Acoustic profiles in
vocal emotion expression. Journal of Personality and
Social Psychology, 70, 614–636. http://dx.doi.org/
10.1037/0022-3514.70.3.614
Bänziger, T., & Scherer, K. R. (2005). The role of
intonation in emotional expressions. Speech
Communication, 46, 252–267. http://dx.doi.org/
10.1016/j.specom.2005.02.016
Baumann, O., & Belin, P. (2010). Perceptual scaling of
voice identity: Common dimensions for different
vowels and speakers. Psychological Research, 74,
110–120. http://dx.doi.org/10.1007/s00426-008-
0185-z
Belin, P., Fecteau, S., & Bédard, C. (2004). Thinking the
voice: Neural correlates of voice perception. Trends
in Cognitive Sciences, 8, 129–135. http://dx.doi.org/
10.1016/j.tics.2004.01.008
Belin, P., & Zatorre, R. J. (2003). Adaptation to speaker’s
voice in right anterior temporal lobe. NeuroReport,
14, 2105–2109. http://dx.doi.org/10.1097/00001756-
200311140-00019
Belin, P., Zatorre, R. J., Lafaille, P., Ahad, P., & Pike, B.
(2000). Voice-selective areas in human auditory
cortex. Nature, 403, 309–312. http://dx.doi.org/
10.1038/35002078
Bestelmeyer, P. E. G., Belin, P., & Grosbras, M. H.
(2011). Right temporal TMS impairs voice detection.
Current Biology, 21, R838–R839. http://dx.doi.org/
10.1016/j.cub.2011.08.046
Bricker, P. D., & Pruzansky, S. (1966). Effects of stimulus
content and duration on talker identification. Journal
of the Acoustical Society of America, 40, 1441–1449.
http://dx.doi.org/10.1121/1.1910246
Cameron, D. (2008). The myth of Mars and Venus: Do men
and women really speak different languages? Oxford,
England: Oxford University Press.
Cartei, V., Cowles, H. W., Banerjee, R., & Reby, D.
(2014). Control of voice gender in pre-pubertal
children. British Journal of Developmental Psychology,
32, 100–106. http://dx.doi.org/10.1111/bjdp.12027
Chartrand, T. L., & Bargh, J. A. (1999). The chameleon
effect: The perception–behavior link and social
interaction. Journal of Personality and Social
Psychology, 76, 893–910. http://dx.doi.org/
10.1037/0022-3514.76.6.893
Cheshire, J., Kerswill, P., Fox, S., & Torgersen, E. (2011).
Contact, the feature pool and the speech community:
The emergence of Multicultural London English.
Journal of Sociolinguistics, 15, 151–196. http://dx.doi.
org/10.1111/j.1467-9841.2011.00478.x
The Voice
Cooke, M., & Lu, Y. (2010). Spectral and temporal
changes to speech produced in the presence of
energetic and informational maskers. Journal of the
Acoustical Society of America, 128, 2059–2069.
http://dx.doi.org/10.1121/1.3478775
Creel, S. C., & Bregman, M. R. (2011). How talker
identity relates to language processing. Language and
Linguistics Compass, 5, 190–204. http://dx.doi.org/
10.1111/j.1749-818X.2011.00276.x
Darwin, C. (1872). The expression of the emotions in man
and animals. http://dx.doi.org/10.1037/10001-000
Eisner, F., & McQueen, J. M. (2005). The specificity of
perceptual learning in speech processing. Perception
and Psychophysics, 67, 224–238. http://dx.doi.org/
10.3758/BF03206487
Eisner, F., & McQueen, J. M. (2006). Perceptual learning
in speech: Stability over time. Journal of the Acoustical
Society of America, 119, 1950–1953. http://dx.doi.org/
10.1121/1.2178721
Ekman, P. (1972). Universals and cultural differences
in facial expressions of emotions. In J. Cole (Ed.),
Nebraska Symposium on Motivation (pp. 207–282).
Lincoln: University of Nebraska Press.
Ekman, P. (1992). An argument for basic emotions.
Cognition and Emotion, 6, 169–200. http://dx.doi.org/
10.1080/02699939208411068
Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969).
Pan-cultural elements in facial displays of emotion.
Science, 164, 86–88. http://dx.doi.org/10.1126/
science.164.3875.86
Evans, B. G., & Iverson, P. (2004). Vowel normalization
for accent: An investigation of best exemplar
locations in northern and southern British English
sentences. Journal of the Acoustical Society of America,
115, 352–361. http://dx.doi.org/10.1121/1.1635413
Fant, G. (1960). Acoustic theory of speech production.
The Hague, the Netherlands: Mouton.
Farley, S. D., Hughes, S. M., & LaFayette, J. N. (2013).
People will know we are in love: Evidence of
differences between vocal samples directed toward
lovers and friends. Journal of Nonverbal Behavior,
37, 123–138. http://dx.doi.org/10.1007/s10919-
013-0151-3
Fellowes, J. M., Remez, R. E., & Rubin, P. E. (1997).
Perceiving the sex and identity of a talker without
natural vocal timbre. Perception and Psychophysics,
59, 839–849. http://dx.doi.org/10.3758/BF03205502
Fitch, W. T., & Giedd, J. (1999). Morphology and
development of the human vocal tract: A study using
magnetic resonance imaging. Journal of the Acoustical
Society of America, 106, 1511–1522. http://dx.doi.org/
10.1121/1.427148
Formisano, E., De Martino, F., Bonte, M., & Goebel, R.
(2008). “Who” is saying “what”? Brain-based
301
Scott and McGettigan
decoding of human voice and speech. Science, 322,
970–973. http://dx.doi.org/10.1126/science.1164318
Garrod, S., & Pickering, M. J. (2004). Why is conversation
so easy? Trends in Cognitive Sciences, 8, 8–11. http://
dx.doi.org/10.1016/j.tics.2003.10.016
Gawda, B. (2007). Neuroticism, extraversion, and
paralinguistic expression. Psychological Reports,
100, 721–726. http://dx.doi.org/10.2466/
pr0.100.3.721-726
Gordon, M., & Ladefoged, P. (2001). Phonation types:
A cross-linguistic overview. Journal of Phonetics, 29,
383–406. http://dx.doi.org/10.1006/jpho.2001.0147
Gregory, N. D., Chandran, S., Lurie, D., & Sataloff, R. T.
(2012). Voice disorders in the elderly. Journal of
Voice, 26, 254–258. http://dx.doi.org/10.1016/
j.jvoice.2010.10.024
Gunter, C., & Manning, W. (1982). Listener estimates of
speaker height and weight in filtered and unfiltered
conditions. Journal of Phonetics, 10, 251–257.
Harrington, J., Palethorpe, S., & Watson, C. I. (2000).
Does the Queen speak the Queen’s English?
Elizabeth II’s traditional pronunciation has been
influenced by modern trends. Nature, 408, 927–928.
http://dx.doi.org/10.1038/35050160
Hiki, S., & Itoh, H. (1986). Influence of palate shape
on lingual articulation. Speech Communication,
5, 141–158. http://dx.doi.org/10.1016/0167-
6393(86)90004-X
Hollien, H., DeJong, G., Martin, C. A., Schwartz, R., &
Liljegren, K. (2001). Effects of ethanol intoxication
on speech suprasegmentals. Journal of the Acoustical
Society of America, 110, 3198–3206. http://dx.doi.
org/10.1121/1.1413751
Hollien, H., Majewski, W., & Doherty, E. T. (1982).
Perceptual identification of voices under normal,
stress, and disguise speaking conditions. Journal of
Phonetics, 10, 139–148.
Hu, C., Wang, Q., Short, L. A., & Fu, G. (2012). Speech
spectrum’s correlation with speakers’ Eysenck
personality traits. PLoS ONE, 7, e33906. http://
dx.doi.org/10.1371/journal.pone.0033906
Hughes, S. M., & Harrison, M. A. (2013). I like my voice
better: Self-enhancement bias in perceptions of voice
attractiveness. Perception, 42, 941–949. http://dx.doi.
org/10.1068/p7526
Jürgens, U. (2002). Neural pathways underlying vocal
control. Neuroscience and Biobehavioral Reviews,
26, 235–258. http://dx.doi.org/10.1016/S0149-
7634(01)00068-9
Juslin, P. N., & Laukka, P. (2001). Impact of intended
emotion intensity on cue utilization and decoding
accuracy in vocal expression of emotion. Emotion,
1, 381–412. http://dx.doi.org/10.1037/1528-
3542.1.4.381
302
Juslin, P. N., & Laukka, P. (2003). Communication
of emotions in vocal expression and music
performance: Different channels, same code?
Psychological Bulletin, 129, 770–814. http://dx.doi.
org/10.1037/0033-2909.129.5.770
Kamachi, M., Hill, H., Lander, K., & Vatikiotis-Bateson,
E. (2003). “Putting the face to the voice”: Matching
identity across modality. Current Biology, 13,
1709–1714. http://dx.doi.org/10.1016/j.cub.
2003.09.005
Karpf, A. (2007). The human voice: The story of a
remarkable talent. London, England: Bloomsbury
Publishing.
Kirk, P. L., Ladefoged, J., & Ladefoged, P. (1993).
Quantifying acoustic properties of modal, breathy
and creaky vowels in Jalapa Mazatec. In T. Montler &
A. Mattina (Eds.), American Indian linguistics and
ethnography in honor of Laurence C. Thompson
(pp. 433–450). Missoula: University of Montana.
Kirkpatrick, A. W., Koo, J., Zalev, A. H., Burnstein, M. J., &
Warren, R. E. (1999). Endoscopic perforation of
the rectum presenting initially as a change in voice.
Canadian Journal of Surgery, 42, 305–306.
Kohler, K. J. (2008). ‘Speech-smile,’ ‘speech-laugh,’
‘laughter’ and their sequencing in dialogic
interaction. Phonetica, 65, 1–18. http://dx.doi.org/
10.1159/000130013
Kreiman, J. (1997). Listening to voices: Theory and
practice in voice perception research. In K. Johnson &
J. W. Mullennix (Eds.), Talker variability in speech
processing (pp. 85–108). New York, NY: Academic
Press.
Kreiman, J., Gerratt, B. R., Precoda, K., & Berke, G. S.
(1992). Individual differences in voice quality
perception. Journal of Speech and Hearing Research, 35,
512–520. http://dx.doi.org/10.1044/jshr.3503.512
Kyong, J. S., Scott, S. K., Rosen, S., Howe, T. B.,
Agnew, Z. K., & McGettigan, C. (2014). Exploring
the roles of spectral detail and intonation contour
in speech intelligibility: An fMRI study. Journal of
Cognitive Neuroscience, 26, 1748–1763. http://dx.doi.
org/10.1162/jocn_a_00583
Labov, W. (1963). The social motivation of a sound
change. Word, 19, 273–309.
Lachs, L., & Pisoni, D. B. (2004). Crossmodal source
identification in speech perception. Ecological
Psychology, 16, 159–187. http://dx.doi.org/10.1207/
s15326969eco1603_1
Ladefoged, P. (1975). A course in phonetics. New York,
NY: Harcourt Brace Jovanovich.
Lammert, A., Proctor, M., & Narayanan, S. (2013).
Morphological variation in the adult hard palate
and posterior pharyngeal wall. Journal of Speech,
Language, and Hearing Research, 56, 521–530.
http://dx.doi.org/10.1044/1092-4388(2012/12-0059)

Laukka, P., Juslin, P., & Bresin, R. (2005). A dimensional
approach to vocal expression of emotion.
Cognition and Emotion, 19, 633–653. http://dx.doi.
org/10.1080/02699930441000445
Lieberman, D. E. (2011). The evolution of the human head.
Cambridge, MA: Harvard University.
Lieberman, P. (1968). Primate vocalizations and
human linguistic ability. Journal of the Acoustical
Society of America, 44, 1574–1584. http://dx.doi.
org/10.1121/1.1911299
Linville, S. E. (2004, October 19). The aging voice.
ASHA Leader. Retrieved from http://www.asha.org/
publications/leader/2004/041019/041019e.htm
Lisker, L. (1977). Rapid versus rabid: A catalog of
acoustic features that may cue distinction. Journal of
the Acoustical Society of America, 62(Suppl. 1), S77.
http://dx.doi.org/10.1121/1.2016377
Lombard, E. (1911). Le signe de l’elevation de la voix
[The sign of the rise in the voice]. Annales des
Maladies de L’Oreille et du Larynx, 37, 101–119.
Loveday, L. (1981). Pitch, politeness and sexual role—An
exploratory investigation into the pitch correlates of
English and Japanese politeness formulas. Language
and Speech, 24, 71–89.
MacLarnon, A. M., & Hewitt, G. P. (1999). The evolution
of human speech: The role of enhanced breathing
control. American Journal of Physical Anthropology,
109, 341–363. http://dx.doi.org/10.1002/(SICI)1096-
8644(199907)109:3<341::AID-AJPA5>3.0.CO;2-2
Mavica, L. W., & Barenholtz, E. (2013). Matching
voice and face identity from static images. Journal
of Experimental Psychology: Human Perception and
Performance, 39, 307–312. http://dx.doi.org/10.1037/
a0030945
McFarland, D. H. (2001). Respiratory markers of
conversational interaction. Journal of Speech,
Language, and Hearing Research, 44, 128–143. http://
dx.doi.org/10.1044/1092-4388(2001/012)
McGettigan, C., Eisner, F., Agnew, Z. K., Manly, T.,
Wisbey, D., & Scott, S. K. (2013). T’ain’t what
you say, it’s the way that you say it—Left insula
and inferior frontal cortex work in interaction
with superior temporal regions to control the
performance of vocal impersonations. Journal of
Cognitive Neuroscience, 25, 1875–1886. http://dx.doi.
org/10.1162/jocn_a_00427
Murray, I. R., & Arnott, J. L. (1993). Toward the
simulation of emotion in synthetic speech: A review
of the literature on human vocal emotion. Journal
of the Acoustical Society of America, 93, 1097–1108.
http://dx.doi.org/10.1121/1.405558
Murry, T., & Singh, S. (1980). Multidimensional analysis
of male and female voices. Journal of the Acoustical
Society of America, 68, 1294–1300. http://dx.doi.org/
10.1121/1.385122
The Voice
Negus, V. E. (1949). The comparative anatomy and
physiology of the larynx. London, England: William
Heineman Medical Books.
Newton, C., Burns, R., & Bruce, C. (2013). Accent
identification by adults with aphasia. Clinical
Linguistics and Phonetics, 27, 287–298. http://dx.doi.
org/10.3109/02699206.2012.753111
Nolan, F. (1991). Forensic phonetics. Journal of
Linguistics, 27, 483–493. http://dx.doi.org/10.1017/
S0022226700012755
Nolan, F. (2001, December). Speaker identification
evidence: Its forms, limitations, and roles. In
Proceedings of the Conference on Law and Language:
Prospect and Retrospect (pp. 1–19). Retrieved from
http://www.ling.cam.ac.uk/francis
Nolan, F. (2012). Degrees of freedom in speech
production: An argument for native speakers in
LADO. International Journal of Speech Language and
the Law, 19, 263–289.
Nygaard, L. C., Sommers, M. S., & Pisoni, D. B. (1994).
Speech perception as a talker-contingent process.
Psychological Science, 5, 42–46. http://dx.doi.org/
10.1111/j.1467-9280.1994.tb00612.x
Pardo, J. S. (2006). On phonetic convergence during
conversational interaction. Journal of the Acoustical
Society of America, 119, 2382–2393. http://dx.doi.
org/10.1121/1.2178720
Pardo, J. S., & Remez, R. E. (2006). The perception
of speech. In M. Traxler & M. A. Gernsbacher
(Eds.), The handbook of psycholinguistics (2nd ed.,
pp. 201–248). http://dx.doi.org/10.1016/B978-
012369374-7/50008-0
Patterson, R. D., Smith, D. R. R., van Dinther, R., &
Walters, T. C. (2008). Size Information in the
production and perception of communication
sounds. In W. A. Yost, A. N. Popper, & R. R. Fay
(Eds.), Auditory perception of sound sources (pp.
43–75). New York, NY: Springer.
Perrachione, T. K., Del Tufo, S. N., & Gabrieli, J. D.
(2011). Human voice recognition depends on
language ability. Science, 333, 595. http://dx.doi.
org/10.1126/science.1207327
Perry, T. L., Ohde, R. N., & Ashmead, D. H. (2001).
The acoustic bases for gender identification from
children’s voices. Journal of the Acoustical Society of
America, 109, 2988–2998. http://dx.doi.org/
10.1121/1.1370525
Rauschecker, J. P., & Scott, S. K. (2009). Maps and
streams in the auditory cortex: Nonhuman primates
illuminate human speech processing. Nature
Neuroscience, 12, 718–724. http://dx.doi.org/10.1038/
nn.2331
Remez, R. E., Fellowes, J. M., & Rubin, P. E. (1997).
Talker identification based on phonetic information.
303
Scott and McGettigan
Journal of Experimental Psychology: Human Perception
and Performance, 23, 651–666. http://dx.doi.
org/10.1037/0096-1523.23.3.651
Sauter, D. A., Eisner, F., Calder, A. J., & Scott, S. K.
(2010). Perceptual cues in nonverbal vocal
expressions of emotion. Quarterly Journal of
Experimental Psychology, 63, 2251–2272. http://
dx.doi.org/10.1080/17470211003721642
Sauter, D. A., Eisner, F., Ekman, P., & Scott, S. K.
(2010). Cross-cultural recognition of basic emotions
through nonverbal emotional vocalizations.
Proceedings of the National Academy of Sciences,
USA, 107, 2408–2412. http://dx.doi.org/10.1073/
pnas.0908239106
Sauter, D. A., & Scott, S. K. (2007). More than one kind
of happiness: Can we recognize vocal expressions
of different positive states? Motivation and Emotion,
31, 192–199. http://dx.doi.org/10.1007/s11031-007-
9065-x
Scherer, K. R. (1978). Personality inference from voice
quality: Loud voice of extroversion. European Journal
of Social Psychology, 8, 467–487. http://dx.doi.
org/10.1002/ejsp.2420080405
Scherer, K. R. (1986). Vocal affect expression: A review
and a model for future research. Psychological
Bulletin, 99, 143–165. http://dx.doi.org/
10.1037/0033-2909.99.2.143
Scherer, K. R. (1994). Affect bursts. In S. van Goozen, N. E.
van de Poll, & J. A. Sergeant (Eds.), Emotions: Essays on
emotion theory (pp. 161–196). Hillsdale, NJ: Erlbaum.
Scott, S. K. (2008). Voice processing in monkey and
human brains. Trends in Cognitive Sciences, 12,
323–325. http://dx.doi.org/10.1016/
j.tics.2008.06.003
Scott, S. K., Blank, C. C., Rosen, S., & Wise, R. J. S.
(2000). Identification of a pathway for intelligible
speech in the left temporal lobe. Brain: A Journal of
Neurology, 123, 2400–2406. http://dx.doi.org/
10.1093/brain/123.12.2400
Scott, S. K., Sauter, D., & McGettigan, C. (2009). Brain
mechanisms for processing perceived emotional
vocalizations in humans. In S. Brudzynski (Ed.),
Handbook of mammalian vocalization: An integrative
neuroscience approach (pp. 187–197). London,
England: Academic Press.
Scott, S. K., Young, A. W., Calder, A. J., Hellawell, D. J.,
Aggleton, J. P., & Johnsons, M. (1997). Impaired
auditory recognition of fear and anger following
bilateral amygdala lesions. Nature, 385, 254–257.
http://dx.doi.org/10.1038/385254a0
Shannon, R. V., Zeng, F.-G., Kamath, V., Wygonski, J., &
Ekelid, M. (1995). Speech recognition with primarily
temporal cues. Science, 270, 303–304. http://dx.doi.
org/10.1126/science.270.5234.303
304
Sidtis, D., & Kreiman, J. (2012). In the beginning
was the familiar voice: Personally familiar voices
in the evolutionary and contemporary biology
of communication. Integrative Psychological and
Behavioral Science, 46, 146–159. http://dx.doi.
org/10.1007/s12124-011-9177-4
Singh, S., & Murry, T. (1978). Multidimensional
classification of normal voice qualities. Journal of
the Acoustical Society of America, 64, 81–87. http://
dx.doi.org/10.1121/1.381958
Skodda, S., Grönheit, W., Mancinelli, N., & Schlegel, U.
(2013). Progression of voice and speech impairment
in the course of Parkinson’s disease: A longitudinal
study. Parkinson’s Disease, 2013, 389195. http://
dx.doi.org/10.1155/2013/389195
Stuart-Smith, J., Pryce, G., Timmins, C., & Gunter, B.
(2013). Television can also be a factor in
language change: Evidence from an urban dialect.
Language, 89, 501–536. http://dx.doi.org/10.1353/
lan.2013.0041
Tagliamonte, S., & Denis, D. (2010). The stuff of change:
General extenders in Toronto, Canada. Journal
of English Linguistics, 38, 335–368. http://dx.doi.
org/10.1177/0075424210367484
Tanford, J. A., Pisoni, D. B., & Johnson, K. (1991). Novel
scientific evidence of intoxication: Acoustic analysis
of voice recordings from the Exxon Valdez. Journal of
Criminal Law and Criminology, 82, 579–609. http://
dx.doi.org/10.2307/1143746
Thomas, E. R., & Carter, P. M. (2006). Prosodic rhythm
and African American English. English World-Wide,
27, 331–355. http://dx.doi.org/10.1075/eww.
27.3.06tho
Tiede, M. K., Gracco, V. L., Shiller, D. M., Espy-Wilson, C., &
Boyce, S. E. (2005). Perturbed palatal shape and
North American English /r/ production. Journal of
the Acoustical Society of America, 117, 2568. http://
dx.doi.org/10.1121/1.4788555
van Bezooijen, R. (1995). Sociocultural aspects of pitch
differences between Japanese and Dutch women.
Language and Speech, 38, 253–265.
Van Lancker, D., Kreiman, J., & Emmorey, K. (1985a).
Familiar voice recognition: Patterns and parameters
Part I: Recognition of backward voices. Journal of
Phonetics, 13, 19–38.
Van Lancker, D., Kreiman, J., & Emmorey, K. (1985b).
Familiar voice recognition: Patterns and parameters.
Part II: Recognition of rate-altered voices. Journal of
Phonetics, 13, 39–52.
Van Lancker, D. R., Kreiman, J., & Cummings, J.
(1989). Voice perception deficits: Neuroanatomical
correlates of phonagnosia. Journal of Clinical and
Experimental Neuropsychology, 11, 665–674. http://
dx.doi.org/10.1080/01688638908400923

Verhaegen, M., & Munro, S. (2004). Possible
preadaptations to speech: A preliminary comparative.
Human Evolution, 19, 53–70. http://dx.doi.org/
10.1007/BF02438909
Vukovic, J., Jones, B. C., Feinberg, D. R., Debruine, L. M.,
Smith, F. G., Welling, L. L., & Little, A. C. (2011).
Variation in perceptions of physical dominance
and trustworthiness predicts individual differences
in the effect of relationship context on women’s
The Voice
preferences for masculine pitch in men’s voices.
British Journal of Psychology, 102, 37–48. http://
dx.doi.org/10.1348/000712610X498750
Wernicke, C. (1974). Der aphasische
Symptomencomplex: Eine psychologische Studie
auf anatomischer Basis [The aphasic symptom
complex: A psychological study from an anatomical
basis]. Berlin, Germany: Springer. (Original work
published 1874)
305
 Chapter 12
Gesture
Erica A. Cartmill and Susan Goldin-Meadow
Human communicators move. Heads tilt, eyebrows
raise, hands wave, fingers point, bodies lean, faces
contort, and all synchronize with vibration of vocal
chords and the opening and closing of the mouth.
You might say that human communication is a tri-
umphant study in multimodality. However, what is
the point of all of this movement? Is it purposeful?
Is it communicative? Is it part of language? The
communicative utility of extralinguistic features of
language (communicative elements not governed
by the linguistic rules of the language) has been
acknowledged for thousands of years (see Kendon,
2004), but researchers are finally coming to a better
understanding of the ways in which the body is an
integral part of both language and thought.
WHAT IS GESTURE?
In this chapter, we focus on manual gesture and its
relationship to both spoken and signed languages.
We aim to illustrate what gesture is, but also what
it is not, addressing both the scope and limitations
of gesture as a communicative medium. We begin
by defining gesture and the ways in which it is dif-
ferent from signed languages. We discuss the range
of meanings that gesture can communicate and the
way in which gesture integrates with linguistic struc-
tures. We ask what happens when gesture becomes
the dominant communicative medium, and we
explore what is unique about gesture when it is com-
pared to other nonverbal forms of human commu-
nication and to the manual communication of other
species. We end by discussing gesture’s relationship
http://dx.doi.org/10.1037/14669-012
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
to cognition, and we raise the question of whether
gesture is really for communicating at all.
Gesture Is Not Sign Language
Sign languages (like gesture) are produced in the
manual modality, but (unlike gesture) sign lan-
guages display the same underlying structural fea-
tures as spoken languages (Klima & Bellugi, 1979;
Sandler & Lillo-Martin, 2006). Signed languages
are fully structured languages with phonological
(e.g., Stokoe, 1960), morphological (e.g., Klima &
Bellugi, 1979), and syntactic (e.g., Liddell, 1980)
rules. Some signs iconically represent things in the
world (meaning that the form of the sign is related
to the physical features of the gesture’s referent). For
example, the sign for “bird” in American Sign Lan-
guage is made by pinching the thumb and forefinger
together in front of the mouth like a bird’s beak.
However, many signs have no iconic elements what-
soever. Moreover, iconicity does not appear to play a
central role in guiding young children’s acquisition
of sign language (Bonvillian, Orlansky, & Novack,
1983; but see R. L. Thompson, Vinson, Woll, &
Vigliocco, 2012). When signs do have iconic ele-
ments, they are not holistic iconic representations
of entire events, and there are linguistic rules that
constrain the form of the signs; for example, rules
that determine whether events that co-occur in the
world can be expressed simultaneously within a
single sign. Although it would be easy enough to
indicate the manner of motion in a sign describing
a skate boarder moving in a circle, to be grammati-
cally correct, the American Sign Language signer
307
Cartmill and Goldin-Meadow
must produce separate, serially linked signs, one
for the manner (illustrating how the foot works the
board) and one for the path (illustrating the circling
movement; Supalla, 1990). Sign languages thus do
not always take advantage of the iconic potential
that the manual modality offers.
This separation of manner and path illustrates
one aspect of the compositionality of sign language
(i.e., it is structured and assembled on multiple
levels). Each sign language has several layers of
structure—rules governing the use of signs (mor-
phological structure), the use of subsign elements
(phonological structure), and the assembly of signs
into sentences (syntactic structure). It is important
to note that although these levels of structure are
shared across all signed and spoken languages, each
sign language follows its own particular rules about
the signs, elements, and combinations that are per-
missible in that language.
Sign languages do not depend on the verbal lan-
guages spoken in their communities (e.g., British
Sign Language and American Sign Language are dif-
ferent languages), but they do differ from country
to country or from region to region in the same way
spoken languages do. Many countries have stan-
dardized national sign languages, whereas others
have regional languages shared by the deaf indi-
viduals in a local community. Just as with spoken
languages, different sign languages are not mutually
intelligible (Klima & Bellugi, 1979).
Gesture shares a modality with signed languages;
they are both produced manually rather than
vocally. However, gesture lacks the regular struc-
tures and rules that are central to language. Unlike
sign language, gesture does not exhibit combinato-
rial properties and lacks syntactic rules. This lack
of structure may restrict gesture’s communicative
potential, but it does provide greater freedom for
individual variation.
Gesture Involves the Hands
Gesture is typically considered to be movement of
the hands and arms, but other parts of the body may
sometimes be used (e.g., shrugging the shoulders
to express uncertainty or nodding the head in affir-
mation). Sometimes the whole body can be used to
illustrate an action or event (e.g., when explaining
308
an unusual walk or touchdown dance), but these
whole-body gestures are not typical (see Chapter 15,
this handbook). The head and shoulders are also
often involved in gesture (e.g., head tilts can be used
to mark perspective shift during narrative speech;
McClave, 2000).
What Counts as Gesture?
In 1969, Ekman and Friesen attempted to identify
and categorize the kinds of nonverbal behavior pro-
duced during spontaneous communication. They
identified five categories (affect displays, regulators,
adaptors, emblems, and illustrators) that are still
used to frame the field of nonverbal communication
today. These categories are differentiated by form,
function, and communicative intent. They have
been further refined by McNeill (1992) and are also
described in Chapter 19 of this handbook.
According to Ekman and Friesen’s (1969) clas-
sification, affect displays convey internal emotional
states (e.g., frowning suddenly or walking with a
bounce in your step). Regulators maintain the give-
and-take between speakers during a spoken conver-
sation (e.g., leaning in to indicate a desire to speak
or raising the chin toward another person to cede
the floor to him or her). Adaptors (also called self-
adaptors) are routinized movements directed toward
one’s own body that have been maintained by habit
and are not produced with intent to communicate
(e.g., retucking hair behind one’s ear even when it
is already there). The last two categories—emblems
and illustrators—are what people typically visualize
when they think of gesture.
Emblems are gestures that have shared conven-
tional meanings. They are frequently called conven-
tional gestures (Cartmill, Demir, & Goldin-Meadow,
2012; McNeill, 1992, 2005). These gestures are
culturally specific and come with expectations about
how they should look and when they should be
used. Variation from these learned forms and uses
makes a gesture difficult to understand in much the
same way that variation in the pronunciation or use
of a word would lead to misunderstanding. It would
be difficult, for example, to interpret an “OK” sign if
only the pinky finger were extended, or a “thumbs
up” if the other fingers were not curled into the palm
or if the thumb pointed to the side rather than up.

Emblems, more than other types of gestures, are
like words or sentences in that they have specific,
shared meanings and, as just noted, can be pro­­
duced incorrectly. Speakers use them consciously
to communicate particular concepts, and they can
be produced either with or without accompanying
speech. For example, in the United States it would
be perfectly acceptable for a speaker to respond to a
yes/no question with a horizontal shake of the head
instead of the word “no.” However, emblems are
mainly culturally distinct. For example, in Bulgaria,
where the gesture for “no” is a vertical “head toss,”
shaking the head horizontally would not convey
the desired meaning (Kita, 2009; McClave, Kim,
Tamer, & Mileff, 2007). However, some emblematic
forms are shared across cultures within particular
regions of the world, and a few emblems appear to
be similar across multiple regions (­Matsumoto &
Hwang, 2013). The standards of form that are
applied to emblems make them comparable to
words and signs. However, unlike words or signs,
emblems are not assembled into sequences. Each
emblem has a specific, learned meaning (e.g.,
“OK,” “great,” “bye-bye,” “I don’t know,” “wait”),
but emblems are not combined with one another
according to structural rules.
Illustrators are movements that are produced
alongside speech and often illustrate the concepts
conveyed in speech. As such, they are referred to as
cospeech gestures (this is the term we adopt here)
or, sometimes, just gestures (McNeill, 1992). This
broad category of gestures includes most of what
is studied by gesture researchers and is the main
focus of our chapter. From this point on, we use the
general term gesture to refer to the category of “com-
municative” acts that includes both illustrators and
emblems. We use the term cospeech gesture to refer
to illustrators only.
Cospeech gestures differ from emblems in that
(as their name suggests) they are produced along
with speech, and their meanings depend on that
speech. Emblems can convey their meanings in
the absence of speech (at least to others who have
learned the same gesture; e.g., holding up a finger
to tell someone to wait for a moment while saying
nothing). In contrast, the meaning of a cospeech
gesture relies heavily on the spoken contexts in
Gesture
which it is produced. Someone might illustrate the
phrase “all the beads hit the floor and scattered” by
moving both hands slightly out to one side while
wiggling their fingers as if playing the piano. If the
same gesture were accompanied by the sentence
“I need to respond to that e-mail,” it would be
interpreted as typing. In other contexts, the same
movement might represent playing the piano or a
rainstorm.
Because cospeech gestures are produced in the
context of spoken communication, they are, in
this sense, deliberate (unlike adaptors). However,
people are not typically aware of the specific move-
ments they make when producing cospeech gestures
(i.e., illustrators), and thus these gestures are rarely
under conscious control. This combination of being
produced unconsciously but within a consciously
produced communicative act sets cospeech gesture
apart from other forms of nonverbal behavior and
makes it a powerful tool with which to ask questions
about the mind and intent of the gesturer. Gesture
can add information to speech by illustrating fea-
tures of objects (such as size or shape) or properties
of events (such as speed or path) that are not explic-
itly conveyed in speech. Taking gesture into account
when looking at language provides a more complete
understanding of the reasoning and intent of the
speaker. In this way, cospeech gesture can be said to
provide a “window on the mind” (Goldin-Meadow,
2003a; McNeill, 1992).
Types of Cospeech Gesture
Cospeech gesture can be further categorized accord-
ing to the presence of iconic, deictic, or emphatic ele-
ments (McNeill, 1992). These categories are defined
by the ways they convey meaning: Some employ
imagery to represent the world, either iconically or
metaphorically; others convey meaning by directing
attention to things in the environment or rhythmi-
cally highlighting parts of the accompanying speech.
Iconic gestures convey meaning by recreating
an aspect of their referent’s shape or movement.
These gestures represent physical objects or events
by mimicking an aspect of the shape, size, or move-
ment of the object or event. The hand can represent
a hand performing an action (e.g., a cupped hand
pushing an imaginary toy train along a curved
309
Cartmill and Goldin-Meadow

track), but it can also represent an object or entity
directly (e.g., a flat hand moving in a curved line to
represent the same train). The hand can also take on
a neutral handshape and serve only as a “tracer” in
space, tracing the outline of an object or the path of
an action (e.g., an extended index finger drawing a
curved line in the air to show the path of the train).
Figures 12.1A–12.1D provide further examples of
these different handshapes.
Metaphoric gestures represent abstract ideas or
concepts, but by illustrating the concepts with a
gesture, the concept is given physical characteris-
tics (see Figure 12.1E). The speech accompanying
these gestures may already contain a metaphor, or
the gesture may add a metaphoric element to the
speech. For example, a person might produce a
lifting gesture with the sentence, “I need to raise
my grade in that class,” or with the sentence, “I
need to improve my grade in that class.” In the
first case, the gesture is illustrating a metaphor
present in speech; in the second case, the gesture
is adding the metaphoric element by mapping the

FIGURE 12.1.  Examples of different types of cospeech gestures and different handshapes. Figure 12.1A
shows a 38-month-old boy depicting a bat flying through an iconic gesture. His arms are outstretched, and
he flaps them up and down as wings; his hand (and arm) represents a hand (and arm) acting in the world.
Figure 12.1B shows a 50-month-old girl producing an iconic gesture to depict a towel by gesturing as if she
were holding a towel and wrapping it around her body. Her hands represent hands acting on an invisible
object (the towel). Figure 12.1C shows a 22-month-old girl producing an iconic gesture of a spider; she
holds her fingers downward and wiggles them and, thus, uses her hand to represent another object (in this
case, the spider). Figure 12.1D shows a 38-month-old boy gesturing to the uncompleted edges of a puzzle;
he uses his finger to trace along the bottom and right edges of the puzzle frame, producing an iconic gesture
in which his hand serves as a pointer or tracer, outlining the space that the puzzle will fill. Figure 12.1E
shows an adult woman producing a metaphoric gesture while being interviewed about her acting career. She
gestures outward with her thumb while describing a project that someone wrote “years ago.” The gesture
is metaphoric because it attributes physical spatial features to the concept of time, situating the past event
to the left of her body. Figure 12.1F shows a 14-month-old boy producing a deictic gesture by pointing to a
framed picture on top of a table.

310

more abstract word “improve” to the act of physi-
cally lifting.
Deictic or indexical gestures direct attention
toward objects, people, events, or locations in the
surrounding environment (see Figure 12.1F). Point-
ing with the index finger is the paradigmatic deictic
gesture, but deictic gestures may be made with other
handshapes (e.g., a flat hand with the palm up or
to the side) or with other body parts (e.g., pointing
with the chin by lifting it in the desired direction;
Wilkins, 2003). Holding up or touching objects
to draw attention to them may also be considered
deictic gestures. The direction of the point gives
some indication as to its meaning, but, without
accompanying speech, it is often difficult to discern
the precise meaning of the point. This phenomenon
can be seen when observing young infants pointing.
Sometimes parents can infer the meaning immedi-
ately, but other times it will take many attempts to
guess the thing that the child has in mind.
Beat gestures are rhythmic movements of the
hands or head that correspond to, and serve to
highlight, the prosody of speech (Ekman & Friesen,
1969, referred to these gestures as rhythmic ges-
tures). Beat gestures do not have imagistic or indexi-
cal meaning, but they can segment and emphasize
elements in speech by moving during certain words.
For example, a person listing off a number of items
or events might “beat” the air rhythmically by bring-
ing the hand sharply downward with each element
in the list: “We need to get bread (beat), apples
(beat), and carrots (beat), and then we should go get
cheese (beat).” You see many of these gestures when
people are giving speeches (try looking for them in
politicians). They emphasize the speech they accom-
pany, sort of like a gestural highlighter.
These categories are useful in conceptualizing
the range of cospeech gestures, but they should not
be thought of as mutually exclusive. A gesture may
easily fall into more than one category. For example,
a person giving directions might trace the shape of
a bridge over a road (an iconic gesture) and also
produce the gesture in the direction of the bridge’s
location (adding a deictic element). Gestures can be
extremely complex, and understanding the meaning
of a gesture requires more than simply observing the
shape of its movement.
Gesture
Understanding the meaning of a gesture can
require shared knowledge of conventions (in the
case of emblems) or the significance of objects in the
environment (in the case of deictic gestures). How-
ever, most importantly, understanding a gesture’s
meaning requires understanding the speech that
accompanies the gesture and identifying the rela-
tionship between gesture and speech.
Gesture Relates to Speech
Speech and gesture are intimately entwined. More
than 90% of all gestures occur in the presence of
speech (McNeill, 1992). This close relationship
between gesture and speech emerges early in life and
is strengthened as children learn language. In adults,
the gesture–speech relationship is characterized by
both temporal and semantic integration (further dis-
cussed in the How Does Gesture Fit Into a Linguis-
tic System? section).
The temporal relationship between gesture and
speech is highly synchronous, and the alignment
of the two modalities can be seen by looking at the
moments of greatest intensity in both speech and
gesture. The movement phase of a speaker’s gesture
co-occurs with the point of peak prosodic empha-
sis in the accompanying clause in speech (Kendon,
1980; McClave, 1998). Importantly, it is not the case
that people synchronize speech with any manual
movement: Gestures show greater synchrony with
accompanying speech than manual actions do
(Church, Kelly, & Holcombe, 2014).
The semantic relationship between gesture
and speech can be defined by the degree to which
gesture conveys information that is not found in
speech. On one end of the spectrum, gesture can
complement the information in speech (essentially
duplicating the information in speech). On the
other end, gesture can supplement the informa-
tion in speech, conveying information that is not
found anywhere in that speech. Often, the relation-
ship between speech and gesture falls somewhere
in between, with gesture echoing the information
found in speech but providing some additional
details (e.g., about the size, shape, or location of
an object). This dual function is particularly clear
when gesture is used to disambiguate a referent in
speech—for example, when a point is produced
311
Cartmill and Goldin-Meadow

along with “that one” or “put it there.” Without the
gesture, both of these utterances would be under-
specified, providing insufficient information in
speech to be understood without some other clues
to meaning. Figure 12.2 illustrates gestures that
have a complementary, supplementary, or disambig-
uating relation to speech. The categories are more
fully described next.
Gesture can complement information in speech
by reinforcing size, shape, movement, path, or loca-
tion information conveyed in speech. The degree
of specificity in speech and overlap in meaning
between speech and gesture can vary. For example,
the utterance “there was a huge bird” could be
accompanied by flapping hands like a bird’s wings
(reinforcing the meaning “bird”). However, the
same flapping gesture could incorporate the mean-
ing “huge” by making the flapping motion larger
and using the whole arm (thus reinforcing the
meaning of both “huge” and “bird”).
Gesture can supplement information in speech
by adding information about size, shape, movement,
path, or location that is not expressed in speech.
For example, a child might request help in opening
a jar by pointing to it and saying “open.” By taking
into account both the spoken word (specifying the
requested action, opening) and the gesture (indicat-
ing the item to be opened), a complete imperative
can be seen, “open jar.” Supplementary relationships
between gesture and speech can indicate transitional
periods in the acquisition of language or learning.
This type of gesture–speech relationship is particu-
larly revealing in the language of young children
who are not yet combining words into sentences.
A child is very likely to produce his or her first
sentence-like meaning entirely in speech (“open jar”)
several months after he or she produces his or her
first sentence-like meaning in gesture plus speech
(point at jar + “open”; Goldin-Meadow & Butcher,
2003; Iverson & Goldin-Meadow, 2005). By observ-
ing children’s gesture–speech combinations, it is
possible to predict linguistic achievements before
they emerge entirely in speech. We discuss this
developmental phenomenon more extensively in the
How Does Gesture Fit Into a Linguistic System?
section.

FIGURE 12.2.  Examples of the three different types of gesture–speech relationships: complementary, supplemen-
tary, and disambiguating. Figure 12.2A shows a 50-month-old girl pointing to fish in a fish tank. She says “It’s a
guppy.” Her gesture indicates the same thing (the fish) that she is naming in speech, and thus her gesture comple-
ments the speech it accompanies. Figure 12.2B shows a mother using an iconic gesture to describe the lights on a fire
truck to her 14-month-old infant. She rotates her hands while saying “The fire truck goes woo woo woo woo.” She
identifies the noise that the fire truck makes in speech (“woo woo”), but the gesture provides information about the
lights on top of the truck flashing. This information appears nowhere in speech, and thus her gesture supplements
the speech it accompanies. Figure 12.2C shows a 46-month-old girl producing an iconic gesture in which she con-
veys the path traveled by an animal in a story. She is holding a marker, and although her handshape does not convey
any information about the animal, the movement of her gesture represents the trajectory of the action she describes.
She says “he went over there and over there” while sweeping her arm back and forth across her body. The terms
“here” and “there” are ambiguous when conveyed in speech alone. Her gesture thus disambiguates her speech by
clarifying which directions the character traveled.

312

Gesture can disambiguate information in speech
by specifying the referent of an underspecified
speech act. For example, the sentences, “I’ll have
two of those,” “It was right there,” and “It went
like this,” are ambiguous without an accompanying
gesture. Adding a deictic or iconic gesture makes
these sentences highly informative. Deictic words in
speech (here, there, this, that) are, in fact, frequently
accompanied by gesture. Even young children make
use of gesture to disambiguate their speech, and
children speaking a language that permits a great
deal of ambiguity (e.g., Turkish, which allows more
omission than English) use gesture to fully specify
their utterances (Demir, So, Özyürek, & Goldin-
Meadow, 2012). Gesture can thus give children
learning structurally different languages a way to
achieve comparable levels of specification while
adhering to the referential expressions dictated by
their language.
These categorical distinctions are useful in con-
ceptualizing the range of relationships that gesture
can hold to speech, but attributing a gesture–speech
relationship to a particular communicative act can
be problematic. This difficulty arises because a ges-
ture can complement or disambiguate speech while
at the same time adding information that is not
expressed in speech. In the huge bird example,
the gesture is adding information about the action of
the bird. However, it is not clear from the sentence
in isolation whether the bird was flying. The ges-
ture could simply be referencing a bird by means of
the stereotypical action associated with the animal.
It gets even more complicated if we allow for the
possibility that the movement indicates something
about the flying style of the bird (say, fast shallow
wing beats vs. long, slow ones). A speaker may not
vary this information intentionally but might never-
theless gesture differently to depict a flamingo and a
buzzard.
There are times when the information con-
veyed in gesture contradicts the information con-
veyed in speech, for example, pointing to the left
while saying, “then you take a right.” These are
true errors, and they are rare. Errors of this sort
should not be confused with what has come to
be called gesture–speech mismatches (Church &
Goldin-Meadow, 1986; Goldin-Meadow, 2003a).
Gesture
The gesture in a gesture–speech mismatch conveys
different information from the information conveyed
in speech, but that information can (in principle) be
integrated with the information in speech, although
the speaker may not yet have integrated the informa-
tion. For example, consider a child learning about
mathematical equivalence who is asked to solve the
problem, 5 + 9 + 3 = __ + 3, and to explain her
answer. After putting 17 in the blank (an incorrect
answer), the child explains that to get the answer
she “added the five, the nine, and the three” (an add-
to-equal-sign strategy in speech); at the same time,
she points to the 5, the 9, and the 3 on the left and to
the 3 on the right of the blank (an add-all-numbers
strategy in gesture). Note that to solve the problem
correctly, the child must recognize that the equal
sign breaks the equation into parts (reflected in
her speech) and that there is an additional number
on the right side of the equation (reflected in her
gestures). The child in this example does not yet
appear to have fully integrated these two pieces of
information, but her gestures, taken in conjunction
with her speech, suggest that she is aware of both
pieces (Alibali & Goldin-Meadow, 1993). Producing
gesture–speech mismatches of this sort indicates
that the learner is in a transitional period with
respect to a concept and, if given instruction in the
concept, is likely to make significant progress
(Goldin-Meadow & Alibali, 2013).
WHAT KINDS OF INFORMATION CAN BE
COMMUNICATED THROUGH GESTURE?
Gesture can be used to communicate a wide range
of meanings, but those meanings rely heavily on the
surrounding linguistic, social, and physical contexts.
The meaning of a gesture, particularly cospeech
gesture, is often not transparent without speech.
If you were to watch a video of person speaking
and gesturing with the sound turned off, you are
not likely to guess the person’s message (Krauss,
Morrel-Samuels, & Colasante, 1991; emblems are,
of course, an exception because, by definition, they
can be interpreted without speech). Even seemingly
transparent gestures, such as pointing, require a
close inspection of the accompanying speech to be
correctly interpreted.
313
Cartmill and Goldin-Meadow
Indexing the Environment
(Deictic Gestures)
Deictic gestures index the environment by directing
the attention of others. However, this process is not
always as straightforward as it seems. Deictic gesture
may refer to whole objects in the immediate envi-
ronment (e.g., “Can you hand me that?” used with
a point to a glass). However, these gestures can also
refer to parts, features, or properties of objects (e.g.,
pointing to the same glass while saying “It’s a little
dirty,” or “I think we need a new washer”). In all
these cases, the gesture is indexing an object pres-
ent in the immediate environment, but the specific
meaning of the gesture is made clear by the accom-
panying speech.
Surprisingly, deictic gestures can also be used to
refer to objects that are not present, either by point-
ing out their absence or by referring to an object that
has a salient relationship to a nonpresent object. For
example, while pointing to the glass, a person can
say, “Would you get me some more please?” In this
case, the gesture does not refer to the glass itself, but
to the absence of the liquid inside it. Similarly, at a
party, someone might point to the glass of a friend
who is out of the room and ask, “Does anyone know
where she went?” In this case, the glass is serving
as an anchor for a particular person because it is
associated with that person; the point to the glass is
referencing the absent individual. Points at a pres-
ent object to refer to a nonpresent object can also
be found in young children (Liszkowski, Schäfer,
Carpenter, & Tomasello, 2009; see also Butcher,
Mylander, & Goldin-Meadow, 1991). For example,
a child can point to a chair and say “daddy.” If
daddy is present, the gesture–speech combination
could be a request for daddy to sit in the chair. If,
however, daddy is not there, then the combination
could be a statement that it is “daddy’s chair.”
Referencing Shared Meanings (Emblems)
In that same way that a word or idiomatic expres-
sion requires both speaker and listener to have
preexisting knowledge of the expression, emblems
rely on a shared understanding of the relationship
between form and meaning. Because they require
prior knowledge of meaning, emblems function
like a code and can refer to an essentially unlimited
314
range of concepts. For example, emblems can refer
to concepts that correspond to single words (e.g.,
nodding to indicate “yes”) or to entire phrases (e.g.,
shrugging the shoulders to indicate “I don’t know”).
Emblems can bear an iconic relationship to the
things they symbolize (e.g., holding the palm against
the cheek and tilting the head to refer to sleeping),
but they more often have an arbitrary relationship
(e.g., scraping one index finger against the other to
scold someone).
Emblems vary between cultures in both form
and meaning (e.g., Americans cross their index and
middle finger as an emblem of good luck, whereas
Germans make a fist and tuck their thumb under
their index finger). Though emblems are typically
shared broadly within a culture, emblems may be
established in smaller groups (e.g., a baseball coach
developing a code to signal plays to the pitcher).
They may even emerge in particular families. Just
as families may invent unique words shared only
within the family (e.g., by adopting a young child’s
mispronunciation of a word as a slang term), so may
they develop local customs in their emblems.
Depicting Objects, Actions, and Events
(Iconic Gestures)
Iconic gestures capture an aspect of the objects or
actions they represent. For example, a speaker uses
two flat hands, palms facing each other, to indicate
the width of a container while saying, “It’s not very
wide,” or moves his hand across a table wiggling his
fingers while saying, “He crawled over.” Iconic ges-
tures also reveal information about the perspective
speakers take vis-à-vis the event they are describing
and can be produced from two different perspec-
tives: (a) In character viewpoint gestures, the ges-
ture portrays an event from the character’s point of
view (e.g., pumping the arms as though running to
describe a character who is moving quickly; moving
a closed-hand away from the torso to describe a char-
acter giving something away) and (b) in observer
viewpoint gestures, the gesture portrays the event
from the observer’s point of view (e.g., moving the
two fingers of an upside-down V-hand back and
forth representing the moving legs of a character in
a running event; moving an index finger up to repre-
sent the ascent of the character in a climbing event).

Depicting Abstract Thoughts (Metaphoric
Gestures)
Metaphoric gestures are defined by their relation-
ship to speech. They can have exactly the same
forms as iconic gestures, but they are considered
metaphoric because they refer to things that do not
have physical features. In principle, any abstract
concept may be depicted in gesture; it just needs
to activate a visuo-spatial schema (such as height,
weight, containment, or forward movement).
Even highly abstract concepts, such as “justice” or
“truth,” can be illustrated in gesture by tying them
to images of balance or containment. However,
without the accompanying speech, there would be
no way to differentiate a metaphoric gesture from an
iconic gesture.
HOW DOES GESTURE FIT INTO
A LINGUISTIC SYSTEM?
In the sections that follow, we describe the ways
in which gesture is an integral part of the human
linguistic system by drawing on evidence from stu­
dies of gesture production, gesture comprehension,
and gesture’s role during language development.
We also consider what happens when gesture is the
dominant communicative medium (e.g., in pro-
foundly deaf children who are not exposed to a sign
language, or during communication games in which
hearing participants are not allowed to speak).
Gesture Is an Integral Part of Language
Gesture is not a supplemental or secondary system
applied on top of speech. Rather, gesture and speech
together form an integrated linguistic system. Ges-
ture is part of the planning process of language, and
this deep integration reveals itself in the alignment
of both semantic and temporal features between
gesture and speech. Evidence from behavioral,
neurological, and developmental studies provides
support for the view that language is an integrated,
multimodal system.
Integration of gesture and speech during ­language
production.  Gesture is linked to spoken lan-
guage at every level of analysis. For example, at
the phonological level, producing hand gestures
Gesture
influences the voice spectra of the accompany-
ing speech for deictic gestures (Chieffi, Secchi, &
Gentilucci, 2009), emblems (Barbieri, Buonocore,
Volta, & Gentilucci, 2009; Bernardis & Gentilucci,
2006), and beat gestures (Krahmer & Swerts, 2007).
When phonological production breaks down, as in
stuttering or aphasia, gesture production stops as
well (Mayberry & Jaques, 2000; McNeill, Levy, &
Pedelty, 1990). The simultaneous disruption of
gesture during an arrest of speech provides further
evidence for the tight temporal synchrony between
gesture and speech.
At the lexical level, gesture can both reflect and
compensate for gaps in a speaker’s verbal lexicon.
For example, when speakers of English, Japanese,
and Turkish are asked to describe a scene in which
an animated figure swings on a rope, English speak-
ers overwhelmingly use the verb “swing” along
with an arced gesture (Kita & Özyürek, 2003). In
contrast, Japanese and Turkish speakers, who speak
languages that do not have single verbs that express
an arced trajectory, use generic motion verbs along
with the comparable gesture, that is, a straight ges-
ture (Kita & Özyürek, 2003). However, gesture can
also compensate for gaps in the speaker’s lexicon by
conveying information that is not encoded in the
accompanying speech. For example, complex shapes
that are difficult to describe in speech can be con-
veyed in gesture (Emmorey & Casey, 2001).
At the syntactic level, gestures are influenced
by the structural properties of the accompanying
speech. For example, English expresses manner
and path within the same clause, whereas Turk-
ish expresses the two in separate clauses. The
gestures that accompany manner and path con-
structions in these two languages display a parallel
structure—English speakers produce a single ges-
ture combining manner and path (a rolling move-
ment produced while moving the hand forward),
whereas Turkish speakers produce two separate
gestures (a rolling movement produced in place,
followed by a moving forward movement; Kita &
Özyürek, 2003; Kita et al., 2007).
Integration of gesture and speech during language
comprehension.  Listeners glean information from
both gesture and speech and seamlessly integrate
315
Cartmill and Goldin-Meadow
the two pieces of information. One strong piece of
evidence that listeners are truly integrating infor-
mation across modalities and not perceiving speech
and gesture separately comes from experimental
work showing that people will report in their
speech information that was conveyed only in ges-
ture (Cassell, McNeill, & McCullough, 1999). For
example, a person is told a story in which the nar-
rator says, “She whacks him one,” while producing
a punching gesture. When retelling the story, the
person says, “She punches Sylvester,” integrat-
ing information conveyed only in gesture into the
spoken account (see also Goldin-Meadow, Kim, &
Singer, 1999; Goldin-Meadow & Singer, 2003).
Being able to integrate information across gesture
and speech is a skill found early in development,
even in one-word speakers (Morford & Goldin-
Meadow, 1997).
Gesture can help listeners understand speech by
providing information that complements or elabo-
rates on that in speech. Listeners are more likely to
correctly perceive and recall information conveyed
in speech when it is accompanied by gesture that
complements the meanings in speech than when it
is accompanied by no gesture (Beattie & Shovelton,
1999, 2000; Graham & Argyle, 1975; McNeil,
­Alibali, & Evans, 2000; L. A. Thompson & ­Massaro,
1994). However, if the information conveyed in
gesture conflicts with the information conveyed in
speech, listeners may have greater difficulty process-
ing the speech (Kelly, Özyürek, & Maris, 2010).
When faced with information in gesture that dif-
fers from the information in speech, listeners are
less likely to understand the information in speech
than if there is no gesture at all (Goldin-Meadow &
Sandhofer, 1999; Kelly & Church, 1998; McNeil
et al., 2000). This decrement in the perception of
speech when gesture provides different information
provides further evidence that gesture and speech
form an integrated system. It further suggests that
this integration is automatic. If gesture–speech
integration was under voluntary control, listeners
could choose to ignore gesture and focus solely on
speech. However, experimental evidence suggests
that they cannot ignore gesture, even when explic-
itly instructed to do so (Kelly et al., 2010; Langton,
O’Malley, & Bruce, 1996).
316
The semantic integration between gesture and
speech can be seen on a neurological level. Gesture
affects the neural processing of language in that
speech receives a different response when it is per-
ceived with or without gesture. This effect has been
found using both event-related potentials (Kelly,
Kravitz, & Hopkins, 2004) and functional magnetic
resonance imaging designs (Dick, Goldin-Meadow,
Hasson, Skipper, & Small, 2009). Importantly, these
processing differences are not explained by the mere
presence or absence of movement during speech
comprehension. Noncommunicative actions (such
as scratching one’s chin) do not elicit the same
pattern of activation as semantically meaningful
gestures (Dick et al., 2009). One study found that
gestures and actions accompanying speech are pro-
cessed differently in Broca’s area; gestures showed a
pattern more consistent with semantic processing,
and actions showed a pattern more consistent with
mirror system activation (Skipper, Goldin-Meadow,
Nusbaum, & Small, 2007).
Furthermore, the specific relationship between
gesture and speech, rather than the presence of
gesture itself, is visible in the neural processing of
language. Using an event-related potential design,
Kelly et al. (2004) found that video stimuli in which
gestures and speech conveyed contradictory informa-
tion (gesturing “short” while saying “tall”) produced
a large negativity at 400 ms after stimulus presenta-
tion (the so-called N400 effect indicating semantic
distance between linguistic items). Interestingly,
gestures conveying information that is different
from, but integratable with, information conveyed
in speech (gesturing “thin” while saying “tall” to
describe a tall, thin container; i.e., a gesture–speech
mismatch) are processed no differently at this stage
from gestures that convey the same information
as speech (gesturing “tall” while saying “tall”; i.e.,
a gesture–speech match). Neither one produces a
large negativity at 400 ms; that is, neither one is rec-
ognized as a semantic anomaly (Kelly et al., 2004).
It is important to note, however, that at early
stages of sensory/phonological processing (P1–N1
and P2), speech accompanied by a mismatching
gesture (e.g., gesturing “thin” while saying “tall”) is
processed differently from speech accompanied by a
matching gesture (gesturing “tall” while saying “tall”).

Thus, information conveyed in gesture that is dif-
ferent from, but has the potential to be integrated
with, information conveyed in speech is noted at
early stages of processing but not at later higher
level stages. Neurological studies of gesture are still
in their infancy, but further work in this area holds
great promise for untangling the precise relationship
between gesture and speech in perceiving and inter-
preting meaning in language.
Integration Emerges During Typical
Language Development
Gesture plays a particularly important role dur-
ing early language development. Children begin to
gesture before they can talk, and during the early
years of language development, gesture provides
children with a means of supplementing and modi-
fying their spoken language. By using gestures when
they cannot yet produce words, and by combining
words and gestures when their spoken repertoires
are limited, children increase their communicative
potential and extend beyond the proficiency of their
speech.
Onset of gesture–speech integration.  Children
typically begin to gesture between 8 and 12 months
of age (Bates, 1976; Bates, Benigni, Bretherton,
Camaioni, & Volterra, 1979). These early gestures
are mainly deictic points and hold-ups (holding up
an object to draw attention to it). Deictic gestures
are grounded in the physical environment because
they have meaning only by directing others’ atten-
tion to objects, events, or locations. However, just
like adults, young children can use deictic gestures
to refer to absent objects (Butcher et al., 1991;
Liszkowski et al., 2009). Along with points and
hold-ups, very young children begin to use the con-
ventional gestures (emblems) that are common to
their culture (Guidetti, 2002). These include ges-
tures such as side-to-side head shakes used to indi-
cate “no,” and hands at shoulder height with palms
facing up used to indicate “I don’t know.” Iconic
gestures are rare in 1-year-old children, but they do
occur, and children differ in how frequently they
produce them (Acredolo & Goodwyn, 1988). Beat
and metaphoric gestures, however, do not appear
until much later in development (McNeill, 1992).
Gesture
Once children begin to communicate in gesture,
they begin the process of integrating their visual
and vocal channels. During the period when chil-
dren are using gesture but have not yet acquired
spoken words, children frequently combine their
gestures with nonword vocalizations (Iverson &
Thelen, 1999). However, once children acquire
their first spoken words, they do not immediately
combine those words with gesture. In the earliest
stages of speech, children produce either gesture or
speech alone—rarely combining the two (Butcher &
Goldin-Meadow, 2000).
During the one-word period (where children
are using single words but not yet combining those
words into sentences), the relationship between
gesture and speech changes (Goldin-Meadow,
2006). At the beginning of this period, children
are using more gestures than words and are not
habitually combining words with gestures. On
the rare occasions when children of this age do
combine a word—or more likely, a meaningless
vocalization—with a gesture, the two are not tempo-
rally synchronized in an adult-like manner (i.e., the
word or vocalization is not aligned with the stroke
or peak of the gesture). However, there comes a
critical point in this one-word period where children
begin to combine words with gestures in earnest
and to synchronize their production of gesture and
speech in those combinations (Butcher & Goldin-
Meadow, 2000). These features—semantic integra-
tion and temporal synchrony—characterize the
relationship between gesture and speech in adults
(McNeill, 1992).
The onset of gesture–speech combinations
heralds a new phase in children’s linguistic devel-
opment and dramatically expands the scope of
children’s communicative systems. Children’s early
gesture–speech combinations are complementary in
that they reference the same object or event in both
modalities (e.g., point to shoe + “shoe”; Capirci,
Iverson, Pizzuto, & Volterra, 1996; de Laguna,
1927; Greenfield & Smith, 1976; Leopold, 1949).
Children’s gestures can also add information by
conveying an idea that is found nowhere in speech
(e.g., point to shoe + “gimme”). These additive or
supplementary combinations emerge only after (or
at the same time as) complementary combinations
317
Cartmill and Goldin-Meadow
(Goldin-Meadow & Butcher, 2003; Iverson &
Goldin-Meadow, 2005). Supplementary combina-
tions thus do not appear until after gesture and
speech have become temporally synchronized
(Goldin-Meadow & Butcher, 2003). The fact that
supplementary gesture–speech combinations (com-
binations in which gesture and speech convey dif-
ferent, but potentially integratable, information) do
not appear until after gesture and speech achieve
temporal and semantic integration provides further
evidence that gesture and speech are part of a single,
integrated system, rather than two separate commu-
nicative systems.
Gesture precedes and predicts structures in
speech.  Throughout early language development,
linguistic phenomena tend to appear in gesture
before they emerge in speech. The deictic gestures
and emblems that children produce before they can
speak are relatively simple, but they mark the onset
of children’s linguistic development. Points and
hold-ups, in particular, indicate a growing desire to
communicate about things and presage the acquisi-
tion of verbal labels for objects. At this early stage
of language development (when children are learn-
ing their first words and building a vocabulary),
the number of different meanings children com-
municate in their gestures predicts the total number
of spoken words they will acquire in the next few
years (Rowe & Goldin-Meadow, 2009a). Indeed,
the gestures children produce in the early stages
of language learning have reliably been found to
foreshadow subsequent vocabulary development
(Bavin et al., 2008; Goodwyn & Acredolo, 1993;
Rowe & Goldin-Meadow, 2009a, 2009b; Rowe,
Özçali kan, & Goldin-Meadow, 2008). For exam-
ple, a child’s early deictic gestures reliably predict
which nouns are likely to enter that child’s spoken
vocabulary in the next 3 months (Iverson & Goldin-
Meadow, 2005).
New linguistic constructions are also foreshad-
owed in gesture. By combining a gesture with a
word, children are able to convey two different ideas
within a single communicative act (e.g., pointing at
a cup while saying “gimme”). Importantly, the age
when a child first produces these supplementary
gesture–speech combinations predicts the age when
318
that child will produce his or her first two-word
utterance (e.g., “gimme cup”; Goldin-Meadow &
Butcher, 2003; Iverson & Goldin-Meadow, 2005).
Gesture continues to forecast children’s verbal mile-
stones beyond the transition from one-word to two-
word speech. For example, children produce their
first complex sentence containing two predicates
in gesture and speech (e.g., “I like it,” said while
producing an “eat” gesture) several months before
producing their first complex sentence entirely in
speech (“I like eating it”; Özçali kan & Goldin-
Meadow, 2005).
What Happens When Gesture Becomes
the Dominant Linguistic System?
We have seen that gesture assumes a holistic form
when it is used along with speech. However, what
happens when gesture is called upon to replace
speech and thus fulfill all of the functions typically
served by speech?
Homesign and emerging sign languages.  We
know that the manual modality can assume linguis-
tic properties—as described earlier, sign languages
of the deaf are segmented and combinatorial in form
(Klima & Bellugi, 1979; Sandler & Lillo-Martin,
2006), as are spoken languages. However, what
would happen if a child was not exposed to a con-
ventional sign language and had only gesture with
which to communicate?
Deaf children born to deaf parents learn their par-
ents’ sign language as naturally as hearing children
learn spoken language from their hearing parents
(Newport & Meier, 1985). However, most deaf chil-
dren are born, not to deaf parents, but to hearing par-
ents who do not know a sign language and want their
child to learn to speak. Unfortunately, most children
with profound hearing losses are unable to learn the
spoken language that surrounds them, even with
hearing aids and intensive instruction. In addition,
they frequently do not have access to a sign language
model. Despite their lack of an accessible model for
language, deaf children under these circumstances
communicate with the hearing people in their worlds
and use gestures, called homesign, to do so.
Homesign is characterized by many, although
not all, of the properties found in natural languages

(Goldin-Meadow, 2003b). For example, home-
signers’ gestures form a lexicon, and these lexical
items are composed of parts, comparable in struc-
ture to a morphological system (Goldin-Meadow,
Mylander, & Butcher, 1995; Goldin-Meadow,
Mylander, & Franklin, 2007). Moreover, the lexi-
cal items combine to form structured sentences,
comparable in structure to a syntactic system
(Feldman, Goldin-Meadow, & Gleitman, 1978;
Goldin-Meadow & Mylander, 1984, 1998). In addi-
tion, homesigners use gestural lexical markers that
modulate the meanings of their gesture sentences
(negation and questions; Franklin, Giannakidou, &
Goldin-Meadow, 2011) and grammatical categories
(nouns, verbs, and adjectives; Goldin-Meadow,
Butcher, Mylander, & Dodge, 1994). Homesigners
display hierarchical structure in their sentences by
building structure around the nominal constituent
(Hunsicker & Goldin-Meadow, 2012) or by adding
a second proposition to create a complex sentence
(Goldin-Meadow, 1982). Finally, homesigners use
their gestures not only to make requests of others
but also to comment on the present and nonpresent
(Butcher et al., 1991; Morford & Goldin-Meadow,
1997), to make generic statements about classes of
objects (Goldin-Meadow, Gelman, & Mylander,
2005), to tell stories about real and imagined events
(Morford & Goldin-Meadow, 1997; Van Deusen-
Phillips, Goldin-Meadow, & Miller, 2001), to talk
to themselves (Goldin-Meadow, 2003b), and to talk
about language (Goldin-Meadow, 1993)—that is, to
serve the typical functions that all languages serve,
signed or spoken.
In countries such as the United States, child
homesigners are likely to learn a conventional sign
language at some later point in their lives, often
around adolescence. However, in other countries
(Nicaragua is a good example), many homesigners
are never integrated into the deaf community and
continue to use their gesture systems with the hear-
ing people who surround them as their sole means
of communication. Analyses of adult homesigners in
Nicaragua have uncovered linguistic structures that
may (or may not) turn out to go beyond the struc-
tures found in child homesigners: the grammati-
cal category subject (Coppola & Newport, 2005),
pointing devices representing locations versus
Gesture
nominals (Coppola & Senghas, 2010), morphopho-
nological finger complexity patterns (Brentari,
Coppola, Mazzoni, & Goldin-Meadow, 2012), and
morphological devices that mark number (Coppola,
Spaepen, & Goldin-Meadow, 2013). By contrasting
the linguistic systems constructed by child and adult
homesigners, the impact that cognitive and social
maturity has on language can be seen.
We can also examine gesture as it continues on
the path toward becoming a fully established lan-
guage. In the late 1970s, deaf individuals in Nica-
ragua (who were likely to have been homesigners)
came together for the first time and began to fashion
a shared communication system, which has come
to be called Nicaraguan Sign Language (NSL; Kegl,
­Senghas, & Coppola, 1999; Senghas & Coppola,
2001). By contrasting the linguistic systems con-
structed by adult homesigners in Nicaragua with the
structures used by the first cohort of NSL signers,
the impact that a community of users has on lan-
guage can be seen.
However, NSL has not stopped growing. Every
year, new students enter the school and learn to
sign among their peers. This second cohort of sign-
ers has as its input the sign system developed by the
first cohort and, interestingly, changes that input
so that the product contains increasingly complex
linguistic structure (e.g., Senghas, 2003). The mem-
bers of the second cohort, in a sense, stand on the
shoulders of the first cohort and can therefore take
the transformation process one step further. By con-
trasting the linguistic systems developed by the first
and second cohorts of NSL, the impact that passing
a language through a new generation of learners
has on language structure can be seen. Once learn-
ers are exposed to a system that contains linguistic
structure (i.e., Cohort 2 and beyond), the processes
of language change may be identical to the processes
studied in historical linguistics. One interesting
question is whether the changes seen in NSL in its
earliest stages are of the same type and magnitude
as the changes that occur in mature languages over
historical time.
Cospeech gesture versus silent gesture.  A defining
feature of homesign is that it is not shared in the
way that conventional communication systems
319
Cartmill and Goldin-Meadow
are shared. Deaf homesigners produce gestures to
communicate with the hearing individuals in their
homes. However, the hearing individuals, particu-
larly hearing parents who are committed to teaching
their children to talk and thus to oral education, use
speech back. As a result, when the children’s parents
gesture, those gestures are produced along with
speech and, as we have shown, form an integrated
system with that speech. The parents’ cospeech ges-
tures are thus not free to take on the properties of
homesign, and, indeed, the structures found in
children’s homesigns cannot be traced back to the
spontaneous gestures that the children’s hearing par-
ents produce while talking to them (Goldin-Meadow
et al., 1994, 1995; Goldin-Meadow & Mylander,
1983, 1984). Homesigners see the global and
unsegmented gestures that their parents produce.
However, when gesturing themselves, they use ges-
tures that are characterized by segmentation and
linearization.
Cospeech gestures thus do not assume the lin-
guistic properties found in homesign. However, what
would happen if hearing speakers were asked to
abandon speech and to create a manual communica-
tion system on the spot? Would that system contain
the linguistic properties found in homesign? Exam-
ining the gestures that hearing speakers produce
when requested to communicate without speech
allows us to explore the robustness of linguistic con-
structions created online in the manual modality.
Hearing gesturers asked to gesture without
speaking are able to construct some properties of
language with their hands. For example, the order
of the gestures they construct on the spot indi-
cates who does what to whom (Gershkoff-Stowe &
Goldin-Meadow, 2002; Goldin-Meadow, McNeill, &
Singleton, 1996). However, hearing gesturers do
not display other linguistic properties found in
established sign languages and even in homesign
(Goldin-Meadow, 2015). For example, they do not
use consistent form-meaning pairings akin to mor-
phemes (Singleton, Morford, & Goldin-Meadow,
1993), and they do not use the same finger com-
plexity patterns that established sign languages and
homesign display (Brentari et al., 2012).
Interestingly, the gestures that hearing speak-
ers construct on the spot without speech do not
320
appear to be derived from their spoken language.
When hearing speakers of four different languages
(English, Spanish, Chinese, Turkish) are asked to
describe animated events using their hands and
no speech, they abandon the order typical of their
respective spoken languages and produce gestures
that conform to the same order—agent, object,
action (e.g., captain–pail–swings; Goldin-Meadow,
So, Özyürek, & Mylander, 2008). This order is also
found when hearing speakers of these four lan-
guages perform a noncommunicative, nongestural
task (Goldin-Meadow et al., 2008). Recent work on
English, Turkish, and Italian speakers has replicated
this finding in hearing gesturers but has found that
gesturers move away from the agent–object–action
order when asked to describe reversible events
involving two animates (“girl pulled man”; Meir,
Lifshitz, Ilkbasaran, & Padden, 2010) and when
asked to describe more complex events (“man tells
child that girl catches fish”; Langus & Nespor,
2010). Studies of hearing gesturers give researchers
the opportunity to manipulate conditions that have
the potential to affect communication and to then
observe the effect of those conditions on the
structure of the emerging language.
WHAT IS UNIQUE ABOUT GESTURE AS
A COMMUNICATIVE MEDIUM?
In the sections that follow, we compare gesture
to different kinds of human communication and
behavior (speech, sign language, vocal cues, and
manual actions) and explore how it differs from
these other ways of communicating and interacting.
We also ask whether human gesture is unique in
any ways from the gestures and communicative
body movements of other animals.
Less Digital Than Speech (or Sign)
Gesture conveys meaning in a somewhat different
way than speech (or sign language) does. Speech
is discrete and combinatorial: Words are made up
of meaningful subunits and are, in turn, combined
into meaningful sequences according to predefined
rules. Gesture is more holistic, and gestures can
combine with speech or blend into other gestures in
flexible ways (though gestures are rarely combined

with other gestures). Because of this difference, ges-
ture is a relatively analog form of communication,
whereas speech is relatively digital. Gesture is not
constrained by the rules and conventionally defined
forms that govern speech and thus lacks the expres-
sive power that comes with combinatoriality and
syntax.
However, gesture is spatial and imagistic in a way
that speech is not. This property means that gesture
can be a powerful tool for mapping concepts onto
spatial representations or for embodying mental
representations. Even ideas that are not inherently
spatial can be described in gesture. For example,
when asked to reason about moral dilemmas, speak-
ers of all ages use gestures that reveal whether they
are reasoning from one person’s perspective (e.g.,
the speaker holds both hands out and curls the
fingers in as though grasping an object, indicating
the acquisitiveness of a single character) or from
multiple perspectives (e.g., the speaker lays out the
viewpoint of one character in her right hand and the
viewpoint of a second character in her left hand; she
then moves the two viewpoints together and apart,
indicating that the two views are incompatible).
Interestingly, if children are told to gesture when
explaining their moral reasoning immediately before
receiving a lesson on moral dilemmas, they produce
significantly more multiple-perspective responses
in speech after the lesson than children told not to
gesture, or than children given no instructions in
how to use their hands (Beaudoin-Ryan & Goldin-
Meadow, 2014). Because it is so tightly tied to space,
gesture allows speakers to literally take one perspec-
tive on one hand and another perspective “on the
other hand.” Doing so may allow speakers to make
use of spatial learning mechanisms (Newcombe,
2010) that they would not have used had they not
gestured.
How Does Gesture Differ From Other
Nonverbal Cues?
In the sections that follow, we compare gesture to
other nonverbal behaviors like qualities of the voice
and physical action. We describe the characteris-
tics that gesture shares with these other behaviors
and identify some of the differences that set gesture
apart. We also compare human gesture to gestures
Gesture
produced by nonhuman primates and identify the
ways in which human gesture stands out.
Voice.  Like gesture, the voice can convey extra-
linguistic information that complements or adds to
the semantic content of speech. Vocal features such
as pitch, loudness, and formant frequency convey
information about the speaker’s physical character-
istics and emotional state. Characteristics such as
size, age, and individual identity are all marked in
acoustic parameters of the voice (e.g., Harnsberger,
Shrivastav, Brown, Rothman, & Hollien, 2008; Sell
et al., 2010). These features are what allow listeners
to identify speakers on the phone. Emotion is also
conveyed in the voice, as is made clear by imagin-
ing the different emotions that can be conveyed by
changing the way you might utter the phrase “I’m
fine” (see also Chapter 11, this handbook).
Much of the extralinguistic information carried
in the voice marks properties of the speaker rather
than the language itself. However, in some cases,
vocal features contribute information that is seman-
tically related to the information conveyed in the
words. In these cases, the vocal features display a
nonarbitrary relationship to the semantic content of
the speech. This kind of vocal modulation (deemed
acoustic analog expression by Shintel, Nusbaum, &
Okrent, 2006) resembles the way iconic gesture
conveys meaning and might be thought of as a type
of vocal gesture. Much like gesture, vocal features
may have varying relationships to the content of
speech. The voice may complement the idea con-
veyed in speech—for example, by lowering the
pitch of the voice when saying “It’s going down”
(Perlman, 2010; Shintel et al., 2006). However,
the voice may also add information to speech—for
example, by slowing down the rate of speech while
saying “It’s going down” if the object is descending
slowly (Shintel et al., 2006). Importantly, listeners
are able to successfully integrate the information in
these supplementary relationships—they correctly
interpret the earlier sentence as involving a slowly
descending object (Shintel & Nusbaum, 2007).
While the modulation of vocal features during
speech has much in common with gesture–speech
combinations, the use of a single modality for both
linguistic and extralinguistic content may restrict
321
Cartmill and Goldin-Meadow
the range of possible meanings conveyed uniquely
in the features of the voice. The study of vocal ges-
ture is still in its infancy, but hopefully the com-
ing years will see increased research in this area. It
would be particularly useful to compare the use of
vocal gesture during speech to the use of manual
gesture during sign language (see Goldin-Meadow,
Shield, Lenzen, Herzig, & Padden, 2012). Compari-
sons of this sort have the potential to provide insight
into how linguistic and gestural systems are shaped
by the auditory versus visual modality and by con-
straining communication to a single modality.
Action.  Gesture is similar to action in that it
involves physical movement (see also Chapter 15,
this handbook). Iconic or metaphoric gestures may
closely resemble actions (e.g., turning an imaginary
key while saying “lock it”). However, gesture differs
from action in that it is representational—physically
twisting a key locks the door; gesturing the twisting
motion does not. As a result, unlike action, gesture
is not tied to the affordances of the physical envi-
ronment. A gesture representing an action can be
performed in the location where the action would
be performed, but it can also be displaced and per-
formed in the absence of any physical objects. Take,
for example, the act of turning the crank on a hand-
mixer. A parent teaching a child to use the mixer
might use an action to demonstrate to the child how
to operate the device. In this case, the parent might
perform the action slowly, turning the crank so that
the child can see how the act is done. If however,
the parent was to convey the same information
using gesture, the range of possibilities is greatly
increased. The parent could gesture a facsimile of
the action with the hand held near the handle of
the mixer. Or the parent could produce the same
handle-turning gesture farther away from the mixer
(perhaps while the child was holding it) or even in
another room to refer to the mixer in its absence.
The parent might also use a more abstract represen-
tation of the movement, drawing circles with a fin-
ger to highlight the path of the handle as it rotated.
This flexibility in form and the ability to distance
movement from the affordances of physical objects
clearly differentiates even very action-like gestures
from action itself.
322
Gesture and action also affect mental represen-
tation of events in different ways. Gesturing about
performing an action encodes features of the action
in greater detail than performing the action itself
(Beilock & Goldin-Meadow, 2010). When using
a gesture to represent an action on an object (say,
turning a crank), the gesturer performs an action
on an “invisible” object. To do this, the gesturer has
to form a clear mental representation of the object.
When performing the same action on an object, the
actor does not need to retain a representation of the
object in his or her memory because he or she can
offload the features and affordances of the object
onto the physical environment (Cartmill, Beilock, &
Goldin-Meadow, 2012). Even when gesture closely
resembles the movements of the action it represents,
gesturing about an event has a stronger effect on the
mental representation of the action involved than
performing the action again (Goldin-Meadow &
Beilock, 2010).
Because gesture does not depend on the affor-
dances of the objects and events it represents, it
has the potential to affect learning differently from
the way action affects learning. Novack, Congdon,
Hemani-Lopez, and Goldin-Meadow (2014) asked
whether gesturing promotes learning because it is
itself a physical action or because it uses physical
action to represent abstract ideas. To address this
question, they taught third-grade children a strat-
egy for solving mathematical equivalence problems
that was instantiated in one of three ways: (a) in the
physical action children performed on objects, (b) in
a concrete gesture miming that action, or (c) in an
abstract gesture. All three types of hand movements
helped children learn how to solve the problems
on which they were trained. However, only gesture
led to success on problems that required general-
izing the knowledge gained. The results suggest that
gesture promotes transfer of knowledge better than
action and that the beneficial effects gesture has on
learning may reside in the features that differentiate
it from action.
Is Gestural Communication Unique
to Humans?
Humans are not unique in using manual gestures to
communicate, but humans may be unique in their

ability to exploit gesture’s vast representational
potential. Other primates use manual gestures, and
their gestures share some properties with human
gesture. However, their gestures do not exhibit the
iconic, imagistic properties that make human ges-
ture such a powerful representational medium.
Like humans, great apes (our closest living rela-
tives) use gestures to communicate with one another
in seemingly intentional ways, and their gestures
appear to be meaningful (Call & Tomasello, 2007;
Cartmill & Maestripieri, 2012). Great apes also take
the gaze of their partner into account when gestur-
ing; they use visual gestures more often when others
are looking, but when a potential partner is looking
away, apes will switch to audible gestures (like clap-
ping) or move to a location where they can be seen
(Call & Tomasello, 2007; Liebal, Call, Tomasello, &
Pika, 2004; Poss, Kuhar, Stoinski, & Hopkins,
2006). Great apes are also able to use deictic ges-
tures (such as pointing), but they primarily produce
these gestures when communicating with humans
in captive conditions and not when communicating
with other apes in their natural environments (Leav-
ens, 2004; Leavens, Hopkins, & Bard, 1996).
Ape gestures differ from human gestures in sev-
eral notable ways. First, they are not temporally
synchronized with vocalization in the way human
gestures are synchronized with speech. Second, they
are not richly iconic or representational. Some ape
gestures have been described as iconic (e.g., one
ape swung its arm in the direction it wished another
to go; Savage-Rumbaugh, Wilkerson, & Bakeman,
1977; Tanner & Byrne, 1996), but the degree of ico-
nicity in these gestures is debated (see Cartmill,
Beilock, & Goldin-Meadow, 2012). Third, ape ges-
tures seem to have only imperative meanings (e.g.,
move away, come here, gimme that), whereas human
gesture can convey both imperative and declarative
meanings (Tomasello & Camaioni, 1997). Early in
development, human children start out producing
primarily imperative communicative gestures. For
example, a young infant reaches for or points to
things that he or she wants. However, within the
first 12 months of life, children begin to communi-
cate declaratively—pointing out things because they
want to share the experience with others, not
because they want to obtain an item (Tomasello,
Gesture
Carpenter, & Liszkowski, 2007). This development
can be easily observed if you are around a parent
with a 1-year-old child. Typically, the child will
point to something in the environment (such as a
bird), and the parent will respond by saying some-
thing such as, “Yes, that’s a bird. Isn’t it pretty?”
This exchange will likely satisfy both participants.
To appreciate the difference between imperative and
declarative pointing, imagine a different scene in
which a child points to a favorite toy that has fallen
out of his or her stroller. If the parent responds
with “Yes, it fell down,” it is doubtful that the child
would be content with the response. The child in
the second example is using a point to request a
specific action from the parent; the child in the first
example is using a point to initiate an affiliative
social interaction with the parent.
Finally, human gesture is unique because it is
part of human language. It displays greater flexi­
bility and representational features than the com-
munication systems of other animals. Although
human gesture does not display the linguistic
compositionality and syntactic structures of spoken
language, it is semantically and temporally inte-
grated with speech. Gesture is also flexible in how
it conveys meaning. It can refer to something in the
environment or use something in the environment
to refer to something else (deictic gesture). It can
rely on learned forms and culturally shared mean-
ings (emblems). It can use rhythm to emphasize
concepts in speech (beat gesture). Or, it can convey
meaning imagistically, through iconic similarity to
physical referents (iconic gesture) or by aligning
abstract concepts with spatial schemas (metaphoric
gesture).
IS GESTURE FOR COMMUNICATION
AT ALL?
In this final section, we turn to the question of
whether gesture exists primarily as a tool for com-
munication or for thought. It is clear that speakers
produce meaningful gestures that can comple-
ment and supplement the information conveyed in
speech, and that listeners pick up on the informa-
tion in gesture, integrating across modalities to gain
a fuller understanding of the speaker’s meaning.
323
Cartmill and Goldin-Meadow
It is unclear, however, the degree to which a speaker’s
gestures are intended to communicate versus pro-
duced during speech to aid in the organization of
thought and the production of language. It is possi-
ble that gesture is able to fulfill both communicative
and cognitive functions simultaneously.
There is evidence that speakers produce at least
some of their gestures for the purpose of commu-
nicating with the listener, in other words, for the
listener. If gestures were produced solely to help
with speech planning and production, then gesture
rates should decline when speakers repeat the same
message to different listeners. However, speakers
experience no such decrease in gesturing (Jacobs &
Garnham, 2007). Similarly, if gesture is not pro-
duced for the listener, then gesturing should not
change when the speaker and listener cannot see
each other. However, if a screen is placed between
speaker and listener, the rate of gesturing declines
(Alibali, Heath, & Myers, 2001; Mol, Krahmer,
Maes, & Swerts, 2011). In short, people gesture
more when they can be seen. These findings suggest
that gesture is used for the purpose of communicat-
ing with others. However, other studies demonstrate
the ways that gesture can also benefit the speaker.
Gesturing for Thinking
People gesture in many situations where there is no
obvious benefit to a listener (or perhaps no listener
at all). One common example is the habitual use
of gesture while talking on the phone. Even more
convincing, congenitally blind speakers, who have
never seen another person gesture, produce gestures
when they speak, even when speaking to blind lis-
teners (Iverson & Goldin-Meadow, 1998). The fact
that congenitally blind speakers gesture to blind
listeners suggests that gesture may be playing a role
for speakers as well as listeners. In this section, we
consider a number of ways in which gesturing has
been shown to influence how speakers think.
Facilitating Lexical Access
Gesturing can support the planning and production
of speech by facilitating lexical retrieval—by help-
ing speakers “find” words (Rauscher, Krauss, &
Chen, 1996). For example, if you were asking a
friend for a corkscrew but could not remember
324
the word, you might produce a gesture that rep-
resented the corkscrew while you were searching
for the right word. Support for the lexical access
theory comes from the observation that speakers
are particularly likely to gesture during unrehearsed
speech (Chawla & Krauss, 1994) or when they use
words that are unpredictable given the surrounding
context (Beattie & Shovelton, 2000). When lexical
access is impaired (during experimental manipula-
tion or in patients with aphasia), rates of gesturing
increase. Finally, experimental studies manipulat-
ing gesture rate have demonstrated that both adults
and children are more successful at finding correct
words (as in the corkscrew example) when they are
allowed to gesture than when they are not allowed
to gesture (Frick-Horbury & Guttentag, 1998; Pine,
Bird, & Kirk, 2007). These findings support the
theory that gesturing can aid lexical access.
Reducing Demands on Conceptualization
Speakers gesture on problems that are conceptually
difficult, even when there are no lexical demands
(Alibali, Kita, & Young, 2000; Hostetter, Alibali, &
Kita, 2007; Kita & Davies, 2009; Melinger & Kita,
2007), suggesting that gesture can do more than
facilitate lexical access. For example, when adults
are asked to describe dot patterns, they gesture more
when talking about patterns that do not have lines
connecting the dots (patterns that are more difficult
to conceptualize) than patterns that do have lines
(Hostetter et al., 2007). As a second example, chil-
dren who are asked to solve Piagetian conservation
problems (problems that require conceptualiza-
tion) gesture more than when they are simply asked
to describe the materials used in the conservation
problems (Alibali et al., 2000).
However, we need to be cautious in interpreting
these results. In all of these studies, conceptualiza-
tion difficulty and gesturing go hand-in-hand (the
more conceptually difficult a problem, the more
gesture). However, to be certain that gesturing plays
a causal role in reducing conceptualization demands
(as opposed to merely reflecting those demands),
researchers need to manipulate gesture and demon-
strate that the manipulation has an impact on con-
ceptualization demands. Studies of this type have
not yet been done.

Reducing Demands on Working Memory
Studies have been done that experimentally manipu-
late gesture and explore the impact of that manipu-
lation on working memory. Adults and children
were asked to remember an unrelated list of items
while explaining how they solved a math problem.
One group was allowed to gesture freely during their
explanations; the other group was prevented from
gesturing. Speakers recalled more items (and thus
maintained more items in verbal working m ­ emory)
when they gestured during their explanation
than when they did not gesture (Goldin-Meadow,
Nusbaum, Kelly, & Wagner, 2001; Wagner,
­Nusbaum, & Goldin-Meadow, 2004). This effect
was found even when the gestures were directed at
objects that were not present in the context (Ping &
Goldin-Meadow, 2010), suggesting that gesturing
confers its benefits not just by tying abstract speech
to objects directly visible in the environment.
Importantly, it was not being told not to gesture that
increased demands on working memory—speakers
remembered more words when they gestured than
when they did not gesture, both when they were
instructed not to gesture and also when they chose
not to gesture (Goldin-Meadow et al., 2001).
Bringing in New Knowledge
Gesturing can also affect thinking by bringing new
knowledge into a speaker’s repertoire. To determine
whether gesture can create new ideas, we again need
to manipulate gesture, but this time we need to tell
speakers to move their hands in particular ways.
Speakers’ ideas should change as a function of the
particular hand movements they make. Goldin-
Meadow, Cook, and Mitchell (2009) manipulated
gesturing during a math lesson, asking some chil-
dren to produce gestures that instantiated a correct
procedure for solving the math problem, some to
produce gestures that instantiated a partially cor-
rect procedure for solving the math problem, and
some to produce no gestures at all. Children in all
three groups were taught a different (but also cor-
rect) procedure in speech. They found that children
required to produce correct gestures learned more
than children required to produce partially correct
gestures, who learned more than children required
to produce no gestures. It was clear that children in
Gesture
the gesture conditions had added new information
to their repertoires simply because, after the lesson,
they produced in speech the procedure that had
been instantiated only in their gestures during the
lesson (and that the teacher had not conveyed at all).
Researchers may be able to lay the foundations for
new knowledge simply by telling learners how to
move their hands.
APPLICATIONS OF GESTURE RESEARCH
Growing numbers of researchers are incorporat-
ing gesture into their work because of gesture’s
potential to reveal aspects of mental representation
or reasoning that are not conveyed in speech. For
example, studying gesture can reveal differences in
the ways experts and novices conceptualize prob-
lems (Ping, Larson, Decatur, Zinchenko, & Goldin-
Meadow, 2014), shed light on the use and impact of
nonverbal input in instruction (Alibali & Nathan,
2012; Church, Ayman-Nolley, & Mahootian, 2004;
­Richland, Zur, & Holyoak, 2007), and help iden-
tify students who are on the cusp of learning a new
concept (e.g., Alibali, Flevares, & Goldin-Meadow,
1997; Church & Goldin-Meadow, 1986; Perry,
Church, & Goldin-Meadow, 1988; Pine, Lufkin, &
Messer, 2004). Chapter 19 of this handbook looks
in greater detail at the coding of gesture and how
the study of gesture can be applied to the study of
the mind. Gesture can be particularly insightful
in the study of language production and language
acquisition. The gestures young children produce
often predict the upcoming developments in speech.
Conversely, a lack of gesture can be an early marker
of language delay.
Children begin gesturing before they can speak,
and gesture plays a large part in children’s early
communicative repertoire. As described earlier,
linguistic developments often manifest in gesture
before they become apparent in speech, and this
early window onto language development can be
used to identify atypical developmental trajectories
before they become visible in speech. Low gesture
rates early in development may be a signal that
a child is likely to experience delays in language
learning (Sauer, Levine, & Goldin-Meadow, 2010),
providing an opportunity for early interventions.
325
Cartmill and Goldin-Meadow
Conversely, robust gesturing in children who have
low language skills relative to their peers signals that
they are ready to learn and are “late bloomers” who
will catch up without the need for intervention. One
study of low verbal children (whose vocabularies
were in the lowest 10% of their age group) found
that the children in this group who were the most
competent in gesture caught up with their peers
within the following year, whereas the children who
had fared poorly on gesture at the initial evalua-
tion remained delayed a year later (Thal, Tobias, &
Morrison, 1991). Similarly, a study of children with
early brain injury found that children whose gesture
rates were within the typical range were the most
likely to catch up with their typically developing
peers, whereas children whose gesture rates were
below the typical range were likely to display persis-
tent language delay (Sauer et al., 2010).
Gesture’s potential for early identification of
delay has become increasingly important in autism
research. One study of gesturing in 12-month-old
infants found that infants who were later diagnosed
with autism had gestured less overall and produced
very little pointing when compared to typically
developing infants (Osterling & Dawson, 1994;
see also Bernabei, Camaigni, & Levi, 1998). This
reduced gesturing has also been found in studies of
younger siblings of children with autism—children
who displayed smaller communicative repertoires
in gesture had increased likelihood of receiving an
autism diagnosis later in development (Mitchell
et al., 2006). Although further work is needed to
understand the development of gesture and speech
in atypically developing populations, gesture’s
potential to reveal communicative delays before they
manifest in speech makes gesture a powerful tool in
the early identification (and possible treatment) of
linguistic disorders.
Gesture can also play an important role in the
legal world. The accuracy of information obtained in
forensic interviews is critically important to credibi­
lity in the legal system. It is well-known that the way
interviewers frame questions influences the accuracy
of witnesses’ reports. Broaders and Goldin-Meadow
(2010) studied children interviewed about an event
that they had witnessed. They found that the inter-
viewer’s gestures served as a source of information
326
and, at times, misinformation that led the child
­witnesses to report incorrect details. Conversely,
they also found that the gestures that the child wit-
nesses spontaneously produced during the inter-
views conveyed substantive information that was not
always conveyed in their speech and, thus, would not
appear in written transcripts of the proceedings. The
findings underscore the need to attend to and docu-
ment gestures produced in investigative interviews,
particularly interviews conducted with children.
Finally, gesture can play a role in the classroom.
Because children’s gestures often display informa-
tion about their thinking that they do not express in
speech, gesture can provide teachers with important
information about their pupils’ knowledge. Not only
do teachers pay attention to the information that
children express in gesture (e.g., Alibali et al., 1997)
but they also alter their input to children as a func-
tion of those gestures (Goldin-Meadow & Singer,
2003). In addition, the gestures that teachers them-
selves produce during their lessons have been found
to matter for student learning. Lessons that contain
gestures promote deeper learning (i.e., new forms
of reasoning, generalization to new problem types,
retention of knowledge) better than lessons that do
not contain gestures (Church et al., 2004; Valen-
zeno, Alibali, & Klatzky, 2003). Moreover, because
it is known that the act of gesturing can itself pro-
mote learning (Broaders, Cook, Mitchell, & Goldin-
Meadow, 2007), teachers can consider encouraging
their students to gesture, which has the potential to
activate implicit knowledge and make the students
particularly receptive to instruction.
CONCLUSIONS AND DIRECTIONS FOR
FUTURE RESEARCH
Gesture is a powerful representational tool and
an integral part of human language. It can convey
meaning through deixis, rhythm, convention, ico-
nicity, and even metaphor. Gesture and speech are
tightly integrated both temporally and semantically.
Gesture can have a complementary or supplementary
relationship with speech, and the combination of
gesture and speech often conveys a richer meaning
than either modality does on its own. ­Gesturing can
reveal thoughts that are not expressed in speech, and

professionals involved in learning and assessment
(e.g., teachers and clinicians) would benefit from
taking gestures into account when performing evalu-
ations. Gesture emerges early in language develop-
ment and has great potential as an early indicator
of linguistic delay or other atypical development.
Further research into the clinical and educational
applications of gesture will help clarify when and
how gesture may be used as a diagnostic tool.
Gesture also plays a role in cognition: reduc-
ing demands on memory and conceptualization
and integrating new knowledge during learning. It
is known that gesture can play a role in cognition
and learning—it not only reflects knowledge but it
can also play a causal role in changing that knowl-
edge (e.g., Goldin-Meadow et al., 2009). However,
researchers do not yet know the extent to which
gesturing changes the way a person represents an
idea or reasons through a problem. Researchers also
do not know whether gesture’s role in cognition
stems from its embodied nature (i.e., the fact that
it resembles action) or its visiospatial properties.
These questions are particularly salient in the realm
of language development—children’s gestures pres-
age upcoming linguistic developments and have the
potential to play a causal role in that development
(see, e.g., LeBarton, Goldin-Meadow, & Rauden-
bush, 2013). Understanding when and how gesture
can affect language and thought is a rich area for
future inquiry.
The movements we make with our hands when
we talk are important to both communication and
thought. Understanding the specific ways in which
gesture reflects and affects language and learning is
an important challenge for future research. Integrat-
ing gesture into clinical and scientific approaches to
language and thought will ultimately lead to a deeper
understanding of the nature of the human mind.
References
Acredolo, L., & Goodwyn, S. (1988). Symbolic gesturing
in normal infants. Child Development, 59, 450–466.
http://dx.doi.org/10.2307/1130324
Alibali, M. W., Flevares, L., & Goldin-Meadow, S. (1997).
Assessing knowledge conveyed in gesture: Do
teachers have the upper hand? Journal of Educational
Psychology, 89, 183–193. http://dx.doi.org/10.1037/
0022-0663.89.1.183
Gesture
Alibali, M. W., & Goldin-Meadow, S. (1993).
Gesture–speech mismatch and mechanisms of
learning: What the hands reveal about a child’s state
of mind. Cognitive Psychology, 25, 468–523.
http://dx.doi.org/10.1006/cogp.1993.1012
Alibali, M. W., Heath, D. C., & Myers, H. J. (2001).
Effects of visibility between speaker and listener on
gesture production: Some gestures are meant to be
seen. Journal of Memory and Language, 44, 169–188.
http://dx.doi.org/10.1006/jmla.2000.2752
Alibali, M. W., Kita, S., & Young, A. J. (2000). Gesture
and the process of speech production: We think,
therefore we gesture. Language and Cognitive
Processes, 15, 593–613. http://dx.doi.org/
10.1080/016909600750040571
Alibali, M. W., & Nathan, M. J. (2012). Embodiment
in mathematics teaching and learning: Evidence
from learners’ and teachers’ gestures. Journal of the
Learning Sciences, 21, 247–286. http://dx.doi.org/
10.1080/10508406.2011.611446
Barbieri, F., Buonocore, A., Volta, R. D., & Gentilucci, M.
(2009). How symbolic gestures and words interact
with each other. Brain and Language, 110, 1–11.
http://dx.doi.org/10.1016/j.bandl.2009.01.002
Bates, E. (1976). Language and context: The acquisition of
pragmatics. New York, NY: Academic Press.
Bates, E., Benigni, L., Bretherton, I., Camaioni, L., &
Volterra, V. (1979). The emergence of symbols:
Cognition and communication in infancy. New York,
NY: Academic Press.
Bavin, E. L., Prior, M., Reilly, S., Bretherton, L.,
Williams, J., Eadie, P., . . . Ukoumunne, O. C. (2008).
The Early Language in Victoria Study: Predicting
vocabulary at age one and two years from gesture and
object use. Journal of Child Language, 35, 687–701.
http://dx.doi.org/10.1017/S0305000908008726
Beattie, G., & Shovelton, H. (1999). Mapping the range
of information contained in the iconic hand gestures
that accompany spontaneous speech. Journal of
Language and Social Psychology, 18, 438–462.
http://dx.doi.org/10.1177/0261927X99018004005
Beattie, G., & Shovelton, H. (2000). Iconic hand gestures
and the predictability of words in context in
spontaneous speech. British Journal of Psychology,
91, 473–491. http://dx.doi.org/10.1348/
000712600161943
Beaudoin-Ryan, L., & Goldin-Meadow, S. (2014).
Teaching moral reasoning through gesture.
Developmental Science, 17, 984–990. http://dx.doi.org/
10.1111/desc.12180
Beilock, S. L., & Goldin-Meadow, S. (2010). Gesture
changes thought by grounding it in action.
Psychological Science, 21, 1605–1610. http://dx.doi.org/
10.1177/0956797610385353
327
Cartmill and Goldin-Meadow
Bernabei, P., Camaigni, L., & Levi, G. (1998). An evaluation
of early development in children with autism and
pervasive developmental disorders from home movies:
Preliminary findings. Autism, 2, 243–258. http://dx.doi.
org/10.1177/1362361398023003
Bernardis, P., & Gentilucci, M. (2006). Speech and
gesture share the same communication system.
Neuropsychologia, 44, 178–190. http://dx.doi.org/
10.1016/j.neuropsychologia.2005.05.007
Bonvillian, J. D., Orlansky, M. O., & Novack, L. L. (1983).
Developmental milestones: Sign language acquisition
and motor development. Child Development, 54,
1435–1445. http://dx.doi.org/10.2307/1129806
Brentari, D., Coppola, M., Mazzoni, L., & Goldin-
Meadow, S. (2012). When does a system become
phonological? Handshape production in gesturers,
signers, and homesigners. Natural Language and
Linguistic Theory, 30, 1–31. http://dx.doi.org/
10.1007/s11049-011-9145-1
Broaders, S. C., Cook, S. W., Mitchell, Z., & Goldin-
Meadow, S. (2007). Making children gesture brings
out implicit knowledge and leads to learning. Journal
of Experimental Psychology: General, 136, 539–550.
http://dx.doi.org/10.1037/0096-3445.136.4.539
Broaders, S. C., & Goldin-Meadow, S. (2010). Truth
is at hand: How gesture adds information during
investigative interviews. Psychological Science,
21, 623–628. http://dx.doi.org/10.1177/
0956797610366082
Butcher, C., & Goldin-Meadow, S. (2000). Gesture and
the transition from one-to two-word speech: When
hand and mouth come together. In D. McNeill (Ed.),
Language and gesture (pp. 235–257). http://dx.doi.org/
10.1017/CBO9780511620850.015
Butcher, C., Mylander, C., & Goldin-Meadow, S. (1991).
Displaced communication in a self-styled gesture
system: Pointing at the nonpresent. Cognitive
Development, 6, 315–342. http://dx.doi.org/
10.1016/0885-2014(91)90042-C
Call, J., & Tomasello, M. (Eds.). (2007). The gestural
communication of apes and monkeys. Mahwah, NJ:
Erlbaum.
Capirci, O., Iverson, J. M., Pizzuto, E., & Volterra,
V. (1996). Communicative gestures during the
transition to two-word speech. Journal of Child
Language, 23, 645–673.
Cartmill, E. A., Beilock, S., & Goldin-Meadow, S. (2012).
A word in the hand: Action, gesture and mental
representation in humans and non-human primates.
Philosophical Transactions of the Royal Society of
London B: Biological Sciences, 367, 129–143.
http://dx.doi.org/10.1098/rstb.2011.0162
Cartmill, E. A., Demir, Ö. E., & Goldin-Meadow, S.
(2012). Studying gesture. In E. Hoff (Ed.), The guide
328
to research methods in child language (pp. 208–225).
http://dx.doi.org/10.1002/9781444344035.ch14
Cartmill, E. A., & Maestripieri, D. (2012). Socio-cognitive
specializations in nonhuman primates: Evidence
from gestural communication. In J. Vonk & T.
K. Shackelford (Eds.), The Oxford handbook of
comparative evolutionary psychology (pp. 166–193).
Oxford, England: Oxford University Press.
Cassell, J., McNeill, D., & McCullough, K. E. (1999).
Speech–gesture mismatches: Evidence for one
underlying representation of linguistic and
nonlinguistic information. Pragmatics and Cognition,
7, 1–34. http://dx.doi.org/10.1075/pc.7.1.03cas
Chawla, P., & Krauss, R. (1994). Gesture and speech
in spontaneous and rehearsed narratives. Journal of
Experimental Social Psychology, 30, 580–601.
http://dx.doi.org/10.1006/jesp.1994.1027
Chieffi, S., Secchi, C., & Gentilucci, M. (2009). Deictic
word and gesture production: Their interaction.
Behavioural Brain Research, 203, 200–206.
http://dx.doi.org/10.1016/j.bbr.2009.05.003
Church, R. B., Ayman-Nolley, S., & Mahootian, S.
(2004). The role of gesture in bilingual education:
Does gesture enhance learning? International Journal
of Bilingual Education and Bilingualism, 7, 303–319.
http://dx.doi.org/10.1080/13670050408667815
Church, R. B., & Goldin-Meadow, S. (1986). The
mismatch between gesture and speech as an index of
transitional knowledge. Cognition, 23, 43–71.
http://dx.doi.org/10.1016/0010-0277(86)90053-3
Church, R. B., Kelly, S., & Holcombe, D. (2014).
Temporal synchrony between speech, action and
gesture during language production. Language,
Cognition and Neuroscience, 29, 345–354.
http://dx.doi.org/10.1080/01690965.2013.857783
Coppola, M., & Newport, E. L. (2005). Grammatical
subjects in home sign: Abstract linguistic structure
in adult primary gesture systems without linguistic
input. Proceedings of the National Academy of
Sciences, USA, 102, 19249–19253. http://dx.doi.org/
10.1073/pnas.0509306102
Coppola, M., & Senghas, A. (2010). Deixis in an
emerging sign language. In D. Brentari (Ed.),
Sign languages: A Cambridge language survey
(pp. 543–569). http://dx.doi.org/10.1017/
CBO9780511712203.025
Coppola, M., Spaepen, E., & Goldin-Meadow, S. (2013).
Communicating about quantity without a language
model: Number devices in homesign grammar.
Cognitive Psychology, 67, 1–25. http://dx.doi.org/
10.1016/j.cogpsych.2013.05.003
de Laguna, G. (1927). Speech: Its function and
development. Bloomington: Indiana University Press.

Demir, O. E., So, W.-C., Özyürek, A., & Goldin-Meadow, S.
(2012). Turkish- and English-speaking children
display sensitivity to perceptual context in the referring
expressions they produce in speech and gesture.
Language and Cognitive Processes, 27, 844–867.
http://dx.doi.org/10.1080/01690965.2011.589273
Dick, A. S., Goldin-Meadow, S., Hasson, U., Skipper, J. I., &
Small, S. L. (2009). Co-speech gestures influence
neural activity in brain regions associated with
processing semantic information. Human Brain
Mapping, 30, 3509–3526. http://dx.doi.org/10.1002/
hbm.20774
Ekman, P., & Friesen, W. V. (1969). The repertoire of
nonverbal behavior: Categories, origins, usage and
coding. Semiotica, 1, 49–98.
Emmorey, K., & Casey, S. (2001). Gesture, thought and
spatial language. Gesture, 1, 35–50. http://dx.doi.org/
10.1075/gest.1.1.04emm
Feldman, H., Goldin-Meadow, S., & Gleitman, L. (1978).
Beyond Herodotus: The creation of language by
linguistically deprived deaf children. In A. Lock
(Ed.), Action, symbol, and gesture: The emergence of
language (pp. 351–414). New York, NY: Academic
Press.
Franklin, A., Giannakidou, A., & Goldin-Meadow, S.
(2011). Negation, questions, and structure building
in a homesign system. Cognition, 118, 398–416.
http://dx.doi.org/10.1016/j.cognition.2010.08.017
Frick-Horbury, D., & Guttentag, R. E. (1998). The
effects of restricting hand gesture production on
lexical retrieval and free recall. American Journal of
Psychology, 111, 43–62. http://dx.doi.org/
10.2307/1423536
Gershkoff-Stowe, L., & Goldin-Meadow, S. (2002).
Is there a natural order for expressing semantic
relations? Cognitive Psychology, 45, 375–412.
http://dx.doi.org/10.1016/S0010-0285(02)00502-9
Goldin-Meadow, S. (1982). The resilience of recursion: A
study of a communication system developed without
a conventional language model. In E. Wanner &
L. R. Gleitman (Eds.), Language acquisition: The state
of the art (pp. 51–77). New York, NY: Cambridge
University Press.
Goldin-Meadow, S. (1993). When does gesture become
language? A study of gesture used as a primary
communication system by deaf children of hearing
parents. In K. R. Gibson & T. Ingold (Eds.), Tools,
language, and cognition in human evolution (pp. 63–85).
New York, NY: Cambridge University Press.
Goldin-Meadow, S. (2003a). Hearing gesture: How
our hands help us think. Cambridge, MA: Harvard
University Press.
Goldin-Meadow, S. (2003b). Resilience of language: What
gesture creation in deaf children can tell us about
Gesture
how all children learn language. New York, NY:
Psychology Press.
Goldin-Meadow, S. (2006). Nonverbal communication:
The hand’s role in talking and thinking. In
W. Damon, R. Lerner, D. Kuhn, & R. S. Siegler (Eds.),
Handbook of child psychology: Vol. 2. Cognition,
perception, and language (6th ed., pp. 336–372).
New York, NY: Wiley.
Goldin-Meadow, S. (2015). The impact of time on predicate
forms in the manual modality: Signers, homesigners,
and silent gesturers. Topics in Cognitive Science, 7,
169–184. http://dx.doi.org/10.1111/tops.12119
Goldin-Meadow, S., & Alibali, M. W. (2013). Gesture’s
role in speaking, learning, and creating language.
Annual Review of Psychology, 64, 257–283. http://
dx.doi.org/10.1146/annurev-psych-113011-143802
Goldin-Meadow, S., & Beilock, S. (2010). Action’s
influence on thought: The case of gesture.
Perspectives on Psychological Science, 5, 664–674.
http://dx.doi.org/10.1177/1745691610388764
Goldin-Meadow, S., & Butcher, C. (2003). Pointing
toward two-word speech in young children. In
S. Kita (Ed.), Pointing: Where language, culture, and
cognition meet (pp. 85–107). Hillsdale, NJ: Erlbaum.
Goldin-Meadow, S., Butcher, C., Mylander, C., & Dodge,
M. (1994). Nouns and verbs in a self-styled gesture
system: What’s in a name? Cognitive Psychology, 27,
259–319. http://dx.doi.org/10.1006/cogp.1994.1018
Goldin-Meadow, S., Cook, S. W., & Mitchell, Z. A.
(2009). Gesturing gives children new ideas about
math. Psychological Science, 20, 267–272. http://
dx.doi.org/10.1111/j.1467-9280.2009.02297.x
Goldin-Meadow, S., Gelman, S. A., & Mylander, C.
(2005). Expressing generic concepts with and
without a language model. Cognition, 96, 109–126.
http://dx.doi.org/10.1016/j.cognition.2004.07.003
Goldin-Meadow, S., Kim, S., & Singer, M. (1999). What
the teacher’s hands tell the student’s mind about
math. Journal of Educational Psychology, 91, 720–730.
http://dx.doi.org/10.1037/0022-0663.91.4.720
Goldin-Meadow, S., McNeill, D., & Singleton, J. (1996).
Silence is liberating: Removing the handcuffs on
grammatical expression in the manual modality.
Psychological Review, 103, 34–55. http://dx.doi.org/
10.1037/0033-295X.103.1.34
Goldin-Meadow, S., & Mylander, C. (1983). Gestural
communication in deaf children: Noneffect of
parental input on language development. Science, 221,
372–374. http://dx.doi.org/10.1126/science.6867713
Goldin-Meadow, S., & Mylander, C. (1984). Gestural
communication in deaf children: The effects and
noneffects of parental input on early language
development. Monographs of the Society for Research
329
Cartmill and Goldin-Meadow
in Child Development, 49, 1–151. http://dx.doi.org/
10.2307/1165838
Goldin-Meadow, S., & Mylander, C. (1998). Spontaneous
sign systems created by deaf children in two cultures.
Nature, 391, 279–281. http://dx.doi.org/10.1038/
34646
Goldin-Meadow, S., Mylander, C., & Butcher, C. (1995).
The resilience of combinatorial structure at the word
level: Morphology in self-styled gesture systems.
Cognition, 56, 195–262. http://dx.doi.org/10.1016/
0010-0277(95)00662-I
Goldin-Meadow, S., Mylander, C., & Franklin, A.
(2007). How children make language out of
gesture: Morphological structure in gesture systems
developed by American and Chinese deaf children.
Cognitive Psychology, 55, 87–135. http://dx.doi.org/
10.1016/j.cogpsych.2006.08.001
Goldin-Meadow, S., Nusbaum, H., Kelly, S. D., &
Wagner, S. (2001). Explaining math: Gesturing
lightens the load. Psychological Science, 12, 516–522.
http://dx.doi.org/10.1111/1467-9280.00395
Goldin-Meadow, S., & Sandhofer, C. M. (1999). Gestures
convey substantive information about a child’s
thoughts to ordinary listeners. Developmental
Science, 2, 67–74. http://dx.doi.org/10.1111/
1467-7687.00056
Goldin-Meadow, S., Shield, A., Lenzen, D., Herzig, M., &
Padden, C. (2012). The gestures ASL signers
use tell us when they are ready to learn math.
Cognition, 123, 448–453. http://dx.doi.org/10.1016/
j.cognition.2012.02.006
Goldin-Meadow, S., & Singer, M. A. (2003). From
children’s hands to adults’ ears: Gesture’s role in
the learning process. Developmental Psychology,
39, 509–520. http://dx.doi.org/10.1037/0012-
1649.39.3.509
Goldin-Meadow, S., So, W.-C., Özyürek, A., & Mylander,
C. (2008). The natural order of events: How speakers
of different languages represent events nonverbally.
Proceedings of the National Academy of Sciences,
USA, 105, 9163–9168. http://dx.doi.org/10.1073/
pnas.0710060105
Goodwyn, S. W., & Acredolo, L. P. (1993). Symbolic
gesture versus word: Is there a modality advantage
for onset of symbol use? Child Development, 64,
688–701. http://dx.doi.org/10.2307/1131211
Graham, J. A., & Argyle, M. (1975). A cross-cultural
study of the communication of extra-verbal
meaning by gestures. International Journal of
Psychology, 10, 57–67. http://dx.doi.org/10.1080/
00207597508247319
Greenfield, P., & Smith, J. (1976). The structure of
communication in early language development.
New York, NY: Academic Press.
330
Guidetti, M. (2002). The emergence of pragmatics: Forms
and functions of conventional gestures in young
French children. First Language, 22, 265–285.
Harnsberger, J. D., Shrivastav, R., Brown, W. S., Jr.,
Rothman, H., & Hollien, H. (2008). Speaking
rate and fundamental frequency as speech cues to
perceived age. Journal of Voice, 22, 58–69.
http://dx.doi.org/10.1016/j.jvoice.2006.07.004
Hostetter, A. B., Alibali, M. W., & Kita, S. (2007). I see
it in my hands’ eye: Representational gestures
reflect conceptual demands. Language and Cognitive
Processes, 22, 313–336. http://dx.doi.org/10.1080/
01690960600632812
Hunsicker, D., & Goldin-Meadow, S. (2012). Hierarchical
structure in a self-created communication system:
Building nominal constituents in homesign.
Language, 88, 732–763. http://dx.doi.org/10.1353/
lan.2012.0092
Iverson, J. M., & Goldin-Meadow, S. (1998). Why people
gesture when they speak. Nature, 396, 228.
http://dx.doi.org/10.1038/24300
Iverson, J. M., & Goldin-Meadow, S. (2005). Gesture
paves the way for language development.
Psychological Science, 16, 367–371. http://dx.doi.org/
10.1111/j.0956-7976.2005.01542.x
Iverson, J. M., & Thelen, E. (1999). Hand, mouth and
brain: The dynamic emergence of speech and
gesture. Journal of Consciousness Studies, 6, 19–40.
Jacobs, N., & Garnham, A. (2007). The role of
conversational hand gestures in a narrative task.
Journal of Memory and Language, 56, 291–303.
http://dx.doi.org/10.1016/j.jml.2006.07.011
Kegl, J., Senghas, A., & Coppola, M. (1999). Creation
through contact: Sign language emergence and sign
language change in Nicaragua. In M. DeGraff (Ed.),
Language creation and language change: Creolization,
diachrony, and development (pp. 179–237).
Cambridge, MA: MIT Press.
Kelly, S. D., & Church, R. B. (1998). A comparison between
children’s and adults’ ability to detect conceptual
information conveyed through representational
gestures. Child Development, 69, 85–93. http://dx.doi.org/
10.1111/j.1467-8624.1998.tb06135.x
Kelly, S. D., Kravitz, C., & Hopkins, M. (2004).
Neural correlates of bimodal speech and gesture
comprehension. Brain and Language, 89, 253–260.
http://dx.doi.org/10.1016/S0093-934X(03)00335-3
Kelly, S. D., Özyürek, A., & Maris, E. (2010). Two sides
of the same coin: Speech and gesture mutually
interact to enhance comprehension. Psychological
Science, 21, 260–267. http://dx.doi.org/10.1177/
0956797609357327
Kendon, A. (1980). Gesticulation and speech: Two aspects
of the process of utterance. In M. R. Key (Ed.),

Relationship of verbal and nonverbal communication
(pp. 207–228). The Hague, the Netherlands:
Mouton.
Kendon, A. (2004). Gesture: Visible action as utterance.
Cambridge, England: Cambridge University Press.
Kita, S. (2009). Cross-cultural variation of speech-
accompanying gesture: A review. Language and
Cognitive Processes, 24, 145–167. http://dx.doi.org/
10.1080/01690960802586188
Kita, S., & Davies, T. S. (2009). Competing
conceptual representations trigger co-speech
representational gestures. Language and Cognitive
Processes, 24, 761–775. http://dx.doi.org/10.1080/
01690960802327971
Kita, S., & Özyürek, A. (2003). What does cross-
linguistic variation in semantic coordination of
speech and gesture reveal? Evidence for an interface
representation of spatial thinking and speaking.
Journal of Memory and Language, 48, 16–32.
http://dx.doi.org/10.1016/S0749-596X(02)00505-3
Kita, S., Özyürek, A., Allen, S., Brown, A., Furman, R., &
Ishizuka, T. (2007). Relations between syntactic
encoding and co-speech gestures: Implications for a
model of speech and gesture production. Language
and Cognitive Processes, 22, 1212–1236.
http://dx.doi.org/10.1080/01690960701461426
Klima, E., & Bellugi, U. (1979). The signs of language.
Cambridge, MA: Harvard University Press.
Krahmer, E., & Swerts, M. (2007). The effects of visual
beats on prosodic prominence: Acoustic analyses,
auditory perception and visual perception. Journal of
Memory and Language, 57, 396–414. http://dx.doi.org/
10.1016/j.jml.2007.06.005
Krauss, R. M., Morrel-Samuels, P., & Colasante, C.
(1991). Do conversational hand gestures
communicate? Journal of Personality and Social
Psychology, 61, 743–754. http://dx.doi.org/10.1037/
0022-3514.61.5.743
Langton, S. R. H., O’Malley, C., & Bruce, V. (1996).
Actions speak no louder than words: Symmetrical
cross-modal interference effects in the processing
of verbal and gestural information. Journal of
Experimental Psychology: Human Perception and
Performance, 22, 1357–1375. http://dx.doi.org/
10.1037/0096-1523.22.6.1357
Langus, A., & Nespor, M. (2010). Cognitive systems
struggling for word order. Cognitive Psychology,
60, 291–318. http://dx.doi.org/10.1016/
j.cogpsych.2010.01.004
Leavens, D. A. (2004). Manual deixis in apes and
humans. Interaction Studies, 5, 387–408.
http://dx.doi.org/10.1075/is.5.3.05lea
Leavens, D. A., Hopkins, W. D., & Bard, K. A. (1996).
Indexical and referential pointing in chimpanzees
Gesture
(Pan troglodytes). Journal of Comparative Psychology,
110, 346–353. http://dx.doi.org/10.1037/
0735-7036.110.4.346
LeBarton, E. S., Goldin-Meadow, S., & Raudenbush, S.
(2013). Experimentally induced increases in early
gesture lead to increases in spoken vocabulary.
Journal of Cognition and Development. Advance online
publication. http://dx.doi.org/10.1080/15248372.20
13.858041
Leopold, W. (1949). Speech development of a bilingual
child: Vol. 3. A linguist’s record. Evanston, IL:
Northwestern University Press.
Liddell, S. (1980). American Sign Language syntax. The
Hague, the Netherlands: Mouton.
Liebal, K., Call, J., Tomasello, M., & Pika, S. (2004).
To move or not to move: How apes adjust to the
attentional state of others. Interaction Studies, 5,
199–219. http://dx.doi.org/10.1075/is.5.2.03lie
Liszkowski, U., Schäfer, M., Carpenter, M., &
Tomasello, M. (2009). Prelinguistic infants, but not
chimpanzees, communicate about absent entities.
Psychological Science, 20, 654–660. http://dx.doi.org/
10.1111/j.1467-9280.2009.02346.x
Matsumoto, D., & Hwang, H. C. (2013). Cultural
similarities and differences in emblematic gestures.
Journal of Nonverbal Behavior, 37, 1–27. http://dx.doi.
org/10.1007/s10919-012-0143-8
Mayberry, R. I., & Jaques, J. (2000). Gesture production
during stuttered speech: Insights into the nature
of gesture–speech integration. In D. McNeill (Ed.),
Language and gesture (pp. 199–214). http://dx.doi.org/
10.1017/CBO9780511620850.013
McClave, E. (1998). Pitch and manual gestures. Journal of
Psycholinguistic Research, 27, 69–89. http://dx.doi.org/
10.1023/A:1023274823974
McClave, E., Kim, H., Tamer, R., & Mileff, M. (2007).
Head movements in the context of speech in Arabic,
Bulgarian, Korean, and African-American Vernacular
English. Gesture, 7, 343–390. http://dx.doi.org/
10.1075/gest.7.3.04mcc
McClave, E. Z. (2000). Linguistic functions of head
movements in the context of speech. Journal of
Pragmatics, 32, 855–878. http://dx.doi.org/10.1016/
S0378-2166(99)00079-X
McNeil, N. M., Alibali, M. W., & Evans, J. L. (2000).
The role of gesture in children’s comprehension of
spoken language: Now they need it, now they don’t.
Journal of Nonverbal Behavior, 24, 131–150.
http://dx.doi.org/10.1023/A:1006657929803
McNeill, D. (1992). Hand and mind: What gestures reveal
about thought. Chicago, IL: University of Chicago Press.
McNeill, D. (2005). Gesture and thought. Chicago, IL:
University of Chicago Press.
331
Cartmill and Goldin-Meadow
McNeill, D., Levy, E., & Pedelty, L. (1990). Speech and
gesture. In G. Hammond (Ed.), Cerebral control of
speech and limb movements (pp. 203–256). North
Holland, the Netherlands: Elsevier.
Meir, I., Lifshitz, A., Ilkbasaran, D., & Padden, C.
(2010). The interaction of animacy and word
order in human languages: A study of strategies in
a novel communication task. In A. D. M. Smith,
M. Schouwstra, B. de Boer, & K. Smith (Eds.),
Proceedings of the Eighth Evolution of Language
Conference (pp. 455–456). Singapore: World
Scientific Publishing.
Melinger, A., & Kita, S. (2007). Conceptualisation load
triggers gesture production. Language and Cognitive
Processes, 22, 473–500. http://dx.doi.org/10.1080/
01690960600696916
Mitchell, S., Brian, J., Zwaigenbaum, L., Roberts, W.,
Szatmari, P., Smith, I., & Bryson, S. (2006). Early
language and communication development of infants
later diagnosed with autism spectrum disorder.
Journal of Developmental and Behavioral Pediatrics,
27, S69–S78. http://dx.doi.org/10.1097/
00004703-200604002-00004
Mol, L., Krahmer, E., Maes, A., & Swerts, M. (2011).
Seeing and being seen: The effects on gesture
production. Journal of Computer-Mediated
Communication, 17, 77–100. http://dx.doi.org/
10.1111/j.1083-6101.2011.01558.x
Morford, J. P., & Goldin-Meadow, S. (1997). From here
and now to there and then: The development of
displaced reference in homesign and English. Child
Development, 68, 420–435. http://dx.doi.org/
10.2307/1131669
Newcombe, N. (2010). Picture this: Increasing math
and science learning by improving spatial thinking.
American Educator, 34(2), 29–43.
Newport, E. L., & Meier, R. P. (1985). The acquisition of
American Sign Language. In D. I. Slobin (Ed.), The
cross-linguistic study of language acquisition: Vol. 1.
The data (pp. 881–938). Hillsdale, NJ: Erlbaum.
Novack, M. A., Congdon, E. L., Hemani-Lopez, N., &
Goldin-Meadow, S. (2014). From action to
abstraction: Using the hands to learn math.
Psychological Science, 25, 903–910. http://dx.doi.org/
10.1177/0956797613518351
Osterling, J., & Dawson, G. (1994). Early recognition of
children with autism: A study of first birthday home
videotapes. Journal of Autism and Developmental
Disorders, 24, 247–257. http://dx.doi.org/10.1007/
BF02172225
Özçali kan, S., & Goldin-Meadow, S. (2005). Gesture is
at the cutting edge of early language development.
Cognition, 96, B101–B113. http://dx.doi.org/
10.1016/j.cognition.2005.01.001
332
Perlman, M. (2010). Talking fast: The use of speech rate
as iconic gesture. In F. Parrill, M. Turner, & V. Tobin
(Eds.), Meaning, form, and body (pp. 245–262).
Stanford, CA: Center for the Study of Language and
Information.
Perry, M., Church, R. B., & Goldin-Meadow, S. (1988).
Transitional knowledge in the acquisition of
concepts. Cognitive Development, 3, 359–400.
http://dx.doi.org/10.1016/0885-2014(88)90021-4
Pine, K. J., Bird, H., & Kirk, E. (2007). The effects of
prohibiting gestures on children’s lexical retrieval
ability. Developmental Science, 10, 747–754.
http://dx.doi.org/10.1111/j.1467-7687.2007.00610.x
Pine, K. J., Lufkin, N., & Messer, D. (2004). More
gestures than answers: Children learning about
balance. Developmental Psychology, 40, 1059–1067.
http://dx.doi.org/10.1037/0012-1649.40.6.1059
Ping, R., & Goldin-Meadow, S. (2010). Gesturing saves
cognitive resources when talking about nonpresent
objects. Cognitive Science, 34, 602–619. http://dx.doi.
org/10.1111/j.1551-6709.2010.01102.x
Ping, R., Larson, S. W., Decatur, M.-A., Zinchenko, E., &
Goldin-Meadow, S. (2014). Unpacking the gestures
of chemistry learners: What the hands tell us about
correct and incorrect conceptions of stereochemistry.
Manuscript submitted for publication.
Poss, S. R., Kuhar, C., Stoinski, T. S., & Hopkins, W. D.
(2006). Differential use of attentional and visual
communicative signaling by orangutans (Pongo
pygmaeus) and gorillas (Gorilla gorilla) in response
to the attentional status of a human. American
Journal of Primatology, 68, 978–992. http://dx.doi.org/
10.1002/ajp.20304
Rauscher, F., Krauss, R., & Chen, Y. (1996). Gesture,
speech, and lexical access: The role of lexical
movements in speech production. Psychological
Science, 7, 226–231. http://dx.doi.org/10.1111/
j.1467-9280.1996.tb00364.x
Richland, L. E., Zur, O., & Holyoak, K. J. (2007).
Mathematics: Cognitive supports for analogies in the
mathematics classroom. Science, 316, 1128–1129.
http://dx.doi.org/10.1126/science.1142103
Rowe, M. L., & Goldin-Meadow, S. (2009a). Differences
in early gesture explain SES disparities in child
vocabulary size at school entry. Science, 323,
951–953. http://dx.doi.org/10.1126/science.1167025
Rowe, M. L., & Goldin-Meadow, S. (2009b). Early
gesture selectively predicts later language learning.
Developmental Science, 12, 182–187. http://dx.doi.org/
10.1111/j.1467-7687.2008.00764.x
Rowe, M. L., Özçali kan, S., & Goldin-Meadow, S.
(2008). Learning words by hand: Gesture’s role in
predicting vocabulary development. First Language,

28, 182–199. http://dx.doi.org/10.1177/
0142723707088310
Sandler, W., & Lillo-Martin, D. (2006). Sign language
and linguistic universals. http://dx.doi.org/10.1017/
CBO9781139163910
Sauer, E., Levine, S. C., & Goldin-Meadow, S. (2010).
Early gesture predicts language delay in children
with pre- or perinatal brain lesions. Child
Development, 81, 528–539. http://dx.doi.org/10.1111/
j.1467-8624.2009.01413.x
Savage-Rumbaugh, E. S., Wilkerson, B. J., & Bakeman,
R. (1977). Spontaneous gestural communication
among conspecifics in the pygmy chimpanzee (Pan
paniscus). In G. H. Bourne (Ed.), Progress in ape
research (pp. 97–116). http://dx.doi.org/10.1016/
B978-0-12-119350-8.50017-3
Sell, A., Bryant, G. A., Cosmides, L., Tooby, J., Sznycer,
D., Von Rueden, C., . . . Gurven, M. (2010).
Adaptations in humans for assessing physical
strength from the voice. Proceedings of the Royal
Society B: Biological Sciences, 277, 3509–3518.
Senghas, A. (2003). Intergenerational influence and
ontogenetic development in the emergence of spatial
grammar in Nicaraguan Sign Language. Cognitive
Development, 18, 511–531. http://dx.doi.org/
10.1016/j.cogdev.2003.09.006
Senghas, A., & Coppola, M. (2001). Children creating
language: How Nicaraguan Sign Language acquired a
spatial grammar. Psychological Science, 12, 323–328.
http://dx.doi.org/10.1111/1467-9280.00359
Shintel, H., & Nusbaum, H. C. (2007). The sound of
motion in spoken language: Visual information
conveyed by acoustic properties of speech.
Cognition, 105, 681–690. http://dx.doi.org/10.1016/
j.cognition.2006.11.005
Shintel, H., Nusbaum, H. C., & Okrent, A.
(2006). Analog acoustic expression in speech
communication. Journal of Memory and Language,
55, 167–177. http://dx.doi.org/10.1016/
j.jml.2006.03.002
Singleton, J. L., Morford, J. P., & Goldin-Meadow, S.
(1993). Once is not enough: Standards of well-
formedness in manual communication created over
three different timespans. Language, 69, 683–715.
http://dx.doi.org/10.2307/416883
Skipper, J. I., Goldin-Meadow, S., Nusbaum, H. C., &
Small, S. L. (2007). Speech-associated gestures,
Broca’s area, and the human mirror system. Brain and
Language, 101, 260–277. http://dx.doi.org/10.1016/
j.bandl.2007.02.008
Stokoe, W. C. (1960). Sign language structure: An
outline of the visual communication systems of the
American deaf (Studies in Linguistics: Occasional
Gesture
Papers, No. 8). Buffalo, NY: University of Buffalo,
Department of Anthropology and Linguistics.
Supalla, T. (1990). Serial verbs of motion in ASL. In S. D.
Fischer & P. Siple (Eds.), Theoretical issues in sign
language research: Vol. 1. Linguistics (pp. 127–152).
Chicago, IL: University of Chicago Press.
Tanner, J. E., & Byrne, R. W. (1996). Representation of
action through iconic gesture in a captive lowland
gorilla. Current Anthropology, 37, 162–173. http://
dx.doi.org/10.1086/204484
Thal, D., Tobias, S., & Morrison, D. (1991). Language
and gesture in late talkers: A 1-year follow-up.
Journal of Speech and Hearing Research, 34, 604–612.
http://dx.doi.org/10.1044/jshr.3403.604
Thompson, L. A., & Massaro, D. W. (1994). Children’s
integration of speech and pointing gestures in
comprehension. Journal of Experimental Child
Psychology, 57, 327–354. http://dx.doi.org/10.1006/
jecp.1994.1016
Thompson, R. L., Vinson, D. P., Woll, B., & Vigliocco,
G. (2012). The road to language learning is iconic:
Evidence from British Sign Language. Psychological
Science, 23, 1443–1448. http://dx.doi.org/10.1177/
0956797612459763
Tomasello, M., & Camaioni, L. (1997). A comparison
of the gestural communication of apes and human
infants. Human Development, 40, 7–24. http://
dx.doi.org/10.1159/000278540
Tomasello, M., Carpenter, M., & Liszkowski, U. (2007).
A new look at infant pointing. Child Development,
78, 705–722. http://dx.doi.org/10.1111/j.1467-8624.
2007.01025.x
Valenzeno, L., Alibali, M. W., & Klatzky, R. L. (2003).
Teachers’ gestures facilitate students’ learning: A
lesson in symmetry. Contemporary Educational
Psychology, 28, 187–204. http://dx.doi.org/10.1016/
S0361-476X(02)00007-3
Van Deusen-Phillips, S. B., Goldin-Meadow, S., &
Miller, P. J. (2001). Enacting stories, seeing worlds:
Similarities and differences in the cross-cultural
narrative development of linguistically isolated deaf
children. Human Development, 44, 311–336.
http://dx.doi.org/10.1159/000046153
Wagner, S. M., Nusbaum, H., & Goldin-Meadow, S.
(2004). Probing the mental representation of gesture:
Is handwaving spatial? Journal of Memory and
Language, 50, 395–407. http://dx.doi.org/10.1016/
j.jml.2004.01.002
Wilkins, D. (2003). Why pointing with the index finger is
not a universal (in sociocultural and semiotic terms).
In S. Kita (Ed.), Pointing: Where language, culture,
and cognition meet (pp. 171–215). Hillsdale, NJ:
Erlbaum.
333
 Chapter 13
Eye Behavior and Gaze
Reginald B. Adams Jr. and Anthony J. Nelson
The eyes have one language everywhere.
—George Herbert (1593–1633)
Eyes hold special prominence in human exchange
and have across time and culture. They serve both
as a salient channel through which people send non-
verbal messages as well as the primary mechanism by
which people perceive nonverbal messages sent by
others. An analysis of famous portraits dating back
5 centuries (Tyler, 1998) underscores the perceptual
salience of eyes by showing that one eye was almost
always painted on or very near the horizontal center
of the portrait, presumably reflecting the inner char-
acter of the subject. Such an assumption matches
popular folk wisdom that “the eyes are the window
to our souls”—wisdom that has infiltrated people’s
most personal and even most global of interactions.
Although remaining silent on matters of the soul,
contemporary science nevertheless strongly supports
the claim that the eyes are a richly informative social
cue, critical to people’s social exchange. For humans,
the “language of the eyes” appears innately prepared.
An ability to process gaze information is believed
to play a pivotal role in the development of theory
of mind—that is, the ability to understand that oth-
ers have mental and emotional experiences differ-
ent than your own (Baron-Cohen, 2002), which is
evidenced also by the fact that psychopathological
disorders marked by deficits in theory of mind
(e.g., autism) are likewise linked to a failure to
attend to the eyes (Emery, 2000). Whether this
“language” is universal remains open to question,
although we often assume it is. Take, for example,
http://dx.doi.org/10.1037/14669-013
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
George W. Bush’s proclamation after meeting
Vladimir Putin for the first time: “I looked the man
in the eye. I found him to be very straightforward
and trustworthy. . . . I was able to get a sense of his
soul” (Perlez, 2001, para. 9). Despite such assump-
tions, there are important individual, contextual, and
cultural influences that can affect how people inter-
pret messages conveyed by eyes. In this chapter, we
explore eye behaviors, how and why people engage
in them, what meanings people derive from them,
and the factors that influence such perceptions.
The first part of the chapter focuses on why
people are drawn to the eyes. The second part
focuses on two relatively less-studied eye behaviors:
tear production and pupil dilation/constriction.
The third part of the chapter focuses on the far more
researched behavior of gaze, both as a channel for
information gathering and as a critical mechanism
for social signaling, including discussion of the role
of gaze in theory of mind and emotion perception.
A central assumption of all this work is that the
eyes capture a disproportionate amount of attention
compared to other aspects of nonverbal behavior,
particularly considering their relatively small size.
Before delving into eye behavior, therefore, we first
review evidence that humans are exquisitely tuned
to the eye region of the face.
WHY PEOPLE ARE DRAWN TO THE
EYE REGION
It has been argued that people’s attraction to the
eye region of the face is innate (Argyle & Cook,
335
Adams and Nelson
1976). Early research on young adults in England
demonstrated that nearly 70% of fixations are allo-
cated to the eye region of a given face stimulus
(Walker-Smith, Gale, & Findlay, 1977), revealing a
preference for looking at the eyes over other facial
features. Moreover, adults spend more time looking
at faces with eyes open than those with eyes closed
(Batki, Baron-Cohen, Wheelwright, Connellan, &
Ahluwalia, 2000). Infants naturally attend to the
presence of two large, horizontally arranged eye-
spots, ignoring similar, single- and triple-spot sche-
matic arrays (Hess, 1975). Further, the eye region
of the face has been found to be a minimal facial cue
necessary to generate smiling responses in infants
(Argyle & Cook, 1976). Ahrens (1954) found that
within the first 2 months of life, the presentation of
two horizontally configured dots is enough to evoke
a smile response in an infant; yet, a whole face pre-
sented with eyes concealed failed to produce such
smiling. By 14–26 weeks, more smiling occurs in
response to direct versus averted eye gaze (Hains &
Muir, 1996). Even cross-species evidence reflects an
inherent attraction to the eyes in the form of eye-
spot configurations prevalently brandished on much
of the animal kingdom (Argyle & Cook, 1976).
Such eyespots are thought to have evolved to mimic
the eyes of larger animals to ward off predators and
to attract potential mates.
Because the eyes offer such rich social informa-
tion, adults and infants alike show a natural attrac-
tion to the whole face. The human face is equated
with one’s personal identity (Weitz, 1979) and is
astonishingly well remembered compared to other
visual stimuli (Bruck, Cavanagh, & Ceci, 1991;
Phillips, 1979). Further, researchers have discovered
patches of neurons in the fusiform gyrus (fusiform
face area [FFA]) that appear selectively devoted to
the processing of facial identity in both humans and
nonhuman primates (Kanwisher, McDermott, &
Chun, 1997). Interestingly, people recognize others
through eyes more than mouths, suggesting that the
eyes are particularly powerful in conveying identity
(Joseph & Tanaka, 2003; Tanaka & Farah, 1993).
Likewise, the FFA appears particularly sensitive to
the eyes (Bentin, Allison, Puce, Perez, & McCarthy,
1996), helping to explain why individuals with
autism, who tend to avoid eye contact, also show
336
hypoactivation of the FFA, except when instructed
to look at the eyes (Dalton et al., 2005).
How, When, and Why Facial Actions
Around the Eyes Occur
Some early theories (Buck, 1988; Fridlund,
Ekman, & Oster, 1987; Rinn, 1984) highlighted
evidence for two brain systems that govern the com-
munication of affect; one is considered a voluntary
emotion system stemming from the cortical pyra-
midal motor system, and the other is an involuntary
emotion system stemming from the phylogenically
older extrapyramidal system. Volitional movement
of the muscles around the mouth area, therefore, is
more finely tuned for complex movement, includ-
ing speech production. In contrast the upper face
muscles, including around the eye region, are more
tightly connected to the extrapyramidal motor sys-
tem, which is relatively more difficult and grossly
coordinated (Fridlund et al., 1987). Thus, research-
ers have suggested that the eye region of the face is
more revealing, as it is less prone to volitional, decep-
tive displays (see also Chapter 10, this handbook).
The muscles around the eyes are particularly
instrumental in expressive behavior (Ekman &
Friesen, 1975). The eye region is predominantly
controlled by groups of muscles in the upper face
(Cacioppo, Martzke, Petty, & Tassinary, 1988).
The frontalis muscles and depressor supercilii raise
and lower the eyebrows, respectively, whereas the
corrugator supercilii pull the brows together (i.e., fur-
rowed). The orbicularis oculi surrounds the eyeballs
and is responsible for the actual opening and closing
of the eyes as well as the wrinkles found alongside
the eyes during Duchenne smiles—or smiles that are
automatically elicited in response to spontaneous
positive emotion (Duchenne de Boulogne, 1862/1990).
These muscles are connected to other facial muscle
groups that are associated with facial expressions
(i.e., zygomatic major). Given that these muscle
groups help protect the eyes, they operate relatively
involuntarily and thus are difficult to control.
The Social Dynamics of Facial Actions
Around the Eyes
Basic emotions are accurately decoded from just the
eye region when presented separately from other

regions of the face (Baron-Cohen, Wheelwright, &
Jolliffe, 1997; Nummenmaa, 1964). The eye region
alone is even enough to elicit expressive mimicry
in others (Blaison, Hareli, Strauss, & Hess, 2012).
Participants are also particularly good at identifying
complex mental/emotional states from the eye
regions alone, such as playful, insisting, and regret-
ful, as well as from the whole face (Baron-Cohen
et al., 1997). Because of this latter finding, com-
bined with knowledge that certain psychopathic
disorders, such as autism, are marked by impaired
social processing and an avoidance of the eyes,
the Reading the Mind in the Eyes Test (RME) was
developed to assess abilities to decode mental states
from the eye region alone (Baron-Cohen, Wheel-
wright, Hill, Raste, & Plumb, 2001; Baron-Cohen
et al., 1997). The RME has been associated with
many different trait-related (particularly autism),
gender, and cultural factors, which are described
in more detail next.
Individual Differences in Reading Facial
Actions Around Eye Region Traits
Although the region around the eyes is a rich
source of social information, people vary widely in
the extent to which they can meaningfully interpret
that information. An inability to process social and
emotional meaning from the eye region is a defining
characteristic of some psychopathological disorders.
This ability and attention to the eye region has also
been found to vary as a function of gender, age,
and culture.
Traits.  The ability to reason about the mental
states of others from the eyes has primarily been
studied in regard to deficits associated with autism.
Adults and children on the autism spectrum perform
worse on this task than typically developed children
(Baron-Cohen et al., 1997, 2001). Even parents of
children who are diagnosed with autism perform
worse than parents of nonautistic children (Baron-
Cohen & Hammer, 1997). Some evidence suggests
that the limitations for individuals with autism
are limited to more complex mental state reason-
ing, such as considering whether others are being
thoughtful or arrogant. For instance, individuals
with autism do not show the same degree of deficits
Eye Behavior and Gaze
when decoding basic emotions (Adolphs, Sears, &
Piven, 2001).
In addition to autism, the RME has been used
to assess impairments in other disorders that are
accompanied by social difficulties. For instance,
individuals with schizophrenia will show deficits
on the RME (Irani et al., 2006; Russell et al., 2000).
These deficits are accompanied by hypoactiva-
tion of the left inferior frontal gyrus (Russell et al.,
2000). It is believed that deficits in theory of mind
contribute to symptoms of schizophrenia (see
Brüne, 2005, for review). Additionally, individu-
als who are higher in the possession of borderline
personality traits, which are associated with emo-
tional volatility and negative social outcomes, show
enhanced performance on the negative stimuli rela-
tive to control participants (Scott, Levy, Adams, &
Stevenson, 2011).
Gender.  Consistent, yet small, gender differences
have been found on the RME (Kirkland, Peterson,
Baker, Miller, & Pulos, 2013), with females tending
to score higher than males. When factoring in the
gender of the eyes, male participants show deficits
in RME performance for female eyes, and this was
accompanied by decreases in their limbic system
activation (Schiffer, Pawliczek, Müller, Gizewski, &
Walter, 2013). Most of the research has implicated
sex differences as a causal mechanism. Notably, fetal
testosterone levels, which differ strongly between
males and females in utero, predict children’s perfor-
mance on the RME (Chapman et al., 2006).
Age.  Only one study, to our knowledge, has
examined age-related changes in RME perfor-
mance. Relative to older adolescents and young
adults, young adolescents (10–12 years of age)
demonstrate similar performance on the RME task.
However, they demonstrate increased activation of
frontal areas, including the inferior frontal gyrus,
ventromedial prefrontal cortex, and temporal pole,
likely in an effort to overcome a lack of experience
or underdevelopment of the social brain (Gunther
Moor et al., 2012).
Context and culture.  Only one study, to our knowl-
edge, has directly looked at cross-cultural influences
on mental state reasoning from the eye regions.
337
Adams and Nelson
Adams, Rule, et al. (2010) demonstrated that behav-
iorally, White American participants demonstrated
greater performance on the RME for White rela-
tive to Asian eyes, whereas the opposite pattern
was found for Japanese participants. In addition,
these differences were accompanied with greater
posterior superior temporal sulcus (STS) activa-
tion for same-culture relative to other-culture faces.
Follow-up work, using university identification as a
social group, suggested that these differences may be
due to motivation associated with group identifica-
tion. When U.S. White university students identi-
fied Asian stimulus persons as being a part of one’s
ingroup (i.e., students that attend their university),
they were decoded as accurately as the ingroup
White stimulus persons (M. T. Stevenson, Soto, &
Adams, 2012).
Additionally, culture interacts with gender to
influence mental state reasoning from the eyes. For
instance, in follow-up analyses from functional mag-
netic resonance imaging data collected by Adams,
Rule, et al. (2010), no significant behavioral gender
differences were found, yet men and women did
engage in differential neural processing as a function
of culture of the eyes being presented. Men showed
a greater difference than women between ingroup
and outgroup eyes, with more mentalizing-related
activation to ingroup eyes. This gender difference
was found among White Americans as well as Japa-
nese participants (Franklin, Stevenson, Ambady, &
Adams, 2015).
THE LESSER STUDIED EYE BEHAVIORS
As indicated earlier, much of the research on the
role of eyes in nonverbal perception has centered on
the facial muscle patterning around the eyes, not the
actual behavior of the eyes themselves. In fact, many
researchers have equated the two referring to the
behavior surrounding the eyes as “the eyes”
(e.g., eye region: Baron-Cohen et al., 2001; eye
lids: Matsumoto, 1989), concluding that the “eyes
have it” with regard to emotion perception. Criti-
cally, this work demonstrates that the eye region
is a highly salient facial cue. Yet, as already noted,
the eyes can produce distinct behaviors known to
convey social and emotion meaning. We begin with
338
lesser studied aspects of eye behavior, tear produc-
tion and pupil dilation/constriction, before moving
onto gaze behavior.
Tears
At the end of any major sporting event, you will
often see the flow of tears on both sides of the field.
The winners shed tears of joy as they celebrate a
hard-earned victory, whereas the losers shed tears of
sadness after not achieving their goals. What causes
this seemingly similar response to two different ante-
cedents? How are these tears perceived by others?
How, when, and why tears occur.  Tears are pro-
duced by the lacrimal glands of many vertebrates as
a method of lubricating the eyes (Frey & Langseth,
1985). There are three types of tears: basal, reflexive,
and emotional (Murube, Murube, & Murube, 1999).
Basal tears provide a lubricated layer of lacrimal
film over the cornea of the eye. Reflexive tears are
produced automatically in response to an intense
physiological stimulus. This response helps to flush
out foreign objects and to protect the eyes. Finally,
emotional tears are produced in response to an emo-
tion-eliciting event. Emotional tears are believed to
belong solely to humans (Frey & Langseth, 1985).
Emotional tears may play a role in amplify-
ing the intensity of the emotional experience. For
instance, participants who spontaneously cried in
response to a sadness-inducing video demonstrated
more autonomic nervous system activity relative to
a precrying baseline (Gross, Fredrickson, & Leven-
son, 1994). This may serve as a motivational mech-
anism to drive behavior to eliminate the aversive
feeling. Additionally, they may serve a cathartic, or
homeostatic, purpose. For instance, levels of respira-
tory sinus arrhythmia—a measure of vagal control
that is associated with self-control—increases after
crying in nondepressed individuals (Rottenberg,
Wilhelm, Gross, & Gotlib, 2003).
Dysfunction in tear secretion reveals important
functions served by the production of tears. For
instance, Sjögren’s syndrome is a disorder that pre-
vents the lacrimal gland from producing a sufficient
amount of tears. One major side effect of Sjögren’s
syndrome is the inability to produce emotional tears.
Consequently, many individuals with Sjögren’s

syndrome are also diagnosed with alexithymia, or
an inability to properly express and experience emo-
tions, and are more at risk for depression (H. A.
Stevenson, Jones, Rostron, Longman, & Field,
2004). This suggests that the act of shedding tears
serves as a release valve of emotional buildup.
The social dynamics of tears.  Not only do tears
serve important physiological functions, but they
are perceived as an informative social cue. For
instance, when presented with sad faces, those with
tears removed were rated as significantly less sad
than faces with the tears. Additionally, the tear-
less faces were rated as more ambiguous in valence
(Provine, Krosnowski, & Brocato, 2009). Hasson
(2009) argued that tears evolved partly as a social
signal of appeasement and a call for help. Because
tears obscure vision, Hasson claimed that tears can
signal to others that aggressive acts toward them are
unlikely. This self-handicapping signals submission,
making an approach from another to offer assistance
more likely.
Individual differences in tears.  We now turn our
attention to individual differences that have been
associated with tearful behavior and perception.
Traits.  Surprisingly, individuals who are
depressed are not any more likely to cry in response
to an intensive sad stimulus than nondepressed indi-
viduals (Rottenberg, Gross, Wilhelm, Najmi, & Gotlib,
2002). However, depressed individuals who do cry do
not show a decrease in respiratory sinus arrhythmia,
which (as noted earlier) is important in self-regula-
tion (Rottenberg et al., 2003). Thus, shedding tears
appears to not help individuals who are depressed.
Additionally, there are personality differences associ-
ated with crying. Individuals who are higher in neu-
roticism report more crying episodes. Additionally,
extraverted individuals are more likely to report feel-
ing cathartic relief from crying (De Fruyt, 1997).
Gender.  Numerous studies have shown that
women tear up more than men (De Fruyt, 1997;
Fischer, Rodriguez Mosquera, van Vianen, & Manstead,
2004; Frey & Langseth, 1985; Vingerhoets, Cornelius,
Van Heck, & Becht, 2000). One suggested reason
for this is that women hold less power and status in
society, and as a result, they are expected to display
more submissive expressions (i.e., sadness), which
Eye Behavior and Gaze
are more associated with emotional tearing (Fischer,
Eagly, & Oosterwijk, 2013). Men, on the other
hand, are expected to be stoic and to not display
their emotions (Fabes & Martin, 1991). When they
do shed tears, men who do so gently (i.e., not sob-
bing) are viewed more favorably (Warner & Shields,
2007); however, this appears to be moderated by
some additional factors described later.
Perceptions of and responses to tears differ
depending on whether they are being displayed by a
male or a female. For instance, Fischer et al. (2013)
found that when information about the catalyst of
a crying episode was not given, participants rated
women as less stoic and more sad than males. This
suggests that gender stereotypes are used to fill in
the gaps in context. Therefore, without context,
crying hurts perceptions of women more than men.
However, when contextual details are given, crying
behavior is evaluated based on the appropriate-
ness of displaying such behavior in that particular
context. In a work context, crying men are judged
as more emotional and less competent than crying
women, a judgment that would likely lead to nega-
tive outcomes for men. However, in the context of a
relationship, where crying is more acceptable, men
and women are not perceived differently. Thus,
the context around crying behavior is important in
determining how crying in males will differ relative
to females.
Further evidence suggests that endorsement of
gender roles moderates crying. For instance, men who
endorse gender roles are less likely to cry than those
who do not (Ross & Mirowsky, 1984). The ten-
dency for men not to express “powerless” displays,
such as crying, has also shown strong cross-cultural
consistency (Fischer et al., 2004). Expectations
concerning the amount of tear flow also affect per-
ceptions across genders. For instance, philosopher
Immanuel Kant noted that tears should simply glis-
ten a man’s eye to demonstrate his sensitivity but
the full flow of tears would be inappropriate (see
Tuppini, 2012). Tears also exert a perceptual impact
through chemosignals. For instance, men that sniffed
the odorless tears of women produced during a nega-
tive emotive state showed decreases in sexual arousal
(Gelstein et al., 2011) and decreases in testosterone
(Gelstein et al., 2011; Oh, Kim, Park, & Cho, 2012).
339
Adams and Nelson
This may lead to a shift in motivation toward help-
ing and assistance.
Age.  Crying is a behavior primarily ascribed
to infants and young children. Tears shed by young
ones serve to promote bonding and to elicit helping
behavior from caretakers (Nelson, 2005). The pro-
duction of tears in crying begins to appear around
3–4 weeks after birth (Illingworth, 1955). Children
also learn to use tears to their advantage. For
instance, children will use voluntary crying as a way
to get their parents’ attention, particularly during
the toddler years (Illingworth, 1951, 1955).
Crying among older adults is more common than
it is among younger adults. The reasons for crying,
however, are quite varied. Although there may be
increased levels of depression and sadness among
older adults, particularly as health declines and the
loss of loved ones becomes more probable, older
adults cry for other reasons as well. For instance,
tears are commonplace when an older adult reflects
back on his or her own life. Often, these tears are
associated with a deep emotional experience rather
than sadness (Trice, 1990). Additionally, gender dif-
ferences appear to change in older adulthood. Older
adult women appear to cry as frequently as younger
adult women, but older adult men cry more than
younger men (Hastrup, Baker, Kraemer, & Born-
stein, 1986). This likely reflects downward changes
in status and power for older adult men.
Context and culture.  Emotional tears have
been found throughout every known culture in
history (Lutz, 2001). Despite this, there are sig-
nificant cross-cultural differences that shape tear-
ful responses to various eliciting events. The role
that culture plays in crying was not lost on Charles
Darwin (1872/1965), who was quoted as saying that
women from New Zealand “can voluntarily shed
tears in abundance; they meet for this purpose to
mourn the dead, and they take pride in crying ‘in
the most affecting manner’” (p. 156).
By and large, the acceptance of shedding tears is
tied with cultural norms regarding the expression
of emotions more broadly. Thus, in many Western
cultures, where men are expected to mask their
emotions (Fabes & Martin, 1991; Shields, 2002),
this too carries over to crying. One major cultural
difference in crying behavior involves responses to
340
the death of loved ones. For instance, in Victo-
rian culture, religious influences restricted crying
because it demonstrated too much of a focus on the
current life and too little focus on the afterlife (see
Vingerhoets & Cornelius, 2001). Of course, there are
differences within religions as well. For instance, in a
comparison of two different predominantly Muslim
countries, Egypt and Bali, much different responses
to death are found. Tears are readily shed in Egypt,
whereas tears are held back in Bali (Wikan, 1988).
Even when one’s own child dies, Balinese parents
shed few tears, further underscoring how strong of
an impact that culture has on displays of tears.
Pupils
The pupils are another potential dynamic source
of social information. Pupillary dilation is a physi-
ological marker of heightened arousal (Hess & Polt,
1960) and, as such, may be co-opted as a social
cue. Surprisingly, very little research has examined
pupillary dilation from a perceptual standpoint.
The vast majority of work that has been done has
looked at the effects of pupil dilation on attractive-
ness, though some recent work has been done on
emotion perception as well, which is outlined later.
How, when, and why pupillary behavior
occurs.  The pupil is the opening at the center of
the iris that admits light into the eye. The constric-
tion and dilation of the pupil is primarily mediated
by autonomic nervous system control exerted on the
muscles of the iris, the sphincter pupillae, and the
dilator pupillae. These movement patterns form the
basis of several optical reflexes, including reflexes in
response to light and to psychosensory stimulation.
Early work conducted by Eckhard H. Hess
and colleagues popularized the study of pupillary
responses to social stimuli and as a signal of social
responsivity. Hess and Polt (1960) presented male
and female subjects with various pictures—photos
of a baby, a mother with a baby, a nude male, a nude
female, and a landscape—and pupil size was mea-
sured after viewing each picture. Men showed great-
est dilation to the nude female photos, less dilation
to the mother and the baby, and minimal response
to the baby. Conversely, women showed the greatest
dilation to the nude male. While the baby and the

mother/baby pictures elicited less dilation, women
still showed significantly greater response to those
images than men did. Pupil dilation also increases
in responses to the visual presentation of two con-
centric circles (eye-like shapes) as opposed to one
or three concentric circles, evidencing the possibil-
ity of an innately prepared tendency for responding
to, and interest in, eye-like stimuli (Hess & Petro-
vich, 1987). The researchers concluded that pupil
size could be used to index level of interest in a
visual stimulus.
Pupillary constriction is often found in response
to viewing negative emotional stimuli (e.g., muti-
lated bodies or crippled children; Hess, 1965).
Hess (1972) concluded, “There is a continuum of
pupil responses to stimuli, ranging from extreme
dilation for interesting or pleasing stimuli to
extreme constriction for material that is unpleas-
ant or distasteful to the viewer” (p. 511). This
contention was supported in a number of other
studies (see Metalis, Hess, & Beaver, 1982). In a
meta-review of this work, however, Janisse (1974)
challenged Hess’s “bidirectional” view and argued
that pupil size is linearly related to the affective
intensity of a stimulus, and thus pupil size varies
curvilinearly as a function of valence: Dilation is
largest at the negative and positive ends of the con-
tinuum and smallest in the center, where minimal
stimulation occurs. Loewenfeld (1993) similarly
argued that emotion, sensory stimulation, spon-
taneous thoughts, and physical/mental effort all
elicit pupillary dilation. She noted, however, that
only increases in light intensity elicit pupillary con-
striction. However, recent work has demonstrated
that observing constricted pupils on sad faces also
leads to automatic mimicry of pupil constriction in
perceivers (Harrison, Singer, Rotshtein, Dolan, &
Critchley, 2006).
The social dynamics of the pupillary
response.  Recent neuroimaging work is demon-
strating the signaling value of pupils. The amyg-
dala is sensitive to perceiving pupil size, even in
the absence of subjective awareness (Amemiya &
Ohtomo, 2012; Demos, Kelley, Ryan, Davis, &
Whalen, 2008; Harrison, Gray, & Critchley, 2009).
Additionally, amygdala activation is found in
Eye Behavior and Gaze
response to human (but not cat) eyes, suggesting
that the response is driven by relevance. Given the
amygdala’s role in many aspects of emotion pro-
cessing and self-relevance (N’Diaye, Sander, &
Vuilleumier, 2009), it seems likely that the pupil size
of others is a social cue that can be utilized to under-
stand others’ intentions.
Historically, the pupils have played an important
role in perceptions of beauty for women. As far back
as the 15th century, Italian women were putting
Belladonna (Italian for “beautiful woman”) drops in
their eyes, dilating their pupils to make themselves
appear more attractive (Heiser, 1987). Heterosexual
males judged the female faces with larger pupils to
be more attractive (Bull & Shead, 1979; Hess, 1965;
Stass & Willis, 1967; Tomlinson, Hicks, & Pel-
legrini, 1978). From a sexual selection perspective,
it makes sense that men seek out women who are
attracted to them, given that their pupils increase
when they look at attractive males (Cunningham,
1986; Hess & Petrovich, 1987; Tombs & Silver-
man, 2004). The effects have been more mixed for
male attractiveness (Tombs & Silverman, 2004),
though pupillary responses for gay men match
those found in straight men, with greater responses
toward sexually explicit pictures of those whom
they are sexually attracted to. Homosexual females
also show a preference for female stimuli, but het-
erosexual females do not show differential pupillary
dilation to male versus female stimuli (Rieger &
Savin-Williams, 2012). Thus, it appears that gen-
der differences are important when examining the
attraction–pupil dilation link.
Recent research has implicated pupil size as a
factor in perceiving emotional expressions. For
instance, the perceived intensity and valence of sad
facial expressions are moderated by pupil size, with
smaller pupils leading to higher ratings for both
intensity and sadness (Harrison et al., 2006; Har-
rison, Wilson, & Critchley, 2007). Furthermore,
participants mimic the pupil size of the stimulus
(Harrison et al., 2006), and the degree of this
relationship predicts levels of emotional empathy
(Harrison et al., 2007). Interestingly, no other basic
expressions were impacted by pupil size. Pupil size may
only moderate perceptions of sadness because of the
importance of the eye region in identifying sadness.
341
Adams and Nelson
Additionally, sadness is a short-distance emotion
designed to draw others in to assist. Other emotions
(e.g., joy, anger), which are designed to communi-
cate across longer distances, may not be impacted by
minor changes in pupil size.
Another social factor that has been implicated
in pupillary behavior is deception. An early study
by Berrien and Huntington (1943) demonstrated
that participants who were engaging in deception
had an initial period of pupillary dilation (∼1–5 s),
followed by a return to baseline. More recent
research has corroborated this dilation (Dionisio,
Granholm, Hillix, & Perrine, 2001) and has argued
that this is likely due to the extra mental effort
exerted. Thus, it is possible that pupil dilation
can be used to detect deception, although no evi-
dence exists yet that humans use this information
spontaneously.
Individual differences in pupillary responses and
perception.  We now turn our focus to individual
differences in pupillary behavior.
Traits.  By and large, most research has looked
at pupillary responses that differ across individu-
als as a function of physical and psychological
health. For instance, illness (Loewenfeld, 1993),
chronic fatigue (Pressman, Diphillipo, & Fry,
1986), neuroticism (Rubin, 1970), and depression
(Sokolski, Nguyen, & DeMet, 2000) all diminish
pupillary dilation reflexes. However, the degree to
which humans spontaneously utilize pupil size as
a cue for these conditions has not been addressed.
Individuals with autism show less pupillary
responsiveness to faces while maintaining a higher
baseline level of pupil dilation than nonautistic
controls (Anderson & Colombo, 2009). Although
depression diminishes pupillary dilation broadly,
individuals who are depressed will actually show
greater dilation in response to emotional stimuli, in
particular stimuli that are negative and self-relevant
(Siegle, Steinhauer, Carter, Ramel, & Thase, 2003).
Gender.  Females, but not males, show greater
pupillary dilation in response to seeing direct
gaze faces versus averted gaze faces (Porter, Hood,
Troscianko, & Macrae, 2006). The authors sug-
gested that this response is an autonomic measure
of increased processing of direct gaze—hence,
342
socially relevant—faces. In response to high levels of
pain, women show greater pupil dilation than men
(Ellermeier & Westphal, 1995).
Age.  Infants begin demonstrating reflexive
pupil dilation to light approximately 1 day after
birth (Sherman & Sherman, 1925). By 1 month
of age, they demonstrate greater pupillary dilation
to social stimuli, including more responsiveness
to images of their mother relative to a stranger
(Fitzgerald, 1968). Additionally, infants demonstrate
pupil dilation to unexpected events (Jackson &
Sirois, 2009). Older adults tend to exhibit less pupil-
lary responsiveness to psychosensory stimulation as
well as possessing a smaller absolute pupil size. In
young people, however, absolute pupil size is larger,
and the same pupillary reflexes are stronger than in
healthy adults (Bernick, 1972; Hess & Petrovich,
1987). In a memory load task, young adults showed
greater pupillary dilation to greater cognitive load at
retrieval relative to older adults, even though there
were no age differences at encoding (Van Gerven,
Paas, Van Merriënboer, & Schmidt, 2004).
As mentioned earlier, pupillary dilation has been
differentially associated with sexual orientation and
gender (Rieger & Savin-Williams, 2012). Hetero-
sexual and homosexual males as well as homosexual
females show differential pupillary dilation to sexu-
ally explicit images depending on their sexual pref-
erences, whereas heterosexual women do not.
Context and culture.  We know of no cross-
cultural research that has been conducted looking at
pupils. Given the historical practice of beautification
via administration of belladonna to induce pupillary
dilation (Feinsod, 2000), it is possible that percep-
tions of pupils may be universal. One potentially
fruitful area of research may include an examination
of the influence that iris color has on social percep-
tion. The iris is responsible for constricting/dilating
the pupil. Light-colored irises (e.g., blue, green) are
found predominantly in individuals of European
descent (Sturm & Larsson, 2009). Individuals
with light-colored irises tend to have larger pupils,
and their pupils tend to dilate more in response to
stimuli (see Hess & Petrovich, 1987). As such, it
may be interesting to look at how this may factor
into cross-cultural standards of beauty and emotion
perception.

EYE GAZE BEHAVIOR
We now move our focus to a much more studied
aspect of eye behavior—gaze. Despite of the subtlety
of movement in eye gaze (mere millimeters), adults
are more sensitive to direct eye gaze than postural
behaviors, and they tend to become frustrated when
others do not respond to their looks (Ellsworth,
1975). Direct gaze indicates that another is attend-
ing to us, and elicits our attention toward him or
her. When gaze is averted, it can be inferred that
the person doing the looking is having some kind of
mental or emotional experience about that object of
his or her attention that may be informative to our
own goals.
How, When, and Why Gaze Behavior
Occurs
Six muscles, called the extraocular muscles, control
eye movements. There are four rectus muscles that
control left to right and up and down movement,
and two oblique muscles that move the eyes inward
toward each other versus outward away from each
other, involved in visual focus and depth perception
(see also Chapter 20, this handbook). With these
simple movements, eye gaze conveys much social
information and function. The eyes provide people
with an important source of sensory information.
Various species appear to have developed perceptual
capacities to process the eyes of conspecifics. Like-
wise, the morphology of the eyes has likely evolved
to more effectively communicate. For example,
some have argued that humans have evolved a con-
trasting white sclera to facilitate detecting gaze of
others (Kobayashi & Kohshima, 1997).
The Social Dynamics of Gaze Behavior
There are a number of ways that eye gaze detection
has been argued to be adaptively important (Baron-
Cohen, 1995): (a) It can alert one to food sources,
rivals, mates, and predators that are otherwise hid-
den from sight; (b) it can serve an affiliative func-
tion, through inviting intimacy or communicating
the need for help; and (c) it can serve to maintain a
stable social hierarchy, as the more dominant mem-
bers of the group will be socially referenced more
often thereby communicating group information
Eye Behavior and Gaze
regarding who to defer to and who is allied with
whom, without the need for aggressive contact to
be made (i.e., “attention structure”; Chance, 1967).
Historically, eye gaze behavior has been studied
from two perspectives: as a channel for information
gathering and as a signal of nonverbal information
(Argyle & Cook, 1976). As a channel for informa-
tion gathering, eye gaze conveys where others are
attending, and people can use this information to
infer what they are likely thinking or feeling toward
the object of their gaze. As a social signal, research
has focused on the meaning conveyed by gaze. For
this reason, we include two sections on gaze, focus-
ing on (a) gaze as a channel for information gather-
ing and (b) gaze as a social signal.
Gaze as a Channel for Information
Gathering
Eye gaze tells where in the environment someone is
looking. It allows an individual to follow the atten-
tion of another to some other point in the environ-
ment. Such shared, or joint attention, has been
demonstrated in monkeys and apes as well as in
human infants and adults (see Frischen, Bayliss, &
Tipper, 2007, for review). Shared attention appears
to be an obligatory behavior, necessarily occurring
in response to perceiving another’s averted gaze
(Driver et al., 1999; Perrett & Emery, 1994).
While seeing another’s gaze directed toward
us signals that we are the target of another’s atten-
tion, averted gaze reveals that his or her attention is
directed toward a different object. It is possible that
this individual has spotted a threat in the environ-
ment or, perhaps, is looking at a rewarding stimulus
(e.g., attractive individual). Or perhaps the individ-
ual is talking about something and is looking toward
said object. Regardless of what is being looked at,
it is easy to see the adaptive function that follow-
ing the gaze of another can be. If he or she thinks
it is important enough to look at, it may be just as
important for us to look at as well.
Gaze cueing, or following one’s gaze to a spe-
cific target, has been widely studied using visual-
attention and eye-tracking paradigms. In general,
seeing a face with its gaze averted to the side will
lead to facilitated identification of a target in the
gazed at location (C. K. Friesen & Kingstone, 1998).
343
Adams and Nelson
This effect is so powerful that even instructing par-
ticipants that the target is four times more likely
to appear opposite of the gazed at location does
not reduce early attentional biases toward gazed at
locations (Driver et al., 1999). The primacy of gaze
following can be seen in its existence across numer-
ous species of primates (Bräuer, Call, & Tomasello,
2007; Tomasello, Call, & Hare, 1998) and in human
infants (Farroni, Massaccesi, Pividori, & Johnson,
2004; Hood, Willen, & Driver, 1998; Reid & Stri-
ano, 2005). Even goats can utilize gaze following of
other goats to find food (Kaminski, Riedel, Call, &
Tomasello, 2005).
The findings have been mixed regarding whether
eye gaze is actually a special cue in this regard. Sym-
bolic cues (i.e., arrows) can also trigger gaze shifts
following the direction of the cue (Kuhn & Benson,
2007; Kuhn & Kingstone, 2009). Despite such find-
ings, Hietanen, Nummenmaa, Nyman, Parkkola,
and Hämäläinen (2006) found, using functional
magnetic resonance imaging, that cueing via eye
gaze versus arrows yield differential activations in
the brain. Gaze cueing is primarily mediated by
bilateral activity in the inferior occipital gyrus and
right medial occipital gyrus. Arrow cueing, how-
ever, shows more extensive activations, including
those subsumed by gaze cueing, along with activ-
ity in medial temporal and left intraparietal sulcus.
This suggests that even if gaze cues and arrow cues
do similarly cue attention, they may do so using
different processes, with gaze cueing eliciting
more automatic shifts of attention, whereas arrow
cueing appears to be driven by more voluntary
attentional shifts.
According to Baron-Cohen (1995), the innate
capacity to process gaze information plays a critical
role in the development of a form of social cognition
referred to as theory of mind. He considered it a crit-
ical mechanism for forming high-level dyadic rep-
resentations of an agent and its object of attention.
He proposed four mechanisms that he argued were
evolved via natural selection and constituted what
he referred to as the “mindreading” system, which
enables individuals to make attributions concerning
the mental states and behavioral intentions of others
based largely on the capacity to detect and follow
gaze. The first three of these systems rely heavily on
344
the ability to perceive eye gaze direction. The fourth
and ultimate theory of mind mechanism, which sub-
sumes the other mechanisms, represents an ability
for “epistemic” mental states—such as dreaming,
deception, pretending, fantasy, and belief—and is
not discussed in detail here. The three gaze-related
mechanisms proposed by Baron-Cohen are dis-
cussed next include the (a) Intentionality Detector,
(b) Eye Direction Detector, and (c) Shared Attention
Mechanism.
The Intentionality Detector is considered a
mechanism that allows for motion to be interpreted
in terms of volitional mental states driving goals or
behavioral intent. A neurocognitive module con-
sistent with an intentionality detector would need
to be preferentially sensitive to biological versus
random or object motion. Such cells have in fact
been isolated in the STS of monkeys in response to
viewing whole body locomotion as well as hands
interacting with objects (Bonda, Petrides, Ostry, &
Evans, 1996; Bruce, Desimone, & Gross, 1986; Per-
rett & Emery, 1994). Similarly, cells in areas of the
amygdala of monkeys have been found responsive
to viewing complex social interactions of conspecif-
ics, such as viewing behavior that is characteristic
of a social approach intention (Brothers & Ring,
1992). Amygdala activation has also been associ-
ated with high-level encoding of facial cues, such as
when viewing a conspecific’s direct gaze or an open
mouth expression (Brothers & Ring, 1993). Broth-
ers and Ring (1993) concluded that the amygdala
activation found in these examples likely represents
the integration of activation patterns passed on from
the STS, because such input would be necessary for
generating an appropriate behavioral response to
complex social interactions. In humans, seemingly
homologous STS activation to that just described
in monkeys has been found using positron emis-
sion topography for the viewing of both whole body
biological motion and goal-directed hand action
(Bonda et al., 1996). STS activation has also been
found in response to eye and mouth movement
(Puce, Allison, Bentin, Gore, & McCarthy, 1998;
S. Watanabe, Kakigi, & Puce, 2001) as well as in
response to static eye gaze detection (Hoffman &
Haxby, 2000; Sato, Kochiyama, Uono, & Yoshikawa,
2008). Based on these findings, Hoffman and Haxby

(2000) concluded that the STS is likely associated
with the processing of “changeable” aspects of face,
even static representations thereof, such as those
found in the muscle patterning of emotional facial
expressions.
The Eye Direction Detector, like the Intentional-
ity Detector, is considered an important mechanism
for forming dyadic representations of an agent and
its object of attention. Information from these two
mechanisms are further argued by Baron-Cohen
(1995) to be integrated, allowing gaze to be inter-
preted in terms of desire or intent thereby making
visual perception more meaningful (e.g., she is
looking at me/she likes me). Before sharing atten-
tion and drawing inference, one must first be able to
distinguish between direct and averted gaze. Infants
as young as 2 days old are able to discern gaze direc-
tion (Farroni, Csibra, Simion, & Johnson, 2002),
suggesting this ability is innately prepared.
Although research suggests that staring eyes
are generally perceived as threatening stimuli across
a number of species, the eye-spot configuration has
also been documented as a means of sexual attrac-
tion for some species, including peacocks and male
guppies (see Argyle & Cook, 1976, for review).
Similarly, in most primates, direct eye contact
accompanies anger and attack, whereas averted eye
gaze generally accompanies fear and flight (Hinde &
Rowell, 1962; Redican, 1982; van Hooff, 1976);
yet, in many Old World monkeys and apes, direct
eye gaze has been reported to also serve an affili-
ative function, such as for greeting and grooming
(van Hooff, 1972). The notion of an eye direction
detector presumes that there are brain cells special-
ized for the processing of different eye gaze direc-
tions. In support of this contention, Perrett and
colleagues (Perrett & Emery, 1994; Perrett et al.,
1985) found cells in areas of the STS that are sensi-
tive to particular eye gaze directions (note that these
cells neighbor those responsive to biological motion
discussed in the last section). In fact, many of these
cells have been found to be simultaneously sensi-
tive to head and body orientation, though they are
preferentially sensitive to eye gaze (Perrett et al.,
1985). Furthermore, reciprocal projections from the
STS to the parietal cortex, implicated in the process-
ing of spatial awareness (Harries & Perrett, 1991),
Eye Behavior and Gaze
suggest that gaze direction perception is obligatorily
involved in reflexive orienting in the direction of
another’s gaze. Likewise, in humans similar STS and
parietal activations have been documented during
eye gaze detection (Hoffman & Haxby, 2000).
Building on the ability to detect eye gaze direc-
tion, the Shared Attention Mechanism is defined
as the compulsion to follow another’s line of sight.
Shared attention in this sense requires eye direction
perception to form a triadic representation of an
agent and its object or goal in relation to the self.
This mechanism therefore allows an individual
to determine whether an object that he or she is
attending to is the same object as that which another
is attending to. It also allows an individual to follow
the attention of another to some other point in the
environment. As discussed earlier, such shared
(or joint) attention has been demonstrated in mon-
keys and apes as well as in human infants and adults
(see Frischen et al., 2007, for review)
Individual Differences in Gaze as
Information Gathering
Finally, there are a number of individual differences
associated with the directing of one’s gaze as well as
in the ability to decode and utilize this information.
Traits.  The face communicates the identity, emo-
tional state, and direction of attention of others.
Because of this, visual scanning of the face has been
a topic of interest to researchers. In particular, exam-
ining how normal scanning of faces occurs relative
to scanning abnormalities associated with psycho-
pathologies has been fruitful in terms of identifying
deficits in social functioning that are associated with
these various disorders. Autism is a disorder that is
characterized by deficits in face processing (Adolphs
et al., 2001). One area where this is particularly
relevant is in visual scanning of faces. In general,
people look at features of the face when engaging in
face processing, including eyes, nose, and mouth.
However, individuals with autism spend more time
looking at nonfeatures (Pelphrey et al., 2002). They
will look more at the mouth region, whereas con-
trols will look more at the eye region, when engag-
ing in general face processing (Klin, Jones, Schultz,
Volkmar, & Cohen, 2002) as well as emotion
345
Adams and Nelson
recognition (Spezio, Adolphs, Hurley, & Piven,
2007) tasks. This effect may be driven by amygdala
abnormalities, which are associated with autism
(Baron-Cohen et al., 2000). Amygdala-lesioned
patients make less eye contact (Spezio, Huang,
Castelli, & Adolphs, 2007), and autistic individu-
als show less amygdala and FFA activation, except
when looking at the eye region (Dalton et al., 2005).
Schizophrenia is another disorder that is charac-
terized by deficits in face processing and dysfunc-
tional visual scanning of faces (Streit, Wölwer, &
Gaebel, 1997). Like autistic individuals, schizo-
phrenics tend not to look at the core features of the
face when making expression judgments (Lough-
land, Williams, & Gordon, 2002). This is reflected
in an overall reduction in scanning behavior. Some
studies have found that individuals with social pho-
bia also avoid the eye region, as well as other core
features, when making expression judgments (Hor-
ley, Williams, Gonsalvez, & Gordon, 2003, 2004).
In contrast to people with schizophrenia, people
with social phobia show an increase in scanning
while avoiding fixations on features of the face
(but see Wieser, Pauli, Alpers, & Mühlberger, 2009).
Gender.  During facial emotion recognition, men
look more at nose and mouth than females (Vassallo,
Cooper, & Douglas, 2009). Conversely, women look
at eyes more, and this is associated with better accu-
racy and faster reaction times when asked to rec-
ognize faces (J. K. Hall, Hutton, & Morgan, 2010).
This may help explain the slight benefit for females
over males in facial affect recognition (J. A. Hall,
1978). Of course, we cannot say for certain whether
this is a biologically hardwired sex difference or
whether socialization for females to be more sensi-
tive to facial affect leads to their adoption of a supe-
rior recognition strategy. It is interesting, however,
to see males show similar patterns of facial scanning
as autistic individuals, given the conceptualiza-
tion of autism as an “extreme male-brain” disorder
(Baron-Cohen, 2002).
Age.  With regard to gaze following, infants will
follow the gaze of an experimenter but only when
preceded by communication with the infant,
either through mutual eye contact through direct
gaze (Farroni, Mansfield, Lai, & Johnson, 2003)
346
or speech directed at the infant (Senju & Csibra,
2008). This suggests that gaze following in infants
is contingent on its communicative relevance to
the infant. In terms of looking behavior, older
adults tend to demonstrate a positivity bias, where
they show a greater propensity to engage with
positive stimuli while avoiding negative stimuli
(Carstensen & Mikels, 2005; Mather & Carstensen,
2005). Older adults may show this positivity bias
because, as they approach the end of their lives, their
goals shift from the acquisition of resources and
information to the enhancement of present experi-
ence (Carstensen, Isaacowitz, & Charles, 1999).
In general, older adults show a positivity bias
in visual attention by showing preferential looking
behavior toward positive images and an avoidance
of negative images (Isaacowitz & Murphy, 2010).
Mather and Knight (2005) argued that although
there are predispositions to focus on negative stim-
uli, older adults are able to control these predisposi-
tions in favor of positive stimuli. In line with this,
they later found that divided attention reverses the
positivity bias in looking behavior for older adults
(Knight et al., 2007). This suggests that sufficient
cognitive resources are necessary to control the
natural tendency to focus on negative stimuli. This
is further supported by the finding that positivity
biases are not evident until 500 ms poststimulus
onset (Isaacowitz, Allard, Murphy, & Schlangel,
2009), which is beyond initial visual orienting
(Posner, 1980). Despite its controlled nature, posi-
tivity biases in looking behavior are relatively effort-
less (Allard, Wadlinger, & Isaacowitz, 2010).
Context and culture.  Cultures provide display
rules that dictate when, where, and how much of
an emotion one may express (Matsumoto, 1990;
Matsumoto, Yoo, & Fontaine, 2009; see also
Chapter 4, this handbook). In interdependent/
collectivist cultures (i.e., Japan), individuals are
expected to mask their expressions, whereas people
in independent/individualist cultures (i.e., the
United States) are more freely allowed to express
their emotions (e.g., W. V. Friesen, 1972). Within
the face, however, there are differences in the con-
trollability of expressions. Two prominent muscles
underlying facial expressions are the zygomatic

major, which controls the mouth region, and the
orbicularis oculi, which controls the eye region.
Whereas the zygomatic major can be controlled
when attempting to mask facial expressions, as
noted earlier, it is much more difficult to control
the orbicularis oculi around the eyes (Ekman, 1992;
Ekman & Friesen, 1975). As such, people in cul-
tures that encourage the masking of expressions
would be better off attempting to glean information
from the eyes when attempting to decode facial
affect. Yuki, Maddux, and Masuda (2007) examined
this using emoticons and actual faces. Indeed, they
found that American participants based their inter-
pretation of facial expressions more on the mouth,
whereas Japanese participants based their interpreta-
tions more on the eye region. When making non-
emotion judgments (i.e., identity recognition, race
categorization), Western Caucasians look first at the
eyes and then at the mouth, whereas Eastern Asians
focus more on the center, nose region of the face
(Blais, Jack, Scheepers, Fiset, & Caldara, 2008).
Interestingly, one study found that deaf individu-
als look more at the eyes than hearing individuals
when judging expressions, whereas hearing individu-
als looked more at the nose (K. Watanabe, Matsuda,
Nishioka, & Namatame, 2011). The participants
used in this study were all Japanese, replicating
previous findings that East Asian participants focus
more on the nose than the eyes (Blais et al., 2008).
These findings counter those reported earlier for
Yuki et al.’s (2007) study, which may be explained
in terms of social factors in that attention to the eyes
of other Japanese faces (as opposed to emoticons or
Caucasian faces) may be interpreted as a form of dis-
respect (Argyle & Cook, 1976).
Returning back to the positivity bias in older
adults, positive looking biases in older adults
appears to be moderated by culture. Older Chinese
adults look away from positive facial expressions
(Fung et al., 2008), and this effect is moderated
by level of interdependence (Fung, Isaacowitz,
Lu, & Li, 2010). In interdependent cultures, nega-
tive information may be a cue that group cohesion
is threatened, therefore those who have their self
intertwined with the group may find this more
informative (Fung et al., 2010). This adds further
support for the notion that age-related changes in
Eye Behavior and Gaze
the positivity bias are driven by controlled processes
that can be culturally transmitted.
GAZE AS A MECHANISM FOR SOCIAL
SIGNALING
Ellsworth (1975) suggested that eye gaze does not
carry a specific “sign-referent” meaning and that its
interpretation is dependent on context. That said,
various species, including humans, have developed
special perceptual capacities to process social infor-
mation conveyed by the eyes. Just as the region
around the eyes is believed to have evolved as an
effective social signaling system, so too has eye
gaze behavior, though it does appear that the eyes
are sensitive to combinatorial processes, including
the integration of shared signals conveyed by other
facial and contextual cues. Next, we review evidence
for the signal value of gaze across various domains
of social exchange.
Gaze as a Signal of Self-Relevance
When others make eye contact with us, they signal
that we are the target of their attention. Anything
they are thinking or feeling likely implicates us in
some way, thereby increasing the self-relevance of
the social information they are conveying. Given that
direct gaze signals social relevance to the perceiver,
social cognition should be differentially impacted by
direct gaze versus averted gaze. One area where this
has been examined is in face memory. Faces with
direct gaze are remembered better by infants (Far-
roni, Massaccesi, Menon, & Johnson, 2007), chil-
dren (Hood, Macrae, Cole-Davies, & Dias, 2003),
and adults (Hood et al., 2003; Mason, Hood, &
Macrae, 2004). Additionally, the cross-race effect,
an effect in which memory for same-race faces is
better than memory for cross-race faces, is only
found when the eyes are looking toward the viewer
and not when they are averted—thus supporting the
notion that direct gaze is a cue of relevance to the
perceiver (Adams, Pauker, & Weisbuch, 2010).
In addition to face memory, the process of social
categorization is facilitated by direct eye gaze. Cat-
egorizing faces by gender is faster, and stereotype-
congruent information is easier to access, when
the faces display direct gaze versus averted gaze
347
Adams and Nelson
(Macrae, Hood, Milne, Rowe, & Mason, 2002). This
effect is also found in children (Pellicano & Macrae,
2009). Additionally, individuals are rated as more
likable and more attractive when they gaze toward
us rather than away from us (Mason, Tatkow, &
Macrae, 2005). Ratings of friendliness and trustwor-
thiness are also influenced by gaze direction, with
direct gaze leading to higher ratings of friendliness,
and with averted gaze leading to lower ratings of
trustworthiness, especially when the perceiver is in a
positive mood and therefore more likely to use heu-
ristics in processing (Wyland & Forgas, 2010).
Gaze direction can also moderate neural activ-
ity in response to other-race faces. When viewing
White and Black faces with direct or averted gaze,
White participants showed greater amygdala activa-
tion to Black faces with their gaze directed at per-
ceivers (Richeson, Todd, Trawalter, & Baird, 2008).
Direct gaze also covertly shifts attention more
toward a Black face that is looking toward them
versus away from them, presumably because the
direct gaze signals an intent to approach, which the
authors argued that in combination with race-ste-
reotypic attributions would increase the perceived
threat-value of the face (Trawalter, Todd, Baird, &
Richeson, 2008). These studies both suggest that,
at least for White participants viewing Black faces,
other race faces can be threatening when gaze, and
therefore attention, is directed at the perceiver.
Cooperative behavior can also be impacted by
the perception of direct gaze. For example, simply
displaying posters with pictures of eyes with direct
gaze in a cafeteria reduced the amount of litter-
ing behavior that ensued (Ernest-Jones, Nettle, &
Bateson, 2011). Interestingly, the verbal message
that was included, either an antilittering message
or unrelated message, had no impact on litter-
ing behavior. The mere reminder that one is being
watched, via presentation of direct eye gaze, is suf-
ficient enough to promote prosocial behaviors.
Gaze as a Signal of Social Affiliation
Argyle and Dean (1965) put forth an equilibrium
model of affiliativeness, suggesting that there is an
optimal level of psychological closeness communi-
cated by immediacy cues. Immediacy cues are ones
that signal the need for intimacy and psychological
348
closeness. When the equilibrium is disrupted, com-
pensation is sought to restore it. Much supporting
evidence comes from studies in which physical
distance was manipulated, whereby increasing
the space between participants has been found to
increase direct gaze (see Kleinke, 1986, for review).
Argyle and Dean manipulated eye contact by having
an experimenter either make eye contact with the
participant or stand with eyes closed. Participants
would more closely approach the experimenter (by
23%) when the experimenter’s eyes were closed ver-
sus open. Additionally, individuals will make less
eye contact with a listener as the level of intimacy
of a topic they are discussing increases (Exline,
Gray, & Schuette, 1965; Schulz & Barefoot, 1974).
According to Patterson’s (1976) intimacy-arousal
conceptually related model of immediacy, imme-
diacy cues also increase arousal in the perceiver. If
the perceiver appraises this arousal as positive, he
or she will reciprocate the immediacy. If he or she
appraises this arousal as negative, compensation will
likely occur, resulting in an effort to restore equi-
librium. Helminen, Kaasinen, and Hietanen (2011)
found support for the first step of this model, show-
ing that perceiving direct eye gaze increases skin
conductance responses, which are associated with
increases in arousal, relative to averted gaze.
Another area in which immediacy should have an
impact is in deception. Because engaging in decep-
tion can be anxiety and guilt-provoking, efforts to
increase psychological distance via “nonimmediacy”
behaviors should occur (Zuckerman, DePaulo, &
Rosenthal, 1981). Interestingly, despite the common
belief that liars can be identified by these behaviors
(i.e., gaze aversion), a meta-analysis by DePaulo
et al. (2003) shows inconsistent evidence. Whereas
analysis of subjective ratings of eye contact showed
that liars make slightly less eye contact than non-
liars, an objective measure of this yielded no effect.
Thus, when determining whether someone is lying
to you, it is important to look beyond his or her eye
gaze. As stated earlier, one promising avenue within
the context of the eyes is pupillary dilation.
Gaze as a Signal of Social Dominance
The maintenance of group cohesion in primates is
essential for a group of individuals to survive.

It reduces the risk of attack from predators, particu-
larly for primates living in open terrains (Eisenberg,
1983). One important method of maintaining this
cohesion in a vertical social structure is to under-
stand one’s status in the hierarchy and to monitor
the behavior of more dominant members of the
group (Chance, 1967). By paying more attention
to dominant members, conflicts can be avoided by
keeping distance away from the dominant member.
Additionally, high-status individuals are more likely
to hold resources needed by the individual (Neu-
berg & Fiske, 1987). As such, it behooves low-status
individuals to pay close attention to high-status
individuals.
Supporting this, low-status patas monkeys
engage in more looking behavior toward higher
status conspecifics, but not vice versa (McNelis &
Boatright-Horowitz, 1998). Subordinate brown
capuchin monkeys engage in more looking behavior
toward all members of the group, whereas domi-
nant members do not (Pannozzo, Phillips, Haas, &
Mintz, 2007). In a task where male rhesus macaques
were able to look toward one target to receive a juice
reward, or toward a visual target of a conspecific,
they were more willing to look toward the visual
target when it depicted a high-status male conspe-
cific. They only looked toward low-status conspecif-
ics after they opted for the juice reward (Deaner,
Khera, & Platt, 2005). In humans, participants also
engage in more looking behavior toward high-status
males than low-status males or females (Maner,
DeWall, & Gailliot, 2008). High-status individuals
in a short video clip were looked at more often and
for longer periods of time (Foulsham, Cheng, Tracy,
Henrich, & Kingstone, 2010). On the contrary,
dominant individuals will look less at subordinate
individuals while listening to them speak (Exline,
Ellyson, & Long, 1975).
Under some conditions, however, dominant
group members can use eye contact themselves as a
tool to encourage subordinates to behave appropri-
ately. Direct gaze can be used as a cue of dominance
(Ellsworth, 1975; Exline, 1971; Exline et al., 1975)
and can serve to increase the potency of a message
(J. A. Graham & Argyle, 1975). Individuals sitting
at the head of a table are in a position of “visual
dominance,” which allows him or her to look at
Eye Behavior and Gaze
all members of a group (Webbink, 1986). Students
in a classroom will behave more when verbal rep-
rimands are paired with direct gaze (Van Houten,
Nau, MacKenzie-Keating, Sameoto, & Colavecchia,
1982). Additionally, individuals (particularly men)
who engage in more direct gaze looking behavior are
rated as more dominant (Hillabrant, 1974; Thayer,
1969) or powerful (Dovidio & Ellyson, 1982). Sub-
ordinate individuals, however, will avert eye contact
when looked at by dominant members (Ellsworth,
Carlsmith, & Henson, 1972). This communicates
submissiveness and serves to avoid an aggressive
encounter with the dominant individual.
Gaze as a Signal of Approach and
Avoidance
Davidson and Hugdahl (1995) stated that “approach
and withdrawal are fundamental motivational
dimensions that are present at any level of phy-
logeny where behavior itself is present” (p. 362).
Because people’s behavior is so fundamentally
motivated by approach and avoidance tenden-
cies, it should not be surprising that visual signals
given off by the face, particularly by gaze, convey
these underlying tendencies. Indeed, when a per-
son makes eye contact with others, it conveys an
increased probability of approach and social engage-
ment (Ellsworth & Ross, 1975; Grumet, 1999),
whereas averted gaze signals active avoidance and
is at times considered itself an act of hiding
(Redican, 1982).
Stern (1977) provided evidence for the signifi-
cance of eye gaze behavior in signaling approach
and avoidance in social development. It is a way
for infants to approach or withdraw from others in
an effort to regulate an ideal level of arousal (recall
that direct gaze has been shown to increase arousal,
whereas averted eye gaze reduces it). In fact, before
learning to crawl, gaze behavior is the primary
mechanism by which an infant can approach or
avoid others. This ability to approach or avoid using
gazing behavior has been extensively documented
in adults as well (Argyle & Cook, 1976). Accord-
ing to Mehrabian’s (1967) immediacy model of
social intimacy, eye gaze is a critical cue for enhanc-
ing psychological closeness. Argyle and Dean’s
(1965) affiliative equilibrium model extends this by
349
Adams and Nelson
specifically implicating approach–avoidance forces
underlying such immediacy behavior. As shown
in the next section, the shared signal value of gaze
and emotional expressions as cues of approach and
avoidance helps explain their combined effects on
emotional perception.
Emotional Expression and Gaze
In most standardized face sets, eye gaze is pur-
posefully directed into the camera when models
are displaying the “basic” emotional displays (see
also Chapter 10, this handbook). Making a similar
observation, Fehr and Exline (1987) concluded the
following: “One can imagine the pictured expres-
sions . . . with the gaze directed at a number of dif-
ferent angles. What seems to change in this exercise
is not the emotion expressed, but rather the target
or stimulus of the emotion” (p. 286). Yet, recently
significant interdisciplinary attention has been
given to the issue of combined processing of eye
gaze and emotion. This work demonstrates both
the complexities and new insights that can result
when examining the combinatorial nature of face
processing. Much remains to be learned, but what
is clear is that eye gaze meaningfully influences the
processing of emotional expression and vice versa.
Although both of these cues are expressive and are
thought to be predominantly processed along simi-
lar neural pathways (Haxby, Hoffman, & Gobbini,
2000), they also have been demonstrated to have
distinct neural systems (Engell & Haxby, 2007).
Nevertheless, mounting evidence supports an inter-
dependency of processing of eye gaze and emotion,
using psychophysical, self-report ratings, and neu-
roimaging paradigms, with clear patterns of func-
tional interactivity being established (for a review,
see R. Graham & LaBar, 2012).
Much of the existing evidence supports the
shared signal hypothesis (Adams, Gordon, Baird,
Ambady, & Kleck, 2003; Adams & Kleck, 2003,
2005). This hypothesis predicted that when paired,
cues relevant to threat that share a congruent under-
lying signal value should facilitate the processing
efficiency of an emotion. In initial support of this
hypothesis, Adams and Kleck (2003, 2005) found
that direct gaze facilitated processing efficiency as
well as accuracy, and it increased the perceived
350
intensity of facially communicated approach-
oriented emotions (e.g., anger, joy), whereas
averted gaze facilitated processing efficiency,
accuracy, and perceived intensity of facially com-
municated avoidance-oriented emotions (e.g., fear,
sadness). Similar effects have now been replicated
by several labs using a variety of techniques. One
such study (Milders, Hietanen, Leppänen, & Braun,
2011) found that congruent pairings of gaze and
threat cues (direct anger and averted fear) were more
likely to be detected in an attentional blink para-
digm than incongruent pairings, revealing functional
interactivity at a remarkably early stage of perceptual
processing.
Facial emotion also influences how eye gaze
is perceived. Direct eye gaze is recognized faster
when paired with angry faces, and averted eye gaze
is recognized faster when paired with fearful faces
(Adams & Franklin, 2009). In addition, perceiv-
ers tend to judge eye gaze more often as looking
at them when presented on happy and angry faces
than on neutral or fearful faces, though this effect
was particularly pronounced for happy faces (Lob-
maier, Tiddeman, & Perrett, 2008; Martin & Rovira,
1982). Further, fear expressions increase orienting
toward that cued by averted gaze and anger expres-
sions increase attention capturing effects of direct
gaze (Fox, Mathews, Calder, & Yiend, 2007), a
pattern that is consistent with the shared signal
hypothesis.
Adams et al. (2003) found evidence for interac-
tivity in neural response, replicating previous pat-
terns found in the literature for greater amygdala
activation to fear than anger when coupled with
direct gaze faces, but opposite patterns for averted
gaze faces. This fit with Whalen’s (1998) proposal
that the amygdala is instrumental in processing
stimulus ambiguity. Other evidence has supported
an opposite effect: greater amygdala activation to
congruent threat-gaze pairs. For instance, in one
study, the amygdala was more active to averted
gaze fear faces than to direct fear faces (Hadjikhani,
Hoge, Snyder, & de Gelder, 2008). In yet another
study, greater amygdala activity was found for anger
expressions looking toward, compared to away
from, the observer (Sato, Yoshikawa, Kochiyama, &
Matsumura, 2004). Whereas Adams et al.’s findings
 Eye Behavior and Gaze

seem to implicate a top-down, reflective process Individual Differences in Gaze Behavior

involved in deciphering ambiguity, these later find- and Perception
ings are more consistent with an early, reflexive
Traits.  The role of facial expression in gaze pro-
integration of congruent combinations of gaze and
cessing is also moderated by individual differences.
emotion.
For instance, Mathews, Fox, Yiend, and Calder
A number of follow-up studies aimed at
(2003) found that gaze cueing effects are greater for
addressing these discrepancies proposed that for
faces showing fear versus those with neutral expres-
these two systems to reach an optimal level of
sions, but only in participants high in anxiety. Fox
adaptive functioning, early and late processing are
et al. (2007) replicated this effect, but again, only
likely differentially attuned to distinct visuoper-
in participants high in anxiety. Additionally, they
ceptual representations of threat. Supporting this
found that direct gaze faces expressing anger cap-
hypothesis, Adams et al. (2011) crossed eye gaze
tured attention longer, but again, this was only in
with threat expressions revealing greater neural
participants high in anxiety. Tipples (2006) found
activation—particularly in the right hemisphere
greater cueing effects for faces expressing fear
of the brain, including the right amygdala—for
than happy, compared to neutral. This was associ-
congruent (i.e., clear) threat cues under condi-
ated with levels of trait fearfulness. This effect has
tions of restricted awareness (i.e., 33 ms); whereas
been replicated with dynamically changing faces
under conditions of sustained processing (i.e., 2 s),
(Putman, Hermans, & van Honk, 2006). Other
greater overall neural activation was found, includ-
studies have found a more general negative expres-
ing the left amygdala, for incongruent (i.e., ambig-
sion effect, with greater cueing on faces expressing
uous) threat cues. Additionally, when looking at
fear and anger, relative to happy and neutral faces,
just fear, greater amygdala activation (particularly
but only for participants high in anxiety (Holmes,
the left amygdala) was found again for incongru-
Richards, & Green, 2006).
ent threat (i.e., direct gaze fear) at sustained levels
One important factor to take into consideration
of processing (i.e., 1 s), whereas greater bilateral
in these studies is that eye aperture, or pupil to
amygdala activation was found for congruent threat
sclera contrast, which differs between expressions,
(i.e., averted gaze fear) at more restricted levels of
impacts gaze cueing. Tipples (2005) found that
processing (i.e., 300 ms; Adams et al., 2012). This
gaze cueing was enhanced when the eye aperture
effect was later replicated in bilateral amygdalae
was larger, regardless of expression. In nonanxious
using a within-subjects paradigm (van der Zwaag,
participants, gaze cueing occurs for fearful and dis-
Da Costa, Zürcher, Adams, & Hadjikhani, 2012).
gusted faces only when evaluating the target of the
Together, these studies reveal that gaze and emo-
gaze (i.e., is it positive or negative; Pecchinenda,
tion share an ecological relevance to the observer,
Pes, Ferlazzo, & Zoccolotti, 2008).
one that meaningfully influences behavioral and
neural processes underlying their perceptual inte- Gender.  There have been reported gender differ-
gration. Notably, neuroanatomical models reveal ences in the degree to which individuals follow the
that although these two cues share overlapping eye gaze of others, with women demonstrating more
neural substrates that allow for perceptual integral- gaze cueing than men (Bayliss, di Pellegrino, &
ity, they are also associated with distinct neural Tipper, 2005). Additionally, women show more gaze
substrates that allow for independent processing as cueing in response to familiar others than men do
well (Engell & Haxby, 2007; Haxby et al., 2000). (Deaner, Shepherd, & Platt, 2007). Bayliss et al.
In other words, it appears that although these (2005) argued that hardwired sex differences may
cues can be perceptually integrated in a function- explain this effect. For instance, they also found
ally adaptive manner, they need not always be that those who score higher on the autism quotient
(see Ganel, Goshen-Gottstein, & Goodale, 2005; demonstrate less gaze cueing. Given that autism is
R. Graham & LaBar, 2007; see also Chapter 10, conceptualized as an “extreme male-brain” disorder
this handbook). (Baron-Cohen, 2002), they argued that the worse

351
Adams and Nelson
performance from men may reflect stable traits
associated with sex (i.e., men are “systemizers,” and
women are “empathizers”; see Baron-Cohen, 2002).
Age.  Limited research has been done looking at
the impact of age on processing eye gaze and emo-
tion. A study using event-related potentials dem-
onstrated that, to some extent, infants as young
as 4 months of age seem to integrate eye gaze and
emotion (i.e., more responsive to direct gaze relative
to averted gaze being displayed by a happy face),
but they did not demonstrate integration for fearful
faces. A follow-up study with newborns also repli-
cated this happy-only effect using a visual prefer-
ence paradigm (Rigato, Menon, Johnson, & Farroni,
2011). The authors argued that infants have less
experience with nonhappy faces and, thus, experi-
ence may be important for the ability to integrate
compound social cues in the face.
Context and culture.  Although eye gaze appears
to be a universally important social cue, it can exert
different meanings across cultures. Argyle and Cook
(1976), for instance, suggested that in Western cul-
tures, eye gaze is generally seen as a sign of respect.
A failure to make eye contact may therefore be inter-
preted as suspicious. In East Asian cultures, how-
ever, making eye contact can be impolite and can
be seen as threatening under certain circumstances,
whereas averted eye gaze, especially downward
shifts, can be seen as respectful (Knapp, Hall, &
Horgan, 2013). Consequently, U.S. Caucasian par-
ticipants tend to look a same-culture experimenter
in the eye when answering a question, whereas
Japanese participants tend to look away (McCarthy,
Lee, Itakura, & Muir, 2006; for related discussion,
see Chapter 4, this handbook).
As far as we know, the influence of culture on
neural responses to gaze information has not been
previously examined. However, one study examined
the influence of gaze on responses to viewing same-
versus other-race faces (Richeson et al., 2008). In
this study, White participants showed more right
amygdala activation to direct than averted gaze
Black faces, an effect not found for White faces.
More bilateral insula, left suprmarginal gyrus, right
hippocampus, right middle frontal gyrus, left infe-
rior occipital gyrus, right precentral gyrus, and
352
medial orbital gyrus also were observed. Similar
responses were not apparent when viewing White
faces, leading the authors to conclude that the
threatening, approach-oriented signal stereotypi-
cally ascribed to Black faces was compounded by
direct gaze (see also Adams & Kleck, 2003, 2005;
Ambady & Adams, 2011).
CONCLUDING REMARKS
The eyes are perhaps the most remarkable source
of nonverbal information that humans have access
to. Not only are they the very organ that helps
initiate the process of decoding information from
others, but they, in turn, give others the ability
to express a rich array of social meaning as well.
Variables such as eye gaze direction and pupillary
size give people troves of information that can be
processed—sometimes in isolation, and sometimes
within the context of other social cues (i.e., emo-
tional expression)—to provide essential nonverbal
cues about the intentions of others. It is remarkable
that behaviors that represent mere millimeter shifts
and dilations can have such a powerful impact. If
a soul could be operationalized, certainly the eyes
would make for the most likely candidate as win-
dows into it.
References
Adams, R. B., Jr., & Franklin, R. G., Jr. (2009). Influence
of emotional expression on the processing of gaze
direction. Motivation and Emotion, 33, 106–112.
http://dx.doi.org/10.1007/s11031-009-9121-9
Adams, R. B., Jr., Franklin, R. G., Jr., Kveraga, K.,
Ambady, N., Kleck, R. E., Whalen, P. J., . . . Nelson,
A. J. (2012). Amygdala responses to averted vs
direct gaze fear vary as a function of presentation
speed. Social Cognitive and Affective Neuroscience, 7,
568–577. http://dx.doi.org/10.1093/scan/nsr038
Adams, R. B., Jr., Franklin, R. G., Jr., Nelson, A. J.,
Gordon, H. L., Kleck, R. E., Whalen, P. J., &
Ambady, N. (2011). Differentially tuned responses
to restricted versus prolonged awareness of threat:
A preliminary fMRI investigation. Brain and
Cognition, 77, 113–119. http://dx.doi.org/10.1016/j.
bandc.2011.05.001
Adams, R. B., Jr., Gordon, H. L., Baird, A. A., Ambady,
N., & Kleck, R. E. (2003). Effects of gaze on
amygdala sensitivity to anger and fear faces. Science,
300, 1536. http://dx.doi.org/10.1126/science.
1082244

Adams, R. B., Jr., & Kleck, R. E. (2003). Perceived gaze
direction and the processing of facial displays of
emotion. Psychological Science, 14, 644–647. http://
dx.doi.org/10.1046/j.0956-7976.2003.psci_1479.x
Adams, R. B., Jr., & Kleck, R. E. (2005). Effects of
direct and averted gaze on the perception of facially
communicated emotion. Emotion, 5, 3–11.
http://dx.doi.org/10.1037/1528-3542.5.1.3
Adams, R. B., Jr., Pauker, K., & Weisbuch, M. (2010).
Looking the other way: The role of gaze direction in
the cross-race memory effect. Journal of Experimental
Social Psychology, 46, 478–481. http://dx.doi.org/
10.1016/j.jesp.2009.12.016
Adams, R. B., Jr., Rule, N. O., Franklin, R. G., Jr., Wang, E.,
Stevenson, M. T., Yoshikawa, S., . . . Ambady, N.
(2010). Cross-cultural reading the mind in the
eyes: An fMRI investigation. Journal of Cognitive
Neuroscience, 22, 97–108. http://dx.doi.org/10.1162/
jocn.2009.21187
Adolphs, R., Sears, L., & Piven, J. (2001). Abnormal
processing of social information from faces in autism.
Journal of Cognitive Neuroscience, 13, 232–240. http://
dx.doi.org/10.1162/089892901564289
Ahrens, R. (1954). An ontogenetic study of reactions to
facial expression. Zeitschrift für Experimentelle und
Angewandte Psychologie, 2, 412–454.
Allard, E. S., Wadlinger, H. A., & Isaacowitz, D. M.
(2010). Positive gaze preferences in older adults:
Assessing the role of cognitive effort with pupil
dilation. Aging, Neuropsychology, and Cognition:
A Journal on Normal and Dysfunctional Development,
17, 296–311. http://dx.doi.org/10.1080/
13825580903265681
Ambady, N., & Adams, R. B., Jr. (2011). Us versus them:
The social neuroscience of perceiving out-groups.
In A. Todorov, S. Fiske, & D. Prentice (Eds.), Social
neuroscience: Toward understanding the underpinnings
of the social mind (pp. 135–144). http://dx.doi.
org/10.1093/acprof:oso/97801953168
72.003.0009
Amemiya, S., & Ohtomo, K. (2012). Effect of the observed
pupil size on the amygdala of the beholders. Social
Cognitive and Affective Neuroscience, 7, 332–341.
http://dx.doi.org/10.1093/scan/nsr013
Anderson, C. J., & Colombo, J. (2009). Larger tonic
pupil size in young children with autism spectrum
disorder. Developmental Psychobiology, 51, 207–211.
http://dx.doi.org/10.1002/dev.20352
Argyle, M., & Cook, M. (1976). Gaze and mutual gaze.
New York, NY: Cambridge University Press.
Argyle, M., & Dean, J. (1965). Eye-contact, distance and
affiliation. Sociometry, 28, 289–304. http://dx.doi.
org/10.2307/2786027
Baron-Cohen, S. (1995). Theory of mind and face-
processing: How do they interact in development
Eye Behavior and Gaze
and psychopathology? In D. Cicchetti & D. Cohen
(Eds.), Developmental psychopathology (Vol. 1,
pp. 343–356). New York, NY: Wiley.
Baron-Cohen, S. (2002). The extreme male brain theory
of autism. Trends in Cognitive Sciences, 6, 248–254.
http://dx.doi.org/10.1016/S1364-6613(02)01904-6
Baron-Cohen, S., & Hammer, J. (1997). Parents of children
with Asperger syndrome: What is the cognitive
phenotype? Journal of Cognitive Neuroscience, 9,
548–554. http://dx.doi.org/10.1162/jocn.1997.9.4.548
Baron-Cohen, S., Ring, H. A., Bullmore, E. T., Wheelwright,
S., Ashwin, C., & Williams, S. C. (2000). The amygdala
theory of autism. Neuroscience and Biobehavioral
Reviews, 24, 355–364. http://dx.doi.org/10.1016/
S0149-7634(00)00011-7
Baron-Cohen, S., Wheelwright, S., Hill, J., Raste, Y., &
Plumb, I. (2001). The “Reading the Mind in the
Eyes” Test Revised Version: A study with normal
adults, and adults with Asperger syndrome or high-
functioning autism. Journal of Child Psychology and
Psychiatry, 42, 241–251. http://dx.doi.org/10.1111/
1469-7610.00715
Baron-Cohen, S., Wheelwright, S., & Jolliffe, T. (1997).
Is there a “language of the eyes”? Evidence from
normal adults, and adults with autism or Asperger
syndrome. Visual Cognition, 4, 311–331. http://
dx.doi.org/10.1080/713756761
Batki, A., Baron-Cohen, S., Wheelwright, S., Connellan, J., &
Ahluwalia, J. (2000). Is there an innate gaze module?
Evidence from human neonates. Infant Behavior and
Development, 23, 223–229. http://dx.doi.org/
10.1016/S0163-6383(01)00037-6
Bayliss, A. P., di Pellegrino, G., & Tipper, S. P. (2005).
Sex differences in eye gaze and symbolic cueing
of attention. Quarterly Journal of Experimental
Psychology, Section A: Human Experimental
Psychology, 58, 631–650. http://dx.doi.org/10.1080/
02724980443000124
Bentin, S., Allison, T., Puce, A., Perez, E., & McCarthy, G.
(1996). Electrophysiological studies of face perception
in humans. Journal of Cognitive Neuroscience, 8,
551–565. http://dx.doi.org/10.1162/jocn.1996.8.6.551
Bernick, N. (1972). Relationship between age and pupil
size: Some additional variables. Perceptual and Motor
Skills, 34, 727–731. http://dx.doi.org/10.2466/
pms.1972.34.3.727
Berrien, F. K., & Huntington, G. H. (1943). An
exploratory study of pupillary responses during
deception. Journal of Experimental Psychology, 32,
443–449. http://dx.doi.org/10.1037/h0063488
Blais, C., Jack, R. E., Scheepers, C., Fiset, D., & Caldara, R.
(2008). Culture shapes how we look at faces. PLoS
ONE, 3, e3022. http://dx.doi.org/10.1371/journal.
pone.0003022
353
Adams and Nelson
Blaison, C., Hareli, S., Strauss, C., & Hess, U. (2012,
May). Imitating eyes: Mimicry of emotions shown in
the eyes. Paper presented at CERE 2012, Canterbury,
England.
Bonda, E., Petrides, M., Ostry, D., & Evans, A. (1996).
Specific involvement of human parietal systems and
the amygdala in the perception of biological motion.
Journal of Neuroscience, 16, 3737–3744.
Bräuer, J., Call, J., & Tomasello, M. (2007). Chimpanzees
really know what others can see in a competitive
situation. Animal Cognition, 10, 439–448. http://
dx.doi.org/10.1007/s10071-007-0088-1
Brothers, L., & Ring, B. (1992). A neuroethological
framework for the representation of minds. Journal
of Cognitive Neuroscience, 4, 107–118. http://dx.doi.
org/10.1162/jocn.1992.4.2.107
Brothers, L., & Ring, B. (1993). Mesial temporal neurons
in the macaque monkey with responses selective
for aspects of social stimuli. Behavioural Brain
Research, 57, 53–61. http://dx.doi.org/10.1016/0166-
4328(93)90061-T
Bruce, C. J., Desimone, R., & Gross, C. G. (1986). Both
striate cortex and superior colliculus contribute to
visual properties of neurons in superior temporal
polysensory area of macaque monkey. Journal of
Neurophysiology, 55, 1057–1075.
Bruck, M., Cavanagh, P., & Ceci, S. J. (1991).
Fortysomething: Recognizing faces at one’s 25th
reunion. Memory and Cognition, 19, 221–228. http://
dx.doi.org/10.3758/BF03211146
Brüne, M. (2005). “Theory of mind” in schizophrenia: A
review of the literature. Schizophrenia Bulletin, 31,
21–42. http://dx.doi.org/10.1093/schbul/sbi002
Buck, R. (1988). Nonverbal communication:
Spontaneous and symbolic aspects. American
Behavioral Scientist, 31, 341–354. http://dx.doi.
org/10.1177/000276488031003006
Bull, R., & Shead, G. (1979). Pupil dilation, sex of
stimulus, and age and sex of observer. Perceptual and
Motor Skills, 49, 27–30. http://dx.doi.org/10.2466/
pms.1979.49.1.27
Cacioppo, J. T., Martzke, J. S., Petty, R. E., & Tassinary,
L. G. (1988). Specific forms of facial EMG response
index emotions during an interview: From Darwin
to the continuous flow hypothesis affect-laden
information processing. Journal of Personality
and Social Psychology, 54, 592–604. http://dx.doi.
org/10.1037/0022-3514.54.4.592
Carstensen, L. L., Isaacowitz, D. M., & Charles, S. T.
(1999). Taking time seriously: A theory of
socioemotional selectivity. American Psychologist, 54,
165–181. http://dx.doi.org/10.1037/0003-066X.
54.3.165
Carstensen, L. L., & Mikels, J. A. (2005). At the
intersection of cognition and emotion: Aging and the
354
positivity effect. Current Directions in Psychological
Science, 14, 117–121. http://dx.doi.org/10.1111/
j.0963-7214.2005.00348.x
Chance, M. R. A. (1967). Attention structure as the basis
of primate rank orders. Man, 2, 503–518. http://
dx.doi.org/10.2307/2799336
Chapman, E., Baron-Cohen, S., Auyeung, B., Knickmeyer, R.,
Taylor, K., & Hackett, G. (2006). Fetal testosterone
and empathy: Evidence from the empathy quotient
(EQ) and the “Reading the Mind in the Eyes” Test.
Social Neuroscience, 1, 135–148. http://dx.doi.org/
10.1080/17470910600992239
Cunningham, M. R. (1986). Measuring the physical
in physical attractiveness: Quasi-experiments on
the sociobiology of female facial beauty. Journal
of Personality and Social Psychology, 50, 925–935.
http://dx.doi.org/10.1037/0022-3514.50.5.925
Dalton, K. M., Nacewicz, B. M., Johnstone, T., Schaefer,
H. S., Gernsbacher, M. A., Goldsmith, H. H., . . .
Davidson, R. J. (2005). Gaze fixation and the
neural circuitry of face processing in autism. Nature
Neuroscience, 8, 519–526.
Darwin, C. (1965). The expression of the emotions in man
and animals. New York, NY: Oxford University Press.
(Original work published 1872)
Davidson, R. J., & Hugdahl, K. (1995). Brain asymmetry.
Cambridge, MA: MIT Press.
Deaner, R. O., Khera, A. V., & Platt, M. L. (2005).
Monkeys pay per view: Adaptive valuation of
social images by rhesus macaques. Current
Biology, 15, 543–548. http://dx.doi.org/10.1016/
j.cub.2005.01.044
Deaner, R. O., Shepherd, S. V., & Platt, M. L. (2007).
Familiarity accentuates gaze cuing in women but
not men. Biology Letters, 3, 64–67. http://dx.doi.
org/10.1098/rsbl.2006.0564
De Fruyt, F. (1997). Gender and individual differences in
adult crying. Personality and Individual Differences,
22, 937–940. http://dx.doi.org/10.1016/S0191-
8869(96)00264-4
Demos, K. E., Kelley, W. M., Ryan, S. L., Davis, F. C., &
Whalen, P. J. (2008). Human amygdala sensitivity
to the pupil size of others. Cerebral Cortex, 18,
2729–2734. http://dx.doi.org/10.1093/cercor/bhn034
DePaulo, B. M., Lindsay, J. J., Malone, B. E.,
Muhlenbruck, L., Charlton, K., & Cooper, H.
(2003). Cues to deception. Psychological Bulletin,
129, 74–118. http://dx.doi.org/10.1037/0033-2909.
129.1.74
Dionisio, D. P., Granholm, E., Hillix, W. A., & Perrine,
W. F. (2001). Differentiation of deception using
pupillary responses as an index of cognitive
processing. Psychophysiology, 38, 205–211. http://
dx.doi.org/10.1111/1469-8986.3820205

Dovidio, J. F., & Ellyson, S. L. (1982). Decoding visual
dominance: Attributions of power based on relative
percentages of looking while speaking and looking
while listening. Social Psychology Quarterly, 45,
106–113. http://dx.doi.org/10.2307/3033933
Driver, J., IV, Davis, G., Ricciardelli, P., Kidd, P., Maxwell,
E., & Baron-Cohen, S. (1999). Gaze perception
triggers reflexive visuospatial orienting. Visual
Cognition, 6, 509–540. http://dx.doi.org/10.1080/
135062899394920
Duchenne de Boulogne, G.-B. (1990). The mechanism
of human facial expression or an electrophysiological
analysis of the expression of the emotions. http://
dx.doi.org/10.1017/CBO9780511752841 (Original
work published 1862)
Eisenberg, J. (1983). The social organisation of mammals.
Berlin, Germany: Walter de Gruyter.
Ekman, P. (1992). Telling lies. New York, NY: Norton.
Ekman, P., & Friesen, W. V. (1975). Unmasking the face:
A guide to recognizing emotions from facial clues.
Englewood Cliffs, NJ: Prentice Hall.
Ellermeier, W., & Westphal, W. (1995). Gender
differences in pain ratings and pupil reactions to
painful pressure stimuli. Pain, 61, 435–439. http://
dx.doi.org/10.1016/0304-3959(94)00203-Q
Ellsworth, P. C. (1975). Direct gaze as a social
stimulus: The example of aggression. In P. Pliner,
L. Krames, & T. Alloway (Eds.), Nonverbal
communication of aggression (pp. 53–75). http://
dx.doi.org/10.1007/978-1-4684-2835-3_3
Ellsworth, P. C., Carlsmith, J. M., & Henson, A. (1972).
The stare as a stimulus to flight in human subjects:
A series of field experiments. Journal of Personality
and Social Psychology, 21, 302–311. http://dx.doi.
org/10.1037/h0032323
Ellsworth, P. C., & Ross, L. (1975). Intimacy in response
to direct gaze. Journal of Experimental Social
Psychology, 11, 592–613. http://dx.doi.org/
10.1016/0022-1031(75)90010-4
Emery, N. J. (2000). The eyes have it: The neuroethology,
function and evolution of social gaze. Neuroscience
and Biobehavioral Reviews, 24, 581–604. http://
dx.doi.org/10.1016/S0149-7634(00)00025-7
Engell, A. D., & Haxby, J. V. (2007). Facial expression
and gaze-direction in human superior temporal
sulcus. Neuropsychologia, 45, 3234–3241. http://
dx.doi.org/10.1016/j.neuropsychologia.2007.06.022
Ernest-Jones, M., Nettle, D., & Bateson, M. (2011).
Effects of eye images on everyday cooperative
behavior: A field experiment. Evolution and Human
Behavior, 32, 172–178. http://dx.doi.org/10.1016/
j.evolhumbehav.2010.10.006
Exline, R., Gray, D., & Schuette, D. (1965). Visual
behavior in a dyad as affected by interview content
and sex of respondent. Journal of Personality
Eye Behavior and Gaze
and Social Psychology, 1, 201–209. http://dx.doi.
org/10.1037/h0021865
Exline, R. V. (1971). Visual interaction: The glances of
power and preference. In J. K. Cole (Ed.), Nebraska
Symposium on Motivation (pp. 162–205). Lincoln:
University of Nebraska Press.
Exline, R. V., Ellyson, S. L., & Long, B. (1975). Visual
behavior as an aspect of power role relationships.
In P. Pliner (Ed.), Nonverbal communication of
aggression (Vol. 2, pp. 21–52). New York, NY:
Plenum Press.
Fabes, R. A., & Martin, C. L. (1991). Gender and age
stereotypes of emotionality. Personality and Social
Psychology Bulletin, 17, 532–540. http://dx.doi.org/
10.1177/0146167291175008
Farroni, T., Csibra, G., Simion, F., & Johnson, M. H.
(2002). Eye contact detection in humans from birth.
Proceedings of the National Academy of Sciences,
USA, 99, 9602–9605. http://dx.doi.org/10.1073/
pnas.152159999
Farroni, T., Mansfield, E. M., Lai, C., & Johnson, M.
H. (2003). Infants perceiving and acting on the
eyes: Tests of an evolutionary hypothesis. Journal of
Experimental Child Psychology, 85, 199–212. http://
dx.doi.org/10.1016/S0022-0965(03)00022-5
Farroni, T., Massaccesi, S., Menon, E., & Johnson, M. H.
(2007). Direct gaze modulates face recognition in
young infants. Cognition, 102, 396–404. http://dx.doi.
org/10.1016/j.cognition.2006.01.007
Farroni, T., Massaccesi, S., Pividori, D., & Johnson,
M. H. (2004). Gaze following in newborns.
Infancy, 5, 39–60. http://dx.doi.org/10.1207/
s15327078in0501_2
Fehr, B. J., & Exline, R. V. (1987). Social visual
interaction: A conceptual and literature review. In
A. W. Siegman & S. Feldstein (Eds.), Nonverbal
behavior and communication (pp. 225–326). Hillsdale,
NJ: Erlbaum.
Feinsod, M. (2000). The blind beautiful eye. Journal
of Neuro-Ophthalmology, 20, 22–24. http://dx.doi.
org/10.1097/00041327-200020010-00008
Fischer, A. H., Eagly, A. H., & Oosterwijk, S. (2013).
The meaning of tears: Which sex seems emotional
depends on the social context. European Journal of
Social Psychology, 43, 505–515.
Fischer, A. H., Rodriguez Mosquera, P. M., van Vianen,
A. E. M., & Manstead, A. S. R. (2004). Gender and
culture differences in emotion. Emotion, 4, 87–94.
Fitzgerald, H. E. (1968). Autonomic pupillary reflex
activity during early infancy and its relation to social
and nonsocial visual stimuli. Journal of Experimental
Child Psychology, 6, 470–482. http://dx.doi.
org/10.1016/0022-0965(68)90127-6
Foulsham, T., Cheng, J. T., Tracy, J. L., Henrich, J., &
Kingstone, A. (2010). Gaze allocation in a dynamic
355
Adams and Nelson
situation: Effects of social status and speaking.
Cognition, 117, 319–331. http://dx.doi.org/10.1016/
j.cognition.2010.09.003
Fox, E., Mathews, A., Calder, A. J., & Yiend, J. (2007).
Anxiety and sensitivity to gaze direction in
emotionally expressive faces. Emotion, 7, 478–486.
http://dx.doi.org/10.1037/1528-3542.7.3.478
Franklin, R. G., Jr., Stevenson, M. T., Ambady, N., &
Adams, R. B., Jr. (2015). Cross-cultural reading
the mind in the eyes and its consequences for
international relations. In J. E. Warnick & D. Landis
(Eds.), Neuroscience in intercultural contexts
(pp. 117–142). New York, NY: Springer.
Frey, W., & Langseth, M. (1985). Crying: The mystery of
tears. Minneapolis, MN: Winston Press.
Fridlund, A. J., Ekman, P., & Oster, H. (1987). Facial
expressions of emotion. In A. W. Siegman &
S. Feldstein (Eds.), Nonverbal behavior and
communication (pp. 143–223). Hillsdale, NJ:
Erlbaum.
Friesen, C. K., & Kingstone, A. (1998). The eyes have
it! Reflexive orienting is triggered by nonpredictive
gaze. Psychonomic Bulletin and Review, 5, 490–495.
http://dx.doi.org/10.3758/BF03208827
Friesen, W. V. (1972). Cultural differences in facial
expressions in a social situation: An experimental test
of the concept of display rules (Unpublished doctoral
dissertation). University of California, San Francisco.
Frischen, A., Bayliss, A. P., & Tipper, S. P. (2007).
Gaze cueing of attention: Visual attention,
social cognition, and individual differences.
Psychological Bulletin, 133, 694–724. http://dx.doi.
org/10.1037/0033-2909.133.4.694
Fung, H. H., Isaacowitz, D. M., Lu, A. Y., & Li, T. (2010).
Interdependent self-construal moderates the age-
related negativity reduction effect in memory and
visual attention. Psychology and Aging, 25, 321–329.
http://dx.doi.org/10.1037/a0019079
Fung, H. H., Isaacowitz, D. M., Lu, A. Y., Wadlinger, H.
A., Goren, D., & Wilson, H. R. (2008). Age-related
positivity enhancement is not universal: Older
Chinese look away from positive stimuli. Psychology
and Aging, 23, 440–446. http://dx.doi.org/10.1037/
0882-7974.23.2.440
Ganel, T., Goshen-Gottstein, Y., & Goodale, M. A.
(2005). Interactions between the processing of gaze
direction and facial expression. Vision Research, 45,
1191–1200. http://dx.doi.org/10.1016/j.visres.
2004.06.025
Gelstein, S., Yeshurun, Y., Rozenkrantz, L., Shushan, S.,
Frumin, I., Roth, Y., & Sobel, N. (2011). Human
tears contain a chemosignal. Science, 331, 226–230.
http://dx.doi.org/10.1126/science.1198331
356
Graham, J. A., & Argyle, M. (1975). The effects of
different patterns of gaze combined with different
facial expressions on impression formation. Journal
of Human Movement Studies, 1, 178–182.
Graham, R., & LaBar, K. S. (2007). Garner interference
reveals dependencies between emotional expression
and gaze in face perception. Emotion, 7, 296–313.
http://dx.doi.org/10.1037/1528-3542.7.2.296
Graham, R., & LaBar, K. S. (2012). Neurocognitive
mechanisms of gaze-expression interactions in face
processing and social attention. Neuropsychologia,
50, 553–566. http://dx.doi.org/10.1016/
j.neuropsychologia.2012.01.019
Gross, J. J., Fredrickson, B. L., & Levenson, R. W.
(1994). The psychophysiology of crying.
Psychophysiology, 31, 460–468. http://dx.doi.org/
10.1111/j.1469-8986.1994.tb01049.x
Grumet, G. W. (1999). Kinesic cues: The body, eyes,
and face. In L. K. Guerrero & J. A. DeVito (Eds.),
The nonverbal communication reader: Classic and
contemporary readings (pp. 48–89). Prospect Heights,
IL: Waveland Press.
Gunther Moor, B., Op de Macks, Z. A., Güroglu, B.,
Rombouts, S. A. R. B., Van der Molen, M. W., &
Crone, E. A. (2012). Neurodevelopmental changes
of reading the mind in the eyes. Social Cognitive and
Affective Neuroscience, 7, 44–52. http://dx.doi.org/
10.1093/scan/nsr020
Hadjikhani, N., Hoge, R., Snyder, J., & de Gelder, B.
(2008). Pointing with the eyes: The role of gaze in
communicating danger. Brain and Cognition, 68, 1–8.
http://dx.doi.org/10.1016/j.bandc.2008.01.008
Hains, S. M., & Muir, D. W. (1996). Infant sensitivity
to adult eye direction. Child Development, 67,
1940–1951. http://dx.doi.org/10.2307/1131602
Hall, J. A. (1978). Gender effects in decoding nonverbal
cues. Psychological Bulletin, 85, 845–857. http://
dx.doi.org/10.1037/0033-2909.85.4.845
Hall, J. K., Hutton, S. B., & Morgan, M. J. (2010). Sex
differences in scanning faces: Does attention to the
eyes explain female superiority in facial expression
recognition? Cognition and Emotion, 24, 629–637.
http://dx.doi.org/10.1080/02699930902906882
Harries, M. H., & Perrett, D. I. (1991). Visual processing
of faces in temporal cortex: Physiological evidence
for a modular organization and possible anatomical
correlates. Journal of Cognitive Neuroscience, 3, 9–24.
http://dx.doi.org/10.1162/jocn.1991.3.1.9
Harrison, N. A., Gray, M. A., & Critchley, H. D. (2009).
Dynamic pupillary exchange engages brain regions
encoding social salience. Social Neuroscience, 4,
233–243. http://dx.doi.org/10.1080/174709108
02553508
Harrison, N. A., Singer, T., Rotshtein, P., Dolan, R. J., &
Critchley, H. D. (2006). Pupillary contagion: Central

mechanisms engaged in sadness processing. Social
Cognitive and Affective Neuroscience, 1, 5–17. http://
dx.doi.org/10.1093/scan/nsl006
Harrison, N. A., Wilson, C. E., & Critchley, H. D.
(2007). Processing of observed pupil size modulates
perception of sadness and predicts empathy.
Emotion, 7, 724–729. http://dx.doi.org/10.1037/
1528-3542.7.4.724
Hasson, O. (2009). Emotional tears as biological signals.
Evolutionary Psychology, 7, 363–370.
Hastrup, J. L., Baker, J. G., Kraemer, D. L., & Bornstein,
R. F. (1986). Crying and depression among older
adults. Gerontologist, 26, 91–96. http://dx.doi.org/
10.1093/geront/26.1.91
Haxby, J. V., Hoffman, E. A., & Gobbini, M. I. (2000).
The distributed human neural system for face
perception. Trends in Cognitive Sciences, 4, 223–233.
http://dx.doi.org/10.1016/S1364-6613
(00)01482-0
Heiser, C. B. (1987). The fascinating world of the
nightshades. New York, NY: Dover.
Helminen, T. M., Kaasinen, S. M., & Hietanen, J. K.
(2011). Eye contact and arousal: The effects of
stimulus duration. Biological Psychology, 88,
124–130. http://dx.doi.org/10.1016/j.biopsycho.
2011.07.002
Hess, E. H. (1965). Attitude and pupil size. Scientific
American, 212, 46–54. http://dx.doi.org/10.1038/
scientificamerican0465-46
Hess, E. H. (1972). Pupillometrics. In N. S. Greenfield &
R. A. Sternbach (Eds.), Handbook of psychophysiology
(pp. 491–531). New York, NY: Holt, Rinehart and
Winston.
Hess, E. H. (1975). The tell-tale eye: How your eyes reveal
hidden thoughts and emotions. New York, NY: Van
Nostrand Reinhold.
Hess, E. H., & Petrovich, S. B. (1987). Pupillary behavior
in communication. In A. W. Siegman & S. Feldstein
(Eds.), Nonverbal behavior and communication
(pp. 327–348). Hillsdale, NJ: Erlbaum.
Hess, E. H., & Polt, J. M. (1960). Pupil size as related
to interest value of visual stimuli. Science, 132,
349–350. http://dx.doi.org/10.1126/science.
132.3423.349
Hietanen, J. K., Nummenmaa, L., Nyman, M. J.,
Parkkola, R., & Hämäläinen, H. (2006). Automatic
attention orienting by social and symbolic cues
activates different neural networks: An fMRI study.
NeuroImage, 33, 406–413. http://dx.doi.org/
10.1016/j.neuroimage.2006.06.048
Hillabrant, W. (1974). The influence of locomotion
and gaze direction on perceptions of
interacting persons. Personality and Social
Eye Behavior and Gaze
Psychology Bulletin, 1, 237–239. http://dx.doi.
org/10.1177/014616727400100181
Hinde, R. A., & Rowell, T. E. (1962). Communication by
postures and facial expressions in the rhesus monkey
(Macaca mulatta). Proceedings of the Zoological
Society of London, 138, 1–21. http://dx.doi.org/
10.1111/j.1469-7998.1962.tb05684.x
Hoffman, E. A., & Haxby, J. V. (2000). Distinct
representations of eye gaze and identity in the
distributed human neural system for face perception.
Nature Neuroscience, 3, 80–84. http://dx.doi.org/
10.1038/71152
Holmes, A., Richards, A., & Green, S. (2006). Anxiety
and sensitivity to eye gaze in emotional faces.
Brain and Cognition, 60, 282–294. http://dx.doi.
org/10.1016/j.bandc.2005.05.002
Hood, B. M., Macrae, C. N., Cole-Davies, V., & Dias, M.
(2003). Eye remember you: The effects of gaze
direction on face recognition in children and adults.
Developmental Science, 6, 67–71. http://dx.doi.
org/10.1111/1467-7687.00256
Hood, B. M., Willen, J. D., & Driver, J. (1998). Adult’s
eyes trigger shifts of visual attention in human
infants. Psychological Science, 9, 131–134. http://
dx.doi.org/10.1111/1467-9280.00024
Horley, K., Williams, L. M., Gonsalvez, C., & Gordon, E.
(2003). Social phobics do not see eye to eye: A visual
scanpath study of emotional expression processing.
Journal of Anxiety Disorders, 17, 33–44. http://dx.doi.
org/10.1016/S0887-6185(02)00180-9
Horley, K., Williams, L. M., Gonsalvez, C., & Gordon,
E. (2004). Face to face: Visual scanpath evidence for
abnormal processing of facial expressions in social
phobia. Psychiatry Research, 127, 43–53. http://
dx.doi.org/10.1016/j.psychres.2004.02.016
Illingworth, R. S. (1951). Sleep problems in the first
three years. BMJ: British Medical Journal, 1, 722–728.
http://dx.doi.org/10.1136/bmj.1.4709.722
Illingworth, R. S. (1955). Crying in infants and children.
BMG: British Medical Journal, 1, 75–78. http://dx.doi.
org/10.1136/bmj.1.4905.75
Irani, F., Platek, S. M., Panyavin, I. S., Calkins, M. E.,
Kohler, C., Siegel, S. J., . . . Gur, R. C. (2006). Self-
face recognition and theory of mind in patients with
schizophrenia and first-degree relatives. Schizophrenia
Research, 88, 151–160. http://dx.doi.org/10.1016/
j.schres.2006.07.016
Isaacowitz, D. M., Allard, E. S., Murphy, N. A., &
Schlangel, M. (2009). The time course of age-related
preferences toward positive and negative stimuli.
The Journals of Gerontology, Series B: Psychological
Sciences and Social Sciences, 64, 188–192. http://
dx.doi.org/10.1093/geronb/gbn036
357
Adams and Nelson
Isaacowitz, D. M., & Murphy, N. A. (2010). Aging eyes
facing an emotional world: The role of motivated
gaze. In R. B. Adams Jr., N. Ambady, K. Nakayama, &
S. Shimojo (Eds.), The science of social vision
(pp. 133–150). http://dx.doi.org/10.1093/acprof:oso/
9780195333176.003.0008
Jackson, I., & Sirois, S. (2009). Infant cognition: Going
full factorial with pupil dilation. Developmental
Science, 12, 670–679. http://dx.doi.org/10.1111/
j.1467-7687.2008.00805.x
Janisse, M. P. (1974). Pupil size, affect and exposure
frequency. Social Behavior and Personality, 2,
125–146. http://dx.doi.org/10.2224/sbp.1974.2.2.125
Joseph, R. M., & Tanaka, J. (2003). Holistic and part-
based face recognition in children with autism.
Journal of Child Psychology and Psychiatry,
44, 529–542. http://dx.doi.org/10.1111/1469-
7610.00142
Kaminski, J., Riedel, J., Call, J., & Tomasello, M. (2005).
Domestic goats, Capra hircus, follow gaze direction
and use social cues in an object choice task. Animal
Behaviour, 69, 11–18. http://dx.doi.org/10.1016/
j.anbehav.2004.05.008
Kanwisher, N., McDermott, J., & Chun, M. M. (1997).
The fusiform face area: A module in human
extrastriate cortex specialized for face perception.
Journal of Neuroscience, 17, 4302–4311.
Kirkland, R., Peterson, E., Baker, C., Miller, S., &
Pulos, S. (2013). Meta-analysis reveals adult female
superiority in “Reading the Mind in the Eyes Test.”
North American Journal of Psychology, 15, 121–146.
Kleinke, C. L. (1986). Gaze and eye contact: A research
review. Psychological Bulletin, 100, 78–100. http://
dx.doi.org/10.1037/0033-2909.100.1.78
Klin, A., Jones, W., Schultz, R., Volkmar, F., & Cohen,
D. (2002). Visual fixation patterns during viewing of
naturalistic social situations as predictors of social
competence in individuals with autism. Archives
of General Psychiatry, 59, 809–816. http://dx.doi.
org/10.1001/archpsyc.59.9.809
Knapp, M., Hall, J., & Horgan, T. (2013). Nonverbal
communication in human interaction. Boston, MA:
Cengage Learning.
Knight, M., Seymour, T. L., Gaunt, J. T., Baker, C.,
Nesmith, K., & Mather, M. (2007). Aging and goal-
directed emotional attention: Distraction reverses
emotional biases. Emotion, 7, 705–714. http://dx.doi.
org/10.1037/1528-3542.7.4.705
Kobayashi, H., & Kohshima, S. (1997). Unique
morphology of the human eye. Nature, 387, 767–768.
http://dx.doi.org/10.1038/42842
Kuhn, G., & Benson, V. (2007). The influence of eye-
gaze and arrow pointing distractor cues on voluntary
358
eye movements. Perception and Psychophysics, 69,
966–971. http://dx.doi.org/10.3758/BF03193934
Kuhn, G., & Kingstone, A. (2009). Look away! Eyes and
arrows engage oculomotor responses automatically.
Attention, Perception, and Psychophysics, 71, 314–327.
http://dx.doi.org/10.3758/APP.71.2.314
Lobmaier, J. S., Tiddeman, B. P., & Perrett, D. I. (2008).
Emotional expression modulates perceived gaze
direction. Emotion, 8, 573–577. http://dx.doi.org/
10.1037/1528-3542.8.4.573
Loewenfeld, I. E. (1993). The pupil: Anatomy, physiology,
and clinical applications. Ames: Iowa State University.
Loughland, C. M., Williams, L. M., & Gordon, E. (2002).
Visual scanpaths to positive and negative facial
emotions in an outpatient schizophrenia sample.
Schizophrenia Research, 55, 159–170. http://dx.doi.
org/10.1016/S0920-9964(01)00186-4
Lutz, T. (2001). Crying: The natural and cultural history of
tears. New York, NY: Norton.
Macrae, C. N., Hood, B. M., Milne, A. B., Rowe, A. C., &
Mason, M. F. (2002). Are you looking at me? Eye
gaze and person perception. Psychological Science, 13,
460–464. http://dx.doi.org/10.1111/1467-9280.00481
Maner, J. K., DeWall, C. N., & Gailliot, M. T. (2008).
Selective attention to signs of success: Social
dominance and early stage interpersonal perception.
Personality and Social Psychology Bulletin, 34,
488–501. http://dx.doi.org/10.1177/0146167207311910
Martin, W., & Rovira, M. (1982). Response biases
in eye-gaze perception. Journal of Psychology:
Interdisciplinary and Applied, 110, 203–209. http://
dx.doi.org/10.1080/00223980.1982.9915341
Mason, M. F., Hood, B. M., & Macrae, C. N. (2004).
Look into my eyes: Gaze direction and person
memory. Memory, 12, 637–643. http://dx.doi.
org/10.1080/09658210344000152
Mason, M. F., Tatkow, E. P., & Macrae, C. N. (2005).
The look of love: Gaze shifts and person perception.
Psychological Science, 16, 236–239. http://dx.doi.
org/10.1111/j.0956-7976.2005.00809.x
Mather, M., & Carstensen, L. L. (2005). Aging and
motivated cognition: The positivity effect in
attention and memory. Trends in Cognitive Sciences,
9, 496–502. http://dx.doi.org/10.1016/j.tics.2005.
08.005
Mather, M., & Knight, M. (2005). Goal-directed
memory: The role of cognitive control in older
adults’ emotional memory. Psychology and Aging,
20, 554–570. http://dx.doi.org/10.1037/0882-
7974.20.4.554
Mathews, A., Fox, E., Yiend, J., & Calder, A. (2003). The
face of fear: Effects of eye gaze and emotion on visual
attention. Visual Cognition, 10, 823–835. http://
dx.doi.org/10.1080/13506280344000095

Matsumoto, D. (1989). Face, culture, and judgments
of anger and fear: Do the eyes have it? Journal of
Nonverbal Behavior, 13, 171–188. http://dx.doi.
org/10.1007/BF00987048
Matsumoto, D. (1990). Cultural similarities and
differences in display rules. Motivation and Emotion,
14, 195–214. http://dx.doi.org/10.1007/BF00995569
Matsumoto, D., Yoo, S. H., & Fontaine, J. (2009).
Hypocrisy or maturity? Culture and context
differentiation. European Journal of Personality, 23,
251–264. http://dx.doi.org/10.1002/per.716
McCarthy, A., Lee, K., Itakura, S., & Muir, D. W. (2006).
Cultural display rules drive eye gaze during thinking.
Journal of Cross-Cultural Psychology, 37, 717–722.
http://dx.doi.org/10.1177/0022022106292079
McNelis, N. L., & Boatright-Horowitz, S. L. (1998). Social
monitoring in a primate group: The relationship
between visual attention and hierarchical ranks.
Animal Cognition, 1, 65–69. http://dx.doi.org/10.1007/
s100710050008
Mehrabian, A. (1967). Attitudes inferred from non-
immediacy of verbal communications. Journal of
Verbal Learning and Verbal Behavior, 6, 294–295.
http://dx.doi.org/10.1016/S0022-5371(67)80113-0
Metalis, S. A., Hess, E. H., & Beaver, P. W. (1982).
Pupillometric analysis of two theories of obesity.
Perceptual and Motor Skills, 55, 87–92. http://dx.doi.
org/10.2466/pms.1982.55.1.87
Milders, M., Hietanen, J. K., Leppänen, J. M., & Braun,
M. (2011). Detection of emotional faces is modulated
by the direction of eye gaze. Emotion, 11, 1456–1461.
http://dx.doi.org/10.1037/a0022901
Murube, J., Murube, L., & Murube, A. (1999). Origin
and types of emotional tearing. European Journal of
Ophthalmology, 9, 77–84.
N’Diaye, K., Sander, D., & Vuilleumier, P. (2009). Self-
relevance processing in the human amygdala: Gaze
direction, facial expression, and emotion intensity.
Emotion, 9, 798–806.
Nelson, J. (2005). Seeing through tears: Crying and
attachment. London, England: Routledge.
Neuberg, S. L., & Fiske, S. T. (1987). Motivational
influences on impression formation: Outcome
dependency, accuracy-driven attention, and
individuating processes. Journal of Personality
and Social Psychology, 53, 431–444. http://dx.doi.
org/10.1037/0022-3514.53.3.431
Nummenmaa, T. (1964). The language of the face.
Jyvaskyla, Finland: Jyvaskylan Yliopistoyhdistys.
Oh, T. J., Kim, M. Y., Park, K. S., & Cho, Y. M. (2012).
Effects of chemosignals from sad tears and
postprandial plasma on appetite and food intake
in humans. PLoS ONE, 7, e42352. http://dx.doi.
org/10.1371/journal.pone.0042352
Eye Behavior and Gaze
Pannozzo, P. L., Phillips, K. A., Haas, M. E., & Mintz,
E. M. (2007). Social monitoring reflects dominance
relationships in a small captive group of brown
capuchin monkeys (Cebus apella). Ethology,
113, 881–888. http://dx.doi.org/10.1111/
j.1439-0310.2007.01392.x
Patterson, M. L. (1976). An arousal model of
interpersonal intimacy. Psychological Review,
83, 235–245. http://dx.doi.org/10.1037/
0033-295X.83.3.235
Pecchinenda, A., Pes, M., Ferlazzo, F., & Zoccolotti,
P. (2008). The combined effect of gaze direction
and facial expression on cueing spatial attention.
Emotion, 8, 628–634.
Pellicano, E., & Macrae, C. N. (2009). Mutual eye
gaze facilitates person categorization for typically
developing children, but not for children with
autism. Psychonomic Bulletin and Review, 16,
1094–1099. http://dx.doi.org/10.3758/PBR.16.6.1094
Pelphrey, K. A., Sasson, N. J., Reznick, J. S., Paul, G.,
Goldman, B. D., & Piven, J. (2002). Visual scanning
of faces in autism. Journal of Autism and
Developmental Disorders, 32, 249–261. http://dx.doi.
org/10.1023/A:1016374617369
Perlez, J. (2001, June 18). Cordial rivals: How Bush and
Putin became friends. New York Times. Retrieved
from http://www.nytimes.com/2001/06/18/world/
cordial-rivals-how-bush-and-putin-became-friends.
html
Perrett, D. I., & Emery, N. J. (1994). Understanding
the intentions of others from visual signals:
Neurophysiological evidence. Current Psychology
of Cognition, 13, 683–694.
Perrett, D. I., Smith, P. A., Mistlin, A. J., Chitty, A. J.,
Head, A. S., Potter, D. D., . . . Jeeves, M. A. (1985).
Visual analysis of body movements by neurones
in the temporal cortex of the macaque monkey:
A preliminary report. Behavioural Brain Research,
16, 153–170. http://dx.doi.org/10.1016/0166-
4328(85)90089-0
Phillips, R. J. (1979). Some exploratory experiments on
memory for photographs of faces. Acta Psychologica,
43, 39–56. http://dx.doi.org/10.1016/0001-6918(79)
90012-X
Porter, G., Hood, B. M., Troscianko, T., & Macrae, C. N.
(2006). Females, but not males, show greater
pupillary response to direct- than deviated-gaze
faces. Perception, 35, 1129–1136. http://dx.doi.
org/10.1068/p5438
Posner, M. I. (1980). Orienting of attention. Quarterly
Journal of Experimental Psychology, 32, 3–25. http://
dx.doi.org/10.1080/00335558008248231
Pressman, M. R., Diphillipo, M. A., & Fry, J. M. (1986).
Senile miosis: The possible contribution of disordered
359
Adams and Nelson
sleep and daytime sleepiness. Journal of Gerontology,
41, 629–634. http://dx.doi.org/10.1093/
geronj/41.5.629
Provine, R., Krosnowski, K., & Brocato, N. (2009).
Tearing: Breakthrough in human emotional
signaling. Evolutionary Psychology, 7, 52–56.
Puce, A., Allison, T., Bentin, S., Gore, J. C., & McCarthy, G.
(1998). Temporal cortex activation in humans
viewing eye and mouth movements. Journal of
Neuroscience, 18, 2188–2199.
Putman, P., Hermans, E., & van Honk, J. (2006). Anxiety
meets fear in perception of dynamic expressive gaze.
Emotion, 6, 94–102.
Redican, W. K. (1982). An evolutionary perspective on
human facial displays. In P. Ekman (Ed.), Emotion
in the human face (2nd ed., pp. 212–280). New York,
NY: Cambridge University Press.
Reid, V. M., & Striano, T. (2005). Adult gaze influences
infant attention and object processing: Implications
for cognitive neuroscience. European Journal of
Neuroscience, 21, 1763–1766. http://dx.doi.org/
10.1111/j.1460-9568.2005.03986.x
Richeson, J. A., Todd, A. R., Trawalter, S., & Baird, A. A.
(2008). Eye-gaze direction modulates race-related
amygdala activity. Group Processes and Intergroup
Relations, 11, 233–246. http://dx.doi.org/10.1177/
1368430207088040
Rieger, G., & Savin-Williams, R. C. (2012). The eyes have
it: Sex and sexual orientation differences in pupil
dilation patterns. PLoS ONE, 7, e40256. http://dx.doi.
org/10.1371/journal.pone.0040256
Rigato, S., Menon, E., Johnson, M. H., & Farroni, T.
(2011). The interaction between gaze direction and
facial expressions in newborns. European Journal of
Developmental Psychology, 8, 624–636. http://dx.doi.
org/10.1080/17405629.2011.602239
Rinn, W. E. (1984). The neuropsychology of facial
expression: A review of the neurological and
psychological mechanisms for producing facial
expressions. Psychological Bulletin, 95, 52–77. http://
dx.doi.org/10.1037/0033-2909.95.1.52
Ross, C. E., & Mirowsky, J. (1984). Men who cry. Social
Psychology Quarterly, 47, 138–146.
Rottenberg, J., Gross, J. J., Wilhelm, F. H., Najmi, S., &
Gotlib, I. H. (2002). Crying threshold and intensity
in major depressive disorder. Journal of Abnormal
Psychology, 111, 302–312.
Rottenberg, J., Wilhelm, F. H., Gross, J. J., & Gotlib,
I. H. (2003). Vagal rebound during resolution of
tearful crying among depressed and nondepressed
individuals. Psychophysiology, 40, 1–6.
Rubin, L. S. (1970). Pupillary reflexes as objective
indices of autonomic dysfunction in the differential
360
diagnosis of schizophrenic and neurotic behavior.
Journal of Behavior Therapy and Experimental
Psychiatry, 1, 185–194. http://dx.doi.org/10.1016/
0005-7916(70)90001-7
Russell, T. A., Rubia, K., Bullmore, E. T., Soni, W.,
Suckling, J., Brammer, M. J., . . . Sharma, T. (2000).
Exploring the social brain in schizophrenia:
Left prefrontal underactivation during mental
state attribution. American Journal of Psychiatry,
157, 2040–2042. http://dx.doi.org/10.1176/appi.
ajp.157.12.2040
Sato, W., Kochiyama, T., Uono, S., & Yoshikawa, S.
(2008). Time course of superior temporal sulcus
activity in response to eye gaze: A combined fMRI
and MEG study. Social Cognitive and Affective
Neuroscience, 3, 224–232. http://dx.doi.org/10.1093/
scan/nsn016
Sato, W., Yoshikawa, S., Kochiyama, T., & Matsumura, M.
(2004). The amygdala processes the emotional
significance of facial expressions: An fMRI
investigation using the interaction between
expression and face direction. NeuroImage,
22, 1006–1013. http://dx.doi.org/10.1016/
j.neuroimage.2004.02.030
Schiffer, B., Pawliczek, C., Müller, B. W., Gizewski, E. R., &
Walter, H. (2013). Why don’t men understand
women? Altered neural networks for reading the
language of male and female eyes. PLoS ONE,
8, e60278. http://dx.doi.org/10.1371/journal.
pone.0060278
Schulz, R., & Barefoot, J. (1974). Non-verbal responses
and affiliative conflict theory. British Journal of Social
and Clinical Psychology, 13, 237–243. http://dx.doi.
org/10.1111/j.2044-8260.1974.tb00116.x
Scott, L. N., Levy, K. N., Adams, R. B., Jr., & Stevenson,
M. T. (2011). Mental state decoding abilities in
young adults with borderline personality disorder
traits. Personality Disorders, 2, 98–112. http://dx.doi.
org/10.1037/a0020011
Senju, A., & Csibra, G. (2008). Gaze following in human
infants depends on communicative signals. Current
Biology, 18, 668–671. http://dx.doi.org/10.1016/
j.cub.2008.03.059
Sherman, M., & Sherman, I. C. (1925). Sensori-motor
responses in infants. Journal of Comparative
Psychology, 5, 53–68. http://dx.doi.org/10.1037/
h0074472
Shields, S. A. (2002). Speaking from the heart: Gender
and the social meaning of emotion. New York, NY:
Cambridge University Press.
Siegle, G. J., Steinhauer, S. R., Carter, C. S., Ramel, W., &
Thase, M. E. (2003). Do the seconds turn into hours?
Relationships between sustained pupil dilation
in response to emotional information and self-
reported rumination. Cognitive Therapy and Research,

27, 365–382. http://dx.doi.org/10.1023/
A:1023974602357
Sokolski, K. N., Nguyen, B. D., & DeMet, E. M. (2000).
Decreases in dilated pupil size in depressed patients
with age may reflect adrenergic changes. Psychiatry
Research, 94, 267–272. http://dx.doi.org/10.1016/
S0165-1781(00)00155-4
Spezio, M. L., Adolphs, R., Hurley, R. S., & Piven, J.
(2007). Abnormal use of facial information in
high-functioning autism. Journal of Autism and
Developmental Disorders, 37, 929–939. http://dx.doi.
org/10.1007/s10803-006-0232-9
Spezio, M. L., Huang, P.-Y. S., Castelli, F., & Adolphs,
R. (2007). Amygdala damage impairs eye contact
during conversations with real people. Journal of
Neuroscience, 27, 3994–3997. http://dx.doi.org/
10.1523/JNEUROSCI.3789-06.2007
Stass, J. W., & Willis, F. N. (1967). Eye contact, pupil
dilation, and personal preference. Psychonomic
Science, 7, 375–376. http://dx.doi.org/10.3758/
BF03331131
Stern, D. N. (1977). The first relationship: Mother and
infant. Cambridge, MA: Harvard University Press.
Stevenson, H. A., Jones, M. E., Rostron, J. L., Longman, L.
P., & Field, E. A. (2004). UK patients with primary
Sjögren’s syndrome are at increased risk from clinical
depression. Gerodontology, 21, 141–145. http://
dx.doi.org/10.1111/j.1741-2358.2004.00027.x
Stevenson, M. T., Soto, J. A., & Adams, R. B., Jr. (2012).
More than meets the eye: The role of self-identity
in decoding complex emotional states. Emotion, 12,
882–886.
Streit, M., Wölwer, W., & Gaebel, W. (1997). Facial-
affect recognition and visual scanning behaviour in
the course of schizophrenia. Schizophrenia Research,
24, 311–317. http://dx.doi.org/10.1016/S0920-
9964(96)00126-0
Sturm, R. A., & Larsson, M. (2009). Genetics of human
iris colour and patterns. Pigment Cell and Melanoma
Research, 22, 544–562.
Tanaka, J. W., & Farah, M. J. (1993). Parts and wholes
in face recognition. Quarterly Journal of Experimental
Psychology, Section A: Human Experimental
Psychology, 46, 225–245. http://dx.doi.org/10.1080/
14640749308401045
Thayer, S. (1969). The effect of interpersonal looking
duration on dominance judgments. Journal of Social
Psychology, 79, 285–286. http://dx.doi.org/10.1080/0
0224545.1969.9922427
Tipples, J. (2005). Orienting to eye gaze and face
processing. Journal of Experimental Psychology:
Human Perception and Performance, 31, 843–856.
Tipples, J. (2006). Fear and fearfulness potentiate
automatic orienting to eye gaze. Cognition and
Eye Behavior and Gaze
Emotion, 20, 309–320. http://dx.doi.org/10.1080/
02699930500405550
Tomasello, M., Call, J., & Hare, B. (1998). Five primate
species follow the visual gaze of conspecifics. Animal
Behaviour, 55, 1063–1069. http://dx.doi.org/10.1006/
anbe.1997.0636
Tombs, S., & Silverman, I. (2004). Pupillometry: A
sexual selection approach. Evolution and Human
Behavior, 25, 221–228. http://dx.doi.org/10.1016/
j.evolhumbehav.2004.05.001
Tomlinson, N., Hicks, R., & Pellegrini, R. (1978).
Attributions of female college students to variations
in pupil size. Bulletin of the Psychonomic Society, 12,
477–478. http://dx.doi.org/10.3758/BF03329742
Trawalter, S., Todd, A. R., Baird, A. A., & Richeson, J. A.
(2008). Attending to threat: Race-based patterns of
selective attention. Journal of Experimental Social
Psychology, 44, 1322–1327. http://dx.doi.org/
10.1016/j.jesp.2008.03.006
Trice, L. B. (1990). Meaningful life experience to the
elderly. Image: The Journal of Nursing Scholarship, 22,
248–251. http://dx.doi.org/10.1111/j.1547-5069.
1990.tb00223.x
Tuppini, T. (2012). Kant, blows of tears. In P. Giordanetti,
R. Pozzo, & M. Sgarbi (Eds.), Kant’s philosophy
of the unconscious (pp. 147–176). http://dx.doi.
org/10.1515/9783110265408.147
Tyler, C. W. (1998). Painters centre one eye in portraits.
Nature, 392, 877–878. http://dx.doi.org/10.1038/
31833
van der Zwaag, W., Da Costa, S. E., Zürcher, N. R.,
Adams, R. B., Jr., & Hadjikhani, N. (2012). A 7
telsa fMRI study of amygdala responses to fearful
faces. Brain Topography, 25, 125–128. http://dx.doi.
org/10.1007/s10548-012-0219-0
Van Gerven, P. W. M., Paas, F., Van Merriënboer, J.
J. G., & Schmidt, H. G. (2004). Memory load
and the cognitive pupillary response in aging.
Psychophysiology, 41, 167–174.
van Hooff, J. (1972). A comparative approach to the
phylogeny of laughter and smiling. In R. A. Hinde
(Ed.), Non-verbal communication (pp. 209–241). New
York, NY: Cambridge University Press.
van Hooff, J. (1976). The comparison of facial expression
in man and higher primates. In M. von Cranach
(Ed.), Methods of inference from animal to human
behaviour: Proceedings (pp. 165–169). Chicago, IL:
Aldine.
Van Houten, R., Nau, P. A., MacKenzie-Keating, S. E.,
Sameoto, D., & Colavecchia, B. (1982). An analysis
of some variables influencing the effectiveness
of reprimands. Journal of Applied Behavior
Analysis, 15, 65–83. http://dx.doi.org/10.190/
jaba.1982.15-65
361
Adams and Nelson

Vassallo, S., Cooper, S. L., & Douglas, J. M. (2009). Whalen, P. J. (1998). Fear, vigilance, and ambiguity:
Visual scanning in the recognition of facial affect: Is Initial neuroimaging studies of the human amygdala.
there an observer sex difference? Journal of Vision, 9, Current Directions in Psychological Science, 7,
11. http://dx.doi.org/10.1167/9.3.11 177–188. http://dx.doi.org/10.1111/1467-8721.
ep10836912
Vingerhoets, A., & Cornelius, R. (2001). Adult crying: A
biopsychosocial approach. Hove, England: Brunner- Wieser, M. J., Pauli, P., Alpers, G. W., & Mühlberger, A.
Routledge. (2009). Is eye to eye contact really threatening
and avoided in social anxiety? An eye-tracking
Vingerhoets, A., Cornelius, R. R., Van Heck, G. L., &
and psychophysiology study. Journal of Anxiety
Becht, M. C. (2000). Adult crying: A model and
Disorders, 23, 93–103. http://dx.doi.org/10.1016/j.
review of the literature. Review of General Psychology,
janxdis.2008.04.004
4, 354–377.
Wikan, U. (1988). Bereavement and loss in two Muslim
Walker-Smith, G. J., Gale, A. G., & Findlay, J. M. (1977).
communities: Egypt and Bali compared. Social
Eye movement strategies involved in face perception.
Science and Medicine, 27, 451–460. http://dx.doi.org/
Perception, 6, 313–326. http://dx.doi.org/10.1068/
10.1016/0277-9536(88)90368-1
p060313
Wyland, C. L., & Forgas, J. P. (2010). Here’s looking at
Warner, L. R., & Shields, S. A. (2007). The perception of
you kid: Mood effects on processing eye gaze as a
crying in women and men: Angry tears, sad
heuristic cue. Social Cognition, 28, 133–144. http://
tears, and the “right way” to cry. In U. Hess &
dx.doi.org/10.1521/soco.2010.28.1.133
P. Philippot (Eds.), Group dynamics and emotional
expression (pp. 92–117). http://dx.doi.org/10.1017/ Yuki, M., Maddux, W. W., & Masuda, T. (2007). Are
CBO9780511499838.006 the windows to the soul the same in the East and
West? Cultural differences in using the eyes and
Watanabe, K., Matsuda, T., Nishioka, T., & Namatame,
mouth as cues to recognize emotions in Japan and
M. (2011). Eye gaze during observation of static
the United States. Journal of Experimental Social
faces in deaf people. PLoS ONE, 6, e16919. http://
Psychology, 43, 303–311. http://dx.doi.org/10.1016/
dx.doi.org/10.1371/journal.pone.0016919
j.jesp.2006.02.004
Watanabe, S., Kakigi, R., & Puce, A. (2001). Occipito­
Zuckerman, M., DePaulo, B. M., & Rosenthal, R. (1981).
temporal activity elicited by viewing eye movements:
Verbal and nonverbal communication of deception.
A magnetoencephalographic study. NeuroImage, 13,
In L. Berkowitz (Ed.), Advances in experimental
351–363. http://dx.doi.org/10.1006/nimg.2000.0682
social psychology (Vol. 14, pp. 1–59). New York, NY:
Webbink, P. (1986). The power of the eyes. New York, NY: Academic Press.
Springer.
Weitz, S. (1979). Nonverbal communication: Readings with
commentary. New York, NY: Oxford University Press.

362
 Chapter 14
Signs, Signals, and Symbols
in Olfactics
Jeannette Haviland-Jones, Patricia Wilson, and Robin Freyberg
In reviewing olfactics—nonverbal communication
using the olfactory system—we use a classic set
of concepts: signs, signals, and symbols. Though
we are using this system to organize research that
is drawn from subfields in psychology, sociology,
anthropology, and neuroscience, we are acutely
aware of both advantages and disadvantages. Con-
ceptualizing olfactory communication as signs,
signals, or symbols brings histories of theory and
interpretation from other areas that are impor-
tant for building new research programs revealing
many gaps in the current basic research but also
potentially putting conceptual constraints on new
research. Put simply, occasionally in this chapter
the reader will be cautioned that using this sys-
tem has given rise to speculation necessary at this
juncture. In spite of any drawbacks, using this
well-established system for human communication
reveals many exciting and previously unsuspected
human nonverbal connections in an area that is
expanding rapidly.
SIGNS
As we use the term, signs are communications that
are part of an actual experience. For example, in
facial movement, a zygomatic muscle movement,
even one that is brief and small, is part of the happy
experience and can both inform and influence
other people. This sign can be shared and becomes
nonverbal communication. We review evidence
for similar chemosensory (see Table 14.1 for defini-
tions of olfactory-related terms) or olfactory cues for
http://dx.doi.org/10.1037/14669-014
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
emotions and other personal communications. This
area has emerged only in the present century with
the discovery that people can produce and detect
odors for fear, happy, and disgust. Personal iden-
tity body odors may also be a type of sign, and we
include some of this research.
SIGNALS
We use the term signals to classify those olfactory
cues that signal something that may be separate
from the actual experience. The separation of signs
and signals may not be absolute, as a signal may
be a modified sign. In modern Asian societies, it is
not uncommon that a woman will eat products that
affect personal body odor—an identity odor qualify-
ing as a sign—intending that she will smell sweeter,
thereby projecting a more contented temperament.
This may be the equivalent of a social smile in its
intent. However, nonhuman natural scents and
manufactured scents provide an enormous variety of
signals for anything from clothes to shops. Some of
these, such as the altered body odor just described,
have sign qualities as well as signal. For example,
some floral shops add a modified (manufactured)
rose fragrance to their air. Though the scent is not
coming from the flowers in the shop any more than
the “sweet” odor of the young woman in the earlier
example is coming from her personal identity, the
artificial floral scent used is a good mimic and sig-
nals desirable qualities for a person or a product.
Later in the chapter, we note that some signals
may have additive power because they are partial
363
Haviland-Jones, Wilson, and Freyberg

TABLE 14.1

Glossary of Olfactics Terms as Used in This Chapter

Term Definition
Chemosensory Pertaining to those senses that are impacted by molecular action (olfaction and gustation)
Chemosignal An odor that conveys a “message”
Fragrance A scent with a pleasant quality
Odor/odorants An airborne chemical that stimulates olfactory receptors that may or may not be consciously detected
Pheromone A chemical emitted by one member of a species that is detected by another member of the same
species and produces physiological, neuroendocrine, and/or behavioral effects traditionally used to
describe processes in nonhuman species
Scent A quality of an odor, implying conscious detection
Semiochemical A broad category including pheromones or other chemicals with the same communicative functions
as pheromones; this term is used for humans and includes interspecies communicative functions
Smell A sensation resulting from adequate stimulation of the olfactory organ, implying conscious detection

Note. The terms and definitions provided here reflect our grasp of common usage in published research. We were sur-
prised not to find an existent glossary for olfactics—probably because of the emergent aspect of the field.

signs and that the border between sign and sig-
nal may be blurred more often than is realized.
However, many signals seem only to have learned
connections to an object. Clothes and many other
products may have “brand” scents, sometimes called
“signatures,” so that the product is recognized by
the scent signal. In these cases, the signal may have
no sign qualities but is a learned association with
some attention-altering features. The research on
the role of olfaction in affecting memory is quite
germane to the power of olfaction signaling and is
briefly noted.
SYMBOLS
Many years ago, one of us watched Paul Ekman,
the founder of much of the modern work on com-
munication in emotion and the ultimate master of
facial movement (see Ekman & Friesen, 1975).
He was using facial movements during a speech to
indicate commas, periods, and exclamation points.
To the informed audience, this was very funny; to
the uninformed, it was incomprehensible. Ekman
had made abstract symbols from movements that
might have been part of signs. He had given these
movements meaning unrelated to their usual sign
value. The symbolic communicative purposes of
olfaction comprise an emerging, exciting research
area crossing linguistic and cognitive research with
sensory. Here, the review includes the recent
research on fishy scents symbolizing “fishy” or
untrustworthy people as well as the effect of clean
scents to signal virtue and compassion. We also
briefly note examples in sociology and anthropol-
ogy pertaining to the smelly symbols of the “great
unwashed” or the heavenly scents in religious
ceremonies.
Although it may seem as if the three categories
of sign, signal, and symbol differ on a continuum
of unintentional to intentional communication, in
this review we show that this is a complex and still
puzzling aspect of olfactics. Although signs are not
likely to be intentionally sent, it is questionable
whether they may be intentionally or consciously
used by the receiver and whether the intent to
detect and use such information affects the result.
Signals and symbols can be intentionally sent and
consciously received, but in normal olfactics dis-
course, it is not required and may not be frequent.
Again, the process and effectiveness may vary with
the degree of conscious use. For example, if one
uses words to describe a person as untrustworthy—
“He is a cheat”—it is generally done intentionally.
Although there is a clear verbal signal, it may be
tempered by the authority of the speaker. If the mere
presence of a fishy smell leads to an attribution of
untrustworthiness, the fishy smell is symbolic, but
it may not be at all intentional, and the process may

364

not necessarily be apprehended. The belief that the
fishy person is untrustworthy may feel more intui-
tive, and less dubious, when the information comes
from an odor not consciously processed than from a
verbal assertion. Later, evidence that subconscious
process in olfactics is more frequent than conscious
process is considered (e.g., Sela & Sobel, 2010).
The present state of research in olfactics contributes
new and unresolved perspectives on questions of
intentionality and conscious or subconscious non-
verbal communication.
A BRIEF BACKGROUND
Until recently, little attention has been given to
olfactics. Historians and anthropologists point to a
derisive attitude toward smells both in the modern
Western culture as well as more particularly in the
Western research culture (see Claussen, Howes, &
Synnott, 1994, for extensive review). Though this
has been countered in some respects by excellent
applied research in the flavor and fragrance industry
(e.g., the Monell Institute; the Sense of Smell Insti-
tute), it still has been a poor cousin in the larger
academic community. Naturally, the methods for
conducting excellent research also were underdevel-
oped (see Chapter 21, this handbook).
Even though olfactics is neglected in human
research, there is an extensive research history
for nonhuman olfactory communication. Among
insects, chemosensory communication is the most
effective and specific communication system (most
famously noted in ant communities by Hölldo-
bler & Wilson, 1990). Among vertebrates, there is
substantial evidence that chemosensory commu-
nication can provide sexual, caretaking, food, and
predator information. Nevertheless, many theories
about human olfaction have argued persuasively
that nonverbal communication using olfactory
information among humans is either lacking or is
insignificant compared to other sensory systems and
cognitive functions. It is not always clear whether
the argument is that humans cannot adequately
sense chemosensory cues (i.e., decode them) or that
humans do not send olfactory cues (i.e., encode
them) or principally that chemo­sensory cues are
not influential in comparison to non­verbal messages
Signs, Signals, and Symbols in Olfactics
from other sensory systems. There is a need for care-
ful research into each of these hypotheses.
The embeddedness of the olfactory system
throughout the neural network may have occurred
because the olfactory system is one of the earli-
est to develop both in phylogeny and ontogeny. In
comparison with other senses, the olfactory system
has direct cortical projections (no thalamic relay
nucleus). Some olfactory fibers reach the orbitofron-
tal cortex (Stockhorst & Pietrowsky, 2004), which
is implicated in emotional processing, judgment,
and decision, thus influencing a wide variety of
behavioral responses (Bechara, Damasio, & Damasio,
2000). Perhaps extending its potential for future
research, it is now clear that more of the human
genome is dedicated to olfaction than to any other
sensory system (see Buck & Axel, 1991). Contrary
to earlier estimates, Bushdid, Magnasco, Vosshall,
and Keller (2014) calculated that discrimination
of more than a trillion olfactory stimuli would be
an underestimate of human capacity. Rather than
being “primitive” and simple, olfaction may be the
most integrated and complex of the sensory systems.
The current challenge is to discover what this pres-
ently means for human olfactics, that is, nonverbal
communication using airborne compounds, which
include odors, scents, and semiochemicals.
HUMAN SIGNS
Some years ago we entered this field with substan-
tial skepticism. For fun, we worked with a middle
school science class to show whether there is a real
fear odor, as folklore attests. We did not expect
to find a fear odor, only to show how research
addresses such superstitions. First, the students
made two short videos to be used for the fear and
happy mood inductions. These were composed of
back-to-back short segments of people giving full-
face fear or happy faces. Second, these two videos
were shown to the science class, fear on one day
and happy on another, and body odor was collected.
Class members did not wear deodorant and collected
body odor by placing gauze pads in the axillary area.
A control collection was made by having an equiva-
lent number of pads in an open container in the sci-
ence room. The pads were stored in −80 °C freezers
365
Haviland-Jones, Wilson, and Freyberg
to prevent bacteria growth. Finally, a few weeks
later, we placed each set of pads—fear, happy, and
control—in identical containers identified only with
codes (a double-blind procedure). Any students,
even those not in the class, who wanted to smell the
buckets and guess which went with each condition
were allowed to do so. To our surprise, the students
were significantly above chance in matching the
fear and happy axillary pads to the mood-inducing
stimuli (Haviland-Jones & McGuire, 1999). In this
and most studies of mood-odor identification that
only require one or two opportunities to identify the
mood odor, the reliability of judgments was only
slightly above chance but still significant. However,
a recent study (Haviland-Jones, Wilson, & McGuire,
2014) has indicated that this low percentage of iden-
tification may be deceptive. With multiple trials and
a more conservative comparison task, about half of
the sample was nearly 100% correct after five expo-
sures (no feedback). The reason for incorrect scores
or scores only slightly above chance in the remain-
ing half of the group requires further study and
perhaps training. Therefore, the question of how
accurately people judge mood odors is still fairly
open, but it seems likely that some people are highly
accurate and possibly that others can be trained.
Clearly, there are many open questions.
Chen and Haviland-Jones (2000) did a larger
and more controlled version of the original school
study of mood odor, finding the same results, and
a new field was launched. One group of people
could “encode” an emotion sign with body scent,
and another group could “decode” it. In spite of our
initial skepticism, we had authenticated a human
olfactory sign for the field of olfactics. It is not just
a superstition; fear and other emotion odors can be
detected.
Mood Signs
Because mammals generally convey signs of alarm,
including chemosensory signs, to conspecifics,
human chemosensory mood signs for alarm should
not have been so surprising (e.g., Brechbühl,
Klaey, & Broillet, 2008). Although most of the sub-
sequent human research has been concerned with
the so-called alarm signs, there is a smaller amount
of research on signs for happiness and disgust,
366
suggesting that this area will expand substantially in
the coming years. There is accumulating evidence
that the semiochemical signs of fear, happiness,
and disgust not only can be “decoded” as well as
“encoded” but that they serve several communica-
tive functions. Even though the following review
solidly demonstrates communicative effects for
mood signs, it remains a puzzle as to why our study
participants have a persistent lack of confidence in
their own ability to detect mood odors.
The chemical compounds used for the signs in
most of the research in this area are gathered much
as we did in the mood-odor identification study
(described earlier) with the young science class,
that is, with the donors of the mood signs generally
wearing a pad tucked into their armpits to collect
body scent while watching a mood-induction video.
Some researchers use more naturalistic situations to
produce their mood signs, such as anxiety producing
exams (Pause, Lübke, Laudien, & Ferstl, 2010); other
researchers use more extreme mood inducements,
such as people sky diving (Mujica-Parodi & Strey,
2006) or following a high rope course (Zernecke
et al., 2011). Very little is known definitively about
the chemical composition of these signs or about how
small differences in components may result in either
individual or in behavioral differences.
Fear.  The emerging literature on fear (anxiety,
alarm) human chemosignals illustrates that they
have sign qualities. We include the wider variety of
terms here because there is no known organic com-
pound that presently distinguishes the mood com-
pounds for fear, alarm, or anxiety. The mood odors
are collected in different circumstances that might
well lead to distinctions that may be made in the
future. Nevertheless, these mood odors are produced
as part of physiological emotional behavior, as they
can be demonstrated to impact brain mechanisms
responsible for mood (e.g., activation of the amyg-
dala; Mujica-Parodi et al., 2009). They are part of a
whole behavior pattern and qualify under our defi-
nition as olfactory signs. They are observed not only
in controlled, brief mood-induction laboratory pro-
cedures but also can be collected for longer periods
of time (Pause et al., 2010). From the early research
(Chen & Haviland-Jones, 2000; Haviland-Jones &

McGuire, 1999), we knew that fear body odors
could be identified. This was the first indication that
they had sign qualities in communicating somewhat
reliable information from one person to another.
Later research has shown a wider array of functions,
particularly ones affecting judgments of visual mood
signs. In all the studies, an alarm/anxiety chemo-
signal leads to attributing more negative and less
positive expression to neutral or ambiguous faces.
For example, Pause, Ohrt, Prehn, and Ferstl (2004)
primed participants with happy, fearful, and sad
faces, asking the participants to then rate the emo-
tion on a neutral face. Usually, priming with a happy
face leads to “seeing” more happiness in the neutral
face; however, the anxiety chemosignal reduced the
effectiveness of the happy prime. Three more stud-
ies used morphing software to produce gradually
changing images of facial expression from neutral to
clear emotion visual signs. The emotional displays
on the faces varied between neutral to angry faces
(Mujica-Parodi et al., 2009), fear to happy faces
(Zhou & Chen, 2009), and neutral to happy faces
(Zernecke et al., 2011). The olfactory mood sign or
anxiety chemosignal affected judgments at points
of ambiguity but not at the extremes—always lead-
ing to a more negative emotional bias. One study
that investigated the impact of fear mood odor on
other processes suggested that olfactory communica-
tion may operate differently than the other sensory
systems. In this study, exposure to the fear mood
odor was expected to decrease reaction times, as
would exposure to a fear visual cue—that is, see-
ing a snake results in a fast response. Instead, the
reaction times were slower when the participants
were exposed to a fear odor (Chen, Katdare, &
Lucas, 2006). The researchers were hard pressed to
explain the slower reactions. However, we suggest
a different process. If you can smell fear from other
humans but not see them or a dangerous target,
the potential threat might be far away. Freezing or
moving very cautiously would be a smart reaction.
On the other hand, if you see the dangerous target,
preparing to move as fast as possible might be the
smarter reaction. This is clearly speculative, but it
is useful speculation because it suggests that differ-
ent sensory systems may serve different functions
and that one important function for olfaction is an
Signs, Signals, and Symbols in Olfactics
implicit search function, an idea we explore later in
the chapter.
Disgust.  Recent research (de Groot, Smeets,
Kaldewaij, Duijndam, & Semin, 2012) has added
odors collected from people who are disgusted to
the list of distinguishable olfactory signs. In the
studies by de Groot et al. (2012), emphasis was
placed on the social communicative function of the
fear and disgust mood odors. Mood odors were col-
lected from donors who were anxious or who were
disgusted. Both disgust and fear mood odor pro-
duced matching facial expressions on the part of the
perceivers or decoders as well as associated fear- or
disgust-related behaviors. In our terms, this shows
that both the fear and disgust odor compounds have
nonverbal communicative sign values. Both the fear
and disgust facial movements that are stimulated by
the odors are clearly motor behaviors. An emerg-
ing direction in the motor system literature (mirror
neurons) is focusing on the likelihood that olfactory
stimuli may impact the motor system via differ-
ent neural pathways than visual stimuli (Aglioti &
Pazzaglia, 2011; Tubaldi et al., 2011). This is note-
worthy because mirror neurons are implicated in
nonverbal behaviors.
Happy.  The mood sign research suggests that
investigators can identify olfactory signs signify-
ing safety, happiness, or calmness/relaxation (e.g.,
Chen & Haviland-Jones, 2000; Haviland-Jones &
McGuire, 1999; Haviland-Jones & Wilson, 2008).
Though these can be identified, there is no exten-
sion to other communicative processes that we are
aware of. It is uncertain why there is little research
impetus here. In the visual sensory field, positive,
happy signs—such as smiles—are discrete, are
easily recognized, emerge early in life, and have
considerable social significance. Olfactory signs for
happiness or calmness have the potential for similar
powerful effects. The fact that in Western culture
people seem to have no awareness that happy or
safety mood odors might be operating is not suf-
ficient evidence that they have negligible impact.
There is little confidence for signs of alarm in
Western culture (or other identity signs; discussed
later), yet they have demonstrable impact. If happy
chemosignals affect moods, behaviors, and cognitive
367
Haviland-Jones, Wilson, and Freyberg
processes, then their presence or absence in homes,
hospitals, or work places could have significant
effects on mental and physical health (see Haviland-
Jones, Hudson, Wilson, Freyberg, & McGuire,
2013). This is not known, clearly, but the hypothesis
reveals a gap in the research area.
Developmental research.  Coming from a different
perspective, developmental research on olfaction
may be related to nonverbal communication of posi-
tive emotion or social bonding. Newborns certainly
are capable of detecting a variety of biological and
artificial odors (e.g., Schaal, 1988). Macfarlane
(1975) was one of the first to discover that neonates
were more likely to orient toward a breast pad worn
by the infant’s mother than to one that had been
worn by an unfamiliar female. These findings revo-
lutionized olfactory research and even today provide
an important entry for future research on nonverbal
communication and mood signs. It is especially
significant because olfaction is the only sensory
system directly connected to the ventral tegmental
area, a dopamine producing area of the brain (­ Ashby,
Isen, & Turken, 1999), which is likely linked to
mood regulation from the earliest moments of life.
Beyond the orienting response, familiar odors have
also been found to exert a soothing effect on infants.
Both maternal breast milk odor and other familiar,
pleasant artificial odorants may reduce crying and
grimacing (Goubet, Strasbaugh, & Chesney, 2007;
Varendi, Christensson, Porter, & Winberg, 1998)
as well as reduce cortisol levels indicative of stress
(Nishitani et al., 2009). This emotional response
in reaction to familiar odors seems to have a long-
lasting impact extending into toddlerhood. For
example, neonates exposed to a chamomile-scented
balm on their mothers’ breasts showed fewer facial
expressions of disgust when exposed to chamomile
scent at 7 and 21 months of age (Delaunay-El Allam,
Soussignan, Patris, Marlier, & Schaal, 2010). Thus,
the positive effects of the chamomile scent persisted
for many months. In children never exposed, the
scent had either no or negative effects, compared to
a no-odor control. Also noting that familiar scents
affect positive mood and behavior, Coffield, Mayhew,
Haviland-Jones, and Walker-Andrews (2014) found
that exposure to the common odors of rose-scented
368
baby powder or fresh-evergreen cleaner increased
infants’ attention to a woman’s sad face and reduced
the infants’ distress facial expression to that face.
In this case, the familiar odors helped modulate
stress. It appears that some odors have positive
mood effects, increasing positive affective behavior,
modulating negative mood, and increasing atten-
tion. Whether human odors for happiness can have
similar effects is unknown, but it seems likely given
the importance of positive emotion. In all these
cases, the mechanisms are unknown. In Delaunay-El
Allam et al.’s (2010) study of chamomile, it is
likely that the odor was a conditioned stimulus
that became associated with feeding, the uncondi-
tioned stimulus, and thus is a classical conditioning
demonstration. This is less clear in Coffield et al.’s
(2014) study. Although the scent similar to baby
powder may have been associated with caretaking,
the evergreen scent would have been far more likely
to have been an ambient odor not associated with
infant care. This leads to the possibility that some
odors have pharmacological sorts of effects, a point
discussed later. In other words, it is likely that there
are several routes for these phenomena.
Sexual Attraction Signs—The Pheromone
Controversy
In 1959, Karlson and Luscher coined the term
pheromones to describe observations of communica-
tion via chemical means that they had noted in their
studies of insects. Generally speaking, pheromones
are airborne chemicals that are emitted by one
member of a species; are detected by a conspecific;
and result in specific behavioral, neural, or endo-
crine changes. In the past half century and more
since Karlson and Luscher’s (1959) work, chemo-
sensory signs that qualify as pheromones have been
documented in most species. Pheromones are asso-
ciated in the popular press with sexual attraction.
Often there is great resistance to the proposition
that human sexual behavior in particular might be
greatly affected by pheromonal compounds.
One line of research related to sexual communi-
cation, whether or not it falls under the pheronomal
definition, focuses on several chemicals commonly
found to elicit particular sexual or courting behav-
iors in animals and also found to be present in

humans. The early research from the mid-1970s to
mid-1990s investigated chemicals such as androste-
none and androstenol, finding that exposure to
such chemicals affected feelings of increased attrac-
tiveness and, possibly, territorial behaviors. For
example, women waiting in a dentist’s office chose
seats that had been treated with androstenone
(Kirk-Smith & Booth, 1980).
The more recent investigations have focused on
androstadienone (AND). AND was first identified
as a putative pheromone by Berliner (1994), who
filed a patent claiming that AND and estratetraenol
were sex hormones affecting females and males dif-
ferentially. In women, but not in men, for example,
AND has been shown to have emotional and atten-
tional effects (Jacob, Kinnunen, Metz, Cooper, &
McClintock, 2001), to modify autonomic nervous
system functioning (Bensafi et al., 2003), and to
impact neuroendocrine function (Savic, Berglund,
Gulyas, & Roland, 2001). AND and other potential
pheromonal-like compounds are likely to impart
information about sexuality or at least sexual iden-
tity. In the search engine described later, we propose
that such compounds would lead people to prepare
or search for sexual information, but the right con-
text would be necessary for any behavioral conse-
quences. Even in the case of the mood signs, there is
little evidence of direct behavior change but a good
deal of evidence for attention and attitude changes.
It is quite likely that signs for sexuality conform to
similar processes in humans.
Another contribution to the controversy about
potential innate chemical communication, perhaps
pheromonal, concerns menstrual synchrony. Anec-
dotally, some women reported that the timing of
their menstrual cycles coincided with other women
who were in close proximity to them; this is called
menstrual synchrony. In the first published observa-
tion, McClintock (1971) documented menstrual
synchrony among college roommates in an all-female
college dormitory. Subsequent observations noted
similar patterns of synchronization in Bedouins
(A. Weller & Weller, 1997), office workers (Matteo,
1987; L. Weller, Weller, Koresh-Kamin, & Ben-Shoshan,
1999), and mothers and daughters living together
(A. Weller & Weller, 1993). These patterns seemed
to be dependent on a group of variables, including
Signs, Signals, and Symbols in Olfactics
close proximity, diet, and a structured social sched-
ule (McClintock, 2000). To add to the complexity,
male chemosensory compounds may also play a role
in menstrual synchrony. In a study investigating
possible effects of male axillary secretions, recipi-
ent females demonstrated an increase in luteniz-
ing hormone pulsing (Preti, Wysocki, Barnhart,
Sondheimer, & Leyden, 2003). Such an increase
would impact both the timing and the length of
the menstrual cycle. Thus, chemical communica-
tion between individuals has been shown to impact
biological functioning, here as a form of regulation.
Whether these results constitute evidence of non-
verbal communication is open to debate, but it does
underscore the importance of studying olfactory
communication on many levels. This is an aspect
of potential olfactory communication that would
require new approaches to the topic of olfactics.
One should note that more recent research has
cast a shadow over the original findings related to
menstrual synchrony, as recent attempts to replicate
have failed—perhaps because the living arrange-
ments and intimate associations of college students
have changed (see Schank, 2001; Ziomkiewicz,
2006). This further illustrates the complex con-
textual requirements for this type of regulatory
communication. Regardless, what the menstrual
synchrony investigations spawned was research into
possible mechanisms of chemosensory coordination
or, possibly, communication.
Identity Signs
Few people studying traditional nonverbal com-
munication would consider that each of us has a
unique odor signature that can be identified by
ourselves about 92% of the time (Lundström, Boyle,
Zatorre, & Jones-Gotman, 2008) or by our near rela-
tives (Lundström, Boyle, Zatorre, & Jones-Gotman,
2009) at about 85% accuracy. People can identify
the shirts of newborn relatives even before they have
met them (Porter, Balogh, Cernoch, & Franchi,
1986). Though the judges in these identification
studies are very highly reliable in the identification,
they tend to think they are not reliable (Porter et al.,
1986; Weisfeld, Czilli, Phillips, Gall, & Lichtman,
2003). Yet, from odor alone, strangers as well as our
family can usually know our gender, often our age,
369
Haviland-Jones, Wilson, and Freyberg
and whether we belong to their group. Strangers
may even gather some information about neurotic
personalities and dominance from body chemosen-
sory signs (Sorokowska, 2013). Even though few
people believe they can identify either their own
belongings or those of relatives by odor alone, even
when faced with the evidence, neuroimaging studies
confirm the basic nature of this olfactory identifica-
tion system showing that people have dedicated
neurological pathways for this process (Lundström
et al., 2008).
Signs for friend and foe.  Even from within the
womb, humans begin to use olfaction as a learning
tool for bonding interactions with others. Schaal,
Marlier, and Soussignan (2000) demonstrated that
human fetuses learn about the olfactory environ-
ment through foods digested by their mothers and
subsequently respond through facial expressions to
exposure to those odors several days following birth.
Some have suggested that such in-utero learning
conveys important family and cultural information,
facilitating adaptation to a particular environment
(Mennella, Jagnow, & Beauchamp, 2001; Mennella,
Johnson, & Beauchamp, 1995). Although this
awaits future research, it is possible that humans not
only learn about food in utero but that mood bio-
chemical compounds from a mother also may have
effects on unborn children. Young infants not only
know their mothers’ own body odor but also the fra-
grances used by their mothers, and they attend lon-
ger to sterile gauze pads containing that fragrance
(Schleidt & Genzel, 1990). By fragrance alone,
very young infants possibly could know that they
are with familiar people, not strangers. Going past
infancy, Mallet and Schaal (1998) showed that older
children can identify the t-shirts of their friends.
The children were significantly more accurate in
identifying the owners of the t-shirts for the same
sex than the other sex, and girls were somewhat
more accurate than boys. Clearly, fragrance is a
powerful tool for identifying others and for defining
personal experience from a very early age.
There is some research on the processes that
underlie identity information. Individual body odor
can be identified from urine or sweat among humans
(e.g., Zavazava, Westphal, & Müller-Ruchholtz,
370
1990). Extensive research indicates that immune
system genes, especially those known as human leu-
kocyte antigens (HLAs), play a role in this. Wede-
kind, Seebeck, Bettens, and Paepke (1995) found
that people tend to prefer odors of potential mates
when HLAs are dissimilar to their own. Ober et al.
(1997) found that mate selection in closed commu-
nities tends to reflect HLA differences, other factors
being equal, and argued for immune system advan-
tages in offspring due to this variance. There is evi-
dence that parents who have very similar HLA types
are more likely to suffer from spontaneous abortions
and are more likely to have low birth weight infants
(Reznikoff-Etievant et al., 1991), suggesting some
utility to this type of identity sign.
Sniffing olfactory compounds left in the air by
a person gives a good deal of information. While
investigating new homes for purchase, one of us,
being aware of this phenomenon, found that sniffing
at the threshold while keeping the eyes closed gave
an immediate impression of content or of malcon-
tent, a sort of “welcome” or “watch out.” Although
a careful study would be needed to show how much
of that impression is given by the lingering body
odor of the previous owners as opposed to odors of
building decay or even sounds or temperature, we
speculate that it is well within the realm of possibil-
ity that many dwellings communicate the ages, gen-
ders, happiness, and even more specific attributes of
their occupants and that these communications lead
the stranger entering the territory to be prepared
for defense or for welcome, for alienation, or for
familiarity.
The olfactory information about the states and
types of occupants in a strange territory might
impart very useful information about the suitability
of the site for new human occupancy. For those
migrating from one place of work to another or
from one town or even one continent to another,
the odors encountered in a new place not only tell
whether the people there are like one’s own fam-
ily or tribe, they also give information about the
supportive nature of the environment. Do these
people eat familiar food, are they happy, or are they
interested in finding new mates for their children?
Choices of place are probably continuously made by
using the nonverbal cues of olfactics. Will a woman

sit where a man has just sat, or will she choose a
seat occupied by a woman? Will a nervous person
choose one sidewalk over another based on intuitive
feelings? Will the depression of a roommate
be contagious even when that roommate is away for
the weekend? There are a wide range of research
studies that could address sociological and wellness
questions in this realm, but little experimental
research exists.
In many cultures, there are rites to cleanse homes
of the lingering olfactory messages of the previous
occupants. Although we might have considered
this superstition prior to the mood-odor research,
we now know that mood odors and probably other
odors can be contagious. Being in a place with lin-
gering odors of fear or disgust could well engender
similar expectations in the person who innocently
enters the contaminated space. Perhaps these old
rituals are more than superstition.
Identity signs morphing into signals.  Not only
is it likely that the identification of relatives and
people’s choices of intimate partners are influenced
by genetic information in olfactory compounds but,
as Hämmerli, Schweisgut, and Kaegi (2012) found,
people’s fragrance choices are also likely to be influ-
enced by genetic information. They found genetic
markers that can be used to predict which perfume
ingredients are preferred. Although nonbody scents
that are applied to the body or to objects around a
person would seem to be signals because they are
not part of a chemosignal generated by a person, the
choice of personal body scents is associated with
the identity scent that we have labeled a sign. This
makes it seem as if the scents selected as signals
may be morphed signs. It may not be random that
one person prefers a scent with woody notes, and
another person prefers a scent with floral notes. It
may not be entirely a learned preference or asso-
ciation but instead an amalgam of associations, an
enhancement of genetic components of body scent.
When you detect a person’s perfume, you may be
detecting more information about that person than
just a reaction to advertising. Further evidence that
these HLA-related choices for fragrances have some
signal value for social and emotional interaction is
shown in Freyberg’s research. Freyberg and Ahren
Signs, Signals, and Symbols in Olfactics
(2011) demonstrated that when adolescent girls
were asked to use a new fragrance, first they found it
to be slightly less pleasant than their usual favorite.
The more important finding was that they had lower
levels of social enjoyment while using it, compared
to when they used their preferred fragrance. They
also used fewer words indicative of intimacy in writ-
ten narratives (an associative task for subconscious
effects; see Chapter 21, this handbook), indicating
overall a slight sense of alienation. Further, evidence
that fragrances have a role in signaling identity was
shown in that adolescents were able to identify the
fragrances worn by close friends (Freyberg, 2014).
Finally, there were physiological effects. The ado-
lescents had elevated heart rates when exposed to
the unfamiliar fragrance compared to the familiar
fragrance (Freyberg & Bart, 2012), suggesting some
heightened stress. Taken as a whole, such findings
suggest that fragrances become integrated into peo-
ple’s social relationships and that changing them to
nonpreferred fragrances may also affect those social
relations.
A part of olfactics therefore includes the many
ways that people vary their personal scent, some of
which could form a type of dialogue. Not only do
humans use fragrances to enhance their personal
scent—to leave a sign about their identity, but they
may also use it to disguise or modulate an identity
scent. Just as a hunter canine may roll in the dung
of its prey to cover its alien scent and move into a
herd, saying with its scent “I am one of you, stay in
place,” humans may apply scents that allow them to
enter territories or blend more easily with particular
people and events. People may also adjust their body
scents to smell younger or older, or more confident
and energetic, for example.
In some cultures, a young woman may eat
flowers or use sweet unguents, both floral and
resinous, for a length time before a romantic encoun-
ter or perhaps a wedding and subtly change her
body scent to one that is sweeter and more floral.
This has ancient roots, as is seen in the Bible when
Esther is preparing for marriage and has an extended
fragrance preparation: “to wit, six months of oil of
myrrh and six months with sweet odours” (Esther 2:12).
In some cultures (Stoddart, 1990), it was common
when arranging marriages for the marriage broker to
371
Haviland-Jones, Wilson, and Freyberg
request clothing worn by the potential bride so her
temperament could be assessed from the odors on
her clothing.
Most commonly in North America, people
use antiperspirants/deodorants to moderate their
body odor and fragrances to enhance or disguise it
(for a related discussion, see also Chapter 15, this
handbook). This is done to be polite, but it carries
a strong message of belongingness when it is com-
mon for the whole community. When making a
long-term stay in a foreign port, one may choose to
announce one’s foreign origin using body products
and eating foods from one’s native land. Alterna-
tively, one can quickly acquire body products and
foods from the new culture and begin to assimilate.
Body odor plays at least as large a role in this deci-
sion to assimilate or not as dress plays in com-
municating (e.g., Havlicek & Lenochova, 2006;
Penn & Potts, 1998). In these ways, body odor
communicates “stranger” or “companion.” It may
be manipulated deliberately, though it is most fre-
quently done without conscious awareness. We note
that most people who use fragrances believe that
the scent becomes somewhat different when applied
to the body. We do not know of research related to
this belief, but it is possible that there are chemical
reactions that literally morph nonhuman scents with
human ones creating unique compounds.
The negative or malodor side of this sort of
­identity odor sign into signal morphing is part of
common social discourse in the sense that polite
people do not eat offensive foods before intimate
encounters. One does not eat garlic, for example,
when intending a first kiss, unless of course the
garlic consumption is mutual, separating the garlic-
loving couple from nongarlicky outsiders. Similarly,
one does not take evidence of sports or work activi-
ties that produce potentially unpleasant body odors
into other social situations, unless of course, one
intends to communicate that one has been unusu-
ally virtuous in working or playing. These two
examples highlight the signal value of body odors
and the manipulation of those odors to mask one’s
identity and activities or to bring a focus onto them.
Communication via olfactics seems to
include both the communication that is possible
when people are together and, importantly, the
372
communication that can be distant over space and
time. The absence of a distant loved one is especially
likely to evoke, not only a picture, but an unwashed
pillow or shirt as additional comfort. Although
research on both the central processing (e.g., Lund-
ström et al., 2008, 2009; Prehn-Kristensen et al.,
2009) as well as the social processing of these
phenomena is emerging, it is still incomplete; we
acknowledge both that there is need for continu-
ing research but also that several of the hypoth-
eses described in the previous few paragraphs are
untested.
NONHUMAN OLFACTORY
COMPOUNDS—SIGNS AND/OR SIGNALS
Odor cues from many sources are used as signals
to influence a wide variety of human motivational
states and behaviors. On ourselves we use scents to
signal special occasions, special feelings, and even
that our unique self is present or has been present.
We also use scents on our clothes, in our homes,
and to signal special events or places, again for aes-
thetic reasons or for more specific purposes. Numer-
ous products are infused with scent, and there is
some evidence that this influences attention to the
products and consumers’ memories for them (see
Jellinek, 1996; Morrin & Ratneshwar, 2000, 2003).
There are good reviews of the many ways in
which we use food odors (e.g., Schiffman & Gra-
ham, 2000) as attractant signals and in which we use
decay or fecal odors for the opposite (e.g., Rozin &
Fallon, 1987). Here, however, we are focusing on
the odors that have less obvious evolutionary signifi-
cance and that are more peculiar to humans.
There are contradictory hypotheses about per-
sonal and place odor signals. Many scientists (e.g.,
Herz, 2012) argue strongly that preferences for,
attitudes toward, and liking of odors are associa-
tions that are learned or idiosyncratic and very
mutable. However, others—particularly practi-
tioners in aromatherapy but also many fragrance
scientists—argue strongly that the scents of specific
florals, herbals, and tree barks or resins have effects
that do not require learning (e.g., Keville & Green,
2009). Although both of these hypotheses can be
supported in particular cases, the space between

them may be densely occupied with examples of
scents that have a hint of meaning in their chemical
composition and are then used in predictable ways
so that learned associations provide stronger con-
nections. The study of signs alerts us to the possibil-
ity that some signals are effective because we have
chosen scents that are likely to present signals, not
randomly but with specific qualities mimicking the
human odor signs for moods, and that many other
sign/signals are possible.
There are many excellent reviews of the his-
tory of perfume, the components and psychology
of perfumes, and their many uses (e.g., Jellinek,
1996). What is important for us in reviewing the
signal value of perfumes is that they usually con-
tain ingredients derived from plants and animals
that have the potential to communicate moods or
sexual interest. Generally, the top notes of a fine
perfume come from flower and fruit secretions and
are designed by the flowering plants to attract other
creatures for pollination. These chemosignals may
not have compelling sign values for human beings,
but because their sign value is designed to resonate
across species, in certain instances, we may be one
of those species. Professional perfumers across the
ages and cultures may have information about such
effects, but they are largely untested in the public
sphere. Middle and bottom notes often come from
plant resins that have phytosteroids and from mam-
malian sex attractants, such as civet or musk. Yet,
Stoddart (1990) supported the unproven consensus
that these close biochemical relatives of human hor-
mones and semiochemicals provide “no chance that
they would release the behavior in man for which
they are designed for their own species” (p. 167).
Nevertheless, he wrote that people use perfumes
because of their sexual associations. Stoddart’s view
appears to be that perfumes have some aphrodisiacal
sign compounds and signal attraction, but modern
science has little to demonstrate actual effects on
human coitus. This area of varied sign or signal
effects from natural chemicals in the air, as well
as those embedded in manufactured products, on
human behavior, cognition, emotion, and well-being
is clearly in need of further research.
An interesting example of group signal use of
scents is given by Howes (1987) with respect to the
Signs, Signals, and Symbols in Olfactics
use of incense in rituals or religious ceremonies.
This seems to be a tradition recorded in the earli-
est human histories of Eastern societies—at least
as early as 3000 BCE, Hindi, Jewish, Muslim, and
Christian ceremonies have used incense or aromatic
spices, though there are also periods in which their
use was deplored or even outlawed. No doubt the
long history of use and association contributes
greatly to the continuing success of aromatics in
ceremonies, but it is also possible that there are
qualities of the chosen fragrances that support and
signal a communion with others, even unseen oth-
ers. Howes wrote that the fragrances or incense
used in ritual, much like the use of drumming in
the same type of group context, leads to a shared
mood in a group of people. The scents signal the
shared associations, certainly, but may be especially
effective for additional reasons. Just as drumming in
rituals or ceremonies affects physiological processes,
it is likely that the particular fragrances used do as
well. The feeling of our hearts beating as one may
be much more than a metaphor. The fragrances of
frankincense and myrrh may influence communion
and when combined with many religious associa-
tions could be a powerful signal. Taking a step
further, Stoddart (1990) pointed to phytosterols in
plants whose fundamental structure is the same as
“hormones that orchestrate the reproductive system
of animals” (p. 195). Further, Stoddart reported
that pine resins of the sort we love in our Christmas
trees contain the male hormones of testosterone
and androstenedione. The choice of incense (as
well as perfumes), he argued, is based on erotogenic
compounds. There are no tests that we are aware of
indicating that there are detectable amounts of the
phytosterols present in the air during ceremonies,
but it is an interesting possibility. Of course, the
compounds used may affect mood facets as well as
sexual attraction. Howes also pointed to the “tran-
sitional” metaphor of scents that makes their use
important in ritual or religious ceremonies. Incense
is a symbol of change, of looking beyond the mun-
dane life, searching for a more paradisiacal life. This
transitional symbolism, Howes argued, is central to
the use of odors as signals generally. Putrid odors
signal the transition to decay, flowering odors signal
the transition to spring, and cooking odors signal the
373
Haviland-Jones, Wilson, and Freyberg
transition from raw to edible. Although this observa-
tion takes us into the later section on odors as sym-
bols, it is a significant part of Howes’s hypothesis
about the way odors change our moods and state.
Florals
We first gave thought to this problem of the sign
and signal effects of plants when trying to find a
good explanation for the claim that “flowers make
you happy” or “flowers make you smile.” Pollen
(2002) stated that a “handful” of flowering plants,
as an “evolutionary strategy,” had “manufactured
chemicals” needed to unlock the pleasure centers
in our brain (p. 177). In an experiment with gifts,
Haviland-Jones, Rosario, Wilson, and McGuire
(2005) found that a gift of flowers (compared to
similar gifts) increased reports of happiness over
several days, elicited a 100% display of the true
smile when presented, changed social behavior
among strangers (even of North American men),
and, finally, improved episodic memory scores
among older adults. Subsequent research showed
that the emotion effects could be produced with a
gardenia fragrance alone, even if the fragrance was
not consciously detected (Haviland-Jones et al.,
2013). From other research disciplines, there is
some support for the memory and mood effects of
the flower and floral scent studies. From the mar-
keting domain, Morrin and Ratneshwar (2003)
confirmed that fragrance may provide long-term
detailed memory of products, so the memory effects
may be robust and warrant further investigation.
From the comparative domain, there is some sup-
port for the effects of floral scents on anxiety. Sev-
eral studies have shown anxiolytic effects of rose
and citrus flower oils on mice that are comparable to
exposure to diazepam (valium; e.g., see de Almeida,
Motta, de Brito Faturi, Catallani, & Leite, 2004).
Although the effects may be mild, there is evidence
that at least some floral odors send an olfactory sig-
nal that includes significant social and emotional
effects. Consciously or not, people have been using
these scent signals with some potential sign values
to let others know they have an intention for social
events and to make people happier—supporting
large floral, gardening, and landscaping industries
along the way. The application of nonhuman scents
374
and odors for cosmetics, home products, and the air
in the space of home or work is a huge multinational
business, and its significance for nonverbal commu-
nication cannot be discounted.
We have intended to use the classification of
signs where the chemical compounds themselves are
a part of the message and to use the classification of
signals where the particular chemical compounds
are most likely sending a message that is based only
on association, but it is proving difficult to sepa-
rate these forms of communication. The important
potential discovery here is that there may be more
sign-like messages in some of our olfactory signals
than we know presently. There may be processes for
creating the strong associations between scents and
products that are not well understood. There are
studies on at least two well-known scents that sup-
port the hypothesis that scents used for signals have
sign-like compounds: lavender and peppermint.
Lavender
Lavender is commonly used to signal relaxation.
There is contradictory evidence that it may actually
influence such a state. Two studies have shown that
exposing older adults to certain fragrances, including
lavender, improved posture (Freeman et al., 2009)
and gait performance (Ebihara et al., 2012), and a
third found both decreased falls and decreased agita-
tion in the older adult population (Y. Sakamoto et al.,
2012). It may be that long-term observations such as
this have led to the use of lavender in many relaxation
products. However, benefits vary with context. For
example, whereas R. Sakamoto, Minoura, Usui,
Ishizuka, and Kanba (2005; see also Field et al., 2005)
found that exposure to lavender during work breaks
maintained performance, Moss, Cook, Wesnes, and
Duckett (2003) found a decrease in working memory
and slower reactions for lavender. Many questions
pertain to these contradictory results (which abound
in the area of aromatherapy). As noted in Chapter
21 of this handbook, there are many technical and
methodological issues. There is little standardization
across studies, and little attention is given to individ-
ual differences of age, gender, or ethnicity.
At least one well-designed and careful study of
lavender documented complete failure for the relax-
ing effects of lavender. This study reveals problems

generated by the specific claims of aromatherapy for
relaxation. Kiecolt-Glaser et al. (2008) exposed par-
ticipants to lavender while placing their hands in ice
water for a cold pressor pain test. Physiological and
self-report indices did not show any benefit for the
lavender in comparison to lemon oil or a no-aroma
condition. It is quite clear that the lavender did not
have a calming pharmacological effect, that is, it did
not counter actual physical pain. Nonetheless, there
may be some effects from lavender under other cir-
cumstances, but we may need a different approach
to understanding the process. Later in this chapter,
we present a hypothesis that scents may affect a
search function, that is, they may orient attention
and discrete motor behaviors in search of goals
­communicated by the scent. If no appropriate goal is
found, the search may cease with no clear effect on
physiological or psychological processes. If a search
for a pillow, a quiet room, and a dim light was pos-
sible, the lavender might prompt such a search and
facilitate relaxation. In painful conditions, such as
the cold pressor test, there may not be any relaxing
potential in the lab context, that is, one might search
but find only a dead end or worse.
Peppermint
Peppermint is also widely used in bathing and home
products; it is used to signal an attentive and refresh-
ing state. Sullivan et al. (1998) showed that simple
exposure to peppermint improved attention and
decreased error rates on several tasks for patients
with significant brain damage. Zoladz, Raudenbush,
and Lilley (2004) also showed improved scores on
a number of tasks, including verbal memory and
discrimination, when inhaling peppermint. Inhaling
cinnamon fragrance was slightly less effective but
still improved performance, whereas jasmine and
cherry did not. Raudenbush (2004) has reported
significant advantages to peppermint, in comparison
to other scents, on several types of athletic perfor-
mance. Peppermint is one of many scents that affect
the trigeminal system (dedicated to perception of
irritations, such as burning, itching, or cooling) as
well as the olfactory system (see Brand, 2006, for
review of such interactions). Peppermint is a type of
irritant that is arousing but also is perceived as pleas-
ant. It is likely that peppermint is used intentionally
Signs, Signals, and Symbols in Olfactics
as a common signal for arousal and refreshment
because of the sign effects it frequently has on physi-
ology, activity, and performance. Whether the sign
effects of common scents, such as lavender and pep-
permint, that are known to signal calm or arousal
are minor and whether they must be enhanced with
associations to achieve any impact is not clear and
requires further research. However, some scents,
no matter how they are associated with otherwise
rewarding events or goals, do not achieve ubiquitous
acceptance in consumer products. Herz and von Clef
(2001) used the example of parmesan cheese odor,
which can be confused with the odor of vomit, to
demonstrate that people can learn to like such odors.
However, it is worth noting that even sophisticated
cheese lovers do not wear it, and no one puts it on
the bed sheets, whereas select floral and herbal and
tree scents, including lavender, peppermint, garde-
nia, frankincense, and a limited number of others,
have been used in all these products.
General Aroma Signals
Even when no known sign compounds seem to
be present, scents are used as signals. According
to Claussen et al. (1994), in commerce there is an
“aroma of the commodity which often works at
a subconscious or barely conscious level, but has
nevertheless been shown by market research to be
crucial to the prospective consumer’s perception of
a product as desirable or worth buying” (p. 180).
Claussen et al. reviewed marketing reports show-
ing that scents make a product appear to be more
desirable. Putting a scent on one of several identi-
cal sweaters often leads a consumer to describe
the scented sweater as having better quality. Signal
scents can be applied to many products. A signal
scent does not emanate from the product but from
other sources and is then added to the product.
The signal scent brings attention to the product,
enhances its value, and, if congruent, increases the
likelihood that details including brand name will be
remembered long term (e.g., Krishna, Lwin, & Mor-
rin, 2010). It is the memory aspect that has captured
the attention of marketing researchers. Morrin and
colleagues (e.g., Morrin & Ratneshwar, 2000, 2003)
have shown that the longer the time delay from
exposure to memory testing, the more
375
Haviland-Jones, Wilson, and Freyberg
significant the effects of the scent signal in affect-
ing recall. This result is reminiscent of the Proust
effect. In Swann’s Way, Marcel Proust (1922/2013)
wrote of the experience of his protagonist smelling
the scent of tea and Madeleine biscuits and being
unaccountably swept away on an emotional jour-
ney all the way back to his childhood, an experi-
ence to which many can attest (Wysocki & Gilbert,
1989). Unlike the research on products where the
remembered product and its attributes are known,
in Proust’s case the memory might have been a con-
structed memory, because it took Proust quite some
time searching for something that would make sense
to him. Even so, the long-term memory effects for
odors are a remarkable signal effect for olfaction
and may well distinguish it from other sensory cues.
This is also an area in which there are many unan-
swered questions.
NONHUMAN OLFACTORY
COMPOUNDS—SYMBOLS
Some odors, scents, or fragrances may come to have
meaning in a social context that may be relatively
independent of the scent’s pleasantness or intensity,
just as words have meanings distinct from their
sounds. Those of us living in 21st century North
America seem to be, if not an exception, at least at
the low end of the distribution of educated mean-
ings attached to olfaction: According to Hall (1969),
“The extensive use of deodorants and the suppres-
sion of odors in public places results in olfactory
blandness that would be difficult to duplicate any-
where in the world” (p. 45). Hall wrote that this
results in a loss of richness and variety and even
obscures the deeper memories that come only from
odors. Hall and others have noted that cultural dif-
ferences in odor symbolism can result in cultural
clashes. If an Arab colleague perceives trust when
he can detect the breath of his North American
colleague but his colleague perceives intrusion or
aggression, the clash of olfactory symbolism will
make it difficult to broach other problems (see also
Chapter 4, this handbook).
Sociologists, such as Claussen et al. (1994), point
to the many examples in history of the symbolic
status of the fragrant or the foul. Individuals with
376
lower status due to age, gender, class, ethnicity,
religion, or occupation have often been described
as foul-smelling, whereas higher ranked individuals
are described as especially fragrant, and heavenly
beings would be the most fragrant of all. Claussen
et al.’s point reminds us that odors have traditionally
been associated with goodness, even godliness (for a
review, see Rozin, Haidt, & McCauley, 2008). There
still is a paucity of research on symbolic communi-
cation in olfaction, but fortunately, there are a few
recent studies that show how meaningful olfactory
symbols may be, affecting not only attitude but also
behavior.
Fishy and Foul
In a series of studies, Lee and Schwarz (2012)
examined how exposure to fishy odors heightened
suspicion and reduced cooperation. In one study,
participants engaged in trust games. When they were
exposed to fish oil during such games, they invested
less money in the games than when they were
exposed to another unpleasant odor or the control,
inferring heightened suspicion and less trust in their
partners. Such findings suggest that odors can have
a symbolic nature and can lead individuals to make
inferences about situations or people that extend
beyond hedonic characteristics.
There is some question about the association
between fishy odor and untrustworthy behav-
ior. Is the link a semantic one within the English
language with the connotation of fishy meaning
untrustworthy by definition? Or, is the seman-
tic link secondary to a primary informational
link—information about the state of a fish? As
anyone who buys and cooks fish will attest, seafood
is selected by odor. Fresh fish should smell like
the ocean; only fish that is not fresh smells fishy
(National Oceanic and Atmospheric Administra-
tion, n.d.). Therefore, it is not fish per se that is
untrustworthy but the smell of decaying fish. Fishy
smells are not just symbolically untrustworthy,
but they inform the purchaser. The fish may look
good, but its smell tells the true story; if it smells
fishy, it is not good. Therefore, in the case of Lee
and Schwarz (2012), fishy odor is not quite the
same type of symbol as a word because it is not
completely constructed but is representational in

some circumstances. Later, we suggest that the fishy
odor leads to a process of search—for decay in this
instance. If the only objects that can be the target of
the search in a social context are people, then the
odor of fish decay may be “owned” by them. Appar-
ently, people who are associated with decay should
not be accepted on the basis of their appearance,
no matter how innocent they look. If there actually
were fish or some other congruent target to focus
the search elsewhere, it remains an open question
whether the suspicion of people would arise.
Future research could take this area much further.
It would be pertinent for environmental as well as
personality domains.
Clean and Virtuous
Symbolic representation in odors extends beyond
negative inferences. Clean odors also impact both
cognition and behavior. Holland, Hendriks, and
Aarts (2005) demonstrated that when participants
were exposed to odors from a citrus-scented cleaner,
cognitive/semantic associations were activated.
Those exposed to the cleaner, even though they did
not identify it, were more likely to identify cleaning-
related words in a cognitive task and to use more
cleaning-related words in an unrelated dialogue.
Further, exposure to the same clean scent led partic-
ipants to engage in more cleaning behaviors, defined
as the number of times participants removed snack
crumbs from their work environment. Similarly,
Liljenquist, Zhong, and Galinsky (2010) found that
participants gave back more money in an anony-
mous trust game and indicated greater willingness
to volunteer and donate money when they were
exposed to the odor from a citrus-scented cleaner.
It is somewhat remarkable that clean odors could
ultimately impact such different types of behaviors
ranging from specific cleansing behaviors and iden-
tification of clean words to increases in behaviors
implying greater virtue.
Again, it is important to question the process by
which clean odors become known as clean odors.
One inference would be that the cleaner is manu-
factured first, becomes popular, perhaps because of
effective marketing, and then the odor becomes a
learned association. In that case, the semantics and
behavior of cleanliness come from using the cleaner;
Signs, Signals, and Symbols in Olfactics
the odor association is learned. Another inference,
just as plausible, is that the odor was selected for
the cleaning product because it comes from plants
that actually have antimicrobial and antibacterial
qualities. In this case, the odor–clean association
comes before the product development. There is
a large literature on the medicinal effects of plant
products (e.g., Craker & Simon, 1990). For our
purposes, Pattnaik, Subramanyam, and Kole (1996)
showed that orange oils and citronella were effective
in inhibiting more than 20 tested bacteria strains as
well as about a dozen strains of fungi. In the case of
the citrus cleaner, the manufacturers have adopted
cleaning odors from plants long associated correctly
with good health. As with the issue of fishy odors
being signs for decay, citrus odors are signs for good
health. When these odors are used in products or
places that have neither fish nor plant, is the resid-
ual odor a symbol or a morphed sign being used as
a signal? The separation of odors into categories is
more difficult than usually suspected.
Future studies should explore whether other
types of clean fragrances, such as pine or other
antibacterial plant oils, would lead to similar clean,
literal, and figurative behaviors or whether citral
compounds are special. Would frequent association
of any odor, perhaps musk, which has no “sign”
value as a cleanser, have as powerful an associa-
tion over time? Taking the associations to another
applied domain, could clean fragrances affect daily
behavior, increasing the likelihood of hand washing
in hospitals or restaurants, for example?
THE SEARCH ENGINE HYPOTHESIS:
IMPLICIT AND EXPLICIT PROCESSES
In each section of this chapter reviewing signs, sig-
nals, and symbols, we have seen that olfactics can
be explicit, but it is more often implicit. On the
explicit side, obviously we know that we smell odors
around us, and generally, when we know we smell
something, we also believe we know what it is or
that we can “follow our nose” to find out what it is.
On the implicit side, most of the questions await-
ing new paradigms concern compounds in the air
that may not be detected consciously but still play
a role in olfactics in nonverbal communication. For
377
Haviland-Jones, Wilson, and Freyberg
example, the mood semiochemical signs and the
unique ­personal identity semiochemicals can be
detected, but few people are aware of that detection,
and few are aware that these odors affect behavior.
When we first noted that people, if they must make
a choice, can discriminate mood semiochemicals,
and as others (e.g., Lundström et al., 2009) noted
that people, again, if they must, can identify kin and
self-identity signs, we tried to develop an hypothesis
that could guide our research in this renaissance of
olfactics. We had to account for the fact that people
in the research studies did not seem to know they
could detect these olfactory cues and that people
generally are poor at labeling odor cues. However,
recent research on the genetic dedication to olfaction
in humans as well as information on the extraordi-
nary discriminatory abilities of people indicate that
the olfactory system has the potential for tracking
an enormous amount of information. At that point,
there was little research on the impact of these che-
mosensory signs, but prior research on the social
impact of perfumes (e.g., Baron, 1997) had shown
that the psychological effects of scents were context
dependent. Putting together these pieces and others,
and relying heavily on some concepts from the larger
comparative literature, we proposed the search engine
hypothesis. Rather than being a system designed for
identification—an information approach more suited
to the other senses, particularly vision—we pro-
posed that the olfactory system is designed more as
a subconscious parallel rapid processor that prompts
searches to be carried out often by the other senses
(Haviland-Jones & Wilson, 2008).
If two people are greeting each other, and there is
a disgust olfactory cue left in the air from a departed
person, we would predict that the semiochemical
prompts a rapid subconscious search for related con-
textual cues, or internally, from a memory network.
The mere presence of such a cue may affect arousal
as the olfactory system is attentive to the cue. Nev-
ertheless, the outcome of the search prompted by
the olfactory cue is not determined but depends on
other available information gathered in the search.
For example, if the persons greeting each other are
relaxed and pleased to be together, nothing will sup-
port the waning disgust cues. The pair will never
consciously notice. If there is some other cue in the
378
context that seems to merit distrust, such as a loud
argument in the corridor, there may be an increased
likelihood that the couple mutually direct distrust
toward it. If one of the couple brings with him some
minor irritation, the olfactory cues in the new con-
text could make a conflict more likely. The partner
greeting him, prompted by the irrelevant disgust
olfactory cue, already is unknowingly searching for
other cues of separation or discord. The result could
be a disgust face in his greeting, and the encounter
could become conflictual. One of the pair might be
characteristically contemptuous or dismissive (see
Magai & Haviland-Jones, 2002), and any prompt to
search for contempt would quickly initiate a well-
learned dismissive set of behaviors and attributions.
Research on olfactory process suggests that the
search engine just illustrated with a disgust semio-
chemical might operate at two levels. First, olfactory
compounds generally may affect arousal states even
when focused attention is on the visual or audi-
tory aspect of the event, object, or person. Thus, we
propose that the odor or scent makes the perceiver
more alert and more likely to scan the environment
or memory for significant changes and cues. It is
therefore part of a multisensory “booster.” These
state changes may emerge along several routes. The
simplest may be similar to any multimodal effect
in that the addition of a sensory mode, in this case
olfaction, but actually it might be any sense, boosts
attention. In addition to a generalized attention
booster, olfactory compounds may be more spe-
cific about the kind of information to search for, as
in the case of mood semiochemicals versus sexual
ones versus social odors or fishy or clean or floral
odors, as our review indicated. Here again, there
may be differences related to whether the odor is
consciously sensed. Although the explicit smell of
flowers might lend itself to a search for a garden,
it might also, implicitly, lend itself to a search for
positive socioemotional encounters, just as fishy or
clean odors could lend to search for a fish or a fruit
but, in certain contexts, may also affect generalized
socioemotional searches (see also the embodiment
discussion later in this section; Bargh, Schwader,
Hailey, Dyer, & Boothby, 2012).
The search engine hypothesis directs us to
consider methods and technologies that examine

implicit or subconscious processes in future research.
Most of the past research on olfaction, including our
own, has centered on the identification of odors in
one form or another and the thresholds for their cor-
rect identification. If we are correct in our thinking
about the search engine, this identification would be
a minor part of olfactics. If the olfactory compound
is often, or even usually, subconsciously detected,
then we would not explicitly acknowledge the olfac-
tory cue when explaining to ourselves or to others
how we make decisions that would be related to the
olfactory cue. When we rapidly search for a target
or event or even memory that is congruent with
the olfactory cue, we are very likely to believe that
the origin of the search was in the seen or heard or
touched target or event that was identified later in
the search, skipping the subconscious olfactory cue
because we were never aware of it.
Since we first tentatively proposed the search
engine hypothesis to describe a potentially highly
developed olfactory communication process (e.g.,
Haviland-Jones & Wilson, 2008), there have been
great strides in olfactics research. The commu-
nicative potential for mood signs is established.
Even when not identified, one person’s mood sign
affects another’s facial movements and attributions
and alters the semantic web so that words related
to the mood odor become more frequent in use
and more quickly identified (see Chapter 21, this
handbook). A similar process has been found for a
few other odors—such as fishy, clean, and floral—
and probably also the identity of relatives versus
strangers, men versus women, and infants versus
adults. The extent of the “vocabulary”—at least
metaphorically—is unknown. Much work remains
to be done before we can be confident about the
extent of the possible discrete cues and the con-
straints on their effects. In a variety of sensory
fields, there are parallel theories. Here, we need to
include the research and theory on ­automaticity
generated by Bargh and colleagues (e.g., Bargh,
Gollwitzer, Lee-Chai, Barndollar, & Trötschel,
2001; Bargh et al., 2012), though to ­thoroughly
expand the connections between their research,
which is largely on other sensory systems, to olfac-
tion goes beyond our present review. For a limited
example, they find that there is an embodiment of
Signs, Signals, and Symbols in Olfactics
sensory processes in which there are metaphoric
associations for sensory inputs. Warm air is associ-
ated with warm feelings, and heavy weights with
seriousness. In olfaction, we find other examples
to bolster their concepts, such as that floral scent
is associated with happiness, or fishy odors with
untrustworthiness. In the case of odors, however,
more attention is needed to the adaptive origins of
the connections (e.g., see coevolution of flowers and
people; Haviland-Jones et al., 2005).
There has long been a suspicion that olfactory
processes have some unusual attributes, such as
the previously mentioned Proust effect on memory.
Recent research has expanded the proposition
that olfactory processes present a new and unique
view for nonverbal communication. Sela and Sobel
(2010) presented elegant testing of their hypothesis
that there is a sensory amnesia for odors that natu-
rally interferes with object identification. This is a
potential explanation for the low level of attribu-
tions to odor itself. They also argue that the olfac-
tory system operates largely at the unconscious
level, that it is a rapid information source for certain
types of information but not for objects. Therefore,
an explanation of the function of olfactics must
address how it functions without excellent object
identification, and the search engine hypothesis
is a potential direction. Recently, there was a sig-
nificant discovery about the olfactory system that
may lead to the understanding of whether olfaction
operates differently from other senses. McGann and
colleagues (see, e.g., Kass, Rosenthal, Pottackal, &
McGann, 2013) found that the synaptic output of
the olfactory sensory neurons, the first cells that
physically contact odor molecules in the nose
and project the information to the olfactory bulb,
changes when an odor is paired with shock. Previ-
ous concepts of odor detection and associations pre-
dicted that the associations occurred in the cortical
limbic brain regions, not at the peripheral edge of
the system. This finding may be related to the obser-
vation that people exposed to chemical compounds
traumatically may report serious reactions to very
low levels of the traumatic odor, levels thought to
be below threshold. The change in their ability to
detect the offending compound is often unconscious
and unlikely to respond to ordinary therapeutic
379
Haviland-Jones, Wilson, and Freyberg

intervention. If the peripheral synaptic output In addition to these general guidelines, here are
has changed, it possibly would account for such some of the quandaries that have emerged within
phenomena. This development is very recent, but the review:
it has profound possibilities for functions special-
1. What is the “vocabulary” of olfactory messages?
ized in the olfactory system. Future research in
That is, what types of messages—including, but
olfactics will profit from extending the research on
not limited to, moods, sexual motivation, and
automaticity (see Bargh et al., 2012) into olfac-
group membership—are encoded in the olfactory
tion, particularly in examining developmental
system? What kinds of messages are decoded?
changes, cross-modality, and semantic areas. The
For example, the attributions related to ingroup
broader theory for automaticity also presents chal-
and outgroup membership and the “male war-
lenges to the emerging field of olfactics and to our
rior” phenomenon (Van Vugt, De Cremer, &
search engine. In particular, given so many parallel
Janssen, 2007) may have some roots in olfactory
findings in other sensory systems for nonconscious
identity signs.
sensory perception and nonverbal communication,
2. Could the frequently observed lack of connection
one might posit that olfaction is just an additional
between knowing the odor of relatives, for exam-
sensory system that seems mysterious only because
ple, and personal confidence in knowing that
there is less empirical research as of yet. Neverthe-
one is sensing a diagnostic odor be affected with
less, there is some evidence that olfaction does
training? Direct tests of training on expectations
have different functions from other systems—as
may solve some of the present uncertainty.
the review often indicates and as the newest
3. Could we use information from olfactics to
research suggests.
­manage well-being in homes, hospitals, or work
places? That is, are there supportive olfactory
FUTURE DIRECTIONS cues that we are eliminating by “scrubbing”
all information from the air in controlled
One of the general conclusions, one that has been
environments?
noted by many in the field, is that paradigms
4. What can be learned of genetic factors in olfac-
and tools to explore the subconscious processes
tics? For example, do people differ in their reac-
inherent in olfactics need to be expanded and
tions to traumatic exposure to chemicals mainly
developed (see also Chapter 21, this handbook).
due to prior experience or mainly due to genetic
There are many possibilities that can be adapted
sensitivity?
from other types of nonverbal communication
5. What can be learned of the messages in simple
research. In particular, the multisensory processes
versus complex organic compounds? For exam-
are likely to be paramount—the effect of olfac-
ple, what is the chemical signature of fear versus
tory cues on movement (facial and postural) on
happy?
directed attention in vision, audition, and touch.
6. Does olfaction affect long-term memory?
The overlap between olfaction and taste is also
7. How does context affect olfactics? For example,
key. How olfactory cues affect social attributions,
undetected floral odors influence positive social
word use in narrative, metaphor, and other subtle
behavior in safe contexts. In a threatening con-
cognitive processes related to nonverbal com-
text, will floral odors still influence positive
munication is another large area with just small
social behavior?
pockets of intriguing research. On the other end
of the spectrum, in studying internal processing,
SUMMARY
the emerging uses of functional magnetic reso-
nance imaging and other neurological technolo- Though there are decades of research on olfactory
gies also are increasingly useful in demonstrating nonverbal communication, 21st century research is
the parallel processing of many chemosignals in a revolutionizing this field, called olfactics. Many of
network. the discoveries have occurred so late in our history

380

because they concern olfactory communication
that is subconscious and often denied. Listing just
a few of the unexpected conclusions, we find the
following:
1. Most people can smell fear odors from other peo-
ple even though they are mostly skeptical of this.
2. Most people can identify the olfactory thumb-
print unique to themselves at nearly 100% accu-
racy without having confidence in being correct,
and they are also very good in identifying the
olfactory thumbprints of their near relatives and,
often, of their close friends.
3. Not only do human odors affect mood and moti-
vation but select plant and animal odors also
have effects on mood, performance, interpersonal
behavior, and physiological reactions and can be
used as signals and symbols in communication.
This includes floral odors that affect moods and
social behavior.
4. Odors that are used in cleansers may lead a per-
son to many virtuous behaviors from cleaning
up crumbs to being more trusting, even when
unaware of any cleanser or odor. In contrast,
fishy odors may cause an opposite set of behav-
iors. These are just a selection of new and often
surprising findings, each of which points to
new research directions in both theoretical and
applied fields.
We organized the research to show how olfac-
tory cues are signs, signals, and symbols. An exam-
ple of a communication sign is the mood odor. We
have learned thus far that we communicate fear,
disgust, and (possibly) happy with body chemicals.
An example of a signal is a product brand odor that
affects long-term memory for details of the product.
Finally, for an example of a symbolic cue, there is a
fishy odor. It is a symbol for nontrusting attitudes
and behaviors. A critical factor in olfactics is that
chemical signals, whether from the individual or
from the environment, can encode valuable infor-
mation that is generally decoded without explicit
knowledge or attention. We proposed that one
of the most important processes in olfactics is to
provoke a search for types of information, rather
than to identify specific objects. We called this pro-
cess the search engine. In spite of the many recent
Signs, Signals, and Symbols in Olfactics
advances, olfactics remains one of the least explored
and most exciting areas in human nonverbal
communication.
References
Aglioti, S. M., & Pazzaglia, M. (2011). Sounds and scents
in (social) action. Trends in Cognitive Sciences, 15,
47–55. http://dx.doi.org/10.1016/j.tics.2010.12.003
Ashby, F. G., Isen, A. M., & Turken, A. U. (1999). A
neuropsychological theory of positive affect and its
influence on cognition. Psychological Review, 106,
529–550. http://dx.doi.org/10.1037/0033-295X.
106.3.529
Bargh, J. A., Gollwitzer, P. M., Lee-Chai, A., Barndollar, K.,
& Trötschel, R. (2001). The automated will: Non­
conscious activation and pursuit of behavioral goals.
Journal of Personality and Social Psychology, 81,
1014–1027. http://dx.doi.org/10.1037/0022-3514.
81.6.1014
Bargh, J. A., Schwader, K. L., Hailey, S. E., Dyer, R. L., &
Boothby, E. J. (2012). Automaticity in social-
cognitive processes. Trends in Cognitive Sciences, 16,
593–605. http://dx.doi.org/10.1016/j.tics.2012.
10.002
Baron, R. A. (1997). The sweet smell of . . . helping:
Effects of pleasant ambient fragrance on prosocial
behavior in shopping malls. Personality and Social
Psychology Bulletin, 23, 498–503. http://dx.doi.org/
10.1177/0146167297235005
Bechara, A., Damasio, H., & Damasio, A. R. (2000).
Emotion, decision making and the orbitofrontal
cortex. Cerebral Cortex, 10, 295–307. http://dx.doi.org/
10.1093/cercor/10.3.295
Bensafi, M., Brown, W. M., Tsutsui, T., Mainland, J. D.,
Johnson, B. N., Bremner, E. A., . . . Sobel, N. (2003).
Sex-steroid derived compounds induce sex-specific
effects on autonomic nervous system function in
humans. Behavioral Neuroscience, 117, 1125–1134.
http://dx.doi.org/10.1037/0735-7044.117.6.1125
Berliner, D. A. (1994). Fragrance compositions containing
human pheromones. Retrieved from http://www.lens.
org/lens/patent/US_5278141_A
Brand, G. (2006). Olfactory/trigeminal interactions
in nasal chemoreception. Neuroscience and
Biobehavioral Reviews, 30, 908–917. http://dx.doi.org/
10.1016/j.neubiorev.2006.01.002
Brechbühl, J., Klaey, M., & Broillet, M.-C. (2008).
Grueneberg ganglion cells mediate alarm pheromone
detection in mice. Science, 321, 1092–1095. http://
dx.doi.org/10.1126/science.1160770
Buck, L., & Axel, R. (1991). A novel multigene family
may encode odorant receptors: A molecular basis for
odor recognition. Cell, 65, 175–187. http://dx.doi.
org/10.1016/0092-8674(91)90418-X
381
Haviland-Jones, Wilson, and Freyberg
Bushdid, C., Magnasco, M. O., Vosshall, L. B., & Keller, A.
(2014). Humans can discriminate more than 1
trillion olfactory stimuli. Science, 343, 1370–1372.
http://dx.doi.org/10.1126/science.1249168
Chen, D., & Haviland-Jones, J. (2000). Human olfactory
communication of emotion. Perceptual and Motor
Skills, 91, 771–781. http://dx.doi.org/10.2466/pms.
2000.91.3.771
Chen, D., Katdare, A., & Lucas, N. (2006). Chemosignals
of fear enhance cognitive performance in humans.
Chemical Senses, 31, 415–423. http://dx.doi.org/10.1093/
chemse/bjj046
Claussen, C., Howes, D., & Synnott, A. (1994). Aroma: A
cultural history of smell. London, England: Routledge.
Coffield, C. N., Mayhew, E. M., Haviland-Jones, J. M., &
Walker-Andrews, A. S. (2014). Adding odor: Less
distress and enhanced attention for 6-month-olds.
Infant Behavior and Development, 37, 155–161. http://
dx.doi.org/10.1016/j.infbeh.2013.12.010
Craker, L. E., & Simon, J. E. (Eds.). (1990). Herbs, spices,
and medicinal plants: Recent advances in botany,
horticulture, and pharmacology (Vol. 4). Boca Raton,
FL: CRC Press.
de Almeida, R. N., Motta, S. C., de Brito Faturi, C.,
Catallani, B., & Leite, J. R. (2004). Anxiolytic-like
effects of rose oil inhalation on the elevated plus-
maze test in rats. Pharmacology Biochemistry and
Behavior, 77, 361–364. http://dx.doi.org/10.1016/
j.pbb.2003.11.004
de Groot, J. H. B., Smeets, M. A. M., Kaldewaij, A.,
Duijndam, M. J. A., & Semin, G. R. (2012).
Chemosignals communicate human emotions.
Psychological Science, 23, 1417–1424. http://dx.doi.
org/10.1177/0956797612445317
Delaunay-El Allam, M., Soussignan, R., Patris, B., Marlier,
L., & Schaal, B. (2010). Long-lasting memory for an
odor acquired at the mother’s breast. Developmental
Science, 13, 849–863. http://dx.doi.org/10.1111/
j.1467-7687.2009.00941.x
Ebihara, S., Nikkuni, E., Ebihara, T., Sakamoto, Y.,
Freeman, S., & Kohzuki, M. (2012). Effects of
olfactory stimulation on gait performance in frail
older adults. Geriatrics and Gerontology International,
12, 567–568. http://dx.doi.org/10.1111/j.1447-0594.
2011.00801.x
Ekman, P., & Friesen, W. V. (1975). Unmasking the face.
Englewood Cliffs, NJ: Prentice Hall.
Field, T., Diego, M., Hernandez-Reif, M., Cisneros, W.,
Feijo, L., Vera, Y., . . . Claire He, Q. (2005). Lavender
fragrance cleansing gel effects on relaxation.
International Journal of Neuroscience, 115, 207–222.
http://dx.doi.org/10.1080/00207450590519175
Freeman, S., Ebihara, S., Ebihara, T., Niu, K., Kohzuki,
M., Arai, H., & Butler, J. P. (2009). Olfactory stimuli
382
and enhanced postural stability in older adults. Gait
and Posture, 29, 658–660. http://dx.doi.org/10.1016/
j.gaitpost.2009.02.005
Freyberg, R. (2014). Fragrance identification in close
adolescent female friendships. Unpublished manuscript.
Freyberg, R., & Ahren, M.-P. (2011). A preliminary trial
exploring perfume preferences in adolescent girls.
Journal of Sensory Studies, 26, 237–243. http://dx.doi.
org/10.1111/j.1745-459X.2011.00339.x
Freyberg, R., & Bart, M. (2012, April). Olfactory
environment influences close relationships through
multiple methods of measurement. Poster presented
at the annual meeting of the Association for
Chemoreception Sciences, Huntington Beach, CA.
Goubet, N., Strasbaugh, K., & Chesney, J. (2007).
Familiarity breeds content? Soothing effect of a
familiar odor on full-term newborns. Journal of
Developmental and Behavioral Pediatrics, 28, 189–194.
http://dx.doi.org/10.1097/dbp.0b013e31802d0b8d
Hall, E. T. (1969). The hidden dimension. Garden City,
NY: Anchor Books.
Hämmerli, A., Schweisgut, C., & Kaegi, M. (2012).
Population genetic segmentation of MHC-correlated
perfume preferences. International Journal of Cosmetic
Science, 34, 161–168. http://dx.doi.org/10.1111/
j.1468-2494.2011.00696.x
Haviland-Jones, J., Hudson, J. A., Wilson, P., Freyberg, R.,
& McGuire, T. (2013). The emotional air in your
space: Scrubbed, wild or cultivated? Emotion, Space
and Society, 6, 91–99. http://dx.doi.org/10.1016/
j.emospa.2011.10.002
Haviland-Jones, J., & McGuire, D. (1999). The scents of
fear and funny. Aroma-Chology Review, 8, 11.
Haviland-Jones, J. M., Rosario, H. H., Wilson, P., &
McGuire, T. R. (2005). An environmental approach
to positive emotion: Flowers. Evolutionary Psychology,
3, 104–132.
Haviland-Jones, J. M., & Wilson, P. (2008). A “nose”
for emotion: Emotional information and challenges
in odors and semiochemicals. In M. Lewis, J. M.
Haviland-Jones, & L. Feldman-Barrett (Eds.),
Handbook of emotions (3rd ed., pp. 235–248).
New York, NY: Guilford Press.
Haviland-Jones, J. M., Wilson, P., & McGuire, T. (2014).
In pursuit of phenotypes for identifying fear and happy
mood chemosignals. Unpublished manuscript.
Havlicek, J., & Lenochova, P. (2006). The effect of meat
consumption on body odor attractiveness. Chemical
Senses, 31, 747–752. http://dx.doi.org/10.1093/
chemse/bjl017
Herz, R. (2012). That’s disgusting: Unraveling the mysteries
of repulsion. New York, NY: Norton.
Herz, R. S., & von Clef, J. (2001). The influence of verbal
labeling on the perception of odors: Evidence for

olfactory illusions? Perception, 30, 381–391. http://
dx.doi.org/10.1068/p3179
Holland, R. W., Hendriks, M., & Aarts, H. (2005). Smells
like clean spirit: Nonconscious effects of scent on
cognition and behavior. Psychological Science, 16,
689–693. http://dx.doi.org/10.1111/j.1467-9280.
2005.01597.x
Hölldobler, B., & Wilson, E. O. (1990). The ants. http://
dx.doi.org/10.1007/978-3-662-10306-7
Howes, D. (1987). Olfaction and transition: An essay on
the ritual uses of smell. Canadian Review of Sociology/
Revue Canadienne de Sociologie, 24, 398–416.
Jacob, S., Kinnunen, L. H., Metz, J., Cooper, M., &
McClintock, M. K. (2001). Sustained human
chemosignal unconsciously alters brain function.
NeuroReport, 12, 2391–2394. http://dx.doi.org/10.1097/
00001756-200108080-00021
Jellinek, P. (1996). The psychological basis of perfumery.
In J. S. Jellinek (Ed.), The psychological basis of
perfumery (4th ed., pp. 1–162). http://dx.doi.org/
10.1007/978-94-009-1567-1
Karlson, P., & Luscher, M. (1959). “Pheromones”: A
new term for a class of biologically active substances.
Nature, 183, 55–56. http://dx.doi.org/10.1038/
183055a0
Kass, M. D., Rosenthal, M. C., Pottackal, J., & McGann, J. P.
(2013). Fear learning enhances neural responses
to threat-predictive sensory stimuli. Science, 342,
1389–1392. http://dx.doi.org/10.1126/science.
1244916
Keville, K., & Green, M. (2009). Aromatherapy: A
complete guide to the healing art. Berkeley, CA:
Crossing Press.
Kiecolt-Glaser, J. K., Graham, J. E., Malarkey, W. B.,
Porter, K., Lemeshow, S., & Glaser, R. (2008).
Olfactory influences on mood and autonomic,
endocrine, and immune function. Psychoneuro­
endocrinology, 33, 328–339. http://dx.doi.org/
10.1016/j.psyneuen.2007.11.015
Kirk-Smith, M. D., & Booth, D. A. (1980). Effect of
androstenone on choice of location in others’
presence. In H. van der Starre (Ed.), Olfaction and taste
(Vol. 7, pp. 397–400). London, England: IRL Press.
Krishna, A., Lwin, M. O., & Morrin, M. (2010). Product
scent and memory. Journal of Consumer Research, 37,
57–67. http://dx.doi.org/10.1086/649909
Lee, S. W., & Schwarz, N. (2012). Bidirectionality,
mediation, and moderation of metaphorical effects:
The embodiment of social suspicion and fishy smells.
Journal of Personality and Social Psychology, 103,
737–749. http://dx.doi.org/10.1037/a0029708
Liljenquist, K., Zhong, C. B., & Galinsky, A. D. (2010).
The smell of virtue: Clean scents promote reciprocity
Signs, Signals, and Symbols in Olfactics
and charity. Psychological Science, 21, 381–383.
http://dx.doi.org/10.1177/0956797610361426
Lundström, J. N., Boyle, J. A., Zatorre, R. J., & Jones-
Gotman, M. (2008). Functional neuronal processing
of body odors differs from that of similar common
odors. Cerebral Cortex, 18, 1466–1474. http://dx.doi.
org/10.1093/cercor/bhm178
Lundström, J. N., Boyle, J. A., Zatorre, R. J., & Jones-
Gotman, M. (2009). The neuronal substrates of
human olfactory based kin recognition. Human Brain
Mapping, 30, 2571–2580. http://dx.doi.org/10.1002/
hbm.20686
Macfarlane, A. (1975). Olfaction in the development
of social preferences in the human neonate. Ciba
Foundation Symposium, 33, 103–117.
Magai, C., & Haviland-Jones, J. (2002). The genius of
emotion. http://dx.doi.org/10.1017/CBO9780511509575
Mallet, P., & Schaal, B. (1998). Rating and recognition
of peers’ personal odors by 9-year-old children:
An exploratory study. Journal of General Psychology, 125,
47–64. http://dx.doi.org/10.1080/00221309809595576
Matteo, S. (1987). The effect of job stress and job
interdependency on menstrual cycle length,
regularity and synchrony. Psychoneuroendocrinology,
12, 467–476. http://dx.doi.org/10.1016/0306-
4530(87)90081-3
McClintock, M. K. (1971). Menstrual synchrony and
suppression. Nature, 229, 244–245. http://dx.doi.org/
10.1038/229244a0
McClintock, M. K. (2000). Human pheromones: Primers,
releasers, signalers, or modulators? In K. Wallen &
J. E. Schneider (Eds.), Reproduction in context:
Social and environmental influences on reproductive
physiology and behavior (pp. 355–420). Cambridge,
MA: MIT Press.
Mennella, J. A., Jagnow, C. P., & Beauchamp, G. K.
(2001). Prenatal and postnatal flavor learning by
human infants. Pediatrics, 107, e88. http://dx.doi.org/
10.1542/peds.107.6.e88
Mennella, J. A., Johnson, A., & Beauchamp, G. K. (1995).
Garlic ingestion by pregnant women alters the odor
of amniotic fluid. Chemical Senses, 20, 207–209.
http://dx.doi.org/10.1093/chemse/20.2.207
Morrin, M., & Ratneshwar, S. (2000). The impact of
ambient scent on evaluation, attention, and memory
for familiar and unfamiliar brands. Journal of Business
Research, 49, 157–165. http://dx.doi.org/10.1016/
S0148-2963(99)00006-5
Morrin, M., & Ratneshwar, S. (2003). Does it make sense
to use scents to enhance brand memory? Journal of
Marketing Research, 40, 10–25. http://dx.doi.org/
10.1509/jmkr.40.1.10.19128
Moss, M., Cook, J., Wesnes, K., & Duckett, P. (2003).
Aromas of rosemary and lavender essential oils
383
Haviland-Jones, Wilson, and Freyberg
differentially affect cognition and mood in healthy
adults. International Journal of Neuroscience, 113,
15–38. http://dx.doi.org/10.1080/00207450390161903
Mujica-Parodi, L. R., & Strey, H. (2006). Identification
and isolation of human alarm pheromones. http://
dx.doi.org/10.1037/e512332010-001
Mujica-Parodi, L. R., Strey, H. H., Frederick, B., Savoy,
R., Cox, D., Botanov, Y., . . . Weber, J. (2009).
Chemosensory cues to conspecific emotional stress
activate amygdala in humans. PLoS ONE, 4, e6415.
http://dx.doi.org/10.1371/journal.pone.0006415
National Oceanic and Atmospheric Administration.
(n.d.). Seafood safety: Simple solutions to handling
seafood safely. Retrieved from http://www.seafood.
nmfs.noaa.gov/pdfs/consumerbrochure.pdf
Nishitani, S., Miyamura, T., Tagawa, M., Sumi, M.,
Takase, R., Doi, H., . . . Shinohara, K. (2009).
The calming effect of a maternal breast milk odor
on the human newborn infant. Neuroscience
Research, 63, 66–71. http://dx.doi.org/10.1016/
j.neures.2008.10.007
Ober, C., Weitkamp, L. R., Cox, N., Dytch, H., Kostyu,
D., & Elias, S. (1997). HLA and mate choice in
humans. American Journal of Human Genetics, 61,
497–504. http://dx.doi.org/10.1086/515511
Pattnaik, S., Subramanyam, V. R., & Kole, C. (1996).
Antibacterial and antifungal activity of ten essential
oils in vitro. Microbios, 86, 237–246.
Pause, B. M., Lübke, K., Laudien, J. H., & Ferstl, R.
(2010). Intensified neuronal investment in the
processing of chemosensory anxiety signals in non-
socially anxious and socially anxious individuals.
PLoS ONE, 5, e10342. http://dx.doi.org/10.1371/
journal.pone.0010342
Pause, B. M., Ohrt, A., Prehn, A., & Ferstl, R. (2004).
Positive emotional priming of facial affect perception
in females is diminished by chemosensory anxiety
signals. Chemical Senses, 29, 797–805. http://dx.doi.
org/10.1093/chemse/bjh245
Penn, D., & Potts, W. (1998). How do major histocompatibility
complex genes influence odor and mating preferences?
Advances in Immunology, 69, 411–436. http://dx.doi.
org/10.1016/S0065-2776(08)60612-4
Pollen, M. (2002). The botany of desire: A plant’s eye view
of the world. New York, NY: Random House.
Porter, R. H., Balogh, R. D., Cernoch, J. M., & Franchi,
C. (1986). Recognition of kin through characteristic
body odors. Chemical Senses, 11, 389–395. http://
dx.doi.org/10.1093/chemse/11.3.389
Prehn-Kristensen, A., Wiesner, C., Bergmann, T. O.,
Wolff, S., Jansen, O., Mehdorn, H. M., . . . Pause, B. M.
(2009). Induction of empathy by the smell of anxiety.
PLoS ONE, 4, e5987. http://dx.doi.org/10.1371/
journal.pone.0005987
384
Preti, G., Wysocki, C. J., Barnhart, K. T., Sondheimer,
S. J., & Leyden, J. J. (2003). Male axillary extracts
contain pheromones that affect pulsatile secretion of
luteinizing hormone and mood in women recipients.
Biology of Reproduction, 68, 2107–2113. http://dx.doi.
org/10.1095/biolreprod.102.008268
Proust, M. (2013). Swann’s way. Retrieved from http://
www.gutenberg.org/files/7178/7178-h/7178-h.htm
(Original work published 1922)
Raudenbush, B. (2004). The effects of peppermint on
enhancing mental performance and cognitive functioning,
pain threshold and tolerance, digestion and digestive
processes, and athletic performance. Unpublished
manuscript, Department of Psychology, Wheeling
Jesuit University, Wheeling, WV.
Reznikoff-Etievant, M. F., Bonneau, J. C., Alcalay, D.,
Cavelier, B., Touré, C., Lobet, R., & Netter, A.
(1991). HLA antigen-sharing in couples with
repeated spontaneous abortions and the birthweight
of babies in successful pregnancies. American Journal
of Reproductive Immunology, 25, 25–27. http://dx.doi.
org/10.1111/j.1600-0897.1991.tb01059.x
Rozin, P., & Fallon, A. E. (1987). A perspective on
disgust. Psychological Review, 94, 23–41. http://
dx.doi.org/10.1037/0033-295X.94.1.23
Rozin, P., Haidt, J., & McCauley, C. R. (2008). Disgust.
In M. Lewis, J. M. Haviland-Jones, & L. Feldman-
Barrett (Eds.), Handbook of emotions (3rd ed.,
pp. 757–776). New York, NY: Guilford Press.
Sakamoto, R., Minoura, K., Usui, A., Ishizuka, Y., &
Kanba, S. (2005). Effectiveness of aroma on work
efficiency: Lavender aroma during recesses prevents
deterioration of work performance. Chemical Senses,
30, 683–691. http://dx.doi.org/10.1093/chemse/bji061
Sakamoto, Y., Ebihara, S., Ebihara, T., Tomita, N.,
Toba, K., Freeman, S., . . . Kohzuki, M. (2012).
Fall prevention using olfactory stimulation with
lavender odor in elderly nursing home residents: A
randomized controlled trial. Journal of the American
Geriatrics Society, 60, 1005–1011. http://dx.doi.org/
10.1111/j.1532-5415.2012.03977.x
Savic, I., Berglund, H., Gulyas, B., & Roland, P. (2001).
Smelling of odorous sex hormone-like compounds
causes sex-differentiated hypothalamic activations in
humans. Neuron, 31, 661–668. http://dx.doi.org/
10.1016/S0896-6273(01)00390-7
Schaal, B. (1988). Olfaction in infants and children:
Developmental and functional perspectives. Chemical
Senses, 13, 145–190. http://dx.doi.org/10.1093/chemse/
13.2.145
Schaal, B., Marlier, L., & Soussignan, R. (2000). Human
foetuses learn odours from their pregnant mother’s
diet. Chemical Senses, 25, 729–737. http://dx.doi.
org/10.1093/chemse/25.6.729

Schank, J. C. (2001). Menstrual-cycle synchrony:
Problems and new directions for research. Journal
of Comparative Psychology, 115, 3–15. http://dx.doi.
org/10.1037/0735-7036.115.1.3
Schiffman, S. S., & Graham, B. G. (2000). Taste and
smell perception affect appetite and immunity in
the elderly. European Journal of Clinical Nutrition,
54(Suppl. 3), S54–S63. http://dx.doi.org/10.1038/
sj.ejcn.1601026
Schleidt, M., & Genzel, C. (1990). The significance of
mother’s perfume for infants in the first weeks of
their life. Ethology and Sociobiology, 11, 145–154.
http://dx.doi.org/10.1016/0162-3095(90)90007-S
Sela, L., & Sobel, N. (2010). Human olfaction: A constant
state of change-blindness. Experimental Brain
Research, 205, 13–29. http://dx.doi.org/10.1007/
s00221-010-2348-6
Sorokowska, A. (2013). Assessing personality using
body odor: Differences between children and adults.
Journal of Nonverbal Behavior, 37, 153–163. http://
dx.doi.org/10.1007/s10919-013-0152-2
Stockhorst, U., & Pietrowsky, R. (2004). Olfactory
perception, communication, and the nose-to-brain
pathway. Physiology and Behavior, 83, 3–11. http://
dx.doi.org/10.1016/j.physbeh.2004.07.018
Stoddart, D. M. (1990). The scented ape: The biology
and culture of human odour. Cambridge, England:
Cambridge University Press.
Sullivan, T. E., Warm, J. S., Schefft, B. K., Dember, W. N.,
O’Dell, M. W., & Peterson, S. J. (1998). Effects of
olfactory stimulation on the vigilance performance of
individuals with brain injury. Journal of Clinical and
Experimental Neuropsychology, 20, 227–236. http://
dx.doi.org/10.1076/jcen.20.2.227.1175
Tubaldi, F., Turella, L., Pierno, A. C., Grodd, W.,
Tirindelli, R., & Castiello, U. (2011). Smelling odors,
understanding actions. Social Neuroscience, 6, 31–47.
http://dx.doi.org/10.1080/17470911003691089
Van Vugt, M., De Cremer, D., & Janssen, D. P. (2007).
Gender differences in cooperation and competition:
The male-warrior hypothesis. Psychological Science,
18, 19–23. http://dx.doi.org/10.1111/j.1467-9280.
2007.01842.x
Varendi, H., Christensson, K., Porter, R. H., & Winberg, J.
(1998). Soothing effect of amniotic fluid smell in
newborn infants. Early Human Development, 51, 47–55.
http://dx.doi.org/10.1016/S0378-3782(97)00082-0
Wedekind, C., Seebeck, T., Bettens, F., & Paepke, A. J.
(1995). MHC-dependent mate preferences in
humans. Proceedings of the Royal Society of London
Signs, Signals, and Symbols in Olfactics
B: Biological Sciences, 260, 245–249. http://dx.doi.
org/10.1098/rspb.1995.0087
Weisfeld, G. E., Czilli, T., Phillips, K. A., Gall, J. A., &
Lichtman, C. M. (2003). Possible olfaction-based
mechanisms in human kin recognition and inbreeding
avoidance. Journal of Experimental Child Psychology,
85, 279–295. http://dx.doi.org/10.1016/S0022-
0965(03)00061-4
Weller, A., & Weller, L. (1993). Menstrual synchrony
between mothers and daughters and between
roommates. Physiology and Behavior, 53, 943–949.
http://dx.doi.org/10.1016/0031-9384(93)90273-I
Weller, A., & Weller, L. (1997). Menstrual synchrony
under optimal conditions: Bedouin families. Journal
of Comparative Psychology, 111, 143–151. http://
dx.doi.org/10.1037/0735-7036.111.2.143
Weller, L., Weller, A., Koresh-Kamin, H., & Ben-Shoshan, R.
(1999). Menstrual synchrony in a sample of working
women. Psychoneuroendocrinology, 24, 449–459.
http://dx.doi.org/10.1016/S0306-4530(98)00092-4
Wysocki, C. J., & Gilbert, A. N. (1989). National Geographic
Smell Survey: Effects of age are heterogenous. Annals
of the New York Academy of Sciences, 561, 12–28. http://
dx.doi.org/10.1111/j.1749-6632.1989.tb20966.x
Zavazava, N., Westphal, E., & Müller-Ruchholtz, W.
(1990). Characterization of soluble HLA molecules
in sweat and quantitative HLA differences in serum
of healthy individuals. International Journal of
Immunogenetics, 17, 387–394. http://dx.doi.org/
10.1111/j.1744-313X.1990.tb00890.x
Zernecke, R., Haegler, K., Kleemann, A. M., Albrecht, J.,
Frank, T., Linn, J., . . . Wiesmann, M. (2011). Effects
of male anxiety chemosignals on the evaluation of
happy facial expressions. Journal of Psychophysiology,
25, 116–123. http://dx.doi.org/10.1027/0269-8803/
a000047
Zhou, W., & Chen, D. (2009). Fear-related chemosignals
modulate recognition of fear in ambiguous facial
expressions. Psychological Science, 20, 177–183.
http://dx.doi.org/10.1111/j.1467-9280.2009.02263.x
Ziomkiewicz, A. (2006). Menstrual synchrony: Fact or
artifact? Human Nature, 17, 419–432. http://dx.doi.
org/10.1007/s12110-006-1004-0
Zoladz, P., Raudenbush, B., & Lilley, S. (2004, April).
Impact of the chemical senses on augmenting memory,
attention, reaction time, problem solving, and response
variability: The differential role of retronasal versus
orthonasal odorant administration. Paper presented
at the annual conference of the Association for
Chemical Reception Sciences, Sarasota, FL.
385
 Chapter 15
The Body: Postures, Gait,
Proxemics, and Haptics
David Matsumoto, Hyisung C. Hwang, and Mark G. Frank
An understanding of the complexity and compre-
hensiveness of the world of nonverbal behavior is
not complete without a discussion of the many mes-
sages and signals produced by postures, body move-
ments, space, and touch. When many people refer
to “body language,” in fact, they are referring to the
messages and signals produced by the whole body,
which we cover in this chapter. Although research
on posture, whole body movements, space, and
touch is not as mature as that on face, voice, gesture,
gaze, or odor (see Chapters 10–14, this handbook),
it is also equally clear that the whole body com-
municates a wealth of information about emotional
states, cognitions, intent, personality, and atti-
tudes. Below, we highlight the most important and
exciting research findings with regard to postures,
body movements, space, and touch, beginning with
postures.
POSTURES
Postures are defined as a position of the body or of
body parts. Early research indicated that postures
communicate attitudinal states and general affect
as opposed to the very specific emotions communi-
cated by face and voice. These attitudinal and gen-
eral affective states include liking versus disliking,
orientation (closed or open), and attention (direct
or indirect). These various dimensions can be sum-
marized as communicating general positivity as well
as status relationships (Mehrabian, 1968a, 1968b,
1969). For example body orientation—the degree
to which a person’s shoulders and legs are turned
http://dx.doi.org/10.1037/14669-015
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
in the direction of rather than away from another
interactant—is associated with attitude toward the
interactant; the least direct orientation occurs for
disliked interactants, whereas more direct orienta-
tions occur for liked others. Open arms and legs in a
seated position generally communicate a more posi-
tive attitude and openness, whereas arms akimbo
(arms on hips) or arms crossed in front of one’s
body generally are associated with more negative
attitudes. Open body postures also communicate
more power (Cashdan, 1998; J. A. Hall, Coats, &
LeBeau, 2005).
Recent studies have examined whether body
postures can be signals of more discrete emotional
states, not just general positivity–negativity. For
example, Coulson (2004) had observers view
176 computer-generated mannequin figures and
asked them to judge the presence or absence of six
emotions—anger, disgust, fear, happiness, sadness,
and surprise. Observers could not distinguish dis-
gust, fear, and surprise beyond chance levels. Anger,
sadness, and happiness, however, were reliably
attributed to a large number of postures, with some
obtaining greater than 90% agreement rates. Anger
was conveyed by a backward head tilt, absence of
a backward chest bend, no abdominal twist, arm
raised forward and upward, and weight transfers
either forward or backward. Happiness was con-
veyed by backward head tilt, no forward movement
of the chest, and arms raised above shoulder level
and straight at the elbow. Sadness was conveyed by
a forward head tilt, forward chest bend, no twisting,
and arms at the sides. Atkinson, Dittrich, Gemmell,
387
Matsumoto, Hwang, and Frank
and Young (2004) used point-light displays of both
dynamic and static behaviors of individuals posing
“whole body” emotions. Emotion judgments of the
still image at the peak of the expression were dis-
criminated at rates greater than chance for the five
emotions tested (anger, fear, happiness, disgust, and
sadness), although dynamic point-light displays of
postures showed higher rates of discrimination than
the static displays. One concern about studies in this
area, however, is that actors are asked to pose emo-
tions; thus, it is possible that the produced behav-
iors are mimes of an emotion but not really what
happens when emotions are elicited spontaneously
and naturally.
Several studies have examined the relation-
ship between nonverbal behaviors in general, and
postures in particular, with rapport, and a few key
nonverbal behaviors have been shown to positively
influence the development of rapport. Mirroring
is known as the chameleon effect, which refers to a
tendency to adopt the postures, gestures, and man-
nerisms of interaction partners (Chartrand & Bargh,
1999), and in general it has been supported by the
available evidence as producing positive regard for
the other. However, some studies also show null
or even negative effects on rapport as a function of
mirroring (La France & Ickes, 1981). These data
suggest that mirroring may positively influence
rapport if it is done naturally and subconsciously
(or at least appears to be so); if an interactant part-
ner feels that one is deliberately trying to mirror
behaviors, the partners are likely to feel manipu-
lated, diminishing the chances for rapport building.
Other nonverbal behaviors found to be associated
with rapport include smiling (Rotenberg et al.,
2003); direct body orientation, uncrossed legs, sym-
metrical arms, and moderate eye contact (Harrigan,
Oxman, & Rosenthal, 1985); and congruent limbs
and forward leaning postures (Trout & Rosenfeld,
1980). Rapport can be reliably judged by observing
nonverbal cues only (Grahe & Bernieri, 1999) and
across cultures (Bernieri & Gillis, 1995). Sensitivity
to such nonverbal behaviors by physicians has been
related to interpersonal success and rapport with
patients in clinical settings (DiMatteo, Friedman, &
Taranta, 1979; DiMatteo, Taranta, Friedman, &
Prince, 1980). Tickle-Degnen and Rosenthal (1990)
388
have suggested that the development of rapport
is a dynamic process involving three interrelated
components—mutual attentiveness, positivity, and
posture coordination—and that the relative weight-
ing of these components changes across time. In
early interactions, positivity and attentiveness may
be more important than posture coordination. Later,
however, coordination and attentiveness may be
more important than positivity.
Research has also shown that various postures
are associated with romantic interest. Muehlen-
hard, Koralewski, Andrews, and Burdick (1986)
had male and female actors portray a heterosexual
couple engaged in a conversation while the female
actor was instructed to pose high, medium, and low
amounts of various behaviors shown by previous
surveys to be associated with romantic intentions.
Male and female judges then rated the likelihood
that the woman would accept a date if asked by the
man. Fifteen behaviors reflected these judgments,
including leaning toward the man, shoulders paral-
lel to the man, being closer to the man, and a brief
touch, particularly on the knee.
One very interesting notion that has received
very little empirical attention is the concept of
posture–gesture mergers, which are defined as a
movement that occurs when a gesture (typically
by the hands) flows directly into a postural shift
(via movement of the whole body; Lamb & Watson,
1979). Winter, Widell, Truitt, and George-Falvy
(1989) conducted four studies in which they dem-
onstrated that posture–gesture mergers that accom-
panied verbal expressions were associated with
truthfulness (vs. deception), frustration, relaxation,
and judgments of sincerity and effectiveness. Unfor-
tunately, we know of no studies since these early
ones that have followed this concept.
There is surprisingly little systematic research
on the production or interpretation of the mean-
ing of postures across cultures. The existing
studies suggest that people of different cultures
interpret postures according to the same dimen-
sions (i.e., positivity, openness, status) but place
different weights of importance on specific aspects
of these dimensions (Kudoh & Matsumoto, 1985;
­Matsumoto & Kudoh, 1987). Certainly this is an
area that is ripe for new research.
 The Body

BODY MOVEMENTS AND GAIT an aggressive component, with facial grimaces or

aggression, fists, punching motions, and shouts or
One type of nonverbal behavior related to postures
utterances. This expression has been documented
is body movement (see also the section on crowds
in individuals from many different cultures around
and situations in Chapter 8, this handbook). The
the world, as well as congenitally blind individuals,
study of the contribution of different features of
immediately after a victory in an agonistic context
body movements as signals of emotion has occurred
(Hwang & Matsumoto, 2014b). There are cultural
for decades (e.g., de Meijer, 1989). In one study
differences in the degree to which individuals pro-
(Montepare, Koff, Zaitchik, & Albert, 1999), observ-
duce this expression after victory; individuals from
ers saw 3-s video clips of actors portraying angry,
status-oriented cultures tend to produce more of
happy, sad, or neutral emotional situations. The
these whole body expressions than do individuals
observers were able to judge which emotion was
from more egalitarian cultures (Hwang & Matsu-
being portrayed beyond chance levels. Subsequent
moto, 2014a). Presumably, status-oriented cultures
ratings of the body movements indicated that angry
will facilitate the expression of nonverbal behaviors
videos were rated as jerkier, stiffer, and harder than
related to dominance more than will egalitarian
the other emotions; anger body movements were
cultures, apropos of the norms regarding social
also more expanded, fast, and had more action.
interaction and social coordination in these cultures.
More recently, Van Dyck, Maes, Hargreaves, Lesaffre,
Furthermore, people of different cultures reliably
and Leman (2013) demonstrated that happiness
label these whole body expressions as triumph
and sadness could be expressed through free dance
(Matsumoto & Hwang, 2012). The expression of
movement; participants in the happy condition
triumph is related to dominance threat displays seen
moved faster, with more acceleration, and made
across the animal kingdom, and it is likely impor-
more expanded and more apparently impulsive
tant in the creation and maintenance of status hier-
movements than in the sad condition. Dael, Mor-
archies in communities.
tillaro, and Scherer (2012) asked 10 actors to pose
Whole body behaviors opposite to those asso-
12 emotions and found evidence for patterns of
ciated with triumph (e.g., head tilted forward or
postural movements associated with three of the 12
down, tongue out, eyes closed, and one or both arms
emotions (anger, amusement/pleasure, and despair).
pulled in toward body) have been studied. Some
Several recent studies have demonstrated that the
researchers (e.g., Tracy & Matsumoto, 2008) have
emotion of pride is expressed nonverbally through
labeled these behaviors as “shame.” However, these
whole body movements (Tracy & Robins, 2004).
same behaviors have also been associated with sub-
The pride expression includes an expanded posture
mission (see also Andrews, Brewin, Rose, & Kirk,
with head tilted back, arms akimbo, and a low-
2000; Dickerson, Gruenewald, & Kemeny, 2004;
intensity smile on the face. This expression is dis-
J. A. Hall et al., 2005; Keltner & Harker, 1998).
tinguished from other positive emotions, and it has
There is yet to be definitive research delineating the
been reliably recognized in adults across cultures
difference between expressive behaviors related to
and in children as young as 4 years of age (Tracy &
shame and submission.
Robins, 2008; Tracy, Robins, & Lagattuta, 2005).
As with postures (reviewed earlier), one area of
Individuals across cultures produce this expres-
research related to body movements concerns rap-
sion, as do individuals who are congenitally blind
port and the quality of interpersonal relations. In
(Tracy & Matsumoto, 2008).
support of the hypothesized relationship between
Other recent studies have provided evidence for
rapport and interpersonal coordination (Tickle-
the whole body display of the emotion of triumph.
Degnen & Rosenthal, 1987), Bernieri (1988)
The triumph expression includes an expanded pos-
examined the degree of movement synchrony in
ture with chest out and head back, arms away from
teacher–student interactions and demonstrated a
the body and raised above the shoulders, and torso
strong relationship between participants’ rapport
pushed out. The triumph expression also includes
and the degree of movement synchrony between the

389
Matsumoto, Hwang, and Frank
pairs. Movement synchrony has also been related
to the quality of mother–infant interactions (Berni-
eri, Reznick, & Rosenthal, 1988) and opposite-sex
encounters related to courtship rituals (Grammer,
Kruck, & Magnusson, 1998).
One specific type of whole body movement is
gait, which is the pattern of movement of the body
when walking. Early studies showed that observ-
ers can accurately judge the sex of a person by his
or her gait alone, even when the targets wear bulky
sweatshirts with only reflectors on their shoulders,
elbows, hips, knees, and other joints (Kozlowski &
Cutting, 1977). Using a point-light technique in
which there are no other visibly observable cues
to use, individuals can reliably recognize people
through their gait (Richardson & Johnston, 2005).
There is also some evidence that age and sex can
be reliably identified across cultures (Montepare &
Zebrowitz, 1993). Furthermore, when actors posed
emotions of pride, happiness, anger, and sadness,
they walked with longer strides and more exagger-
ated foot pounding (heavy footedness) when posing
anger than when posing the other emotions (Monte-
pare, Goldstein, & Clausen, 1987). Gait patterns are
different when people are asked to simulate different
emotional states or while listening to different types
of music (Janssen et al., 2008).
Classic research has suggested that gait is con-
sistent with other expressive behaviors within an
individual and thus partially accountable for one’s
personality (Allport & Vernon, 1933). Recent
research has uncovered fascinating, new findings.
For example, gait can be used as cues to determine
sexual advances or inappropriate touching (Saka-
guchi & Hasegawa, 2006); slow walkers with slow
length strides and personality traits implying vul-
nerability were identified as targets for inappropri-
ate touching. Gait differences manifest themselves
in various disorders. Depressed patients show
decreased stride length, decreased coordination of
arms in movement, and slower velocity (Lemke,
Wendorff, Mieth, Buhl, & Linnemann, 2000).
Parkinson’s patients also show reduced stride, but
they can be trained to expand the stride by employ-
ing various visual clues, again suggesting some
volitional control over the movement (Lewis,
Byblow, & Walt, 2000).
390
Convicted criminals agree on which people to
victimize based on gait; specifically, those chosen
as would-be victims had strides too long or too
short, did not swing their feet, and moved in only two
dimensions compared to three (Grayson &
Stein, 1981). More recently, undergraduate
students—some of whom had been previously
victimized, whereas others had not—were surrep-
titiously videotaped, and then those videos were
shown to convicted criminals (Book, Costello, &
Camilleri, 2013). Criminals who scored higher on
psychopathy were more accurate in identifying
those who were previously victimized, and they
were also more likely to mention gait (particularly
the confidence in the gait) as a clue to identifying a
past victim. Furthermore, one study examined judg-
ments of the emotional state of individuals carrying
a concealed handgun versus a concealed 1-L bottle
and found that observer’s judgments correlated
with arm swing and gait (Blechko, Darker, & Gale,
2009). The suggestion that the additional emotional
load (e.g., anxiety) involved with deceptive behav-
ior may produce gait abnormalities has a number of
interesting practical ­applications, such as security
screening.
To our knowledge, there have been no direct
studies that have examined cultural similarities or
differences in gait. However, there has been some
interesting research in the speed with which individ-
uals across cultures typically move through their cit-
ies (Kirkcaldy, Furnham, & Levine, 2001; Levine &
Bartlett, 1984; Levine, Lynch, Miyake, & Lucia,
1989; Levine & Norenzayan, 1999). These stud-
ies have demonstrated that pace is associated with
punctuality, coronary heart disease, and a variety of
attitudinal and personality traits.
INTERPERSONAL SPACE (PROXEMICS)
The use of space in interpersonal interactions is
another important nonverbal behavior and is called
proxemics. E. T. Hall’s (1966, 1973) classic work
in this area specified four different levels or zones
of interpersonal space use depending on social
relationship type: intimate, personal, social, and
public (see Figure 15.1). These zones are like con-
centric invisible bubbles that surround the person.

Public
space
Social
space
Personal
space
Intimate
space
FIGURE 15.1.  Distinctions of interpersonal space.
Data from E. T. Hall (1966, 1973)
He suggested that interpersonal distance helps to
regulate intimacy by controlling sensory expo-
sures because the possibility of sensory stimulation
(smells, sights, touch) is enhanced at closer dis-
tances. In one of his earlier works, E. T. Hall (1963)
delineated a notational system for interpersonal
proxemic behaviors that included categories such as
physical interaction proximity; amounts of touch-
ing, detectable body heat, detectable body odor, and
visual contact; directness of body orientation; and
loudness of speaking. In the United States, E. T. Hall
suggested that intimate distances are less than 18
in. (45.72 cm), personal distances range from 18 in.
(45.72 cm) to 4 ft (1.22 m), social distances range
from 4 ft (1.22 m) to 12 ft (3.66 m), and public dis-
tances are greater than 12 ft (3.66 m).
People of all cultures appear to use space accord-
ing to the major distinctions proposed by E. T. Hall
(1966, 1973), but they differ in the size of the spaces
they attribute to them (see also Chapter 4, this
handbook). A study of individuals from five differ-
ent cultures (American, Swedish, Greek, Southern
Italian, and Scottish) showed that the cultures were
similar in the order of the distances for different
types of transactions but that there were significant
mean differences in the actual distances used (Little,
1968). In general, results from various studies sug-
gest that cultures around the Mediterranean, Middle
East, or of Latin origin interact at closer distances.
The Body
Arab males, for example, tend to sit closer to each
other than American males, with more direct, con-
frontational types of body orientations (Watson &
Graves, 1966). They also use greater eye contact
and speak in louder voices. Arabs, at least in the
past, learned to interact with others at distances
close enough to feel the other person’s breath
(E. T. Hall, 1963, 1966). Latin Americans tend
to interact more closely than do individuals of
European backgrounds (Forston & Larson, 1968),
and Indonesians tend to sit closer than Austra-
lians (Noesjirwan, 1977, 1978). Italians interact
more closely than either Germans or Americans,
and Colombians interact at closer distances than
do Costa Ricans (Shuter, 1976). When interacting
with someone from their same culture, Japanese
sat the farthest away, Venezuelans the closest, with
Americans in the middle (Sussman & Rosenfeld,
1982); interestingly, in the same study, foreigners
who spoke in English adopted the American conver-
sational distance compared to when speaking with
others from their home country in their native lan-
guage. Cultural differences in the use of space even
occur when individuals use dolls to interact with
each other (Little, 1968).
Interpersonal spacing is not fixed, however;
rather it differs according to context. Worchel
(1986) demonstrated that individuals who had to
interact with others chose relatively greater dis-
tances after having experienced social isolation,
when they believed their interaction would be
observed by others and when they expected a long
conversation to focus on a personal topic; room size
also affected distance but only in rectangular rooms.
The single major factor determining distances
appears to be the relationship of the interactants;
more intimate relationships produce closer interper-
sonal distances (Gifford & O’Connor, 1986). The
specific content or affective tone of the interaction
is the next most important (Little, 1968). Interest-
ingly, topic intimacy may not be an important factor
determining spacing (Baker & Shaw, 1980).
Because the use of interpersonal space is based,
at least in part, on the regulation of sensory pro-
cesses, it makes good sense that cultures regulate
the use of space, as such regulation is necessary
for social coordination. Intermediate distances are
391
Matsumoto, Hwang, and Frank
generally rated as more comfortable, preferable,
and appropriate than more extreme close or far
distances (Thompson, Aiello, & Epstein, 1979).
­Altman (1975) suggested that personal space is used
as a mechanism to regulate interpersonal interaction
and to achieve a desired level of privacy. As such
violations of culturally appropriate boundaries of
space by the intrusion of others bring about aversive
reactions (Sussman & Rosenfeld, 1978), including
feelings of comfort, personal control, and privacy
(Greenberg, Strube, & Myers, 1980). Insults also
result in larger personal space preferences (O’Neal,
Brunault, Carifio, Troutwine, & Epstein, 1980),
and repeated intrusions of personal space (e.g.,
in overcrowded conditions) produce detrimental
health outcomes (Paulus, McCain, & Cox, 1978).
The intimacy equilibrium hypothesis (Argyle &
Dean, 1965) suggests that people take steps to
reinstate their space when space boundaries are
intruded. In an elevator, for example, individu-
als often stare at the floor numbers, thus visually
removing the space offender. When individuals
feel their space has been invaded, they respond
with an array of nonverbal behaviors, such as the
eye gaze aversion described earlier; however, they
also change their body orientation away from the
other, or even deploy buffers (such as books or
briefcases) to demarcate territory (Hayduk, 1983;
see also Chapter 8, this handbook). Similarly, a bar
will have seats all facing forward, thus directing eye
gaze forward, enabling a stranger to sit at the edge
of our intimate zone without causing major offense.
The use of individual technologies, such as cellular
phones, iPods, and noise reducing headsets, likely
serve to create psychological space as well. Conse-
quently, when individuals want to take control of a
situation or avoid being controlled by others, they
use larger spaces (Strube & Werner, 1982).
When people ask for permission to invade space,
males and females react differently. In one study,
male and female students who were seated outdoors
and alone on a university campus were approached
by a confederate who sat either 9 in. (22.86 cm) or
18 in. (45.72 cm) away. In one condition, the con-
federate asked for permission before sitting down;
in another condition, the confederate said nothing.
Males left more quickly when the confederate sat
392
closer and did not ask for permission; females left more
quickly when the confederate sat closer but asked for
permission (Sundstrom & Sundstrom, 1977).
The compensatory mechanisms for establishing
intimacy equilibrium may differ drastically across
cultures and contexts. For instance, anecdotally
it appears that cultures that value relatively more
space among dyadic interactants, such as those in
East Asia, do not have strong rules about maintain-
ing such space among crowds of strangers. Anyone
with experience in a crowded rush hour train in
downtown Tokyo or packed marketplaces in China
can attest to this. However, Americans in the United
States tend to value some degree of personal space
even when among crowds of strangers, and they are
offended or insulted if people “invade” that space.
Another thing to recognize is that much of the infor-
mation researchers have about the use of interper-
sonal space involves dyadic interaction. Researchers
have much less information about the use of space
among groups of people, especially strangers (see
Chapter 8, this handbook, for a related discussion).
One often-overlooked aspect of space usage is
the influence of age. A meta-analysis of studies dem-
onstrated that personal space gradually and steadily
increases between 3 and 21 years of age (Hayduk,
1983). This has been found to be true in both natu-
ralistic and experimental settings. Adults as well as
children are influenced by age differences; when a
5-year-old intrudes on the rear space of an adult, the
adult generally turns toward and speaks to the child.
When 10-year-olds do so, the adults generally move,
lean away, or fidget. However, there are certainly
likely to be culture- and gender-specific differences
in the influence of age on space (see, e.g., Pagán &
Aiello, 1982).
TOUCH (HAPTICS)
Another important nonverbal behavior is touch
(also known as haptics). Touch is related to the
use of interpersonal space because touch requires
close physical contact. Some writers consider inter-
personal touch as one of humankind’s earliest and
most basic forms of communication (Knapp, 1972;
Morris, 1971), probably owing to its importance
in early mother–infant caretaking. Early research

indicated that touch conveys positive emotion,
closeness, comfort, compassion, intimacy, imme-
diacy, interpersonal closeness, liking, acceptance,
warmth, and love (Richmond, McCroskey, & Payne,
1991; Thayer, 1986). Withholding touch can signal
an assortment of negative feelings, such as resent-
ment, hostility, anger, or mistrust (Richmond et al.,
1991). Touch often has different meanings to men
and women. Men are more likely to interpret touch
as communicating affection and sexual attraction,
whereas women are more likely to separate those
meanings (Hollender, Luborsky, & Scaramella,
1969; Nguyen, Heslin, & Nguyen, 1976).
Recent research has gone beyond examining the
relationship between touch and general positive
emotion by providing initial evidence that touch can
also communicate distinct emotions—such as anger,
fear, disgust, love, gratitude, and sympathy—across
cultures (Hertenstein, Holmes, McCullough, &
Keltner, 2009; Hertenstein, Keltner, App, Bulleit, &
Jaskolka, 2006). Among family members the
amount of touching—embraces, handholding, arms
around shoulders—is positively associated with the
smiling intensity of family members, and these rela-
tionships predict positive behaviors outside of the
home as well (Oveis, Gruber, Keltner, Stamper, &
Boyce, 2009). Among professional basketball play-
ers, increased touch is associated with increased
cooperation and improved performance (Kraus,
Huang, & Keltner, 2010). Touch by female wait
staff also improved tips offered at restaurants (Ebesu
Hubbard, Tsuji, Williams, & Seatriz, 2003).
Previous research on touch focused on examin-
ing gender differences in nonreciprocal touching,
where one individual initiates and another receives
touch in an interpersonal setting. Much of this
research received its impetus from the classic works
of Henley (1973, 1977), who proposed that higher
status individuals have a touching privilege, that is,
they are allowed to touch others more often than
lower status individuals to maintain and express
their status differences. Furthermore, because men
were perceived to have higher status than women
in the past, status was therefore expressed more by
men touching women than vice versa.
Early studies and reviews provided contradictory
evidence for the gender asymmetry postulated by
The Body
Henley (1973, 1977). In surveys, men and women
both agreed that being touched by close friends of
the opposite sex is pleasant and that touch from
a same-sex person is unpleasant; however, being
touched by an opposite sex stranger is unpleasant
for women but pleasant for men (Heslin, Nguyen,
& Nguyen, 1983). Major (1981) concluded that
men are more likely to initiate opposite-sex touch
in interpersonal interactions, thereby supporting
Henley’s (1973, 1977) hypothesis. In a meta-analysis
of 25 studies, however, Stier and Hall (1984) failed
to find support for the hypothesis. Subsequently,
Major, Schmidlin, and Williams (1990) conducted
an observational study of 799 intentional touches
occurring in public settings; controlling for inten-
tion, age, and setting, they reported that other-
gender touch was more likely to be initiated by
men, thus supporting Henley’s (1977) hypothesis.
Derlega, Lewis, Harrison, Winstead, and Costanza
(1989) also reported support for the gender asym-
metry hypothesis.
One possible explanation for the contradictory
findings in this area is that context and culture may
moderate the effects of gender asymmetry. J. A. Hall
(1996), for example, examined touch, status, and
gender at professional meetings of psychologists
and philosophers and did not find overall effects for
gender asymmetry in touching. However, there was
gender asymmetry when the male and female inter-
actants were of equal status. The type of relationship
between the interactants also matters, as gender
asymmetry has been reported in specific types of
relationships (e.g., romantic relationships) but
not others (McDaniel & Andersen, 1998; Willis &
Briggs, 1992).
Although the studies described earlier dealt
with opposite-sex touch, there have been a few
studies that have examined same-sex touch.
Women tend to engage in more same-sex touch
than men (Henley, 1973; Major et al., 1990; Stier &
Hall, 1984) and report greater comfort with same-
sex touch than do men (P. A. Andersen & Lei-
bowitz, 1978). One explanation offered for this
gender difference is that homophobic attitudes
underlie the differences—that is, that men more
than women are likely to be seen as homosexual if
they exhibit same-sex touch (Derlega et al., 1989).
393
Matsumoto, Hwang, and Frank
Studies involving questionnaire surveys and actual
observational data have provided evidence in sup-
port of this hypothesis (Roese, Olson, Borenstein,
Martin, & Shores, 1992).
Because touch has strong interpersonal mean-
ings, humans learn what is and is not appropriate
with regard to touching others and being touched.
There are interesting cultural and gender differences
in reactions to touch. When there is no justification
for the touch, men seem to have more aversive reac-
tions to being touched than women, and they like
an intruding toucher less (Sussman & Rosenfeld,
1978). Touch avoidance refers to attitudes toward
being touched and is a communication disposition.
Numerous studies have demonstrated that opposite-
sex touch avoidance is higher for women than men
(J. F. Andersen, Andersen, & Lustig, 1987; P. A.
Andersen & Leibowitz, 1978). Opposite-sex touch
avoidance is also positively related with communica-
tion apprehension and is negatively correlated with
verbal predispositions to communicate, open com-
munication style, and self-esteem (J. F. Andersen,
et al., 1987).
Several interesting studies have demonstrated a
link between touch and compliance. In one of the
first studies to do so (Kleinke, 1977), money was
left on the shelf of a phone booth in an airport. An
experimenter approached the person who next used
the phone, asking if he found the money. There
were large differences in compliance as a func-
tion of whether the experimenter touched or did
not touch the person; 96% of the people who were
touched returned the money, whereas only 63% of
the people who were not touched did. Touch has
also been shown to increase compliance to signing
petitions and completing research surveys (Pat-
terson, Powell, & Lenihan, 1986; Willis & Hamm,
1980) as well as complying with a physician’s
request to take antibiotics (Guéguen et al., 2011).
There are likely cultural differences, however, in
the relationship between touching and compliance.
One study conducted in Poland found that touching
by a man of another man when making a request
negatively affected the chances for compliance
(Dolinski, 2010); the noncompliance was related to
homophobic attitudes among the participants. Fur-
thermore, if the costs to the request are high—for
394
example, donating blood—touch also does not
seem to induce additional compliance (Guéguen &
Vion, 2009).
Just as cultures regulate space in their societies,
cultures also regulate touch in social interactions,
and one way cultures do so is through greeting ritu-
als. Some cultures have institutionalized touch as a
part of their greeting rituals (e.g., handshake, hug,
kiss on the cheek); others do not (e.g., bowing).
As mentioned previously, Watson (1970) classified
30 countries as either a contact culture (those that
facilitated physical touch or contact during interac-
tion) or a noncontact culture. Recent research has
generally confirmed this prediction (McDaniel &
Andersen, 1998) but has also called for sharpening
these generalizations by taking into account context
and relationship (Remland, Jones, & Brinkman,
1991). Regardless, violations of the cultural rules
regarding touch are likely to be interpreted in the
same way as those of space, producing aversive
consequences.
The research discussed in this section until
now has involved one person touching another.
Another form of nonverbal behavior involves a
person touching him- or herself. These move-
ments have been studied under various labels in
the literature and are generally called “adaptors” or
“manipulators” (Ekman & Friesen, 1969; Rosen-
feld, 1966; see also Chapter 12, this handbook).
These behaviors allow the individual to adapt
to internal or external stimuli (e.g., scratching
an itch), because they allow for the manipula-
tion of a part of the body (rubbing one’s arms to
soothe oneself). Some studies have suggested that
self-touching is indicative of negative affect, anxi-
ety, withdrawal, hostility, or depression (Mahl,
1968; Mehrabian, 1972). Self-touching behavior
is also related to information processing (Bar-
roso, Freedman, Grand, & Van Meel, 1978); lan-
guage construction (Freedman & Bucci, 1981);
Type A behaviors such as being ambitious, rigidly
organized, highly status conscious, or impatient
(Friedman, Hall, & Harris, 1985); and attentional
focusing and distractibility (Barroso & Feld, 1986;
Barroso et al., 1978). Taken together, these studies
indicate that self-touching serves as an important
signal of psychological functioning.

CONCLUSION
In this chapter, we have seen how incredibly sophis-
ticated our nonverbal behavior system is by exam-
ining posture, body movements, space, and touch.
As we mentioned at the outset, research on each of
these nonverbal channels is not as mature as that
on face, voice, gesture, gaze, or odor. However, it
is equally clear that each of these nonverbal chan-
nels communicates a wealth of information about
emotional states, cognitions, intent, personality,
and attitudes for diverse social functions and mean-
ings in social life. Despite the importance of these
nonverbal channels, as evidenced by the studies
reviewed earlier, many areas still remain unknown
and unexplored, especially across cultures. Future
continued research on each of these channels is
warranted to continue to expand knowledge about
the signal values of behaviors produced in these
channels. Being knowledgeable about all the vari-
ous channels of nonverbal behaviors is beneficial in
social interaction, as people are more likely to get
more comprehensive and accurate social messages
when communicating with others. For example,
think about many incidents that are related to the
nonverbal behaviors discussed in this chapter that
happened to upset or discomfort us when traveling
in foreign countries or when interacting with people
with different cultural backgrounds at work or in
our communities. Individuals can improve their
social skill strategies by understanding why these
incidents happen and by becoming increasingly aware
of how to more constructively react to them. Future
continued research in this area can continue to bring
important potential benefits in our social lives.
An especially promising road for future studies
is the examination of the signal value of the combi-
nation of the various nonverbal channels because
people’s normal social interactions occur in dynamic
ways, sometimes in face-to-face interaction but
sometimes at a distance. For example, one interest-
ing line of recent research has examined judgments
of facial expressions of emotion linked in different
combinations with whole body responses (Aviezer
et al., 2008; Meeren, van Heijnsbergen, & de Gelder,
2005). Studies combining faces of one emotion with
bodily responses of another have reported a body
The Body
(i.e., context) superiority in judgments (Aviezer
et al., 2008; Meeren et al., 2005). Of course, testing
multiple nonverbal channels at once in and of itself
does not guarantee that the findings are valuable
and valid because there are serious ecological valid-
ity questions that arise when mixing and matching
nonverbal behaviors, as it is not clear that mixed
messages in face and body actually occur in real life
(e.g., a disgusted face and fear body posture with
head and body back and arms up in a defensive
move). Thus, the more channels added, the trickier
it is to obtain valid findings. Future studies therefore
need to conceptualize more carefully the meaning
of the stimuli produced when combining behaviors
from different nonverbal channels. Nevertheless,
when such combinations are examined, especially
within a larger environmental and/or social con-
text, they are especially promising areas of future
research, as will be investigations of the subtle coor-
dination of body movements between individuals
during interactions and the dynamic flow of behav-
iors within an individual (as in gait).
References
Allport, G. W., & Vernon, P. E. (1933). Studies in
expressive movement. http://dx.doi.org/10.1037/
11566-000
Altman, I. (1975). The environment and social behavior:
Privacy, personal space, territory, crowding. Pacific
Grove, CA: Brooks Cole.
Andersen, J. F., Andersen, P. A., & Lustig, M. W. (1987).
Opposite sex touch avoidance: A national replication
and extension. Journal of Nonverbal Behavior, 11,
89–109. http://dx.doi.org/10.1007/BF00990960
Andersen, P. A., & Leibowitz, K. (1978). The development
and nature of the construct touch avoidance.
Environmental Psychology and Nonverbal Behavior, 3,
89–106. http://dx.doi.org/10.1007/BF01135607
Andrews, B., Brewin, C. R., Rose, S., & Kirk, M. (2000).
Predicting PTSD symptoms in victims of violent
crime: The role of shame, anger, and childhood
abuse. Journal of Abnormal Psychology, 109, 69–73.
http://dx.doi.org/10.1037/0021-843X.109.1.69
Argyle, M., & Dean, J. (1965). Eye contact, distance and
affiliation. Sociometry, 28, 289–304. http://dx.doi.org/
10.2307/2786027
Atkinson, A. P., Dittrich, W. H., Gemmell, A. J., &
Young, A. W. (2004). Emotion perception from
dynamic and static body expressions in point-light
395
Matsumoto, Hwang, and Frank
and full-light displays. Perception, 33, 717–746.
http://dx.doi.org/10.1068/p5096
Aviezer, H., Hassin, R. R., Ryan, J., Grady, C., Susskind, J.,
Anderson, A., . . . Bentin, S. (2008). Angry, disgusted,
or afraid? Studies on the malleability of emotion
perception. Psychological Science, 19, 724–732.
http://dx.doi.org/10.1111/j.1467-9280.2008.02148.x
Baker, E., & Shaw, M. E. (1980). Reactions to interperson
distance and topic intimacy: A comparison of
strangers and friends. Journal of Nonverbal Behavior,
5, 80–91. http://dx.doi.org/10.1007/BF00986511
Barroso, F., & Feld, J. (1986). Self-touching and attentional
processes: The role of task difficulty, selection stage,
and sex differences. Journal of Nonverbal Behavior, 10,
51–64. http://dx.doi.org/10.1007/BF00987205
Barroso, F., Freedman, N., Grand, S., & Van Meel, J.
(1978). The evocation of two types of hand movements
in information processing. Journal of Experimental
Psychology: Human Perception and Performance, 4,
321–329. http://dx.doi.org/10.1037/0096-1523.4.2.321
Bernieri, F. J. (1988). Coordinated movement and rapport
in teacher–student interactions. Journal of Nonverbal
Behavior, 12, 120–138. http://dx.doi.org/10.1007/
BF00986930
Bernieri, F. J., & Gillis, J. S. (1995). The judgment of
rapport: A cross-cultural comparison between
Americans and Greeks. Journal of Nonverbal Behavior,
19, 115–130. http://dx.doi.org/10.1007/BF02173170
Bernieri, F. J., Reznick, S. J., & Rosenthal, R. (1988).
Synchrony, pseudosynchrony, and dissynchrony:
Measuring the entrainment process in
mother–infant interactions. Journal of Personality
and Social Psychology, 54, 243–253. http://dx.doi.
org/10.1037/0022-3514.54.2.243
Blechko, A., Darker, I. T., & Gale, A. G. (2009). The role
of emotion recognition from nonverbal behaviour in
detection of concealed firearm carrying. Proceedings
of the Human Factors and Ergonomics Society 53rd
Annual Meeting, 53, 1363–1367.
Book, A., Costello, K., & Camilleri, J. A. (2013).
Psychopathy and victim selection: The use of gait
as a cue to vulnerability. Journal of Interpersonal
Violence, 28, 2368–2383. http://dx.doi.org/10.1177/
0886260512475315
Cashdan, E. (1998). Smiles, speech, and body posture:
How women and men display sociometric status and
power. Journal of Nonverbal Behavior, 22, 209–228.
http://dx.doi.org/10.1023/A:1022967721884
Chartrand, T. L., & Bargh, J. A. (1999). The chameleon
effect: The perception–behavior link and social
interaction. Journal of Personality and Social Psychology,
76, 893–910. http://dx.doi.org/10.1037/0022-
3514.76.6.893
Coulson, M. (2004). Attributing emotion to static
body postures: Recognition accuracy, confusions,
396
and viewpoint dependence. Journal of Nonverbal
Behavior, 28, 117–139. http://dx.doi.org/10.1023/
B:JONB.0000023655.25550.be
Dael, N., Mortillaro, M., & Scherer, K. R. (2012).
Emotion expression in body action and posture.
Emotion, 12, 1085–1101. http://dx.doi.org/10.1037/
a0025737
de Meijer, M. (1989). The contribution of general features
of body movement to the attribution of emotions.
Journal of Nonverbal Behavior, 13, 247–268. http://
dx.doi.org/10.1007/BF00990296
Derlega, V. J., Lewis, R. J., Harrison, S., Winstead, B. A., &
Costanza, R. (1989). Gender differences in the
initiation and attribution of tactile intimacy. Journal
of Nonverbal Behavior, 13, 83–96. http://dx.doi.org/
10.1007/BF00990792
Dickerson, S. S., Gruenewald, T. L., & Kemeny, M. E.
(2004). When the social self is threatened: Shame,
physiology, and health. Journal of Personality, 72,
1191–1216. http://dx.doi.org/10.1111/j.1467-
6494.2004.00295.x
DiMatteo, M. R., Friedman, H. S., & Taranta, A. (1979).
Sensitivity to bodily nonverbal communication as
a factor in practitioner–patient rapport. Journal of
Nonverbal Behavior, 4, 18–26. http://dx.doi.org/
10.1007/BF00986909
DiMatteo, M. R., Taranta, A., Friedman, H. S., & Prince, L. M.
(1980). Predicting patient satisfaction from
physicians’ nonverbal communication skills. Medical
Care, 18, 376–387. http://dx.doi.org/10.1097/
00005650-198004000-00003
Dolinski, D. (2010). Touch, compliance, and homophobia.
Journal of Nonverbal Behavior, 34, 179–192. http://
dx.doi.org/10.1007/s10919-010-0090-1
Ebesu Hubbard, A. S., Tsuji, A. A., Williams, C., &
Seatriz, V., Jr. (2003). Effects of touch on gratuities
received in same-gender and cross-gender dyads.
Journal of Applied Social Psychology, 33, 2427–2438.
http://dx.doi.org/10.1111/j.1559-1816.2003.tb01893.x
Ekman, P., & Friesen, W. V. (1969). The repertoire of
nonverbal behavior: Categories, origins, usage, and
coding. Semiotica, 1, 49–98.
Forston, R. F., & Larson, C. U. (1968). The dynamics of
space: An experimental study in proxemic behavior
among Latin Americans and North Americans.
Journal of Communication, 18, 109–116. http://dx.doi.
org/10.1111/j.1460-2466.1968.tb00061.x
Freedman, N., & Bucci, W. (1981). On kinesic filtering
in associative monologue. Semiotica, 34, 225–249.
http://dx.doi.org/10.1515/semi.1981.34.3-4.225
Friedman, H. S., Hall, J. A., & Harris, M. J. (1985).
Type A behavior, nonverbal expressive style, and
health. Journal of Personality and Social Psychology,
48, 1299–1315. http://dx.doi.org/10.1037/0022-
3514.48.5.1299

Gifford, R., & O’Connor, B. (1986). Nonverbal
intimacy: Clarifying the role of seating distance
and orientation. Journal of Nonverbal Behavior, 10,
207–214. http://dx.doi.org/10.1007/BF00987480
Grahe, J. E., & Bernieri, F. J. (1999). The importance
of nonverbal cues in judging rapport. Journal of
Nonverbal Behavior, 23, 253–269. http://dx.doi.org/
10.1023/A:1021698725361
Grammer, K., Kruck, K., & Magnusson, M. S. (1998).
The courtship dance: Patterns of nonverbal
synchronization in opposite-sex encounters. Journal
of Nonverbal Behavior, 22, 3–29. http://dx.doi.org/
10.1023/A:1022986608835
Grayson, B., & Stein, M. I. (1981). Attracting assault:
Victim’s nonverbal cues. Journal of Communication, 31,
68–75. http://dx.doi.org/10.1111/j.1460-2466.1981.
tb01206.x
Greenberg, C. I., Strube, M. J., & Myers, R. A. (1980).
A multitrait–multimethod investigation of
interpersonal distance. Journal of Nonverbal Behavior,
5, 104–114. http://dx.doi.org/10.1007/BF00986513
Guéguen, N., Afifi, F., Brault, S., Charles-Sire, V.,
Leforestier, P.-M., Morzedec, A., & Piron, E.
(2011). Failure of tactile contact to increase request
compliance: The case of blood donation behavior.
Journal of Articles in Support of the Null Hypothesis,
8, 1–8.
Guéguen, N., & Vion, M. (2009). The effect of a practitioner’s
touch on a patient’s medication compliance. Psychology,
Health, and Medicine, 14, 689–694. http://dx.doi.org/
10.1080/13548500903334739
Hall, E. T. (1963). A system for the notation of proxemic
behaviors. American Anthropologist, 65, 1003–1026.
http://dx.doi.org/10.1525/aa.1963.65.5.02a00020
Hall, E. T. (1966). The hidden dimension. New York, NY:
Doubleday.
Hall, E. T. (1973). The silent language. New York, NY: Anchor.
Hall, J. A. (1996). Touch, status, and gender at
professional meetings. Journal of Nonverbal Behavior,
20, 23–44. http://dx.doi.org/10.1007/BF02248713
Hall, J. A., Coats, E. J., & LeBeau, L. S. (2005). Nonverbal
behavior and the vertical dimension of social relations:
A meta-analysis. Psychological Bulletin, 131, 898–924.
http://dx.doi.org/10.1037/0033-2909.131.6.898
Harrigan, J. A., Oxman, T. E., & Rosenthal, R. (1985).
Rapport expressed through nonverbal behavior.
Journal of Nonverbal Behavior, 9, 95–110. http://
dx.doi.org/10.1007/BF00987141
Hayduk, L. A. (1983). Personal space: Where we now
stand. Psychological Bulletin, 94, 293–335. http://
dx.doi.org/10.1037/0033-2909.94.2.293
Henley, N. M. (1973). Status and sex: Some touching
observations. Bulletin of the Psychonomic Society, 2,
91–93. http://dx.doi.org/10.3758/BF03327726
The Body
Henley, N. M. (1977). Body politics: Power, sex, and
nonverbal communication. Englewood Cliffs, NJ:
Prentice Hall.
Hertenstein, M. J., Holmes, R., McCullough, M., &
Keltner, D. (2009). The communication of emotion
via touch. Emotion, 9, 566–573. http://dx.doi.org/
10.1037/a0016108
Hertenstein, M. J., Keltner, D., App, B., Bulleit, B. A.,
& Jaskolka, A. R. (2006). Touch communicates
distinct emotions. Emotion, 6, 528–533. http://dx.doi.
org/10.1037/1528-3542.6.3.528
Heslin, R., Nguyen, T. D., & Nguyen, M. L. (1983).
Meaning of touch: The case of touch from a stranger
or same sex person. Journal of Nonverbal Behavior, 7,
147–157. http://dx.doi.org/10.1007/BF00986945
Hollender, M. H., Luborsky, L., & Scaramella, T. J.
(1969). Body contact and sexual enticement.
Archives of General Psychiatry, 20, 188–191. http://
dx.doi.org/10.1001/archpsyc.1969.01740140060007
Hwang, H. C., & Matsumoto, D. (2014a). Cultural differences
in victory signals of triumph. Cross-Cultural Research, 48,
177–191. http://dx.doi.org/10.1177/1069397113510875
Hwang, H. C., & Matsumoto, D. (2014b). Dominance threat
display for victory and achievement in competition
context. Motivation and Emotion, 38, 206–214. http://
dx.doi.org/10.1007/s11031-013-9390-1
Janssen, D., Schollhorn, W. I., Lubienetzki, J., Folling, K.,
Kokenge, H., & Davids, K. (2008). Recognition of
emotions in gait patterns by means of artificial neural
nets. Journal of Nonverbal Behavior, 32, 79–92. http://
dx.doi.org/10.1007/s10919-007-0045-3
Keltner, D., & Harker, L. (1998). The forms and
functions of the nonverbal signal of shame. In
P. Gilbert & B. Andrews (Eds.), Shame: Interpersonal
behavior, psychopathology, and culture (pp. 76–98).
New York, NY: Oxford University Press.
Kirkcaldy, B., Furnham, A., & Levine, R. (2001).
Attitudinal and personality correlates of a nation’s pace
of life. Journal of Managerial Psychology, 16, 20–34.
http://dx.doi.org/10.1108/02683940110366551
Kleinke, C. I. (1977). Compliance to requests made by
gazing and touching experimenters in field settings.
Journal of Experimental Social Psychology, 13, 218–223.
http://dx.doi.org/10.1016/0022-1031(77)90044-0
Knapp, M. L. (1972). Nonverbal communication in human
interaction. New York, NY: Holt, Rinehart & Winston.
Kozlowski, L. T., & Cutting, J. E. (1977). Recognizing
the sex of a walker from a dynamic point light
display. Perception and Psychophysics, 21, 575–580.
http://dx.doi.org/10.3758/BF03198740
Kraus, M. W., Huang, C., & Keltner, D. (2010). Tactile
communication, cooperation, and performance: An
ethological study of the NBA. Emotion, 10, 745–749.
http://dx.doi.org/10.1037/a0019382
397
Matsumoto, Hwang, and Frank
Kudoh, T., & Matsumoto, D. (1985). Cross-cultural
examination of the semantic dimensions of body
postures. Journal of Personality and Social Psychology,
48, 1440–1446. http://dx.doi.org/10.1037/0022-3514.­­
48.6.1440
La France, M., & Ickes, W. (1981). Posture mirroring and
interactional involvement: Sex and sex typing effects.
Journal of Nonverbal Behavior, 5, 139–154. http://
dx.doi.org/10.1007/BF00986131
Lamb, W., & Watson, E. (1979). Body code: The meaning
in movement. London, England: Routledge and Kegan
Paul.
Lemke, M. R., Wendorff, T., Mieth, B., Buhl, K., &
Linnemann, M. (2000). Spatiotemporal gait patterns
during over ground locomotion in major depression
compared with healthy controls. Journal of Psychiatric
Research, 34, 277–283. http://dx.doi.org/10.1016/
S0022-3956(00)00017-0
Levine, R. V., & Bartlett, K. (1984). Pace of life, punctuality,
and coronary heart disease in six countries. Journal of
Cross-Cultural Psychology, 15, 233–255. http://dx.doi.
org/10.1177/0022002184015002009
Levine, R. V., Lynch, K., Miyake, K., & Lucia, M. (1989).
The Type A city: Coronary heart disease and the pace
of life. Journal of Behavioral Medicine, 12, 509–524.
http://dx.doi.org/10.1007/BF00844822
Levine, R. V., & Norenzayan, A. (1999). The pace of
life in 31 countries. Journal of Cross-Cultural
Psychology, 30, 178–205. http://dx.doi.org/10.1177/
0022022199030002003
Lewis, G. N., Byblow, W. D., & Walt, S. E. (2000). Stride
length regulation in Parkinson’s disease: The use of
extrinsic, visual cues. Brain: A Journal of Neurology,
123, 2077–2090. http://dx.doi.org/10.1093/brain/­
123.10.2077
Little, K. B. (1968). Cultural variations in social schemata.
Journal of Personality and Social Psychology, 10, 1–7.
http://dx.doi.org/10.1037/h0026381
Mahl, G. F. (1968). Gestures and body movements
in interviews. In J. M. Shlien, H. F. Hunt, J. O.
Matarazzo, & C. Savage (Eds.), Research in psychotherapy
(pp. 295–346). http://dx.doi.org/10.1037/10546-016
Major, B. (1981). Gender patterns in touching behavior.
In C. Mayo & N. M. Henley (Eds.), Gender and
nonverbal behavior (pp. 15–37). http://dx.doi.org/­
10.1007/978-1-4612-5953-4_2
Major, B., Schmidlin, A. M., & Williams, L. (1990).
Gender patterns in social touch: The impact of
setting and age. Journal of Personality and Social
Psychology, 58, 634–643. http://dx.doi.org/10.1037/
0022-3514.58.4.634
Matsumoto, D., & Hwang, H. S. (2012). Evidence
for a nonverbal expression of triumph. Evolution
and Human Behavior, 33, 520–529. http://dx.doi.
org/10.1016/j.evolhumbehav.2012.01.005
398
Matsumoto, D., & Kudoh, T. (1987). Cultural similarities
and differences in the semantic dimensions of body
postures. Journal of Nonverbal Behavior, 11, 166–179.
http://dx.doi.org/10.1007/BF00990235
McDaniel, E., & Andersen, P. A. (1998). International
patterns of interpersonal tactile communication: A
field study. Journal of Nonverbal Behavior, 22, 59–75.
http://dx.doi.org/10.1023/A:1022952509743
Meeren, H. K. M., van Heijnsbergen, C. C. R. J., & de
Gelder, B. (2005). Rapid perceptual integration of
facial expression and emotional body language.
Proceedings of the National Academy of Sciences,
USA, 102, 16518–16523. http://dx.doi.org/10.1073/
pnas.0507650102
Mehrabian, A. (1968a). Inference of attitudes from
the posture, orientation, and distance of a
communicator. Journal of Consulting and Clinical
Psychology, 32, 296–308. http://dx.doi.org/10.1037/
h0025906
Mehrabian, A. (1968b). Relationship of attitude to
seated posture, orientation, and distance. Journal of
Personality and Social Psychology, 10, 26–30. http://
dx.doi.org/10.1037/h0026384
Mehrabian, A. (1969). Significance of posture and
position in the communication of attitude and status
relationships. Psychological Bulletin, 71, 359–372.
http://dx.doi.org/10.1037/h0027349
Mehrabian, A. (1972). Nonverbal communication.
Chicago, IL: Aldine-Atherton.
Montepare, J. M., Goldstein, S. B., & Clausen, A. (1987).
The identification of emotions from gait information.
Journal of Nonverbal Behavior, 11, 33–42. http://
dx.doi.org/10.1007/BF00999605
Montepare, J. M., Koff, E., Zaitchik, D., & Albert, M.
(1999). The use of body movements and gestures as
cues to emotions in younger and older adults. Journal
of Nonverbal Behavior, 23, 133–152. http://dx.doi.org/
10.1023/A:1021435526134
Montepare, J. M., & Zebrowitz, L. A. (1993). A cross-
cultural comparison of impressions created by age
related variations in gait. Journal of Nonverbal Behavior,
17, 55–68. http://dx.doi.org/10.1007/BF00987008
Morris, D. (1971). Intimate behaviour. New York, NY:
Random House.
Muehlenhard, C. L., Koralewski, M. A., Andrews, S. L., &
Burdick, C. A. (1986). Verbal and nonverbal cues
that convey interest in dating. Behavior Therapy,
17, 404–419. http://dx.doi.org/10.1016/S0005-
7894(86)80071-5
Nguyen, T., Heslin, R., & Nguyen, M. L. (1976). The
meaning of touch: Sex and marital status differences.
Representative Research in Social Psychology, 7, 13–18.
Noesjirwan, J. (1977). Contrasting cultural patterns on
interpersonal closeness in doctors: Waiting rooms

in Sydney and Jakarta. Journal of Cross-Cultural
Psychology, 8, 357–368. http://dx.doi.org/10.1177/
002202217783008
Noesjirwan, J. (1978). A rule-based analysis of cultural
differences in social behavior: Indonesia and Australia.
International Journal of Psychology, 13, 305–316. http://
dx.doi.org/10.1080/00207597808246634
O’Neal, E. C., Brunault, M. A., Carifio, M. S., Troutwine,
R., & Epstein, J. (1980). Effect of insult upon
personal space preferences. Journal of Nonverbal
Behavior, 5, 56–62. http://dx.doi.org/10.1007/
BF00987055
Oveis, C., Gruber, J., Keltner, D., Stamper, J. L., & Boyce,
W. T. (2009). Smile intensity and warm touch as
thin slices of child and family affective style. Emotion,
9, 544–548. http://dx.doi.org/10.1037/a0016300
Pagán, G., & Aiello, J. R. (1982). Development of
personal space among Puerto Ricans. Journal of
Nonverbal Behavior, 7, 59–68. http://dx.doi.org/
10.1007/BF00986868
Patterson, M. L., Powell, J. L., & Lenihan, M. G. (1986).
Touch, compliance, and interpersonal affect. Journal
of Nonverbal Behavior, 10, 41–50. http://dx.doi.org/
10.1007/BF00987204
Paulus, P. B., McCain, G., & Cox, V. C. (1978). Death
rates, psychiatric commitments, blood pressure, and
perceived crowding as a function of institutional
crowding. Environmental Psychology and Nonverbal
Behavior, 3, 107–116. http://dx.doi.org/10.1007/
BF01135608
Remland, M. S., Jones, T. S., & Brinkman, H. (1991).
Proxemic and haptic behavior in three European
countries. Journal of Nonverbal Behavior, 15,
215–232. http://dx.doi.org/10.1007/BF00986923
Richardson, M. J., & Johnston, L. (2005). Person
recognition from dynamic events: The kinematic
specification of individual identity in walking style.
Journal of Nonverbal Behavior, 29, 25–44. http://
dx.doi.org/10.1007/s10919-004-0888-9
Richmond, V. P., McCroskey, J. C., & Payne, S. K.
(1991). Nonverbal behavior in interpersonal relations.
Englewood Cliffs, NJ: Prentice Hall.
Roese, N. J., Olson, J. M., Borenstein, M. N., Martin, A.,
& Shores, A. L. (1992). Same-sex touching behavior:
The moderating role of homophobic attitudes.
Journal of Nonverbal Behavior, 16, 249–259. http://
dx.doi.org/10.1007/BF01462005
Rosenfeld, H. M. (1966). Approval-seeking and approval
inducing functions of verbal and nonverbal
responses in the dyad. Journal of Personality and
Social Psychology, 4, 597–605. http://dx.doi.org/
10.1037/h0023996
Rotenberg, K. J., Eisenberg, N., Cumming, C., Smith, A.,
Singh, M., & Terlicher, E. (2003). The contribution
The Body
of adults’ nonverbal cues and children’s shyness
to the development of rapport between adults
and preschool children. International Journal of
Behavioral Development, 27, 21–30. http://dx.doi.
org/10.1080/01650250143000571
Sakaguchi, K., & Hasegawa, T. (2006). Person perception
through gait information and target choice for
sexual advances: Comparison of likely targets in
experiments and real life. Journal of Nonverbal
Behavior, 30, 63–85. http://dx.doi.org/10.1007/
s10919-006-0006-2
Shuter, R. (1976). Proxemics and tactility in Latin
America. Journal of Communication, 26, 46–52. http://
dx.doi.org/10.1111/j.1460-2466.1976.tb01902.x
Stier, D. S., & Hall, J. A. (1984). Gender differences in
touch: An empirical and theoretical review. Journal
of Personality and Social Psychology, 47, 440–459.
http://dx.doi.org/10.1037/0022-3514.47.2.440
Strube, M. J., & Werner, C. (1982). Interpersonal
distance and personal space: A conceptual and
methodological note. Journal of Nonverbal Behavior,
6, 163–170. http://dx.doi.org/10.1007/BF00987065
Sundstrom, E., & Sundstrom, M. G. (1977). Personal
space invasions: What happens when the invader
asks for permission? Environmental Psychology and
Nonverbal Behavior, 2, 76–82. http://dx.doi.org/
10.1007/BF01145823
Sussman, N. M., & Rosenfeld, H. M. (1978). Touch,
justification, and sex: Influences on the aversiveness
of spatial violations. Journal of Social Psychology, 106,
215–225. http://dx.doi.org/10.1080/00224545.1978.
9924173
Sussman, N. M., & Rosenfeld, H. M. (1982). Influence of
culture, language, and sex on conversational distance.
Journal of Personality and Social Psychology, 42, 66–74.
http://dx.doi.org/10.1037/0022-3514.42.1.66
Thayer, S. (1986). History and strategies of research
on social touch. Journal of Nonverbal Behavior, 10,
12–28. http://dx.doi.org/10.1007/BF00987202
Thompson, D. E., Aiello, J. R., & Epstein, Y. M. (1979).
Interpersonal distance preferences. Journal of
Nonverbal Behavior, 4, 113–118. http://dx.doi.org/
10.1007/BF01006355
Tickle-Degnen, L., & Rosenthal, R. (1987). Group
rapport and nonverbal behavior. Review of Personality
and Social Psychology, 9, 113–136.
Tickle-Degnen, L., & Rosenthal, R. (1990). The nature
of rapport and its nonverbal correlates. Psychological
Inquiry, 1, 285–293. http://dx.doi.org/10.1207/
s15327965pli0104_1
Tracy, J. L., & Matsumoto, D. (2008). The spontaneous
expression of pride and shame: Evidence for
biologically innate nonverbal displays. Proceedings
of the National Academy of Sciences, USA, 105,
399
Matsumoto, Hwang, and Frank
11655–11660. http://dx.doi.org/10.1073/pnas.
0802686105
Tracy, J. L., & Robins, R. W. (2004). Show your pride:
Evidence for a discrete emotion expression.
Psychological Science, 15, 194–197. http://dx.doi.
org/10.1111/j.0956-7976.2004.01503008.x
Tracy, J. L., & Robins, R. W. (2008). The nonverbal
expression of pride: Evidence for cross-cultural
recognition. Journal of Personality and Social
Psychology, 94, 516–530. http://dx.doi.org/
10.1037/0022-3514.94.3.516
Tracy, J. L., Robins, R. W., & Lagattuta, K. H. (2005).
Can children recognize pride? Emotion, 5, 251–257.
http://dx.doi.org/10.1037/1528-3542.5.3.251
Trout, D. L., & Rosenfeld, H. M. (1980). The effect of
postural lean and body congruence on the judgment of
psychotherapeutic rapport. Journal of Nonverbal Behavior,
4, 176–190. http://dx.doi.org/10.1007/BF00986818
Van Dyck, E., Maes, P.-J., Hargreaves, J., Lesaffre, M., &
Leman, M. (2013). Expressing induced emotions
through free dance movement. Journal of Nonverbal
Behavior, 37, 175–190. http://dx.doi.org/10.1007/
s10919-013-0153-1
400
Watson, O. M. (1970). Proxemic behavior: A cross-cultural
study. The Hague, the Netherlands: Mouton.
Watson, O. M., & Graves, T. D. (1966). Quantitative
research in proxemic behavior. American
Anthropologist, 68, 971–985. http://dx.doi.org/10.1525/
aa.1966.68.4.02a00070
Willis, F. N., Jr., & Briggs, L. F. (1992). Relationship
and touch in public settings. Journal of Nonverbal
Behavior, 16, 55–63. http://dx.doi.org/10.1007/
BF00986879
Willis, F. N., Jr., & Hamm, H. K. (1980). The use of
interpersonal touch in securing compliance. Journal
of Nonverbal Behavior, 5, 49–55. http://dx.doi.org/­
10.1007/BF00987054
Winter, D. D. N., Widell, C., Truitt, G., & George-Falvy,
J. (1989). Empirical studies of posture–gesture
mergers. Journal of Nonverbal Behavior, 13, 207–223.
http://dx.doi.org/10.1007/BF00990294
Worchel, S. (1986). The influence of contextual variables
on interpersonal spacing. Journal of Nonverbal
Behavior, 10, 230–254. http://dx.doi.org/10.1007/
BF00987482
 Chapter 16
Nonverbal Communication in
Primates: Observational and
Experimental Approaches
Lisa A. Parr, Jérôme Micheletta, and Bridget M. Waller
Primate nonverbal communication has received
considerable scientific attention since Darwin
(1872) first commented on the expressive similarity
between species more than 140 years ago. Here, we
review classic and recent findings in primate com-
munication research and explain how these results
have influenced the understanding of human non-
verbal behavior. First, the modes of communication
used by primates and the various theoretical and
methodological approaches employed to tackle these
processes are reviewed. Second, the major findings
from observational studies are summarized. Finally,
the major findings from experimental approaches
are discussed.
INTRODUCTION TO PRIMATE
NONVERBAL COMMUNICATION
How Do Primates Communicate?
Unlike human language, nonverbal communication
has clear counterparts in other primate species. Pri-
mates communicate with their faces (facial expres-
sions), with their hands and bodies (gestures), using
sounds and vocalizations (vocal cues), and also
through olfactory signaling (chemical cues). There-
fore, examining nonverbal communication systems
in other primates can be highly informative when
trying to understand the form, function, and evolu-
tion of nonverbal signals. Like humans, nonhuman
primates (here­after, primates) produce meaningful,
communicative, and functionally relevant body pos-
tures, gestures, vocalizations, and facial expressions,
many of which appear to be homologous (deriving
http://dx.doi.org/10.1037/14669-016
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
from the same ancestral behavior) across related
species (see also Chapter 10, this handbook).
Facial behavior has received the most attention
because of its similarity between humans and other
primates, so here we treat the facial channel and non-
facial channel separately.
The facial channel.  Facial expressions are particu-
larly conserved across the primate order to the extent
that the same morphologically homologous displays
seem to form the core facial communicative reper-
toires of many different primate species (Andrew,
1963; Bolwig, 1964). The bared-teeth display (see
Figure 16.1), for example, is a common expres-
sion displayed by many primate species (Bout &
Thierry, 2005; De Marco, Petit, & Visalberghi, 2008;
de Waal & Luttrell, 1985; Laidre & Yorzinski, 2005;
Maestripieri, 1996; Pellis, Pellis, & Thierry, 1997;
Waller & Dunbar, 2005), and researchers have
hypothesized that this conspicuous expression is
related to, or homologous with, the human smile
(van Hooff, 1972). However, the specific form and
function of the bared-teeth display can vary across
species depending on a variety of socioecological fac-
tors, but there are key similarities in the expression
of bared-teeth displays between species that help to
explain the importance and ubiquity of this expres-
sion in human social interaction, for example, smiles.
Regardless of the specific social context in
which it is expressed, the bared-teeth display
has an appeasing and/or affiliative function that
serves to reduce or avoid conflict (Preuschoft,
1992; Waller & Dunbar, 2005). This function is
401
Parr, Micheletta, and Waller

FIGURE 16.1.  Facial expression commonly observed in primates. Left: A silent bared-teeth display expressed
by a young male crested macaque. Right: A play face displayed by an adult female crested macaque. From Primate
Communication: A Multimodal Approach, by K. Liebal, B. M. Waller, A. M. Burrows, and K. E. Slocombe, 2014,
p. 224. Copyright 2014 by Cambridge University Press. Reprinted with permission.

extremely applicable for use in humans who have a
highly cooperative and affiliative social style (Mehu,
Grammer, & Dunbar, 2007). Mehu et al. (2007),
for example, showed that spontaneous, genuine
smiles (also referred to as Duchenne smiles, which
are defined by contraction of the orbicularis oculi
around the eyes in addition to the zygomatic major
raising the lip corners) are used most often by
humans in cooperative, sharing contexts, suggesting
that these smiles reliably advertise altruistic inten-
tions and help to form and maintain cooperative
relationships. The interpretation that bared-teeth
displays serve an appeasing, conflict-reducing func-
tion has also been a useful approach in the studies of
material objects from past human cultures that are
adorned with these facial motifs (Samson & Waller,
2010). Previous studies interpreted the bared-teeth
motifs as depicting angry faces or deathlike grimaces,
but Samson and Waller (2010) argued that their use
on objects associated with ritual and inter- and intra-
community interaction was more likely to represent
cooperative intentions, like the human smile.
Another facial expression commonly observed
across the primate order is the play face (see
Figure 16.1). The origin of this display may lie in
the movements surrounding biting, having been
ritualized so that the upper lip covers the upper
teeth, and the mouth appears open and relaxed.
Restricted to social contexts defined as playful, the
play face seems to function as a signal to initiate
and/or prolong playful activities. It is possible that
the expression is used to demonstrate the ability to
harm another individual, for example, its biting ori-
gin, but that the displayer has no intention of doing
so. Thus, the use of this signal may function to con-
firm that play is only play (Pellis & Pellis, 1996).
Human laughter has long been considered to be a
homologue of the play face (van Hooff, 1972), and
some primate species have accompanying vocaliza-
tions that seem akin to vocalized laughter (Davila
Ross, Owren, & Zimmermann, 2009). Davila Ross
et al. (2009), for example, found that the acoustic
similarities between nonhuman primate and human
“laughter” vocalizations match known phylogenetic
relationships, supporting the hypotheses that these
vocalizations are homologous. Much less attention
has been paid to other facial expressions in terms
of their similarity across species and likely human

402

counterparts, presumably due to the extensive time
involved in describing and categorizing these signals
(Parr, Cohen, & de Waal, 2005) but also the lack of
standardized nomenclature to describe them across
different species (see the discussion later in this
chapter on the Facial Action Coding System [FACS];
Ekman, Friesen, & Hager, 2002).
The nonfacial channel.  Similar to facial expres-
sions, primate vocalizations also show a high degree
of conservation in terms of their basic categories
and the contexts in which they are produced. Many
primates have individual vocalizations represent-
ing threats, alarms, warning calls, contact calls,
and calls indicating affiliation; however, with the
exception of laughter, whether structure is preserved
with phylogeny has less often been the focus of
observational or experimental studies. An impor-
tant difference between visual facial expressions
and auditory vocalizations is that calls can travel
over considerable distance and can, therefore, be
detected in the absence of direct visual contact. For
this reason, evolution may have favored maintain-
ing unique properties of call sounds across species
to ensure that individuals are able to discriminate
between similar categories of calls made by con-
specifics (members of the same species) from those
of heterospecifics (members of different species;
Krebs, Davies, & Parr, 1993). Thus, the auditory
and structural properties of vocalizations may be of
less interest in and of themselves than other factors,
including the intent for sending the signal and how
it is perceived by others—for example, the sender/
receiver psychology, its social function, and the rela-
tionship between specific calls and human language.
Thus, although many social primates produce simi-
lar types of vocalizations in similar contexts—for
example, screaming during conflict, alarm calls in
the presence of predators, and so forth—these vocal-
izations may sound very different from one another,
and, in some cases, there may be unique repertoires
within members of the same species.
Displays involving the whole body abound in the
descriptive repertoires of many different primate spe-
cies (Bertrand, 1969; van Hooff, 1970), but there has
been a relatively recent focus on movements, partic-
ularly actions involving the hands, that are referred
Nonverbal Communication in Primates
to as gestures (Liebal, Mueller, & Pika, 2007). Simi-
lar to vocalizations, the research focus is less often
on the specific structural properties of gestures and
more on how gesture use may reveal interesting
underlying cognitive processes, such as intentional-
ity (Pika & Mitani, 2006). Evidence suggests that
primate gestures may be the only nonhuman com-
munication system that can be used intentionally,
minimally qualifying for first-order intentionality,
meaning that the signals are produced with the goal
of changing the behavior of others (Dennett, 1987).
A recent study of wild chimpanzees attempted to
identify the meaning of gestures by focusing on
whether the production of the gesture stopped
the ongoing communication, indicating that the
recipient’s response satisfied the sender and that the
intended meaning was achieved (Hobaiter & Byrne,
2014). These authors found that many gestures used
by the majority of individuals within the community
produced reliable outcomes in other group mem-
bers, indicating that the gestures themselves had
meaning. This refutes an existing claim that gestures
reflect idiosyncratic behaviors specific to particular
individuals (Tomasello, 2008; for a related discus-
sion, see also Chapter 12, this handbook).
What Can We Learn From Primate
Communication?
Traditionally, the three main channels of primate
behavioral communication—facial expression,
vocalization, and gesture—have been studied indi-
vidually, often under different research assumptions
and using different experimental approaches (Liebal,
Waller, Burrows, & Slocombe, 2014; Slocombe,
Waller, & Liebal, 2011). Such differences have an
influence on the perceived relevance of looking
to primate communication to understand human
communication. For example, primate facial expres-
sions are often conceptualized as emotional signals
and are therefore seen as potentially useful in the
attempt to understand the evolution of human emo-
tion (Parr & Waller, 2007). In contrast, vocaliza-
tions and gestures are often studied with an a priori
interest in how these signals relate to the cognitive
state of the signaler, possibly indicating their refer-
entiality, intentionality, or other levels of flexibility
that may help to identify the evolution of human
403
Parr, Micheletta, and Waller
language. For example, primate alarm calls and food
calls may be referential in the sense that they refer to
something external in the environment, in contrast
to the caller’s internal motivation. This referential-
ity is an important characteristic of human language
that is not usually associated with animal communi-
cation systems (Seyfarth, Cheney, & Marler, 1980;
Slocombe & Zuberbühler, 2005b). Another feature
of human language that is regularly pursued in pri-
mate communication studies, particularly research
on gestures, is its intentionality. The default, parsi-
monious assumption is that animal communication
is not voluntary or intentional, and so scientists
are keen to find evidence to the contrary, which,
again, might help in the effort to understand the
evolutionary origins of human language (Leavens,
Russell, & Hopkins, 2005; Schel, Townsend, Mach-
anda, Zuberbühler, & Slocombe, 2013). The goal of
this approach has been to find a hint of the largely,
although not always, intentional communication
of humans in their primate relatives, which could
indicate the early roots of human language. The
accepted criteria for intentionality vary across stud-
ies and research groups (Liebal et al., 2014) and
include behavioral indicators such as persistence,
elaboration, responding to the attentional state of
the recipient, and flexibility of usage. To date, some
of the most convincing evidence of intentionality
has come from primate gestures (as described ear-
lier; Genty, Breuer, Hobaiter, & Byrne, 2009), but
recent findings in chimpanzees suggest that primate
vocalizations might also meet the established cri-
teria for intentionality (Schel et al., 2013). In light
of these differences, there has been a recent call
for a multimodal perspective to better understand
primate communication (Liebal et al., 2014). By
studying primate communication as an integrated
whole, researchers may avoid drawing inaccurate
conclusions from studies that focus on one modal-
ity at a time and/or have little methodologically in
common, all leading to a better understanding of the
true nature of communication systems.
Primate communication research is nested
within longstanding historical traditions, usually
with a biological focus and with the goal of better
understanding human communication. Research
on facial expressions in particular, compared to
404
vocalizations and gestures, has often focused on the
receiver of the signal, where the response given to
a particular signal or display may provide informa-
tion about the signal’s meaning (Fridlund, 1994;
Guilford & Dawkins, 1991). Within this “receiver
psychology” framework, the goal of the sender
is seen as less important than the response of the
receiver, as it is only through achieving responses
in others that a signal can become fixed over time
and reliably communicative. However, there is an
interesting debate in animal behavior surround-
ing the nature of this communication sequence in
terms of what is actually happening to the individual
receiving the information. Rendall, Owren, and
Ryan (2009) argued that conceptualizing a commu-
nicative sequence as information transfer invokes
unnecessary and cognitively loaded anthropomor-
phic similarities with human language. Specifically,
the authors argued that scientists should not assume
that any information has been transferred. Instead,
the authors advocated for an alternative approach
where scientists focus on the concrete, measurable
sensory, physiological, and psychological changes
that occur in receivers, much like that espoused in
the original receiver psychology viewpoint. Impor-
tantly, they argued that we need to separate the roles
of sender and receiver and not assume that there is
shared information between both parties. In other
words, receivers may not interpret the information
that is sent in the way in which it was intended.
Seyfarth et al. (2010) defend the use of informa-
tion transfer as a concept in animal communication
based on their own experimental findings. They
demonstrated that, as a result of communicat-
ing with others, individuals have clearly acquired
new information about things they did not “know”
before. In this case, conceptualizing communica-
tion as a means of transferring information does
not exclude the importance and constraints of the
receiver’s sensory and psychological processes. How
the signal is produced and perceived, and what
information the signal carries, can and should be
kept separate (Scarantino, 2010). In one sense, the
nature of information is that it can be conceptual-
ized in different ways, including the sender’s emo-
tion, intention, or an indication of its forthcoming
behavior. What is key from a functional perspective

is that the receiver has been exposed to something
it was not previously privy to, regardless of whether
the information matches exactly what was “sent”
or was intended to be sent. In this way, receivers
can respond to the information in flexible ways that
essentially enable them to maximize their own self-
interests (Waller & Cherry, 2012). For example,
a primate could use the information present in
another’s face, for example, a play-face expression,
to decide whether to continue playing. The informa-
tion present in the face does not necessarily need
to be experienced by the sender and receiver in the
same way to be conceptualized as information that
the receiver did not have before. The face may not
even be understood and could instead simply stimu-
late a response in others. Such an interpretation,
however, could still be seen as information transfer.
How Can We Make Direct Comparisons
Between Human and Nonhuman Primate
Nonverbal Communication?
With a few exceptions, comparative studies of pri-
mate communication have focused on identifying
the cognitive and behavioral characteristics of com-
municative signals across species, as opposed to
identifying whether these behaviors are homologous
in origin. Some exceptions to this have been stud-
ies of primate laughter (Davila Ross et al., 2009)
and the potentially homologous frequency between
mouth movements used in primate lipsmacks and
human speech (Ghazanfar, 2013; Ghazanfar, Chan-
drasekaran, & Morrill, 2010). The main exception,
however, has been the case of facial expressions,
which have long been examined as homologous
counterparts in a variety of species (Darwin, 1872).
More recently, the study of facial expressions has
benefited from the development of an objective,
observational coding system that provides research-
ers with a framework and nomenclature to verify
that the same movements are being identified as the
same expression. The FACS (Ekman et al., 2002) is
an anatomically based system originally developed
in the 1970s to compare expressions across diverse
human populations. The FACS was designed to
deconstruct facial expressions into their most basic
elements, the most minimally observable move-
ments of the face. These movements correspond to
Nonverbal Communication in Primates
the action of the underlying facial musculature, and
thus the same movement in two different people is
presumed to share a comparable underlying biologi-
cal origin.
The FACS in humans involves a detailed train-
ing and certification process and remains today as
the gold standard for describing facial movement.
Recently, the FACS has been modified to describe
facial movements in species other than humans,
including several species of nonhuman primates and
some domestic animals. This expansion provides a
rigorous method for comparing facial expressions
across species at the level of the facial musculature.
The first modification of FACS for use with nonhu-
man animals, for example, chimpanzees (Pan troglo-
dytes), was ChimpFACS (Vick, Waller, Parr, Smith
Pasqualini, & Bard, 2007), followed by MaqFACS
(Parr, Waller, Burrows, Gothard, & Vick, 2010) for
use with rhesus macaques (Macaca mulatta). Other
primate systems include GibbonFACS (Waller,
Lembeck, Kuchenbuch, Burrows, & Liebal, 2012)
and OrangFACS (Caeiro, Waller, Zimmermann,
Burrows, & Davila-Ross, 2013), and most recently
the system has been modified for use with dogs
(Waller et al., 2013). Development of each system
has relied on an explicit understanding of the facial
musculature of each species and how it is similar to
and/or different from humans (Burrows, 2008). In
some cases, intramuscular electrical stimulations
were performed to provide direct knowledge of how
activation of specific muscles, or combination of
muscles, changes the appearance of the face leading
to the described movement, called an action unit or
AU (Waller, Parr, Gothard, Burrows, & Fuglevand,
2008; Waller et al., 2006).
The FACS versions have been used to make
explicit comparisons between proposed facial
expression homologues in humans and chimpan-
zees, confirming previous observations that many
expressions are morphologically similar (Parr,
Waller, & Vick, 2007), and also to understand how
chimpanzees are discriminating between specific
expressions (Parr, Waller, & Heintz, 2008). A FACS
style approach has also been used to conduct com-
parative analyses on the relationship between facial
movement and socioecological variables across the
primate order (Dobson, 2009b). Another useful way
405
Parr, Micheletta, and Waller
that FACS can be applied has been to make explicit
comparisons between species in identical experi-
mental situations. For example, Waller, Misch,
Whitehouse, and Herrmann (2014) compared the
facial expressions of human children and chimpan-
zees when engaging in a task that had no solution
and was, therefore, impossible to solve. The more
they persisted with the task, the more the children
produced expressive elements of anger, but the
chimpanzees did not, suggesting that there may be
a functional significance to these expressions for
humans, for example, “I’m frustrated, help me.”
In sum, primate communication consists of
many sources of information, including visual, audi-
tory, and gestural signals that have been studied
and interpreted by scientists using many different
practical and theoretical approaches (Liebal et al.,
2014; Slocombe et al., 2011). Drawing broad con-
clusions about the nature of primate communica-
tion and how the different modalities contribute
to communication as a whole is a challenging task.
Nevertheless, the diversity of scientists addressing
primate communication makes for a rich and fruit-
ful discipline, and the findings continue to advance
the understanding of human communication and its
evolution.
MAIN FINDINGS FROM OBSERVATIONAL
STUDIES
Physical, Ecological, and Social Influences
Despite some signals being highly conserved across
species, primate communication remains very
diverse in terms of the range of signals that can be
produced, the contexts in which they are used, and
their specific meaning. Moreover, some species
use a large number of communicative signals that
can be highly complex and involve different sen-
sory modalities, whereas others may rely on only a
few key signals expressed using a restricted range
of modalities. Why do some species have such a
large repertoire of communicative displays com-
pared to others? Furthermore, why do some species
rely more on some sensory channels than others?
Since the rise of behavioral ecology, scientists have
tried to identify features of the environment and
characteristics of the social systems of species that
406
could account for the variations observed in pri-
mate communication. In this section, we report on
three possible influences—physical, ecological, and
social—that may have contributed to the properties
of primate communication systems. Comparative
studies evaluating the importance of these evolu-
tionary relevant variables can help researchers to
understand the diversity of nonhuman primate com-
munication and which of these variables, or com-
bination of variables, may have contributed to the
seemingly uniquely and highly complex communi-
cation system in humans.
Physical Influences on Communication:
Allometry
The evolution of at least some communication sig-
nals is subject to allometric constraints—the idea
that anatomy, physiology, and behavior scales in
relation to one’s body size. In many animals, both
the length of the vocal folds, which determines the
minimum frequency of calls, and the shape of the
vocal tract, which affects its filtering properties, vary
with body size. In primates, a comparative study
revealed that overall, large individuals utter calls
that are longer in duration, lower in fundamental
frequency, and have smaller formant dispersion
than smaller animals (Ey, Pfefferle, & Fischer,
2007). Therefore, specific features of primate vocal-
izations give important clues regarding the body
size of the caller, which is an important correlate
of one’s genetic fitness. In the visual domain, body
size correlates strongly with facial mobility, such
that large-bodied primate species have a higher
facial mobility than small-bodied species, and this
relationship holds even when taking the phyloge-
netic proximity of the different species into account
(Dobson, 2009a).
The idea of allometric constraint in the visual
modality seems to be exercised on the perception
of communicative signals such that large individual
species are better equipped to detect and process
subtle facial movements (Kiltie, 2000). Structural
constraints are also likely to apply to the produc-
tion of facial expressions. Facial mobility depends
on the number and size of facial muscles, which,
in turn, relate to the volume of the facial nucleus,
for example, an estimate of the number of motor

neurons traveling from brain stem to the facial
muscles. Recent comparative studies of the volume
of the facial nucleus indicate that large species have
a larger facial nucleus than smaller bodied spe-
cies (Sherwood, 2005). Because both an increase
in muscle size and the number of muscles would
lead to an increase in facial nucleus volume, this
suggests that larger bodied species, for example,
chimpanzees, have a higher degree of facial mobility
than smaller bodied species, for example, cotton-
top tamarins (Dobson, 2009a). However, a direct
comparison of the number and organization of the
facial musculature across a variety of primates has
revealed a very similar muscular plan (Burrows,
2008). Therefore, the evidence suggests that it is not
the number of facial muscles per se but, perhaps,
how they are used that matters more for commu-
nication. Moreover, within large-bodied primates,
there can be large differences in their facial expres-
sion repertoire that are independent of their capacity
for facial movement. FACS versions in gorillas and
chimpanzees have revealed that these two great ape
species have the same degree of facial mobility, for
example, each has 13 anatomically distinct basic
movements or AUs. However, gorillas are known
for their greater reliance on gestural communica-
tion compared to facial expressions, whereas chim-
panzees display a broad and diverse range of facial
expressions (Genty et al., 2009; Parr et al., 2005).
Furthermore, the allometry of facial mobility has no
implication regarding the complexity of the overall
communication system of a species. Species char-
acterized by a smaller body size can simply rely on
other sensory modalities, such as olfaction or vocal-
izations. Ring-tailed lemurs, for example, express
complex semiochemicals (Scordato, Dubay, & Drea,
2007) that they use in diverse social contexts, such
as mating and intergroup communication (Palagi &
Norscia, 2009).
Ecological Influences on Communication
Signals have to travel from the sender to the receiv-
ers through the environment with as little disrup-
tion as possible. Therefore, the ecological milieu
of a species constrains the extent to which the
species can rely on certain sensory modalities and
also affects specific features of their communicative
Nonverbal Communication in Primates
signals. The study of ecological influences on pri-
mate communication is a difficult topic, mainly
because it is difficult to precisely assess the physi-
cal constraint imposed by the environment, and
environmental influences change frequently. Often,
researchers have to rely on coarse classifications
given the limited data available. For example, over-
all percentage of arboreality can be calculated by
recording the time a species spends on the ground,
but a simple arboreal/terrestrial dichotomy might
not be detailed enough to reveal the effect of ecology
on the evolution of communication systems. Given
the small broadcast range of visual signals and their
poor ability to transmit through obstacles, compared
to vocal signals, one of the most common assump-
tions is that arboreal species will show a greater
reliance on acoustic versus visual modalities in
their communication systems. Moreover, it is often
assumed, although difficult to ascertain, that dense
foliage results in visual barriers and poor lighting
conditions, making visual communication poten-
tially ineffective. Highly terrestrial species should,
therefore, exhibit greater reliance on facial expres-
sions and also display higher facial mobility than
arboreal species. However, a recent comparative
study showed that terrestrial species tended to have
a higher facial mobility than arboreal ones, although
the relationship was not statistically significant, pos-
sibly because arboreality was only measured using a
dichotomous scale (Dobson, 2009b).
Other studies have found that some properties
of visual signals may have experienced ecological
pressures related to thermoregulation, camouflage,
and need to reduce solar glare (Santana, Alfaro, &
Alfaro, 2012; Santana, Alfaro, Noonan, & Alfaro,
2013). Although these ecological pressures may be
primarily related to physiological functions, any
changes in the color of the face or length of facial
hair could also affect communication via the facial
channel. In a series of elegant studies, Santana et al.
(2012, 2013) calculated an index of facial color pat-
tern complexity using multiple areas of the faces
and recording traits describing the hair and skin
color (see Figure 16.2). Generally, these authors
found that species living in tropical, dense forests
had darker parts of their face and darker faces over-
all, whereas species living in more open habitats
407
Parr, Micheletta, and Waller
FIGURE 16.2.  The face of a Barbary macaque illustrat-
ing the procedure used to characterize facial color pat-
tern complexity. Faces were subdivided into 10 areas
that were used to record traits describing the hair and
skin color (1. crown medial; 2. crown lateral; 3. fore-
head; 4. margin dorsal; 5. margin medial; 6. margin
ventral; 7. nose; 8. mouth; 9. eye mask; 10. cheek).
From “Adaptive Response to Sociality and Ecology
Drives the Diversification of Facial Colour Patterns in
Catarrhines,” by S. E. Santana, J. L. Alfaro, A. Noonan,
and M. E. Alfaro, 2013, Nature Communications, 4,
p. 5. Copyright 2013 by the Nature Publishing Group.
Reprinted with permission.
and temperate climates had lighter faces. This pat-
tern could have evolved to make individuals more
cryptic and therefore less vulnerable to predation.
Whether darker faces also provide a better thermo-
regulation and better protection against ultraviolet
radiation remains to be tested. These variations also
have implications for social communication as well
as individual and species recognition (as shown later
in this chapter).
There are also indications that the environment
can affect vocal signals (Ey & Fischer, 2009). The
vocal repertoires of primate species living in closed
habitats seem to contain more discrete rather than
graded calls, and these calls also show lower vari-
ability that those emitted in open habitats. These
factors may help to ensure that the signal is not
degraded by the foliage or other structure in a dense
environment. The frequency and timing of back-
ground noise is another factor that may influence
the quality of communicative signals. The frequency
408
in which calls are omitted must differ from the fre-
quency of background noise to facilitate effective
communication between individuals, particularly
in noisy environments. Individuals can also adjust
the timing of their calls according to ambient noise,
thereby optimizing sound transmission.
Social Influences on Communication
According to the social intelligence hypothesis
(Dunbar, 1993; Humphrey, 1976; Jolly, 1966),
individuals living in complex societies face greater
challenges than individuals living in simpler soci-
eties, leading to larger brains and more advanced
social cognition. For example, individuals need to
recognize a large number of group members and
need to process and remember many complex social
interactions, including third-party interactions, to
strategically and effectively assess the behavior of
group mates. Thus, the social intelligence hypoth-
esis suggests that sociality could be an important
selection pressure leading to the evolution of
complex communication systems. This concept is
referred to as the social complexity hypothesis for
communication complexity (Freeberg, Dunbar, &
Ord, 2012).
Comparative studies have examined some of the
predictions of the social complexity hypothesis. The
majority of these studies have focused on vocal com-
munication, presumably because of its relevance to
the evolution of human language and because of the
tools available for precise and objective analyses of
auditory signals (Slocombe et al., 2011). In a large-
scale study, researchers found that the size of the
vocal repertoire correlated with overall group size
and the time individuals spent grooming—two com-
mon measures of social complexity (McComb &
Semple, 2005). Several authors have reported facial
expressions and vocalizations that are displayed in
unidirectional ways and that are directed to domi-
nant individuals, including the bared-teeth display
in rhesus macaques (de Waal & Luttrell, 1985) and
pant-grunt vocalizations in the chimpanzee (de
Waal, 1982). These are referred to as formal signals
in that they provide information about hierarchy
status. In the visual domain, an analysis of gestures
in several species has revealed that the more toler-
ant species seem to have more complex affiliative

signals, whereas the signals used to convey domi-
nance and submission showed no difference across
species (Maestripieri, 1999). Dobson (2009b) exam-
ined the relationship between group size and facial
mobility while controlling for phylogeny. He found
that facial mobility increased in parallel with group
size in several primates, highlighting the impor-
tant role of facial expressions in maintaining group
cohesion. This finding was later extended to social
tolerance, such that more socially tolerant species of
macaques, characterized by high rates of reconcili-
ation and counteraggression, were found to have a
larger facial expression repertoire than species that
showed less social tolerance (Dobson, 2012). The
assumption is that the social relationships of more
tolerant species are more varied and less rigid than
intolerant species, and, therefore, individuals have a
greater need for communicative flexibility.
Further support for the social complexity
hypothesis in primates comes from the analysis of
the facial color pattern complexity described ear-
lier (Santana et al., 2013). In a comparative study
of 139 primates, species with larger group sizes
had more complex facial color patterns than spe-
cies with smaller group sizes, perhaps facilitating
individual recognition. Interestingly, facial color
complexity also increased with the degree of sym-
patry, for example, the occurrence of species living
in an overlapping geographical area, which could
also facilitate species recognition or ingroup/out-
group recognition. Although it has been known
that brightly colored faces can play a role in sexual
selection, there is little quantitative evidence that
complex facial patterns play a role in other aspect of
social communication. Nevertheless, some studies
have suggested that both face and hair color, as well
as hair length, may be involved (De Marco et al.,
2008; Kingdon, 2007; Micheletta, Engelhardt, Mat-
thews, Agil, & Waller, 2013).
The influence of social complexity on com-
munication is not restricted to the size of primates’
behavioral repertoire. Some facial displays, although
similar in appearance, fulfil different functions in
different species. The best known example of this
phenomenon is the silent bared-teeth (SBT) display,
which is widespread in primates. The SBT display
is considered homologous to the human smile
Nonverbal Communication in Primates
(Preuschoft, 1995), and, in primates, its function
ranges from formal subordination, submission, and
appeasement to peaceful intentions and even play-
fulness. This functional variability, however, can-
not always be predicted by phylogeny (Preuschoft,
1995). In some cases, the variability has been shown
to correspond to differences in power asymmetries
between closely related species, for example, gen-
era Cebus and Macaca, which led Preuschoft and
van Hooff (1995) to propose the power asymmetry
hypothesis of motivational emancipation. In species
where the balance of power is asymmetrical, the
predictability of conflict outcomes is high. In rhesus
macaques (Macaca mulatta), for example, high-rank-
ing individuals are rarely challenged, and violent
contests are largely unnecessary if individuals have
a means to clearly signal their subordinate status.
Indeed, in species with asymmetrical power, the
SBT display was found to be emitted unidirection-
ally, displayed by lower to higher ranking individu-
als as an indication of their willing submission or
subordination (Beisner & McCowan, 2013; Flack &
de Waal, 2007). In contrast, in species where the
balance of power is symmetrical, as is the case for
Tonkean macaques (Macaca tonkeana), the predict-
ability of conflict outcomes is low, and individuals
have a greater need to negotiate their outcome. In
these species, the SBT display has been emancipated
from its narrow meaning. Instead of being displayed
unidirectionally in specific social contexts, its mean-
ing can vary depending on the social context, and
this can vary broadly from submissive contexts to
peaceful interactions and even play. Likewise, the
relaxed open-mouth display (homologue of the
human laughter face; Preuschoft, 1995) functions
as a play signal in species characterized by strong
power asymmetries, whereas it is more flexible when
power asymmetries are low. In these latter species,
the SBT and open-mouth displays are often blended
in their use across multiple contexts, conveying an
appeasing rather than a strictly submissive meaning
(Thierry, Demaria, Preuschoft, & Desportes, 1989).
Dynamic and Multimodal Signals
Although there is a long history of excellent etho-
logical work on primate communication (Marler,
1965; van Hooff, 1967), these studies have almost
409
Parr, Micheletta, and Waller
exclusively discussed the appearance of signals at
their peak intensity. However, just like humans, the
real-world expression of most displays is accom-
panied by a high degree of both inter- and intrain-
dividual variability. This is, in part, due to the fact
that visual and vocal displays can vary gradually
and broadly in both their structural form and level
of intensity, alternating rapidly between different
visual configurations or acoustic characteristics, and
even merging into one another to produce unritual-
ized blends. These variations are generally thought
to reflect the internal motivation of the sender
(Preuschoft, 1995) and/or adaptation to the environ-
ment (Ey & Fischer, 2009).
Graded signals are especially common in primate
vocalizations where different variants of the same
call are recognized by listeners and can provide
information regarding the role of the caller dur-
ing social interactions (Slocombe & Zuberbühler,
2005a) or the intensity of the interaction (Slocombe,
Townsend, & Zuberbühler, 2009). Graded signals
are not restricted to the acoustic channel, how-
ever. One example is lipsmacking, a dynamic facial
expression produced by smacking the lips together
at a rhythmic frequency. This expression is com-
monly used during affiliative interactions in many
primate species. In some species of macaques, for
example, researchers have found that the form of
lipsmacking and its effect on receivers’ behavior is
highly variable: The addition or omission of some
of its visual components can affect the outcome
of the social interaction by increasing or reducing
the occurrence of subsequent affiliative contacts
(Micheletta et al., 2013). Such multicomponent,
flexible, and even subtle signals can be extremely
meaningful because different signal compositions
can modify the behavior of the receiver and the out-
come of the interaction.
Another example comes from studies of the play
face, a relaxed open-mouth expression where the
top teeth are typically fully or partially covered by
the upper lip, and the lower teeth are often exposed
as the lower jaw is opened. In gorillas, a unique
form of the play face, referred to as the full play face,
involves exposing both the upper and the lower
teeth, and this form of play face is found more often
during intense bouts of play (Waller & Cherry,
410
2012; see also Figure 16.1). This subtle change in
the appearance of the full play face compared to the
typical play face could have an important role in
coordinating and maintaining play by reducing the
uncertainty of the receiver regarding the intentions
of the sender. Therefore, instead of describing these
displays as discrete, peak-intensity signals, they
might be more usefully described as occurring in a
dynamic range, where examples of displays across
this range can produce dramatically different social
behavioral outcomes.
Primate communication is also often multi-
modal, meaning vocalizations, gestures, and facial
expressions are used in combination with one
another. Thus, the ability to produce, perceive, and
accurately interpret multimodal signals is essential
during social interaction and may have been an
important defining step in the evolution of com-
plex communication systems like that of humans.
Despite its central role in human communication,
the study of multimodal communication has been
relatively neglected by primatologists. In 1999,
Sarah R. Partan and Peter Marler proposed an excit-
ing theoretical framework for the study of multi-
modal signals in animals (see Figure 16.3; see also
Hebets & Papaj, 2005; Munoz & Blumstein, 2012;
Wilson, Dean, & Higham, 2013). According to
this framework, the combination of signals across
multiple modalities offers several possibilities for
variations in the form and function of communica-
tive displays. Individual components of composite
signals can provide either redundant or nonredundant
information, affecting the behavior of receivers dif-
ferently. Redundant signals elicit similar responses
in a receiver when presented alone regardless of
modality; however, if presented together in a multi-
modal form, they can cause either the same response
(equivalence) or an increase or decrease in the inten-
sity of the response (enhancement and antagonism).
Redundant signals can be particularly efficient in
improving the salience of multimodal signals, ensur-
ing that the message reaches the receiver with mini-
mal distortion, that is, different channels back up
each other (Johnstone, 1996). Nonredundant sig-
nals are those that elicit different responses from a
receiver, but when combined in a multimodal form,
they can elicit an increased or decreased response

FIGURE 16.3.  Theoretical framework for the study
of multimodal communication. The upper panel
depicts redundant signals; the lower panel depicts non­
redundant signals. Panels on the left describe possible
responses to two separate components (“a” and “b”)
represented by geometric shapes (the same shape indi-
cates the same qualitative response; different shapes
indicate different responses). The right panels describe
responses to the composite signal (the size of the shape
indicates quantitatively different responses). From
“Communication Goes Multimodal,” by S. Partan and
P. Marler, 1999, Science, 283, p. 1272. Copyright 1999
by the American Association for the Advancement of
Science. Adapted with permission.
(modulation) or the emergence of a totally new
response. Component signals may also not elicit any
response when presented separately but only when
combined with other signals (Rowe, 1999; Rowe
& Guilford, 1996).
Multimodality offers a way of enriching the
communicative repertoire of a species even if that
repertoire is characterized by only a few distinct
signals. In macaques, for example, lipsmacks are
primarily a visual display of affiliation and appease-
ment. However, if accompanied by a soft grunting
vocalization, lipsmacks are more likely to lead to
an affiliative physical contact than when given
alone (Micheletta et al., 2013; Partan, 1998). This
demonstrates an effect of enhancement in that the
grunting vocalization enhanced the response of
receivers to visual displays of lipsmacks alone. A
recent analysis of the contest hoot, a vocalization
used by bonobos in aggressive and playful interac-
tions, revealed how gestures could be used to com-
plement vocalizations and provide additional cues
concerning the nature of the signaler’s intentions
(Genty, Clay, Hobaiter, & Zuberbühler, 2014).
Although the acoustic structure of contest hoots
Nonverbal Communication in Primates
remains similar across contexts, these vocalizations
were more often associated with soft gestures, such
as an arm raise or touch if given in a playful con-
text. In aggressive interactions, contest hoots were
more often combined with rough gestures such as
slapping or hitting. These results provide strong
evidence that analyzing the dynamic, multimodal
nature of communicative behavior provides a much
richer interpretation of the meaning of primate
communication than when signals are studied in
isolation. Bottom-up approaches, such as the FACS,
combined with the different frameworks for the
study of multimodal communication provide a
powerful toolkit for comparative studies aimed at
understanding the evolution of human nonverbal
behavior.
EXPERIMENTAL APPROACHES TO
STUDYING PRIMATE COMMUNICATION
Discrimination of Facial Expressions:
Identification of Major Categories
Experimental evidence shows that primates are
sensitive to facial expressions and can discriminate
among several major categories. As discussed
in the first part of this chapter, facial expressions
contain both component movements—individual
features such as open mouth, raised lips, and fur-
rowed brows—and unique configurations of these
features. Using FACS coding, researchers have iden-
tified movement configurations that represent pro-
totypical forms of facial expressions. The smile, for
example, must contain a retraction of the lips
(in FACS terms this is Action Unit 12, or AU12),
but it can also include a raised upper lip (AU10)
and/or lowered bottom lip (AU16), exposing the
teeth. The lips could be parted (AU25) or not, and
the jaw could be slightly open (AU26) or stretched
wide open (AU27). Moreover, smiles of enjoyment
are said to contain AU6, cheek raiser, which func-
tions to wrinkle the corner of the eyes, referred to
as crows feet in the Duchenne smile. This example
shows that although many different component
movements may be included in a single expression,
the extent to which these individual or combined
movements are necessary for the discrimination and/
411
Parr, Micheletta, and Waller
or recognition of specific facial expressions in pri-
mates, particularly nonhuman primates, is unknown.
To examine the contribution of individual
component movements to the discrimination of
chimpanzee facial expressions, Parr and colleagues
conducted a series of experiments using Chimp-
FACS (Vick et al., 2007). This first began with
researchers coding the movements (AUs) present
in more than 250 photographs of chimpanzee facial
expressions taken during naturalistic behavioral
interactions (Parr, Waller, Vick, & Bard, 2007).
These expressions had previously been categorized
as belonging to one of nine expression types using
published chimpanzee behavioral ethograms (Good-
all, 1968; Parr et al., 2005; van Hooff, 1967, 1970).
These categories included the bared-teeth display,
relaxed open-mouth expression (play face), pant-
hoot, scream, alert face, pout, whimper, neutral,
and an ambiguous category that contained expres-
sions that were either weak in intensity or blended
between several categories and thus could not be
identified definitively. After coding these photo-
graphs, the AU configurations were analyzed using
discriminant functions analyses, which revealed
more than 70% agreement between the ChimpFACS
coded movements and the a priori expression cat-
egories given to the photographs (see Table 16.1).
Categories where the agreement of expression iden-
tity was poor included, as expected, the ambiguous
expressions but also neutral faces, which were coded
to contain some subtle movements. Pouts were also
frequently confused with pant-hoots, which are very
similar in appearance (Parr, Waller, Vick, & Bard,
2007). Taking this one step further, Parr, Waller,
Vick, and Bard (2007) sorted the individual pho-
tographs according to probability that they were
assigned to the correct category. Then, taking only
those expressions for which the probability of agree-
ment was more than 90%, they listed the AUs that
were common to all photographs in each expression
category. This resulted in a set of AUs that repre-
sented the most prototypical configuration for each
expression type (see Table 16.1).
The second phase of these studies involved cre-
ating a set of chimpanzee faces that were posed to
contain each individual component movement as
well as the set of prototypical movements involved
412
in each facial expression category, for example,
those that predicted more than 90% category assign-
ment. These were created using the software Poser
(Smith Micro Software, n.d.). Figure 16.4 shows the
Poser expression prototypes and lists their associ-
ated ChimpFACS AUs as well as an example of each
component movement. Once created, Parr et al.
(2008) tested the extent to which chimpanzees
could discriminate these expression categories using
only their individual component movements. This
was done using a matching-to-sample task where
subjects were presented with an expression proto-
type (sample) and each pair of component move-
ments involved in the formation of that prototype.
For example, Figure 16.4 shows that the bared-teeth
display is composed of AUs 10 + 12 + 16 (AU25 is
common to all expression types and indicates parted
lips). Thus, matching-to-sample trials testing the
bared-teeth display would pair a prototypical bared-
teeth display with three possible combinations
of component movements, for example, AU10 +
AU12, AU10 + AU16, and AU12 + AU16. Results
of this experiment showed that not all component
movements were equally salient for expression dis-
crimination (Parr et al., 2008). Each expression had
at least one movement that was more representative
of a particular expression, and subjects spontane-
ously chose that movement over the other possible
combinations. Moreover, when discriminating
among the prototypes themselves, the chimpan-
zees became confused, making mistakes, when the
two expressions shared not the greatest number of
individual movements but the most prototypical
movements. Therefore, it appears that the discrimi-
nation of facial expressions can be accomplished
by the presence of a single component movement,
as long as that movement is the most salient in the
configuration.
Less is known about the discrimination, or
recognition of facial expressions in monkeys com-
pared to chimpanzees. One of the earliest articles
was by Kanazawa (1996), who trained Japanese
macaques to discriminate among several expres-
sions, for example, bared-teeth displays and threat
faces, each with examples varying in their inten-
sity from peak intensity to neutral. Then, based on
subjects’ discrimination performance, he used a
 TABLE 16.1

Percentage of Agreement for Classifications of Chimpanzee Facial Expressions Using the Chimpanzee Version of the Facial Action Coding System
(ChimpFACS)

Name Photo Bared teeth Play face Pant-hoot Ambiguous Neutral Scream Alert face Pout Whimper
Bared teeth 70.7 [10 + 12 17.1 0 2.4 0 9.8 0 0 0
+ (16) + 25]

Play face 0 87.1 [12 + 25 0 6.5 0 6.5 0 0 0

+ 26 (or 27)]

Pant-hoot 0 0 94.6 [22 + 25 0 0 0 0 5.4 0

+ 26]

Ambiguous 16.7 8.3 20.8 16.7 [N/A] 8.3 8.3 20.8 0 0

Neutral 0 0 0 6.7 40 [none] 0 40 13.3 0

Scream 0 2.9 0 0 0 97.1 [10 + 12 + 0 0 0

16 + 25 + 27]

Alert face 5.6 16.7 22.2 0 5.6 0 50 [16 + 25] 0 0

Pout 0 0 72.4 0 0 0 0 24.1 [22 + 25] 3.4

Whimper 9.1 0 18.2 0 9.1 0 0 0 63.6 [12 + 22

+ 25]

Note. Rows indicate the percentage of agreement between the expression groups identified by the authors and the ChimpFACS coding. In brackets is the prototypical
configuration of action units for each expression category. From “Classifying Chimpanzee Facial Expressions Using Muscle Action,” by L. A. Parr, B. M. Waller,
Nonverbal Communication in Primates

413
S. J. Vick, and K. A. Bard, 2007, Emotion, 7, p. 176. Copyright 2007 by the American Psychological Association.
Parr, Micheletta, and Waller

FIGURE 16.4.  Examples of each facial expression prototype created using Poser software, noting the unique com-
bination of ChimpFACS (Facial Action Coding System) action units contained in each. At the bottom of the figure is
an example of each of the five individual action units needed to create these expression prototypes.

multidimensional scaling (MDS) analysis to create
a matrix showing the basis of the perceived dif-
ferences between the expression types. The MDS
analysis plots each item (expression) along a two-
dimensional axis that can then be interpreted based
on the identity and location of each item. This study
revealed that the two expression types varied con-
tinuously along the horizontal axis, from bared teeth
on one end to threats on the other. Therefore, this
axis was labeled subordinate–dominant, based on
the meaning of each of these expressions. The verti-
cal axis ranged from neutral faces to tense mouth
expressions, and therefore this axis was labeled
neutral–tense.
Each stimulus was also coded by humans for
the presence of specific features, and then cor-
relations were performed between the individual
features shared by the expressions along each
axis. This revealed eye-width, mouth thrust, and
mouth curve to be the prominent features for the
subordinate–dominant axis, and brow raise to be
the predominant feature for the neutral–threat axis.
When discriminating among human facial expres-
sions, the monkeys were able to classify only the
smile and were unable to discriminate anger and dis-
gust faces from sad faces (Kanazawa, 1996). There-
fore, even though perceptual features could enable
the discrimination of these human expressions, the
macaque seemed unable to attend to these features,
perhaps because they lacked biological relevance for
the monkey.
Parr and Heintz (2009) tested the ability of rhe-
sus monkeys to discriminate among conspecific
facial expressions using a matching-to-sample task.
Monkeys easily learned to discriminate between
expression types when the matching pair showed
identical photographs of a facial expression paired
with another expression type as the nonmatching
stimulus, or when the matching pair showed differ-
ent monkeys making the same expression and the
nonmatching image was a neutral portrait of another
(third) monkey. However, subjects had a much
harder time on these latter trials when the non-
matching image showed a third monkey portraying

414
 Nonverbal Communication in Primates

FIGURE 16.5.  Two of the trial types used in Parr and Heintz’s (2009) study. On the left, subjects were required to
match an identical pair of images (bared-teeth display) versus another monkey portraying one of the other expres-
sion categories (open-mouth threat). The right-side image portrays a trial in which subjects performed the worst.
All three images show a different monkey, but in the matching pair, the monkeys depict the same expression
(scream), whereas the monkey in the nonmatching image portrays a different expression type (open-mouth threat).
From “Facial Expression Recognition in Rhesus Monkeys, Macaca mulatta,” by L. A. Parr and M. Heintz, 2009,
Animal Behaviour, 77, pp. 1510–1511. Copyright 2009 by Elsevier. Reprinted with permission.

one of the other expression categories and not a
neutral portrait (see Figure 16.5). Therefore, the
monkeys had trouble discriminating between differ-
ent expression categories when the expressions were
also portrayed by different individuals. Using a simi-
lar MDS procedure as Kanazawa (1996), the sub-
jects’ error patterns were analyzed and displayed in
a two-dimensional matrix. One axis appeared to rep-
resent mouth shape, ranging from mouths stretched
open to mouths that were funnel shaped, whereas
the other axis appeared to represent lip retraction,
ranging from lips fully retracted to relaxed and open
(Parr & Heintz, 2009). These features alone, how-
ever, could not account for the overall performance
of the subjects and suggest that monkeys may not
rely solely on distinctive features to discriminate
their basic expression types.
Although the experiments described earlier
have employed careful controls and presented only
high-quality photographs of conspecific and het-
erospecific facial expressions taken from naturally
occurring contexts, one could still raise concerns
about the nature of the stimuli and the ques-
tions they are able to address. Facial expressions
are highly dynamic stimuli, ranging from subtle
microexpressions to fully charged affective displays
containing movement and often accompanied by an
auditory cue or specific vocalizations. Presenting
facial expressions as static photographs diminishes
their signal value, resulting in poor performance
and perhaps even forcing subjects to attend to other
cues, such as specific features or combinations of
features. Therefore, many researchers have moved
beyond, or avoided altogether, experimental stud-
ies of facial expression recognition using only a
single, visual modality and instead have addressed
the extent to which nonhuman primates recognize
expressions across modalities or from videos con-
taining auditory and visual information, in addition
to dynamic movement.
Discrimination of Facial Expressions:
Multimodal Components
Affective signals such as facial expressions are
also multimodal, combining auditory and visual
elements. Experimental studies show that pri-
mates are sensitive to the multimodal nature of
expressions. Parr (2004) presented three chim-
panzees with the task of matching facial expres-
sion videos—bared-teeth displays, screams, play

415
Parr, Micheletta, and Waller

faces, and pant-hoots—with a still image of that
expression type. Four different types of trials were
included in the study that manipulated the audio
and visual information presented. Here, we just
review two trial types. Congruent multimodal tri-
als included both the category-specific visual and
auditory information in the sample videos, and the
correct choice was to select the photograph of the
expression type that matched the sample. If correct,
the subject received a treat and moved on to the
next trial. However, also presented were incongru-
ent multimodal trials, in which the sample videos
showed the visual track of one expression type but
the audio track of another expression. Two photo-
graphs were subsequently presented as the match-
ing choices, one matched the visual component in
the video, whereas the other matched the audio
component. In these trials, there was no incorrect
answer, so subjects were rewarded for any response
they made, which was interpreted as reflecting the
salience of one modality over the other. The results
of the task can be seen in Table 16.2. In this table,
the percentage correct indicates the proportion of
trials in which subjects chose the facial expression
photograph that matched the visual information
in the video, so 60% would indicate that subjects
matched the visual information 60% of the time and
the auditory information 40% of the time. Subjects
performed very well on the congruent category,
when both audio and visual information matched,
but there were biases in the modality the subjects
preferred to match for the incongruent trials. In gen-
eral, subjects showed a strong preference to match
screams using the visual modality and to match
pant-hoots using the auditory modality. There was
also a slight bias to match the play face using the
auditory component, for example, a distinctive
laughter vocalization. No preference was found for
the bared-teeth display (Parr, 2004).
In 2003, Ghazanfar and Logothetis tested
whether young rhesus monkeys could integrate the
auditory and visual information in two of their most
prolific expressions: the coo and the open mouth
threat. Using a preferential looking paradigm, mon-
keys were presented with side-by-side video screens
showing a conspecific making each of these two
expressions. Each video was 2 s in length and played
in a 1-min loop. During the presentation, the vocal-
ization of only one of the expressions was played on
the computer. If monkeys detected the similarity,

TABLE 16.2

Subjects’ Performance (% Correct) Matching Facial Expressions to Video Clips Showing the Same
Expression Either With the Accompanying Vocalization (Congruent) or With the Vocalization Replaced
With That From Another Expression Category (Incongruent)

Stimulus categories
Subject Expression type Incongruent Congruent
Jarred Scream 61.6 90.0
Pant-hoot 30.2 100.0
Play face 33.4 100.0
Bared teeth 35.0 100.0
Katrina Scream 96.6 90.0
Pant-hoot 3.4 90.0
Play face 33.4 100.0
Bared teeth 43.2 80.0
Scott Scream 86.8 90.0
Pant-hoot 26.4 44.4
Play face 30.4 100.0
Bared teeth 46.6 10.0

Note. From “Perceptual Biases for Multimodal Cues in Chimpanzee (Pan troglodytes) Affect Recognition,” by L. A. Parr,
2004, Animal Cognition, 7, p. 175. Copyright 2004 by Springer. Reprinted with permission.

416

or congruency, between the visual and auditory
components of these expressions, then it was pre-
dicted that they would look longer at the video that
matched the vocalization. As expected, the monkeys
looked longer at the video that corresponded to the
vocalization played in more than 67% of the trials,
significantly greater than the 50% looking time that
was expected by chance (Ghazanfar & Logothetis,
2003). They also showed significantly longer indi-
vidual fixations overall when the videos were con-
gruent than incongruent. Finally, monkeys showed
greater congruency detection for the coo videos,
possibly because the threat videos were aversive and
may have reduced likelihood of sustained looking.
Therefore, monkeys appear to understand the con-
gruency between specific vocalizations and the cor-
responding mouth movements that are made during
these facial expressions. This may be important
because the frequency of mouth movements dur-
ing nonhuman primate communication is the same
frequency as that seen during human speech, thus
providing evidence for an evolutionary relation-
ship between primate facial expressions and human
speech (Ghazanfar, 2013).
Discrimination of Facial Expressions:
Context Dependence/Meaning
As a final topic, there is sufficient evidence that
nonverbal signals can acquire subtle, and sometimes
not so subtle, changes in meaning when performed
in different social contexts, including when they
are used by different individuals, for example, age
and rank. Moreover, there is also evidence that, like
in humans, the interpretation of signals can vary
according to the mood state of the receiver. Mon-
keys are able to navigate these context-dependent
and individually dependent variations in a signal’s
meaning, although there has been little controlled
experimental study to provide evidence of how this
is achieved. Several detailed behavioral studies of
primates have highlighted some of these interesting
variations in signal meaning.
Flack and de Waal (2007) performed one of the
first studies showing that context could influence
the potential meaning of facial signals. They stud-
ied pig-tailed macaques and found that bared-teeth
displays were sometimes used by individuals in
Nonverbal Communication in Primates
contexts that did not appear to warrant them, for
example, peaceful contexts in which there was no
immediate need to overtly assert subordinate status
to a dominant individual. This was in direct contrast
to bared-teeth displays used in situations of conflict,
where performance of the display has been shown
to reduce immediate aggression toward the signaler
(Flack & de Waal, 2007). These authors analyzed
bared-teeth displays used in peaceful contexts and
found that aggression was less frequent and that
grooming was more frequent among dyads of mon-
keys in which the bared-teeth display was given
outside of conflict contexts, that is, during peaceful
contexts. The authors suggested that the use of this
signal in a new, nonconflict context acted to rein-
force the dominant–subordinate relationship and
function as a type of insurance against aggression
between the pair if conflict was to arise.
Rhesus macaques, unlike many of the other
macaque species that have been discussed in this
chapter, are the most aggressive macaque species.
The rhesus monkey has an extremely strict matri-
lineal dominance hierarchy, and the bared-teeth dis-
play in this species has been described as a “formal”
signal of submission because it is used unidirection-
ally, given by lower ranking individuals to higher
ranking individuals but not in the other direction
(de Waal & Luttrell, 1985). In this way, the bared-
teeth display signals recognition of the dominance
hierarchy and communicates submission toward
dominant individuals. In a recent study, Beisner
and McCowan (2013) performed a detailed assess-
ment of the use and function of bared-teeth displays
in rhesus macaques and identified several new and
exciting findings. First, these authors identified two
usages of the bared-teeth display in nonconflict con-
ditions. In one, the signal is given, and the sender
retreats or withdraws from the target of the display,
that is, the “bared-teeth leave,” whereas in the other
condition, the signal is given, but the sender does
not retreat, that is, “bared-teeth stay.” In detailed
analyses of the subsequent behavior of the senders
and receivers of these signals, Beisner and McCowan
found higher rates of grooming and lower rates of
aggression among individuals who used the bared-
teeth stay signal, suggesting that when used in a
nonconflict, peaceful context, this formal signal of
417
Parr, Micheletta, and Waller
subordination may function to communicate inter-
est in maintaining a peaceful, positive relationship.
These studies suggest that the bared-teeth display,
one of the most conserved expressions among
primates, not only shows variability in when it is
produced but that this variability appears to be asso-
ciated with interest in maintaining positive long-
term relationships, not just the immediate reduction
of aggression from dominant individuals in situa-
tions of imminent conflict.
Finally, Bethell, Holmes, MacLarnon, and
Semple (2012) performed a selective looking study
with rhesus monkeys to determine the effects of
emotion, or mood state, on attention to social
stimuli, for example, negative and neutral facial
expressions. Mood state of the animal was con-
trolled by testing monkeys directly after a stress-
ful/aversive event, such as the routine annual
veterinary health check, or after a positive event,
such as receiving environmental enrichment. Ini-
tially, the attentional stance of the subjects was the
same, regardless of their mood state. They looked
faster and longer at the negative/aversive facial
expression. This is a typical response reflecting
the negativity bias that many animals, including
humans, show toward aversive images (Ito, Larsen,
Smith, & Cacioppo, 1998). However, in the aver-
sive/veterinary condition, subjects followed the
initial vigilance with a period of sustained aver-
sion, avoiding looking at the negative expression.
This was in contrast to the positive/enrichment
condition in which the monkeys maintained their
vigilance toward the negative expression. These
findings are important as they relate to social vigi-
lance in a group context. If threatened, or in aver-
sive contexts such as conflict, an individual may
actively avoid any interactions with others or may
interpret expressions as more negative, facilitating
social withdrawal, whereas in positive, peaceful
contexts, individuals may be more motivated to
seek out and/or engage in social interactions.
CONCLUSIONS
Comparative research of nonverbal communication
in primates can inform us on the basic principles
underlying its evolution in humans. However, to
418
be accurate, this approach needs to incorporate
multiple species, not just rely on data from a few
select species, such as rhesus macaques and chim-
panzees. Ideally, these species would be tested
using the same or highly comparable conditions,
social stimuli, and general experimental methods.
Nonverbal communication is a fundamental aspect
of primate social behavior and can be complex.
Primates and the societies in which they live are
extremely diverse, and researchers can take advan-
tage of this diversity to explore the relationship
between the ecology, social style, and communi-
cation system of a species. In particular, such an
integrative approach can further the understanding
of which selective pressures act on communication
complexity. Additionally, nonverbal communi-
cation research is highly biased toward study-
ing a select few sensory modalities, for example,
auditory/vocalizations, using a restricted set of
methodological tools, for example, playback experi-
ments (Slocombe et al., 2011). With data heavily
influenced by these limitations, our conclusions
regarding species differences are likewise limited.
Primate communication involves multiple channels
of sensory information that are integrated across
time and that occur within a specific social and
ecological framework. As such, communication
is essentially multimodal and contextually deter-
mined. Multimodal signals can lead to qualitative as
well as quantitative changes in receivers’ responses,
and the decision of when and how these signals
are produced remains unknown. Although some
researchers have explored how nonhuman primates
perceive and process these multimodal signals
(Ghazanfar & Logothetis, 2003; Parr, 2004), more
research is needed to better understand their char-
acteristics and, importantly, their social function.
Designing experiments to reveal the intricacies of
social communication is challenging, but techno-
logical progress and researchers’ creativity will lead
to novel and interesting findings (Partan, Larco, &
Owens, 2009).
References
Andrew, R. J. (1963). The origin and evolution of the calls
and facial expressions of the primates. Behaviour, 20,
1–107. http://dx.doi.org/10.1163/156853963X00220

Beisner, B. A., & McCowan, B. (2013). Signaling context
modulates social function of silent bared-teeth displays
in rhesus macaques (Macaca mulatta). American
Journal of Primatology, 76, 111–121. http://dx.doi.org/
10.1002/ajp.22214
Bertrand, M. (1969). The behavioral repertoire of the
stumptail macaque: A descriptive and comparative
study. Bibliotheca Primatologica, 11, 1–273.
Bethell, E. J., Holmes, A., MacLarnon, A., & Semple,
S. (2012). Evidence that emotion mediates social
attention in rhesus macaques. PLoS ONE, 7, e44387.
http://dx.doi.org/10.1371/journal.pone.0044387
Bolwig, N. (1964). Facial expression in primates with
remarks on a parallel development in certain
carnivores (a preliminary report on work in
progress). Behaviour, 22, 167–192. http://dx.doi.org/
10.1163/156853964X00012
Bout, N., & Thierry, B. (2005). Peaceful meaning for the
silent bared-teeth displays of mandrills. International
Journal of Primatology, 26, 1215–1228. http://dx.doi.
org/10.1007/s10764-005-8850-1
Burrows, A. M. (2008). The facial expression musculature
in primates and its evolutionary significance.
BioEssays, 30, 212–225. http://dx.doi.org/10.1002/
bies.20719
Caeiro, C., Waller, B. M., Zimmermann, E., Burrows,
A. M., & Davila-Ross, M. (2013). OrangFACS: A
muscle-based facial movement coding system for
orangutans (Pongo spp.). International Journal of
Primatology, 34, 115–129. http://dx.doi.org/10.1007/
s10764-012-9652-x
Darwin, C. (1872). The expression of emotions in man and
animals. http://dx.doi.org/10.1037/10001-000
Davila Ross, M., Owren, M. J., & Zimmermann, E. (2009).
Reconstructing the evolution of laughter in great apes
and humans. Current Biology, 19, 1106–1111. http://
dx.doi.org/10.1016/j.cub.2009.05.028
De Marco, A., Petit, O., & Visalberghi, E. (2008). The
repertoire and social function of facial displays in Cebus
capucinus. International Journal of Primatology, 29,
469–486. http://dx.doi.org/10.1007/s10764-007-9174-0
Dennett, D. C. (1987). The intentional stance. Cambridge,
MA: MIT Press.
de Waal, F. B. M. (1982). Chimpanzee politics. Baltimore,
MD: Johns Hopkins UniversityPress.
de Waal, F. B. M., & Luttrell, L. M. (1985). The formal
hierarchy of rhesus macaques: An investigation
of the bared-teeth display. American Journal of
Primatology, 9, 73–85. http://dx.doi.org/10.1002/
ajp.1350090202
Dobson, S. D. (2009a). Allometry of facial mobility in
anthropoid primates: Implications for the evolution
of facial expression. American Journal of Physical
Nonverbal Communication in Primates
Anthropology, 138, 70–81. http://dx.doi.org/10.1002/
ajpa.20902
Dobson, S. D. (2009b). Socioecological correlates of facial
mobility in nonhuman anthropoids. American Journal
of Physical Anthropology, 139, 413–420. http://dx.doi.
org/10.1002/ajpa.21007
Dobson, S. D. (2012). Coevolution of facial expression
and social tolerance in macaques. American Journal of
Primatology, 74, 229–235. http://dx.doi.org/10.1002/
ajp.21991
Dunbar, R. I. M. (1993). Coevolution of neocortex size,
group size and language in humans. Behavioral and
Brain Sciences, 16, 681–694. http://dx.doi.org/10.1017/
S0140525X00032325
Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial
Action Coding System. Salt Lake City, UT: Research
Nexus.
Ey, E., & Fischer, J. (2009). The “acoustic adaptation
hypothesis”—A review of the evidence from birds,
anurans and mammals. Bioacoustics, 19, 21–48.
Ey, E., Pfefferle, D., & Fischer, J. (2007). Do age- and
sex-related variations reliably reflect body size
in non-human primate vocalizations? A review.
Primates, 48, 253–267. http://dx.doi.org/10.1007/
s10329-006-0033-y
Flack, J. C., & de Waal, F. (2007). Context modulates
signal meaning in primate communication.
Proceedings of the National Academy of Sciences,
USA, 104, 1581–1586. http://dx.doi.org/10.1073/
pnas.0603565104
Freeberg, T. M., Dunbar, R. I. M., & Ord, T. J. (2012).
Social complexity as a proximate and ultimate
factor in communicative complexity. Philosophical
Transactions of the Royal Society of London B:
Biological Sciences, 367, 1785–1801. http://dx.doi.
org/10.1098/rstb.2011.0213
Fridlund, A. J. (1994). Human facial expression—An
evolutionary view. London, England: Academic Press.
Genty, E., Breuer, T., Hobaiter, C., & Byrne, R. W. (2009).
Gestural communication of the gorilla (Gorilla gorilla):
Repertoire, intentionality and possible origins. Animal
Cognition, 12, 527–546. http://dx.doi.org/10.1007/
s10071-009-0213-4
Genty, E., Clay, Z., Hobaiter, C., & Zuberbühler, K.
(2014). Multi-modal use of a socially directed call in
bonobos. PLoS ONE, 9, e84738. http://dx.doi.org/
10.1371/journal.pone.0084738
Ghazanfar, A. A. (2013). Multisensory vocal communication
in primates and the evolution of rhythmic speech.
Behavioral Ecology and Sociobiology, 67, 1441–1448.
http://dx.doi.org/10.1007/s00265-013-1491-z
Ghazanfar, A. A., Chandrasekaran, C., & Morrill, R. J.
(2010). Dynamic, rhythmic facial expressions and
the superior temporal sulcus of macaque monkeys:
419
Parr, Micheletta, and Waller
Implications for the evolution of audiovisual speech.
European Journal of Neuroscience, 31, 1807–1817.
http://dx.doi.org/10.1111/j.1460-9568.2010.07209.x
Ghazanfar, A. A., & Logothetis, N. K. (2003). Neuro­
perception: Facial expressions linked to monkey calls.
Nature, 423, 937–938. http://dx.doi.org/10.1038/
423937a
Goodall, J. (1968). A preliminary report on expressive
movements and communication in the Gombe
Stream chimpanzees. In P. C. Jay (Ed.), Primates:
Studies in adaptation and variability (pp. 313–519).
New York, NY: Holt, Rinehart & Winston.
Guilford, T., & Dawkins, M. S. (1991). Receiver
psychology and the evolution of animal signals.
Animal Behaviour, 42, 1–14. http://dx.doi.org/
10.1016/S0003-3472(05)80600-1
Hebets, E. A., & Papaj, D. R. (2005). Complex signal
function: Developing a framework of testable
hypotheses. Behavioral Ecology and Sociobiology, 57,
197–214. http://dx.doi.org/10.1007/s00265-004-
0865-7
Hobaiter, C., & Byrne, R. W. (2014). The meanings
of chimpanzee gestures. Current Biology, 24,
1596–1600. http://dx.doi.org/10.1016/j.cub.
2014.05.066
Humphrey, N. K. (1976). The social function of intellect.
In P. P. G. Bateson & R. A. Hinde (Eds.), Growing
points in ethology (pp. 303–317). Cambridge,
England: Cambridge University Press.
Ito, T. A., Larsen, J. T., Smith, N. K., & Cacioppo, J. T.
(1998). Negative information weighs more heavily
on the brain: The negativity bias in evaluative
categorizations. Journal of Personality and Social
Psychology, 75, 887–900. http://dx.doi.org/10.1037/
0022-3514.75.4.887
Johnstone, R. A. (1996). Multiple displays in animal
communication: “Backup signals” and “multiple
messages.” Philosophical Transactions of the Royal
Society of London B: Biological Sciences, 351,
329–338. http://dx.doi.org/10.1098/rstb.1996.0026
Jolly, A. (1966). Lemur social behavior and primate
intelligence. Science, 153, 501–506. http://dx.doi.org/
10.1126/science.153.3735.501
Kanazawa, S. (1996). Recognition of facial expressions
in a Japanese monkey (Macaca fuscata) and humans
(Homo sapiens). Primates, 37, 25–38. http://dx.doi.
org/10.1007/BF02382917
Kiltie, R. A. (2000). Scaling of visual acuity with body
size in mammals and birds. Functional Ecology, 14,
226–234. http://dx.doi.org/10.1046/j.1365-2435.
2000.00404.x
Kingdon, J. (2007). Primate visual signals in noisy
environments. Folia Primatologica, 78, 389–404.
http://dx.doi.org/10.1159/000105151
420
Krebs, J. R., Davies, N. B., & Parr, J. (1993). An
introduction to behavioural ecology (3rd ed.).
Cambridge, MA: Blackwell Scientific.
Laidre, M. E., & Yorzinski, J. L. (2005). The silent
bared-teeth face and the crest-raise of the mandrill
(Mandrillus sphinx): A contextual analysis of signal
function. Ethology, 111, 143–157. http://dx.doi.org/
10.1111/j.1439-0310.2004.01051.x
Leavens, D. A., Russell, J. L., & Hopkins, W. D. (2005).
Intentionality as measured in the persistence and
elaboration of communication by chimpanzees (Pan
troglodytes). Child Development, 76, 291–306. http://
dx.doi.org/10.1111/j.1467-8624.2005.00845.x
Liebal, K., Mueller, C., & Pika, S. (Eds.). (2007). Gestural
communication in nonhuman and human primates.
http://dx.doi.org/10.1075/bct.10
Liebal, K., Waller, B. M., Burrows, A. M., & Slocombe,
K. E. (2014). Primate communication: A multimodal
approach. Cambridge, England: Cambridge
University Press.
Maestripieri, D. (1996). Social communication among
captive stump-tailed macaques (Macaca arctoides).
International Journal of Primatology, 17, 785–802.
Maestripieri, D. (1999). Primate social organization,
gestural repertoire size, and communication
dynamics: A comparative study of macaques. In B. J.
King (Ed.), The origins of language: What nonhuman
primates can tell us (pp. 55–78). Santa Fe, NM:
School of American Research Press.
Marler, P. (1965). Communication in monkeys and apes.
In I. DeVore (Ed.), Primate behavior: Field studies
of monkeys and apes (pp. 544–584). New York, NY:
Holt, Rinehart & Winston.
McComb, K., & Semple, S. (2005). Coevolution of vocal
communication and sociality in primates. Biology
Letters, 1, 381–385. http://dx.doi.org/10.1098/
rsbl.2005.0366
Mehu, M., Grammer, K., & Dunbar, R. I. M. (2007).
Smiles when sharing. Evolution and Human
Behavior, 28, 415–422. http://dx.doi.org/10.1016/
j.evolhumbehav.2007.05.010
Micheletta, J., Engelhardt, A., Matthews, L., Agil,
M., & Waller, B. M. (2013). Multicomponent
and multimodal lipsmacking in crested macaques
(Macaca nigra). American Journal of Primatology, 75,
763–773. http://dx.doi.org/10.1002/ajp.22105
Munoz, N. E., & Blumstein, D. T. (2012). Multisensory
perception in uncertain environments. Behavioral
Ecology, 23, 457–462. http://dx.doi.org/10.1093/
beheco/arr220
Palagi, E., & Norscia, I. (2009). Multimodal signaling in
wild Lemur catta: Economic design and territorial
function of urine marking. American Journal of
Physical Anthropology, 139, 182–192. http://dx.doi.
org/10.1002/ajpa.20971

Parr, L. A. (2004). Perceptual biases for multimodal cues
in chimpanzee (Pan troglodytes) affect recognition.
Animal Cognition, 7, 171–178. http://dx.doi.org/10.1007/
s10071-004-0207-1
Parr, L. A., Cohen, M., & de Waal, F. B. M. (2005). The
influence of social context on the use of blended
and graded facial displays in chimpanzees (Pan
troglodytes). International Journal of Primatology, 26,
73–103. http://dx.doi.org/10.1007/s10764-005-0724-z
Parr, L. A., & Heintz, M. (2009). Facial expression
recognition in rhesus monkeys, Macaca mulatta.
Animal Behaviour, 77, 1507–1513. http://dx.doi.org/
10.1016/j.anbehav.2009.02.024
Parr, L. A., & Waller, B. M. (2007). The evolution of
human emotion. In T. M. Preuss & J. H. Kaas (Eds.),
Evolution of nervous systems (Vol. 4, pp. 447–472).
http://dx.doi.org/10.1016/B0-12-370878-8/00045-8
Parr, L. A., Waller, B. M., Burrows, A. M., Gothard, K. M.,
& Vick, S. J. (2010). Brief communication: MaqFACS:
A muscle-based facial movement coding system for
the rhesus macaque. American Journal of Physical
Anthropology, 143, 625–630. http://dx.doi.org/
10.1002/ajpa.21401
Parr, L. A., Waller, B. M., & Heintz, M. (2008). Facial
expression categorization by chimpanzees using
standardized stimuli. Emotion, 8, 216–231. http://
dx.doi.org/10.1037/1528-3542.8.2.216
Parr, L. A., Waller, B. M., & Vick, S. J. (2007). New
developments in understanding emotional facial
signals in chimpanzees. Current Directions in
Psychological Science, 16, 117–122. http://dx.doi.
org/10.1111/j.1467-8721.2007.00487.x
Parr, L. A., Waller, B. M., Vick, S. J., & Bard, K. A.
(2007). Classifying chimpanzee facial expressions
using muscle action. Emotion, 7, 172–181. http://
dx.doi.org/10.1037/1528-3542.7.1.172
Partan, S., & Marler, P. (1999). Communication goes
multimodal. Science, 283, 1272–1273. http://dx.doi.
org/10.1126/science.283.5406.1272
Partan, S. R. (1998). Multimodal communication: The
integration of visual and vocal signals by rhesus
macaques (Unpublished doctoral dissertation).
University of California, Davis.
Partan, S. R., Larco, C. P., & Owens, M. J. (2009).
Wild tree squirrels respond with multisensory
enhancement to conspecific robot alarm behaviour.
Animal Behaviour, 77, 1127–1135. http://dx.doi.org/
10.1016/j.anbehav.2008.12.029
Pellis, S. M., & Pellis, V. C. (1996). On knowing it’s
only play: The role of play signals in play fighting.
Aggression and Violent Behavior, 1, 249–268. http://
dx.doi.org/10.1016/1359-1789(95)00016-X
Pellis, S. M., Pellis, V. C., & Thierry, B. (1997). The use
of the bared teeth display as a play signal in Tonkean
Nonverbal Communication in Primates
macaques (Macaca tonkeana). In M. Taborsky &
B. Taborsky (Eds.), Advances in ethology: Contributions
to the XXV International Ethological Conference:
Vienna, Austria, 20–27 August 1997 (Vol. 32). Berlin,
Germany: Blackwell Wissenschafts-Verlag.
Pika, S., & Mitani, J. (2006). Referential gestural
communication in wild chimpanzees (Pan
troglodytes). Current Biology, 16, R191–R192. http://
dx.doi.org/10.1016/j.cub.2006.02.037
Preuschoft, S. (1992). “Laughter” and “smiling” in
Barbary macaques (Macaca sylvanus). Ethology, 91,
220–236. http://dx.doi.org/10.1111/j.1439-0310.
1992.tb00864.x
Preuschoft, S. (1995). “Laughter” and “smiling” in
macaques: An evolutionary perspective. Utrecht,
the Netherlands: University of Utrecht, Faculteit
Biologie.
Preuschoft, S., & van Hooff, J. A. (1995). Homologizing
primate facial displays: A critical review of methods.
Folia Primatologica, 65, 121–137. http://dx.doi.org/
10.1159/000156878
Rendall, D., Owren, M. J., & Ryan, M. J. (2009). What do
animal signals mean? Animal Behaviour, 78, 233–240.
http://dx.doi.org/10.1016/j.anbehav.2009.06.007
Rowe, C. (1999). One signal or two? Science, 284, 741.
http://dx.doi.org/10.1126/science.284.5415.741h
Rowe, C., & Guilford, T. (1996). Hidden color aversions
in domestic chicks triggered by pyrazine odors of
insect warning displays. Nature, 383, 520–522. http://
dx.doi.org/10.1038/383520a0
Samson, A. V. M., & Waller, B. M. (2010). Not growling
but smiling: New interpretations of the bared-teeth
motif in the pre-Columbian Caribbean. Current
Anthropology, 51, 425–433. http://dx.doi.org/10.1086/
651090
Santana, S. E., Alfaro, J. L., & Alfaro, M. E. (2012).
Adaptive evolution of facial colour patterns in
Neotropical primates. Proceedings of the Royal Society
B: Biological Sciences, 279, 2204–2211. http://dx.doi.
org/10.1098/rspb.2011.2326
Santana, S. E., Alfaro, J. L., Noonan, A., & Alfaro, M. E.
(2013). Adaptive response to sociality and ecology
drives the diversification of facial colour patterns in
catarrhines. Nature Communications, 4, 2765. http://
dx.doi.org/10.1038/ncomms3765
Scarantino, A. (2010). Animal communication between
information and influence. Animal Behaviour, 79,
e1–e5. http://dx.doi.org/10.1016/j.anbehav.2010.
03.005
Schel, A. M., Townsend, S. W., Machanda, Z.,
Zuberbühler, K., & Slocombe, K. E. (2013).
Chimpanzee alarm call production meets key criteria
for intentionality. PLoS ONE, 8, e76674. http://
dx.doi.org/10.1371/journal.pone.0076674
421
Parr, Micheletta, and Waller
Scordato, E. S., Dubay, G. R., & Drea, C. M. (2007).
Chemical composition of scent marks in the
ringtailed lemur (Lemur catta): Glandular
differences, individual signatures, and seasonal
variation. Chemical Senses, 32, 493–504. http://
dx.doi.org/10.1093/chemse/bjm018
Seyfarth, R. M., Cheney, D. L., Bergman, T., Fischer, J.,
Zuberbuhler, K., & Hammerschmidt, K. (2010).
The central importance of information in studies of
animal communication. Animal Behaviour, 80, 3–8.
http://dx.doi.org/10.1016/j.anbehav.2010.04.012
Seyfarth, R. M., Cheney, D. L., & Marler, P. (1980).
Monkey responses to three different alarm calls:
Evidence of predator classification and semantic
communication. Science, 210, 801–803. http://dx.doi.
org/10.1126/science.7433999
Sherwood, C. C. (2005). Comparative anatomy of the
facial motor nucleus in mammals, with an analysis of
neuron numbers in primates. Anatomical Record Part
A: Discoveries in Molecular, Cellular, and Evolutionary
Biology, 287, 1067–1079. http://dx.doi.org/10.1002/
ar.a.20259
Slocombe, K. E., Townsend, S. W., & Zuberbühler, K.
(2009). Wild chimpanzees (Pan troglodytes
schweinfurthii) distinguish between different scream
types: Evidence from a playback study. Animal
Cognition, 12, 441–449. http://dx.doi.org/10.1007/
s10071-008-0204-x
Slocombe, K. E., Waller, B. M., & Liebal, K. (2011).
The language void: The need for multimodality
in primate communication research. Animal
Behaviour, 81, 919–924. http://dx.doi.org/10.1016/
j.anbehav.2011.02.002
Slocombe, K. E., & Zuberbühler, K. (2005a). Agonistic
screams in wild chimpanzees (Pan troglodytes
schweinfurthii) vary as a function of social role.
Journal of Comparative Psychology, 119, 67–77. http://
dx.doi.org/10.1037/0735-7036.119.1.67
Slocombe, K. E., & Zuberbühler, K. (2005b).
Functionally referential communication in a
chimpanzee. Current Biology, 15, 1779–1784. http://
dx.doi.org/10.1016/j.cub.2005.08.068
Smith Micro Software. (n.d.). Poser [Computer software].
Retrieved from http://poser.smithmicro.com
Thierry, B., Demaria, C., Preuschoft, S., & Desportes,
C. (1989). Structural convergence between silent
bared-teeth display and relaxed open-mouth display
in the Tonkean macaque (Macaca tonkeana). Folia
Primatologica, 52, 178–184. http://dx.doi.org/
10.1159/000156396
Tomasello, M. (2008). Origins of human communication.
Cambridge, MA: MIT Press.
van Hooff, J. A. R. A. M. (1967). The facial displays of the
Catarrhine monkeys and apes. In D. Morris (Ed.),
Primate ethology (pp. 7–68). Chicago, IL: Aldine.
422
van Hooff, J. A. R. A. M. (1970). A component analysis
of the structure of the social behaviour of a semi-
captive chimpanzee group. Experientia, 26, 549–550.
van Hooff, J. A. R. A. M. (1972). A comparative
approach to the phylogeny of laughter and smiling.
In R. A. Hinde (Ed.), Non-verbal communication
(pp. 209–241). Cambridge, England: Cambridge
University Press.
Vick, S. J., Waller, B. M., Parr, L. A., Smith Pasqualini, M. C.,
& Bard, K. A. (2007). A cross-species comparison
of facial morphology and movement in humans and
chimpanzees using the Facial Action Coding System
(FACS). Journal of Nonverbal Behavior, 31, 1–20.
http://dx.doi.org/10.1007/s10919-006-0017-z
Waller, B. M., & Cherry, L. (2012). Facilitating play
through communication: Significance of teeth
exposure in the gorilla play face. American Journal of
Primatology, 74, 157–164. http://dx.doi.org/10.1002/
ajp.21018
Waller, B. M., & Dunbar, R. I. M. (2005). Differential
behavioural effects of silent bared teeth display and
relaxed open mouth display in chimpanzees (Pan
troglodytes). Ethology, 111, 129–142. http://dx.doi.
org/10.1111/j.1439-0310.2004.01045.x
Waller, B. M., Lembeck, M., Kuchenbuch, P., Burrows,
A. M., & Liebal, K. (2012). GibbonFACS: A muscle-
based facial movement coding system for hylobatids.
International Journal of Primatology, 33, 809–821.
http://dx.doi.org/10.1007/s10764-012-9611-6
Waller, B. M., Misch, A., Whitehouse, J., & Herrmann, E.
(2014). Children, but not chimpanzees, have facial
correlates of determination. Biology Letters,
10, 20130974. http://dx.doi.org/10.1098/
rsbl.2013.0974
Waller, B. M., Parr, L. A., Gothard, K. M., Burrows, A. M.,
& Fuglevand, A. J. (2008). Mapping the contribution
of single muscles to facial movements in the rhesus
macaque. Physiology and Behavior, 95, 93–100. http://
dx.doi.org/10.1016/j.physbeh.2008.05.002
Waller, B. M., Peirce, K., Caeiro, C. C., Scheider, L.,
Burrows, A. M., McCune, S., & Kaminski, J. (2013).
Paedomorphic facial expressions give dogs a selective
advantage. PLoS ONE, 8, e82686. http://dx.doi.
org/10.1371/journal.pone.0082686
Waller, B. M., Vick, S. J., Parr, L. A., Bard, K. A.,
Pasqualini, M. C. S., Gothard, K. M., & Fuglevand, A. J.
(2006). Intramuscular electrical stimulation of facial
muscles in humans and chimpanzees: Duchenne
revisited and extended. Emotion, 6, 367–382. http://
dx.doi.org/10.1037/1528-3542.6.3.367
Wilson, A. J., Dean, M., & Higham, J. P. (2013). A game
theoretic approach to multimodal communication.
Behavioral Ecology and Sociobiology, 67, 1399–1415.
http://dx.doi.org/10.1007/s00265-013-1589-3
 Part IV

METHODOLOGY
 Chapter 17
Measuring the Dynamic Stream
of Display: Spontaneous and
Intentional Facial Expression
and Communication
Ross Buck and Michael Miller
The recent past has witnessed the development
of a number of techniques for the analysis of the
stream of emotional display across time, including
facial expressions, body movements and postures,
and vocalic analyses. Such techniques can enable
the objective and efficient documentation of the
dynamic stream of display, and they are of immense
value, permitting the analysis of the detailed
structure of “body language” as never before.
This chapter focuses specifically on facial expres-
sion, which is of particular importance displaying
at personal and close social distances (Buck &
Miller, 2014). We review a number of methods to
code and analyze facial expression—including the
Maximally Discriminative Facial Movement Cod-
ing System (MAX; Izard, 1979), the Facial Action
Coding System (FACS; Ekman & Friesen, 1978),
facial electromyography (EMG; van Boxtel, 2010),
and structural analysis—and we relate these to the
development of new techniques of automatic facial
analysis. We then examine these in terms of four
issues in the field that are thus far unresolved. One
issue involves whether the techniques are category-
based, focused on the expression of specified emo-
tions by facial muscle movements, or bottom-up,
allowing response categories to emerge from the
data. A second issue is the fact that the focus of
facial measurement and analysis techniques is often
on the behavior of the individual responder, or
sender (e.g., the sender’s facial display of emotions
across time). The communicative aspect of the
display—whether it can be easily, accurately, and
effectively “picked up” and responded to by human
http://dx.doi.org/10.1037/14669-017
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
interaction partners—is generally not considered.
One of the aims of this chapter therefore is to distin-
guish emotional display behavior on one hand, and
emotional communication on the other. A third issue
involves the facial data sets used to test measure-
ment techniques, including automatic analysis
techniques: whether they themselves are based on
the same emotion categories used to construct the
measurement technique. A final and particularly
important issue concerns a fundamental distinction
in understanding facial expression: the distinction
between nonvoluntary and spontaneous emotion
displays versus voluntarily controlled and managed
displays. Many techniques for the analysis of facial
expression have employed intentionally posed and
static expressions; and the emerging possibility for
the analysis of spontaneous, dynamic, naturalistic,
and ecologically based expression is a major focus of
this chapter.
After reviewing these issues, we describe the
measurement of dynamic and spontaneous facial
communication—and how such measurement can
be coordinated with other facial measures, includ-
ing automatic analyses. Specifically, we consider the
slide-viewing technique (SVT) for eliciting dynamic
and spontaneous facial behaviors, and the associ-
ated Communication of Affect Receiving Ability Test
(CARAT; Buck, 1976) employing SVT sequences as a
basis for a database of spontaneous, naturalistic, and
ecologically valid facial expression stimuli that can
be used to evaluate automatic analysis techniques.
An additional and related aim in this chapter
is to review a segmentation technique that offers
425
Buck and Miller
a powerful complement to the analysis of facial
expression. In segmentation, human observers or
receivers indicate when important events occur in
the stream of expression. Segmentation brings the
human receiver’s perceptions into the picture, so
that—unlike automatic analysis techniques—the
measure reflects both the sender’s display and the
receiver’s ability to process and understand that dis-
play. By combining the segmentation technique with
the objective analysis of the sender’s display behav-
ior yielded by facial expression analyses, it should
be possible to address powerfully a variety of impor-
tant issues in emotional expression and emotional
communication.
FACIAL DISPLAY MEASUREMENT: MAJOR
APPROACHES AND ISSUES
A number of methods to code and measure facial
expression have been developed over the years, and
in recent years these have been greatly enhanced by
techniques of automated analysis. However, several
basic issues remain unresolved in this literature.
Approaches to Coding Facial Expressions
Silvan S. Tomkins’s tradition.  Tomkins’s (1962,
1963, 1991) theory had a major influence on the
study of facial expressions, a central tenet being that
certain primary affects are associated with innate and
universal patterns of facial responses. The notion
that there are innate patterns of facial expression
universally associated with specific affects was
grounded in the arguments of Charles Darwin,
particularly in The Expression of the Emotions in Man
and Animals (Darwin, 1872/1998; see also Chapter
4, this handbook). Tomkins’s theory put facial
expression research on a more secure conceptual
basis by specifying the display behaviors associ-
ated with specific emotions. Positive affects include
interest/excitement (eyebrows down and stare fixed
or tracking an object), enjoyment/joy (smiling), and
surprise/startle (eyebrows raised, eyes blinking);
negative affects are distress/anguish (crying), fear/
terror (eyes frozen open in fixed stare or averted,
pale cold skin, sweating and trembling, erect hair),
anger/rage (frown, clenched jaw, red face), contempt
426
(sneer), disgust (lower lip protruded), and shame/
humiliation (eyes and head lowered). Secondary
affects represent a combination of primary affects,
and the expressions associated with them represent
combinations of those associated with the primary
affects: Thus, smugness may be a combination of
joy and anger—your worst enemy is run over by
a truck—and may be associated with a combined
smile and frown. Tomkins’s approach set the stage
for the identification of meaningful units of facial
expression, which would, in turn, enable the study
of their timing and organization.
Tomkins and McCarter (1964) found that facial
expressions simulating affect posed by actors pre-
sented in still photographs could be judged by
trained observers with a high degree of accuracy.
Paul Ekman and Carroll E. Izard, who were students
of Tomkins, each carried out cross-cultural studies
involving the judgment of such posed photographs,
and each developed a conceptualization of emotion
based on Tomkins’s general theory and a system for
manually coding facial expressions. Izard’s systems
include the MAX (Izard, 1979) and the System for
Identifying Affect Expressions by Holistic Judg-
ments (AFFEX; Izard, Dougherty, & Hembree,
1983); Ekman’s systems include the Facial Action
Scoring Technique (FAST; Ekman, Friesen, & Tom-
kins, 1971), FACS (Ekman & Friesen, 1978), and
Emotion FACS (EMFACS; Ekman & Friesen, 1982).
Izard’s MAX and AFFEX.  Izard and Dougherty
(1982) described two complementary systems for
measuring facial expression. The MAX is a system
designed to identify primary emotions and blends
based on microanalytic coding of movements
in three areas of the face. Thirteen “appearance
change” units of muscle configuration changes
are coded in the mouth–lips region, eight in the
eyes–nose–cheek region, and six in the brow region.
The codes are used in conjunction with formulas to
distinguish the eight fundamental emotions of joy,
sadness, fear, anger, surprise, disgust, contempt, and
interest—and also the state of physical pain/distress.
In applying the MAX, judges are trained using a
manual and video materials presenting demonstra-
tions of the component movements associated with
the fundamental emotions, plus practice materials.

The three facial regions are coded separately and
independently (Malatesta & Haviland, 1982). A cod-
er’s reliability in MAX is determined by agreement in
identifying emotion expressions or blends; a coder’s
reliability is assessed requiring at least 80% agree-
ment (onset and offset points within 0.5 s) with the
master code in the MAX training manual (Izard &
Dougherty, 1982).
The AFFEX was based on experience suggest-
ing that judgments of facial expressions could be
improved by brief training. As Izard and Dougherty
(1982) put it,
improvement resulted from pointing
out and labeling the discrete changes in
appearance for each emotion expression
in each region of the face, [suggesting
that with training] the impressionistic,
intuitive global judgment could become a
more sophisticated, holistic judgment . . .
based on analysis and integration of the
information available from the appear-
ance changes in all three major regions of
the face. (p. 114)
Judges are trained to observe the whole face, to
identify discrete facial cues, to identify their tempo-
ral sequence and duration, and to label the emotion
directly. Unlike MAX, AFFEX requires integrating
information from the whole face, arguably retaining
advantages of holistic judgments based on simulta-
neously observing movement patterns in all areas of
the face.
MAX and AFFEX represent fundamentally differ-
ent ways to approach the analysis of facial expres-
sion: MAX avoids holistic judgments by requiring
analysis of separated parts of the face. This tech-
nique has the advantage of forcing attention to the
disconnected details of expression, avoiding effects
of general inferences about the emotion being
experienced. AFFEX goes to the opposite extreme,
eschewing serial, linear, and disembodied analysis
and taking full advantage of embodied abilities to
perceive or “pick up” social affordances from the
whole face (see McArthur & Baron, 1983). Interest-
ingly, Izard and Dougherty (1982) presented evi-
dence that MAX and AFFEX were equally effective
Measuring the Dynamic Stream of Display
in predicting judgments and that a combination of
the two improved prediction substantially.
Ekman’s FAST, FACS, and EMFACS.  Ekman
and colleagues approached the problem of the
universality of facial displays by systematically
gathering a large number of photographs—about
3,000—depicting posed emotional expressions.
These were studied using a variety of methods—
asking judges to judge them using the primary affect
labels, choosing their own emotion labels, select-
ing photographs to fit a given label, and so forth
(Ekman et al., 1971). Eventually 30 photographs
were chosen that were reliably associated with one
and only one of the primary affects. Six of Tomkins’s
primary affects were found: Interest and shame were
not reliably associated with posed photographs, and
disgust and contempt were combined in the initial
research. Specifically, photographs were found for
happiness, sadness, fear, anger, surprise, and disgust/
contempt.
The FAST attempted to specify and measure
critical cues reliably associated with these primary
affects. Like MAX, FAST scored movement in three
facial areas: the eyebrow/forehead, eyes/lids, and
lower face. A photographic atlas defined move-
ments in each area that were universally associated
with the primary affects. Coders view each area of
a stimulus face separately and classify the type and
duration of movement according to the atlas. For
example, a coder viewing the eyebrow/forehead
compares any response with eight criterion photo-
graphs in the atlas and assigns a code number. There
are 17 criterion photographs of the eyes/lids and 45
of the lower face. Formulas derive the frequency and
duration of the occurrence of each primary affect in
each facial region.
Unlike MAX, AFFEX, or FAST, the FACS
focused on facial musculature points underlying all
facial movements, not only those associated with
emotion (Ekman & Friesen, 1975, 1977). Like
FAST, coders carefully view and classify the type
and duration of facial movement according to an
atlas. The FACS extended the concept of anima-
tion units first described in Swedish by researcher
Carl-Herman Hjortsjö (1969) in Människans ansikte
och det mimiska språket (Man’s Face and the Mimic
427
Buck and Miller
Language), relating these to specific facial muscle
movements renamed action units (AUs) by
Ekman and Friesen (1975, 1977). Their resulting
technique, the FACS in original and modified forms,
greatly influenced research on facial expression
(Matsumoto & Ekman, 1988).
Like MAX, FACS yields objective scores of mus-
cle movements without any inference about the
holistic meaning of the behavior. Unlike MAX, it
identifies the AUs involved in any facial movement,
their intensity, and degree of asymmetry, whether
or not they are associated with emotion. The
Manual for the FACS (Ekman & Friesen, 1977) and
the FACS Investigator’s Guide (Ekman & Friesen,
1978) are available online; and persons purchasing,
studying, and successfully completing online tests
become certified FACS coders (see http://face-and-
emotion.com/dataface/facs/new_version.jsp).
EMFACS uses the FACS to score only the AUs
and AU combinations considered by Ekman and
colleagues to be relevant to detecting emotion.
EMFACS is intended to save time when the investi-
gation is centered on the study of emotion (Ekman,
Friesen, Irwin, & Rosenberg, 2014). However, the
empirical evidence supporting these choices of AUs
and their combinations is mixed (see Matsumoto &
Hwang, 2014). The EMFACS is made available only
to persons who are certified FACS coders, and the
authors caution that EMFACS should not be used to
study emotion blends, disguised or highly controlled
emotions, or very subtle expressions. As Matsumoto
and Hwang (2014) noted, this relative insensitivity
of the EMFACS (and FACS) to subtle expressions
(due to poor reliability shown by coders) may be a
problem that can be overcome by automatic meth-
ods of facial analysis.
Facial EMG techniques.  Another kind of tech-
nique for studying facial expression does not depend
on visible expressions or muscle movements: This
is facial EMG that records the electrical activation
of facial muscles that may not be associated with
visible movement (van Boxtel, 2010). Facial EMG
is ordinarily recorded with two small surface elec-
trodes (<4 mm) located close to one another over
a facial muscle of interest, which may play a major
role in the expression of specific primary emotions.
428
The muscles involved in frowning (corrugator
supercilii, depressor supercilii: brow lowerer AU4
in the FACS system) and smiling (orbicularis oculi,
pars orbitalis: cheek raiser AU6; and zygomaticus
major: lip corner puller AU12) are often of inter-
est. For example, in a classic experiment, Dimberg
(1982) showed that pictures of happy and angry
facial expressions evoked different facial EMG
response patterns, with increased zygomatic region
activity to happy expressions and increased corruga-
tor region activity to angry expressions.
Compared with the analysis of visible facial
expression, facial EMG has an advantage in sensi-
tivity, in that it is able to detect activations of facial
muscles that are so faint that they are not associ-
ated with any visible facial movement; furthermore,
their time resolution allows rapid changes to be
reliably measured (van Boxtel, 2010). Moreover,
there is evidence that such responses are automatic
and unconscious (Öhman, 2002; Wexler, Warren-
burg, Schwartz, & Janer, 1992). This confers the
significant advantage that facial EMG measurement
can detect emotional tendencies that the responder
may not be aware of; furthermore, it can be used
to study emotional responding in responders who
cannot respond verbally, such as preverbal infants.
With proper electrode placement, facial EMG activ-
ity can potentially discriminate between the primary
affects—although this is challenging both because
of cross-talk artifacts between facial muscles, the
natural complexity and changeableness of emotional
experience itself, and potential confounding by non-
emotional factors such as talking, orienting, mental
effort, and fatigue. Moreover, although participants
typically adapt well to the procedure, the facial EMG
technique is limited by the extensive preparation
required and the facial electrodes themselves (van
Boxtel, 2010).
Structural approaches to facial expression mea-
surement.  Most techniques for facial expression
measurement stem from Darwin’s (1872/1998)
analysis, as noted, and presume that certain patterns
of facial expression are naturally associated with
specific emotions. A contrasting approach, based
on a view of communication that was developed in
the early 1950s, is the structural approach of

Ray L. Birdwhistell and colleagues (e.g., Birdwhistell,
1970). Birdwhistell was an anthropologist who had
been impressed by cultural differences in facial
expression and also by gesture, posture, gait, arm
movements, and other bodily movements and
behaviors. He suggested that together these contrib-
ute to a multichannel code that is learned and pat-
terned within a cultural tradition, much as a child
learns language, complete with regional variations
and accents. Birdwhistell termed the field studying
this nonverbal bodily code kinesics, and he sug-
gested that it could be investigated and measured
in a way analogous to structural linguistics (Sapir,
1921). In structural linguistics, the sounds made
by individual human beings are termed phones, and
the sounds used in a given language are phonemes.
For example, there are 44 phonemes in English, and
most human languages contain fewer than 100.
They are the elements of spoken language and
correspond roughly to the letters in the alphabet.
Phonemes are combined into morphemes, which
have arbitrarily assigned meaning and correspond
roughly to the words in a dictionary—for example,
the vocabulary. A given language has a few hundred
thousand morphemes, and these constitute the sym-
bols of a language. The “magic” of language is that
the limited number of phonemes and morphemes
combine, using the rules of grammar, into an infinite
variety of sentences that are comprehensible by any-
one who has learned the vocabulary and grammar
of a language.
Birdwhistell (1970) suggested that the structure
of language is paralleled by the structure of kinesics.
Specifically, phones correspond to kines, which are
all movements of a given body part distinguishable
by an observer; phonemes correspond to kinemes,
which are all distinguishable movements of a given
body part that are actually used in communica-
tion in a given culture; morphemes correspond
to kinemorphs, which are body movements with
an arbitrarily assigned meaning in a given culture
and roughly constitute the kinesic vocabulary; and
sentences correspond to complex kinemorphs and
complex kinemorphic constructions, which are combi-
nations of kinemorphs based on grammar-like rules
and are infinite in variety but comprehensible to
anyone who has learned the kinesic vocabulary and
Measuring the Dynamic Stream of Display
grammar. This linguistic–kinesic analogy is reflected
in the popular term for Birdwhistell’s analysis: body
language (Fast, 1970). The goal of kinesics is, in
effect, to discover and document the vocabulary and
grammar of body language or, more precisely, of a
diversity of body languages.
The methods for observing and measuring the
elements of kinesics reflect the traditional methods
of structural linguistics, which involve the careful
and comprehensive analysis of examples of natural
language. In the case of kinesics, this involves the
observation and analysis of examples of human
interaction occurring in natural settings—that is,
observing and comprehensively analyzing regu-
larities in the stream of behavior in one natural
context, and seeking similar regularities in other
contexts in a process termed context analysis. The
structural approach and method of context analy-
sis were formalized in Palo Alto (CA) in 1956 by a
group including anthropologists Birdwhistell and
Gregory Bateson, psychoanalysts Frieda Fromm-
Reichmann and Henry Brosin, and linguists Charles
Hockett and Norman McQuown. This collaboration
resulted in the publication of The Natural History
of an Interview, which analyzed portions of conver-
sation between Bateson and members of a middle
class American family (McQuown, Bateson, Bird-
whistell, Brosin, & Hockett, 1971). Communica-
tive behaviors of participants were analyzed using
kinesic, phonetic, and paralinguistic transcription
systems aligned over time (Leeds-Hurwitz, 1987).
The movements of the face are included in context
analysis, but unlike other measurement systems
that we have considered, the face is not regarded
as central. In fact, the structural approach takes a
strong culturally relativist position that, just as with
language, bodily expressive behavior is learned and
differs from culture to culture and, indeed, that
“accents” occur within cultures. Thus, Birdwhistell
insisted that his observations of the structure of
kinesic behavior in the Philadelphia (PA) area might
not generalize to other areas of the United States.
It is noteworthy that the elements of kinesic analy-
sis rest firmly on the role of facial expressions and
other bodily movements in communication: They are
defined according to their being distinguishable to
an interaction partner; are observed and measured
429
Buck and Miller
during the process of face-to-face interaction; and
are posited not only to underlie the communica-
tion of meaning between individuals but, indeed,
to punctuate the informal interpersonal transaction
and, by doing so, are responsible for the maintenance
of the social order (Scheflen & Scheflen, 1972).
Birdwhistell’s (1970) methods were applied to
the analysis of psychotherapy transactions between
patients and therapists-in-training at the Eastern
Pennsylvania Psychiatric Institute near Philadel-
phia. Analysis of therapists’ behaviors revealed that
they often responded to their patients with head
nods, and Birdwhistell reported that the duration
and number of nods were relevant in influencing
patients’ behavior. “Fast nods” of less than 0.3 s
often produced no change in patients’ behavior,
but “slow nods” of more than 0.8 s did: The patient
often slowed or speeded vocalization or elaborated
on a point. Multiple fast nods (three or more in
secession) typically caused a dramatic change, often
with a fading of vocalization on the part of the
patient—an indication of understanding on the part
of the therapist, perhaps. However, one of the most
interesting observations concerned the rhythm of
therapists’ nodding: whether it was in synchrony
or out of synchrony with the rhythm of patients’
behavior. The synchronous “understanding nod”
tended to encourage patients, but the asynchronous
“sore thumb nod” was disruptive, perhaps express-
ing a lack of attention or interest on the part of the
therapist. Indeed, this and other demonstrations
of interpersonal synchrony were among of the most
significant discoveries on the part of the kinesic
approach (Condon & Ogston, 1966; Kendon,
1970). This illustrates how the meaning of a specific
behavior, such as a nod, may differ greatly in inter-
actional/communicative context. Thus, the meaning
of a given facial expression, such as frowning (brow
lowerer AU4) or smiling (cheek raiser AU6 and lip
corner puller AU12), may differ according to its syn-
chrony and whether an interaction partner is frown-
ing or smiling.
A problem with the structural approach has been
the sheer complexity of the raw data. Whereas struc-
tural linguistics involves a limited number of phones
and phonemes that are organized into meaningful
morphemes, the number of behaviors contributing
430
to kinemorphs can be very large. In the face, any
distinguishable movement qualifies as a kine, and as
with language, the kinemes derived from kines can
vary from subculture to subculture. For example,
Birdwhistell (1970) distinguished four eyelid posi-
tions as potential kinemes in his eastern Penn-
sylvania samples, but he emphasized that may be
confined to that subculture. Some or many AUs may
qualify as kinemes but that would depend on their
being actually used communicatively in interaction
within a given subculture, and their theoretical link-
age to emotion is, from the structural point of view,
not relevant. From a practical standpoint, this com-
plexity has made research challenging. Birdwhistell
noted that he spent 100 hr analyzing every second of
interaction at the beginning of his research, but later
he achieved 1 hr per second—a great improvement,
to be sure. However, this disadvantage is signifi-
cantly lessened if facial expressions and other bodily
behaviors can be analyzed automatically. We now
turn to systems designed to achieve the automatic
analysis of emotion from facial expression, or auto-
matic emotion detection systems.
Automated Facial Emotion Analysis
Techniques
The contributions of facial expression scoring tech-
niques over the years led to great interest in develop-
ing automated scoring systems that can potentially
address some of their problems. In particular, prob-
lems with the MAX, FACS, and EMFACS in reliably
detecting subtle and nuanced expressions with human
judges might be remedied by automated systems. In
addition, such systems may address the problems
of complexity and information overload for human
judges in structural analyses and, indeed, the labori-
ous nature of human manual coding in any system.
This section follows the historical timeline from
1973 to the present, highlighting key problems and
advances related to technology, mathematics, and
the understanding of emotive facial displays. The
practical and theoretical uses of automated emo-
tion detection systems (AEDS) are also discussed.
As a whole, the energy devoted to AEDS has led to
the development of computerized systems capable
of reliably detecting facial displays of Ekman and
Friesen’s (1975) six primary affects in controlled

environments using ample display databases—that
is, with full lighting, frontal views of the face, mini-
mal head movements, and basically Caucasian mod-
els. However, capturing and detecting displays in
real-world contexts with limited display databases
still stands as a significant challenge. In this sec-
tion, we continue with a consideration of bottom-up
expression detection that is not focused on specific
emotion categories, which parallels the kinesic
approach in a number of respects, and we conclude
by considering human-to-human communication
applications of current real-time, facial emotion
detection software.
History.  The pursuit of a fully automated system
to detect emotions via facial displays carries with it
a history marked with multiple roadblocks and cre-
ative solutions born from the collaborative efforts of
an array of scientific disciplines. To understand the
multiple approaches that have been used to address
the problem, it is useful to consider the basic ele-
ments involved in formulating a solution. Essentially,
all attempts at automated facial emotion recogni-
tion have required researchers to consider what the
human eye and brain are doing while observing, pro-
cessing, and interpreting facial emotions (Pantic &
Rothkrantz, 2000). Specifically, AEDS generally
require three stages: face acquisition, feature extrac-
tion, and emotion expression recognition. Unlike
natural human processing, AEDS require a great
deal of technology and mathematical modeling; they
also fall well short in their capacity to contextual-
ize the scenarios in which facial emotions are being
displayed. The end goal of AEDS is to reliably detect
emotion expressions in a multitude of contexts
involving individuals who vary by facial anomalies,
ethnicity, age, hairstyle, eyewear, and biological sex.
Though there is some debate (Valstar, Mehu,
Jiang, Pantic, & Scherer, 2012), most sources sug-
gest that the first attempt to address automatically
recognizing emotions through facial displays was a
dissertation by Kanade (1973). This was followed by
a conference article proposing automatic expression
analysis using 20 facial points (Suwa, Sugie, & Fuji-
mora, 1978). Each of these ventures was preceded
by at least 10 years of research (Samal & Iyengar,
1992) focused on computerized facial recognition,
Measuring the Dynamic Stream of Display
which entailed both identification (databases provided
with a face in that database match it to the correct
identity) and verification (databases provided with a
face that might not belong in that database indicate
whether it does or does not belong). Another pre-
cursor that pushed forth the pursuit of AEDS was
the development of the FACS (Ekman & Friesen,
1977). Thus, attempts at AEDS stem from fields
dealing with a range of topics pertaining to pattern
recognition, computer programing, mathematics,
and human behavior.
By 1992, research on computerized facial
recognition (not yet emotion recognition) was
underway—building databases of faces and math-
ematical techniques for face acquisition, feature
extraction, and matching. However, at this time
there were many unsolved issues in facial recogni-
tion: Most researchers attempted to focus on facial
profiles or faces viewed straight on (a) because of
the complexity of detecting and extracting facial
features of tilted heads (Samal & Iyengar, 1992);
(b) because databases of faces were limited (mostly
White males); and (c) because images being used
had to have no occlusion (e.g., a hand on someone’s
face), no facial hair, no glasses, and no scars or
other distinct facial anomalies such deep wrinkles
(Samal & Iyengar, 1992). Between the 1970s and
1990s, researchers explored the best two-dimen-
sional (2D) pixilation and image sizes to use for
facial representation (e.g., Baron, 1981; Kaufman &
Breeding, 1976; Wong, Law, & Tsang, 1989). Com-
mon mathematical analyses to match extracted faces
included using Euclidean distances, clustering, set
partitioning, and correlations between facial feature
vectors (Samal & Iyengar, 1992). Major uses of the
facial recognition technology were focused on auto-
mating police databases and multiple security sys-
tems. The foundation of facial emotion recognition
was being laid during this period.
Detecting emotion categories.  Regarding AEDS,
marginal headway had been made by the early
1990s, with only two significant studies undertaken
(Suwa et al., 1978; Terzopoulos & Waters, 1990).
Terzopoulos and Waters (1990) used emergent
computing power and offered an initial automated
three-dimensional (3D) emotion rendering system,
431
Buck and Miller
although makeup and manual facial feature location
were needed for the system to work efficiently. At
the onset of the 1990s, the major hurdles for those
interested in AEDS centered on a lack of sexually
and ethnically diverse facial emotion expression
databases, a lack of diverse expression databases in
general, and a lack of complex computerized train-
ing programs. The idea of training a system to auto-
matically detect emotions for both males and females
appeared to be a distant goal (Samal & Iyengar,
1992). The early 1990s marked a point where AEDS
were in their infancy, yet poised to grow.
The period from 1991 to 1995 experienced
growth in the interest and efficiency of AEDS. In a
review, Bettadapura (2012) suggested that a catalyst
for interest in the early 1990s was a combination
of cheaper computing power and more advanced
algorithms for facial detection and tracking. At
this time, researchers were attempting to detect
emotions in static images and image sequences. In
tandem, researchers interested in facial animation
and those interested in automated facial detec-
tion were both building models to represent and
interpret emotion displays via facial musculature
movements—though doing so in different ways.
Those interested in emotion detection were mostly
focused on using or improving on the FACS system
and thus paying close attention to AUs. Computer
animation researchers, however, were using a multi-
tude of individual systems that were technologically
advanced but lacked coherency (Bettadapura, 2012).
Based on a review of AEDS by Pantic and Roth-
krantz (2000), systems and techniques designed
from 1991 to 1995 lacked the capacity to classify
emotion expressions in terms of AUs, and only one
system of the 11 reviewed was able to perform in
real-time; both of these were cited as critical features
of an efficient AEDS. In addition, many systems
were plagued with an inability to detect and/or clas-
sify images with occlusions and lighting variation.
Lastly, though some scholars using facial motion
analysis were able to perform automatic facial data
extraction, others still required hand measures
(Pantic & Rothkrantz, 2000). The increase in auto-
mation and efficiency of analysis techniques during
this time, coupled with the problems that they were
unable to overcome, continued to spur on interest
432
in developing new AEDS. Suggested applications
of AEDS focused primarily on human–computer
interaction, animation, and human behavior
interpretation.
The following period from 1996 to 2000 was
reviewed by Pantic and Rothkrantz (2000), who
offered an excellent snapshot of the developments
occurring during that period. Continued increases
in cheap computational power (Fasel & Luettin,
2003) accelerated advances in both 2D and 3D
motion tracking, which spurred on interest to
continue developing more efficient AEDS. On the
downside, despite significant improvements related
to face detection, expression data extraction, and
facial expression classification, there were still
notable flaws that no system was able to overcome.
Specifically, none of the reviewed AEDS were able
to extract data from images with occluded faces or
to accurately classify all 44 facial AUs (Pantic &
Rothkrantz, 2000). Some of the major advances
during this time include work by Essa and Pentland
(1997), who used a combination of previous meth-
ods to generate the FACS+, which was an extension
of the FACS coding system. Also, effective methods
for facial expression data extraction were employed,
centered on the three most common approaches to
face representation—that is, holistic (whole face),
analytical (focusing on specific facial features or
areas), and hybrid approaches.
The complexity and reliability of AEDS increased
in the late 1990s as myriad analysis techniques
emerged that were centered on template-based,
feature-based, rule-based, and neural-network-based
methods for detecting, classifying, and interpreting
static and sequenced emotion expressions (Pantic &
Rothkrantz, 2000). As examples, the use of Gabor
wavlet filters to deal with illumination issues was
becoming common (Fasel & Luettin, 2003), a more
complete facial display database emerged (Tian,
Kanade, & Cohn, 2000), and multimodal frame-
works that included vocal analysis were presented
(Cohn & Katz, 1998). In addition, the Moving Pic-
tures Experts Group provided computer animation
researchers with a standardized facial movement
system that illustrated Facial Animation Parameters:
This system became the international standard in
1999 (Bettadapura, 2012). The new standardized

Facial Animation Parameters are related to FACS
AUs, enabling a smoother merger between anima-
tion and emotion recognition researchers. Amidst
these advances, researchers were still criticized for a
lack of engagement with issues such as head orien-
tation, headwear, and occlusions. Pantic and Roth-
krantz (2000) called for more attention to detecting
facial AUs and cautioned researchers against using
the same set of images for system training and test-
ing. Meanwhile, Fasel and Luettin (2003) suggested
that more attention be paid to measuring emotion
intensities and conversational faces that are likely
rotated out of plane.
After 2000, researchers who were focused on
AEDS began to seriously address issues related to
detecting out-of-plane head rotations, spontaneous
expressions, occlusions, and mixed-emotions (Val-
star, Gunes, & Pantic, 2007; Valstar et al., 2012).
Over the next decade, the databases of static and
video expressions increased extensively to include
audio cues, more natural contexts, 3D recordings,
and spontaneous expressions (Zeng, Pantic, Rois-
man, & Huang, 2009). In their review on the state
of automated facial analysis Zeng et al. (2009) cited
five major advancements since 2000: detection of
attitudinal and nonbasic affective states, early efforts
to analyze posed and spontaneous displays, early
attempts at vision-based analysis of spontaneous
mixed displays (e.g., facial expression and gesture)
with 3D models, some attempts at context-dependent
interpretation of facial displays, and utilization of
advanced feature extraction and classification tech-
niques being applied in real-time using object track-
ing. Regarding face acquisition, multiple methods
were used at this time, and many researchers pres-
ently use the piecewise Bézier volume deformation
tracker (Tao & Huang, 2002; see Figure 17.1). A
multitude of machine learning algorithms for extrac-
tion and classification had been developed, such
as Bayesian nets, support vector machines (SVMs),
multilayer perceptrons, and decision trees (for a
review, see Sebe et al., 2007). By this time, numer-
ous applications for AEDS had been proposed,
including the following: E-learning, lie detection,
patient monitoring, emotion training (e.g., for autis-
tic individuals), driver monitoring, consumer reac-
tions to products, surveillance, affective computing,
Measuring the Dynamic Stream of Display
and human interaction with robots and computers
(Suneetha & Sandhy Arani, 2013).
Competitions.  After a decade of significant
advances in AEDS, notable competitions provided a
catalyst for those interested in dealing with some of
the lingering challenges in creating efficient AEDS.
In 2011, the first Facial Expression Recognition and
Analysis (FERA) challenge urged researchers to deal
with small, context-specific, and, often, out of plane
data sets. In 2013, the Emotion Recognition in the
Wild (EmotiW) challenge pressed researchers to try
and classify emotions expressions in contexts analo-
gous to real-world conditions. The FERA challenge
was reviewed using a meta-analysis (Valstar et al.,
2012). The data set used in this challenge was a por-
tion of the Geneva Multimodal Emotion Portrayals,
a collection of more than 7,000 audiovisual emotion
portrayals (of 18 emotions, 26 facial AUs labeled
frame-by-frame) enacted by 10 actors under the
direction of a professional director (Valstar et al.,
2012). The challenge entailed two contests: one to
detect AUs (12 possible) per frame, and the other
to detect five emotion expressions (anger, fear,
joy, relief, sadness). One observation, related to
machine learning techniques, was that during this
competition many competitors used SVMs, includ-
ing the team that won the AU detection competi-
tion (Senechal et al., 2011). The challenge did not
provide sufficient data to suggest whether modeling
with 3D or 2D images was more robust; however,
the data did suggest that modeling appearance
dynamics over geometric features alone was more
efficient. In the meta-analysis of the competition,
Valstar et al. (2012) concluded that “inferring dis-
crete emotions from video of known subjects may
well be considered solved” (p. 976)—that is, auto-
matic detection of the six basic emotions has only
marginal room for improvement. In terms of the
future, Valstar et al. suggested that the detection of
specific AUs and their intensity remains a significant
challenge for researchers.
In 2013, the EmotiW challenge used the Acted
Facial Expression in the Wild data set collected from
movies; the data set aimed to closely reflect real-
world contexts and included both visual and audio
dimensions. The challenge focused on researchers
433
Buck and Miller

Figure 17.1.  Deformation tracker: Snapshots of the real-time demonstration system. The ratio between the heights
of the two bars on the right side indicates the quality of the tracking result. From “Visual Estimation and Compression
of Facial Motion Parameters—Elements of a 3D Model-Based Video Coding System,” by H. Tao and T. S. Huang, 2002,
International Journal of Computer Vision, 50, p. 122. Copyright 2002 by Springer. Reprinted with permission.

attempting to detect emotions (anger, sadness, hap-
piness, fear, disgust, surprise, and neutral) using
audio or visual stimuli—or both. This challenge
has yet to be extensively reviewed; we offer some of
the insights gleaned from the winning team. Kahou
et al. (2013) won the emotion recognition compe-
tition with a recognition accuracy rate of 41.03%
(well above the test fusion baseline of 25%). Their
most prominent findings related to a problem of
overfitting using deep neural networks. Specifically,
they found that typical SVMs were overfitting mod-
els by scanning very small spaces and that multi-
layer perceptrons were also overfitting by scanning
moderately small spaces. Their solution was to uti-
lize a unique method of aggregating per model and
per class predictions via random search over simple
weighted averages (Kahou et al., 2013). In effect,
they were using multiple complex deep network
models to explore moderate areas of space rapidly
and without having to adjust their model param-
eters. Their findings suggest that when overfitting
is an issue in dual modality emotion detection, it is
useful to step back from the standard SVM and mul-
tilayer perceptron techniques.
Current challenges.  Over the course of the last
40 years, the capacity for automated facial emotion
recognition has made great progress. Still, there are
clearly matters related to real-world recognition that
need to be addressed. For example, the promising
EmotiW challenge winners showed a comparatively
low accuracy rate (41.03%) compared to real-time

434

recognition rates of known faces using automated
programs such as the Computer Expression
Recognition Toolbox (CERT) tool box, which has
now been modified and is called FACET (90%
posed, 80% spontaneous; Littlewort et al., 2011).
The FACET software allows an integration between
physiological responses and overt displays by using
a system that synchronizes, in real-time, the collec-
tion of facial affect, skin conductance responses, heart
rate, and eye movement. Other notable automated
programs include 3D Facial Imaging (nViso, 2011),
Affdex (Affectiva, 2011), and FaceReader 6 (Noldus
Information Technology, n.d.): Each program uses
variations of state of the art techniques to automati-
cally detect, extract, and classify emotional facial
expressions.
Unresolved Issues in the Field
As the technical challenges of automatic facial
expression detection have been met with progress,
several issues in the general facial coding field need
to be addressed. In this section, we address four of
these issues: the question of bottom-up versus top-
down analytic techniques, the difference between
facial emotional detection and communication, the
question of a facial expression database indepen-
dent of given detection techniques, and the issue of
static and posed versus dynamic and spontaneous
facial expression.
Bottom-up facial expression detection.  Research
based on the FACS and MAX divided facial expres-
sions into predefined emotion categories based on
facial muscle movements. These approaches can
be considered “top-down” in the sense that they
target facial muscle movement patterns and do not
consider other possible sources of facial expression
and display. However, there is evidence that the face
can communicate emotion without scorable FACS
movements (e.g., Vanger & Ellgring, 1989). The
cues used in such communication are unclear: Slight
tendencies to raise or lower the head or look to the
left or right, blanching or blushing, and changes
in pupillary response are all possible candidates.
Indeed, Tomkins’s (1962, 1963, 1991) original pri-
mary affects included interest/excitement displayed
by eye behavior and shame/humiliation displayed
Measuring the Dynamic Stream of Display
by head lowering; however, these emotions were
excluded from the primary affects of Ekman and
Friesen (1978), perhaps because they could not be
expressed in static photographs of posed expression.
In contrast to this, we have seen that kinesic
analysis uses a “bottom-up” approach—considering
all observable facial/body movement as potentially
meaningful. An analogous approach has used the
advances in expression detection and machine
learning to select the most consistent and rel-
evant expressions from massive collections of raw
data (Ahn, Bailenson, Fox, & Jabon, 2009, 2010).
Unlike Birdwhistell and his colleagues, these inves-
tigators have assumed that facial expressions are
important—valid reflections or readouts of internal
states—and as such can allow the prediction of
future behavior. They also assumed that in situa-
tions where social factors are minimized, stimuli
can generate emotions, and therefore facial expres-
sions, automatically and universally across cultures.
From this point of view, detecting and analyzing
facial expressions can yield insights about internal
motivational–emotional states generalizable across
cultures. They furthermore suggested that a bottom-
up approach can leverage raw computer power
that avoids disadvantages associated with objective
coding schemes such as MAX and FACS: extensive
training of coders, labor-intensive frame-by-frame
analysis, intercoder reliability problems, inability
to reliably detect brief and subtle nuances of facial
expression, and human error. We saw the problems
faced by structural approach, as it takes approxi-
mately 1 hr to score 1 min of facial behavior in
terms of FACS AUs into temporal segments (Pantic,
Valstar, Rademaker, & Maat, 2005). According to
Ahn et al. (2010), this automated technique prom-
ises “cost-effective detection with greater accuracy,
reliability, and the ability to detect micro-expres-
sions that may be lost to the human eye” (p. 357).
These investigators cited several examples using
computer vision and machine-learning algorithms to
model and predict behavior. The facial feature
data extracted from video clips can involve either
22 or 37 points on the face, based on the Neven
Vision or OKAO Vision (OMRON, n.d.) libraries,
respectively. These are inputted and mapped to
participant behavior. For example, 3–5 s of facial
435
Buck and Miller
expressions are extracted 1 or 2 s before the behav-
ior to be predicted occurs. A training set of observa-
tions is taken independently of a testing set, using
different participants to avoid falsely high accuracy
estimates from the machine learning algorithms.
According to Ahn et al. (2010), overall accuracy is
defined as the percentage of behaviors correctly clas-
sified by the system, and precision is defined as the
“percentage of correctly categorized instances in a
given class” (p. 357).
In a study of real-time classification of evoked
emotions, the spontaneous facial expressions of
participants to amusing or sad films were recorded
while physiological responses were taken (Bailenson
et al., 2008). The behavior predicted was the sec-
ond-by-second intensity of the two emotions rated
by FACS-trained coders. The percentages of correct
classifications in classifying amusement or sadness
were 90% and 72%, respectively. The predictions
of emotion intensity were correlated with the rated
intensity, yielding significant correlations of .53 for
amusement and .23 for sadness. Ahn et al. (2010)
concluded that “computer vision coupled with
machine learning algorithms may be used to detect,
classify, and predict emotion and emotion intensity
with relatively high accuracy” (p. 358). Other studies
reported by Ahn et al. (2010) applied AEDS analysis
to detect and predict emotions in a driving simulator
(Jabon, Bailensen, Pontikakis, Takayama, & Nass,
2011) and while shopping (Ahn, Jabon, & Bailenson,
2008) as well as to detect operator fatigue (Ahn,
Jabon, & Bailenson, 2009). All in all, the bottom-up
approach may have advantages over the category-
based systems in terms of sensitivity to all sorts of
facial behaviors, not only those preselected as involv-
ing specific muscle movements or emotions.
Emotion detection versus communication.  A
second area relevant to the facial coding literature
is emotion detection versus human-to-human
communication (Fasel & Luettin, 2003; Pantic &
Rothkrantz, 2000; Zeng et al., 2009). Little research
has applied facial coding or AEDS software to study
human communicative scenarios as opposed to one-
way, behavior recognition (in varied contexts), ani-
mation, and human–computer interaction. Recent
articles provide examples of how AEDS could be
436
integrated to help address spontaneous human
communication phenomena. Proyer, Meier, Platt,
and Ruch (2013) studied how individuals with a
fear of being laughed at (gelotophobia) and indi-
viduals with a joy in being laughed at (gelotophilia)
react emotionally (e.g., shame, sadness, anger, and
joy) when viewing images of potentially embarrass-
ing situations (i.e., a person slipping on a snowy
street). Participants rated their emotional responses
to four images considering how they would feel if
they saw this happen and how they would feel if
it happened to them. The study provides interest-
ing results regarding emotional responses to being
laughed at but could benefit from AEDS and consid-
eration of a more communicative design. As another
example, Powers, Rauh, Henning, Buck, and West
(2011) studied brief conversations in dyads over
video to study (a) the effects of a delay in the audio/
video signal on the rated emotions experienced by
the participants, (b) assumed similarity in rated
emotions, (c) actual similarity in rated emotions,
and (d) emotion communication accuracy (e.g.,
their ability to detect the emotion ratings of the
partner). The addition of facial coding, and particu-
larly AEDS analysis, in such a design could signifi-
cantly enhance such an investigation, resulting in
assessments of the nature, similarity, and perhaps
synchrony of actual expressions to compare with
participant ratings.
Facial expression databases.  Another issue in
the field of facial coding and AEDS analysis con-
cerns the nature of the facial expression database
used to evaluate the system. As noted, Pantic and
Rothkrantz (2000) cautioned against using the same
images for training and testing. Pantic et al. (2005)
noted the need for an easily accessible database for
exploring issues involved in AEDS analysis. They
considered the characteristics of five existing face
databases and presented the MMI Facial Expression
Database, which includes more than 1,500 samples
of posed expressions—both static images and brief
sequences showing the onset, apex, and offset of the
expression. Faces are presented in both frontal and
profile views. However, all of the databases consid-
ered by Pantic et al., including their own, employed
intentionally posed facial expressions.

The recent shift to the analysis of dynamic, spon-
taneous, and ecologically valid facial expressions
recorded in natural settings (Nicolaou, Gunes, &
Pantic, 2011) is exemplified in the Ahn and col-
leagues’ (Ahn, Bailenson, et al., 2009; Ahn et al.,
2010) research. Such a shift requires a database of
such expressions to be used to test facial coding and
AEDS systems. In his survey of the state of the art of
FERA in articles published from 2001 to the present,
Bettadapura (2012) pointed out the need for a shift
from the analysis of posed to spontaneous expres-
sion. He noted the need for a standardized spontane-
ous expression database containing video sequences
in which the participant is unaware of being filmed
and in conditions where spontaneous expressions
are encouraged. In addition, such sequences should
be labeled, that is, they should include informa-
tion about the participant’s emotional response,
either from participant self-ratings or from ratings
of observers, or both. Also, such sequences should
show a complete temporal pattern including the
onset, apex, and offset of the emotional response.
All of these characteristics are found in video
sequences taken in the SVT: In the next section, we
describe this technique and relevant research.
Spontaneous versus intentional displays of
emotion.  The final and arguably most critical issue
in the facial coding literature involves its empha-
sis on intentionally posed facial expressions. It is
widely recognized that facial expressions reflect a
combination of an innately based emotion display
and learned management of the display (Buck &
Powers, 2013). Static photographs or films of posed
displays are valuable: Indeed, Ekman and Friesen’s
(1975) analysis of posed displays convincingly
demonstrated pancultural features of facial displays.
However, posed expressions are lacking in that they
depict persons not actually experiencing the emo-
tion in question.
An alternative is to film natural displays in
response to effective emotional stimuli in a situation
where influences from display rules are minimized,
as exemplified in the technique employed by Ahn
and colleagues (Ahn, Bailenson, et al., 2009; Ahn
et al., 2010). That has been the approach of the
SVT, developed from the Cooperative Conditioning
Measuring the Dynamic Stream of Display
Technique used by Robert E. Miller (1967) to study
the communication of affect in rhesus monkeys. The
following section describes the SVT and how it can
address the unresolved issues in the facial coding
literature.
MEASURING DYNAMIC AND
SPONTANEOUS FACIAL EMOTION
COMMUNICATION
Spontaneous facial expressions can be elicited by
filming participants with an unobtrusive camera
while they are responding to effective emotional
stimuli in minimally social situations. In the sec-
tions that follow, we examine different applications
of this technique.
The Slide-Viewing Technique
In the SVT, participants (senders) view a series of
emotionally laded images while being filmed by an
unobtrusive camera. After each slide, the sender
rates the emotions felt while viewing the slide. The
filmed expressions are then rated by receivers, who
attempt to judge (a) what sort of slide was pre-
sented on each trial and (b) the sender’s emotional
response to the slide (Buck, Miller, & Caul, 1974;
Buck, Savin, Miller, & Caul, 1972). This procedure
yields two measures of communication accuracy.
The percent correct measure is the percentage of
slides that receivers correctly categorize. This is a
“situational reference” task, where the goal is to
identify the situation (e.g., slide type) that initiated
the expression. The other is the emotion correlation
measure, where the goal is to detect specific emo-
tions as rated by the sender. The receivers’ rating of
each emotion is correlated with the sender’s ratings
across the slides, yielding measures of the accuracy
with which each emotion (happiness, sadness, fear,
anger, etc.) was communicated. These two mea-
sures of communication accuracy are analogous,
respectively, to the overall accuracy and precision
measures employed by Ahn and colleagues (Ahn,
Bailenson, et al., 2009; Ahn et al., 2010). In the
SVT studies, communication accuracy is defined in
terms of sending accuracy, an individual’s spontane-
ous tendency to send accurate nonverbal emotional
437
Buck and Miller
messages across receivers, as well as receiving ability,
an individual’s ability to receive accurate nonverbal/
emotional messages from others across senders.
Studies in young adults.  The SVT has been used
to study sending accuracy in a variety of participant
groups. Initial studies used young adults (Buck
et al., 1972, 1974). The slides used as stimuli were
chosen by showing a large number of selected slides
to observers and asking them to categorize them
into five content categories: Sexual, Scenic, Pleasant
People, Unpleasant, and Unusual. Senders watched
each picture and, when a light came on, verbally
described the feelings it evoked. When a slide was
presented, senders looked at the slide (slide period);
and when the light was turned on, they verbally
described their emotional reaction (talk period).
After the light and slide were turned off, senders
rated the emotions they experienced to the slide:
For the initial studies, only Pleasant–Unpleasant and
Strong–Weak scales were used, but a larger variety
of emotions was used in later studies. Receivers
viewed the sender (without audio) and made judg-
ments about the type of slide the sender viewed and
the sender’s emotional response.
Results revealed significant communication on
all measures. Women were significantly better send-
ers than men on both the percent correct and emo-
tion correlation measures. Also, better senders had
smaller increases in skin conductance deflections
(SCDs) and heart rate accelerations when begin-
ning to speak.1 This was used to define two expres-
sive styles: An externalizing style combined high
sending accuracy with low autonomic responding,
whereas an internalizing style showed low sending
accuracy with high autonomic responding. Results
also revealed a tendency for males to show an inter-
nalizing style of response and for females to show
an externalizing style of response. The internalizing
style was associated with higher introversion and
lower self-esteem than externalizing. Also, the exter-
nalizing style was associated with personal verbal
descriptions of feelings (e.g., “that makes me feel
good”), and the internalizing style was associated
I t should be noted that the skin conductance measure to show significant negative between-subjects relationship with sending accuracy was the
1
change in SCDs, not change in skin conductance level. SCD changes are thought to reflect inhibition (Fowles, 1980).
438
with more of a description of the slide (e.g., “that’s
a nice slide”). The results were compatible with
suggestions that internalizing is associated with the
inhibition of overt expression and that externalizing
is associated with a relative lack of such inhibition.
Studies in preschool children.  The results of the
studies with adult participants had clear implica-
tions for understanding how emotional displays are
socialized in children, and the next studies using the
SVT were conducted with 3.5- to 5-year-olds serving
as senders, with their mothers and groups of under-
graduates serving as receivers (Buck, 1975, 1977).
The studies aimed to investigate the developmental
origins of externalizing and internalizing styles of
response. To create emotionally loaded slides appro-
priate to young children, a new slide category was
added: Familiar People slides showed the child and
friends at the nursery school. Each child was photo-
graphed in the school before the experiment. During
the experiment, the child saw two slides of him- or
herself and two general views of the school. This
slide category had the important advantages of reli-
ably eliciting strongly positive emotional responses,
which are otherwise difficult to elicit, while also
increasing the overall positivity of the experiment
for the children. Other categories were Unfamiliar
People, Unusual, and Unpleasant, showing pictures
mildly unpleasant to children.
The child was brought to the experiment by the
mother and sat with an experimenter whom the
children knew from school. The child faced a back-
lighted projection screen on a small table, and a
video camera was focused on the child’s face from
the next room. The experimenter sat beside and
slightly behind the child to avoid cuing the child’s
expressive behavior and kept her attention on the
slides. This established a “minimally social” situa-
tion, helping to keep the child on task and minimiz-
ing any effect of the experimenter on the child’s
expressive behavior. The mother sat in a separate
room with a television monitor. The image of the
child appeared just before the slide was presented,
the mother watched her child’s face, and when the

image disappeared, the mother indicated which kind
of slide she thought had been presented and rated
how pleasant or unpleasant her child felt about the
slide. Later, children’s expressions were shown to
groups of undergraduates, who made similar ratings.
Results showed substantial differences between
children in their sending accuracy, but this was not
strongly related to gender. The difference between
externalizing and internalizing response modes was
found in the children, with a negative correlation
between sending accuracy and SCD responses. How-
ever, response mode and gender were not related as
they had been in the adults. Interestingly, sending
accuracy was negatively correlated with age among
boys—that is, as boys got older between 3.5 and
5 years, their sending accuracy tended to decline.
To investigate whether sending accuracy as mea-
sured in the laboratory would have implications for
the children’s behavior in the classroom, teachers
were asked to rate the children on scales reflecting
characteristics of persons manifesting internalizing
and externalizing response modes. Sending accuracy
was positively related to teacher ratings of extraver-
sion, activity, and having many friends; it was also
positively related to being aggressive, bossy, and
often hard to get along with. Sending accuracy was
negatively related to ratings of introversion, shyness,
and being solitary; it was also negatively related to
being quiet and reserved, cooperative, and emotion-
ally controlled. Thus, externalizing and internaliz-
ing each were related to both positive and negative
social behavior. Overall, results were consistent with
a temperament basis for externalizing and internal-
izing response modes, with a social learning account
explaining the emerging gender difference.
In the context of facial measurement, these stud-
ies have demonstrated evidence of external validity
in sending accuracy measures that use a method of
expression elicitation that speak to the unresolved
issues in the field. They fulfill the desiderata pointed
out by Bettadapura (2012): involving spontaneous
expression, with video sequences in which the par-
ticipant is unaware of being filmed, in a minimally
social situation where spontaneous expressions are
encouraged; where sequences include information
about the sender’s emotional response both from
self-ratings and ratings of observers; and where
Measuring the Dynamic Stream of Display
sequences show the onset, apex, and offset of the
emotional response. Moreover, sending accuracy
measures have been significantly associated with
gender, age, patterns of physiological responding,
verbal reports of emotion, personality measures, and
teachers’ ratings of classroom behavior. Importantly,
the percent correct and pleasantness measures are
not simply measures of the behavior of the individual
sender; they are accuracy scores reflecting the extent
to which receivers are able accurately to make judg-
ments about senders, and this accuracy is an objec-
tive measure that can be related to chance. However,
they can be used to assess and measure the behaviors
of the individual sender, as we show later.
Emotional Expression and Communication
in Patients With Brain Damage
It is widely recognized that disorders associated
with brain damage and psychiatric illness are asso-
ciated with deficits in emotional expression and
communication, but the exact nature of these defi-
cits is poorly understood. The SVT offers a way to
approach such questions, and it has been applied to
several clinical groups, including patients with brain
damage, children with emotional and behavioral
disorders (EBDs), and patients with schizophrenia.
The results of these studies are significant in under-
standing facial measurement, and specifically the
basis of the distinction between spontaneous and
intentional facial expression, and how facial expres-
siveness differs from facial communication.
SVT studies in patients with brain damage. 
Damage to the left cerebral hemisphere has long
been associated with deficits in verbal communica-
tion termed aphasias. Disagreement in the clini-
cal and research literature concerning the effects
of this brain damage on “nonverbal” aspects of
communication prompted the use of the SVT to
study emotional expression and communication in
patients with brain damage. Some aspects of non-
verbal emotional expression seemed to be intact in
such patients; however, others—including gestures,
pantomime, and mimicry—showed serious deficits
(Goodglass & Kaplan, 1963).
Reasoning that training in pantomime might
allow aphasic patients to employ nonverbal
439
Buck and Miller
communication abilities that may be spared, Duffy,
Duffy, and Pearson (1975) developed a training
program including tests for assessing progress:
the Pantomime Expression Test and Pantomime
Recognition Test. However, experience indicated
that the training did not greatly help the patients,
and impairment in pantomime recognition and/or
expression was found to be strongly correlated with
the degree of verbal impairment. Recognizing that
the SVT may tap a different sort of “nonverbal com-
munication” than intentional pantomime, Buck and
Duffy (1980) studied emotional facial expression
in four patient groups, all older men in a Veterans
Affairs hospital. Aphasic patients showed symp-
toms of expressive aphasia with right hemiplegia
(e.g., paralysis); Right-Hemisphere-Damaged (RHD)
patients showed left hemiplegia without aphasia
symptoms; Parkinson’s Disease patients had a medi-
cal diagnosis of Parkinson’s disease, a prominent
symptom of which is a lack of facial expression due
to damage to the brain’s motor system; and Compar-
ison patients had no known brain damage. Patients
viewed 12 slides in four categories: Familiar People
(hospital personnel known to the patient; e.g., rec-
reation staff, nurses), Scenic, Unusual, and mildly
Unpleasant. Videotapes of the patients’ expressions
were rated by observers who guessed the type of
slide being presented on each trial. There was no
attempt to ask the patients to rate their feelings, so
emotion correlation measures were not taken.
Results indicated that aphasic patients were quite
expressive when viewing the slides, even though
they could not speak. For example, a patient seeing
a picture of a favored nurse smiled broadly, gestured
with the left hand, and vocalized enthusiastically
with strong affective prosody but without utter-
ing any words. Similarly, when viewing a picture
of a starving child, one of the aphasic patients was
moved to tears. In contrast, an RHD patient seeing
a nurse’s picture said in a monotone: “That is nurse
Jones. We love her very much. She is wonderful to
us.” There was thus much emotion in the verbal
content of his statement, but it was said without
Of course, there are cases in which people intentionally display emotions that they do not feel, and such behaviors are clearly intentional. Buck and
2
Van Lear (2002) termed this pseudospontaneous communication: It involves the intentional manipulation of innately based displays, regardless of
whether they are facial expressions, gestures, postures, or vocalizations. From the viewpoint of the sender, the pseudospontaneous display is
voluntary, but if sufficiently convincing, then receivers will respond to it as if it were spontaneous.
440
prosody and his face remained unexpressive. Apha-
sic patients actually had the highest percent correct
scores in the study: not significantly different from
the Comparison patients but significantly higher
than the RHD and Parkinson groups. The latter two
groups did not differ significantly from one another.
The Porch Index of Communicative Ability (PICA;
Porch, 1967), a measure largely of verbal ability, and
pantomime measures were highly intercorrelated
with each other (rs > .90), whereas the percent cor-
rect score had zero-order correlations with the PICA
and pantomime measures.
These results demonstrated that, indeed, distinct
kinds of nonverbal communication are associated
with the SVT measure of sending accuracy on one
hand and pantomime abilities on the other. Buck
and Duffy (1980) suggested that the common fea-
ture in the PICA and pantomime measures is the
intention to communicate; that such communication
involves arbitrary symbols, is learned, and is cultur-
ally patterned; and that the content of the message
is a proposition. A proposition is a statement capable
of logical analysis, and as the most elementary sort
of logical analysis is the test for truth or falsity, so a
proposition is a statement that can be false. In con-
trast, the SVT measures a kind of communication
that is spontaneous (e.g., not intentional) and expres-
sive of a presently existing motivational–emotional
state. The display is a sign of a state, just as smoke
is a sign of fire and as dark clouds are a sign of rain,
so that if the motivational–emotional state is not
present, the sign by definition cannot be present.
Therefore, such communication is nonpropositional
because it cannot be false.2
FACS analysis of patients with brain damage. 
Buck and Duffy’s (1980) conclusion that left-hemi-
sphere damage (LHD) is associated with greater
spontaneous sending accuracy than is RHD was
supported in a study by Borod, Koff, Lorch, and
Nicholas (1985), who additionally found that
frontal RHD produced particularly severe deficits.
These results were questioned, however, by

Mammucari et al. (1988), who used the FACS to
score facial behaviors of RHD and LHD persons.
Their critique is relevant to this chapter because it
illustrates the difference between using an objective
scoring technique as opposed to communication
scores in measuring facial expression.
Mammucari et al. (1988) filmed with a hidden
camera the facial expressions of LHD, RHD, and
comparison patients without brain damage to four
brief films. Two were positive (a fat man chasing a
thin man; a puppy playing with a flower), one was
negative (a surgical toilette of a man with the face
and skull heavily burned), and one was neutral
(waves lapping a beach). Autonomic measures were
also taken. Facial muscular actions were coded using
the FACS, and a computer dictionary converted the
muscle action scores into emotion scores. Rating
scales similar to those of Buck and Duffy (1980) and
Borod et al. (1985) were also employed, but Mam-
mucari et al. (1988) did not employ communication
scores (e.g., whether observers could judge when
the patients were watching positive, negative, or
neutral films). Results indicated that comparison
persons produced more scorable facial expressions
than RHD patients to the unpleasant film, more
scorable facial expressions than LHD and RHD
patients to the neutral film, and more emotional
facial expressions than RHD patients to the unpleas-
ant film. There were no significant FACS differences
between LHD and RHD patients, or between groups
on the rating scales. However, comparison and LHD
groups showed more “aversion movements” to the
unpleasant film than did RHD patients—that is,
they were more likely to avert their gaze from the
unpleasant film than the RHD patients. Gaze aver-
sion was not considered to be an “emotional facial
expression” in the FACS system, but had communi-
cation scores been taken, it is probable that the gaze
aversion would cue observers that the film being
viewed was negative. It was also found that the
autonomic responses of RHD patients were reduced,
particularly to the negative film (Zoccolotti,
Caltagirone, Benedetti, & Gainotti, 1986).
Buck (1990) responded to Mammucari et al.’s
(1988) article, pointing out aspects of their study
that could account for their null results. Specifically,
it was suggested that FACS may be insufficiently
Measuring the Dynamic Stream of Display
sensitive for the measurement of the nuances of
spontaneous expressions that can be important in
communication. Indeed, Vanger and Ellgring (1989)
reported significant facial communication in the
absence of scorable FACS movement, suggesting
that subtle and scarcely visible bulging resulting
from muscle tension, rather than muscle move-
ment, can be detected by communication measures.
Indeed, Mammucari et al. (1988) acknowledged that
the FACS procedure does not engage the capacities
of human beings for gestaltic direct face percep-
tion, but they argued that the analytic and objective
aspects of FACS are superior to subjective scoring
procedures. However, communication measures can
be both objective and take advantage of direct face
perception (Buck, 1990).
Emotional Expression and Communication
in Psychopathology
Measurement of facial expression and communica-
tion by the SVT also has the potential to clarify the
nature of the emotional deficits associated with
psychopathology. There is evidence that the occur-
rence of facial expressive behavior may not always
be associated with accurate communication, and
this fact has important implications for social func-
tioning. The SVT has been used with two kinds of
patient groups: children with EBDs and children
with schizophrenia.
EBDs.  EBDs in children and adolescents is a broad
category of disorders characterized by serious pat-
terns of hostile, aggressive, and disruptive behavior
that are inappropriate for a child’s age and persist
for more than 6 months. They are characterized
by hostility toward authority figures; frequent tan-
trums; harming or threatening to harm themselves,
other people, or pets; damaging or destroying prop-
erty; lying or stealing; doing poorly at school; early
involvement in smoking, drinking, drug use, and/
or sex; and difficulty building and/or maintaining
satisfactory interpersonal relationships with peers
and adults. EBDs may take the form of internal-
izing behaviors (involving depression and anxiety)
or externalizing behaviors (involving violence and
aggression). Children with EBDs tend to act out
rather than holding in their emotions.
441
Buck and Miller

Goldman (1993) noted that studies of the 4

EBD
relationships between nonverbal communication 3.5
Comp
abilities and social competence tend to emphasize
3
receiving rather than sending accuracy and, there-

Communication Score
fore, used the SVT to study relationships between 2.5

sending accuracy and social competence in children 2

with EBDs and age-matched comparison children
1.5
(both groups of children were 8–13 years of age).
1
Participants were 30 children with EBDs (25 boys
and 5 girls) enrolled in psychiatric school pro- 0.5
grams and 30 sex- and age-matched comparison 0
children from a local school system. The children Happy Pleas-Unp Surprised Sad Afraid Angry

served as senders in the SVT, and their expres- Emotion

sive behaviors were judged and rated by groups of

4
observers. EBD
Children viewed eight slides in four content 3.5
Comp
categories: Familiar People (the child and his or her 3
teacher), Unfamiliar People, Unpleasant, and Unusual. Communication Score
2.5
Each slide presentation lasted 20 s, after which the
child rated his or her feelings about the slide (happy, 2

pleasant or unpleasant, surprised, sad, afraid, angry). 1.5

The videotapes were viewed and rated by observer
1
groups, who guessed on each trial the kind of slide
presented and the child’s feelings. The child’s self- 0.5

reported response to slides was measured for each 0

of the six emotions for each of the eight slides. The Happy Pleas-Unp Surprised Sad Afraid Angry

Emotion
expressive response to slides was computed as the
mean observer’s rating of the child on each of the
0.6
seven emotions for each of the eight slides. Emotion EBD

communication accuracy was measured as the cor- 0.5

Comp

relation computed, across the eight slides, of the

Communication Score

child’s self-report of each emotion and the mean of 0.4

the judges’ ratings of the sender on that emotion.
This yielded a specific communication score for each 0.3

emotion—happiness, sadness, fear, anger, surprise,

0.2
and pleasant–unpleasant—for each child.
Results are presented in Figure 17.2. The top 0.1
figure illustrates that there was no significant dif-
ference between the average self-reported emotions 0
Happy Pleas-Unp Surprised Sad Afraid Angry
of the children with EBDs and the comparison
Emotion
children. The middle figure shows that there was
a significant interaction between the children with
EBDs and the comparison children on the observ- Figure 17.2.  Emotion expression and communica-
tion in children with emotional or behavioral disor-
ers’ ratings of their emotional experience computed ders and healthy age-matched comparison children.
across the slides. The children with EBDs were rated Pleas–Unp = pleasant–unpleasant.
as being significantly more negative (sadness, fear,

442

anger) and less positive (happiness, pleasantness)
across the slides. There was no difference for sur-
prise. The lower figure shows that there was a large
difference in communication accuracy between the
children with EBDs and the comparison children:
The children with EBDs were poorer overall in
the communication of all of the emotions (effect
size = .44). Perhaps significantly, the ability of
children with EBDs to communicate anger was not
statistically significant.
Results with the percent correct measure paral-
leled the finding of significant communication defi-
cits among children with EBDs: The average percent
correct was 44% for the children with EBDs and was
60% for the comparison group (effect size = .56).
Also, the percent correct measure was significantly
correlated with the emotion correlation measures of
communication accuracy (average r = .46). More-
over, both communication measures were positively
correlated with teachers’ and parents’ ratings of
children’s social skills and were negatively corre-
lated with ratings of problem behaviors. In particu-
lar, children’s abilities to communicate anger were
positively related to social skills as rated by both
teachers (r = .31) and parents (r = .34) and were
negatively related to problem behaviors as rated by
both teachers (r = −.45) and parents (r = −.40).
These results are of interest because they demon-
strate the importance of measuring emotion send-
ing accuracy in social and emotional functioning.
It is noteworthy that children with EBDs showed
the greatest deficit in communicating anger, and
that the extent to which anger was accurately com-
municated was positively related to social skills and
was negatively related to problem behaviors. This
suggests the possibility that intervention programs
seeking to teach children to control or manage—to
“hold in”—their anger may be off the mark. It may
be that these children are failing to express their
anger at low levels where it can be managed socially,
so that others miss the cues until the child “blows
up” in uncontrollable rage.
Inappropriate affect and schizophrenia.  asserts something about its own assertion, in such
Inappropriate affect has long been cited as charac-
teristic of schizophrenia, but it is unclear precisely
Measuring the Dynamic Stream of Display
what is “inappropriate” in the emotional experi-
ence, expression, and communication of patients
with schizophrenia. Some have argued that nega-
tive symptoms in schizophrenia are associated with
inability to experience emotion, and particularly
positive emotion (anhedonia), which leads to flat
affect and emotional withdrawal (e.g., Limpert &
Amador, 2001). There is evidence of impaired
expression in posing or intentionally enacting facial
expressions of emotion (Kohler, Martin, Milonova,
et al., 2008; Kohler, Martin, Stolar, et al., 2008).
However, apparent deficits in emotional experi-
ence among schizophrenia patients may be illusory.
Earnst and Kring (1999) found that patients with
deficits were less expressive than patients without
deficits and controls when viewing emotional films,
but they did not report experiencing less emotion.
Also, patients with schizophrenia and with flat affect
described happy and sad experiences with less pro-
sodic inflection and fluency, but they used the same
number of words describing pleasure and distress
(Alpert, Rosenberg, Pouget, & Shaw, 2000). Trémeau
et al. (2005) described blunted affect in schizophre-
nia as involving expressive deficits not necessarily
corresponding to deficits in emotional experi-
ence. Kring and Caponigro (2010) concluded that
although patients with schizophrenia may show few
displays of emotion, they tend to report experiencing
strong feelings; furthermore, recent meta-analyses
have concluded that affective experience in schizo-
phrenia is relatively intact (Green & Lee, 2012;
Llerena, Strauss, & Cohen, 2012; Strauss, 2013).
Although there is an established literature on
emotional experience, and some work on emotional
expression and schizophrenia, few studies have
investigated emotional communication, despite the
fact that communication deficits are at the heart
of the classic double-bind (DB) theory of schizo-
phrenia (Bateson, Jackson, Haley, & Weakland,
1956). The DB hypothesis suggested that intractable
paradoxes in family communication play a specific
role in the etiology of schizophrenia. The DB was
defined as a message that asserts something, and
a way that these two assertions are mutually exclu-
sive (Watzlawick, Beavin, & Jackson, 1967). The
443
Buck and Miller
DB was seen to involve paradoxical propositional
communication, but emotional DBs were often cited
as examples (Watzlawick et al., 1967; Watzlawick,
Beavin, & Jackson, 1969).
There is in fact evidence that schizophrenia is
commonly associated with deficits of emotional
communication that could potentially interfere with
the establishment and maintenance of personal rela-
tionships and, furthermore, that disrupted personal
relationships are associated with the intensification
of schizophrenia symptoms. High levels of expressed
emotion (EE) within the family—including hostil-
ity, criticism, and emotional overinvolvement—have
been found to be associated with increased severity
of schizophrenia symptoms (see Wearden, Tarrier,
Barrowclough, Zastowny, & Rahill, 2000). In some
cultures, EE seems to be “schizophrenogenic”: pre-
dicting risk and relapse in schizophrenia (Hooley &
Gotlib, 2000). For example, patients with schizo-
phrenia and with family members high in EE are
more likely to experience clinical relapse than per-
sons from low EE families (Bebbington & Kuipers,
1994; Hahlweg, 2005). Also, there is evidence that
family education programs reducing high EE may
prevent or delay relapse and lower need for psycho-
tropic drugs, and that such intervention in the early
phases of schizophrenia may be effective in prevent-
ing later illness (Wahlberg & Wynne, 2001).
The EE evidence suggests that the development of
schizophrenia may be associated not with problems
of propositional or logical implication but rather
with problems in emotional expression and commu-
nication. However, there are aspects of EE that are
poorly understood: In particular, there is evidence
that EE may have different significance in differ-
ent cultures (Cheng, 2001, 2002; Jenkins & Karno,
1992). Indeed, high levels of EE, and especially emo-
tional overinvolvement, appear to be the norm in
some families (Hashemi & Cochrane, 1999).
It is possible that the critically schizophreno-
genic agent of stress may not be not hostility, criti-
cism, and emotional overinvolvement per se but
rather their detrimental effects on emotional commu-
nication, which may well vary with culture. What is
experienced as stressful EE in one culture may be an
expected and appreciated sign of attention, love, and
affection in another—actually enhancing emotional
444
communication and the growth of positive and
stress-buffering personal relationships. If the criti-
cal variable in EE effects indeed is communication,
studies of emotional communication may cast a
new light on the nature of environmental stress in
schizophrenia, potentially affording opportunities
to conceptualize and understand interactive effects
of emotional communication on the biological con-
comitants of schizophrenia. This, in turn, suggests a
need to develop objective and standard measures of
emotional experience, expression, and communica-
tion in schizophrenia, which could be used to deter-
mine just what is “inappropriate” in the responses of
these persons.
The SVT in schizophrenia.  The SVT has the
potential to shed light on these issues in schizo-
phrenia because, unlike other facial measurement
techniques, it assesses communication. As with
Goldman’s (1993) study of children with EBDs, it
affords the simultaneous analysis of the sender’s
rating of his or her own emotional experience,
observers’ judgments of the emotion experienced by
the sender based on the sender’s display, and, cru-
cially, the relationship between these: whether the
observer can accurately “pick up” the sender’s rated
emotional experience, as revealed in the emotion
correlation measures of communication accuracy.
Using the SVT, Easton (1994) studied 20 patients
with schizophrenia, 11 patients with depression,
and 12 age-matched comparison persons. She dem-
onstrated the reliability and validity of the SVT as a
measure of emotional expression and communica-
tion in these groups. Also, patients with schizophre-
nia rated their emotional responses to the slides
appropriately, suggesting that the subjective experi-
ence of emotion as well as the higher order cognitive
processing that underlies the ability to report one’s
experience of emotion were both reasonably intact
in these patients. However, the expressive behav-
iors of the patients were seen as less appropriate, in
that they appeared to have more negative reactions
to positive slides and positive reactions to negative
slides, and they showed serious deficits in commu-
nication accuracy.
Buck, Cartwright-Mills, Sheehan, and Ross (2014)
carried on this research with expanded schizophrenia

and comparison samples. Fifty patients with schizo-
phrenia (6 females, 48 males) served as senders.
The comparison sample consisted of healthy adults
(35 females, 33 males). Because of the relatively
small number of women in the schizophrenia sample,
formal statistical tests compared only males with
schizophrenia and healthy males. Slide categories
were Familiar, Scenic, Unpleasant, and Unusual, and
senders were rated by groups of observers.
The sender’s self-reported responses, expressive
responses, and emotion communication accuracy
were assessed as in Goldman’s (1993) study. For
self-rated emotional experience, there were small
to moderate differences between males with schizo-
phrenia and comparison males in self-rated emo-
tional experience (mean effect size = .16). There
were no significant main effects between the self-
ratings of males with schizophrenia and comparison
males, but there were significant interactions show-
ing tendencies for males with schizophrenia to rate
their experience as relatively more negative on the
conventionally positive Familiar and Scenic slides
and as relatively less negative on the conventionally
0.8
0.7
0.6
0.5
Mean emotion correlation
0.4
0.3
0.2
0.1
0
AFRAID DISGUST
−0.1
−0.2
Figure 17.3.  Specific emotion communication accuracy in schizophrenia and healthy comparison groups.
PL-UNPL = pleasant–unpleasant ; Schiz-M = schizophrenic male; Schiz-F= schizophrenic female; COMP-M =
comparison male; COMP-F = comparison female.
Measuring the Dynamic Stream of Display
negative Unpleasant slides. Thus, males with schizo-
phrenia rated themselves as feeling slightly to
moderately less appropriate relative to comparison
males. For observer-rated expressive response, there
were moderate differences in observers’ ratings of
males with schizophrenia versus comparison males
(mean effect size = .19). Observers rated males with
schizophrenia as appearing generally less happy
and angrier than comparison males. There were
significant interactions that showed tendencies for
observers to rate males with schizophrenia relatively
more negatively on Familiar Experimenter and Scenic
slides and less negatively on Unpleasant slides. Thus,
males with schizophrenia were rated as feeling mod-
erately less appropriate than comparison males.
For the emotion correlation measures, Figure 17.3
illustrates the mean emotion correlation scores with
data for female participants presented for compari-
son. In the comparison sample, women were
better senders of all emotions: Analysis of the
emotion correlation measures revealed that women
(mean emotion correlation = .63) were better send-
ers than men overall (mean emotion correlation = .47).
Schiz-M
Schiz-F
COMP-M
COMP-F
SAD ANGRY HAPPY PL-UNPL SURP
Specific Emotions
445
Buck and Miller
There were no significant differences between males
and females in the schizophrenia sample, although
as noted, with only six females in the schizophrenia
sample, these data cannot be taken to be conclusive.
As Figure 17.3 illustrates, comparison males
were substantially better senders than males with
schizophrenia in communicating specific emotions
(mean effect size = .44). For individual emotions,
communication accuracy was markedly higher for
comparison males than males with schizophrenia
for fear and disgust, and it was moderately higher
for sadness, anger, happiness, and pleasantness.
For communicating surprise, the difference between
males with schizophrenia and comparison males
was not significant.
Implications.  Again, the ability of the SVT mea-
sures to assess communication yielded findings that
could not be demonstrated using other methods of
facial measurement. Like the children with EBDs,
there were relatively small differences between
patients with schizophrenia and comparison par-
ticipants in self-reported emotions, and their emo-
tional expressions were judged to be moderately
inappropriate—more negative on the positive slides
and more positive on the negative slides. However,
the difference in communication was large. This
finding is consistent with the notion that deficits in
emotional communication may be at the heart of
the effects demonstrated in EE research, and they
are not inconsistent with cultural differences in EE
patterns. It is also consistent with a reinterpreta-
tion of the DB hypothesis in terms of emotional,
as opposed to propositional, communication. This
allows a new approach integrating EE findings
with the DB hypothesis, suggesting that emotional
DBs may indeed be specifically schizophrenogenic
sources of stress. As Kymalainen and Weisman de
Mamani (2008) noted, “what is experienced as
critical . . . to patients from some ethnic groups may
be experienced as non-stressful or even comforting
to patients from other backgrounds” (p. 90).
As the children with EBDs had particular dif-
ficulty communicating anger, the patients with
schizophrenia had particular difficulty communicat-
ing fear. Fear was also not communicated by the 20
patients with schizophrenia in the study by Easton
446
(1994): The average emotion correlation was .00.
The reasons that fear may be poorly communicated
in schizophrenia are not immediately apparent, but
they may relate to patterns of altered emotion pro-
cessing in the amygdala in schizophrenia (e.g., Pan-
kow et al., 2013; Phelps & LeDoux, 2005; Taylor
et al., 2012).
The SVT results in schizophrenia patients can
be compared with those of Hamm, Kohler, Gur,
and Verma (2011), who used an automated FACS
to analyze facial expressions in schizophrenia and
comparison samples. Senders were filmed as they
listened to a verbal recounting of their own emo-
tional experiences involving happiness, sadness,
fear, and anger. The films were analyzed frame-
by-frame for AUs and AU combinations. Flatness
was defined in terms of frames without AUs, and
inappropriateness was defined in terms of frames
containing disqualifying AUs for the target emotion
or qualifying AUs for a different emotion (e.g., AUs
that did not belong or that belonged to a different
emotion). Patients with schizophrenia were found
to display both more flatness and more inappropri-
ateness relative to comparison participants. Hamm
et al. noted that the inappropriate expressions were
ambiguous, and it seems probable that, had commu-
nication measures been employed, those ambiguous
and inappropriate expressions would be associated
with low sending accuracy.
The Segmentation Technique
The SVT has proven to be a reliable and valid para-
digm for measuring emotional communication in
normal and clinical samples, but by itself it does not
specify why those behaviors are important or when
they occur. The former question can be addressed
by facial coding techniques such as the MAX and
FACS, and the latter question can be addressed with
the segmentation technique.
The SVT produces a dynamic representation
of the sender’s stream of spontaneous nonverbal
display. Dynamic facial expressions have a tem-
poral organization that varies from expression to
expression. A problem with identifying effective
cues is that spontaneous behavior occurs in a com-
plex stream, and it is difficult to determine exactly
when events of communicative importance occur.

The temporal organization of spontaneous facial
expression can be studied by behavior unitization or
segmentation techniques used by Newtson (1976).
These can investigate what behaviors constitute
important cues to the receiver, resulting in accurate
(or inaccurate) communication, and the nature of
these cues can be assessed by facial coding systems.
Newtson, Engquist, and Bois (1977) devised a
simple and flexible procedure to assess how naive
observers segment continuous displays. Participants
were given a button attached to an event recorder
and viewed a film of an action sequence, typically
depicting a person performing simple actions such
as walking across a room, clearing a table, shelving
books, and so forth. They were instructed to press
the button when in their judgment one meaningful
action ended and another one began. Newtson et al.
demonstrated that the button presses of different
observers tended to occur together at high-informa-
tion points of change (consensual breakpoints [CPs])
in the stimulus sequence. The still photographs
taken at CPs preserved in a sort of comic-book
form all of the essential information in the behavior
stream so that a person given these photographs
without seeing the film could accurately put them
into their correct temporal sequence and know
exactly what happened in the film. Moreover, when
single frames were removed at CPs as opposed to
nonconsensual points, a new group of observers
viewing the film for the first time was much more
able to detect breakpoints at the former (78% vs.
35%), suggesting that observers naturally increase
their attention at CPs.
The segmentation technique was applied to
film sequences taken during the SVT. Buck, Baron,
Goodman, and Shapiro (1980) showed SVT
sequences showing adults’ and preschool children’s
expressions to adult observers. Adult sequences
were from the CARAT (Buck, 1976), assembled
from videotape segments taken in SVT studies
depicting senders responding to Sexual, Scenic,
Unpleasant, and Unusual slides. CARAT showed
32 sequences selected via item analysis from a
600-item pool. Twenty-five different senders—12
females and 13 males—were represented. Each
sequence was broken into a 3-s preslide period, a
10-s slide period when the sender viewed the slide
Measuring the Dynamic Stream of Display
silently, and an 11-s talk period when the sender
described his or her emotional response. Children’s
expressions were from a children’s form termed
CARAT–C, showing facial expressions of preschool
children to Familiar, Unpleasant, and Unusual slides
from Buck’s (1975) study. CARAT–C was used to
study nonverbal receiving ability in preschool chil-
dren (Alper, Buck, & Dreyer, 1978) and included
18 items 12 s in length, showing three examples
each of girls or boys responding to one of the three
kinds of slides.
In Buck et al.’s (1980) study, observers indicated
every time a meaningful event occurred in the
senders’ expressive behavior; and after each
sequence, they guessed what kind of slide the sender
was watching. Results indicated that adult female
senders generated many more breakpoints than
males on Unpleasant slides, and they tended to gen-
erate more breakpoints on Scenic slides, whereas
adult male senders generated more breakpoints on
Sexual and Unusual slides. For children, boys gen-
erated more breakpoints than girls on Unpleasant
slides, whereas girls elicited more breakpoints than
boys on Unusual slides. Interestingly, the number of
breakpoints generated by the sender was positively
and significantly correlated with sending accuracy
for adult female senders and girls—but not for adult
male senders or boys.
These findings were replicated by Buck, Baron,
and Barrette (1982) using an event recorder to
indicate when meaningful behaviors occurred.
A timing track inserted on CARAT made it possible
to record the 1-s signals on an event recorder, allow-
ing localization of events relative to the stream of
behavior. Half the participants first took CARAT
as a test of receiving ability, after which they were
given segmentation instructions and viewed CARAT
again, this time segmenting it by pressing the but-
ton every time they thought a meaningful event
occurred. The remaining observers were given the
segmentation instructions first, segmented CARAT,
and then viewed it again as a test of receiving abil-
ity. A third “expert” group of graduate and advanced
undergraduate students who were well versed in
nonverbal communication literature and theory, and
all of whom had taken CARAT as a test of receiving
ability, also segmented CARAT.
447
Buck and Miller

The resulting communication data can be com- 40

Pre- Slide period Talk period
bined across observers to assess characteristics of slide
30 Sexual
slides
senders, and they can be combined across senders 20
to assess characteristics of observers. Regarding
10
senders, differences in expressive behavior of male
0
versus female senders during the slide period versus
40
the talk period were examined for the four kinds Scenic
slides
of slides and related to the sending accuracy of the 30

sequence, defined as the average ability of observers 20

Percent Ss Indicating a Point

to guess what kind of slide the sender was viewing. 10
Segmentation measures were number of breakpoints
0
(mean number of button presses) and number of
40
CPs, defined as a 1-s period receiving from expert Unpleasant
slides
receivers a number of button presses greater than one 30

standard deviation over the mean number of presses 20

per second for that sequence (Newtson, 1976).
Three sorts of behavior occurred at CPs: facial
expressions (smiles, frowns, eyebrow lifts), gestures
(shrugs, hand movements, cocking the head), and
body movements (leaning forward or back, squirm-
ing, turning). To analyze whether events occur-
ring at CPs were facial expressions, a new group of
observers viewed CARAT while an experimenter,
using the timing track as a guide, called out “now”
when a CP occurred. Observers judged whether a
facial expression occurred at the CP. Occurrence of
facial expressions was defined as the percentage of
observers indicating that they were certain that a
facial expression occurred at that point.
Results indicated that, despite the procedural
differences between the studies, the number of
breakpoints elicited by the CARAT sequences in
Buck et al.’s (1982) study was closely related to
the number elicited in Buck et al.’s (1980) study
(r = .79). Results also revealed substantial gender
differences in the relationships between segmenta-
tion measures and communication accuracy (Buck
et al., 1974, 1980). Regarding segmentation, adult
female senders were higher than males on the num-
ber of breakpoints, the number of CPs, and the num-
ber of CPs judged to be facial expressions. Thus,
compared with males, females showed more overall
expressive behavior, and observers had greater
consensus regarding locating meaningful behavior
in their streams of expression. Also, of points con-
sensually identified as being meaningful, more were
associated with facial expressions in female than
10
0
40 Male Sender
Female Sender
30
Unusual
20
slides
10
0
0 5 10 15 20
Time in Seconds
Figure 17.4.  Segmentation patterns for male
and female senders to different slide categories.
Ss = subjects. From “The Temporal Organization of
Spontaneous Emotional Expression: A Segmentation
Analysis,” by R. Buck, R. Baron, and D. Barrette,
1982, Journal of Personality and Social Psychology, 42,
p. 512. Copyright 1982 by the American Psychological
Association.
male senders. Thus, not only were there more CPs
in female senders, a greater proportion was associ-
ated with facial expressions. The mean number of
breakpoints for each second of the four slide types is
presented in Figure 17.4. It shows a 1–2-s response
latency for breakpoints after slide onset as well as
peaks of meaningful behavior at about 3–4 s.
In conclusion, a combination of the SVT, seg-
mentation, and facial coding will allow a compre-
hensive and efficient approach to the measurement
and analysis of facial expression and communi-
cation. The SVT produces a dynamic stream of
spontaneous and ecologically valid expression in
response to an emotional stimulus, and it allows the

448

assessment of the sender’s self-reported emotional
response, ratings of the sender’s emotional response
by observers, and emotional communication accu-
racy. The segmentation technique can identify
consensual high-information points in the stream
of spontaneous expression, which may constitute
moments of peak expression. These high informa-
tion points can then be the focus of facial coding by
techniques such as MAX and FACS, enabling their
efficient utilization.
CARAT as a Standard Spontaneous
Emotional Expression Database
The segmentation studies demonstrated the value of
using CARAT as a standard database of spontaneous
expressions. Indeed, as noted, the SVT sequences
that make up CARAT embody the desiderata cited
by Bettadapura (2012) for a standardized spon-
taneous expression database. They contain video
sequences in which participants were unaware of
being filmed, in conditions where spontaneous
expression was encouraged. Sequences are labeled
by both information about the participant’s self-rat-
ings and ratings of observers, and sequences show
a complete temporal pattern including the onset,
apex, and offset of the emotional response.
The latest versions are the digital CARAT—
Spontaneous (CARAT–S, in two equivalent forms;
Buck, Powers, & Derven, 2012) and CARAT:
Spontaneous-Posed-Regulated (CARAT–SPR; Buck,
Powers, & Kapp, 2011). These include sequences
demonstrated to be clearly judged by clinically
normal individuals (90%+ accuracy), and they are
intended for use in studies of brain responses to facial
expressions (e.g., functional magnetic resonance
imaging) as a supplement to the Pictures of Facial
Affect (Ekman, 1993), which employs static posed
expressions (Buck et al., 2012; Buck et al., 2011).
The CARAT–SPR includes not only spontaneous
sequences but also sequences in which participants
posed (e.g., “pretend you are seeing a picture of a
familiar person”) and regulated their facial expres-
sion. For regulated expressions, participants posed
one expression (e.g., viewing a familiar person) while
actually viewing an incompatible image (e.g., a severe
facial injury). These are designed to assess the brain’s
response to spontaneous versus posed versus regulated
Measuring the Dynamic Stream of Display
expressions of, say, responses to a familiar person or an
unpleasant scene. Sequences were chosen from 1,250
digital sequences, and they were edited to standardize
the size of the sender’s face on the screen (only the
head and upper shoulders are visible), to show a uni-
form blue background, and to last approximately 12 s
(see Figure 17.5). The CARAT–S and CARAT–SPR
also present senders only once to avoid familiarity
effects. Senders include both males and females as
well as the ethnic/racial mix typical of a large state
university. Peak expressions for most sequences were
FACS-analyzed by Powers and colleagues.
Expressiveness Versus Communication
Accuracy
One of the aims of this chapter has been to com-
pare emotional display on one hand, and emotional
communication on the other. Display behavior is
an aspect of the individual sender, whereas com-
munication involves the function of the display as
analyzed by Darwin (1872/1998) and others: the
social communication of the sender’s emotional
state to interaction partners. Thus, if a display
is to function appropriately, it must express the
motivational–emotional state of the sender in a form
accessible to, usable by, and meaningful to interac-
tion partners.
Emotion is unique in that different aspects of its
expression: physiological responses (Emotion I),
displays (Emotion II), and subjective experience
(Emotion III) are differently accessible to the
responder and the interaction partner (Buck, 1984,
1999). The responder has “sole access” to Emotion
III subjective experience, the interaction partner
has the best access to Emotion II display behavior,
and neither have very good access to Emotion I
physiological arousal (Buck, 1999). The way that
a child learns to label and understand Emotion III
experience is through social biofeedback from the
partner’s response to the display. This requires emo-
tional communication. Individuals learn to label
and understand emotion based on social experi-
ence involving emotional communication, which
naturally leads to social biofeedback, emotional
education, emotional competence, and social bioreg-
ulation. If a sender is insufficiently expressive, or too
expressive, the interaction partner will be confused
449
Buck and Miller

Figure 17.5.  Peak spontaneous expressions from Communication of Affect Receiving Ability Test (CARAT)
items. Top = familiar person; Center = unpleasant; Bottom = unusual.

about the sender’s actual emotional state, and social competence, will naturally result in deficits in label-
biofeedback will be compromised (see Figure 17.6). ing and understanding Emotion III feelings and
The resulting lack of social biofeedback, and desires. Nemiah and Sifneos (1970) coined the
consequent deficits in emotional education and term alexithymia (no words for mood) to describe

450
 Measuring the Dynamic Stream of Display

Emotion III Accessible

Subjective experience of
feelings and desires to self Social
biofeedback

Inhibition

Other persons

Emotion II Affords EMOTION EDUCATION

to others leading to the CONTROL of expression
Expressive Display: Accessible
Facial expressions,
postures, pheromones
and EMOTION COMPETENCE
Autonomic,
Endocrine,
Emotion I Immune System
Responding

Not accessible

Inhibition Inhibition

Other Other
persons persons

Figure 17.6.  Social biofeedback (top) and the consequences of too little (bottom left) and too much (bottom
right) facial expression. Inhibition is excessive in the bottom left figure and is insufficient in the bottom right figure.
From “The Biological Affects: A Typology,” by R. Buck, 1999, Psychological Review, 106, p. 307. Copyright 1999 by
the American Psychological Association.

emotional expression in patients with psycho-
somatic disease. Such persons cannot describe
emotions in words, and they show high levels of
autonomic nervous system (ANS) responses. Impul-
sive persons, with low ANS responses, also have dif-
ficulty describing feelings. The common factor may
be poor emotion communication during develop-
ment and, therefore, deficits in social biofeedback,
emotional education and emotional competence.
The foregoing data on senders with EBDs
and schizophrenia demonstrated that emo-
tional expression does not guarantee emotional

451
Buck and Miller

HIGH FUTURE DIRECTIONS: ANALYZING

Hypoexpression Normal Hyperexpression SPONTANEOUS FACIAL EXPRESSION AND
Communication Expression COMMUNICATION
(Sending) High
Accuracy Communication
Inhibitory Accuracy Disinhibitory The SVT is a relatively simple, straightforward, and
Psychopathology Psychopathology
Low comprehensively tested method for measuring emo-
Psychopathology
LOW tional communication via dynamic, spontaneous,
LOW Expressiveness HIGH and ecologically valid facial displays. It affords the
study of emotional communication—a dyadic pro-
Alexithymia Hypoexpressive Normal Hyperexpressive cess involving both sender and receiver. At the same
time, the SVT affords in the CARAT a standardized
Inhibition Inhibited Disinhibited
database of such displays not influenced by any par-
Expressive ticular analysis technique. It has been used to study
Style Internalizing Externalizing
emotional communication in a variety of participant
Fight or Flight HIGH LOW groups, including adults, children, and clinical
Response
groups.
Emotional LOW HIGH LOW We have seen that, when it comes to distinguish-
Education/
Competence ing between facial expression and facial communi-
cation, the principle of the “golden mean” applies:
Figure 17.7.  Relationship between expressiveness A moderate degree of display favors accurate com-
and sending accuracy. From Nonverbal Communication munication in human beings. Communication can
(p. 417) by J. A. Hall and M. L. Knapp, 2013, Boston, be disrupted by too little but also too much expres-
MA: de Gruyter Mouton. Copyright 2013 by de Gruyter
Mouton. Reprinted with permission. sion. Both hypo- and hyperexpression can disrupt
accurate emotional communication, potentially
contributing to deficits in emotional education
and consequent alexithymia: “no words for mood”
(Nemiah & Sifneos, 1970).
communication. Sending accuracy and expressive- The SVT therefore can answer the unresolved
ness may actually show a curvilinear relationship issues that confront the literature in the automatic
(see Figure 17.7). This is due to the effects of the analysis of facial expression: It involves spontane-
extremes of expressiveness on the accuracy of ous, dynamic, and ecologically valid facial expres-
emotional communication during development. sions; it allows a bottom-up analysis where response
Sending accuracy will tend to be greatest (all categories can emerge from the data; it is sensitive
else equal) at moderate levels of expressiveness, to communicative aspects of the expression; and
and it will be low at the extremes of low or high it generates a facial database independent of the
expressiveness, as Figure 17.7 indicates. This has emotion categories used to construct the analysis
important implications for understanding the role technique.
of emotional expression and communication in The foregoing analysis suggests a strategy for
personality and psychopathology. Low expressive- the comprehensive analysis of facial expression
ness may be related to hypoexpressive alexithymia, and communication that addresses some of the
which would be associated with extreme introver- unanswered issues in the literature on coding facial
sion, internalizing, high ANS responding, and low expressions of emotion. First, communication
sending accuracy. In contrast, hyperexpressive- scores assess the functional significance of expres-
ness may be related to hyperexpressive alexithymia, sive behavior in organizing social behavior. Based on
which would be associated with extreme extraver- the ability of observers to make correct judgments
sion and externalizing, low ANS responding, and about the sender, they are objective, concrete mea-
low sending accuracy. sures that can be evaluated relative to chance; they

452

do not require trained raters; and they have been
associated with a wide variety of characteristics in
senders known to be critical to social and emotional
functioning. Sending accuracy scores are reliable
and valid, and they assess a stable and important
aspect of personality. Sending accuracy indicates
the presence and functional utility of an expressive
display but does not by itself specify the nature or
timing of the behaviors upon which the display is
based, which is the forte of facial coding techniques.
The next step in the analysis of emotional expres-
siveness is to determine when important communi-
cative displays occur. The segmentation procedure
is an efficient and effective way to do this. Segmen-
tation scores also are objective, concrete measures
that reflect the consensual responding on the part of
observers as to when important points occur in the
stream of expression. Observers’ instructions can be
modified to refer to specific kinds of expression—for
example, whenever the sender looks happy, sad, or
angry. One might use the segmentation technique
for each kind of emotion on which significant com-
munication was demonstrated.
Neither communication nor segmentation scores
by themselves indicate the nature of the expressive
displays underlying the scores—that is precisely
where facial coding systems including AEDS come
in. This involves a third step in the analysis of emo-
tional expression and communication: the analysis
of the nature of the behavior by instruments such
as MAX, FACS, facial EMG, and their technological
offspring AEDS. Indeed, one is not confined to the
analysis only of facial expression. Notation systems
for measuring objectively body movement (i.e.,
dance notation techniques), paralanguage, or any-
thing else that seems appropriate may be employed.
However, arguably, the critical variable of interest
in the analysis of expressive behavior is not only the
display behavior of the individual sender divorced
from the functional communicative context. The
ultimate function of the display, its reason for being,
is communication.
References
Affectiva. (2011). Exploring the emotion classifiers behind
Affdex facial coding. Retrieved from http://www.
affdex.com/clients/affdex-resources
Measuring the Dynamic Stream of Display
Ahn, S. J., Bailenson, J., Fox, J., & Jabon, M. (2009).
Using automated facial expression analysis for
emotion and behavior prediction. In Proceedings of
the 95th Annual National Communication Association
Conference (pp. 349–369). Retrieved from https://
vhil.stanford.edu/pubs/2010/ahn-hemm-facial-
expression.pdf
Ahn, S. J., Bailenson, J., Fox, J., & Jabon, M. (2010).
Using automated facial expression analysis for
emotion and behavior prediction. In K. Döveling,
C. von Scheve, & E. Konijn (Eds.), Handbook of
emotions and mass media (pp. 349–369). London,
England: Routledge.
Ahn, S. J., Jabon, M. E., & Bailenson, J. N. (2008, May).
Facial expressions as predictors of online buying
intention. Paper presented at the 58th Annual
International Communication Association
Conference, Montreal, Quebec, Canada.
Ahn, S. J., Jabon, M. E., & Bailenson, J. N. (2009, May).
Judging a book by the cover: Using facial expressions
to predict performance. Paper presented at the 59th
Annual International Communication Association
Conference, Chicago, IL.
Alper, S., Buck, R., & Dreyer, A. (1978). Nonverbal
sending accuracy and receiving ability in preschool
children. Unpublished manuscript, University of
Connecticut, Storrs.
Alpert, M., Rosenberg, S. D., Pouget, E. R., & Shaw, R. J.
(2000). Prosody and lexical accuracy in flat affect
schizophrenia. Psychiatry Research, 97, 107–118.
http://dx.doi.org/10.1016/S0165-1781(00)00231-6
Bailenson, J. N., Pontikakis, E. D., Mauss, I. B., Gross, J. J.,
Jabon, M. E., Hutcherson, C. A. C., . . . John, O.
(2008). Real-time classification of evoked emotions
using facial feature tracking and physiological
responses. International Journal of Human–
Computer Studies, 66, 303–317. http://dx.doi.org/
10.1016/j.ijhcs.2007.10.011
Baron, R. (1981). Mechanisms of human facial
recognition. International Journal of Man–Machine
Studies, 15, 137–178. http://dx.doi.org/10.1016/
S0020-7373(81)80001-6
Bateson, G., Jackson, D., Haley, J., & Weakland, J.
(1956). Toward a theory of schizophrenia. Behavioral
Science, 1, 251–264. http://dx.doi.org/
10.1002/bs.3830010402
Bebbington, P., & Kuipers, L. (1994). The predictive
utility of expressed emotion in schizophrenia:
An aggregate analysis. Psychological Medicine, 24,
707–718. http://dx.doi.org/10.1017/S00332917
00027860
Bettadapura, V. (2012). Face expression recognition and
analysis: The state of the art. Retrieved from http://
arxiv.org/ftp/arxiv/papers/1203/1203.6722.pdf
453
Buck and Miller
Birdwhistell, R. L. (1970). Kinesics and context.
Philadelphia: University of Pennsylvania Press.
Borod, J. C., Koff, E., Lorch, M. P., & Nicholas, M.
(1985). Channels of emotional expression in
patients with unilateral brain damage. Archives of
Neurology, 42, 345–348. http://dx.doi.org/10.1001/
archneur.1985.04060040055011
Buck, R. (1975). Nonverbal communication affect in
children. Journal of Personality and Social Psychology,
31, 644–653. http://dx.doi.org/10.1037/h0077071
Buck, R. (1976). A test of nonverbal receiving ability:
Preliminary studies. Human Communication Research,
2, 162–171. http://dx.doi.org/10.1111/
j.1468-2958.1976.tb00708.x
Buck, R. (1977). Nonverbal communication of affect in
preschool in children: Relationships with personality
and skin conductance. Journal of Personality and
Social Psychology, 35, 225–236. http://dx.doi.org/
10.1037/0022-3514.35.4.225
Buck, R. (1984). The communication of emotion. New
York, NY: Guilford Press.
Buck, R. (1990). Using FACS vs. communication scores
to measure spontaneous facial expression of emotion
in brain-damaged patients: A reply to Mammucari
et al. (1988). Cortex, 26, 275–280. http://dx.doi.org/
10.1016/S0010-9452(13)80357-7
Buck, R. (1999). The biological affects: A typology.
Psychological Review, 106, 301–336. http://dx.doi.
org/10.1037/0033-295X.106.2.301
Buck, R., Baron, R., & Barrette, D. (1982). The temporal
organization of spontaneous emotional expression:
A segmentation analysis. Journal of Personality and
Social Psychology, 42, 506–517. http://dx.doi.org/
10.1037/0022-3514.42.3.506
Buck, R., Baron, R., Goodman, N., & Shapiro, B. (1980).
The unitization of spontaneous nonverbal behavior
in the study of emotion communication. Journal
of Personality and Social Psychology, 39, 522–529.
http://dx.doi.org/10.1037/0022-3514.39.3.522
Buck, R., Cartwright-Mills, J., Sheehan, M., & Ross,
E. (2014). Expressed emotion and the double-bind:
Experience, expression, and communication of
specific emotions in schizophrenia. Manuscript in
preparation.
Buck, R., & Duffy, R. J. (1980). Nonverbal communication
of affect in brain-damaged patients. Cortex, 16,
351–362. http://dx.doi.org/10.1016/S0010-9452(80)
80037-2
Buck, R., & Miller, M. (2014). Beyond facial expression:
Spatial distance as a factor in the communication of
discrete emotions. In A. Kostić & D. Chadee (Eds.),
The social psychology of nonverbal communication (pp.
173–197). New York, NY: Palgrave Macmillan.
454
Buck, R., Miller, R. E., & Caul, W. F. (1974). Sex,
personality, and physiological variables in the
communication of affect via facial expression.
Journal of Personality and Social Psychology, 30,
587–596. http://dx.doi.org/10.1037/h0037041
Buck, R., Powers, S. R., & Derven, R. (2012, October).
Introducing the Communication of Affect Receiving
Ability Test (CARAT): Ecologically valid spontaneous
and dynamic facial displays for social neuroscience.
Paper presented at the annual meeting of the Society
for Social Neuroscience, New Orleans, LA.
Buck, R., Powers, S. R., & Kapp, W. (2011, May).
Developing the Communication of Affect Receiving
Ability Test: Spontaneous-Posed-Regulated
(CARAT–SPR). Paper presented at the 61st
International Communication Association
Convention, Boston, MA.
Buck, R., & Van Lear, C. A. (2002). Verbal and nonverbal
communication: Distinguishing symbolic, spontaneous,
and pseudo-spontaneous nonverbal behavior. Journal
of Communication, 52, 522–541. http://dx.doi.org/
10.1111/j.1460-2466.2002.tb02560.x
Buck, R. W., & Powers, S. R. (2013). Encoding and
display: A developmental-interactionist model
of nonverbal sending accuracy. In J. A. Hall &
M. L. Knapp (Eds.), Nonverbal communication
(pp. 403–440). http://dx.doi.org/10.1515/97831102
38150.403
Buck, R. W., Savin, V. J., Miller, R. E., & Caul, W. F.
(1972). Communication of affect through facial
expressions in humans. Journal of Personality and
Social Psychology, 23, 362–371. http://dx.doi.
org/10.1037/h0033171
Cheng, A. T. (2001). Case definition and culture: Are
people all the same? British Journal of Psychiatry,
179, 1–3. http://dx.doi.org/10.1192/bjp.179.1.1
Cheng, A. T. (2002). Expressed emotion: A cross-
culturally valid concept? British Journal of
Psychiatry, 181, 466–467. http://dx.doi.org/10.1192/
bjp.181.6.466
Cohn, J. F., & Katz, G. S. (1998, September). Bimodal
expression of emotion by face and voice. Paper
presented at the Sixth ACM International Multimedia
Conference, Bristol, England.
Condon, W. S., & Ogston, W. D. (1966). Sound film
analysis of normal and pathological behavior
patterns. Journal of Nervous and Mental Disease,
143, 338–347. http://dx.doi.org/10.1097/00005053-
196610000-00005
Darwin, C. (1998). The expression of the emotions in man
and animals. New York, NY: Oxford University Press.
(Original work published 1872)
Dimberg, U. (1982). Facial reactions to facial expressions.
Psychophysiology, 19, 643–647. http://dx.doi.
org/10.1111/j.1469-8986.1982.tb02516.x

Duffy, R. J., Duffy, J. R., & Pearson, K. L. (1975).
Pantomime recognition in aphasics. Journal of Speech
and Hearing Research, 18, 115–132. http://dx.doi.
org/10.1044/jshr.1801.115
Earnst, K. S., & Kring, A. M. (1999). Emotional responding
in deficit and non-deficit schizophrenia. Psychiatry
Research, 88, 191–207. http://dx.doi.org/10.1016/
S0165-1781(99)00083-9
Easton, C. (1994). Expression and communication
of emotion in schizophrenic patients (Doctoral
dissertation). Retrieved from http://digitalcommons.
uconn.edu/dissertations/AAI9503461
Ekman, P. (1993). Pictures of Facial Affect (POFA).
Retrieved from http://www.paulekman.com/product-
category/research-products
Ekman, P., & Friesen, W. (1975). Pictures of Facial
Affect. Palo Alto, CA: Consulting Psychologists Press.
Ekman, P., & Friesen, W. (1977). Manual for the Facial
Action Coding System. Palo Alto, CA: Consulting
Psychologists Press.
Ekman, P., & Friesen, W. V. (1978). Facial Action
Coding System: Investigator’s guide. Palo Alto, CA:
Consulting Psychologists Press.
Ekman, P., & Friesen, W. V. (1982). Felt, false, and
miserable smiles. Journal of Nonverbal Behavior, 6,
238–252. http://dx.doi.org/10.1007/BF00987191
Ekman, P., Friesen, W. V., Irwin, W., & Rosenberg, E.
(2014). Introduction to EMFACS. Retrieved from
http://face-and-emotion.com/dataface/facs/emfacs_
intro_authors.html
Ekman, P., Friesen, W. V., & Tomkins, S. S. (1971).
Facial affect scoring technique: A first validation
study. Semiotica, 3, 37–58. http://dx.doi.org/10.1515/
semi.1971.3.1.37
Essa, I. A., & Pentland, A. P. (1997). Coding, analysis,
interpretation, and recognition of facial expressions.
IEEE Transactions on Pattern Analysis and Machine
Intelligence, 19, 757–763. http://dx.doi.org/10.1109/
34.598232
Fasel, B., & Luettin, J. (2003). Automatic facial
expression analysis: A survey. Pattern Recognition, 36,
259–275. http://dx.doi.org/10.1016/S0031-3203(02)
00052-3
Fast, J. (1970). Body language. New York, NY: Simon &
Schuster.
Fowles, D. C. (1980). The three arousal model:
Implications of Gray’s two-factor learning theory for
heart rate, electrodermal activity, and psychopathy.
Psychophysiology, 17, 87–104. http://dx.doi.
org/10.1111/j.1469-8986.1980.tb00117.x
Goldman, C. (1993). The relationship of emotional
communication to social competence: The role of
communication accuracy in behaviorally-disordered
Measuring the Dynamic Stream of Display
children’s social functioning (Unpublished doctoral
dissertation). University of Connecticut, Storrs.
Goodglass, H., & Kaplan, E. (1963). Disturbance of
gesture and pantomime in aphasia. Brain: A Journal
of Neurology, 86, 703–720. http://dx.doi.org/
10.1093/brain/86.4.703
Green, M. F., & Lee, J. (2012). Neural bases of emotional
experience versus perception in schizophrenia.
Biological Psychiatry, 71, 96–97. http://dx.doi.org/
10.1016/j.biopsych.2011.11.005
Hahlweg, K. (2005). The shaping of individuals’ mental
structures and dispositions by others: Findings from
research on expressed emotion. Interaction Studies, 6,
131–144. http://dx.doi.org/10.1075/is.6.1.10hah
Hamm, J., Kohler, C. G., Gur, R. C., & Verma, R. (2011).
Automated Facial Action Coding System for dynamic
analysis of facial expressions in neuropsychiatric
disorders. Journal of Neuroscience Methods, 200,
237–256. http://dx.doi.org/10.1016/j.jneumeth.
2011.06.023
Hashemi, A. H., & Cochrane, R. (1999). Expressed
emotion and schizophrenia: A review of studies
across cultures. International Review of Psychiatry, 11,
219–224.
Hjortsjö, C.-H. (1969). Människans ansikte och det
mimiska språket [Man’s face and mimic language].
Lund, Sweden: Studentlitteratur.
Hooley, J. M., & Gotlib, I. H. (2000). A diathesis-
stress conceptualization of expressed emotion and
clinical outcome. Applied and Preventive Psychology,
9, 135–151. http://dx.doi.org/10.1016/S0962-
1849(05)80001-0
Izard, C. E. (1979). The maximally discriminative facial
action coding system (MAX). Newark: University of
Delaware Instructional Resources Center.
Izard, C. E., & Dougherty, L. M. (1982). Two
complementary systems for measuring facial
expressions in infants and children. In C. E. Izard &
P. B. Read (Eds.), Measuring emotions in infants and
children (pp. 97–126). New York, NY: Cambridge
University Press.
Izard, C. E., Dougherty, T. M., & Hembree, E. A. (1983).
A System for Identifying Affect Expressions by Holistic
Judgments (AFFEX). Newark: University of Delaware,
Instructional Resources Center.
Jabon, M. E., Bailensen, J. N., Pontikakis, E. D.,
Takayama, L., & Nass, C. (2011). Facial expression
analysis for predicting unsafe driving behavior.
IEEE Pervasive Computing, 10, 84–95. http://dx.doi.
org/10.1109/MPRV.2010.46
Jenkins, J. H., & Karno, M. (1992). The meaning of
expressed emotion: Theoretical issues raised
by cross-cultural research. American Journal of
Psychiatry, 149, 9–21. http://dx.doi.org/10.1176/
ajp.149.1.9
455
Buck and Miller
Kahou, S. E., Pal, C., Bouthillier, X., Froumenty, P.,
Gülçehre, Ç., Memisevic, R., . . . Wu, Z. (2013).
Combining modality specific deep neural networks
for emotion recognition in video. In Proceedings
of the 15th ACM on International conference on
multimodal interaction (pp. 543–550). http://dx.doi.
org/10.1145/2522848.2531745
Kanade, T. (1973). Picture processing system by computer
complex and recognition of human faces (Unpublished
doctoral dissertation). Tokyo University, Japan.
Kaufman, G. J., & Breeding, K. J. (1976). The automatic
recognition of human faces from profile silhouettes.
IEEE Transactions on Systems, Man and Cybernetics,
SMC-6, 113–121. http://dx.doi.org/10.1109/TSMC.
1976.5409181
Kendon, A. (1970). Movement coordination in social
interaction: Some examples described. Acta
Psychologica, 32, 101–125. http://dx.doi.org/
10.1016/0001-6918(70)90094-6
Kohler, C. G., Martin, E. A., Milonova, M., Wang, P.,
Verma, R., Brensinger, C. M., . . . Gur, R. C. (2008).
Dynamic evoked facial expressions of emotions in
schizophrenia. Schizophrenia Research, 105, 30–39.
http://dx.doi.org/10.1016/j.schres.2008.05.030
Kohler, C. G., Martin, E. A., Stolar, N., Barrett, F. S.,
Verma, R., Brensinger, C., . . . Gur, R. C. (2008). Static
posed and evoked facial expressions of emotions in
schizophrenia. Schizophrenia Research, 105, 49–60.
http://dx.doi.org/10.1016/j.schres.2008.05.010
Kring, A. M., & Caponigro, J. M. (2010). Emotion in
schizophrenia: Where feeling meets thinking. Current
Directions in Psychological Science, 19, 255–259.
http://dx.doi.org/10.1177/0963721410377599
Kymalainen, J. A., & Weisman de Mamani, A. G. (2008).
Expressed emotion, communication deviance, and
culture in families of patients with schizophrenia:
A review of the literature. Cultural Diversity and
Ethnic Minority Psychology, 14, 85–91. http://dx.doi.
org/10.1037/1099-9809.14.2.85
Leeds-Hurwitz, W. (1987). The social history of The
Natural History of an Interview: A multidisciplinary
investigation of social communication. Research on
Language and Social Interaction, 20, 1–51. http://
dx.doi.org/10.1080/08351818709389274
Limpert, C., & Amador, X. F. (2001). Negative symptoms
and the experience of emotion. In R. S. E. Keefe &
J. P. McElroy (Eds.), Negative symptom and cognitive
deficit treatment response in schizophrenia (pp.
111–137). Washington, DC: American Psychiatric
Press.
Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M.,
Movellan, J., & Bartlett, M. (2011). The Computer
Expression Recognition Toolbox (CERT). In
Proceedings of the 2011 IEEE International Conference
on Automatic Face and Gesture Recognition and
456
Workshops (pp. 298–305). Retrieved from http://
mplab.ucsd.edu/∼marni/Projects/CERT.htm
Llerena, K., Strauss, G. P., & Cohen, A. S. (2012).
Looking at the other side of the coin: A meta-analysis
of self-reported emotional arousal in people with
schizophrenia. Schizophrenia Research, 142, 65–70.
http://dx.doi.org/10.1016/j.schres.2012.09.005
Malatesta, C. Z., & Haviland, J. M. (1982). Measuring
change in infant emotional expressivity: Two
approaches applied in longitudinal investigation. In
C. E. Izard & P. B. Read (Eds.), Measuring emotions
in infants and children (pp. 51–74). New York, NY:
Cambridge University Press.
Mammucari, A., Caltagirone, C., Ekman, P., Friesen, W. V.,
Gainotti, G., Pizzamiglio, L., & Zoccolotti, P. (1988).
Spontaneous facial expression of emotions in brain-
damaged patients. Cortex, 24, 521–533. http://dx.doi.
org/10.1016/S0010-9452(88)80046-7
Matsumoto, D., & Ekman, P. (1988). Japanese and
Caucasian facial expressions of emotion [Slides].
San Francisco, CA: Intercultural and Emotion
Research Laboratory, Department of Psychology,
San Francisco State University.
Matsumoto, D., & Hwang, H. C. (2014). Judgments of
subtle facial expressions of emotion. Emotion, 14,
349–357. http://dx.doi.org/10.1037/a0035237
McArthur, L. A., & Baron, R. (1983). Toward an
ecological theory of social perception. Psychological
Review, 90, 215–238. http://dx.doi.org/10.1037/
0033-295X.90.3.215
McQuown, N., Bateson, G., Birdwhistell, R., Brosin, H., &
Hockett, C. (1971). The natural history of an
interview [Microfilm collection of manuscripts on
cultural anthropology, ser. XV, 95–97]. Chicago, IL:
University of Chicago Library.
Miller, R. E. (1967). Experimental approaches to the
physiological and behavioral concomitants of
affective communication in rhesus monkeys. In
S. A. Altmann (Ed.), Social communication among
primates (pp. 125–134). Chicago, IL: University of
Chicago Press.
Nemiah, J. C., & Sifneos, P. E. (1970). Psychosomatic
illness: A problem in communication. Psychotherapy
and Psychosomatics, 18, 154–160. http://dx.doi.org/
10.1159/000286074
Newtson, D. (1976). Task and observer skill factors in
accuracy of assessment of performance (ARI Technical
Report TR-77-A7). Retrieved from http://oai.dtic.mil/
oai/oai?verb=getRecord&metadataPrefix=html&ide
ntifier=ADA038679
Newtson, D., Engquist, G., & Bois, J. (1977). The
objective basis of behavior units. Journal of
Personality and Social Psychology, 35, 847–862.
http://dx.doi.org/10.1037/0022-3514.35.12.847

Nicolaou, M. A., Gunes, H., & Pantic, M. (2011).
Continuous prediction of spontaneous affect from
multiple cues and modalities in valence-arousal
space. IEEE Transactions on Affective Computing, 2,
92–105. http://dx.doi.org/10.1109/T-AFFC.2011.9
Noldus Information Technology. (n.d.). FaceReader 6
[Computer software]. Retrieved from http://www.
noldus.com/human-behavior-research/products/
facereader
nViso. (2011, September). Say it to my face! Applying
facial imaging to understanding consumer
emotional response. Paper presented at the 2011
AMSRS National Conference, Sydney, Australia.
Retrieved from http://www.nviso.ch/assets/pdf/
nViso_AMSRS_2011_3D_Facial_Imaging_Best_
Presentation_Award.pdf
Öhman, A. (2002). Automaticity and the amygdala:
Nonconscious responses to emotional faces. Current
Directions in Psychological Science, 11, 62–66. http://
dx.doi.org/10.1111/1467-8721.00169
OMRON. (n.d.) OKAO Vision [Computer software].
Retrieved from http://www.omron.com/r_d/coretech/
vision/okao.html
Pankow, A., Friedel, E., Sterzer, P., Seiferth, N., Walter, H.,
Heinz, A., & Schlagenhauf, F. (2013). Altered
amygdala activation in schizophrenia patients
during emotion processing. Schizophrenia Research,
150, 101–106. http://dx.doi.org/10.1016/j.schres.
2013.07.015
Pantic, M., & Rothkrantz, L. J. M. (2000). Automatic
analysis of facial expressions: The state of the art.
IEEE Transactions on Pattern Analysis and Machine
Intelligence, 22, 1424–1445.
Pantic, M., Valstar, M., Rademaker, R., & Maat, L.
(2005). Web-based database for facial expression
analysis. In Proceedings of the IEEE International
Conference on Multimedia and Expo, 2005 (Accession
No. 8893257). http://dx.doi.org/10.1109/
ICME.2005.1521424
Phelps, E. A., & LeDoux, J. E. (2005). Contributions of
the amygdala to emotion processing: From animal
models to human behavior. Neuron, 48, 175–187.
http://dx.doi.org/10.1016/j.neuron.2005.09.025
Porch, B. E. (1967). Porch Index of Communicative Ability.
Palo Alto, CA: Consulting Psychologists Press.
Powers, S. R., Rauh, C., Henning, R. A., Buck, R. W., &
West, T. V. (2011). The effect of video feedback
delay on frustration and emotion communication
accuracy. Computers in Human Behavior, 27,
1651–1657. http://dx.doi.org/10.1016/j.chb.
2011.02.003
Proyer, R. T., Meier, L. E., Platt, T., & Ruch, W. (2013).
Dealing with laughter and ridicule in adolescence:
Relations with bullying and emotional responses.
Measuring the Dynamic Stream of Display
Social Psychology of Education, 16, 399–420. http://
dx.doi.org/10.1007/s11218-013-9221-y
Samal, A., & Iyengar, P. A. (1992). Automatic recognition
and analysis of human faces and facial expression: A
survey. Pattern Recognition, 25, 65–77. http://dx.doi.
org/10.1016/0031-3203(92)90007-6
Sapir, E. (1921). Language: An introduction to the study of
speech. New York, NY: Harcourt.
Scheflen, A. E., & Scheflen, A. (1972). Body language and
the social order: Communication as behavioral control.
Englewood Cliffs, NJ: Prentice Hall.
Sebe, N., Lew, M. S., Sun, Y., Cohen, I., Gevers, T., &
Huang, T. S. (2007). Authentic facial expression
analysis. Image and Vision Computing, 25, 1856–1863.
http://dx.doi.org/10.1016/j.imavis.2005.12.021
Senechal, T., Rapp, V., Salam, H., Seguier, R., Bailly, K., &
Prevost, L. (2011). Combining AAM coefficients
with LGBP histograms in the multi-kernel
SVM framework to detect facial action units.
In Proceedings of the 2011 IEEE International
Conference on Automatic Face and Gesture
Recognition and Workshops (pp. 860–865).
http://dx.doi.org/10.1109/FG.2011.5771363
Strauss, G. P. (2013). The emotion paradox of anhedonia
in schizophrenia: Or is it? Schizophrenia Bulletin, 39,
247–250. http://dx.doi.org/10.1093/schbul/sbs192
Suneetha, J., & Sandhy Arani, K. (2013). An overview of
automatic facial expression recognition. International
Journal of Engineering Sciences Research, 4,
1330–1335.
Suwa, M., Sugie, N., & Fujimora, K. (1978). A preliminary
note on pattern recognition of human emotional
expression. In Proceedings of the Fourth International
Joint Conference on Pattern Recognition (pp.
408–410). Piscataway, NJ: Institute of Electrical and
Electronics Engineers.
Tao, H., & Huang, T. S. (2002). Visual estimation and
compression of facial motion parameters—Elements of
a 3D model-based video coding system. International
Journal of Computer Vision, 50, 111–125.
Taylor, S. F., Kang, J., Brege, I. S., Tso, I. F., Hosanagar, A., &
Johnson, T. D. (2012). Meta-analysis of functional
neuroimaging studies of emotion perception
and experience in schizophrenia. Biological
Psychiatry, 71, 136–145. http://dx.doi.org/10.1016/j.
biopsych.2011.09.007
Terzopoulos, D., & Waters, K. (1990). Physically-based
facial modelling, analysis, and animation. Journal of
Visualization and Computer Animation, 1, 73–80.
Tian, Y. L., Kanade, T., & Cohn, J. F. (2000). Dual-state
parametric eye tracking. In Proceedings of the Fourth
IEEE International Conference on Automatic Face and
Gesture Recognition (pp. 110–115). http://dx.doi.
org/10.1109/AFGR.2000.840620
457
Buck and Miller
Tomkins, S. S. (1962). Affect, imagery, and consciousness:
Vol. 1. The positive affects. New York, NY: Springer.
Tomkins, S. S. (1963). Affect, imagery, and consciousness:
Vol. 2. The negative affects. New York, NY: Springer.
Tomkins, S. S. (1991). Affect, imagery, and consciousness:
Vol. 3. The negative affects: Fear and anger. New York,
NY: Springer.
Tomkins, S. S., & McCarter, R. (1964). What and where
are the primary affects? Some evidence for a theory.
Perceptual and Motor Skills, 18, 119–158. http://
dx.doi.org/10.2466/pms.1964.18.1.119
Trémeau, F., Malaspina, D., Duval, F., Corrêa, H.,
Hager-Budny, M., Coin-Bariou, L., . . . Gorman, J.
M. (2005). Facial expressiveness in patients with
schizophrenia compared to depressed patients and
nonpatient comparison subjects. American Journal of
Psychiatry, 162, 92–101. http://dx.doi.org/10.1176/
appi.ajp.162.1.92
Valstar, M. F., Gunes, H., & Pantic, M. (2007). How to
distinguish posed from spontaneous smiles using
geometric features. In Proceedings of the Ninth
International Conference on Multimodal Interfaces (pp.
38–45). http://dx.doi.org/10.1145/1322192.
1322202
Valstar, M. F., Mehu, M., Jiang, B., Pantic, M., &
Scherer, K. (2012). Meta-analysis of the first facial
expression recognition challenge. IEEE Transactions
on Systems, Man, and Cybernetics, Part B:
Cybernetics, 42, 966–979. http://dx.doi.org/10.1109/
TSMCB.2012.2200675
van Boxtel, A. (2010). Facial EMG as a tool for inferring
affective states. In A. J. Spink, F. Grieco, O. E.
Krips, L. W. S. Loijens, L. P. J. J. Noldus, & P.
H. Zimmerman (Eds.), Proceedings of Measuring
Behavior (pp. 104–108). Wageningen, the
Netherlands: Noldus Information Technology.
Vanger, P., & Ellgring, H. (1989). Nonverbal cues during
the recognition of affect during imagery. Unpublished
manuscript, Max Planck Institute for Psychiatry,
Munich, Germany.
458
Wahlberg, K.-E., & Wynne, L. C. (2001). Possibilities
for prevention of schizophrenia: Suggestions from
research on genotype–environment interactions.
International Journal of Mental Health, 30, 91–103.
Watzlawick, P., Beavin, J. H., & Jackson, D. D. (1967).
The pragmatics of human communication. New York,
NY: Norton.
Watzlawick, P., Beavin, J. H., & Jackson, D. D. (1969).
Human communication: Forms, disturbances,
paradoxes. Oxford, England: Hans Huber.
Wearden, A. J., Tarrier, N., Barrowclough, C., Zastowny,
T. R., & Rahill, A. A. (2000). A review of expressed
emotion research in health care. Clinical Psychology
Review, 20, 633–666. http://dx.doi.org/10.1016/
S0272-7358(99)00008-2
Wexler, B. E., Warrenburg, S., Schwartz, G. E., & Janer,
L. D. (1992). EEG and EMG responses to emotion-
evoking stimuli processed without conscious
awareness. Neuropsychologia, 30, 1065–1079. http://
dx.doi.org/10.1016/0028-3932(92)90099-8
Wong, K. H., Law, H. H. M., & Tsang, P. W. M. (1989).
A system for recognizing human faces. In
Proceedings of the 1989 International Conference
on Acoustics, Speech, and Signal Processing (Vol.
3, pp. 1638–1642). http://dx.doi.org/10.1109/
ICASSP.1989.266760
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S.
(2009). A survey of affect recognition methods:
Audio, visual, and spontaneous expressions. IEEE
Transactions on Pattern Analysis and Machine
Intelligence, 31, 39–58. http://dx.doi.org/10.1109/
TPAMI.2008.52
Zoccolotti, P., Caltagirone, C., Benedetti, N., &
Gainotti, G. (1986). Pertubation des responses
vegetatives aux stimuli emotionnels au cours des
lesions hemispheriques unilaterales [Disturbance
of vegetative responses to emotional stimuli in the
hemispherical unilateral lesions]. Encephale, 12,
263–268.
 Chapter 18

Measuring the Voice

Andrew Rosenberg, Frank Enos, and Julia Hirschberg

How a speaker produces an utterance, in the
context of the lexical content of the utterance itself,
conveys important information not only about the
meaning of the utterance but about the speaker’s
mental, social, and physical states. Analyzing
phenomena (such as pitch, loudness, and speak-
ing rate) and voice quality (such as hoarseness,
harshness, or creakiness) provides useful objective
cues to speakers’ intentions, emotions, mental and
physical health, and social relationship with their
listeners.
From the point of view of language meaning,
variation in vocal production can signal differences
in syntactic attachment, disambiguate scope of
modifiers and negation, information status (whether
an item is old or new information, focused or not),
the type of speech act the utterance represents
(a backchannel or an affirmation), and the structure
of the discourse. From the point of view of research
in speaker state, variation in the voice can provide
valuable indicators of medical conditions (such as
depression or schizophrenia) and speaker emotions,
intentions, and attitudes (such as anger, deception,
or trust). The psychological aspects of variations
in voice measures are described in Chapter 11 of
this handbook. In this chapter, we describe the
quantifiable features commonly used in analyzing
these vocal contributions, together with common
approaches to extracting them. We examine features
at the lower, acoustic level—including fundamental
frequency (f0), intensity, duration, rhythm, and
speaking rate—as well as voice quality measures;
we also examine features at a higher, prosodic
level—including pitch accent, phrasing, and overall
intonational contour.
When discussing the higher prosodic level, we
also describe the relationship between some shal-
low lexical features and acoustic realizations. We
describe not only the features themselves and
the different individual states and behaviors they
have been shown to correlate with but we also
describe different techniques that have been used
to measure them in prior literature. Finally, we pro-
vide pointers to tools that can be used by others to
extract and analyze these features.
LOW-LEVEL ACOUSTIC FEATURES
A low-level, acoustic-prosodic feature is defined as
one that is objectively quantifiable and automati-
cally and unambiguously extractable using standard
acoustic analysis software tools. These include, inter
alia, the following:
■■ Pitch (fundamental frequency): maximum,
mean, minimum, and standard deviation;
■■ Intensity (energy): maximum, mean, minimum,
and standard deviation;
■■ Duration; and
■■ Voice quality: jitter, shimmer, and harmonics-
to-noise ratio.

Parts of this chapter are from Automatic Detection and Classification of Prosodic Events, by A. Rosenberg, 2009, Columbia University, New York, NY.
Any views expressed here are the authors’ own, and any errors that may appear here are errors on the part of the authors; in no way do they represent
the views of, or errors on the part of, the D. E. Shaw Group.

http://dx.doi.org/10.1037/14669-018
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
459
Copyright © 2016 by the American Psychological Association. All rights reserved.
Rosenberg, Enos, and Hirschberg
These and other low-level, acoustic-prosodic
features derived from them have proven extremely
useful in analyzing all types of speaker state and
distinguishing one state from another, such as
distinguishing anger from happiness or fear, certainty
from uncertainty, and deceptive intention from non-
deceptive. Commonly examined features include
duration (how long is a word or u ­ tterance?), paus-
ing (how frequent and how long are silences and
where do they occur?), pitch (including pitch range,
minimum, maximum, and mean pitch), and energy
features (also examined in terms of minimum, maxi-
mum, and mean). Voice quality features—such as
shimmer, jitter, and harmonics-to-noise ratio—are
used to measure breathiness, harshness, and vocal
fry, which can be indicators of age and of emotion.
All these are usually examined using a large variety
of time scales, ranging from a few milliseconds to an
entire utterance. Features are typically normalized in a
variety of ways, exploiting long-term, speaker-specific
methods and using local (segmental) context, so that
information from within and across speakers can be
compared. Pitch and loudness can be estimated using
automatic pitch trackers, which track variation in the
fundamental frequency (f0) and energy of speech.
Durational features can be obtained from forced
(i.e., automatic) alignment of hand-segmented data
or from the output of automatic speech recognition
(ASR). Tools to extract all of the features described in
this chapter are discussed at the end of the chapter.
Pitch and Energy Features
Pitch features are computed from the voiced regions
of each phrase, utterance, or longer segment under
analysis and can be used in various forms, includ-
ing raw f0, median-filtered f0, or f0 that has been
­stylized by fitting linear splines (described later in
this section). Second-order features can be computed
from this pitch information, including maximum
pitch; mean pitch; minimum pitch; standard devia-
tion; range of pitch; number of frames that are rising,
falling, doubled, halved, or voiced; the length of the
first/last slope; the number of changes from fall to
rise; and the value of the first, last, and average slope.
Features can be used in raw and normalized forms,
where normalization can be done in a variety of ways:
These include dividing by the mean for each f0 value,
460
subtracting the mean from that value, a cumulative
distribution function value for the feature, and sub-
tracting the mean and dividing by the standard devia-
tion (z-score normalization). Features measuring
pitch halving errors of the pitch tracker can also be
useful. There is some evidence that this error occurs
in the presence of vocal fry or diplophonia (where two
pitches are produced by the speaker at the same time;
Johnson, 2003), which may occur as a consequence
of “forced” or overly energetic speech production.
These have been found to correlate positively with
deception (Enos, 2009) and have been useful in clas-
sifying emotions such as sadness (Liscombe, 2007;
Liscombe, Venditti, & Hirschberg, 2003). Local pro-
sodic features can also be extracted automatically,
such as slope of pitch of last syllable of an utterance
and duration of first syllable of an utterance. Such
local measures can be useful in studying alignment
in turn-taking behavior, for example, as in Benus
(2009) and Levitan, Gravano, and Hirschberg (2011).
Basic energy features can be extracted either
from raw energy values in the segment or only from
voiced regions. These values can also be used in raw,
median-filtered, or stylized form, similarly to pitch
features. Second-order features such as maximum,
minimum, mean, slope, and standard deviation
can also be examined. More sophisticated energy
features may be extracted by examining the energy
components of various frequency sub-bands. Infor-
mation in these sub-bands is also useful in predict-
ing higher level prosodic information, such as pitch
accents, or intonational prominences.
Pitch features are only defined during voiced
regions of speech, where the vibration of vocal folds
results in periodic acoustics. However, energy fea-
tures are defined everywhere in speech, during both
voiced and unvoiced speech sounds. The rationale
for only extracting energy features from voiced
regions is that there is something fundamentally
distinct about the acoustics of speech during voic-
ing; thus, the energy features extracted from voiced
speech are more robust and informative. Energy
extracted from unvoiced regions can be highly
­variable; in Standard American English (SAE), for
example, stops such as “k” and “t” are often char-
acterized by short bursts of energy; fricatives such
as “sh” and “ch” can have very high energy in word

initial and medial contexts but can have a long low-
energy decay at the end of a word.
Pitch and energy reset and stylization.  Pitch
and energy reset features can also be calculated to
identify differences in acoustic phenomena. For
example, calculating the difference of mean, maxi-
mum, minimum, z score of maximum, and standard
deviation of pitch and intensity contours across
words or larger phrases can provide information
on the dynamics of phenomena such as emotion
or for higher level prosodic phenomena such as
phrasing prediction. At the word level, differences
can be c­ alculated at varying length regions, ranging
from 20 to 200 ms preceding or following a word.
As pitch and intensity contours are typically con-
structed with a 10-ms frame rate, these correspond
to a mean of 2, 5, 10, and 20 data points. These
windows can of course be varied independently
for regions preceding and following the segment
of interest. These features are also calculated over
delta (change in) pitch and intensity contours, that
is, the change in these features over time, to capture
dynamic properties of the speech. For example,
identification of emotions such as anger can be
improved by examining the previous context of
vocal features such as pitch, duration, and loudness.
As previously mentioned, pitch and energy
changes can also be calculated by fitting line
­segments to the pitch and energy contour using
linear regression. Linear splines provide a linear
“stylization” of the pitch contour, leading to a rep-
resentation of pitch as a series of variable length-
ened linear segments. Some have argued (Hirst &
Espesser, 1993) that this sort of stylization retains the
salient elements of prosodic variation while ignoring
the “micro”-prosodic influences of particular speech
sounds (e.g., long “e” sounds in SAE are produced at
higher pitch than short “u” or “uh” sounds due to the
stress on articulators required for producing these
segments). This approach is based on the hypothesis
that pitch and energy gradually decrease over the
course of a phrase or an utterance. Although this
assumption is oversimplified, a linear fit can capture
an approximate trend over the course of a longer
stretch of speech. From the information we can
derive about these linear segments, it is possible to
Measuring the Voice
construct features that can be used to identify longer
term phenomena and changes in broader segments
over time. The window over which to generate a
regression fit line can be defined in a variety of ways,
including presence of silence. One can then fit linear
segments to pitch and energy contours between silent
regions, for example, to obtain information about the
general shape of pitch and energy contours for the
region. As pitch and energy contours are sensitive to
changes due to accenting and segmental effects, one
should perform the linear regression over subsets
of the pitch and energy points, constructing regres-
sion lines to (a) the maxima, (b) the minima, (c) the
mean values within each word, in addition to using
(d) every point. The intuition here is that one or
more of these aggregations may be more reliable than
others in reflecting the declension of pitch or energy
over the course of a phrase. Using coefficients (slope
and intercept) and error of these best fit lines, one
can construct features to represent useful information
such as pitch and energy reset. These, for example,
can be calculated from the following:
■■ Slope and error: Include the slope and root-
mean-square error (RMSE) of the previous and
following best fit lines in the feature vector.
■■ Regression reset: Evaluate two regression lines
at the word boundary. Regression reset is the
difference in the linear regression estimates of
the preceding and following best fit lines at the
word boundary. This is designed to calculate the
degree of intonational “reset” that occurs at the
start of a new phrase.
Shape modeling of pitch and energy contours.  A
number of approaches have been proposed to model
the shape of acoustic contours in speech. These are
usually applied to pitch information, but shape mod-
eling of energy contours has also been investigated.
Taylor (1998) proposed the tilt model to com-
pactly capture the shape of the pitch (f0) contour.
This model is based on identifying the location and
value of peak values of f0 within a window. Based
on a hypothesis that interesting pitch behavior is
characterized by a “peak” or rise–fall, tilt attempts
to parameterize a peak by where its peak is in time
and how high it reaches relative to previous and fol-
lowing pitch valleys. Four values are determined,
461
Rosenberg, Enos, and Hirschberg
identifying how much the peak rises from the start
of the event, A[rise], and how much it falls to the
end of the event, A[fall], and identifying the duration
from the start of the event to the peak, D[rise], and
the duration from the peak to the end of the event,
D[fall]. The tilt parameterization is decomposed into
two elements, tilt[amplitude] and tilt[duration].
Tilt[amplitude] is calculated as A[rise] − A[fall]/
A[rise] + A[fall]; tilt[duration] is calculated as
D[rise] − D[fall]/D[rise] + D[fall]. These two are
highly correlated, and the combined tilt parameter-
ization is calculated as the average of tilt[amplitude]
and tilt[duration]. Note that either the rise or the fall
can have zero duration and amplitude, allowing tilt
to describe strictly rising or falling pitch. We discuss
the application of the tilt model in later sections of
this chapter. A simpler approach to this is to examine
the minima and maxima locations directly. Although
this is somewhat less comprehensive than the tilt
parameterization, these simple features can capture
useful information about the voice.
The Tonal Center of Gravity (TCoG) has
emerged as a compelling way to represent the per-
ceptual distinction between early and late peak-
ing pitch contours (Barnes, Veilleux, Brugos, &
­Shattuck-Hufnagel, 2010). This feature has been
used to distinguish acoustic excursions that may
have a similar peak value and position as rapidly
rising, or slowly rising. It is calculated as sum(t ×
x)/sum(x). TCoG represents the point in time at
which there is an equal amount of acoustic “mass”
on either side. Higher TCoG values indicate more
rapid rise characterized by a convex curve approach-
ing the peak, whereas lower TCoG values indicate a
concave approach to the peak.
Mishra, Kumar, Sridhar, and Conkie (2012)
described a measure called the area under the f0
curve. This feature captures the f0 peak and the
increased duration that is often associated with
prominent words. It is the discrete integral of the
smoothed f0 and duration within the interval of a
word. This feature is a compact way to incorporate
both duration and pitch. The intuition is that words
with higher pitch are more salient, as are words
with longer durations. By summing the “total” pitch
within a word, longer words with lower pitch can be
recognized as being similarly perceptually salient as
462
shorter words with high pitch. Rosenberg (2012)
applied this calculation to the intensity curve, find-
ing it to be a highly reliable predictor of prominence
and phrasing, and more reliable than any pitch
feature or duration alone.
Isotonic regression provides a mechanism to
describe a contour as rising, falling, a peak, or a
valley (Mishra, 2008). This approach fits a mono-
tonically increasing or decreasing line to a contour.
The error (RMSE) provides a measure of how good a
fit a rising or falling function has to a given contour.
A simple modification allows the use of this metric
for the identification of peaks and valleys. At each
candidate point t[i] in the contour, the remaining
contour t[i . . . end] is reflected across the current
point. The point t[i] where the isotonic regression
RMSE is minimized represents the best peak fit, or
the best valley fit. An attractive quality of this
algorithm is that it is robust to minor perturbations
and spurts that are higher or lower than the true
peak of a contour. Based on the inverse RMSE of
the isotonic regression fit, a distribution of the four
scores can be calculated. This can serve as a “likeli-
hood” of a contour being one of each of the shapes:
rising, falling, peak, or valley.
Duration and Rhythm
Durational features can be extracted over differ-
ent regions. Maximum and average phone duration
in the segment can be used either as raw values,
normalized using speaker specific durations, or
normalized using durations computed from an
entire corpus. Speaking rate is typically measured in
syllables per second. Where manual orthographic
transcription is not available, ASR output can serve
as a reasonable proxy. It is also possible to obtain
fairly reliable syllable nucleus and boundary detec-
tion using algorithms based on Mermelstein (1975)
and Pfau and Ruske (1998). Segmental duration
at different granularities can help to measure, for
example, the presence of any lengthening effect at
phrase boundaries (cf. Wightman, Ostendorf, &
Shattuck-Hufnagel, 1992). To do this, one can cal-
culate the length of each word-final vowel, syllable,
and syllable rhyme (the vowel and any following
consonants). Because it is possible that prebound-
ary lengthening influences phone durations earlier

than the final phone, one can also investigate the
mean vowel length and mean syllable length within
each word. Phones have different inherent dura-
tions. To account for these inherent differences, one
should normalize vowel durations based on vowel
identity (e.g., whether it is a long or short “a,” “e,”
and so on, in English). In addition to this phone-
identity normalization, one should also normalize
phone durations by speaker. This normalization has
been used to detect preboundary lengthening. Under
this normalization, the mean and standard deviation
of phone durations are calculated separately for each
speaker. A number of normalization approaches
have been explored to account for this, including
z-score normalization of the log of the duration as
well as half-normal, z-score normalization.
To capture rhythm and timing information,
percentages of vocalic intervals (%V), the standard
deviation of vocalic intervals (ΔV), and the standard
deviation of intervocalic intervals (ΔC) are typi-
cally calculated. These measures have been shown
to capture the complexity of the syllabic structure
of a language/dialect in addition to the existence of
vowel reduction. Languages/dialects that have a high
variability of consonantal intervals are likely to have
more clusters of consonants, which lead to more
complex syllables. The complexity of syllabic struc-
ture of a language/dialect and the existence of vowel
reduction in a language/dialect are good correlates
with the rhythmic structure of the language/dialect.
It is possible to identify vocalic intervals using an
automatic phone recognizer. A sequence of consecu-
tive vowels is considered as a single vocalic interval.
Similarly, a sequence of consecutive consonants is
considered as one intervocalic interval.
Speaker Normalization
It can be challenging to determine the most appro-
priate way to normalize for speaker or segmental
differences. Two relatively simple approaches are
range normalization and z-score normalization
(mentioned earlier). Range normalization scales
data points (e.g., f0 or intensity values) such that the
maximum value is 1, and the minimum is 0. To range
normalize a point x, one uses x′ = (x− minimum)/
(maximum − minimum). A requirement of z-score
normalization is that the mean and standard
Measuring the Voice
deviation of the data need to be calculated.
Normalization is then calculated as x′ = (x − mean)/
standard deviation. Both approaches have limita-
tions. Range normalization is very sensitive to the
presence of outliers. A few points with exceed-
ingly high or low values can dramatically impact
the normalization. In pitch extraction, halving and
doubling errors are relatively common and can cre-
ate problems when applying range normalization.
On the other hand, z-score normalization implicitly
assumes that the data that is being normalized fol-
lows a normal or Gaussian distribution. The cal-
culation of mean and standard deviation are based
on maximum likelihood estimates of a univariate
Gaussian distribution. This assumption can lead to
bias in z-score normalization if the underlying data
does not follow a Gaussian distribution. For this
reason, it is more reliable to use z-score normaliza-
tion on logf0 rather than f0, and log energy rather
than raw energy values.
Pauses
Features such as filled and unfilled pauses and
other disfluencies can also be valuable indicators of
speaker state; however, filled pauses are not always
easy to extract automatically despite considerable
research on this topic. When comparing different
conditions (such as emotions, deceptive behavior,
and other speaker or interpersonal states), the ratio
of the number of filled pauses in Condition 1 to the
number of filled pauses in Condition 2 may be help-
ful in distinguishing the two, as well as the ratio
of segments containing filled or unfilled pauses to
total speaker segments. The total duration of filled
or unfilled pauses in a segment and the maximum
duration of a silent pause for the segment can also
be useful indicators as can features that capture
the length of pauses immediately preceding and
following a given segment. Laughter can be com-
puted automatically with some success and used
in similar fashion (Salamin, Polychroniou, & Vin-
ciarelli, 2013). Audible breaths may also be useful,
although their automatic extraction is also an ongo-
ing research problem. Features in dialogue, such as
response latency and the amount of time between
speaker turns, may also be useful indicators of
speaker uncertainty or deception and are simple to
463
Rosenberg, Enos, and Hirschberg
calculate based on the starting and ending times of
speaker turns.
Regions of Analysis
All acoustic-prosodic features may be calculated
over a variety of segments. The most natural choice
is to identify a region of analysis based on the
domain of the analysis itself. For example, if you
want to know whether an utterance is a question,
look at the utterance itself. If you want to know
whether a person is deceptive in response to a
­question, analyze only the material in the response.
However, there is often considerable benefit in com-
paring the current segment to previous context, as
in anger detection, when increase in f0 maximum,
intensity, and duration over time is a useful signal
(Liscombe et al., 2003).
HIGHER LEVEL PROSODIC FEATURES
Higher level prosodic variations are those typically
associated with variation in the symbolic description
of the overall intonational contour. These variations
typically include the location of pitch prominences
and phrase boundaries as well as in the overall shape
of the pitch contour they comprise. These variations
can provide important communicative information
to utterances in many languages (for excellent over-
views, cf. Bolinger, 1985, 1989; Ladd, 1980, 1996).
In most languages that have been studied, such
variation is an important indication of speaker state
as well as utterance meaning. Emotional states, such
as anger, frustration, and happiness, are displayed
in a speaker’s speech through prosodic variation.
Prosody also carries indicators of speaker states of
incredulity, uncertainty (Hirschberg & Ward, 1992;
Liscombe, Hirschberg, & Venditti, 2005; Ward &
Hirschberg, 1985), sarcasm (Rakov & Rosenberg,
2013; Tepperman, Traum, & Narayanan, 2006),
as well as deception (Enos, 2009; Hirschberg et al.,
2005). Prosody plays a critical role in the participa-
tion in and interpretation of dialog. In addition to
directly impacting the meaning of some utterances,
such as “okay” (Gravano, Benus, Hirschberg,
German, & Ward, 2008), prosody is used to struc-
ture spoken material (Grosz & Sidner, 1986; Naka-
jima & Allen, 1993), affecting the intended context
464
of interpretation. Analysis of higher level prosody
has also been shown to be useful in improving
performance in many spoken language processing
tasks, such as ASR (Zhang, Hasegawa-Johnson, &
Levinson, 2004), speech synthesis (Demenko,
Grocholewski, Wagner, & Szymanski, 2006),
speaker identification (Weber, Manganaro,
Peskin, & Shriberg, 2002), language (Tong, Ma,
Zhu, Li, & Chang, 2006) and dialect identification
(Rouas, 2007), story/topic segmentation (Rosenberg,
Sharifi, & Hirschberg, 2007; Stolcke, Shriberg,
Hakkani-Tur, & Tur, 1999), sentence segmenta-
tion (Shriberg, Stolcke, Hakkani-Tur, & Tur, 2000),
discourse segmentation (Hirschberg & Nakatani,
1996; Nakatani, Hirschberg, & Grosz, 1995), extrac-
tive speech summarization (Maskey, Rosenberg, &
Hirschberg, 2008), punctuation insertion
(Christensen, Gotoh, & Renals, 2001), and speech
act classification (Ananthakrishnan, Ghosh, &
Narayanan, 2008; Shriberg et al., 1998).
Conventions for Describing Prosody
Many conventions have been developed for describ-
ing prosodic variation so that this variation can be
described and compared among different research-
ers. These conventions are also employed to
annotate corpora for statistical analysis, automatic
classification experiments, and speech technologies
such as text-to-speech (TTS) synthesis. Manually
annotated data can be used to test hypotheses about
human communication and its underlying cognitive
processes. It also enables the training of statisti-
cal models and supervised classifiers to enable the
automatic annotation of prosody, which, in turn,
allows prosodic information to be used in computa-
tional tasks.
Prosodic labeling schemas vary in terms of the
decisions they make about what prosodic features
to annotate and which labels to employ. Prosodic
annotation standards can be categorized in a num-
ber of dimensions. Annotations can be categorical or
continuous in nature.
Categorical annotations describe prosodic events
as elements of a prosodic “vocabulary” or fixed
inventory of prosodic phenomena, whereas con-
tinuous annotations describe prosody by “degree”
rather than “type.” For example, continuous

standards define a degree of prominence or the
height of an f0 peak, rather than a categorical label
for the type of prominence carried by a spoken
word or syllable. Examples of categorical annota-
tion standards include the British (e.g., Halliday,
1967; O’Connor & Arnold, 1973; Palmer, 1922)
and the American (e.g., Beckman & Hirschberg,
1994; Hirschberg, 2002b) schools for English. The
British school defines prosody as pitch movements
(e.g., rises and falls), whereas the American school
uses pitch targets or tones (e.g., high vs. low tones).
Annotation standards are also differentiated by the
amount of community resources available to facili-
tate instruction, train new annotators, and provide
transparency to the community. These resources
come in the form of annotation manuals, tutorials,
and available data.
The Tones and Break Indices conventions and other
categorical approaches.  For SAE, the Tones and
Break Indices (ToBI; Pitrelli, Beckman, & Hirschberg,
Figure 18.1.  A declarative contour.
A fuller description of the ToBI systems may be found in the ToBI conventions documentation (Beckman & Hirschberg, 1994) and the training
1
materials (Beckman & Elam, 1997) available at http://www.cs.columbia.edu/∼agus/tobi/tobi_convent.pdf. Newer materials are available from
http://www.ling.ohio-state.edu/research/phonetics/E_ToBI.
Measuring the Voice
1994; Silverman et al., 1992) is a widely used pros-
ody annotation standard that is currently the most
popular exemplar of the American school.1 This
convention includes a set of labeling tiers: an ortho-
graphic tier, which contains the speech transcription
and is time-aligned with the speech wave form; a
break index tier, in which five degrees of juncture
between words can be identified; a tone tier, in which
pitch accents, phrase accents, and boundary tones
(largely following Pierrehumbert, 1980) are labeled;
and a miscellaneous tier, in which phenomena such
as disfluencies can be identified. All of these tiers are
also aligned with the speech file and a representa-
tion of the f0 contour. Differences among ToBI break
indices denote differences in word or phrase bound-
aries, with higher numbers marking greater degree
of juncture. An SAE standard declarative contour,
which typically falls, is modeled with one of more
simple high pitch accents, a low phrase accent, and a
low boundary tone (see Figure 18.1); a yes–no ques-
tion contour is modeled by sequences of low pitch
465
Rosenberg, Enos, and Hirschberg
accents and a high phrase accent and high bound-
ary tone (see Figure 18.2). Schematic versions of
other SAE contours—combinations of pitch accents,
phrase accents, and boundaries tones—are shown in
Appendix 18.1.
In SAE ToBI, words may be prominent (accented)
or not (deaccented) and, if accented, may bear one
of five different accents, including the two simple
accents (H* and L*) and three complex accents
(L+H*, L*+H, and H+!H*), where the “*”
­indicates alignment of the accent with the primary
stress of the word, and “H” and “L” indicate high
and low tones, respectively. HiF0 is annotated on
the highest accent peak in a minor (intermediate)
phrase and serves the function of indicating varia-
tion in pitch range from phrase to phrase.
Hirschberg and Pierrehumbert (1986) pro-
posed a set of possible meanings for different pitch
accent types in SAE that have formed the basis for
a number of perception experiments (see also Pier-
rehumbert & Hirschberg, 1990). In their proposal,
H* accents, which are typically found in standard
declarative utterances, convey that the accented
Figure 18.2.  A yes–no question contour.
466
item should be treated as new information in the
discourse—as part of what is being asserted by the
speaker. L* accents, on the other hand, convey that
an accented item should be treated as salient in the
discourse but not part of what is being asserted.
These are typically seen in prominent items in
yes–no question contours and are often employed
to make prepositions or adverbs or discourse mark-
ers prominent in utterance-initial position. L+H*
accents can be used to produce a sense of contrast,
often termed contrastive stress. In this complex
accent, the H tone is aligned with the stressed syl-
lable of the prominent word, and the f0 rise from the
initial L tone is rapid. In the L*+H accent, the L
tone is associated with the word’s stressed syllable,
and the rise to the H is gentler, resulting in more of
a “scooped” accent. This difference in alignment is
critical to interpretation, conveying an impression
either of uncertainty or of incredulity depending on
other prosodic and contextual features. H+!H*
accents are realized as a fall onto the stressed
­syllable and have been proposed by Hirschberg and
Pierrehumbert as conveying some sense that speaker

and hearer are already familiar with the mentioned
item, as in a “reminding” context. There is a ToBI
manual, tutorial with audio examples, and several
available labeled corpora (Heeman, 1999; Held-
ner & Megyesi, 2003).
Although ToBI was originally developed to
describe the prosody of American English, the stan-
dard has since been extended and adapted for many
other languages. There are available data and training
manuals for ToDI (Gussenhoven, 2005), a system
somewhat similar to the ToBI framework developed
for Dutch. Annotation guidelines are available for
ToBI variants in Portuguese, Mandarin Chinese
(C-ToBI), Korean (K-ToBI), Japanese (J-ToBI),
German (G-ToBI and variants), Greek (GRToBI),
Italian (several dialectal variants), and ToBI for Stan-
dard Serbo-Croatian (SC-ToBI). A description of these
variants has been compiled in Jun’s (2005) work. The
Intonational Variation in English (IViE) system for
British English is modeled on ToBI but differs with
regard to its inventory of tone targets (expanded to
cover more varieties) and two additional labeling tiers
(rhythm tier and pitch movement tier), and it has
no Break Index tier (Grabe, Nolan, & Farrar, 1998).
There is a tutorial for IViE annotation available online.
Bruce’s model of Swedish prosody (Bruce, Granstrom,
Gustafson, House, & Touati, 1994) includes an inven-
tory of accent types and boundary-related categories
using a high/low inventory similar to ToBI and ToDI.
Koehler’s (1991) Kiel intonation model annotation
scheme, developed for German, describes intona-
tion in terms of peaks and valleys rather than tones
or movements, and it distinguishes a variety of pitch
prominence categories. There is as yet no publicly
distributed annotation manual or annotated data
available for the Kiel intonation model. The only
prepared tutorials that we have located to date are
for the ToBI, ToDI, and IViE annotation standards.
We have identified available prosodically labeled data
in American English, European Portuguese, Italian,
French, German, Mandarin Chinese, and Australian.
Conventions developed by the Institute for
Perception Research are a frequently cited pro-
sodic annotation standard, developed for Dutch.
It describes prosody as an inventory of pitch move-
ments assigned to syllables and classified as rises or
falls by the range of movement and whether they are
Measuring the Voice
accent-lending or nonaccent-lending. The use
of this standard is supported by work by ’t Hart,
Collier, and Cohen (1990), but is limited by the lack
of a tutorial and publicly available annotated data.
Finally, Dilley and Brown (2005) have developed
Rhythm and Pitch (RaP), a labeling system for anno-
tating the prosody of American English that empha-
sizes rhythm as well as intonation, a topic that is of
growing interest among speech researchers.
Continuous descriptions of intonation.  In con-
trast to these categorical descriptions of intonation
contours, continuous descriptions are typically
aimed at describing the continuous f0 track of
an utterance. These standards have been devel-
oped to facilitate the assignment of f0 targets for
speech synthesis. The Fujisaki superpositional
model (Fujisaki & Hirose, 1982) is based on
acoustico-physiological observation and has been
used primarily for Japanese synthesis. It describes
intonation as a superposition of accent contours
on a phrasal contour via phrase and accent com-
mands that control f0; parameterizations of phrase
and accent signals are continuous rather than cat-
egorical. Möbius (1993) extended this model for
English and German, but it has rarely been used
for annotation. Hirst’s International Transcription
System for Intonation (Hirst & Di Cristo, 1998)
describes intonation in terms of sequences of labels
representing target points for f0, where target points
are defined based on a speaker’s current and previ-
ous pitch range. Typically, these targets employ
specific speaker-dependent f0 values based on the
speaker’s “normal” pitch range. The tilt model
(Taylor, 1998) describes the pitch contour of spe-
cific intonational events—accented and phrase-
ending syllables—using a compact parameterization
describing the slope, height, and peak position of the
rise and fall of the pitch contour within a syllable.
Whatever system is adopted, reliable hand-
annotation of prosody requires an investment of
time to train annotators in the standard. Prosody
has also been studied using untrained annota-
tors. These studies forgo specific annotation
standards, focusing instead on investigations of
particular prosodic phenomena. Mahrt et al. (2011)
used untrained students to annotate prosodic
467
Rosenberg, Enos, and Hirschberg
prominence. In this study, a group of students was
simultaneously presented with an utterance, and
asked to mark the prominent words. This is part of
a broader effort to annotate prosody using untrained
annotators that is being conducted at the Univer-
sity of Illinois’s Linguistic Laboratory for Speech
Prosody under the principal investigator Jennifer
Cole. At the Educational Testing Service, Evanini
and Zechner (2011) investigated the use of crowd-
sourcing to generate prosodic annotations, finding
agreement that approaches expert levels for prosodic
prominence and major phrase boundaries. Buhmann
et al. (2002) ran a pilot study to determine the costs
of using naive rather than expert labelers to generate
prosodic annotations of prominence and prosodic
breaks. They found that reliable annotation was
feasible provided that labelers were provided with
strict written guidelines, supervision, and feedback.
There are also corpora that annotate only a subset of
prosodic phenomena. The C-PROM corpus (Avanzi,
Simon, Goldman, & Auchlin, 2010) of French
speech in a variety of speaking styles is annotated
for two levels of prosodic prominence using trained
labelers. The Linguistic Data Consortium distrib-
utes a subset of the Switchboard corpus (K. Chen,
Hasegawa-Johnson, & Cohen, 2004) that has been
annotated for prosodic prominence and break
indices.
The reader should note that the different ways
to describe “higher level” variation in the speech
described earlier correspond to numerous different
aspects of utterance meaning as well as differences
in speakers’ mental and physical states. The next
section demonstrates how these variations can be
analyzed to identify important information about
the speaker’s meaning and state. We organize the
findings around the different components of higher
level variation, their realization in terms of the low-
level acoustic features discussed earlier, some of the
information they can convey, and some ways that
they can be identified automatically.
Pitch Accent
Accenting, or intonational prominence, has the
effect of drawing a listener’s attention to a particular
section of a spoken utterance. Consider the follow-
ing pair of utterances, where the most prominent
468
item in the utterance is indicated by all capital
letters:
A. John only introduced MARY to Sue.
B. John only introduced Mary to SUE.
In A, John has only introduced one person to
Sue, and that person is Mary. However, in B, John
has introduced Mary to only one person, and that
person is Sue. Identifying intonation prominence, or
pitch accent, can be important for studies of cogni-
tion and social interaction as well as in the develop-
ment of speech processing applications.
Accenting is an acoustic highlighting of a word
in contrast with other words in an utterance. This
effect may be produced by increasing the intensity
with which a word is produced, by employing a
markedly higher or lower pitch, or by increasing
the duration of the word versus its deaccented
(unaccented, nonprominent) version (Bolinger,
1985; Ladd, 1980).
Pitch excursions, in particular, have been
largely taken to be the most salient cue to pitch
accent. Clark and Yallop (1990) described pitch
as “the most salient determinant of prominence”
(p. 280). Recent studies have examined the cor-
relation between speech energy and pitch accent.
Silipo and Greenberg (2000) found that in spon-
taneous speech, duration and energy are the most
important acoustic parameters underlying accent,
with pitch playing only a minor role. Kochanski,
Grabe, Coleman, and Rosner (2005) expanded on
this result with a lengthy analysis-by-classification
of British and Irish English, finding f0 to be a weak
predictor of prominence, with loudness and dura-
tion being much more discriminative of prominent
and nonprominent syllables. Spectral tilt is a less
studied acoustic quality. In perception and produc-
tion studies (Sluijter & van Heuven, 1996; Sluijter,
van Heuven, & Pacilly, 1997), Dutch researchers
showed that accent in Dutch strongly correlates
with the energy within a frequency sub-band greater
than 500 Hz. This observation concerning spectral
emphasis led to a number of other studies examining
relationship between spectral balance or spectral tilt
and pitch accent (Campbell, 1995; Fant, Krucken-
berg, & Liljencrants, 2000; Heldner, 2001; Heldner,
Strangert, & Deschamps, 1999), all finding similar

strong correlations. These terms, though they may
be calculated in different ways, all extract energy
information from a particular frequency sub-band
and compare this energy to the total signal energy.
Note that human agreement on the pitch accent
detection task falls somewhere between 81% and
91% word accuracy (Pitrelli et al., 1994; Syrdal &
McGory, 2000), depending on the experience of
the labelers and the genre of the material. Also, a
majority class baseline on the Boston University
Radio News Corpus (BURNC; Ostendorf, Price, &
Shattuck-Hufnagel, 1995) yields 54.7% pitch accent
detection accuracy, by hypothesizing that all words
are accented.
Accenting may signal many things. In general, con-
tent words (e.g., nouns, verbs, and modifiers) are more
likely to be accented than function words (e.g., prepo-
sitions, articles), although not all content words are
accented, and not all functions words are deaccented.
Words that have been uttered in an earlier phase of a
conversation (given information) are usually less likely
to be accented when produced again, whereas items
introducing new information are more likely to be
intonational prominent, or accented—however, this
is not always the case (Brown, 1983; Dahan, Tanen-
haus, & Chambers, 2002; Gravano & Hirschberg,
2006; Grice & Savino, 1997; Terken & Hirschberg,
1994). Items may also be accented to convey focus or
contrast (Bolinger, 1961; Gundel, 1999) and the topic
(Hedberg, 2003) of an utterance.
Pitch accent types.  In many languages, words that
are intonationally prominent can be made so in dif-
ferent ways to convey different meanings. In SAE, for
example, the most common realization of intonational
prominence is produced with a simple increase in
pitch up to a point high in the speaker’s pitch range.
The use of different pitch accent types can be
used to convey a broad range of information, but
two common usages (that are relatively easy to
describe in text) are to indicate contrast and incre-
dulity. Consider the following dialog:
A. John makes paper airplanes. (Accen-
tuation to indicate topic and focus.)
B. John makes vapor airplanes?
(Accentuation to indicate narrow
focus of the question.)
Measuring the Voice
A. J ohn makes paper airplanes.
(Accentuation to indicate contrast
with the misrecognized word.)
B. John makes paper airplanes?
(Accentuation to indicate incredu-
lity at John’s immaturity.)
Although each of the bolded words are made
acoustically prominent, the realization of this
prominence—the pitch accent type—is used to
convey different meanings.
The ToBI standard describes this as an H*
accent. Simple low (L*) accents are more often used
to accent concepts that are already in the discourse
space, and they are typically used in yes–no question
contours. The ToBI prosodic annotation framework
describes intonation as a series of high (H) and low
(L) tones which are associated with prosodic events
(Pitrelli et al., 1994; Silverman et al., 1992). The
annotation standard defines five pitch accent types.
These include the basic H* and L* accents as well
as three complex tones, L+H*, L*+H, and H+!H*.
L+H* accents are characterized by a low tone
followed quickly by a high tone, whereas L*+H
are roughly defined as a scooped accent that begins
at an accented low tone, followed by an unaccented
high tone. In these two complex tones, L+H* and
L*+H, the starred tone is aligned with the lexically
stressed syllable of the word. The fifth accent type,
H+!H*, is characterized by a sharp pitch drop
from a previously unaccented high tone. In addition
to these, high tones are produced in a compressed
pitch range, that is, compressed from a previous
high tone. These downstepped high tones are indi-
cated as !H, and each accent produced with a high
tone, can be produced with a downstepped high.
The downstepped phenomenon is also called
catathesis. This leads to the additional accent types
!H*, L+!H*, and L*+!H.
Although there is general agreement that accent
shape and alignment impacts the interpretation
of an utterance, such is not the case concerning
whether pitch accents indeed occur in discrete
­categories. Taylor (2000) laid out the argument
against the categorical hypothesis in the presenta-
tion of his tilt model, a continuous parameteriza-
tion of intonational events. Arguing against the
469
Rosenberg, Enos, and Hirschberg
segmental phonology view implicit in ToBI and
other intonational theories, he has drawn a parallel
with phone perception. Although there are indeed
minimal phonetic pairs such as /b/ and /p/ that
semantically distinguish bill from pill, the voicing of
the /b/ in bill can be incrementally reduced such that
the word is eventually perceived as pill. ­However,
at no point does the perception of the word evoke a
combined semantic representation that is partially
bill and partially pill. There is a perceptual bound-
ary prior to the interpretation of the phone or word.
Although Taylor then claimed that there is no
evidence of such a boundary between pitch accent
types, there is in fact evidence from a number of
empirical studies. Ladd and Morton (1997) found
evidence of abrupt, categorical, shifts in the percep-
tion of “normal” versus “emphatic” accent peaks,
although they found no evidence of categorical
­perception in rating how emphatic accents were.
The perception of degree of emphasis appeared
continuous, whereas the interpretation appeared
categorical. Pierrehumbert and Steele (1987) found
evidence that subjects also produce categorical
accent types. In this study, subjects heard L+H
accents where the alignment between f0 peak and
vowel onset was varied in small increments—that
is, from L+H* to L*+H—and were asked to imitate
the contour that they heard. Analysis of these pro-
ductions revealed that subjects in fact produced two
distinct categories of accent, either L+H* or L*+H:
They did not reproduce the continuum they heard.
This provides support for the claim that tonal align-
ment is a binary distinction in SAE intonation.
Automatic detection of pitch accent.  The task of
automatically detecting pitch accent has received
a considerable amount of research attention,
­supported by these and other studies of acoustic
correlates to accenting. A wide range of supervised
machine learning techniques have been applied
to this problem. F. R. Chen and Withgott (1992)
experimented with using a supervised hidden
Markov model (HMM) trained with smoothed pitch
and intensity features to detect emphasis. This work
used these emphasis hypotheses to improve summa-
rization of a spontaneous two-party discourse. The
results of the emphasis detection are presented as a
470
receiver operating characteristic curve, making
comparisons to other studies very difficult. That
said, this approach is the first of many techniques
to apply HMMs and other short-frame based models
to this task.
For example, Conkie, Riccardi, and Rose (1999)
used a HMM to detect pitch accent using speaker
normalized pitch and energy values extracted at
10-ms frames, with delta and delta deltas of these
values. This approach was evaluated on the speech
of a single professional speaker, reading three types
of text: prompts, newspaper text, and phonetically
balanced utterances. This acoustic HMM achieved
82.8% accuracy. A syntactic-prosodic model, associ-
ating part-of-speech (POS) sequences with prosodic
labels, correctly predicted 84.0% of the accents,
and in combination, the two performed at 88.3%
accuracy. This represents one of the better results
in detecting pitch accents using speaker dependent
acoustic modeling.
Sequential modeling of acoustic modeling
extracted at 10-ms frames has been incredibly suc-
cessful in ASR systems. It is, therefore, unsurprising
that this approach would be used in the analysis
of intonation. Sequential models take a number of
forms. A distributed Time Delay Recursive Neural
Network (TDRNN) was described by Ren, Kim,
Hasegawa-Johnson, and Cole (2004). The TDRNN
is a different style of sequential modeling from the
HMM, where the output of a neural network from
one time step is used to provide features in the input
layer of itself at a later time step. This distributed
TDRNN incorporated four distinct TDRNN models,
one for each feature type: pitch, intensity, duration,
and filtered energy coefficients. The filtered energy
coefficients were based on wavelet transformation
of 14 frequency-band filters of the energy in the
speech signal. A discrete cosine transform then con-
verted these to a discrete set of coefficients, which
the authors dubbed spectral balance based cepstral
coefficients (which together represent the short-
term power spectrum of the speech signal). A fifth
TDRNN model was used to combine the output of
the independent TDRNN models. This distributed
approach was able to detect pitch accents on syl-
lables using only acoustic information with 83.64%
accuracy, using three female speakers from the

BURNC as data. When this detection was performed
using a single TDRNN instead of four distributed
models with a combining model, the performance
was 2.43% worse.
Taking a similar approach, Ananthakrishnan and
Narayanan (2005) applied a coupled multistream
HMM (CHMM) to the task of automatic pitch
accent detection. Using this modeling technique,
the HMM used inputs from the pitch, energy, and
duration domains independently, training a coupled
model to combine the information. This work
presented its results in terms of both syllable- and
word-based accuracy. Evaluated on a single BURNC
speaker, and using only acoustic information, the
CHMM model performed with 72.03% word-based
accuracy (73.97% syllable-based). In combination
with a syntactic language model, its word-based
accuracy is improved to 79.5%.
C. Li, Liu, and Xia (2007) developed a technique
for pitch accent detection concurrent with phone
recognition for language learning. Whereas oth-
ers have applied the same techniques for ASR and
pitch accent detection, this work couples the two
tasks, performing simultaneous accent detection and
phone recognition. This technique used normal-
ized pitch values along with Mel-frequency cepstral
coefficients, typically used in ASR, to recognize
phones in accent-bearing and nonaccent bearing
forms. Using a standard HMM and evaluated on
seven BURNC speakers, this model was able to pre-
dict accents on syllables with 80.6% accuracy. For
the accent detection evaluation, phone errors were
ignored from the analysis. If an accent-bearing form
of a phone was correctly detected, the detection was
considered correct whether or not the phone iden-
tity was accurate.
Evaluating the differences in short-frame
HMM modeling and maximum entropy modeling,
Sridhar and colleagues (Sridhar, Bangalore, &
Narayanan, 2008; Sridhar, Narayanan, & Bangalore,
2006, 2007) applied both supervised detection
approaches using acoustic and syntactic features.
The acoustic features were f0 and root-mean-
squared (RMS) energy with delta and delta deltas.
An HMM was evaluated on four BURNC speakers
and correctly identified 70.58% of accents. The
corresponding maximum entropy model achieved
Measuring the Voice
80.09% word level accuracy. The researchers
explored the use of supertags (richer POS tags that
capture syntactic dependencies), modeling uni-,
bi-, and trigrams as well as punctuation-based
features. The inclusion of these syntactic features
dramatically improved the performance of both
the HMM and maximum entropy acoustic mod-
els to 85.13% and 84.53% accuracy, respectively.
Although these results are impressive, there is
one major caveat to the use of lexico-syntactic
features for prosodic detection and classification
on the BURNC material. This corpus is composed
of professional news speakers each reading the
same news stories. When performing a speaker-
independent evaluation, the training and testing
sets necessarily contain overlapping, if not iden-
tical, lexical information. Although accent and
phrasing decisions are not completely determined
by lexical information (Bolinger, 1972), they are
also not independent from it. Therefore, when
training lexico-syntactic models on BURNC data,
it is critical to guarantee that any story appearing
in the training data does not also appear in the
testing data, regardless of the speaker. Without
this guarantee, it is quite likely that the syntactic
results will be artificially inflated due to the con-
sistency between training and test material. The
experiments also described in Ananthakrishnan
and Narayanan (2006), Hirschberg (1993), and
Sridhar et al. (2007, 2008) may have suffered from
this issue. These articles do not directly address
this potential problem, making it difficult to assess
its impact.
Whereas several approaches have used short
frame (10 ms) acoustic features for accent detection,
others have extracted aggregated features of acoustic
information over syllables or words. These features
are then used to train supervised machine learning
classifiers to detect which syllables or words bear
accent.
One of the earliest efforts at automatic pitch
accent detection was performed by Wightman
and Ostendorf (1994). They used decision trees
to simultaneously predict accented syllables and
phrase boundaries. The decision tree outputs were
then used as input to an HMM to model the likeli-
hood of observing a given sequence of prosodic
471
Rosenberg, Enos, and Hirschberg
labels. This was evaluated on a single speaker from
the BURNC and demonstrated 81.51% pitch accent
detection accuracy. The decision tree model used
pitch features—aggregations within the syllable, and
differences from surrounding syllables—as well as
syllable structure—position within a word, lexical
stress—and the presence of a following pause or
breath. Ostendorf and Ross (1997) later proposed a
stochastic modeling framework to simultaneously
predict accent and boundary locations. The pro-
posed input to this structure was pitch, duration,
and energy features and segmental characteristics of
the syllable sequence. Optimizing this joint model-
ing technique on material from a single BURNC
speaker, Ostendorf and Ross were able to detect
pitch accents with 89% syllable-based accuracy.
In this modeling, syllables were classified as unac-
cented, high, low, or downstepped, whereas the 89%
detection accuracy was derived by a post hoc col-
lapsing of high, low, and downstepped classes. This
is an optimistic assessment of the automatic per-
formance of the approach, as it assumes knowledge
of the presence of intermediate phrase boundaries,
and the automatic detection of intermediate phrase
boundaries is a very difficult task.
Ensemble learning techniques were explored
by Sun (2002). Using boosting and bagging
with Classification and Regression Tree (CART)
models, high accuracy pitch accent detection
was achieved using acoustic and syntactic fea-
tures. This approach, like Ostendorf and Ross
(1997), classifies pitch accents as unaccented,
high, low, or downstepped. However, the pub-
lication includes confusion matrices from the
experiments, enabling the construction of binary
detection performance. Using acoustic features a
syllable level accuracy of 89.90 was obtained on
a single BURNC speaker (f2b). At the time, this
represented the highest published accuracy on
pitch accent detection of a single speaker using
only acoustic information. Using lexico-syntactic
features, such as vowel identity, syllable stress of
current and surrounding syllables, and POS, the
bagged CART models performed with 86.99%
accuracy—worse than the acoustic modeling.
However, when combined, they achieved a syllable
level accuracy of 92.78%.
472
In general, words are considered to bear accent
or not. However, some researchers have made
finer distinctions. Emphatic pitch accents were
defined as pitch accents that were perceived as
more prominent than normal pitch accents, adding
a third level to the level of prominence (Brenier,
Cer, & Jurafsky, 2005). It is unclear how consis-
tently this distinction is labeled, but approximately
40% of pitch accents were considered emphatic in
their experimental material—“child directed” sto-
ries read by a single female speaker. Brenier et al.
(2005), however, used pitch, energy, and duration
features to detect emphatic pitch accents, not-
ing that pitch range—defined by the height of the
maximum pitch value—is particularly useful in
distinguishing regular pitch accents from emphatic
ones. Aggregated features were extracted from the
current, previous, and following words and were
normalized by the surrounding intonational phrase.
Lexical features (term frequency–inverse document
frequency, whether the word was in an exclamation
or negation, and word class information) were also
explored. Although this work was intended to detect
emphatic accents, performance in detecting all
accents was also reported. The acoustic and lexical
feature sets performed comparably, detecting pitch
accent with 78.2% and 78.4% word level accuracy,
respectively. Combining them led to a significant
improvement, up to 84.4% word level accuracy.
The classifier was able to detect emphatic pitch
accents with 87.8% accuracy using both acoustic
and lexico-syntactic features, over a 79.8% baseline.
Manual annotation of prosody is very resource
intensive. It can take an expert labeler up to
100–200 times real time to perform a full ToBI
annotation (Syrdal, Hirschberg, McGory, & Beck-
man, 2001). To avoid the resource requirements of
manual annotation of training data for supervised
learning, several researchers have looked into unsu-
pervised and semisupervised learning techniques for
pitch accent detection. Ananthakrishnan and Naray-
anan (2006) examined the use of K-means, Fuzzy
K-means, and Gaussian mixture model clustering
for pitch accent detection. Using intensity-, pitch-,
duration-, and pause-based acoustic features—as
well as lexico-syntactic features capturing syllable
identity, POS, and lexical stress—the authors were

able to detect accents with accuracy up to 77.8%. This
technique was evaluated using seven BURNC speak-
ers, and it potentially suffers from the pitfall of using
BURNC syntactic features in speaker-independent
evaluations. Tamburini (2003a, 2003b) defined a
function, called the Prom function, which is a linear
combination of energy, duration, pitch and spec-
tral tilt features. An unsupervised thresholding of
the value of this function was able to detect pitch
accents with 80.2% accuracy at the syllable level.
Extending this to a supervised approach, manually
annotated data was used to learn weights for each
of the terms in the function; this improved accu-
racy to 82.5%. Levow (2006) evaluated the use of a
semisupervised technique—manifold regularization
(Belkin, Niyogi, & Sindhwani, 2006) in Laplacean
Support Vector Machines (SVMs)—and a spec-
tral clustering technique—asymmetric clustering
(Fischer & Poland, 2004)—for the task of automatic
pitch accent classification. Both of these techniques
were applied using feature vectors containing only
acoustic information. The maximum and mean of
pitch and intensity was extracted from the current
and surrounding syllable nuclei (usually the vowel),
as was the slope of the pitch. Differences between
these features and those extracted from surround-
ing syllables were also included in the feature rep-
resentation. Laplacean SVMs, the semisupervised
technique, classified pitch accents with 81.5%
syllable-based accuracy, whereas the unsupervised
technique, asymmetric clustering, achieved 78.4%
accuracy. Similar to Sun (2002) and Ostendorf and
Ross (1997), this technique classified pitch accents
into four classes—unaccented, high, low, and down-
step. The binary, accented versus unaccented, accu-
racy, however, cannot be determined from the error
reporting described in the article. Braunschweiler
(2006) developed a module dubbed the “Prosodizer”
that uses manually written scoring rules over pitch
energy duration and POS features to detect pitch
accent. These rules, despite being designed by hand,
performed remarkably well. They were able to
recognize pitch accents at the syllable level with
81% accuracy on a single BURNC speaker.
Using lexical information to recognize pitch
accent.  The approaches discussed earlier all have
used acoustic information to detect pitch accents,
Measuring the Voice
though some also employed lexico-syntactic fea-
tures. The task of identifying pitch accent locations
using only text-based features is clearly related to
acoustic accent detection, though it typically has a
somewhat distinct use. In general, when approaches
use only text features, it is for prosodic assignment,
as opposed to prosodic analysis. Prosodic assign-
ment is the task of describing a plausible intonation
for a piece of text. This is frequently used in TTS
systems, to assign the intended intonation for the
synthesized speech.
From a more theoretical point of view, Chomsky
and Halle (1968) posited that accent is entirely
predictable from the surface of an utterance.
Bolinger (1972) took an opposing point of view,
retorting with his famous article, “Accent Is Pre-
dictable (If You’re a Mind Reader).” There do seem
to be multiple valid accent placements for a given
utterance. The repetition of lexical content in the
BURNC, even within a standard speaking style, does
not reveal entirely predictable accent and phrase
boundary placement. However, lexico-syntactic
analysis is able to predict prosodic labels with
accuracy well above chance. This suggests that,
although not completely predicable, there is sig-
nificant information in the lexical content of an
utterance that informs which tokens are likely to be
accented. The following automatic approaches take
advantage of features extracted only from text to
predict pitch accent locations.
Hirschberg (1993) described a technique for pitch
accent assignment using POS information, complex
nominal analysis, and surface position informa-
tion. Also, a technique for modeling the information
status—Given versus New (Prince, 1981)—was
proposed. Following the discourse structure theories
posited in Grosz and Sidner (1986), a first-in-first-out
queue is used to store word roots to represent the
concepts in the current discourse. Although the
attentional and intentional structures are not explic-
itly modeled in this approach, word roots serve to
represent concepts that may not be strictly present
in the linguistic structure of the discourse. Ortho-
graphic cues, punctuation, and paragraph boundaries
are used to determine how the queue is manipulated
throughout the course of the analysis—with dif-
ferent signals indicating whether tokens should be
473
Rosenberg, Enos, and Hirschberg
added or removed from the queue representing local
focus. Using these features to train a CART classifier,
a cross-validation accuracy of 76.5% was achieved
on three speakers of BURNC material. Evaluations
on Audix (read news) and the Air Travel Informa-
tion System corpus (travel domain data) showed
improved performance (80% and 85%, respectively).
Ross and Ostendorf (1996) applied an HMM
over decision tree posteriors to detect pitch accents
from text. Similar to Wightman and Ostendorf
(1994), the decision tree models the likelihood of a
single syllable or word being accented, whereas the
HMM models any sequential phenomena concern-
ing accent and phrase final tone sequences. This
system used a multistage approach, first detecting
pitch accent, then assigning pitch accent type, and
finally assigning phrase boundary intonation. In
their experiments, intonational phrase boundary
location is assumed to be given. Based on the text
from a single BURNC speaker (f2b), POS, prosodic
phrase structure, Given/New status—simplified,
though similar to the representation in Hirschberg
(1993)—lexical stress information, and paragraph
structure were all extracted for training the deci-
sion tree model. Although this approach was trained
using syllables, the article reports accuracy at both
the syllable and word level, facilitating comparison
to other approaches. This technique was able to
predict pitch accent placement from text with 87.7%
syllable-based accuracy, corresponding to 82.5%
word accuracy. Many approaches to prosodic assign-
ment operate similarly to these two early articles.
The common form is to apply some supervised
learning algorithm to associate features derived from
POS tags, syntactic chunks, or syntactic parse trees
to pitch accent locations. The differences in these
approaches come down to the machine learning
algorithm used and the features derived. Hirschberg
and Rambow (2001) evaluated the application of
more sophisticated syntactic features to the task of
prosodic assignment. They applied the Ripper
(W. W. Cohen, 1995) rule-induction machine
learning algorithm using POS tags, supertags, and
parse tree derived features, as well as the length
and relative position of the current sentence. This
technique was evaluated on a single professional
speaker—material collected for use in an AT&T
474
speech synthesis application—and 88.2% word-
based accuracy was reported. They found word and
POS features to be sufficient for predicting pitch
accents, whereas the parse tree features were more
useful for phrase boundary prediction.
Marsi, Reynaert, van den Bosch, Daelmans, and
Hoste (2003) compared the performance of CART
modeling in pitch accent assignment to the perfor-
mance to memory-based learning. The lexico-
syntactic features that were examined included
the word identity, punctuation, POS, noun phrase
(NP) and verb phrase (VP) chunk location, infor-
mation content, term frequency–inverse document
frequency, and distance to the previous occurrence
of word and the sentence boundary. This was evalu-
ated on 201 articles from the Induction of Linguistic
Knowledge corpus, a collection of Dutch newspaper
text. Although the performance of both of the mod-
eling techniques are high—f-measure of 82.0 for
CART and 83.6 for memory-based learning—without
corresponding accuracy information, it is difficult
to assess the relative strength of these features and
modeling techniques compared to others.
Gregory and Altun (2004) applied conditional
random field modeling to the task of accent assign-
ment. POS tags were collapsed to broad classes
of Function, Verb, Noun, and Other. Based on
these broad features, probabilistic variables were
extracted, representing the log probability of a word
being accented given POS information. Unigram,
bigram, reverse bigram, and joint and reverse joint
log probabilities were all used in the model, as were
the number of phones and syllables in a word and
the position in the utterance. Using a conditional
random field with a window size of 5, 76.36% word-
based accuracy was reported. This evaluation was
performed on manually annotated Switchboard
corpus data (telephone conversation) with male
and female speakers with a variety of American
English dialects. It is unclear from this article how
many speakers are represented in their prosodically
annotated data, but the whole Switchboard corpus
contains more than 500 speakers (Godfrey, Holli-
man, & McDaniel, 1992). Similar approaches have
been applied with considerable success to the task
of phrase boundary assignment in addition to the
accent assignment approaches discussed here.

Nenkova et al. (2007) identified a simple lexical
attribute that is remarkably successful in detecting
pitch accent type given its simplicity. They defined
Accent Ratio to capture the rate at which a given
word in a training corpus was accented, and they
found that this feature yielded 75.59% accent detec-
tion accuracy on Switchboard data (Godfrey et al.,
1992); inclusion of other lexical features increased
accuracy to 76.65%.
In a slightly different application, Hasegawa-
Johnson et al. (2004) found that prosodic depen-
dent acoustic and language models can be applied
to improve ASR performance. They reported an
improved word error rate on the BURNC corpus from
24.8% using prosody independent models to 21.7%
using prosody-dependent models. This approach
used a prosodic assignment technique in the con-
struction of its prosody-dependent language model.
Syntactic parse tree and POS features from a range
of windows were used to train an Artificial Neural
Network model to assign pitch accents and phrase
boundaries (A. Cohen, 2004). Evaluated on six
speakers from the BURNC, this approach was able to
predict pitch accents with 83.1% accuracy.
The impact of speaking style on pitch accent
detection.  Rosenberg (2009) examined the role of
pitch, energy, duration, and voice quality features
for detecting pitch accent, finding across multiple
­corpora and machine learning algorithms that
energy features were the most predictive of pitch
accent. Duration features, in general, performed
with 2% less absolute accuracy. The relative per-
formance of pitch features differed by genre. On
professional broadcast news speech, pitch features
performed with accuracy approximately 10% lower
than energy features. On read material, the relation-
ship was less pronounced, with pitch features only
5% worse than energy features. On spontaneous
speech, pitch features performed as well or better
than the duration features, and roughly 2% worse
than the energy feature set. This suggests that energy
is a reliable predictor of prominence, supporting the
findings of Kochanski et al. (2005) and Silipo and
Greenberg (2000), who also found that pitch is a
weak predictor of pitch accent. However, although
energy features produced the best performance
in isolation, the inclusion of pitch and duration
Measuring the Voice
features improved the overall detection accuracy
on read speech. Because accented words are regions
that are perceived as more prominent acoustically
than their context, Rosenberg examined contextual
features, including the same pitch, energy, and dura-
tion features examined for the target word, testing
different amounts of context before and after the tar-
get, and obtained a boost in performance. Rosenberg
obtained best performance for pitch accent pre-
diction by constructing a classifier combination
technique in which energy-based predictions were
corrected by pitch and duration-based classifiers:
For each of 210 energy-based pitch accent detectors,
a correcting classifier was trained using pitch and
duration features to determine whether an energy-
based prediction should be trusted. If the correcting
classifier predicts that the classification is incorrect,
the prediction is inverted. Finally, the 210 cor-
rected energy-based predictors are combined using
a weighted majority voting decision. This technique
produces results for word-based acoustic pitch
accent detection performance at 85% on the BURNC
that out-perform previous attempts.
Automatic classification of pitch accent
type.  Detecting pitch accent as a binary classifica-
tion task is of course simpler than identifying the
different types of accents which occur in many
languages and can convey important differences in
meaning. As noted earlier, the ToBI prosodic annota-
tion framework describes intonation as a series of
high (H) and low (L) tones that are associated with
prosodic events (Silverman et al., 1992). For SAE,
the accent type inventory includes two simple pitch
accents (H* and L*) and three complex accents
(L+H*, L*+H, and H+!H*). Accents including
H tones may also be downstepped or uttered in a
compressed pitch range. Detecting accent types can
reveal much richer semantic information about what
a speaker is trying to communicate. For example,
in SAE, a sentence such as “John didn’t leave” can
convey that a speaker is certain or uncertain about
whether John left, depending on the type of pitch
accent on “John,” L+H*, or L*+H. Although L*
pitch accents are typically used with a high phrase
ending in yes–no questions, use of H* accents with
the same phrase ending can convey something
rather difference—a query about the listener’s
475
Rosenberg, Enos, and Hirschberg
f­ amiliarity with the topic being introduced. Pitch
accent type has also been found to be useful in
identifying contrastiveness (e.g., “A not B”) or focus
of attention (e.g., “Money is all John cares about”).
However, despite the importance of identifying
accent type, it is considerably more difficult than
binary accented/deaccented classification.
Pitch accent types are difficult to classify auto-
matically for two reasons. First, the distinctions
among some accent types (e.g., L+H* vs. L*+H)
may be subtle. Even human agreement on this task
is not high, especially compared to human agree-
ment on a binary classification into accented and
deaccented words. Pitrelli et al. (1994) reported
human agreement of only 64.1% on accent clas-
sification for SAE in the ToBI framework. If down-
stepped variants of accents are collapsed with their
nondownstepped forms, this agreement improves to
76.1%. Second, most pitch accents are H* in current
labeled corpora, including the BDC and BURNC
material. This skewed class distribution leads to a
very high baseline, at or above the rate of human
agreement, and also to a dearth of training data
available for the minority classes.
One technique that has been used in attempts
to simultaneously detect and classify pitch accent
in SAE represents pitch accent detection and clas-
sification as a four-way classification task, where
each word, syllable, or accent may be classified as
unaccented, high, low, or downstepped. This accent
inventory does not distinguish H* from L+H*
accent types. This technique was explored by Ross
and Ostendorf (1996) for prosodic assignment for
TTS. The input features explored were thus exclu-
sively lexically based, capturing POS information,
positional and structural information, as well as a
representation of information status. When evalu-
ated on one BURNC speaker (f2b), a decision tree
trained was able to classify accent types with 72.4%
accuracy over the 71.8% majority class baseline.
Although the classifier performed a four-way clas-
sification, this performance reflects the classifier
accuracy in differentiating accent types on syllables
that, in fact, bear accent. Incorporating acoustic
information, Ostendorf and Ross (1997) addressed
the same classification task using stochastic models.
This modeling structure predicts the joint probability
476
of pitch accent and boundary classes, simultane-
ously combining information from acoustic and
lexical streams. The approach described the combi-
nation of acoustic—pitch range, pitch contour, and
duration—and phonotactic models for classification.
Isolating a three-way (high, low, or downstepped)
classification performance, and assuming the
presence of intermediate phrase boundaries, the
technique classifies pitch accent with 83.4%
accuracy, over a baseline of 77.0%. This performance
is, again, achieved on speech from a single BURNC
speaker (f2b).
This four-way simultaneous classification and
detection approach was also used by Sun (2002).
In this work, boosting and bagging CART decision
trees were used for prosodic analysis. Similar to
Hirschberg (1993) and Ostendorf and Ross (1997),
this approach was evaluated on a single BURNC
speaker (f2b). Using only acoustic features, boosted
CART trees were able to classify accented words
with 74.3% accuracy, whereas incorporation of text
features improved the performance to 79.5%. Levow
also used this four-way classification for pitch
accent detection and classification under supervised
(Levow, 2005), as well as unsupervised and semisu-
pervised (Levow, 2006), learning approaches. Using
SVMs with context sensitive acoustic features, four-
way classification accuracy of 81.3% accuracy at the
syllable level is achieved. Using unsupervised spec-
tral clustering, 78.4% accuracy is reported, whereas
using the semisupervised technique (involving a
small amount of labeled data and a larger amount of
unlabeled data), 81.5% accuracy of Laplacian SVMs
is achieved. Although this work is also evaluated
on the same BURNC speaker, f2b, the results do
not isolate the accent classification from detection
performance. This makes it impossible to directly
compare the performances of these techniques with
previous techniques from a strictly pitch accent
classification point of view.
Read and Cox (2007a) also used this four-way
detection and classification approach for evaluat-
ing three prosodic assignment techniques: dynamic
programming (memory-based learning), N-grams
(language modeling), and decision trees. They
used POS and syntactic parse tree features. These
included representations of structural and identity

information such as parse tree distance between
tokens, surface position, and lexical stress of
­surrounding syllables, as well as vowel and syn-
tactic constituent identity. This work implicitly
acknowledged the impact of the skewed pitch accent
class distribution on the evaluation of classification
­techniques. The authors evaluated their four-way
classification performance using the balanced error
rate, the average recall over each class. Equally
weighting the contribution of each class balanced
error rate increases the contribution of minority
class performance.
Ananthakrishnan and Narayanan (2008) used
Rise/Fall/Connection (RFC; Taylor, 1998) and tilt
(Taylor, 2000) parameters along with word and
POS language modeling to classify pitch accents
as H*, !H*, L+H*, or L*. It is unclear how other
pitch accent types are addressed in this work. The
inclusion of the complex tone, L+H*, makes this
evaluation most similar to the approach we take in
this chapter. The major difference is the presence of
the downstepped high accent (!H*); the other two
complex tones, L*+H and H+!H*, are infrequently
observed. The best performing approach reported in
this work used only RFC parameters in a multilayer
perceptron model. When evaluated on six BURNC
speakers using leave-one-speaker-out cross-valida-
tion, accuracy of 56.4% over a majority class base-
line of 54.0% was obtained.
There has been some research on classifying
pitch accent types in languages other than English.
Fach and Wokurek (1995) described work on the
classification of accents in German as high or low
using HMMs. A forward looking series of pitch (f0)
estimates were used as input to the HMMs. Evalu-
ated on 60 sentences spoken by a single female
speaker, with a look-ahead of three 10-ms pitch
frames, 81% accent type classification accuracy was
reported. Obviously the language difference makes
comparison with other results impossible, but this
work suggests that HMM modeling or other sequen-
tial models may be able to capture contour shapes
with some success. GRToBI (Arvaniti & Baltazani,
2000) annotation uses a similar inventory of pitch
accent types as English, though they are used with
much more equal rates. The majority class of the
Modern Greek corpus is L*+H, with 30% of words
Measuring the Voice
bearing this accent type—compare this to the obser-
vation that nearly 80% of words in the BDC c­ orpora
are accented with H* and !H* accents. Xydas,
Spiliotopoulos, and Kouroupetroglou (2004) used
the Festvox wagon decision tree learning algorithm
(Taylor, Caley, Black, & King, 1999) to simultane-
ously predict accent location and type on a cor-
pus of 516 Modern Greek utterances describing
museum exhibits. Using POS and structural features
as well as punctuation and phrasing information,
they were able to correctly classify 71.9% of pitch
accent types.
The approaches described earlier rely on manu-
ally labeled annotation of accent types. Other into-
nation standards instead describe accent types using
continuously parameterized functions, such as Tay-
lor’s (1998, 2000) tilt model and the Fujisaki model
(Fujisaki & Hirose, 1982). Still other research-
ers attempt to identify an inventory of categorical
accent types directly from f0 data. This approach
involves identifying accented words and then auto-
matically clustering the pitch contours into a tax-
onomy. Oliver (2005) used this approach to cluster
accents in Polish into three classes using self-
organizing maps, whereas Iwano and Hirose (1998)
clustered Japanese accents into 11 contour types.
Both used accent type clustering as an intermediate
stage in another task: Iwano and Hirose found that
syntactic boundary detection could be improved by
the use of accent clusters. Oliver hypothesized that
accent clusters could be used for prosodic assign-
ment for speech synthesis applications.
Rosenberg (2009) explored a wide variety of fea-
tures and techniques for accent type identification in
SAE on the BDC and BURNC corpora. He first iden-
tified features that appear distinctive for the differ-
ent SAE accent types. Features considered included
the minimum, maximum, mean, and standard devia-
tion of pitch and energy extracted over the whole
word, as well as the word’s duration. These were
extracted over the raw pitch and energy contours, as
well as speaker-normalized contours using z-score
normalization. Tilt and skew coefficients of both the
raw pitch and energy tracks and relative locations
of the maxima of these contours as well as those of
immediately preceding and following slopes were
also considered. Results of a statistical analysis of
477
Rosenberg, Enos, and Hirschberg
variance of these features and their relationship to
pitch accent type showed statistically significant
(p < .001) correlations except for one feature—raw
(unnormalized) minimum pitch extracted from the
full word.
In subsequent classification experiments,
Rosenberg (2009) compared the earlier described
features when extracted from the full word versus
the energy-peak region of the word and compared
the relationship between acoustic features and
accent type in multiple genres. He also examined
the role of POS information, which could be useful
in combination with acoustic and prosodic features
if it can be inferred from automatically tagged ASR
output. To measure classification performance
on minority classes, he defined a combined error
rate metric to address the skewness of the data. He
examined the use of a number of aggregation of
pitch and duration information extracted from each
accented word, as well as three pseudo-syllabification
techniques: (a) one based on envelope valleys,
(b) a more complicated technique that includes
spectral and zero-crossing-rate information
(Villing, Timoney, Ward, & Costello, 2004), and
(c) alignment of lexicon based syllabification with
forced-alignment predictions. He also examined a
number of approaches to capturing the shape of the
pitch contour within the accented word or region.
When available, the lexicon-based syllabification
defines a region of analysis that leads to the best
performing pitch accent type classification. He also
reported that extraction of acoustic features from
the full duration of an accented word does not per-
form significantly better than extraction of features
from the pseudo-syllabification region derived
from the technique presented in (Villing et al.,
2004). Also, pitch shape features including tilt
parameters, which were designed to differentiate
types of prosodic events, were not able to classify
pitch accents as well as the simple aggregations of
pitch and intensity information. Furthermore, the
combination of these shape features with the aggre-
gation features did not improve classification per-
formance. He also proposed a Bayesian modeling
technique, quantized contour modeling, in which
acoustic contours are first quantized into a fixed
number of time and value bins. Models (he evaluated
478
Gaussian and multinomial models but others could
be used) are then trained for each time bin. Dur-
ing evaluation, the class that generates the greatest
posterior is selected as the classifier prediction.
However, the best performing parameterization
of quantized contour model is able to classify
pitch accents with a combined error rate that did
not significantly differ from the best performance
using acoustic aggregation features. Nonetheless,
combining the predictions of the two classification
approaches enables one to significantly reduce the
error rate below that of the aggregation classifier.
He also found that undersampling and ensemble
sampling the training data for SVM classifiers also
significantly improved pitch accent classification.
Using ensemble sampling, the combined error rate
on BDC-read data is reduced from 0.371 to 0.270.
This training modification is, by far, the most sig-
nificant factor in improving pitch accent type clas-
sification above the majority class baseline. He also
observed an influence of phrase accents (phrase-
final tonal phenomena which control the shape of
a pitch contour preceding an intermediate phrase
boundary) on pitch accent type prediction: When
a pitch accent is realized on the final word of an
intermediate phrase, classification performance is
significantly worse. Overall classification accuracy
could be improved if intermediate-phrase-final
accents could be modeled separately from accents
that fall earlier in an intermediate phrase. However,
identifying intermediate phrase boundaries accu-
rately is itself a difficult classification task. He also
confirmed an influence of POS-based, word-class
information on pitch accent type distributions.
However, this distinction also fails to improve over
acoustic classification; it also does not improve
classifier performance when combined with
acoustic features.
Intonational Phrasing
Intonational phrasing is the mechanism by which
speakers break up speech into meaningful chunks of
information. These boundaries typically occur at lin-
guistically meaningful points in and between utter-
ances to contribute meaning to the sentences uttered
(Pierrehumbert, 1979). For example, phrasing can
disambiguate syntactically ambiguous sentences

such as the following pair, where “|” indicates a
phrase boundary.
A. B ill doesn’t drink because he’s
unhappy.
B. Bill doesn’t drink | because he’s
unhappy.
In A, with no phrase boundary, the speaker conveys
that Bill drinks, but not because he is unhappy. In
B, Bill does not drink and the reason is because he is
unhappy. Phrasing can also disambiguate prepositional
phrase attachment:
A. I saw the man on the hill with the
telescope.
B. I saw the man on the hill | with the
telescope.
In A, with no internal prosodic phrase bound-
ary, it is most likely that the man is holding the
telescope—the prepositional phrase is attached to
the noun man. In B, with a boundary between hill
and with, it is more likely that the speaker is holding
the telescope—the prepositional phrase is attached
to the verb saw.
In the ToBI standard for SAE there are two lev-
els of prosodic phrase boundaries corresponding to
two levels of perceived disjuncture; for other lan-
guage such as Japanese, there may be more juncture
distinctions. For SAE ToBI, the phrase hierarchy
consists of an intonational phrase that, in turn, is
composed of one or more intermediate phrases, each
of which contains at least one accented word. Into-
national phrase boundaries are often indicated by
the presence of silence, and they are differentiated
from intermediate phrase boundaries by the pres-
ence of a final tonal event, called a boundary tone.
Intermediate phrase boundaries represent a smaller
degree of perceptual disjuncture and frequently are
not accompanied by silence. There are five major
acoustic indicators of prosodic phrasing: (a) the
presence of silence, (b) pitch and energy reset,
(c) preboundary lengthening, (d) changes in speak-
ing rate across the phrase boundary, and (e) tonal
changes at the end of a phrase. These all contribute
to the perception of increased disjuncture at a word
boundary. Note that although some features that
signal an upcoming phrase boundary occur quite
Measuring the Voice
locally, listeners can perceive the upcoming bound-
ary from features earlier in the phrase (Carlson,
Hirschberg, & Swerts, 2005). Silence is the most
reliable indicator of disjuncture. Pitch and energy
reset occur when at the beginning of new phrases,
when words are typically spoken louder and with
increased pitch, which decays over the duration of
the phrase. In the f0 domain, this phenomenon is
known as declination (Pierrehumbert, 1979). Phrase-
ending tones may indicate varying degrees of rise or
fall in the f0 contour, which themselves are mean-
ingful. For example, in SAE, declarative utterances
typically end with falling intonation and yes–no
questions with rising.
Intonational phrasing is critical in structuring
spoken information. A significant amount of
research has focused on techniques to identify
intonational phrase boundaries from speech as a
way to access this structural information. Also,
unexpected or erroneous phrasing decisions can
severely degrade the naturalness and intelligibility
of synthesized speech. TTS systems typically include
a prosodic assignment module that is responsible
for performing lexical analysis and assigning either
prosodic event locations or f0, energy, and duration
targets. Many prosodic assignment techniques can
be used in the analysis of spoken intonation.
If a hypothesized word stream is available, along
with the speech signal, prosodic assignment can be
used to generate syntactic hypotheses for phrase
boundary locations. This, in effect, represents a
prior on the likelihood of a phrase boundary occur-
ring at a given word boundary based on the lexical
content. The acoustic information carries the per-
ceptual qualities that determine whether a phrase
boundary is present or not, whereas the lexical and
syntactic information in the lexical stream can con-
tribute by indicating how likely, in fluid, natural
speech, a phrase boundary is to occur. In this sec-
tion, we provide a survey of previous approaches to
automatic phrase boundary detection, first discuss-
ing approaches that use acoustic information, and
then we present some of the prosodic assignment
techniques that have been applied to this task.
Automatic detection of phrase boundaries has
been shown to successfully contribute to the per-
formance of many automatic spoken language
479
Rosenberg, Enos, and Hirschberg
processing tasks. As discussed in the earlier exam-
ples, the relationship between syntax and prosodic
phrasing has been the subject of considerable
research. There is evidence that syntactic ambiguity
can be reduced by referring to intonational informa-
tion (Beach, 1991; Veilleux & Ostendorf, 1991) and
that intonational information can improve auto-
matic parsing (Ostendorf, Wightman, & Veilleux,
1993). The speech synthesis community has been an
important factor motivating the investigation of the
relationship between syntactic and prosodic phras-
ing. Prosodic structure plays a powerful role in the
naturalness and intelligibility of synthesized speech
(Díaz, van Santen, & Banga, 2009; Ostendorf &
Veilleux, 1994; Silverman, Kalyanswamy, Silverman,
Basson, & Yashchin, 1993). Taking account of the
syntactic and semantic content in intonation genera-
tion for synthesized speech can be used to improve
the naturalness of the synthesized intonation
(Prevost & Steedman, 1993). From a recognition
point of view, there is evidence that the inclusion of
prosodic phrasing and other prosodic information
can be used to improve ASR performance (Hasegawa-
Johnson et al., 2004; Heeman, 1999; Stolcke, Shri-
berg, Hakkani-Tur, & Tur, 1999). Hypothesized
intonational phrase boundaries can also be used to
improve the performance of automatic story/topic
segmentation of broadcast news (Rosenberg et al.,
2007) and extractive speech summarization systems
(Maskey et al., 2008). From a human perspective,
success in these tasks provides some evidence that
speakers provide reliable signals to topic boundaries
through their placement of intonation boundaries,
and, in addition, intonational phrases appear to play
a role in defining phrases that can usefully summa-
rize topics, at least in the news domain.
Automatic detection of phrase boundaries.  In an
early effort to detect intonational phrase boundaries,
Wightman and Ostendorf (1994) used a cascaded
approach, building decision trees with acoustic
features to serve as input to an HMM classifier. They
performed a four-way classification, simultaneously
detecting (binary) pitch accents and intonational
phrase boundaries. The system extracted syllable-
level features that included length of following
silence, presence of an audible breath, acoustic
480
features such as the mean pitch and energy, and
the ratio of these values across syllable boundaries.
The feature-set also included measures of prebound-
ary lengthening (detected from syllable rhyme dura-
tions normalized by phone and speaker identity)
and changes in speaking rate, and the difference in
syllable duration of the mean of three syllables prior
to a candidate boundary and three syllables follow-
ing was used to capture changes in speaking rate.
On speech from a single BURNC speaker (f2b), this
approach achieved phrase boundary detection accu-
racy of 94.15% with an F measure of 0.76.
In a similar study, Batliner et al. (1995)
employed a number of acoustic features to predict
phrase boundaries in German, which were deter-
mined syntactically via a set of hand written rules
prescribing the relationship between syntactic struc-
tures and the presence of phrase boundaries. The
acoustic features used by these researchers included
length of pause after the potential boundary loca-
tion (if any) and syllable and nucleus durations,
normalized by phoneme class of syllable nucleus.
Pitch and energy reset features were determined by
extracting the maximum energy within two syllables
surrounding boundary as well as the location of this
maximum. To model reset associated with pitch
declination, linear regression coefficients of the
pitch contour over two and four syllables on either
side of the boundary were also calculated. From
these features, a three-level phrasing classification
had an accuracy of 81%. Although this experiment
confirms the general understanding that syntax
does not determine prosodic boundary location, its
degree of success does show a relatively close asso-
ciation between syntactic and prosodic phrasing—at
least in German.
In other work on phrase classification in the
BURNC, K. Chen et al. (2004) combined a neural net-
work syntactic-prosodic model with an acoustic Gauss-
ian mixture model. Using the leave-one-speaker-out,
cross-validation approach they were able to achieve
93.07% accuracy. Their syntactic models generated
90.09% accuracy when evaluated separately, whereas
the acoustic-only model yielded only 68.15%. How-
ever, like many other classification experiments run
on the BURNC data (e.g., Ananthakrishman & Naray-
anan, 2008; Sridhar et al., 2008), this work does not

control for the repetition of stories (and thus lexical
and syntactic information) by multiple speakers, so
that training and test sets are truly distinct and that
lexico-syntactic models are not trained on material
they are tested on. In fact, speaker independent evalu-
ation of lexico-syntactic features is not possible on this
corpus, as the material spoken by each speaker is also
present in the corpus produced by another. Therefore,
it is difficult to evaluate the success of this and other
syntactic-prosodic classification experiments using
a leave-one-speaker-out validation approach. Yoon
(2006) used a subset of BURNC to predict intonational
phrase boundaries using the Tilburg Memory-Based
Learner. Features used in this prediction included
numbers of phones, numbers of syllables, the presence
of lexical stress, as well as POS tags, syntactic chunking
information, named entity tags, and argument struc-
ture class (subject, object, and predicate). This work
achieved an accuracy of 92.23% and an F1 of 0.861 in
a binary classification of boundaries. A three-way clas-
sification of intonational, intermediate, and no-phrase
boundary attained an accuracy of 88.06% with an F1
of 0.832 for intonational phrases and an intermediate
phrase boundary F1 of 0.345. This represents the best
reported performance on intonational and intermediate
phrase boundary detection on BURNC, although direct
comparisons are difficult due to different training and
test splits. However, although the author acknowl-
edged that there are multiple productions of the same
story in the test data, it is unclear whether stories in
the test material are also present in the training data
in these experiments, so the status of the numbers
achieved is unclear.
Most research on phrase detection employs
manually annotated or hypothesized word or syllable
segments, although a few frame-based approaches
have also been attempted. In early work, Hirschberg
and Nakatani (1998) classified intonational phrases
from 10-ms frames pitch values, a binary voicing fea-
ture, and RMS energy (with context of up to
27 frames, or 270 ms) speaker normalization and
delta values. Using CART, they achieved a mean F1
of 0.697 and accuracy of 83.6% on the BDC corpus,
where both the read and spontaneous materials were
analyzed as a single data set. Sridhar et al. (2008)
also experimented with short (10 ms) frame acous-
tic information. These researchers employed f0 and
Measuring the Voice
RMS energy values, along with change from previous
frames, in a maximum entropy model. They reported
detection accuracy of 84.1% on material from four
BURNC speakers (f1, f2, m1, m2) and 83.53% on
BDC material from both the read and spontaneous
subcorpora. Under leave-one-speaker-out cross-
validation, incorporating a syntactic model using
POS tags and supertags with HMM acoustic models,
they reported accuracies of 92.91% on BURNC and
90.58% on BDC.
Using 20 acoustic features including raw and
normalized pitch and intensity features for each
word plus duration and silence features and a
number of different machine learning techniques,
Rosenberg (2009) found that, not surprisingly,
silence was the best cue to boundary location, with
pitch and intensity features contributing only a
small additional performance boost. However, pitch
and energy together could predict boundaries when
no silence was present—particularly important for
detecting intermediate phrase boundaries. With
only very simple features, he could predict whether
a word is followed by a prosodic boundary with
F1 = 0.807 on spontaneous speech from the BDC.
Adding contextual information did improve per-
formance for intensity features, with difference in
normalized intensity slope across word boundaries
proving useful. The mean intensity slope of a word
tends to be greater than the slope of the subsequent
word—except at phrase boundaries, where the
energy slope resets, with a dramatic increase. This
reset in slope may contribute to human perception
of disjuncture. Duration was also a moderately suc-
cessful predictor of phrase boundary location: On
all corpora examined, words that end intonational
phrases were significantly longer than those that did
not, signifying preboundary lengthening. However,
this effect may also be due to the additional finding
that significantly more of the words that ended into-
national phrases were accented, compared to those
that did not—nearly twice as many across all cor-
pora, because accented words tend to be longer than
deaccented ones. Also, more phrase ending words
tend to be content words, which also tend to be lon-
ger than function words.
Overall, it is clear that silence is the single
most powerful indicator of intonational phrase
481
Rosenberg, Enos, and Hirschberg
boundaries. Although there are instances of intona-
tional phrase boundaries that do not coincide with
silence, it is rare for silence to be present within
an intonational phrase. Energy is more reliable for
detecting phrase boundaries than pitch is. Phrasing
is indicated by perceived disjuncture, commonly
signaled by acoustic reset. Features representing a
change in intensity are more discriminative to the
presence of phrase boundaries than similar features
capturing changes in pitch. POS tags that surround
a candidate boundary are consistently superior to
traditional N-gram features. Modeling the likeli-
hood of a phrase boundary falling between POS tags
is dramatically better than modeling the likelihood
that a phrase boundary falls after a sequence of POS
tags. Finally, detection of intermediate phrases is
more difficult than intonational phrases. Interme-
diate phrases are indicated by a smaller degree of
disjuncture than intonational phrases. This differ-
ence in degree may lead to intermediate phrasing
being a fundamentally subtler phenomenon than
intonational phrasing. This difference may also lead
to reduced human annotation consistency, which, in
turn, would limit automatic detection performance.
This current technology limitation also limits
researchers’ ability to use these more fine-grained,
linguistically meaningful units of human speech
in the automatic assessment of the correlation of
intonational variation with speaker meaning and
speaker state.
Detecting phrase boundaries in non-English
languages.  There has also been a significant
amount of research in detecting phrase boundaries
in languages other than English. Some correlates
to phrasing are likely to be language independent.
For example, the presence of an audible breath,
or silence, most likely indicates a prosodic phrase
boundary in all languages. Also, speakers typi-
cally insert prosodic phrase boundaries at the end
of a sentence. This phenomenon is not unique to
English. However, other acoustic and syntactic
factors that indicate intonational phrase boundar-
ies may differ across languages in both degree and
dimension. Although research in other languages
can be used to identify areas for investigation or
feature engineering techniques, there is no expec-
tation that approaches that are successful in one
482
language will necessarily be successful in another.
Heldner and Megyesi (2003) exploited preboundary
lengthening features, representations of speaking
rate changes, and a variety of word-class features
and the presence of silence to detect intonational
phrases in Swedish radio interview speech. Using
discriminant analysis, they achieved a three-way
accuracy of 85.3%. In this study, lexical features
performed better than acoustic information in
the detection of “strong boundaries”—intona-
tional phrase boundaries. Thus, acoustic features
were more helpful for detecting weak (intermedi-
ate) phrase boundaries. Although most believe
Germanic languages such as Swedish are similar
to SAE regarding prosodic variation, this conclu-
sion has not been confirmed on SAE material. On
German and English material, Schmid and Atterer
(2004) applied a HMM over POS tags to the task.
They reported an intonational phrase boundary
F1 of 0.778 on Machine Readable Spoken English
Corpus data, 0.8023 on German read newspaper
data, and 0.853 on German radio news data using
a syllable level of analysis. They found syllable-
based analysis to outperform word-based analysis
when measured by F1. They model the POS of at
a syllable level by inserting a dummy tag at word
internal syllables. The fact that this syllable level
representation performs better than a correspond-
ing representation of words suggests that the
modeling of syllable length helps detect phrase
boundaries. These sequences of dummy POS tags
correspond to number of syllables per word. Nakai,
Singer, Sagisaka, and Shimodaira (1995) used
memory-based classification with clusters of f0
contours to detect phrase boundaries in Japanese.
Following the Fujisaki superpositional intonation
model (Fujisaki & Hirose, 1982), accent and phrase
components were stored and clustered. Dynamic
programming is used in a technique similar to
dynamic time warping to generate predictions.
Approximately 75% of accent phrase boundaries
were correctly detected in phoneme-balanced TTS
data based with male speakers. This is impressive
performance considering the technique uses only
pitch contour information. The performance of this
approach could likely be improved further with the
inclusion of other observed correlates of prosodic

phrasing, such as energy, lengthening, and syntactic
information.
Speaking rate is another useful indicator of
phrasing. In predicting phrase boundaries in Korean,
Kim and Oh (1996) found that modeling speaking
rate could improve automatic boundary detection,
reducing error by approximately 20% on a three-
level classification task. Wightman and Ostendorf
(1994) also used representations of speaking rate in
their work on phrase boundary detection.
On Modern Greek material, Zervas, Maragouda-
kis, Fakotakis, and Kokkinakis (2003) and Mara-
goudakis, Zervas, Fakotakis, and Kokkinakis (2003)
have applied naive Bayes, Bayes net, and CART
­classifiers to the task of phrase boundary assign-
ment. Using surrounding POS and shallow chunk-
ing features tags, Bayes net classification achieved an
intonational phrase F1 of 0.796. Xydas et al. (2004)
used manually enriched text markup to predict
phrase boundaries in modern Greek. NPs in the text
are marked with intonational focus features incorpo-
rating newness and “validation” factors—whether
the NP is the second argument of a verb, whether
deixis (e.g., pronouns or words such as here and
that whose referent must be obtained from context)
is present, and whether the NP contains a proper
noun. These features along with POS information
classified the degree of disjuncture following each
NP into four classes with 89.2% accuracy. This
is an example of how additional lexical analysis
can be used to improve prosodic phrase boundary
prediction. Although in this work the markup of
additional information is done manually, the results
show that extraction of this type of focus informa-
tion can contribute to hypotheses of phrase bound-
ary location.
Using Chinese news data, J.-F. Li, Hu, and Wang
(2004) and J.-F. Li, Hu, Wang, and Dai (2005)
applied a two-stage approach to phrase break assign-
ment. An initial set of hypotheses was generated
with maximum entropy (logistic regression) model-
ing using word identity, POS tags, and the number
of syllables in a word. A prosodic phrase length
distribution function was then composed with
these hypotheses to model the expected length of
an intonational phrase. In isolation, the maximum
entropy model generated an F1 of 0.662; the length
Measuring the Voice
distribution function improved this to 0.697, com-
pared to a human consistency F1 of 0.750. This is a
combination technique that could be applied to SAE
speech as a way to merge individual phrase bound-
ary hypotheses with phrase length information.
Also working with Chinese material, Hu, Chen,
Fan, and Wang (2003) hypothesized that phrase
breaks are more likely to occur between uncor-
related words, and they used mutual information
to capture this. They found a 2.4% improvement
to a decision tree classifier by including mutual
information measures. By applying an argmax func-
tion to a weighted combination of the likelihood of
the POS model and mutual information likelihood
(calculated at different windows surrounding the
candidate), they achieved an intonational phrase
boundary F1 of 0.708.
Using lexical information to detect phrase
boundaries.  Other approaches to phrase boundary
detection use only text-based information, relying
largely on correlations between phrase boundary
location and syntactic and positional informa-
tion. Many of these have a similar flavor. A typical
approach is to train syntactic-prosodic models to
learn associations between text analysis and phrase
boundary locations that have been identified from
transcribed speech by hand. Most of these have been
done in aid of TTS synthesis, although findings have
also been applied more generally to phrase bound-
ary detection using acoustic-prosodic as well as
text information. Major work in this area has been
done by Bachenko and Fitzpatrick (1990), who
defined hand-written rules for identifying candidate
boundaries, and who subsequently identified the
“salience” of these. Using a small evaluation cor-
pus of 35 sentences, their system predicted spoken
intonational phrase boundaries with 80% accuracy.
Veilleux and Ostendorf’s work on the interac-
tion between prosody and syntax (Ostendorf &
Veilleux, 1994; Veilleux, 1994) took a hierarchical
approach in which output from single models was
used as input to a hierarchical modeling technique.
In effect, this approach first predicted intermedi-
ate phrase boundaries, using these hypotheses to
predict intonational boundaries. Overall accuracy of
nearly 93% was reported by applying this technique
to a 400-word data set. Collapsing over both phrase
483
Rosenberg, Enos, and Hirschberg

types, 81% correct predictions with only 4% false information can be helpful in detection as well.
predictions were reported. Other approaches have Such information includes the following:
made use of decision trees and have employed a
■■ Constituent features: For example, for predicting
wider variety of features. Wang and Hirschberg
whether there is a boundary between subsequent
(1991, 1992) applied CART trees to a feature vector
words w[i] and w[i + 1], use the following as
containing parse tree and phrase positional features
features: the identity of (1) the largest constituent
along with POS tags. This approach was able to
containing w[i] and not w[i + 1], (2) the larg-
achieve more than 90% accuracy. Hirschberg and
est constituent containing w[i + 1] and not w[i],
Prieto (1996) extracted POS tags and morphologi-
and (3) the smallest constituent containing both
cal category information from a symmetric four-
w[i] and w[i + 1]. A special “None” token is cre-
word window surrounding each word boundary.
ated for instances when these constituents do not
Using these nominal features along with numeri-
exist. These features were introduced by Wang and
cal features representing the number of words and
Hirschberg (1991, 1992).
syllables in the utterance, the relative position of
■■ Positional features: The absolute and relative
the current boundary, distance from punctuation,
position of each word within its narrowest con-
and location within an NP, they found that CART
stituent, from both the start and end. Wang and
trees could assign intonational phrase boundaries
Hirschberg (1991) found a strong influence of
with 95% accuracy on English material and 94%
phrase length on the presence of a phrase bound-
in Spanish. Koehn, Abney, Hirschberg, and Collins
ary. That is, phrase boundaries are more likely
(2000) extended this approach by including syntac-
to occur if it has been a long time since the last
tic parse tree features, improving accuracy by 1.8%
phrase boundary. Phrase boundary position
and F1 of intonational phrase boundaries by 0.0471
information can be approximated by including
using these and other syntactic parse features,
the number of words since the last silent region
along with supertags (Bangalore & Joshi, 1999).
more than 200 ms was observed. Note that this
Hirschberg and Rambow (2001) used the Ripper
is not strictly a lexical feature, but it serves as a
rule-induction algorithm to assign intonational
crude proxy for sentence boundary information.
and intermediate phrase boundary locations from
■■ Parse tree distance features: The degree of syn-
text. On read Wall Street Journal text, they were able
tactic disjuncture by the parse tree, calculated as
to achieve an F1 of 0.701 on intonational phrase
distance between w[i] and w[i + 1]. Treating a
boundary detection with 89.8% overall accuracy.
parse tree as a graph, the parse tree distance is the
When these results were collapsed across interme-
path length between the nodes representing w[i]
diate and intonational phrases, the F1 increased to
and w[i + 1] (as used by Read & Cox, 2007b).
0.873, with an accuracy of 86.1%. Syntactic parse
To avoid overpenalizing complex syntactic
tree features similar to those used in Koehn et al.
constructions, this distance may be normalized
and Hirschberg and Rambow were examined by
by the parse tree depth of the deepest common
Read and Cox (2007b). These features treat a parse
ancestor of w[i] and w[i + 1].
tree as a graph, representing numerical syntactic
distances between two words by the graph distances Of course, if w[i] and w[i + 1] appear in differ-
between their corresponding parse tree nodes. ent sentences, they will not have a smallest covering
Although syntax does not determine prosodic syntactic constituent. Intonational phrase boundar-
phrasing, there is a relationship between the two. ies are very common (98.7% of the time) when the
The study of the relationship between the words and smallest syntactic constituent covering w[i] and
syntax of a sentence and the appropriate prosody w[i + 1] does not exist at such sentence boundar-
to use in realizing that sentence in speech has been ies. On the other hand, phrase boundaries are quite
important in the automatic assignment of prosody rare within prepositional phrases; in only 1.74%
in TTS systems. Although this task is distinct of instances where the smallest covering constitu-
from phrase boundary detection, lexico-syntactic ent is a prepositional phrase does {w[i], w[i + 1]}

484

represent an intonational phrase boundary. Intona-
tional phrase boundaries are more common when
the smallest covering constituent is a VP rather than
an NP—with rates of 23.5% and 16.0%, respectively.
Intonational phrase boundaries are relatively com-
mon, with a rate of 77.8%, when the largest con-
stituent covering w[i] but not w[i + 1] is an unlike
coordinated phrase. Unlike coordinated phrases are
instances of a coordinated phrase across syntactic
categories. Moreover, intonational phrase bound-
aries are more likely to occur at the end of VPs
(53.4%) than at the end of NPs (45.3%). However,
they are more likely to fall at the start of an NP than
a VP; 41.1% of NPs are immediately preceded by an
intonational phrase boundary, whereas only 22.5%
of VPs are. Moreover, 66.6% of constituents with a
coordinating conjunction as its head word are pre-
ceded by an intonational phrase—as are 50.6% of
adjectival phrases. The expected rate of intonational
phrase boundary placement is 19.27%. Each of these
rates is significantly different from these expected
rates of intonation phrase boundary placement.
In addition to these parse tree features, POS-
based, word-class features can be useful in predict-
ing phrase boundaries. Unigram information as
well as bigram, trigram, and surrounding bigram
and four-gram windows can be examined. These
prosodic/syntactic “language models” operate only
on the surface form of the syntactic structure and
have no access to the deeper parse information.
They are therefore able to capture a distinct, more
linear aspect of syntactic information than the parse
tree features and often perform quite competitively.
These two approaches to capturing syntactic indica-
tors of prosodic boundaries can also be combined
to attempt improvements in classification; however,
the information they provide has so far appeared to
be redundant.
Combining syntactic and acoustic features and
using AdaBoost (adaptive boosting; a machine
learning meta-algorithm) and single split decision
trees, Rosenberg (2009) achieved the best text-
independent intonational phrase boundary detection
on the BURNC and BDC corpora. (Note that higher
performance has been reported on the BURNC.
However, these results are based on training and test
divisions that contain readings of the same lexical
Measuring the Voice
material.) Previous results have reported similar
accuracies and in some cases slightly higher F1
values on other corpora. Schmid and Atterer (2004)
reported an F1 of 0.778 in predicting phrase bound-
aries on Machine Readable Spoken English Corpus
data—a corpus of standard southern British English.
Hirschberg and Prieto (1996) reported 95%
accuracy on a corpus of newswire text.
Examining phrasing generation in a concept-
to-speech system, McKeown and Pan (2000) took
advantage of the greater amounts of information
available from concept systems to improve pro-
sodic assignment. By using features based on lexical
content, concepts, syntactic functions, semantic
boundaries (the length of the current “semantic con-
stituent”), as well as POS and information content
(a representation of the “semantic informativeness”
of a word), they applied memory-based learning
to the task of assigning ToBI labels to concept-to-
speech output. They achieved an accuracy of 78.69%
in the assignment of all levels of break indices—a
five-way classification task.
Automatic classification of phrase boundary
types.  The ToBI standard segments phrase-final
intonation into two components, a phrase accent and
a boundary tone, either of which can be described as
having a high (H) or low (L) tone. The phrase accent
occurs at the end of each intermediate phrase, and it
describes the pitch contour from the last pitch accent
to the end of the phrase. High phrase accents (H-)
indicate that the pitch is rising or stable, whereas low
phrase accents (L-) indicate a falling pitch. There
is also a downstepped high phrase accent, which
appears similar to a stable H- phrase accent but falls
to somewhere in the middle of the speaker’s pitch
range, as opposed to high in the pitch range. At full,
intonational phrase boundaries, the phrase accent is
followed by a boundary tone. High boundary tones
(H%) indicate a final rise immediately prior to the
boundary, whereas low tones (L%) indicate the pres-
ence of a stable or falling pitch.
Each intonational phrase boundary thus has
associated with it a phrase accent and boundary
tone. Phrase boundary type is often performed on
combinations of phrase accent and boundary tone,
that is, on the ToBI label combinations of L-L%,
485
Rosenberg, Enos, and Hirschberg
L-H%, !H-L%, H-L%, and H-H%. The rationale for
this is the fact that the acoustic realization of phrase
accents and boundary tones is highly interdepen-
dent. Boundary tones are realized quite differently
in the context of different phrasal tones. Conversely,
phrase accents are realized with different slopes and
at different locations within a speaker’s pitch range
depending on the following boundary tone. Stylized
versions of these combinations appear in Appendix
18.1. Hirschberg and Pierrehumbert (1986) pos-
ited a compositional approach to the interpretation
of categorical events, but they had some difficulty
decoupling the impact on communication between
phrase accents and boundary tones (see also Pier-
rehumbert & Hirschberg, 1990). They posit that H%
can signal a hierarchical relationship between the
intentions of the current and subsequent utterance,
or satisfaction-precedence relationship (Grosz &
Sidner, 1986), whereas phrases ending with L% can
be interpreted without reference to a subsequent
utterance. Moreover, they found that H- phrase
accents and H% boundary tones both indicate that a
phrase should be interpreted with particular respect
to the following phrase, whereas boundary tones
refer to relationships between intonational phrases,
and phrase accents refer to relationships between
intermediate phrases. They found evidence that H-
can be seen to form part of a larger composite when
used within an intonational phrase, with
L- phrase accents highlighting the separation of the
component intermediate phrases. However, when
both a phrase accent and boundary tone are present,
particularly in the case of intonational phrases con-
taining a single intermediate phrase, Pierrehumbert
and Hirschberg (1990) noted “it is more difficult to
separate the meaning of the phrase accent from the
meaning of the boundary tone” (p. 302). In broad
strokes, their findings suggest that high tones
(H- or H%) indicate some degree of connection to
the subsequent phrase, whereas low tones (L- or L%)
indicate greater completeness of the phrase. Bartels
(1999) similarly proposed that L- phrase accents are
essential to declarative intonation, where H- phrase
accents are rarely used in statements, being much
more common in questioning intonation. Bartels
described boundary tone effects as “potential discourse
finality” and “continuation dependence” (p. 58).
486
Phrases that end with a low boundary tone, L%, are
potentially discourse final, whereas those containing
high boundary tones, H%, contain some continua-
tion dependence. This understanding of boundary
tones is consistent with that of Pierrehumbert and
Hirschberg.
In addition to these general compositional
approaches to contour interpretation, interpreta-
tions of phrase-final tones have also been inves-
tigated empirically. The falling contour, L-L%
phrase-final ending, appears to indicate a greater
degree of finality in SAE. The L-H%, or “continu-
ation rise” contour, is thought to communicate
that there is more to come (Bolinger, 1989) and is
frequently used in list intonation (Hirschberg &
Ward, 1992). Combined with L*+H pitch accents
it can indicate uncertainty or incredulity (Hirsch-
berg & Ward, 1992, 1995; Ladd, 1980). H-H%
endings convey the impression that the speaker
is seeking confirmation—as in yes–no questions,
tag questions, and declarative questions (Gussen-
hoven, 1983)—or may be interpreted as deferen-
tial (Lakoff, 1975). Contours ending in a plateau,
H-L%, convey a degree of incompleteness and
may be used to hold the turn (Duncan, 1972; Gra-
vano, 2009). This same contour and use has been
observed in many languages, including German
(Selting, 1996), Japanese (Koiso, Horiuchi, Tutiya,
Ichikawa, & Den, 1998), and Dutch (Caspers,
2003). Contours ending with a plateau also carry a
bored, recitation effect with the implication, “you
already know this” (Hirschberg, 2002a). More-
over, it may also be indicative of negative emotion
(Benus, Gravano, & Hirschberg, 2007).
There is relatively little research on automatic
phrase-final type classification compared to binary
phrase boundary identification. A commonly cited
evaluation of human agreement of ToBI annota-
tion (Syrdal & McGory, 2000) evaluated pairwise
inter-annotator agreement for phrase-final tones
at approximately 85%. This established the high-
est automatic classification performance that can
be expected without overfitting the behavior to a
particular human labeler. Moreover, manually anno-
tated corpora are often labeled by multiple ToBI
experts. In these situations, the internal consistency
of the corpus itself is limited by the interannotator

agreement, limiting the maximum performance
achievable by supervised learning algorithms.
Early work on phrase-final classification
used only lexical information to determine the
desired phrase-final type for a speech synthesis
application (Ross & Ostendorf, 1996). In this work,
a decision tree was trained to assign one of three
phrase-final types—L-L%, L-H%, and H-L%—using
POS, information status, the location of the phrase
boundary with respect to the surrounding sentence
and paragraph, and the number of syllables in
the phrase and since the previous pitch accent.
When trained and evaluated on the material of a
single speaker in the BURNC, this approach
correctly predicted 72.4% of phrase-finals over a
majority class (L-L%) baseline of 61.1%. This is a
heavily speaker dependent evaluation but represents
an early success in the prediction of phrase finals.
Tepperman and Narayanan (2008) used a tree
grammar to model prosodic structure similar to that
developed by Hirschberg and Rambow (2001). They
were able to model the relationship between syl-
lable stress and accents as well as intermediate and
intonational phrases with this tree grammar better
than with N-gram models. The authors evaluated
this model’s performance, trained as a weighted
regular tree grammar, by predicting prosodic event
categories with all but the target label given. Under
this speaker-independent evaluation routine, the
weighted tree grammar was able to correctly predict
75.08% of boundary tones and 88.22% of phrase
accents. This accuracy is quite high. However, the
leave-one-label-out evaluation provides substantial
additional information to the classification tech-
nique that cannot be expected to be available in an
automatic classification setting.
These approaches analyzed the use of lexical
content to predict phrase-finals. Other work has
addressed the acoustic correlates of phrase-finals in
speech. Early work by Price, Ostendorf, and Wight-
man (1989) was able to distinguish L-L% and
L-H% phrase finals with 76% accuracy. This clas-
sification was performed using an HMM model
decoding quantized f0 values and was evaluated on a
single radio news speaker.
Murray (2001) used boundary tone classification
as an application to evaluate the negative impact of
Measuring the Voice
pitch tracking errors. This work classified final-rise
(H%) and final-fall (L%) boundary tones using a
single, hand written rule. The rule was that if the
mean pitch of the final vocalized region was higher
than the mean pitch of the rest of the utterance, or
if the slope of the pitch within the final vocalized
region was positive, the phrase final was consid-
ered a final-rise (H%). Otherwise, it was considered
final-fall (L%). Although the goal of this study was
to examine the impact of pitch tracking errors, not
to generate high accuracy phrase-final predictions,
this single rule was able to differentiate these two
boundary tones with 68% accuracy using automatic
pitch information. This accuracy increased to
77% if the pitch track is manually corrected. This
work provides objective measurement of the degree
to which pitch tracking errors negatively impact
the performance of all automatic phrase-final
classification techniques.
Ishi (2005) classified phrase-finals in Japanese
using an inventory of six types: long rise, short
rise, long flat, short flat, weak flat, and long fall
(as proposed by Toki & Murata, 1987). This work
extracts aggregations of the pitch contour from the
first and second half of the phrase-final region. The
phrase-final region is defined as the vowel and coda
of the phrase-final syllable. Using features that relate
the aggregations drawn from the first and second
half of the phrase-final region, Ishi trained a deci-
sion tree that was able to classify the phrase-finals
of a single speaker with 79.2% accuracy. For SAE,
Ananthakrishnan and Narayanan (2008) studied
the importance of various f0 models for boundary
tone classification. They evaluated the performance
of tilt and RFC parameters (as discussed earlier)
and found that they could classify L-L% from L-H%
phrase final boundaries with 67.7% accuracy using
speaker-independent evaluation. This performance
was achieved using RFC f0 parameterization and a
language model trained over categorical prosodic
events and evaluated on the BURNC.
Rosenberg (2009) examined simple acoustic
features as well as features capturing f0 shape and
parse tree features to achieve the best results to date
on phrase-final classification in the BURNC. He
found that pitch was more useful than energy for
classifying phrase ending intonation, although both
487
Rosenberg, Enos, and Hirschberg
contribute. He also found that examining data from
the 200 ms prior to a phrase boundary was sufficient
for high accuracy classification. Parse-tree informa-
tion was useful in improving classification, although
less discriminative than the acoustic features. He also
noted that the same approaches can be used to clas-
sify intermediate phrases and intonational phrases.
TOOLS FOR PROSODIC ANALYSIS
Over the years a large number of tools have been
developed for speech annotation, visualization,
manipulation, and analysis. These differ not only in
their capabilities but in their availability, support
provided, and ease of use. Free software includes
Praat, Wavesurfer, the OGI Speech Toolkit, Sound
eXchange (SoX), and Transcriber. Praat (Boersma,
2001; http://www.fon.hum.uva.nl/praat) was devel-
oped by Paul Boersma and David Weenink at the
Institute of Phonetic Sciences, University of Amster-
dam. It is a general purpose speech tool with its own
scripting language that supports the editing, segmen-
tation and labeling, and prosodic manipulation of
audio files. There is a large community of Praat users
who have written and shared numerous scripts for a
wide variety of manipulation tasks. Praat is well-
supported, and there are a number of useful mailing
lists and user groups that also provide help. Wave-
surfer is a similar open source tool for speech manip-
ulation derived from software developed by ESPS
XWaves and is now available from http://source-
forge.net/projects/wavesurfer/. It is primarily used
in speech research labs. More special purpose tools
include Transcriber (http://trans.sourceforge.net/en/
presentation.php), which is a tool used for speech
annotation. It provides a graphical user interface for
segmenting longer speech recordings, transcribing
them, and labeling things such as speech turns, topic
changes, and acoustic conditions. It was designed
for the annotation of broadcast news recordings but
is also of more general-purpose usefulness. It was
developed with the scripting language Tcl/Tk and
C extensions. It relies on the Snack sound exten-
sion, which allows support for most common audio
formats, and on the tcLex lexer generator. The OGI
Speech Tools (http://www.speech.cs.cmu.edu/comp.
speech/Section1/Labs/ogi.html) were developed at
488
the Center for Spoken Language Understanding at
the Oregon Graduate Institute of Science and Tech-
nology in Portland—now the Department of Science
and Engineering at the Oregon Health and Science
University. The toolkit, which runs on Unix and is
written in C, includes tools for displaying, search-
ing, analyzing, and manipulating speech data as
well as implementation of various speech process-
ing algorithms. SoX (http://sox.sourceforge.net) is a
command line library that supports the conversion
of digital audio formats to other format. It works on
Windows, Mac, and Linux and allows users to apply
different effects to audio files as well as to play and
record them. Other tools available as freeware or
by license include Matlab (http://www.mathworks.
com); WinPitch (http://www.winpitch.com), which
runs under Windows; PhonEdit (http://www.lpl-aix.
fr/∼lpldev/phonedit); and Adobe Audition (formerly
Cool Edit Pro; http://www.adobe.com/special/products/
audition/syntrillium.html).
A useful tool for the extraction of acoustic features
from speech is openSMILE. openSMILE was
developed and is maintained by Florian Eyben and
others at and associated with the Technische Uni-
versität München (Eyben, Weninger, Gross, &
Schuller, 2013). This tool operates by first extracting
a set of low-level descriptors (LLDs) and then apply-
ing functionals to these. LLDs of the speech signal
are frame-level properties such as intensity, pitch,
and Mel-frequency cepstral coefficients. These are
extracted once per short frame, where the frame length
is typically 10 ms. openSMILE functionals are func-
tions such as mean, standard deviation, and location of
maximum that take a series of LLDs as inputs and out-
put a single feature. All functionals can be applied to
all LLDs. This makes it very easy to extract thousands
of features using openSMILE. One recent configura-
tion file, provided as part of the Interspeech 2014
Paralinguistics Challenge, extracted 6,373 features.
OpenSMILE can extract features based on a full input
file, fixed time units, or for user-specified regions.
AuToBI is a tool that is more specifically for pro-
sodic analysis and extraction of prosodic ­features.
Based on Rosenberg (2009), AuToBI extracts fea-
tures relevant for Automatic ToBI labeling. The task
of complete ToBI labeling is broken down into six
component tasks: (a) detecting pitch accents,

(b) detecting intonational phrase boundaries,
(c) detecting intermediate phrase boundaries,
(d) classifying pitch accent types, (e) classifying
phrase accent/boundary tone pairs that end intona-
tional phrase boundaries, and (f) classifying phrase
accents that end intermediate phrase boundaries.
In this way, AuToBI is a two-stage labeling system
that first detects prosodic events and subsequently
classifies them. If available, AuToBI generates pre-
dictions based on input word segmentation. If this
is not available, AuToBI includes an acoustic-based
syllable segmentation algorithm and will generate
ToBI labels based on this segmentation. AuToBI
represents state-of-the-art performance on each
ToBI labeling task (Rosenberg, 2010, 2012). Impor-
tantly, AuToBI has been evaluated across corpus,
training on one corpus and testing on another.
Performance suffers modestly, by 2%–5% in this
context, demonstrating the system’s robustness
across various data. More recently, AuToBI models
have been trained on languages other than English
(Moniz et al., 2014; Rosenberg, Cooper, Levitan, &
APPENDIX 18.1
Schematic Versions of ToBI Contours for Standard American English
Measuring the Voice
Hirschberg, 2012; Soto, Cooper, Rosenberg, &
Hirschberg, 2013).
CONCLUSION
In this chapter, we have described a number of dif-
ferent approaches to analyzing the voice, at both
lower and higher prosodic levels. All of these can be
useful when associating voice features with a variety
of different discourse, emotional, and mental states
as well as social relationships with listeners. Both
low-level and higher level features have been shown
to correlate with changes in speakers’ physical and
mental states, and both types of features can be
extracted automatically for use in studies including
age, personality, disability, disease, emotion, like-
ability, deception, or trust as topics of interest. The
speech field is currently exploring numerous other
correlates of vocal variation aspects, including the
role of vocal cues in signaling degrees of autistic
dysfunction, degree of cognitive and physical load,
empathy, and many other social behaviors.
489
Rosenberg, Enos, and Hirschberg
APPENDIX 18.1(continued)
Schematic Versions of ToBI Contours for Standard American English
References
Ananthakrishnan, S., Ghosh, P., & Narayanan, S.
(2008). Automatic classification of question
turns in spontaneous speech using lexical and
prosodic evidence. In Proceedings of ICASSP 2008
(pp. 5005–5008). http://dx.doi.org/10.1109/
ICASSP.2008.4518782
Ananthakrishnan, S., & Narayanan, S. (2005). An automatic
prosody recognizer using a coupled multi-stream
acoustic model and a syntactic-prosodic language
model. In Proceedings of ICASSP 2005 (pp. 269–272).
http://dx.doi.org/10.1109/ICASSP.2005.1415102
Ananthakrishnan, S., & Narayanan, S. (2006). Combining
acoustic, lexical, and syntactic evidence for automatic
unsupervised prosody labeling. In Proceedings of
Interspeech 2006 (pp. 297–300). Retrieved from http://
citeseerx.ist.psu.edu/viewdoc/download?
doi=10.1.1.109.7084&rep=rep1&type=pdf
Ananthakrishnan, S., & Narayanan, S. (2008). Fine-
grained pitch accent and boundary tone labeling
with parametric f0 features. In Proceedings of ICASSP
2008 (pp. 4545–4548). http://dx.doi.org/10.1109/
ICASSP.2008.4518667
Arvaniti, A., & Baltazani, M. (2000). Greek ToBI: A
system for the annotation of Greek speech corpora.
490
In Proceedings of LREC 2000 (pp. 555–562). Retrieved
from http://www.lrec-conf.org/proceedings/lrec2000/
pdf/7.pdf
Avanzi, M., Simon, A. C., Goldman, J.-P., & Auchlin, A.
(2010, May). C-prom: An annotated corpus for
French prominence studies. Paper presented at the
Fifth International Conference on Speech Prosody,
Chicago, IL.
Bachenko, J., & Fitzpatrick, E. (1990). A computational
grammar of discourse-neutral prosodic phrasing in
English. Computational Linguistics, 16, 155–170.
Bangalore, S., & Joshi, A. K. (1999). Supertagging:
An approach to almost parsing. Computational
Linguistics, 2, 237–265.
Barnes, J., Veilleux, N., Brugos, A., & Shattuck-Hufnagel, S.
(2010). The effect of global f0 contour shape on the
perception of tonal timing contrasts in American
English intonation. In Proceedings of Speech Prosody
2010 (pp. 1–4). Retrieved from http://www.
speechprosody2010.illinois.edu/papers/100445.pdf
Bartels, C. (1999). The intonation of English statements and
question: A compositional interpretation. New York, NY:
Garland.
Batliner, A., Kompe, R., Kiessling, A., Nöth, E.,
Niemann, H., & Kilian, U. (1995). The prosodic

marking of phrase boundaries: Expectations and
results. In A. J. Rubio-Ayuso & J. M. Lopez-Soler
(Eds.), Speech recognition and coding: New advances
and trends (pp. 89–92). Berlin, Germany: Springer.
Beach, C. M. (1991). The interpretation of prosodic
patterns at points of syntactic structure ambiguity:
Evidence for cue trading relations. Journal of
Memory and Language, 30, 644–663. http://dx.doi.
org/10.1016/0749-596X(91)90030-N
Beckman, M. E., & Elam, G. A. (1997, March). Guidelines
for ToBI labelling. Retrieved from http://www.ling.
ohio-state.edu/research/phonetics/E_ToBI
Beckman, M. E., & Hirschberg, J. (1994). The ToBI
annotation conventions. Retrieved from http://www.
cs.columbia.edu/∼agus/tobi/tobi_convent.pdf
Belkin, M., Niyogi, P., & Sindhwani, V. (2006). Manifold
regularization: A geometric framework for learning
from labeled and unlabeled examples. Journal of
Machine Learning Research, 7, 2399–2434.
Benus, S. (2009). Are we “in sync”: Turn-taking in
collaborative dialogues. In Proceedings of Interspeech
2009 (pp. 2167–2170). Retrieved from http://www1.
cs.columbia.edu/∼sbenus/Research/Benus_Are_we_
in_sync_IS09_proc.pdf
Benus, S., Gravano, A., & Hirschberg, J. (2007). Prosody,
emotions, and . . . “whatever.” In Proceedings of
Interspeech 2007 (pp. 2629–2632). Retrieved from
http://www1.cs.columbia.edu/nlp/papers/2007/
benus_al_07a.pdf
Boersma, P. (2001). Praat, a system for doing phonetics
by computer. Glot International, 5, 341–345.
Bolinger, D. (1961). Contrastive accent and contrastive
stress. Language, 37, 83–96. http://dx.doi.
org/10.2307/411252
Bolinger, D. (1972). Accent is predictable (if you’re a
mind-reader). Language, 48, 633–644. http://dx.doi.
org/10.2307/412039
Bolinger, D. (1985). Intonation and its parts: Melody in spoken
English. Stanford, CA: Stanford University Press.
Bolinger, D. (1989). Intonation and its uses: Melody in
grammar and discourse. Stanford, CA: Stanford
University Press.
Braunschweiler, N. (2006, May). The Prosodizer—
Automatic prosodic annotations of speech synthesis
databases. Paper presented at Speech Prosody 2006,
Dresden, Germany.
Brenier, J., Cer, D., & Jurafsky, D. (2005). The detection
of emphatic words using acoustic and lexical features.
In Proceedings of Interspeech 2005 (pp. 3297–3300).
Retrieved from http://web.stanford.edu/∼jurafsky/
p1034.pdf
Brown, G. (1983). Prosodic structure and the given/new
distinction. In A. Cutler & D. Ladd (Eds.), Prosody:
Measuring the Voice
Models and measurements (pp. 67–77). Berlin,
Germany: Springer-Verlag.
Bruce, G., Granstrom, B., Gustafson, K., House, D., &
Touati, P. (1994). Modelling Swedish prosody in
a dialogue framework. In Proceedings of ICSLP 1994
(pp. 1099–1102). Yokohama, Japan: International
Speech Communication Association.
Buhmann, J., Caspers, J., van Heuven, V. J., Hoekstra, H.,
Martens, J. P., & Swerts, M. (2002). Annotation
of prominent words, prosodic boundaries and
segmental lengthening by nonexpert transcribers
in the spoken Dutch corpus. In Proceedings of
LREC 2002 (pp. 779–785). Paris, France: European
Language Resources Association.
Campbell, N. (1995). Loudness, spectral tilt and
perceived prominence in dialogues. In Proceedings
of the 13th International Congress of Phonetic Sciences
(pp. 676–679). Stockholm, Sweden: International
Congress of Phonetic Sciences.
Carlson, R., Hirschberg, J., & Swerts, M. (2005). Cues to
upcoming Swedish prosodic boundaries: Subjective
judgment studies and acoustic correlates. Speech
Communication, 46, 326–333. http://dx.doi.org/
10.1016/j.specom.2005.02.013
Caspers, J. (2003). Local speech melody as a limiting
factor in the turn-taking system in Dutch. Journal of
Phonetics, 31, 251–276. http://dx.doi.org/10.1016/
S0095-4470(03)00007-X
Chen, F. R., & Withgott, M. (1992). The use of emphasis
to automatically summarize a spoken discourse.
In Proceedings of ICASSP 1992 (pp. 229–232). http://
dx.doi.org/10.1109/ICASSP.1992.225930
Chen, K., Hasegawa-Johnson, M., & Cohen, A. (2004).
An automatic prosody labeling system using ANN-
based syntactic-prosodic model and GMM-based
acoustic-prosodic model. In Proceedings of ICASSP
2004 (pp. 509–512). http://dx.doi.org/10.1109/
ICASSP.2004.1326034
Chomsky, N., & Halle, M. (1968). The sound pattern of
English. New York, NY: Harper & Row.
Christensen, H., Gotoh, Y., & Renals, S. (2001).
Punctuation annotation using statistical prosody
models. In Proceedings of the ISCA Workshop on
Prosody in Speech Recognition and Understanding
(pp. 35–40). Retrieved from http://www.cstr.ed.ac.
uk/downloads/publications/2001/pros01-punc.pdf
Clark, J., & Yallop, C. (1990). Introduction to phonology
and phonetics. New York, NY: Blackwell.
Cohen, A. (2004). A survey of machine learning methods
for predicting prosody in radio speech (Unpublished
master’s thesis). University of Illinois at Urbana-
Champaign.
Cohen, W. W. (1995). Fast effective rule induction. In
Proceedings of the 12th International Conference on
491
Rosenberg, Enos, and Hirschberg
Machine Learning (pp. 115–123). Tahoe City, CA:
Morgan Kaufmann.
Conkie, A., Riccardi, G., & Rose, R. (1999). Prosody
recognition from speech utterances using acoustic and
linguistic based models of prosodic events.
In Proceedings of Eurospeech 1999 (pp. 523–526).
Retrieved from http://www2.research.att.com/∼ttsweb/
tts/papers/1999_Eurospeech/prosodyrec.pdf
Dahan, D., Tanenhaus, M., & Chambers, C. (2002).
Accent and reference resolution in spoken-language
comprehension. Journal of Memory and Language,
47, 292–314. http://dx.doi.org/10.1016/S0749-
596X(02)00001-3
Demenko, G., Grocholewski, S., Wagner, A., &
Szymanski, M. (2006). Prosody annotation for
corpus based speech synthesis. In Proceedings of the
Australian International Conference on Speech Science
and Technology 2006 (pp. 460–465). Retrieved from
http://www.assta.org/sst/2006/sst2006-133.pdf
Díaz, F. C., van Santen, J., & Banga, E. R. (2009).
Integrating phrasing and intonation modelling
using syntactic and morphosyntactic information.
Speech Communication, 51, 452–465. http://dx.doi.
org/10.1016/j.specom.2009.01.007
Dilley, L., & Brown, M. (2005). The RaP (Rhythm and
Pitch) labeling system. Retrieved from http://web.
mit.edu/tedlab/tedlab_website/RaP%20System/RaP_
Labeling_Guide_v1.0.pdf
Duncan, S. (1972). Some signals and rules for taking
speaking turns in conversations. Journal of Personality
and Social Psychology, 23, 283–292. http://dx.doi.
org/10.1037/h0033031
Enos, F. (2009). Detecting deception in speech
(Unpublished doctoral dissertation). Columbia
University, New York, NY.
Evanini, K., & Zechner, K. (2011). Using crowdsourcing to
provide prosodic annotations for non-native speech.
In Proceedings of Interspeech 2011 (pp. 3069–3072).
Retrieved from http://evanini.com/papers/evanini_
INTERSPEECH11.pdf
Eyben, F., Weninger, F., Gross, F., & Schuller, B. (2013).
Recent developments in openSMILE, the Munich
open-source multimedia feature extractor. In
Proceedings of the 21st ACM International Conference on
Multimedia (pp. 835–838). New York, NY: Association
for Computing Machinery.
Fach, M., & Wokurek, W. (1995). Pitch accent
classification of fundamental frequency contours by
hidden Markov models. In Proceedings of Eurospeech
1995 (pp. 2047–2050). Madrid, Spain: Universidad
Politecnica.
Fant, G., Kruckenberg, A., & Liljencrants, J. (2000).
Acoustic-phonetic analysis of prominence in
Swedish. In A. Botinis (Ed.), Intonation, analysis,
492
modelling and technology (pp. 55–86). Dordrecht, the
Netherlands: Kluwer.
Fischer, I., & Poland, J. (2004, June 21). New methods for
spectral clustering (Technical Report No. ISDIA-12-04).
Retrieved from http://repository.supsi.ch/5528/1/
IDSIA-12-04.pdf
Fujisaki, H., & Hirose, K. (1982). Modelling the dynamic
characteristics of voice fundamental frequency with
applications to analysis and synthesis of intonation.
In Proceedings of the 13th International Congress of
Linguists (pp. 57–70). Tokyo, Japan: International
Permanent Committee of Linguists.
Godfrey, J., Holliman, E., & McDaniel, J. (1992).
Switchboard: Telephone speech corpus for research
and development. In Proceedings of ICASSP 1992
(Vol. 1, pp. 517–520). http://dx.doi.org/10.1109/
ICASSP.1992.225858
Grabe, E., Nolan, F., & Farrar, K. (1998,
November–December). IViE: A comparative
transcription system for intonational variation in
English. Paper presented at ICSLP ‘98, Sydney,
Australia.
Gravano, A. (2009). Turn-taking and affirmative cue
words in task-oriented dialog (Unpublished doctoral
dissertation). Columbia University, New York, NY.
Gravano, A., Benus, S., Hirschberg, J., German, E. S., &
Ward, G. (2008). The effect of contour type and
epistemic modality on the assessment of speaker
certainty. In Proceedings of Speech Prosody 2008 (pp.
401–404). Retrieved from http://www.cs.columbia.
edu/nlp/papers/2008/gravano_al_08.pdf
Gravano, A., & Hirschberg, J. (2006). Effect of genre,
speaker and word class on the realization of given
and new information. In Proceedings of Interspeech
2006 (pp. 557–560). Retrieved from http://www1.
cs.columbia.edu/nlp/papers/2006/gravano_
hirschberg_06.pdf
Gregory, M., & Altun, Y. (2004). Using conditional
random fields to predict pitch accents in
conversational speech. In Proceedings of ACL 2004
(pp. 677–683). Retrieved from http://www.aclweb.
org/anthology/P04-1086
Grice, M., & Savino, M. (1997). Can pitch accent type
convey information status in yes–no questions?
In Proceedings of the Workshop on Concept to Speech
Generation Systems (pp. 29–38). Retrieved from
http://aclweb.org/anthology/W97-1205
Grosz, B. J., & Sidner, C. L. (1986). Attention, intentions,
and the structure of discourse. Computational
Linguistics, 12, 175–204.
Gundel, J. K. (1999). On different kinds of focus.
In P. Bosch & R. van der Sandt (Eds.), Focus:
Linguistic, cognitive, and computational perspectives
(pp. 293–305). Cambridge, England: Cambridge
University Press.

Gussenhoven, C. (1983). A semantic analysis of the
nuclear tones of English. Dordrecht, the Netherlands:
Kluwer.
Gussenhoven, C. (2005). Transcription of Dutch
intonation. In S.-A. Jun (Ed.), Prosodic typology: The
phonology of intonation and phrasing
(pp. 118–145). http://dx.doi.org/10.1093/acprof:oso/
9780199249633.003.0005
Halliday, A. M. (1967). Intonation and grammar in British
English. The Hague, the Netherlands: Mouton.
Hasegawa-Johnson, M., Cole, J., Shih, C., Chen, K.,
Cohen, A., Chavarria, S., . . . Choi, J.-Y. (2004).
Speech recognition models of the interdependence
among syntax, prosody and segmental acoustics. In
Proceedings of the HLT–NAACL 2004 Workshop on
Spoken Language Understanding for Conversational
Systems and Higher Level Linguistic Information
for Speech Processing (pp. 56–63). Retrieved from
https://www.aclweb.org/anthology/W/W04/W04-
3010.pdf
Hedberg, N. (2003). The prosody of contrastive topic and
focus in spoken English. Retrieved from http://www.
sfu.ca/∼hedberg/Stuttgart_paper.pdf
Heeman, P. A. (1999). Modeling speech repairs and
intonational phrasing to improve speech recognition.
In Proceedings of the IEEE Workshop on Automatic
Speech Recognition and Understanding (pp. 2–273).
Keystone, CO: IEEE.
Heldner, M. (2001). Spectral emphasis as an additional
source of information in accent detection. In
Proceedings of the ISCA Workshop on Prosody in
Speech Recognition and Understanding (pp. 57–60).
Retrieved from http://www.speech.kth.se/prod/
publications/files/710.pdf
Heldner, M., & Megyesi, B. (2003). Exploring the
prosody-syntax interface in conversations. In
Proceedings of ICPhS 2003 (pp. 2501–2504).
Retrieved from http://www.speech.kth.se/∼heldner/
papers/PS021248.pdf
Heldner, M., Strangert, E., & Deschamps, T. (1999).
Focus detection using overall intensity and high
frequency emphasis. In Proceedings of ICPhS 1999
(Vol. 2, pp. 1491–1493). Retrieved from http://www.
speech.kth.se/prod/publications/files/1188.pdf
Hirschberg, J. (1993). Pitch accent in context:
Predicting intonational prominence from text.
Artificial Intelligence, 63, 305–340. http://dx.doi.
org/10.1016/0004-3702(93)90020-C
Hirschberg, J. (2002a). Communication and prosody:
Functional aspects of prosody. Speech
Communication, 36, 31–43. http://dx.doi.org/10.1016/
S0167-6393(01)00024-3
Hirschberg, J. (2002b). The pragmatics of intonational
meaning. In Proceedings of Speech Prosody 2002 (pp.
Measuring the Voice
65–68). Retrieved from http://www.cs.columbia.
edu/∼julia/papers/pragmatics_sp02_065.pdf
Hirschberg, J., Benus, S., Brenier, J. M., Enos, F.,
Friedman, S., Gilman, S., . . . Stolcke, A. (2005).
Distinguishing deceptive from non-deceptive speech.
In Proceedings of Interspeech 2005 (pp. 1833–1836).
Retrieved from http://www.cs.columbia.edu/∼julia/
files/hirschberg_al_05.pdf
Hirschberg, J., & Nakatani, C. (1996). A prosodic
analysis of discourse segments in direction-giving
monologues. In Proceedings of the 34th Annual Meeting
of the Association for Computational Linguistics (pp.
286–293). http://dx.doi.org/10.3115/981863.981901
Hirschberg, J., & Nakatani, C. (1998). Using machine
learning to identify intonational segments. In
Proceedings of AAAI 1998 (pp. 52–59). Madison,
WI: Association for the Advancement of Artificial
Intelligence.
Hirschberg, J., & Pierrehumbert, J. (1986). The
intonational structuring of discourse. In Proceedings
of the 24th Annual Meeting of the Association for
Computational Linguistics (pp. 136–144). http://
dx.doi.org/10.3115/981131.981152
Hirschberg, J., & Prieto, P. (1996). Training intonational
phrasing rules automatically for English and Spanish
text-to-speech. Speech Communication, 18, 281–290.
http://dx.doi.org/10.1016/0167-6393(96)00017-9
Hirschberg, J., & Rambow, O. (2001). Learning prosodic
features using a tree representation. In Proceedings
of Interspeech 2001 (pp. 1175–1178). Aalborg,
Denmark: International Speech Communication
Association.
Hirschberg, J., & Ward, G. (1992). The influence of pitch
range, duration, amplitude and spectral features on the
interpretation of the rise-fall-rise intonation contour in
English. Journal of Phonetics, 20, 241–251.
Hirschberg, J., & Ward, G. (1995). The interpretation
of the high-rise question contour in English.
Journal of Pragmatics, 24, 407–412. http://dx.doi.
org/10.1016/0378-2166(94)00056-K
Hirst, D., & Di Cristo, A. (1998). Intonation systems.
A survey of twenty languages. Cambridge, England:
Cambridge University Press.
Hirst, D., & Espesser, R. (1993). Automatic modelling
of fundamental frequency using a quadratic spline
function. Travaux de l’Institut de Phonétique d’Aix, 15,
71–85.
Hu, G. P., Chen, B. F., Fan, M., & Wang, R. (2003).
The contribution of mutual information in the
intonational phrase prediction in Chinese text. In
Proceedings of the 2003 International Conference
on Natural Language Processing and Knowledge
Engineering (pp. 407–412). http://dx.doi.
org/10.1109/NLPKE.2003.1275939
493
Rosenberg, Enos, and Hirschberg
Ishi, C. T. (2005). Perceptually-related f0 parameters
for automatic classification of phrase final tones.
IEICE Transactions on Information and Systems, E88-
D, 481–488. http://dx.doi.org/10.1093/ietisy/e88-
d.3.481
Iwano, K., & Hirose, K. (1998, November–December).
Representing prosodic words using statistical models of
moraic transition of fundamental frequency contours
of Japanese. Paper presented at ICSLP ‘98, Sydney,
Australia.
Johnson, K. (2003). Acoustic and auditory phonetics.
Malden, MA: Blackwell.
Jun, S.-A. (Ed.). (2005). Prosodic typology: The phonology of
intonation and phrasing. http://dx.doi.org/10.1093/acpr
of:oso/9780199249633.001.0001
Kim, Y.-J., & Oh, Y.-H. (1996). Prediction of prosodic
phrase boundaries considering variable speaking
rate. In Proceedings of ICSLP 1996 (Vol. 3,
pp. 1505–1508). http://dx.doi.org/10.1109/
ICSLP.1996.607902
Kochanski, G., Grabe, E., Coleman, J., & Rosner, B.
(2005). Loudness predicts prominence: Fundamental
frequency lends little. Journal of the Acoustical
Society of America, 118, 1038–1054. http://dx.doi.
org/10.1121/1.1923349
Koehler, K. (1991). A model of German intonation.
Arbeitsberichte des Instituts für Phonetik und digitale
Sprachverarbeitung der Universität Kiel (AIPUK), 25,
295–360.
Koehn, P., Abney, S., Hirschberg, J., & Collins, M.
(2000). Improving intonational phrasing with
syntactic information. In Proceedings of ICASSP 2000
(Vol. 3, pp. 1289–1290). http://dx.doi.org/10.1109/
ICASSP.2000.861813
Koiso, H., Horiuchi, Y., Tutiya, S., Ichikawa, A., &
Den, Y. (1998). An analysis of turn-taking and
backchannels based on prosodic and syntactic
features in Japanese map task dialogs. Language and
Speech, 41, 295–321.
Ladd, D. R. (1980). The structure of intonational meaning.
Bloomington: Indiana University Press.
Ladd, D. R. (1996). Intonational phonology. Cambridge,
England: Cambridge University Press.
Ladd, D. R., & Morton, R. (1997). The perception of
intonational emphasis: Continuous or categorical.
Journal of Phonetics, 25, 313–342. http://dx.doi.
org/10.1006/jpho.1997.0046
Lakoff, R. (1975). Language and women’s place. New York,
NY: Harper & Row.
Levitan, R., Gravano, A., & Hirschberg, J. (2011).
Entrainment in speech preceding backchannels. In
Proceedings of ACL 2011 (pp. 113–117). Retrieved
from http://www.aclweb.org/anthology/P11-2020
494
Levow, G.-A. (2005). Context in multi-lingual tone and
pitch accent recognition. In Proceedings of Interspeech
2005 (pp. 1809–1812). Retrieved from https://faculty.
washington.edu/levow/papers/IS05_context.pdf
Levow, G.-A. (2006). Unsupervised and semi-
supervised learning of tone and pitch accent. In
Proceedings of HLT–NAACL 2006 (pp. 224–231).
Retrieved from https://www.aclweb.org/
anthology/N/N06/N06-1029.pdf
Li, C., Liu, J., & Xia, S. (2007). English sentence stress
detection system based on HMM framework. Applied
Mathematics and Computation, 185, 759–768. http://
dx.doi.org/10.1016/j.amc.2006.06.081
Li, J.-F., Hu, G.-P., & Wang, R. (2004). Chinese
prosody phrase break prediction based on
maximum entropy model. In Proceedings of
Interspeech 2004 (pp. 729–732). Jeju Island, Korea:
International Speech Communication Association.
Li, J.-F., Hu, G.-P., Wang, R., & Dai, L.-R. (2005). Sliding
window smoothing for maximum entropy based
intonational phrase prediction in Chinese.
In Proceedings of ICASSP 2005 (pp. 285–288). http://
dx.doi.org/10.1109/ICASSP.2005.1415106
Liscombe, J. (2007). Prosody and speaker state:
Paralinguistics, pragmatics, and proficiency
(Unpublished doctoral dissertation). Columbia
University, New York, NY.
Liscombe, J., Hirschberg, J., & Venditti, J. J. (2005).
Detecting certainness in spoken tutorial dialogue.
In Proceedings of Interspeech 2005 (pp. 1837–1840).
Retrieved from http://www.cs.columbia.edu/∼julia/
files/liscombe_al_05b.pdf
Liscombe, J., Venditti, J., & Hirschberg, J. (2003).
Classifying subject ratings of emotional speech
using acoustic features. In Proceedings of Eurospeech
2003 (pp. 725–728). Retrieved from http://www.
cs.columbia.edu/∼julia/files/euro03-emotion.pdf
Mahrt, T., Huang, J.-T., Mo, Y., Fleck, M., Hasegawa-
Johnson, M., & Cole, J. (2011). Optimal models
of prosodic prominence using the Bayesian
information criterion. In Proceedings of Interspeech
2011 (pp. 2037–2040). Retrieved from http://
prosody.beckman.illinois.edu/pubs/59_Mahrt_et_al_
Interspeech2011_abstract.pdf
Maragoudakis, M., Zervas, P., Fakotakis, N., &
Kokkinakis, G. (2003). A data-driven framework for
intonational phrase break prediction. Text, Speech
and Dialogue, 2807, 189–197.
Marsi, E., Reynaert, M., van den Bosch, A., Daelmans,
W., & Hoste, V. (2003). Learning to predict pitch
accents and prosodic boundaries in Dutch. In
Proceedings of ACL 2003 (pp. 489–496). Retrieved
from http://www.aclweb.org/anthology/P03-1062
Maskey, S. R., Rosenberg, A., & Hirschberg, J. (2008).
Intonational phrases for speech summarization. In

Proceedings of Interspeech 2008 (pp. 2430–2433).
Retrieved from http://www.cs.columbia.edu/nlp/
papers/2008/maskey_al_08.pdf
McKeown, K., & Pan, S. (2000). Prosody modeling
in concept-to-speech generation: Methodological
issues. Philosophical Transactions of the Royal Society,
358, 1419–1431.
Mermelstein, P. (1975). Automatic segmentation of speech
into syllabic units. Journal of the Acoustical Society of
America, 58, 880–883.
Mishra, T. (2008). Decomposition of fundamental frequency
contours in the general superpositional intonation model
(Unpublished doctoral dissertation). Oregon Health
and Science University, Portland, OR.
Mishra, T., Kumar, V., Sridhar, R., & Conkie, A. (2012).
Word prominence detection using robust yet simple
prosodic features. In Proceedings of Interspeech
2012 (pp. 1864–1867). Retrieved from http://www.
research.att.com/techdocs/TD_100837.pdf
Möbius, B. (1993). Perceptual evaluation of rule-
generated intonation contours for German
interrogatives. In Proceedings of the ESCA Workshop
on Prosody (pp. 216–219). Retrieved from http://
www.isca-speech.org/archive_open/archive_papers/
prosody_93/pro3_216.pdf
Moniz, H., Mata da Silva, A. I., Hirschberg, J., Batista, F.,
Rosenberg, A., & Trancoso, I. (2014, May). Extending
AuToBI to prominence detection in European Portuguese.
Paper presented at Speech Prosody 2014, Dublin,
Ireland.
Murray, K. (2001). A study of automatic pitch tracker
doubling/halving “errors.” Retrieved from http://www.
aclweb.org/anthology/W01-1613
Nakai, M., Singer, H., Sagisaka, Y., & Shimodaira, H.
(1995). Automatic prosodic segmentation by f0
clustering using superpositional modeling. In
Proceedings of ICASSP 1995 (Vol. 1, pp. 624–627).
http://dx.doi.org/10.1109/ICASSP.1995.479675
Nakajima, S., & Allen, J. F. (1993). A study on
prosody and discourse structure in cooperative
dialogues. Phonetica, 50, 197–210. http://dx.doi.
org/10.1159/000261940
Nakatani, C., Hirschberg, J., & Grosz, B. (1995).
Discourse structure in spoken language: Studies on
speech corpora. In Proceedings of the AAAI Spring
Symposium on Empirical Methods in Discourse
Interpretation and Generation (pp. 106–112).
Retrieved from http://www.aaai.org/Papers/Symposia/
Spring/1995/SS-95-06/SS95-06-018.pdf
Nenkova, A., Brenier, J., Kothari, A., Calhoun, S.,
Whitton, L., Beaver, D., & Jurafsky, D. (2007).
To memorize or to predict: Prominence labeling in
conversational speech. In Proceedings of NAACL-HLT
2007 (pp. 9–16). Retrieved from http://www.cis.
upenn.edu/∼nenkova/papers/hlt07.pdf
Measuring the Voice
O’Connor, J., & Arnold, G. (1973). Intonation of colloquial
English (2nd ed.). London, England: Longman.
Oliver, D. (2005). Deriving pitch accent classes using
automatic f0 stylisation and unsupervised clustering
techniques. In Proceedings of the Second Baltic
Conference on Human Language Technologies
(pp. 161–166). Tallinn, Estonia: Institute of
Cybernetics, Tallinn University of Technology.
Ostendorf, M., Price, P. J., & Shattuck-Hufnagel, S.
(1995). The Boston University Radio News Corpus.
In Proceedings of the Linguistic Data Consortium (pp.
1–19). Retrieved from http://www.inf.ed.ac.uk/
teaching/courses/icl/pdfs/bur_crps.pdf
Ostendorf, M., & Ross, K. (1997). Multi-level recognition
of intonation labels. In Y. Sagisaka, N. Campbell, &
N. Higuchi (Eds.), Computing prosody: Computational
models for processing spontaneous speech
(pp. 291–308). Berlin, Germany: Springer-Verlag.
Ostendorf, M., & Veilleux, N. (1994). A hierarchical
stochastic model for automatic prediction of
prosodic boundary location. Computational
Linguistics, 20, 27–54.
Ostendorf, M., Wightman, C., & Veilleux, N. (1993).
Parse scoring with prosodic information: An analysis/
synthesis approach. Computer Speech and Language,
7, 193–210.
Palmer, H. (1922). English intonation with systematic
exercises. Cambridge, England: Cambridge University
Press.
Pfau, T., & Ruske, G. (1998). Estimating the speaking
rate by vowel detection. In Proceedings of ICASSP
1998 (Vol. 2, pp. 945–948). http://dx.doi.
org/10.1109/ICASSP.1998.675422
Pierrehumbert, J. (1979). The perception of fundamental
frequency declination. Journal of the Acoustical
Society of America, 66, 363–369.
Pierrehumbert, J. (1980). The phonology and phonetics
of English intonation (Unpublished doctoral
dissertation). Massachusetts Institute of Technology,
Cambridge, MA.
Pierrehumbert, J., & Hirschberg, J. (1990). The meaning
of intonational contours in the interpretation of
discourse. In P. Cohen, J. Morgan, & M. Pollack
(Eds.), Intentions in communication (pp. 271–311).
Cambridge, MA: MIT Press.
Pierrehumbert, J., & Steele, S. (1987, August). How many
rise-fall-rise countours? Paper presented at the 11th
International Congress of Phonetic Sciences, Taillin,
Estonia.
Pitrelli, J., Beckman, M., & Hirschberg, J. (1994).
Evaluation of prosodic transcription labeling
reliability in the ToBI framework. In Proceedings
of ICSLP 1994 (pp. 123–126). Yokohama, Japan:
International Speech Communication Association.
495
Rosenberg, Enos, and Hirschberg
Prevost, S., & Steedman, M. (1993). Generating
contextually appropriate intonation. In Proceedings of
EACL 1993 (pp. 332–340). Retrieved from
http://www.aclweb.org/anthology/E93-1039
Price, P., Ostendorf, M., & Wightman, C. (1989). Prosody
and parsing. In Proceedings of the Workshop on Speech
and Natural Language (pp. 5–11). Stroudsburg, PA:
Association for Computational Linguistics.
Prince, E. F. (1981). Toward a taxonomy of give-new
information. In P. Cole (Ed.), Radical pragmatics (pp.
223–255). New York, NY: Academic Press.
Rakov, R., & Rosenberg, A. (2013, August). “Sure, I did the
right thing”: A system for sarcasm detection in speech.
Paper presented at Interspeech 2013, Lyon, France.
Read, I., & Cox, S. (2007a). Automatic pitch accent
prediction for text-to-speech synthesis. In Proceedings
of Interspeech 2007 (pp. 482–485). Retrieved from
http://www.uea.ac.uk/polopoly_fs/1.77246!/read-cox-
interspeech-07.pdf
Read, I., & Cox, S. (2007b). Stochastic and syntactic
techniques for predicting phrase breaks. Computer
Speech and Language, 21, 519–542.
Ren, Y., Kim, S.-S., Hasegawa-Johnson, M., & Cole, J.
(2004). Speaker-independent automatic detection of
pitch accent. In Proceedings of Speech Prosody 2004 (pp.
521–524). Retrieved from http://prosody.beckman.
illinois.edu/pubs/05_Ren_etal_2004_SP04.pdf
Rosenberg, A. (2009). Automatic detection and classification
of prosodic events (Unpublished doctoral dissertation).
Columbia University, New York, NY.
Rosenberg, A. (2010, September). AuToBI: A tool for
automatic ToBI annotation. Paper presented at
Interspeech 2010, Makuhari, Japan.
Rosenberg, A. (2012). Modeling intensity contours and the
interaction between pitch and intensity to improve
automatic prosodic event detection and classification.
In Proceedings of the 2012 IEEE Workshop on Spoken
Language Technology (pp. 376–381). Miami, FL: IEEE.
Rosenberg, A., Cooper, E. L., Levitan, R., & Hirschberg,
J. B. (2012). Cross-language prominence detection.
In Proceedings of Speech Prosody 2012 (pp. 278–281).
Retrieved from http://academiccommons.columbia.
edu/item/ac:163728
Rosenberg, A., Sharifi, & Hirschberg, J. (2007). Varying
input segmentation for story boundary detection in
English, Arabic and Mandarin broadcast news. In
Proceedings of Interspeech 2007 (pp. 2589–2592).
Retrieved from http://www.cs.cmu.edu/∼mehrbod/
SSeg07.pdf
Ross, K., & Ostendorf, M. (1996). Prediction of abstract
prosodic labels for speech synthesis. Computer Speech
and Language, 10, 155–185.
Rouas, J. L. (2007). Automatic prosodic variations
modeling for language and dialect discrimination.
496
IEEE Transactions on Audio, Speech, and Language
Processing, 15, 1904–1911.
Salamin, H., Polychroniou, A., & Vinciarelli, A. (2013).
Automatic detection of laughter and fillers in
spontaneous mobile phone conversations. In
Proceedings of the 2013 IEEE International Conference
on Systems, Man, and Cybernetics (pp. 4282–4287).
Miami, FL: IEEE.
Schmid, H., & Atterer, M. (2004). New statistical
methods for phrase break prediction. In Proceedings
of COLING 2004 (pp. 659–665). Retrieved from
http://www.aclweb.org/anthology/C04-1095
Selting, M. (1996). On the interplay of syntax and
prosody in the constitution of turn constructional
units and turns in conversation. Pragmatics, 6,
357–388.
Shriberg, E., Bates, R., Stolcke, A., Taylor, P., Jurafsky,
D., Ries, K., . . . Ess-Dykema, C. V. (1998). Can
prosody aid the automatic classification of dialog acts
in conversational speech? Language and Speech, 41,
443–492.
Shriberg, E., Stolcke, A., Hakkani-Tur, D., & Tur, G. (2000).
Prosody based automatic segmentation of speech
into sentences and topics. Speech Communication, 32,
127–154.
Silipo, R., & Greenberg, S. (2000, May). Prosodic stress
revisited: Reassessing the role of fundamental frequency.
Paper presented at the NIST Speech Transcription
Workshop, College Park, MD. Retrieved from http://
www1.icsi.berkeley.edu/∼steveng/2000/PDF/Prosody.
pdf
Silverman, K., Beckman, M., Pitrelli, J., Ostendorf, M.,
Wightman, C., Price, P., & Pierrehumbert, J. (1992).
ToBI: A standard for labeling English prosody. In
Proceedings of ICSLP 1992 (Vol. 2, pp. 12–16). Banff,
Alberta, Canada: University of Alberta.
Silverman, K., Kalyanswamy, A., Silverman, J., Basson,
S., & Yashchin, D. (1993). Synthesiser intelligibility
in the context of a name-and-address information
service. In Proceedings of Eurospeech 1993 (pp.
2169–2172). Berlin, Germany: International Speech
Communication Association.
Sluijter, A. M., & van Heuven, V. J. (1996). Spectral balance
as an acoustic correlate of linguistic stress. Journal of
the Acoustical Society of America, 100, 2471–2485.
Sluijter, A. M., van Heuven, V. J., & Pacilly, J. J. (1997).
Spectral balance as a cue in the perception of
linguistic stress. Journal of the Acoustical Society of
America, 101, 503–513.
Soto, V., Cooper, E., Rosenberg, A., & Hirschberg, J.
(2013). Cross-language phrase boundary detection.
In Proceedings of ICASSP 2013 (pp. 8460–8464).
Retrieved from http://www.cs.columbia.edu/speech/
PaperFiles/2013/soto-Columbia-icassp13-v2.pdf

Sridhar, V. R., Bangalore, S., & Narayanan, S. (2008).
Exploiting acoustic and syntactic features for
prosody labeling in a maximum entropy framework.
IEEE Transactions on Audio, Speech, and Language
Processing, 16, 797–811.
Sridhar, V. R., Narayanan, S., & Bangalore, S. (2006,
December). Acoustic-syntactic maximum
entropy model for automatic prosody labeling. Paper
presented at the IEEE/ACL 2006 Workshop on
Spoken Language Technology, Palm Beach, Aruba.
Sridhar, V. R., Narayanan, S., & Bangalore, S. (2007).
Exploiting acoustic and syntactic features for prosody
labeling in a maximum entropy framework. In
Proceedings of HLT–NAACL 2007 (pp. 1–8). Retrieved
from http://citeseerx.ist.psu.edu/viewdoc/download?
doi=10.1.1.129.5480&rep=rep1&type=pdf
Stolcke, A., Shriberg, E., Hakkani-Tur, D., & Tur, G.
(1999). Modeling the prosody of hidden events
for improved word recognition. In Proceedings of
Eurospeech 1999 (pp. 311–314). Retrieved from
http://www.sri.com/sites/default/files/publications/
modeling_the_prosody_of_hidden_events.pdf
Stolcke, A., Shriberg, E., Hakkani-Tur, D., Tur, G., &
Sonmez, K. (1999). Combining words and speech
prosody for automatic topic segmentation.
In Proceedings of the DARPA Broadcast News
Transcription and Understanding Workshop
(pp. 61–64). Retrieved from http://citeseerx.ist.
psu.edu/viewdoc/download?doi=10.1.1.53.379&
rep=rep1&type=pdf
Sun, X. (2002). Pitch accent prediction using ensemble
machine learning. In Proceedings of ICSLP 2002
(pp. 561–564). Retrieved from http://citeseerx.ist.
psu.edu/viewdoc/download?doi=10.1.1.12.4438&
rep=rep1&type=pdf
Syrdal, A., Hirschberg, J., McGory, J., & Beckman, M.
(2001). Automatic ToBI prediction and alignment
to speed manual labeling of prosody. Speech
Communication, 33, 135–151.
Syrdal, A., & McGory, J. (2000). Inter-transcriber
reliability of ToBI prosodic labeling. In Proceedings
of ICSLP 2000 (pp. 235–238). Retrieved from http://
www2.research.att.com/∼ttsweb/tts/papers/2000_
ICSLP/tobiRel.pdf
Tamburini, F. (2003a). Automatic prosodic
prominence detection in speech using acoustic
features: An unsupervised system. In Proceedings
of Interspeech 2003 (pp. 129–132). Geneva,
Switzerland: International Speech Communication
Association.
Tamburini, F. (2003b). Prosodic prominence detection
in speech. In Proceedings of the Seventh International
Symposium on Signal Processing and Its Applications
(Vol. 1, pp. 385–388). http://dx.doi.org/10.1109/
ISSPA.2003.1224721
Measuring the Voice
Taylor, P. (1998, November–December). The tilt
intonation model. Paper presented at ICSLP ‘98,
Sydney, Australia. Retrieved from http://www.
cstr.ed.ac.uk/downloads/publications/1998/
Taylor_1998_e.pdf
Taylor, P. (2000). Analysis and synthesis of intonation
using the tilt model. Journal of the Acoustical Society
of America, 107, 1697–1714.
Taylor, P., Caley, R., Black, A., & King, S. (1999).
Edinburgh Speech Tools Library: System
Documentation Edition 1.2 [Technical report].
Edinburgh, Scotland: The University of
Edinburgh.
Tepperman, J., & Narayanan, S. (2008). Tree grammars
as models of prosodic structure. In Proceedings
of Interspeech 2008 (pp. 2286–2289). Brisbane,
Australia: International Speech Communication
Association.
Tepperman, J., Traum, D., & Narayanan, S. (2006).
Yeah right: Sarcasm recognition for spoken
dialogue systems. In Proceedings of Interspeech 2006
(pp. 1838–1841). Retrieved from http://www1.
cs.columbia.edu/∼julia/papers/teppermanetal06.pdf
Terken, J., & Hirschberg, J. (1994). Deaccentuation of
words representing “given” information: Effects of
persistence of grammatical function and surface
position. Language and Speech, 37, 125–145.
’t Hart, J., Collier, R., & Cohen, A. (1990). A perceptual
study of intonation: An experimental-phonetic approach
to speech melody. Cambridge, England: Cambridge
University Press.
Toki, S., & Murata, M. (1987). Pronunciation and task
listening—Innovative workbooks in Japanese. Tokyo,
Japan: Aratake.
Tong, R., Ma, B., Zhu, D., Li, H., & Chang, E. S. (2006).
Integrating acoustic, prosodic and phonotactic
features for spoken language identification. In
Proceedings of ICASSP 2006 (205–208). http://dx.doi.
org/10.1109/ICASSP.2006.1659993
Veilleux, N. M. (1994). Computational models of the
prosody/syntax mapping for spoken language systems
(Unpublished doctoral dissertation). Boston
University, MA.
Veilleux, N. M., & Ostendorf, M. (1991). Probabilistic
parse scoring based on prosodic phrasing. In
Proceedings of the Workshop on Speech and Natural
Language (pp. 429–434). Retrieved from http://www.
aclweb.org/anthology/H92-1087
Villing, R., Timoney, J., Ward, T., & Costello, J.
(2004). Automatic blind syllable segmentation for
continuous speech. In Proceedings of ISSC 2004
(pp. 41–46). Retrieved from http://citeseerx.ist.psu.
edu/viewdoc/download?doi=10.1.1.81.2336&rep=
rep1&type=pdf
497
Rosenberg, Enos, and Hirschberg
Wang, M. Q., & Hirschberg, J. (1991). Predicting
intonational boundaries automatically from text:
The ATIS domain. In Proceedings of the Workshop on
Speech and Natural Language (pp. 378–383). http://
dx.doi.org/10.3115/112405.112739
Wang, M. Q., & Hirschberg, J. (1992). Automatic
classification of intonational phrase boundaries.
Computer Speech and Language, 6, 175–196.
Ward, G., & Hirschberg, J. (1985). Implicating
uncertainty: The pragmatics of fall-rise intonation.
Language, 61, 747–776.
Weber, F., Manganaro, L., Peskin, B., & Shriberg, E.
(2002). Using prosodic and lexical information
for speaker identification. In Proceedings of ICASSP
2002 (pp. 141–144). Retrieved from
ftp://ftp.icsi.berkeley.edu/pub/speech/papers/
icassp02-spid.pdf
Wightman, C., & Ostendorf, M. (1994). Automatic
labeling of prosodic patterns. IEEE Transactions on
Speech and Audio Processing, 2, 469–481.
Wightman, C., Ostendorf, M., & Shattuck-Hufnagel, S.
(1992). Segmental durations in the vicinity of prosodic
498
phrase boundaries. Journal of the Acoustical Society of
America, 91, 1707–1717.
Xydas, G., Spiliotopoulos, D., & Kouroupetroglou, G.
(2004). Modeling prosodic structures in linguistically
enriched environments. Text, Speech and Dialogue,
3206, 521–528.
Yoon, T.-J. (2006, May). Predicting prosodic phrasing using
linguistic features. Paper presented at Speech Prosody
2006, Dresden, Germany. Retrieved from http://
sprosig.isle.illinois.edu/sp2006/contents/papers/PS4-
14_0211.pdf
Zervas, P., Maragoudakis, M., Fakotakis, N., &
Kokkinakis, G. (2003). Bayesian induction of
intonational phrase breaks. In Proceedings of
Eurospeech 2003 (pp. 113–116). Geneva, Switzerland:
International Speech Communication Association.
Zhang, T., Hasegawa-Johnson, M., & Levinson, S. E.
(2004). Automatic detection of contrast for speech
understanding. In Proceedings of Interspeech 2004
(pp. 581–584). Retrieved from http://www.isle.
illinois.edu/sst/pubs/2004/zhang04interspeech_
contrast.pdf
 Chapter 19

Measuring Gesture
R. Breckinridge Church, Spencer D. Kelly, and Elizabeth Wakefield

Researchers in any science domain create
measurement techniques to answer three fundamen-
tal questions: (a) How do we describe a particular
phenomenon? That is, what are the appropriate
units of analysis for measurement, or how can a
particular phenomenon be operationalized?
(b) What are the functional causes (or purposes)
served by a particular phenomenon? (c) What
are the causal mechanisms or proximal explana-
tions underlying the occurrence of a phenomenon?
­Techniques used to measure gesture are no different
and attempt to answer these same three questions,
here couched in terms of gestural phenomena:
(a) How do we describe gesture? What are the
units of analysis we should use to measure gesture?
(b) What are the functional causes or the purpose
served by gesture—that is, how does gesture
function for human thinking and communication?
(c) What are the proximal mechanisms or explana-
tions underlying gesture’s occurrence?
In the first two sections of this chapter, we focus
on the descriptive question: How do we code and
measure gesture’s form and meaning? We draw
and synthesize from many laboratories (previous
and current) common typologies and schemes for
­coding gesture. In the third section of the chapter,
we discuss the function that gesture serves in
language and cognitive processes and describe
how these coding systems have been applied. In
the fourth section, we discuss brain mechanisms
underlying gesture processing. Given this focus
on gesture’s association with language and cog-
nitive processes, we orient our chapter around
cospeech gestures, or gestures that frequently occur
with speech. However, in an effort to contextual-
ize cospeech gestures, we briefly situate cospeech
gestures among broader categories of nonverbal
communication.
HOW DO WE DESCRIBE GESTURE IN THE
CONTEXT OF NONVERBAL BEHAVIOR?
Measurement begins by identifying and clarifying
categories of phenomena. What constitutes a
gesture? Are there different categories or types of
gesture? Using the broadest conceptualization,
gestures have been defined as any action with the
hands, arms, fingers, facial features, or even the
entire body that is used to communicate to another
individual (Crais, Watson, & Baranek, 2009).
Indeed, all of these types of movement were
emphasized in teaching rhetoric in early Rome
(Kendon, 2004) and are discussed in other chapters
in this handbook (see, e.g., Chapters 10–14).
These behaviors can be subdivided into the
­following three broad categories as conceptualized

We acknowledge all those individuals who participate in gesture research. These individuals have helped shape our thinking about gesture. We also
thank the psychology departments at Northeastern Illinois University, Colgate University (and the Center for Brain and Language at Colgate), and
The University of Chicago for their generous support, and we thank Indiana University for support via National Institutes of Health T32 Training
Grant HD07475. Finally, we thank those who pioneered the study of gesture, forging the paths that have now become major highways in the exami-
nation of gesture: David McNeill, Adam Kendon, William Stokoe, Paul Ekman, Wallace V. Friesen, and Susan Goldin-Meadow.

http://dx.doi.org/10.1037/14669-019
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
499
Copyright © 2016 by the American Psychological Association. All rights reserved.
Church, Kelly, and Wakefield
by the authors: (a) emotional gestures, which express
emotion on the part of the gesturer, as with smil-
ing, frowning, or looking confused (Coulson, 2004;
Darwin, 1872; Ekman & Friesen, 1969; Ekman,
Friesen, & Hager, 2002; Izard, 1994; Mehrabian &
Ferris, 1967); (b) regulating gestures, which are used
to regulate the flow of interactions or conversations
(Bavelas, Chovil, Coates, & Roe, 1995; Bavelas,
Chovil, Lawrie, & Wade, 1992; Ekman & Friesen,
1969); and (c) language-associated gestures, which
are gestures that are used as part of the speaking
process in discourse (e.g., Ekman & Friesen, 1969;
Kendon, 2004; McNeill, 1992). In this chapter, we
focus on a particular subcategory of these language-
associated gestures: cospeech gesture.1 Before we do
so, it is important to describe in a little more detail
how these cospeech gestures are classified within
this larger category of language-associated gestures.
Moving Our Hands to Express Language:
Language-Associated Gestures
Language-associated gestures are interesting
because of their unique relation with language
and cognitive processes (Goldin-Meadow, 2005;
Kendon, 2004; McNeill, 1992). These gestures are
seen as symbolic and convey information related
to the subject matter transmitted in discourse.
Different from emotional and regulating gestures,
language-associated gestures reflect aspects of
individuals’ mental representations during the
process of speaking. In some cases, these mental
representations are conveyed though arbitrary
gesture symbols (e.g., emblems); in other cases,
mental representations are conveyed through idio-
syncratic iconic imagery expressed through the
hands and body. The link between language-asso-
ciated gestures and mental imagery has provided
researchers with a behavioral tool for accessing
those mental processes underlying language and
cognitive activities. A more detailed descrip-
tion of gestures is provided in Chapter 12 of this
handbook.
Different types of language-associated gestures
have been described and classified in terms of their
We created this as an umbrella term for all those gestures that occur on Kendon’s continuum. Although not explicitly stated in prior literature on this
1
topic, these language-associated gestures are a unique category that is differentiated from the other categories of nonverbal communication (emo-
tional and regulating gestures).
500
relation to speech and language (Kendon, 2004;
McNeill, 1992). The framework used to classify how
gesture relates to spoken language is referred to as
Kendon’s continuum—informally outlined by Kendon
(2004) and formalized by McNeill (1992). Kendon’s
continuum classifies gesture along a continuum
reflecting its relation to speech. We describe this
classification system next.
All Language-Associated Gestures
Are Not the Same: Kendon’s Continuum
McNeill (1992) provided a framework based on
Kendon’s classification of gesture types that could
encompass the variety of ways that gesture func-
tions with speech. Essentially, language-associated
gestures can be classified according to two dimen-
sions: (a) the degree to which speech is present or
absent when language-associated gestures occur
and (b) the degree to which language-associated
gestures are conventionalized (McNeill, 1992,
2005). At one end of the continuum are sign lan-
guages, which like spoken languages, are conven-
tionally agreed upon. Therefore, sign languages
can and do operate independently of spoken lan-
guage. Moreover, sign languages possess conven-
tionalized elements that correspond to phonemic,
morphemic, and syntactic elements in spoken lan-
guages. That is, the sign usage is not random but
follows systematic rules. Therefore, for example,
certain hand forms, in order to be understood,
must be placed in rule-bound locations (with
respect to the face, body, or hands) such that a
change in location constitutes a new sign and a dif-
ferent meaning.
Next on the continuum are emblems (Ekman &
Friesen, 1969; also known as conventional ges-
tures; Goldin-Meadow, 1997). Emblems are
defined by a particular culture (i.e., many
emblems are not universal) and may or may
not be used independently of spoken language.
That is, the OK sign (thumb and index finger
form a circle with the other fingers straight and
fanned out; typically used in the United States)
can be associated with the idea of something

being okay without supplementary words; how-
ever, fuller interpretation of the emblem may
require some accompanying verbal context. For
example, if an individual says, “My interview
went [provides the OK sign],” the listener will
know that the OK sign refers to the outcome of
the interview.
Pantomime gestures are next on the continuum
because like emblems, they do not need to occur
with speech to be understood. In fact, in order to
be used, speech has to be absent—think of a good
game of charades. However, unlike sign languages,
pantomimes do not have conventionalized rules
governing their form and meaning. P ­ antomimes
are idiosyncratic, created uniquely by the indi-
vidual who pantomimes (hence the fun—and
frustration—of charades).
Finally, at the other end of the continuum
are cospeech gesticulations, which are the focus
of this chapter (i.e., nonconventional gestures;
Goldin-Meadow, 1997). These gesture movements
are almost always produced in the presence of
speech. These types of gestures may or may not
convey imagery and are linked in some way with
the ideas or concepts communicated through
accompanying speech (e.g., Kendon, 2004;
McNeill, 2005). Thus, the reference or meaning of
these gesticulations is best interpreted in the con-
text of the accompanying speech.2 Gesticulations
are idiosyncratic and driven by the speaker’s own
unique conceptualizations. Gesticulations are not
conventionalized by any formal rules dictated by a
given culture.
The absence of conventions dictating the form
and meaning of gesticulations has been a great
source of consternation for gesture researchers.
We focus on this type of gesture, for this chapter
on gesture measurement, precisely because the
development in measuring these gestures has been
important for understanding two basic human
behaviors pervasively accompanied by gesticu-
lations: language and thinking. We replace the
term gesticulations from here on in with the term
cospeech gestures.
There are exceptions to this that are discussed when we delve further into the application of cospeech gestures. In the study of gestures in educational
2
discourse, gestures can be coded independently of speech when the discourse context is sufficiently constrained. Kendon’s continuum was developed
before this educational research was known, making the characteristic of “speech is required” appropriate.
Measuring Gesture
Moving Hands While Talking: Cospeech
Gestures
The reason researchers are fascinated with cospeech
gestures is that they are a unique and promising way
to access mental processes. The idea that cospeech
gestures may provide a “window on the mind”
(Goldin-Meadow, 2005, p. 1) suggests that they are
a wonderful tool for the study of human thought but
a tool that requires tremendous scrutiny and con-
tinuous refinement.
We describe the taxonomy of gestures first artic-
ulated in Ekman and Friesen’s (1969) research and
further refined by McNeill (1992). This taxonomy
is generally agreed upon and is used (or slight varia-
tions of it) across all gesture research laboratories.
This taxonomy is critical for coding and measuring
gesture. These classifications allow gesture research-
ers to isolate gestures from other types of bodily
behavior. That is, gesture coders approach their
data prepared to determine whether certain behav-
iors map onto the following categories in terms of
both form and function. The first main distinction
made in this taxonomy differentiates imagistic from
nonimagistic cospeech gestures. Imagistic gestures
are those that represent visuo-spatial information
related to the accompanying speech (illustrators
in Ekman and Friesen’s, 1969, research; character-
izing gestures in Goldin-Meadow and colleagues’
research: Church & Goldin-Meadow, 1986; Goldin-
Meadow & Feldman, 1977). These imagistic ges-
tures are classified in two ways: (a) Iconic gestures
represent concrete attributes and actions through
visuo-spatial representation (referred to in Ekman
and Friesen’s, 1969, system as pictorial and kinesic).
For example, an individual might say, “Let’s go play
football,” while simultaneously enacting throwing a
football. (b) In contrast, metaphoric gestures repre-
sent abstract concepts using concrete, visuo-spatial
imagery. For instance, an individual could say,
“To save my business I did a Hail Mary Pass to
potential clients,” and represent this idea with the
same throwing football gesture. In this case, however,
one cannot literally throw a company into a winning
situation. Together, these examples illustrate how
501
Church, Kelly, and Wakefield

the same gesture may be used to represent both con- Step 4. Analyze cospeech gesture to address a
crete and abstract ideas, depending on the spoken particular research question.
message.
We now elaborate the process of coding cospeech
Nonimagistic gestures do not use imagery and
gesture following the four steps just described.
are of two types: (a) Beat gestures (referred to as
rhythmic gestures in Ekman and Friesen’s, 1969,
Step 1. Differentiate Cospeech Gestures
system) are rhythmic movements that occur with
From Other Behaviors
speech to highlight speech, structure speech,3 or
The first decision when coding cospeech gesture is
reflect a distilled version (a version without imag-
how to distinguish it from other body movements
ery) of a more complex gesture (McNeill, 2012).
occurring during speech activity. Answering this
For example, when listing items, one might move
question has practical as well as epistemological
the hand up and down for each item: “First I had
implications for gesture researchers and therefore is
to go to the store [beat movement], then I had to
not easy. Interestingly, very little research has been
get gas [beat movement], and then the dry cleaners
conducted to determine how observers of commu-
[beat movement].” (b) Deictic gestures (also iden-
nication (including gesture coders) actually distin-
tified in Ekman and Friesen’s, 1969, system) are
guish gestures from other kinesthetic movements.
indexical points, whose meanings are derived by
Cospeech gestures occur in contexts where there is
what is pointed to, instead of being contained in the
a cacophony of movements, some related to com-
form of the gesture itself. For example, when some-
munication (e.g., regulating gestures) and some not
one is indicating where an object is located, a point
related to communication (e.g., yawning or scratch-
at the object is combined with a spoken reference
ing your leg). How can cospeech gestures be isolated
to the object: “I put your glasses on the table [point
from all these other movements?
at glasses on the table].”
Distinguishing cospeech gesture from nonlanguage
HOW TO MEASURE COSPEECH GESTURES movement.  It still remains unclear whether there
is something in the form of the gesture itself or the
In this section, we attempt to articulate the process
context in which gesture occurs that allows the
of coding cospeech gestures. The process is not nec-
observer to interpret a movement as a gesture as
essarily the same across laboratories but involves
opposed to motion of the hand or body that is not
essentially the same sequence of steps, further elabo-
intended for communication. Nonetheless, there
rated in the rest of this section. The research dictates
are several criteria that coders use when observing
the type of communicative context under examina-
­communication that allow them to isolate gestures
tion (e.g., observing teachers explaining algebraic
from other behavioral phenomena. Cospeech ges-
concepts during a lesson or one individual telling a
tures satisfy all of the following criteria: (a) the hand
story to another). Once a context is identified, the
or body movement occurs in close temporal connec-
researcher will proceed through the following steps:
tion with speech (see Church, Kelly, & Holcombe,
Step 1. Differentiate cospeech gesture from other 2014, for timing differences between gestures and
behaviors. cospeech actions); (b) the hand or body movement
Step 2. Isolate the gesture word, morpheme, and occurs when the speaker is actively engaged with
utterance. another in communication (e.g., such as establishing
Step 3. Describe the form and meaning of the eye contact; Ekman & Friesen, 1969); (c) the hand
cospeech gesture. or body movement is semantically or pragmatically

Despite the fact that many gesture coding systems include these rhythmic gestures as part of the typologies, the experimental research establishing what
3

function these gestures serve in communication has been limited. Recently, studies have begun to experimentally test the functions implied in Ekman
and Friesen’s (1969) system and McNeill’s (2005) system (e.g., suggesting a highlighting function; Biau & Soto-Faraco, 2013; Chieffi & Ricci, 2007).

502
 Measuring Gesture

related to the speech content it accompanies and, does not hold a comb or come in direct contact
therefore, falls into one of the cospeech gesture cate- with the head). Thus, this combing movement
gories described in the previous section (i.e., is clas- would be considered a cospeech gesture convey-
sified either as iconic, metaphoric, beat, or deictic ing comb (hair).
gesture); and (d) the hand or body movement is not
Distinguishing cospeech gesture from other
a direct functional act on the self, another person,
­language associated gestures.  These same criteria
or an object, such as a self-adaptor (e.g., twirling
can be used to distinguish cospeech gestures from
one’s hair) or a transitive action (e.g., picking up a
other types of language-associated gestures, such
cup; Goldin-Meadow & Feldman, 1977). Movement
as those on Kendon’s continuum (emblems, signs,
that satisfies all four criteria would be classified as a
and pantomimes). Whereas Criteria b and d apply
cospeech gesture.
to emblems; Criterion a (the movement is synchro-
Therefore, for example, if a child is looking at
nized with the accompanying speech) and Criterion
his mother, holds his hand above a jar, and makes
c (the image is aligned with the content of speech
a twisting motion without actually twisting the lid
and is classified as one of the cospeech gesture cat-
off the jar while saying “Can I open this mommy?”
egories: iconic, metaphoric, beat, or deictic gesture)
then this twisting movement is considered a gesture
do not. For example, take the emblem signifying
meaning twist (the jar). In contrast, had the child
hello or goodbye (palm is raised away from the ges-
been looking intently at a jar (not at mommy) while
turer and moved back and forth). This emblem fails
actually opening it (a direct act on an object), and
to meet Criterion a because it is often used without
not speaking, this would be considered a functional
speech, making timing with speech a noncritical
act not intended for communication and, therefore,
feature. It also lacks iconic imagery so it fails to
not a gesture.
fit Criterion c (there is no image that aligns with
Not all instances are this clear, however. Take,
speech). Therefore, because this emblem does not fit
for example, someone holding a comb close to her
all four criteria, it cannot be classified as a cospeech
head and saying, “I really need to comb my hair.”
gesture.
Is this hand movement a gesture or a functional
Similarly, in the case of pantomimes and sign
act? This type of decision occurs all the time in
languages, whereas Criterion 2 (the movement is
coding and can be quite ambiguous. In some cases,
addressed to another person) and Criterion 4 (the
functional acts can be used with speech as demon-
movement is not a functional act) apply, Criterion 1
strations and, thus, really seem like gestures. How-
does not apply because the nature of pantomimes
ever, in this example, the “combing of the hair”
and signs is that they are performed without
would not be considered a gesture because the
speech, and Criterion 3 only applies to pantomimes
hand performs a direct act on a real object and the
(movement depicts an iconic image), whereas sign
self (Criterion d).
language does not because sign symbols are arbi-
In contrast, think of someone who uses a
trary forms.
hand movement to depict the act of combing hair
synchronized with the word “comb” in the state-
ment, “I really have to comb my hair.” This par- Step 2. What Is a Gesture Word?
ticular movement fits all four gesture criteria: (a) Verbal languages have at their root, the word unit.
The comb movement is temporally aligned with Words are easily parsed according to agreed upon
the word “comb”; (b) the statement and move- convention. Words are strung together in a linear
ment are addressed to another person; (c) the fashion. Gestures, however, are not as easily parsed
movement goes along with the semantic content into linear units. Whereas words are two-dimen-
of accompanying speech (i.e., combing) and, sional and static, gestures are three-dimensional,
therefore, would be classified as an iconic gesture seemingly fluid, and nonstatic. What constitutes a
(depicting the image of combing); and (d) the gesture word? How do we determine the beginning
movement is not a direct functional act (the hand and end of a gesture word?

503
Church, Kelly, and Wakefield
In spoken and written languages, morphemes
are combined in linear fashion. In contrast, gestural
“morphemes” can be combined in a linear, as well
as a nonlinear, fashion to reflect complex units
of information. Take, for example, the following
­gesture representing snow falling: Two spread hands
with palms facing down, the fingers flutter while mov-
ing down in front of the torso of the gesturer, produced
while saying, “Look how the snow is falling.” This
gesture has two components: (a) the flutter move-
ment of the fingers and (b) the downward move-
ment of the hand. Note that these two components
could have been represented sequentially (first the
fingers flutter, and then the hands move downward)
in the same way that words are strung together.
However, the gesturer represented these two move-
ments simultaneously (the flutter moves down-
ward). The sequential and simultaneous production
of gestural morphemes contains elements that can
be unitized, coded, and measured.
Gestural descriptions in some systems include
marking the entire gesture event: (a) the preparation
phase, in which the hand/arm or body movement
moves from a rest position to an enacting position;
(b) the stroke phase, in which the gesture forms its
peak image; and (c) the recovery phase, in which the
body/hand/arm returns to a rest position (Kendon,
2004; McNeill, 1992). Note, however, that to mark
these phases, the gesture itself (the stroke of the
gesture) has to first be identified and distinguished
from other bodily movements, as suggested in the
previous section. Including codes for the phases of
gesturing allows for a precise measurement of ges-
ture timing with respect to speech events. Coding
gesture phases anticipates techniques to precisely
measure the timing of speech and associated ges-
tures. We return to this application of coding sys-
tems later.
So, how can the flow of gesture movements be
parsed? How do we know when one gesture ends
and another gesture starts? This can be determined
by a number of features. First, a change in any
dimension of the gesture (e.g., in the form or loca-
tion of the gesture) can be used to differentiate
gesture units. Take, for example, a pointing gesture.
One could point to a table while saying, “I saw your
glasses on the table,” and then redirect the point in
504
one smooth movement to a countertop while saying,
“or maybe I saw them on the countertop.” Despite
the fact that the same finger point was used, the
change in location from the table to the countertop
indicates the production of two distinct gestures.
Second, a pause between presentations of two ges-
tures can be used to parse gestural units. Being able
to detect a pause between gestures is often a luxury,
and it has to be reliably observed.
Step 3. How to Describe Gesture
Form and Meaning
Once a gesture has been identified, gesture research-
ers attempt to describe its form and infer its mean-
ing. To determine meaning of gestures, the form of
the gesture and the context in which it is performed
(speech, discourse, or physical referencing) are
used. The form of the gesture resembles what is
being described. The gesture depicts its referent
through imagery. Take, for example, a child who is
describing milk being poured into a bowl of cereal.
The hand could be shaped as if holding a pitcher
(a fist) that moves along an arced path toward a
bowl while saying, “I poured milk into my cereal.”
The gestural act resembles a pouring movement, and
the verbal context and the orientation of the gesture
to an actual bowl allows for the interpretation of the
gesture to mean pour (milk into bowl).
One goal for coding gesture is to record a
description detailed enough to enable replication of
the gesture. In particular, there are three primary
dimensions of gesture form that are typically coded.
These dimensions are then used to infer gesture
meaning. Borrowing from Stokoe’s (1960) Ameri-
can Sign Language (ASL) coding system, these
three gesture dimensions include the following:
(a) location; (b) hand shape; and (c) the movement
trajectory of the hand, arm, or body. The infer-
ences made, however, do depend on how the form
of the gesture relates to the discourse context. We
illustrate this with examples in later sections of this
chapter.
Location.  The location of the gesture can provide
important information about its meaning. Location
has been noted with respect to particular locations
on the gesturer’s body (e.g., gestures occurring

in front of the face, the torso, or below) or with
respect to objects or other people in the speaker’s
environment.
Body location: Neutral.  Body locations can be
used in two ways. In the first use, the location is
considered neutral (called neutral space by McNeill,
1992). This neutral space includes the space in front
of the torso, including areas from the chin down to
the waist. When gestures are produced in n ­ eutral
space, the body location is not used for ascrib-
ing meaning to the gesture. Rather the neutrality
of the location is viewed as a stage for depicting a
particular image. For example, consider the follow-
ing gesture depicting running. One takes the index
and middle fingers of one’s hand, points the top of
the fingers downward in front of the gesturer, and
moves the fingers back and forth. The body is not
part of the interpretation but is a backdrop for the
running gesture.
Body location: The body is part of the meaning.  to an inference that the gesture represents holding
In the second use, location is used for meaning
inference. In this case, the gesture may depict a
body in action. The gesture performed on a body
part helps determine the meaning of the gesture, as
in the case of producing a stroking gesture oriented
toward the head to represent the act of combing
hair.
Object location.  Finally, location can be noted
in reference to outside objects. In work with chil-
dren performing spatial tasks, the location of ges-
tures performed near task objects enables coders
to infer gesture meaning. For example, a balancing
gesture (right and left hands with palms facing up)
placed near numbers on either side of mathemati-
cal equation displayed on a blackboard (i.e., 3 + 4 +
5 = __ + 5) can be used to infer the representation
of the idea equal.
Hand shape.  Hand shape can be used to represent
information about a person or object. Most cod-
ing systems use ASL alphabet hand shapes (e.g., a
B-shaped hand is a flat hand with fingers and thumb
closed; see Appendix 19.1) to describe a gesture
hand shape in the first pass of coding. The use of
the ASL alphabet allows for a systematic and uni-
versal way to code hand shapes across gesture labs.
Measuring Gesture
The shape of the hand, however, is compared to
the shape of the referent to determine meaning of
the hand shape. For example, a straight hand and
arm might reflect an object with the same straight
shape, such as a tree, and thus is interpreted as tree,
provided that the speech or discourse context sup-
ports this interpretation. A hand that is shaped like
a C held by the head is the same shape as a hand
holding a comb and, thus, is interpreted as hold-
ing, again, provided that the speech supports this
interpretation.
Orientation of the hand and fingers.  There are
additional characteristics of the gesturing hand that
are recorded to provide a replicable description.
In addition to the shape of the hand, the orienta-
tion of the palm and the fingers with respect to the
speaker’s torso is documented. Take, for example, a
C-shaped hand where the palm is facing toward the
center of neutral space. This orientation could lead
something (i.e., provided the context supports this
interpretation). However, if the C-shaped hand is
held with the palm facing down toward the floor,
this orientation could lead (again, depending on the
context) to an inference that the gesture is repre-
senting a round-shaped object, such as a turtle.
Use of one or two hands.  Whether a right hand,
a left hand, or both hands are used is also recorded.
If two hands are used, the relationship between the
hands is typically described. Therefore, for example,
if right and left C-shaped hands face toward the
speaker’s stomach, then both hands are each a part
of one referent, such as the attribute of fat (stom-
ach). In another case, two hands could each be
used to represent two entities that are conceptually
related, as, for example, when one hand represents a
hand holding a nail while the other hand represents
a hand holding a hammer and hitting the hand that
represents the nail.
Movement.  Movement of the hand is the third
dimension coded for gesture inference. Motion of
the hand or the body can represent either a pictorial
image or an action. Motion is coded either as
(a) within hand motion (e.g., fingers fluttering) or
(b) crossing space motion (e.g., a finger point that
505
Church, Kelly, and Wakefield
moves from in front of the speaker to behind the
speaker). A within hand motion is typically inferred
as reflecting a change in the state of an entity. A
hand that open and closes, for example, can repre-
sent a clam opening and closing or a hand gripping.
A movement of a hand that crosses space is typi-
cally understood either as an entity moving from
one location to another or as an attribute. A finger
point moving over a map is seen as representing
movement from one geographic location to another.
However, a finger point that traces the shape of an
object, such as a triangle, is seen as representing the
shape of a triangle.
Viewpoint.  Viewpoint is another dimension that
can be coded but is not a required dimension for
coding. It does bear mentioning, however, because
it provides interesting information about a speak-
er’s perspective—information that can be useful
in examining social discourse. Viewpoint can be
inferred from the combination of hand shape and
location (McNeill, 1992). McNeill (1992) identified
two viewpoints in the gestures produced by his par-
ticipants: (a) Observer viewpoint is reflected in a ges-
turing hand that represents an animated character
(e.g., a person or animal)—the hand is like a pup-
pet, on stage, in front of the speaker—whereas
(b) character viewpoint is reflected in a gesturing
hand that represents a bodily action. For example,
the act of running could be depicted using either
observer or character viewpoint. If expressing the
act of running through observer viewpoint, the mid-
dle and index fingers could represent a person’s legs,
and the wiggling of the fingers could represent the
act of running. This event is set in front of a neutral
body location, suggesting that the gesture is being
produced on an imaginary stage, with the speaker
at a distance from this event—thus, an observer’s
viewpoint. In contrast, the act of running could
also be depicted in character viewpoint. Using the
entire body, the gesturer could pump her arms
to depict what her arms would look like if she
were running. The body location in this character
viewpoint gesture is not neutral; rather, it pro-
vides important information for interpreting the
meaning and viewpoint of the gesture. The gesture
movements are oriented toward the speaker’s body,
506
in a sense employing the body to represent a body
in motion.
It is important to highlight that the form of
cospeech gestures are interpreted using the accom-
panying speech or discourse context to infer mean-
ing. That is, the discourse context surrounding the
gesture is often critical for interpreting the gesture.
Take, for example, the tree gesture described earlier.
The straight hand and arm was interpreted as a tree
because the accompanying speech was “The tree
fell over.” If that same gesture had been produced
with the following speech, “The tower fell,” then
it would have been interpreted as meaning tower.
However, when we describe how cospeech gesture
can be used to understand cognitive processes such
as learning, there is an exception to using speech
information to infer gesture meaning. This is dis-
cussed further in the Gesture Function section.
Step 4. Analyzing Gesture: What Are
the Important Questions?
The way gesture is analyzed depends on the ques-
tion being asked by the researcher. The variety of
ways that gesture is analyzed is captured in the
Gesture Function section. Essentially, however,
cospeech gestures are measured and analyzed
to address language and thinking. Specifically,
cospeech gestures, as we pointed out before, pro-
vide a possible window onto mental representa-
tions associated with language and cognitive
activities. Precisely because cospeech gestures are
not dictated by any formal rules, they can pro-
vide a rich and raw reflection of those processes
underlying language and thinking. The way we
use gestures to address questions about language
and thinking is exemplified later in this chapter.
Laboratories studying gesture have at their disposal
the schemes just presented for accurately identify-
ing and describing gestures. The approach of how
to measure and analyze a gesture is determined by
the questions that interest a researcher and gener-
ally falls into one of two categories: linguistic or
cognitive.
Linguistic approach.  The linguistic, or a psycho-
linguistic, approach is characterized by the work of
Kendon and McNeill and focuses on “descriptive

facts that contribute to a theory of human language”
(Duncan, 2013, p. 1004). This approach is less
guided by a coding system and is more guided by
heuristics or principles developed over decades of
research on gesture and language. This approach
attempts to describe how gesture is used in the
­context of narrative and conversational discourse.
The discourse context being analyzed is typically
with respect to a single person or between two peo-
ple. The gesture data are correlated with other fea-
tures of the discourse context, and these correlations
are used to determine how gesture may function. As
described later, researchers, such as McNeill and his
colleagues, have been able to identify iconic gestures
that provide cohesive functions across a particular
narrative theme.
Cognitive approach.  The second approach char-
acterizes cognitive research in which coded ges-
tures are tabulated, not only for a particular person
but also across a number of individuals engaged in
discourse. In this approach, gestures can be treated
as a scale or continuous variable or as a categori-
cal variable to determine patterns of frequency.
Patterns of gesture production can then be exam-
ined as a function of the type of discourse context
or an experimental treatment. For example, one
question that has been asked in gesture research is
whether teachers in classroom situations gesture
when explaining particular concepts. This ques-
tion can be addressed by asking whether teachers
gesture at all during instruction in the domain in
question or by conducting a more detailed analysis
of how often gesture is used. In the first case, a cat-
egorical analysis could be used: This would require
a researcher to determine how many teachers in a
sample produce gesture at any point in a classroom
lesson versus those who produce no gestures at
all. In the second case, a frequency analysis could
be used: It may be important to know not just
whether a teacher does or does not gesture but also
how often gesture occurs during a lesson—a con-
tinuous measure of gesture use. Analyzing gesture
frequency would show how often teachers produce
gestures and the variability across a sample of
teachers in frequency of producing gestures.
In either case, one can apply the typical statistical
Measuring Gesture
methods appropriate for the treatment of gesture as
a categorical or continuous variable.
Tools for Measuring Gesture
Measurement of gesture is often a painstaking
activity relying on keen human observation. With
the advent of a number of tools, the scientific study
of gesture has greatly improved.
Video recording.  One such tool is video
recording. Prior to the advent of video record-
ing, researchers studied gesture by observing live
communication during naturalistic interactions
(Kendon, 2004). Live observations are limited and
do not allow for the precise coding or transcription
of all discourse activities. With the development
of video recording, researchers were able to per-
form more precise and detailed coding of gesture.
Video recordings could be taken back to labs and
examined using slow motion, time lapse, and event
capturing software. Through this more detailed
analysis, researchers could examine the minute
details of cospeech gesture events, such as the exact
timing differences between the onset of spoken
words and accompanying gestures. The importance
of coding the timing of speech and gesture speaks
to the question of whether speech and cospeech
gesture are an integrated communication system
(a significant question in the literature, as we show
later in the chapter).
The advent of video coding tools has improved
the ability to code and measure gesture. Tools
designed for video viewing and editing are used for
observation, transcription, and analysis of gesture
data (e.g., Kendon, 1972). Many such software
packages (e.g., Apple Final Cut; Adobe Premiere
Pro) are used in research examining gesture pro-
duction in the context of complex discourse activi-
ties. There has been, however, recent development
of multimedia software that is custom-designed
for examining digitally recorded discourse com-
munication data. Programs such as ELAN (Wit-
tenburg, Brugman, Russel, Klassmann, & Sloetjes,
2006), Anvil (Kipp, 2012), and Transana (Woods &
­Dempster, 2011) allow the researcher to set up crite-
ria for identifying discrete behavioral units (speech,
gesture, or any other behavior) that may occur
507
Church, Kelly, and Wakefield
in the natural course of communication. After
identifying these units within the discourse data,
researchers can export their codes to programs
such as Excel, through which they can then count
these units, compare the counts across different
preidentified situations or prescribed variables,
and evaluate timing among different behaviors of
interest, to name a few functions. In addition, these
programs keep detailed accounts of behavior so a
researcher can return to a particular piece of foot-
age, identify a particular gesture, or highlight a
context in which gesture appears. In essence, these
programs combine both a qualitative and a quanti-
tative approach to analyzing complex multimodal
communication situations (Duncan, Rohlfing,
Bielefeld, & Loehr, 2013).
For example, with a software package such as
ELAN, a variety of descriptions of speech and ges-
ture can be applied to videos containing complex
language information. These descriptions are called
annotations. The types of annotations can include
verbatim transcripts of speech, gesture, or any other
behaviors occurring in the context of the video
recorded activity. These transcriptions can be given
a variety of codes or classifications for all behaviors
under examination. These transcriptions can be
parsed in a multitude of ways capturing different
levels of granularity. Thus, annotations can focus
on a variety of features depending on the particular
interest of the researcher. The annotation can be a
word, a sentence, or a meaning inference attributed
to the particular language unit being measured.
These annotations can occur within layers of infor-
mation (e.g., speech, right-hand gesture, left-hand
gesture), which are referred to as tiers (see Appendix
19.2). Layers or tiers can be interconnected hierar-
chically, and they are often aligned with time mea-
sured at a variety of levels (milliseconds, seconds,
minutes, etc.). In the case of cospeech gesture, this
tool can answer many interesting questions: How
synchronized are gestures to certain speech phe-
nomena? Does a particular gesture always co-occur
with a particular word, phrase, or idea? Does the
onset of gesture occur with the onset of a particular
word? At what level of time measurement can or
should this synchrony be assessed? The question of
granularity of synchrony depends on the research
508
question. It may be enough to know that an action
gesture occurs with a particular verb in a sentence
as opposed to, for example, an object of the verb
in that sentence. In this case, the granularity of the
time onsets is somewhat gross. However, imagine
determining whether the temporal onset of pro-
ducing a gesture occurs before, during, or after the
onset of producing a particular phoneme in a word.
This level of synchrony requires timing decisions at
the level of milliseconds. A software package such
as ELAN allows for all levels of measurement and
the determination of many behavioral features all at
the same time. These technological innovations have
improved the precision of the process of identifica-
tion, classification, and explanations for gestural
data, although the software is still not capable of
identifying individual gestures—a researcher must
still manually annotate the video before making use
of these features.
Interrater reliability.  Reliability is perhaps one of
the most critical tools for measuring gesture. The
success of a coding system relies on how well it can
direct two (or more) independent coders to describe
the same phenomenon. Moreover, reliability can be
established through repeated observations of behav-
ioral events, an impossible task in live observations.
In virtually all articles that describe coding (gestural
or otherwise), agreement between two coders has to
be significantly greater than chance. High reliability
scores (80% agreement or more) in coding gesture is
key in measuring gesture and making any assump-
tions about its meaning or function in communica-
tion. We next describe how video recording and
interrater reliability are used to classify and describe
cospeech gesture by showing how gesture measure-
ment is applied to questions about certain language
and cognitive processes.
GESTURE FUNCTION: APPLICATION OF
GESTURE MEASUREMENT
In this section, we demonstrate how gesture cod-
ing and measurement can be applied to research
that attempts to understand cognitive and language
activity. Specifically, we use particular research
contexts to exemplify (a) how different laboratories

code gesture and (b) how these labs address gesture
functions. We focus next on the application of ges-
ture measurement for determining the function of
gesture using these two approaches.
Gesture and Linguistic Research
This research orientation focuses primarily on the
use of speech and gesture in language activity
(e.g., Gullberg & Holmqvist, 2006; Kendon,
2004; Kita & Özyürek, 2003; McNeill, 2005;
Stam, 2006). Typically, narrative descriptions or
redescriptions are elicited by asking a participant
to retell what happened in a story (e.g., a car-
toon or a written story; McNeill, 1992) or event
vignette (Özyürek, Kita, & Allen, 2001). If there
are units to be identified during analysis, they are
speech–gesture units marked by the combination
of speech clauses and gesture phrases. The combi-
nation of gesture with speech is of primary interest
because gestures, according to these researchers,
reflect a visuo-motor manifestation of an idea.
Important to this conceptualization is the notion
that speech and gesture are an integrated system of
communication. Gesture stems from the same idea
units as speech but in a different, raw, imagistic,
and nonconventionalized form (Duncan, 2013;
Kendon, 2004).
As an example, we focus on the work of McNeill
(1992, 2005, 2012), as he was one of the original
researchers to focus on gesture as a psycholin-
guistic phenomenon. Moreover, McNeill’s coding
system has been adapted by many current gesture
researchers. McNeill elicits speech and gesture by
showing individuals a simple, nearly wordless car-
toon (Warner Brothers series on Tweety Bird and
Sylvester the Cat) and then asks viewers to retell
the cartoon story. These retellings are recorded, and
the recordings are transcribed, annotated, and ana-
lyzed. The transcriptions and annotations are the
result of many passes through these data. The first
pass “involves transcription of everything that is
audible in speech” (Duncan, 2013, p. 1007), includ-
ing words, partial words (or aborted word produc-
tion, such as th—th—th stutter in place of the word
the), speech repairs (i.e., fixing a word that was
misspoken, such as “he—I meant she”), breaths,
speech pauses, laughs, and other sounds that may
Measuring Gesture
be produced as part of the speech event. The speech
is transcribed in separate lines from gesture. This
layered transcription allows for gesture and other
bodily movements related to the narration to be
temporally compared with the speech.
The annotator infers the meaning of gestures
using the story context surrounding the gesture
(both the cartoon and the speech narrations) and
the features of the gesture form (discussed earlier
in this chapter). Moreover, the annotator makes
inferences about the discourse function of
gestures. These inferences require, however,
examining the appearance of gestures in combina-
tion with speech in the larger context of the nar-
ration. Often, repeated passes to transcribe and
analyze these narrations can result in reinterpre-
tation of gestures and their roles in the narration
(Duncan, 2013).
Coding gesture in linguistic research.  McNeill’s
coding system has gone through a number of itera-
tions (Duncan, 2013). However, for the purpose
of how to measure gesture in linguistic analyses,
we talk about his original system (borrowed from
Stokoe’s, 1960, system; see also McNeill, 1992).
That is, in McNeill’s research, gestures are identified
and described in terms of their hand shape, location,
and motion (see the Appendix in McNeill, 1992,
for an elaborate description of McNeill’s coding sys-
tem). This information is interpreted in the context
of the speaker’s narrative discourse and the stimulus
subject matter (e.g., a Warner Brothers cartoon).
For example, we can consider how the following
gesture-plus-speech event was coded. A right-hand
index finger (the hand shape) placed in front of the
speaker in neutral space (location) moves upward
(motion) while the speaker says, “He went up the
drainpipe.” In McNeill’s system, meaning is attrib-
uted to both the hand and the motion, but not to
location, which is neutral. In this example, the index
finger represents Sylvester the cat, and the upward
motion reflects Sylvester ascending up the drain-
pipe (in search of Tweety Bird). The gesture was
interpreted as Sylvester moves up the drainpipe. Note
that the context of the cartoon events and the co-
occurring speech event are used for interpreting the
gesture. In this case, the gesture depicts imaginary
509
Church, Kelly, and Wakefield
elements in space. The drainpipe, which Sylvester
climbed, was implied in the gesture.
How does gesture function for story
­narration?  This type of research uses these
detailed descriptions of gesture to make inferences
about their function in discourse. These infer-
ences are typically made at the level of exemplars
identified in the data by examining the speech con-
text surrounding the production of gestures. For
example, McNeill created a classification of gesture
based on the function gestures served in ­narration.
This class of gestures is referred to as catchments
(McNeill, 2005). Catchments are gestures, or fea-
tures of a gesture, that recur over a piece of dis-
course for the purpose of narrative cohesion. In the
transcription of one of McNeill’s (2005) p ­ articipants,
he identifies the recurrent use of a door gesture (two
flat hand gestures, palms facing out away from the
body, representing her front doors) during a descrip-
tion she gave of her house. Analyses that support
the contention that certain gestures are catchments
are based on the recurrence of a particular gesture
(such as the door gesture) throughout the story, cor-
relating with other speech events. In the example of
the door catchment gesture, prosody and intonation,
which are also vehicles for cohesion, were found to
correlate with the occurrence of this door gesture
(McNeill, 2005).
In keeping with his training as a linguist,
McNeill’s work entails in-depth, intensive coding
of an individual or a few individuals to determine
patterns of gesture functioning. As in the catchment
gesture example, analysis is done over the course of
a single narration to draw conclusions about how
gesture functions for an individual. In applications
of gesture measurement for understanding cognitive
processes, larger gesture databases are used to cap-
ture patterns of gesture function across individuals.
We turn to this research next.
Gesture and Cognitive Research
Research examining gesture in the context of
­cognitive problem solving began with a focus on
children and has since expanded to adults (e.g.,
Beilock & Goldin-Meadow, 2010; Hostetter &
Alibali, 2008). We illustrate this type of research
510
using the work conducted by Goldin-Meadow and
colleagues. Church and Goldin-Meadow (1986)
created a coding system that was modified from
Goldin-Meadow’s system for coding language-
like properties in the gestures of deaf children
(Feldman, Goldin-Meadow, & Gleitman, 1978;
Goldin-Meadow & Mylander, 1984). Church and
Goldin-Meadow’s coding system extended McNeill
and Kendon’s systems in two important ways:
(a) The system was designed to describe chil-
dren’s gestures, and (b) the system was designed
to describe ­gestures produced in an educational,
problem-solving context.
Coding gesture in cognitive development
research.  Goldin-Meadow and colleagues’ cod-
ing system was also a variant of Stokoe’s (1960)
system. The hand shape (using the ASL alphabet),
location, and movement were used to describe the
form and meaning of gesture produced by children
who had been asked to solve conceptual problems.
Meaning inferred from the gestures was also deter-
mined by the context of the communicative situ-
ation. We explore one application of this coding
system in the context of a Piagetian conservation
task.
Using gesture to analyze problem solving.  In a
study conducted by Church and Goldin-Meadow
(1986), children were asked to solve Piagetian
conservation problems and to explain their solu-
tions. One Piagetian problem requires children
to observe the transformation of a quantity—for
example, a tall glass of water poured into a short,
wide dish. Children are asked whether the amount
of water in the dish is the same as the amount of
water in the original glass. Children younger than
7 years of age are likely to be compelled by one
dimension of the containers to make their decision:
Children see the different heights of the contain-
ers and conclude that the water in the dish is not
the same amount—that is, a tall glass contains
more water than a short dish. Church and Goldin-
Meadow found that when children explained their
judgments about quantity transformations, their
­verbal explanations were frequently accompanied
by cospeech gestures (Church, 1999; Church &
Goldin-Meadow, 1986).

Children’s problem-solving strategies can be
revealed in gesture.  Using Stokoe’s (1960)
­elements (hand shape, location, and motion),
Church and Goldin-Meadow (1986) coded these
gestures and interpreted them in the context of the
Piagetian problem-solving task. Church and Goldin-
Meadow’s system focused on two essential categories
of gesture: (a) deictics or points and (b) character-
izing gestures (or iconic gestures that represent
either attribute or action information). Finger
points were interpreted as a way to index Piagetian
conservation task objects, such as the glass or the
dish. Characterizing gestures (much like Ekman &
Friesen’s, 1969, illustrators or McNeill’s, 1992,
iconic gestures) were interpreted as reflecting either
action or attribute information related to the conser-
vation problems, such as the width or height of the
glasses (Church & Goldin-Meadow, 1986; Goldin-
Meadow & Feldman, 1977).
Church and Goldin-Meadow’s (1986) system
focused on gesture combinations or gesture utter-
ances produced when children had to reason about
conservation task objects. Therefore, Church and
Goldin-Meadow established criteria for delimiting
gestures and how they were strung together. Dif-
ferentiating one gesture word from another was
determined using a change in either location, hand
shape or movement, or a pause between one gesture
and another.
A gestural utterance was coded according to its
appearance with a speech explanation of a task judg-
ment and could contain one or more gestures. That
is, speech explanations were delimited by a particu-
lar argument or strategy that supported a judgment
about a quantity transformation (e.g., “They’re dif-
ferent because the water is taller in that glass”). All
of the gestures that occurred in the bounds of that
speech explanation were considered part of one
gesture utterance. Gesture utterances occurring
with speech explanations of conservation were seen
as conveying a strategy for a conservation problem
solution—that is, the logic or reasoning behind a
conservation solution. Let’s consider an example.
One child, when asked to explain her nonconserv-
ing solution (the dish and glass do not contain the
same amount of water), said that “The dish is short,
and the glass is tall.” The gesture accompanying this
Measuring Gesture
explanation was a flat hand, with palm facing down
(hand shape) placed near the dish (location) indi-
cating short (dish). This flat hand then moved to the
top of the glass indicating tall (glass). This gesture
utterance contains two gestures: (a) short (dish) and
(b) tall (glass), using the criteria that the location of
the hand changed (hand moved from the dish loca-
tion to the glass location). The inference made for
the meaning of the gestures is based on the Piagetian
problem context and the placement of the gestures
near the objects being explained (i.e., the dish and
the glass). The interpretation of the gesture strat-
egy was much like the interpretation of the speech
strategy. Both speech and gesture used a comparison
of heights strategy (the heights of the glass and the
dish are different, therefore, the amount of water is
different) to justify the belief that the liquid quality
had in fact changed, once poured from tall glass to a
short dish.
Different from the psycholinguistic approach,
which uses speech to interpret gesture, Church
and Goldin-Meadow (1986; see also Church, 1999)
coded the speech strategies of the children inde-
pendently of the gesture strategies. This was pos-
sible because the Piagetian conservation context
constrained the interpretation of the gestures. In
addition, the gestures children used were iconic
depictions of the actions and attributes of the con-
servation task objects, which were easy to interpret.
Coding speech independently from gesture revealed
that in many speech-plus-gesture explanations, a
strategy expressed in one modality differed distinctly,
or mismatched, the strategy expressed in the accom-
panying modality (more on this later in the chapter).
So what role does gesture play with the speech
communication in the Piagetian problem-solving
tasks? Cospeech gesture researchers ask whether
speech and gesture work in concert as an integrated
system during communication (e.g., McNeill, 2005).
Evidence for speech and gesture working as an
integrated system comes from the work that has
examined how gesture impacts speech processing
and learning in the context of children solving these
Piagetian conservation problems.
Does gesture affect learning?  Church and Goldin-
Meadow (1986; see also Church, 1999) found that
511
Church, Kelly, and Wakefield
gesture explanations convey important information
about a child’s mental representation of the prob-
lem being solved. In particular, measurement of
gesture in combination with speech could be used
to assess transitions in learning new material. In
the case of the concept of conservation, measuring
gesture could indicate when a child was in transi-
tion toward conserving quantity after a transforma-
tion. Church and Goldin-Meadow noted that the
gestures failed to directly match the conservation
speech justifications in roughly half of the children
they studied. For example, one child justified her
nonconserving response by saying, “Because this
glass is taller than that short dish.” However, rather
than produce gestures conveying tall (glass)–short
(dish)—information that would match the accompa-
nying speech—the child gestured tall + thin (glass)–
short + wide (dish)—information that supports a
conserving judgment; the water has not changed
in amount because change in the height dimen-
sion (tall glass and short dish) is compensated by
change in the width dimension (thin glass and wide
dish). Thus, gesture information in speech–gesture
mismatches suggested some implicit understand-
ing of the conservation principle not yet evidenced
explicitly in speech. The frequency distributions of
speech–gesture mismatches observed across chil-
dren suggested a distribution with peak productions
at three mismatches. This distribution was used to
classify children into two groups: (a) transitional
children who produced 50% or more mismatches
and (b) stable children who produced fewer than
50% mismatches. It was hypothesized that the tran-
sitional children (who produced speech–gesture
mismatches greater than 50% of the time) were more
ready to learn than stable children (who produced
speech–gesture mismatches fewer than 50% of the
time). Using a pretest–instruction–posttest design,
Church and Goldin-Meadow found that transitional
children were significantly more likely to profit from
instruction in the conservation principle than stable
children (Church, 1999; Church & Goldin-Meadow,
1986). Mismatching gesture has also been shown
to index readiness to learn in abstract mathematics
problems (e.g., Perry, Church, & Goldin-Meadow,
1988). Thus, in cognitive research, cospeech ges-
tures appear to reflect mental representations
512
underlying the way children solve mathematical
problems. That is, the imagery depicted in these
cospeech gestures represents the mental images chil-
dren use when attempting to think about a problem.
Does gesture affect speech processing?  Church
and Goldin-Meadow’s (1986; see also Church, 1999)
research on speech–gesture mismatches led to ques-
tions about the role that gesture plays in speech
comprehension. Gesture occurs frequently when
children attempt to explain how they solve prob-
lems. Could gesture affect the way speech is pro-
cessed? This question becomes particularly relevant
when considering classroom instruction. Indeed,
research has demonstrated that teachers frequently
gesture during their instruction of concepts, par-
ticularly math concepts (Alibali & Nathan, 2012).
Are these gestures detected by learners? Can these
gestures benefit learners’ acquisition of classroom
concepts? Several researchers have manipulated
the presence of gesture with speech statements to
address how gesture affects speech processing
(e.g., Church, Garber, & Rogalsky, 2007; Church,
Kelly, & Lynch, 2000; Kelly & Church, 1997, 1998).
In one such study (Kelly & Church, 1998), video-
tapes of children performing the Piagetian conserva-
tion task were culled to create the following stimuli:
(a) speech explanations combined with matching
gesture, (b) speech explanations combined with
mismatching gesture, and (c) speech explanations
with no accompanying gesture. Participants were
shown these three types of stimuli and asked to
repeat the explanation shown on the video or to
answer questions about the information conveyed
in the explanations (e.g., Did the child indicate that
the dish was short?). Both child and adult observers
of these video stimuli were better able to repeat the
speech information when it was accompanied by a
matching gesture than when accompanied by a mis-
matching gesture or no gesture at all.
Thus, the behavioral evidence suggests that
cospeech gesture can be reliably identified and
described. Moreover, studies that examine how
gesture functions in communicative discourse sug-
gest that gesture’s relation with speech is symbi-
otic. Results from research examining discourse
in an educational context provide compelling

evidence that speech and cospeech gesture work as
an i­ ntegrated system in production and also when
­perceived by observers of communication. Next, we
focus on a relatively new way to measure gesture:
measurement at the neurological level. Specifically,
we discuss methods used to explore the underly-
ing brain mechanisms driving the way speech and
cospeech gesture operate in communication.
BRAIN MECHANISMS: APPLICATION OF
GESTURE MEASUREMENT AT THE
NEUROLOGICAL LEVEL
Traditional approaches to measuring gestures have
relied on behavioral measures, such as annotated
transcripts, memory tests, reaction times, temporal
synchrony, learning outcomes, subjective evalu-
ations, and developmental trajectories. Although
researchers have learned a great deal about gesture
from measuring it in these ways, there are many
interesting questions that cannot be addressed
using these overt behavioral measures. Indeed,
by looking beneath the behavioral surface of ges-
ture processing and more directly at its neural
mechanisms, researchers might gain new insights
that would be missed by focusing only on higher
levels of analysis (e.g., social and cognitive). For
example, does the brain process gesture in the same
way as words? Does the brain integrate gestures
and speech? If so, when and where? In the follow-
ing two sections, we attempt to shed light on these
important questions.4
Functional Magnetic Resonance Imaging
(fMRI) Measurement of Gesture
One of the neuroimaging techniques that has been
used to study gesture processing is fMRI. This
technique is popular because it is noninvasive and
provides data with excellent spatial resolution. fMRI
allows researchers to look where in the brain popu-
lations of neurons are activated when people per-
form different tasks, and it also allows researchers
to ask how these regions are functionally connected
Due to limitations in the current capabilities of neuroimaging techniques, very little research has measured brain activity while people produce gestures.
4
However, in light of the research reviewed in the previous section on the linguistic and cognitive processes involved in gesture production, this limita-
tion offers an exciting challenge for gesture researchers to tackle in the future when imaging techniques become more flexible.
Measuring Gesture
to each other. In the context of gesture research, it
is possible to ask questions such as the following:
What regions of the brain are recruited when some-
one is presented with a message in speech alone, or
the same message accompanied by cospeech ges-
ture? By comparing brain activation to each of these
message types, researchers can actually measure ges-
ture processing neurologically to assess how gesture
contributes to message comprehension.
Although fMRI measures brain activity, it does
not do this directly. With fMRI, researchers actu-
ally measure the blood-oxygenation-level depen-
dent (BOLD) signal, which indicates where oxygen
is being used in the brain (e.g., Amaro & Barker,
2006; Heeger & Ress, 2002). To elaborate, when an
individual performs a task, it is assumed that popu-
lations of neurons that are recruited to complete
the task use their supply of oxygen and glucose.
When researchers perform BOLD contrasts, they
take advantage of how the brain responds to this
depletion: Oxygenated blood is sent to replace the
deoxygenated blood in the localized area. Research-
ers can measure differences in oxygenation level
with BOLD imaging because the properties of oxy-
genated and deoxygenated blood affect the signal
differently. Oxygenated blood has little effect on
the strength of the BOLD signal. In contrast, deoxy-
genated blood introduces inhomogeneities into
the magnetic field that cause the BOLD signal to
decrease in intensity (for more details, see Amaro &
Barker, 2006; Heeger & Ress, 2002). Simply put, a
stronger percentage signal change will be seen in
the BOLD signal as the proportion of oxygenated to
deoxygenated blood increases, and this will occur
when an oversupply of oxygenated blood is sent to a
region to compensate for the blood that has become
deoxygenated as active neurons use oxygen. The
BOLD signal is taken as a proxy for neural activity,
and research has been conducted to substantiate
the relation between the BOLD signal and neural
signaling (e.g., Logothetis, 2003; Logothetis, Pauls,
Augath, Trinath, & Oeltermann, 2001; Logothetis &
Wandell, 2004).
513
Church, Kelly, and Wakefield
Importantly, the physiological response that is
needed to increase the blood supply to an area of
the brain in response to neural signaling is rela-
tively slow. The BOLD signal does not increase in a
region until approximately 2 s after stimulus onset
during an experiment because of this physiologi-
cal constraint—for this reason, the BOLD signal is
considered sluggish in relation to the speed at which
neural signaling actually takes place. This is why the
temporal resolution of fMRI is considered inferior to
techniques such as event-related potentials (ERPs;
discussed in the next section), which can be used
to consider temporal effects on the level of milli-
seconds. In summary, fMRI is a method that can be
used to determine where in the brain neural activity
is taking place. As mentioned earlier, researchers
ask participants to complete different tasks dur-
ing a scan (this can include simply lying supine
in the scanner and perceiving sets of stimuli), and
they measure brain activation in these tasks in
relation to each other, and in relation to epochs of
rest (i.e., periods of time during a scan in which a
participant views a fixation cross, and no response
is required). Researchers have used elegant study
designs to address many questions that are impor-
tant in the field of gesture research. Here, we dem-
onstrate how fMRI studies have shed light on two
issues introduced earlier in this chapter: (a) Does
gesture affect speech processing? (b) Does gesture
affect neural processes involved in learning?
Does gesture affect speech processing?  As
­discussed earlier, one of the overarching questions
about gesture is how it contributes to the under-
standing of spoken language. With fMRI, researchers
have directly addressed this question by asking how
the brain responds to communicative input in which
gesture is present or absent, as well as input in
which the relation between gesture and speech has
been manipulated (e.g., gesture and speech express
identical information, complementary information,
or conflicting information). To interpret results
from these studies, researchers typically consult
neuroimaging literature that has established typical
neural responses during language processing, multi-
modal integration, and action observation (i.e., how
humans process actions produced by others that are
514
not gestural in nature), which are all presumably
related to cospeech gesture processing.
When asking whether gesture is incorporated
into the understanding of the meaning of a message,
one of the major regions of the brain that has been
considered is the left inferior frontal gyrus (IFG),
because it is known to be involved in language pro-
cessing and meaning integration. In the first study to
ask whether the IFG was sensitive to gesture, which
would support the notion that gesture could affect
understanding of a spoken message, participants
were presented with spoken sentences that were
accompanied by an iconic gesture produced simul-
taneously with a target word (Willems, Özyürek, &
Hagoort, 2007). Some messages were congruent:
Sentences made sense, and the gesture was con-
gruent in meaning with the target word (e.g., “He
should not forget the items he wrote on the shop-
ping list”; gesture: write). Other messages contained
incongruencies: (a) The target word could be incon-
gruent with the rest of the sentence and with the
gesture (e.g., “He should not forget the items he hit
on the shopping list”; gesture: write), (b) the gesture
could be incongruent with the target word and sen-
tence (e.g., “He should not forget the items he wrote
on the shopping list”; gesture: hit), or (c) the target
word and gesture could be congruent with each
other but incongruent with the rest of the spoken
message (e.g., “He should not forget the items he hit
on the shopping list”; gesture: hit). The researchers
predicted that if speech and gesture were part of the
same communicative system (and, thus, gesture was
semantically processed along with speech), then any
incongruencies in a message, whether they came
from gesture or speech, should elicit activation of
an overlapping network. Further, this overlapping
network would include the left IFG, which is impor-
tant for unifying lexical information. Results from
Willems et al.’s (2007) study suggest just such an
overlap between spoken language and gesture pro-
cessing systems in the brain. Their research shows
that the left IFG responds similarly to all three of the
incongruent conditions but differently for the con-
gruent condition.
Many other studies have considered this region
as an important focus for understanding whether
gesture is processed as part of language. Whereas

Willems et al.’s (2007) early interpretation of the
role the IFG plays in gesture processing has gar-
nered some support (Green et al., 2009), o­ thers
have suggested that this region may show an
increase in activation with an increase in message
complexity. This would mean that in Willems et al.’s
study, the incongruencies added complexity to the
message participants were working to interpret.
Evidence for this theory of IFG function comes
from studies in which the difficulty of processing a
message is manipulated, but difficulty is modulated
by the complexity or ambiguity of a message, not
from the presence of blatant incongruencies that
make a sentence nonsensical (e.g., Dick, Mok, Raja
Beharelle, Goldin-Meadow, & Small, 2014; Straube,
Green, Weis, Chatterjee, & Kircher, 2009; Villarreal
et al., 2008). For example, in a study conducted by
Dick et al. (2014), participants were presented with
spoken sentences that either contained a target word
that matched a gesture or a target word that was
disambiguated by a gesture. The IFG showed greater
activation to the ambiguous speech that was disam-
biguated by the gesture, in which the information
from gesture and speech had to be integrated before
sentence meaning could be determined. Evidence
has also come from studies that use functional con-
nectivity analyses—a type of analysis that allows
researchers to ask which parts of the brain have
an influence on other parts of the brain during the
processing of various experimental stimuli. Skip-
per, Goldin-Meadow, Nusbaum, and Small (2007)
presented stories that were either accompanied by
meaningful gestures, self-adaptors, or no gestures,
or a condition in which participants heard stories
but received no visual input. The IFG showed the
weakest functional connections with regions asso-
ciated with motor and language processing (e.g.,
premotor regions, superior temporal regions) dur-
ing the meaningful gesture condition. The authors
interpreted this result as suggesting that because
meaningful gestures added necessary s­ emantic
information to the speech, the story was easier to
interpret and required less semantic selection and
retrieval than conditions not containing those help-
ful gestures.
This short review of select findings related to the
IFG illustrates how fMRI can provide insight into
Measuring Gesture
gesture processing in a region that has often been
implicated in the processing of spoken language.
Although not discussed in this chapter, these and
other studies have not only explored the role of the
IFG in gesture processing but also a larger network
of regions that are recruited when gesture is used
in communication. Taken together, the current lit-
erature suggests a general network of regions that
are recruited for gesture processing, although the
specific contributions of each of these regions are
still being debated. Importantly, the literature gen-
erally supports the view that gesture is processed
as a meaningful part of a message—once a point of
contention in the behavioral literature (see Krauss,
Morrel-Samuels, & Colasante, 1991).
Does gesture affect neural processes involved
in learning?  The question “Does gesture affect
speech processing?” is a question of perception, as
the main concern is how an individual perceives and
interprets speech accompanied by cospeech gesture.
In the case of learning, researchers have asked
whether learning (measured behaviorally) is
enhanced when an individual perceives speech
instruction accompanied by cospeech gesture. The
other side of this question is how self-produced
gesture in a learner affects his or her own cogni-
tive state.
Very few fMRI studies have asked how self-
produced gesture during learning changes the way
learned information is processed. Because motion
during the scan can cause artifacts in the functional
data and increase overall noise of the data (Evans,
Todd, Taylor, & Strother, 2010), it is difficult to
measure actual gesture production while in the
scanner. Although researchers can “correct” for
motion to an extent, data are generally excluded
from studies with adult participants when motion
exceeds 1 mm in any direction. Perhaps because of
this, the effect of self-produced gesture on cogni-
tion has been understudied. However, we can ask
how information learned through gesture, outside
of the scanner, is processed by the brain during a
subsequent imaging session. For example, Macedonia,
Müller, and Friederici (2011) trained adults on
new vocabulary words outside of the scanner, and
they assessed how they processed these words in a
515
Church, Kelly, and Wakefield
subsequent scan. To learn words, participants were
shown videos in which (a) the novel word appeared
on a screen, followed by a translation of the word
in their native language, and (b) a short clip was
shown of an iconic gesture related to the novel word
or an unrelated control movement. After seeing the
video, participants were asked to imitate the ges-
ture or unrelated movement and to say the novel
word. Training on words occurred over 4 days, and
a translation assessment showed that by the end
of training, participants had learned almost all of
the new words, across both conditions. During a
subsequent scan, participants were presented with
learned words and unlearned words and were asked
to identify words as such. Macedonia et al. found
that premotor regions showed significantly greater
activation to the words that were learned with self-
produced iconic gesture than the words that were
learned with unrelated movements. This suggests
that the brain responds to learning stimulated by
self-produced gesture, and thus gesturing during the
learning process can impact cognitive change.
ERP Measurement of Gesture
Whereas fMRI is well suited for measuring the spa-
tial location of gesture processing in the brain, the
primary cognitive neuroscience technique used to
measure the neural timing of gesture processing
is electroencephalography (EEG). EEG measures
electrical activity that is emitted from postsyn-
aptic discharges of large groups of neurons, and
these electrical signals pass through the skull and
are recorded by electrodes on the scalp (Allison,
Wood, & McCarthy, 1986). This electrical activity
varies according to its frequency (number of cycles
per second) and its amplitude (positive and negative
microvolts). Although there is interesting research
looking at how the frequency and amplitude of EEG
signals correlate with different neurocognitive states
(e.g., alertness, drowsiness, sleep), the majority of
research on gesture processing focuses on a segment
of the EEG signal called an ERP.
ERPs are portions of the ongoing EEG signal
that are time-locked to a stimulus. The idea of time-
locking is to present different conditions of stimuli
(e.g., speech having two different relationships to
gesture) many times and then to average a person’s
516
EEG response to the onsets of those different types
of stimuli. This time-locked averaging reduces the
signal-to-noise ratio and produces an averaged brain
response that shows how the brain differentiates
(or does not differentiate) stimuli. The result is a
brainwave with many peaks and valleys of elec-
trical activity. These peaks and valleys are called
ERP components, and depending on when (on
the order of milliseconds) and, to a lesser extent,
where (e.g., left or right side of the scalp) they dif-
fer, researchers are able to make inferences into
what neuropsychological processes were used to
distinguish the stimuli. As a general rule of thumb,
­components that occur early in a brainwave reflect
lower level sensory, perceptual, and early attentional
processes, whereas components that occur later
reflect higher level semantic, memory, and meta-
cognitive processes (Key, Dove, & Maguire, 2005).
Next, we explore how researchers have used ERPs
to measure gesture processing addressing the same
questions outlined in the previous section.
Does gesture affect speech processing?  As
described in the previous section, techniques such
as fMRI are well suited for uncovering spatial mech-
anism of gesture processing. However, understand-
ing where gestures are processed in the brain is only
part of the puzzle in understanding how the brain
processes gesture. For example, knowing when
gestures connect with speech during language com-
prehension can give us insights into what it means
to say that gesture and speech form an integrated
system (McNeill, 1992).
In the first study to use ERPs to measure the
timing of gesture processing, Kelly, Kravitz, and
Hopkins (2004) presented gestures that had differ-
ent relations with the accompanying speech (e.g.,
gesturing either short, tall, or wide while saying the
word “short”) and measured ERPs to the spoken
words. They found that incongruent gesture–speech
pairs (saying “short” and gesturing tall) produced
a larger negativity at roughly 400 ms post stimu-
lus at bilateral temporal regions than congruent
gesture–speech pairs (saying “short” and gesturing
short). This N400 effect indexes difficulties with
semantic integration (the larger the N400, the more
difficult it is to process the meaning of the word),
 Measuring Gesture

so this result provided insights into the stage—a measurements of timing, and together with spatial
distinctly semantic stage, which fits nicely with neuroimaging techniques, such as fMRI, ERPs can
McNeill’s (1992) theory—at which the brain inte- help researchers map out the network of when and
grates gesture and speech. Interestingly, the N400 where different brain regions communicate with one
effect for incongruent gestures is no different than another to process gesture and speech.
incongruent speech (Özyürek, Willems, Kita, &
Does gesture affect neural processes involved in
Hagoort, 2007), suggesting that the two modalities
learning?  One of the advantages of ERPs is that
are treated as equals in the brain’s comprehension of
they are very sensitive and can capture processes
language—a claim corroborated through the use of
that are often missed by more overt behavioral mea-
fMRI by Willems et al. (2007), which we discussed
sures. Because of this sensitivity, they can identify
in the previous section.
more precise mechanisms underlying behavioral
The result described earlier is interesting because
change. This makes them very well suited for
if the effect had been earlier in the brainwave, it
measuring outcomes of learning (Key, Molfese, &
would suggest that the integration occurred at per-
Ratajczak, 2006; Osterhout et al., 2008).
ceptual stages of processing rather than at semantic
Despite these clear advantages of ERPs, very few
stages. More recent research has demonstrated that,
researchers have used them to explore how gestures,
indeed, some gestures do affect the brain’s percep-
specifically, affect different learning outcomes. In
tual intake of gesture. In a recent study by Biau and
one of the few published studies investigating this,
Soto-Faraco (2013), participants watched and lis-
Kelly, McDevitt, and Esch (2009) explored how
tened to ongoing discourse in which beat gestures
instructing native English-speakers in foreign vocab-
occasionally emphasized certain words while ERPs
ulary words (Japanese) affected memory for the
were recorded to the speech. They found that beats
meaning of those words. The behavioral results were
produced very early perceptual differences (as early
clear: People remembered many more words—up to
as 100 ms after the onset of a spoken word) in how
1 week later—when they learned the foreign words
the brain processed speech compared to when beats
with versus without iconic gestures (e.g., the teacher
did not accompany speech.
made a drinking gesture while translating the Japa-
There is also evidence that gesture affects late
nese word for drink, Nomu). In addition, ERPs were
stages of speech processing. For example, Holle
recorded after instruction during a recognition task,
et al. (2012) presented syntactically ambiguous
and the brainwaves showed that words learned with
sentences to participants and found that when beat
gesture produced a larger late positive complex than
gestures accompanied the ambiguous words, there
words learned without gesture. The late positive
was a reduced P600 component compared to when
complex is a component that peaks around 600 ms
there were no beat gestures. The P600 is thought
post stimulus in bilateral parietal scalp regions and
to reflect syntactic processing—with a reduction
is thought to reflect, among other things, depth of
reflecting ease of processing (Osterhout & Hol-
imagistic memory encoding (Rugg & Curran, 2007).
comb, 1992)—so this finding shows that gestures
Thus, the reason why people remember more words
facilitate language comprehension even after per-
learned with gesture is that gesture strengthens
ceptual and semantic stages of processing.
iconic memory traces of the meaning of those words
Taken together, this sampling of studies suggests
in the brain. Understanding this “imagistic encod-
that gestures affect speech processing at multiple
ing” mechanism would not be possible from gather-
stages. Beat gestures influence early perception
ing behavioral data alone.
(within 200 ms) and late syntactic processing (after
600 ms) of spoken words, and iconic gestures influ-
CONCLUSION, FUTURE DIRECTIONS, AND
ence the intermediary semantic processing of words
QUALIFICATIONS
(around 400 ms) once they have been perceived and
just prior to their syntactic sequencing in the brain. Gesture research is continually expanding not only
Behavioral methods do not allow for such precise in the sheer volume of research but also in the

517
Church, Kelly, and Wakefield
tools used to measure it. We describe a few of these
innovations. This chapter ends raising a philosophi-
cal as well as a practical question about studying
gesture—is gesture a behavior that can be appropri-
ately described and measured?
Neuroimaging Innovation
Does gesture offer a window into the brain’s readi-
ness to learn? One exciting application of neu-
roimaging techniques—because of their extreme
sensitivity—is that they can be used to predict
future behavior and learning outcomes that may
not be apparent from observable behavior. For
example, Molfese (2000) measured auditory ERPs
in newborns and identified early perceptual com-
ponents (N1, P2, N2) that differentiated—with
85% accuracy—children who would later develop
reading difficulties. Similarly, it is possible that ges-
ture researchers could use ERPs to identify “neural
profiles” of children revealing who may be more or
less open to instruction with and without gesture.
This sort of neural marker would be particularly
useful for very young children (as in Molfese’s,
2000, study) or children who have developmental
disorders (e.g., pervasive developmental disorder
or autism) that make traditional behavioral assess-
ments difficult.
Whereas ERPs may be used in the future to
determine a temporal neural profile for children who
may be more or less open to learning through the
use of gesture, fMRI could be used to ask how the
network recruited in the brain changes at different
stages in its “readiness to learn.” There is already
evidence that adults process gesture differently if it
represents the same information as speech versus
complementary information (e.g., “She has a pet”
[gesture indicates that this pet is a bird] vs. “She has
a bird” [gesture again represents a bird]; Dick et al.,
2014). Specifically, there are regions of the brain,
including the IFG and posterior middle temporal
gyrus (pMTG), that show greater activation when
speech and gesture information must be integrated
for a message to be understood. Wakefield, James,
and James (2013) showed that children recruit the
pMTG to a lesser extent than adults and argued
that this may represent a difference in children and
518
adult’s ability to integrate information in speech and
gesture. If children who show a behavioral readi-
ness to learn also show greater activation in the
pMTG and other regions identified by Dick et al.
(2014) when processing mismatches, compared to
their peers, this would suggest that the mechanism
that leads to learning has to do with the ability to
integrate gestural and speech information. This is
simply one potential use of fMRI to help researchers
understand why children producing certain types of
gestures may respond differently to instruction.
Human Avatars and Other
Technical Innovations
Another interesting future direction in the measure-
ment of gesture is the use of human avatars that
gesture along with speech when providing instruc-
tion on video (Cassell, 2000; Kopp & Wachsmuth,
2004). Creating the gesture repertoire of a human
avatar could not have been done without the
development of gesture coding schemes. One such
software-based system, GALE (Gesturing Avatar for
Learning and Education), has been incorporated in
research activities at The University of Wisconsin
by Martha Alibali and colleagues. Using a human
avatar that gestures can provide researchers with
control over a number of human features that typi-
cally correlate with gesture during communication
(such as facial expression, eye gaze, intonation, etc.)
so the influence of gesture can be measured inde-
pendently from these other features.
In another exciting new trend, computer tech-
nologies designed for the consumer have created
ways to capture human movement through points
of light (e.g., rAndom International; PSFK Labs,
2014), which can then be projected in three-
dimensional space. These technologies are created
for entertainment purposes. However, this type of
technology could be interfaced with software cod-
ing programs such as ELAN to capture and code
gestures.
What Makes a Gesture a Gesture?
(No Gesture Is an Island)
The conceptualization of nonverbal behaviors in
this handbook (for obvious organizational reasons)

focuses on different aspects of the human body (eyes,
voice, face, hands, etc.) as separate from one another.
Thus, one might assume that these different parts
of the body have dedicated and mutually exclusive
purposes. However, it is not the case that these dif-
ferent aspects of the human body neatly map onto
distinct types of nonverbal functions. For example,
all of the types of bodily behaviors that we laid out
in the beginning of the chapter (emotion, regulating,
and language-associated nonverbal behaviors) could
at different moments and in different contexts have
many overlapping functions. Take hand gestures and
facial expressions as an example. A cospeech gesture,
such as pointing, could be used to regulate turn-
taking in an interaction, but this same function might
also be accomplished by a smile and a head nod. Or
consider that cospeech gesture can be just as effective
as facial expressions at communicating about emo-
tion, as in the case of a hand metaphorically depicting
the ups and downs of a rollercoaster while verbally
expressing the emotional ups and downs of an event.
Moreover, these behaviors can be layered on top of
one another to create quite nuanced meaning. Take,
for example, the depiction of a drinking gesture com-
bined with a stylized grimace while saying “He had
a late night.” Note how the meaning of that message
would change if the facial expression was a smile
instead of a grimace.
The take home message of these examples is that
in most cases, there is not a one-to-one correspon-
dence between a bodily action and its meaning. We
are flexible in the way we communicate through
our bodies, and we often draw on layered bodily
communication to express ideas. Just as one idea
can be expressed through multiple body move-
ments, we can also express multiple ideas through
Measuring Gesture
one movement. It is the communicative context
in which we produce a body movement that will
determine the function of nonverbal behavior,
as opposed to some physical characteristic of the
nonverbal behavior itself. Although this chapter
has focused primarily on the hand movements we
call gestures, the broader picture is that gesture is
just one way that humans communicate. In reality,
the way to reference (e.g., through gesture, facial
expressions, tone) is not what is important; what is
important is the fact that human beings reference
using whatever is affordable in the moment.
Can Gesture be Measured?
Should Gesture be Measured?
According to Kendon (2004), “no attempt should
be made to develop a single, unified classification
scheme [of gesture], since so many different dimen-
sions of comparison are possible . . . [and] gesture
cannot be pinned down into a typology in any fixed
way” (p. 84). Kendon and others (e.g., Duncan,
2013) make an interesting argument—indeed,
­gesture is so rich that it could be a mistake to cor-
ral all its diversity in an attempt to categorize it in
a homogenous and uniform way. The key is to find
different ways to measure gesture in a reliable fash-
ion without constraining its richness. In the present
review, we have attempted to highlight this delicate
balance by showing that the questions researchers
have about gesture ultimately guide the precise ways
that they attempt to classify, describe, and measure
it. Hopefully, we have shown that the ability to mea-
sure gestures is crucial for the experimental study of
gesture, which, in turn, is necessary for addressing
fundamental questions about human thinking and
behavior.
519
Church, Kelly, and Wakefield

APPENDIX 19.1 American Sign Language Alphabet

Note. In the public domain.

APPENDIX 19.2 Description of Gesture and Speech ELAN Annotations

Note. This is a screen shot from the ELAN coding program. ELAN software is open-source freeware and can be downloaded at http://www.cs.ru.nl/elan.

520

References
Alibali, M. W., & Nathan, M. J. (2012). Embodiment
in mathematics teaching and learning: Evidence
from learners’ and teachers’ gestures. Journal of the
Learning Sciences, 21, 247–286. http://dx.doi.org/
10.1080/10508406.2011.611446
Allison, T., Wood, C. C., & McCarthy, G. M. (1986).
The central nervous system. In M. G. H. Coles,
E. Donchin, & S. W. Porges (Eds.), Psychophysiology:
Systems, processes, and applications (pp. 5–25).
New York, NY: Guilford Press.
Amaro, E., Jr., & Barker, G. J. (2006). Study design in fMRI:
Basic principles. Brain and Cognition, 60, 220–232.
http://dx.doi.org/10.1016/j.bandc.2005.11.009
Bavelas, J. B., Chovil, N., Coates, L., & Roe, L. (1995).
Gestures specialized for dialogue. Personality and
Social Psychology Bulletin, 21, 394–405. http://dx.doi.
org/10.1177/0146167295214010
Bavelas, J. B., Chovil, N., Lawrie, D. A., & Wade, A.
(1992). Interactive gestures. Discourse Processes,
15, 469–489. http://dx.doi.org/10.1080/
01638539209544823
Beilock, S. L., & Goldin-Meadow, S. (2010). Gesture
changes thought by grounding it in action.
Psychological Science, 21, 1605–1610. http://dx.doi.
org/10.1177/0956797610385353
Biau, E., & Soto-Faraco, S. (2013). Beat gestures
modulate auditory integration in speech perception.
Brain and Language, 124, 143–152. http://dx.doi.
org/10.1016/j.bandl.2012.10.008
Cassell, J. (Ed.). (2000). Embodied conversational agents.
Cambridge, MA: MIT Press.
Chieffi, S., & Ricci, M. (2007). Are beats gestures unrelated
to meaning of verbal production? Comment on
Mizuguchi’s view. Perceptual and Motor Skills, 105,
556–558.
Church, R. B. (1999). Using gesture and speech to capture
transitions in learning. Cognitive Development,
14, 313–342. http://dx.doi.org/10.1016/S0885-
2014(99)00007-6
Church, R. B., Garber, P., & Rogalsky, K. (2007). The role of
gesture in memory and social communication. Gesture,
7, 137–158. http://dx.doi.org/10.1075/gest.7.2.02bre
Church, R. B., & Goldin-Meadow, S. (1986). The
mismatch between gesture and speech as an index of
transitional knowledge. Cognition, 23, 43–71. http://
dx.doi.org/10.1016/0010-0277(86)90053-3
Church, R. B., Kelly, S., & Holcombe, D. (2014).
Temporal synchrony between speech, action and
gesture during language production. Language and
Cognitive Processes, 29, 345–354. http://dx.doi.org/
10.1080/01690965.2013.857783
Church, R. B., Kelly, S. D., & Lynch, K. (2000).
Immediate memory for mismatched speech and
Measuring Gesture
representational gesture across development. Journal
of Nonverbal Behavior, 24, 151–174. http://dx.doi.
org/10.1023/A:1006610013873
Coulson, M. (2004). Attributing emotion to static
body postures: Recognition accuracy, confusions,
and viewpoint dependence. Journal of Nonverbal
Behavior, 28, 117–139. http://dx.doi.org/10.1023/
B:JONB.0000023655.25550.be
Crais, E. R., Watson, L. R., & Baranek, G. T. (2009).
Use of gesture development in profiling children’s
prelinguistic communication skills. American Journal
of Speech-Language Pathology, 18, 95–108. http://
dx.doi.org/10.1044/1058-0360(2008/07-0041)
Darwin, C. (1872). The expression of the emotions in man
and animals. http://dx.doi.org/10.1037/10001-000
Dick, A. S., Mok, E. H., Raja Beharelle, A., Goldin-
Meadow, S., & Small, S. L. (2014). Frontal and
temporal contributions to understanding the
iconic co-speech gestures that accompany speech.
Human Brain Mapping, 35, 900–917. http://dx.doi.
org/10.1002/hbm.22222
Duncan, S. (2013). Transcribing gesture with speech.
In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig,
D. McNeill, & S. Teßendorf (Eds.), Body—Language—
Communication (HSK 38.1) (pp. 1003–1010). Berlin,
Germany: de Gruyter Mouton.
Duncan, S., Rohlfing, K., Bielefeld, X., & Loehr, D. (2013).
Multimodal annotation tools. In C. Müller, A. Cienki,
E. Fricke, S. H. Ladewig, D. McNeill, & S. Teßendorf
(Eds.), Body—Language—Communication
(HSK 38.1) (pp. 1011–1017). Berlin, Germany: de
Gruyter Mouton.
Ekman, P., & Friesen, W. V. (1969). The repertoire of
nonverbal behavior: Categories, origins, usage and
coding. Semiotica, 1, 49–98.
Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial
Action Coding System: The manual [CD ROM]. Salt
Lake City, UT: Research Nexus.
Evans, J. W., Todd, R. M., Taylor, M. J., & Strother,
S. C. (2010). Group specific optimisation of
fMRI processing steps for child and adult data.
NeuroImage, 50, 479–490. http://dx.doi.org/10.1016/
j.neuroimage.2009.11.039
Feldman, H., Goldin-Meadow, S., & Gleitman, L. (1978).
Beyond Herodotus: The creation of a language by
linguistically deprived deaf children. In A. Lock (Ed.),
Action, symbol, and gesture: The emergence of language
(pp. 351–414). New York, NY: Academic Press.
Goldin-Meadow, S. (1997). When gestures and words
speak differently. Current Directions in Psychological
Science, 6, 138–143. http://dx.doi.org/10.1111/1467-
8721.ep10772905
Goldin-Meadow, S. (2005). Hearing gesture: How our
hands help us think. Cambridge, England: Harvard
University Press.
521
Church, Kelly, and Wakefield

Goldin-Meadow, S., & Feldman, H. (1977). The Kendon, A. (1972). Some relationships between body
development of language-like communication motion and speech. In A. W. Siegman & B. Pope
without a language model. Science, 197, 401–403. (Eds.), Studies in dyadic communication (pp. 177–210).
http://dx.doi.org/10.1126/science.877567 http://dx.doi.org/10.1016/B978-0-08-015867-
Goldin-Meadow, S., & Mylander, C. (1984). Gestural 9.50013-7
communication in deaf children: The effects and Kendon, A. (2004). Gesture: Visible action as utterance.
noneffects of parental input on early language Cambridge, England: Cambridge University Press.
development. Monographs of the Society for Research
Key, A. P. F., Dove, G. O., & Maguire, M. J. (2005).
in Child Development, 49, 1–151.
Linking brainwaves to the brain: An ERP primer.
http://dx.doi.org/10.2307/1165838
Developmental Neuropsychology, 27, 183–215. http://
Green, A., Straube, B., Weis, S., Jansen, A., Willmes, K., dx.doi.org/10.1207/s15326942dn2702_1
Konrad, K., & Kircher, T. (2009). Neural integration
Key, A. P. F., Molfese, D. L., & Ratajczak, E. D. (2006).
of iconic and unrelated coverbal gestures: A
ERP indicators of learning in adults. Developmental
functional MRI study. Human Brain Mapping, 30,
Neuropsychology, 29, 379–395. http://dx.doi.
3309–3324. http://dx.doi.org/10.1002/hbm.20753
org/10.1207/s15326942dn2902_5
Gullberg, M., & Holmqvist, K. (2006). What speakers
Kipp, M. (2012). Multimedia annotation, querying
do and what addressees look at: Visual attention to
and analysis in ANVIL. In M. T. Maybury (Ed.),
gestures in human interaction live and on video.
Multimedia information extraction (pp. 351–367).
Pragmatics and Cognition, 14, 53–82. http://dx.doi.
http://dx.doi.org/10.1002/9781118219546.ch21
org/10.1075/pc.14.1.05gul
Kita, S., & Özyürek, A. (2003). What does cross-
Heeger, D. J., & Ress, D. (2002). What does fMRI tell us
linguistic variation in semantic coordination of
about neuronal activity? Nature Reviews Neuroscience,
speech and gesture reveal? Evidence for an interface
3, 142–151. http://dx.doi.org/10.1038/nrn730
representation of spatial thinking and speaking.
Holle, H., Obermeier, C., Schmidt-Kassow, M., Friederici, Journal of Memory and Language, 48, 16–32.
A. D., Ward, J., & Gunter, T. C. (2012). Gesture http://dx.doi.org/10.1016/S0749-596X(02)00505-3
facilitates the syntactic analysis of speech. Frontiers
Kopp, S., & Wachsmuth, I. (2004). Synthesizing
in Psychology, 3, 74. http://dx.doi.org/10.3389/
multimodal utterances for conversational agents.
fpsyg.2012.00074
Computer Animation and Virtual Worlds, 15, 39–52.
Hostetter, A. B., & Alibali, M. W. (2008). Visible http://dx.doi.org/10.1002/cav.6
embodiment: Gestures as simulated action.
Krauss, R. M., Morrel-Samuels, P., & Colasante,
Psychonomic Bulletin and Review, 15, 495–514. http://
C. (1991). Do conversational hand gestures
dx.doi.org/10.3758/PBR.15.3.495
communicate? Journal of Personality and Social
Izard, C. E. (1994). Innate and universal facial expressions: Psychology, 61, 743–754. http://dx.doi.org/
Evidence from developmental and cross-cultural 10.1037/0022-3514.61.5.743
research. Psychological Bulletin, 115, 288–299. http://
Logothetis, N. K. (2003). The underpinnings of the
dx.doi.org/10.1037/0033-2909.115.2.288
BOLD functional magnetic resonance imaging signal.
Kelly, S. D., & Church, R. B. (1997). Can children Journal of Neuroscience, 23, 3963–3971.
detect conceptual information conveyed through
Logothetis, N. K., Pauls, J., Augath, M., Trinath,
other children’s nonverbal behaviors? Cognition and
T., & Oeltermann, A. (2001). Neurophysiological
Instruction, 15, 107–134. http://dx.doi.org/10.1207/
investigation of the basis of the fMRI signal. Nature,
s1532690xci1501_4
412, 150–157. http://dx.doi.org/10.1038/35084005
Kelly, S. D., & Church, R. B. (1998). A comparison between
Logothetis, N. K., & Wandell, B. A. (2004). Interpreting
children’s and adults’ ability to detect conceptual
the BOLD signal. Annual Review of Physiology,
information conveyed through representational
66, 735–769. http://dx.doi.org/10.1146/annurev.
gestures. Child Development, 69, 85–93. http://dx.doi.
physiol.66.082602.092845
org/10.1111/j.1467-8624.1998.tb06135.x
Macedonia, M., Müller, K., & Friederici, A. D. (2011).
Kelly, S. D., Kravitz, C., & Hopkins, M. (2004).
The impact of iconic gestures on foreign language
Neural correlates of bimodal speech and gesture
word learning and its neural substrate. Human Brain
comprehension. Brain and Language, 89, 253–260.
Mapping, 32, 982–998. http://dx.doi.org/10.1002/
http://dx.doi.org/10.1016/S0093-934X(03)00335-3
hbm.21084
Kelly, S. D., McDevitt, T., & Esch, M. (2009). Brief
training with co-speech gesture lends a hand to McNeill, D. (1992). Hand and mind: What gestures reveal
word learning in a foreign language. Language and about thought. Chicago, IL: University of Chicago Press.
Cognitive Processes, 24, 313–334. http://dx.doi. McNeill, D. (2005). Gesture and thought. Chicago, IL:
org/10.1080/01690960802365567 University of Chicago Press.

522

McNeill, D. (2012). How language began: Gesture and
speech in human evolution. http://dx.doi.org/10.1017/
CBO9781139108669
Mehrabian, A., & Ferris, S. R. (1967). Inference of
attitudes from nonverbal communication in two
channels. Journal of Consulting Psychology, 31,
248–252. http://dx.doi.org/10.1037/h0024648
Molfese, D. L. (2000). Predicting dyslexia at 8 years
of age using neonatal brain responses. Brain and
Language, 72, 238–245. http://dx.doi.org/10.1006/
brln.2000.2287
Osterhout, L., & Holcomb, P. J. (1992). Event-related
brain potentials elicited by syntactic anomaly. Journal
of Memory and Language, 31, 785–806. http://dx.doi.
org/10.1016/0749-596X(92)90039-Z
Osterhout, L., Poliakov, A., Inoue, K., McLaughlin, J.,
Valentine, G., Pitkanen, I., . . . Hirschensohn, J.
(2008). Second-language learning and changes in
the brain. Journal of Neurolinguistics, 21, 509–521.
http://dx.doi.org/10.1016/j.jneuroling.2008.01.001
Özyürek, A., Kita, S., & Allen, S. (2001). Tomato Man
movies: Stimulus kit designed to elicit manner, path and
causal constructions in motion events with regard to
speech and gestures. Nijmegen, the Netherlands: Max
Planck Institute for Psycholinguistics, Language and
Cognition Group.
Özyürek, A., Willems, R. M., Kita, S., & Hagoort, P.
(2007). On-line integration of semantic information
from speech and gesture: Insights from event-
related brain potentials. Journal of Cognitive
Neuroscience, 19, 605–616. http://dx.doi.org/10.1162/
jocn.2007.19.4.605
Perry, M., Church, R. B., & Goldin-Meadow, S. (1988).
Transitional knowledge in the acquisition of
concepts. Cognitive Development, 3, 359–400. http://
dx.doi.org/10.1016/0885-2014(88)90021-4
PSFK Labs. (2014). The future of retail: Trends to act
on. Retrieved from http://69.59.162.218/NRF/
BIGShow/2014/Future%20of%20Retail%202014_1.pdf
Rugg, M. D., & Curran, T. (2007). Event-related
potentials and recognition memory. Trends in
Cognitive Sciences, 11, 251–257. http://dx.doi.
org/10.1016/j.tics.2007.04.004
Measuring Gesture
Skipper, J. I., Goldin-Meadow, S., Nusbaum, H. C., &
Small, S. L. (2007). Speech-associated gestures,
Broca’s area, and the human mirror system. Brain and
Language, 101, 260–277. http://dx.doi.org/10.1016/
j.bandl.2007.02.008
Stam, G. (2006). Thinking about speaking and motion:
L1 and L2 gesture. International Review of Applied
Linguistics, 44, 145–171. http://dx.doi.org/10.1515/
IRAL.2006.006
Stokoe, W. C. (1960). Sign language structure: An outline
of the visual communication systems of the American
deaf (Studies in Linguistics: Occasional Papers,
No. 8). Buffalo, NY: University of Buffalo,
Department of Anthropology and Linguistics.
Straube, B., Green, A., Weis, S., Chatterjee, A., & Kircher,
T. (2009). Memory effects of speech and gesture
binding: Cortical and hippocampal activation in
relation to subsequent memory performance. Journal
of Cognitive Neuroscience, 21, 821–836. http://dx.doi.
org/10.1162/jocn.2009.21053
Villarreal, M., Fridman, E. A., Amengual, A., Falasco, G.,
Gerscovich, E. R., Ulloa, E. R., & Leiguarda, R. C.
(2008). The neural substrate of gesture recognition.
Neuropsychologia, 46, 2371–2382. http://dx.doi.
org/10.1016/j.neuropsychologia.2008.03.004
Wakefield, E. M., James, T. W., & James, K. H. (2013).
Neural correlates of gesture processing across human
development. Cognitive Neuropsychology, 30, 58–76.
http://dx.doi.org/10.1080/02643294.2013.794777
Willems, R. M., Özyürek, A., & Hagoort, P. (2007). When
language meets action: The neural integration of
gesture and speech. Cerebral Cortex, 17, pp. 2322–
2333. doi: 10.1093/cercor/bhl141
Wittenburg, P., Brugman, H., Russel, A., Klassmann,
A., & Sloetjes, H. (2006). ELAN: A professional
framework for multimodality research. In
Proceedings of the Fourth Language Resources and
Evaluation Conference (pp. 1556–1559). Retrieved
from http://www.lrec-conf.org/proceedings/lrec2006/
pdf/153_pdf
Woods, D. K., & Dempster, P. G. (2011). Tales from the
bleeding edge: The qualitative analysis of complex
video data using Transana. FQS Forum: Qualitative
Social Research, 12, 17.
523
 Chapter 20
Measuring Eye Behavior
Frank M. Marchak
As described in Chapter 13 of this handbook, eye
behavior and gaze afford a valuable source of informa-
tion regarding nonverbal communication. The pur-
pose of this chapter is to provide an overview of the
theory and practice of measuring eye behavior, includ-
ing eye movements, blinks, and changes in pupil
diameter. It begins with a description of the anatomy
of the eye and reasons why we move our eyes, as well
as a discussion of the different types of eye move-
ments. This is followed by an overview of eye-tracking
technologies and methods, and factors for consider-
ation in selecting a system for a particular applica-
tion. The types of ocular measures—and the forms of
information each can provide—serve to introduce the
data analysis approaches available and their relevance
to answering specific questions. Finally, a description
of current advances in eye-tracking technology and
future directions are discussed.
BACKGROUND
Eye movements are a complex interaction that
results from both the anatomy of the human eye and
the need for detailed visual information. The sec-
tions that follow provide information on the struc-
ture of the eye as well as the types and functions of
eye movements.
Eye Anatomy
Before discussing the details of measuring eye
behavior, we need to know why we move our eyes.
The answer can be found, in part, in the structure of
the eye, illustrated in Figure 20.1.
http://dx.doi.org/10.1037/14669-020
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
For purposes of this chapter, the eye compo-
nents of interest are the cornea, pupil, lens, retina,
and fovea. The basic process of vision involves light
reflected from objects in the environment passing
through the cornea, pupil, and lens and forming an
image on the retina at the location of the fovea.
The cornea is the transparent tissue on the front
of the eye and is responsible for approximately 80%
of its focusing power (E. Goldstein, 2013). However,
because the cornea is fixed, it is unable to adjust its
focus. The remaining 20% of the focusing power of
the eye is supplied by the lens. The lens changes its
shape through the ciliary muscles, which alter the
curvature of the lens and hence its power to focus.
The pupil, the aperture in the iris that controls
the amount of light admitted to the eye, is made up
of two antagonistic smooth muscles. The sphincter
muscle constricts the iris and is under parasympa-
thetic nervous system control, whereas the dilator
muscle, which opens the iris, is under sympathetic
nervous system control (Beatty & Lucero-
Wagoner, 2000).
The retina is made up of visual receptors, rods,
and cones, which contain light-sensitive chemicals
that transform light into neuronal signals. Cones
function in daylight conditions and provide the
perception of color. Rods function under low-level
illumination and provide only shades of gray.
A further difference between the two photoreceptors
is the level of visual acuity they provide, with cones
affording a higher level of detail (Boyton, 1979).
The fovea is an area on the retina approximately
0.25 mm in diameter that contains only cones
525
Frank M. Marchak
FIGURE 20.1.  Basic human eye anatomy. From How Do
Your Eyes Work? by the National Eye Institute. Retrieved
from http://nei.nih.gov/health/healthyeyes/. In the public
domain.
(see Figure 20.2). It lies at the point of focus of the
optical system of the eye and provides the ability to
see fine detail. The periphery of the retina is com-
posed primarily of rods and some cones.
This background neuroanatomy helps explain
why we move our eyes. Because only a small por-
tion of the retina is capable of providing a high
level of visual acuity, to focus attention on an item
of interest, it is necessary to direct the fovea to
that location. Eye movements permit the selection,
fixation, and tracking of objects and scenes in the
environment.
FIGURE 20.2.  Distribution of rods and cones on the retina. From Foundations of Vision (p. 46), by B. A. Wandell,
2014. Copyright 2014 by Brian A. Wandell. Reprinted with permission.
526
Eye Movements
Each eye has three pairs of extraocular muscles:
superior and inferior rectus, superior and inferior
oblique, and the lateral and medial rectus (Oyster,
1999). The function of these muscles is shown in
Figure 20.3. The extraocular muscles control the
two major types of eye movements: gaze stabiliza-
tion and gaze shifting. For the purposes of this
chapter, gaze shifting eye movements are of primary
interest. There are three types of gaze shifting eye
movements. Vergence eye movements are disjunctive
movements of the eyes that change the angle of axis
so that images of a single object are aligned on the
fovea of each eye. This involves moving the eyes in
opposite directions so that they converge on object
of interest. Smooth pursuit eye movements keep a
moving target on the fovea and are under voluntary
control, such as following a bird flying across the
sky. Saccadic eye movements are ballistic move-
ments that abruptly change the point of fixation,
such as when viewing an image or reading, and are
most relevant to the current application.
Saccadic eye movements are characterized by a
series of fixations, when the eye is stationary, fol-
lowed by rapid movements—saccades—that reposi-
tion the fovea to a different location in the visual
field. Saccades are very fast and can range in dura-
tion from 10 ms to 100 ms, which is the response
time of the visual system to perceive an object
(Leigh & Zee, 2006). Because fixations are overt
indicators of the point of visual attention, tracking

FIGURE 20.3.  Eye muscles and functions. From
Neuroscience (p. 429), by D. Purves et al., 2001,
Sunderland, MA: Sinauer Associates. Copyright 2001
by Sinauer Associates. Reprinted with permission.
their locations and the paths between them can
provide insight into the underlying processing of
the observer. The goal of eye-tracking systems is to
accurately and reliably establish the position of the
eyes, usually in relation to the external environment,
for use in determining the relevance of those move-
ments under particular conditions. The next section
describes approaches to measuring eye behavior.
EYE-MOVEMENT RECORDING
Recording eye movements can be accomplished
through a variety of methods and configurations,
depending on the specific needs of the question to be
answered. Each of these arrangements involves trade-
offs, in terms of both ease of use and data fidelity.
The details of these techniques and considerations of
each are discussed in the sections that follow.
Eye-Tracking Techniques
The systematic study of eye movements has been an
area of research interest since at least the mid-1800s.
In recent times, the major approaches have included
scleral search coils, electro-oculography, video-
oculography, and pupil–corneal reflection methods.
Because the vast majority of eye-tracking systems in
use today employ some form of video-based, pupil-
corneal reflection methods, the discussion focuses on
this approach. Wade and Tatler (2005) have provided
a thorough overview of the origins and history of eye-
movement measurement methods and research.
Pupil–corneal reflection eye tracking involves illu-
minating the eye with infrared light—to avoid natural
Measuring Eye Behavior
light reflections—and determining the center of the
pupil and the reflection from the front of the cornea
(see Figure 20.4A). The infrared light is reflected
from both the front and back of the cornea as well
as the front and back of the lens (see Figure 20.4B),
known as Purkinje images, with the first Purkinje
image being the brightest (Holmqvist et al., 2011).
The details of determining the center of the pupil
differ among eye-tracking manufacturers, but most
employ some method of ellipse-fitting to the pupil
and measurement of the longest axis. Ellipses are used
to find the center of the pupil because interference
from the eyelids and eyelashes can prevent detecting a
true circle, and they provide a more robust and accu-
rate method for determining the pupil’s center of mass
(Lin, Craig, Dean, Klette, & Klette, 2003).
There are two methods of illuminating the pupil
depending on the configuration of the cameras and
infrared illuminators. Bright-pupil tracking involves
locating the illuminators coaxially, or parallel, with
the camera, producing an effect similar to that seen
when a camera flash causes “red eye.” Dark-pupil
(A)
1st
2nd
3rd
4th
(B)
FIGURE 20.4.  (A) Pupil center and corneal reflection.
(B) Purkinje images.
527
Frank M. Marchak
tracking locates the illuminators off-axis from the
cameras and produces a dark image of the pupil
(see Figure 20.5). Some systems employ both light-
and dark-pupil methods, switching between the two
to determine which provides the best tracking per-
formance. There is no consensus regarding which of
the two approaches provides the best data, as each
technique’s ability to detect the pupil is affected by
different factors, including pupil size and ethnicity.
Bright pupil techniques have been shown to work
well with Hispanic and Caucasian individuals
(Tobii Technology, 2010). Dark pupil systems per-
form better when the pupil is small such as when
outdoors under conditions of bright ambient illu-
mination, for older participants who tend to have
smaller pupils (Holmqvist et al., 2011), as well as
for Asian populations (Tobii Technology, 2010).
Configurations
Eye trackers come in a variety of configurations, the
utility of which is a function of the particular appli-
cation. Chin rest systems involve the participant
placing his or her chin in a device that keeps the
head stable, and they provide some of the highest
degrees of accuracy. They are useful in situations
where stimulus presentation is fixed, such as on a
computer monitor. However, they impose a level of
intrusiveness and are not suitable for use in situa-
tions such as real-world interactions. For example,
such systems are best suited for circumstances in
which stimuli are presented in a fixed location and
no interaction is required outside the specific task.
FIGURE 20.5.  Dark pupil (A) and bright pupil (B) infrared images. Corneal
reflection is shown in (C).
528
Head-mounted and eyeglass-mounted eye trackers
have cameras and illuminators affixed to either a
device worn over the head, including hats, or incor-
porated into a pair of nonmagnifying eyeglasses.
Many also include a scene camera that provides
images of what the participant is viewing, which can
then be correlated with the eye-movement behavior.
These configurations can permit freedom of move-
ment if part of a portable system, and they permit
capturing interactions across a variety of situations.
Such systems allow the participant to interact freely
with the environment, engaging in tasks such as
shopping, playing sports, or driving. The resulting
data can provide overlays of gaze patterns on the
video of the viewed scene and permit examination
of interaction in a dynamic environment. However,
because it is not possible to differentiate between
smooth pursuit eye movements and saccadic eye
movements in this situation, such systems cannot
provide fixation information but measures based on
raw gaze data. This means that it is possible to cal-
culate measures such as gaze duration or dwell time
but not number of fixations.
Desk-mounted systems allow remotely tracking
eye movements with no attachments to the partici-
pant over a somewhat wide but limited area. Most
systems are tolerant of reasonable degrees of move-
ment but still require that the participant remain
in a fixed location. These systems can be used with
a computer monitor to present stimuli or can be
used alone to track viewing of two-dimensional,
real-world surfaces or scenes. One factor common
 Measuring Eye Behavior

to all remote eye-tracking systems is differentiating

movement of the eyes from movement of the head.
Some systems employ hardware devices that track

High
the head separately and update the eye-tracking data
accordingly by controlling for head movements and

Precision
separating them from movements of the eyes. Most
deal with the issue through the use of multiple
cameras and algorithms that compensate for the
head movement.

Low
System Factors
Independent of the specific eye tracking technique Low High
Accuracy
and configuration, elements of the data collection
capabilities determine the level of data fidelity and
the types of analyses that can be performed. Defini- FIGURE 20.6.  Comparison of accuracy (validity) and
precision (reliability).
tions of the relevant system factors are provided in
the sections that follow, along with relevant aspects
of system performance. their particular location, precision can suffice. For
example, certain measures such as saccade length
Validity and reliability.  As with any type of spe- and pupil diameter are often of interest in terms of
cialized data collection system, a researcher must their relative change. Therefore, when one is using
consider how well the device performs in terms pupil diameter as a measure of arousal, the relative
of the validity and reliability of its measures, and change in the pupil—the ability to characterize this
a researcher must also consider the granularity change correctly over multiple measurements
of measurement so that the device can address (precision)—is more important than the absolute
the requirements of the specific question under size of the pupil (accuracy).
investigation.
System latency.  Other important factors in eye-
The first factors involved in tracking gaze are
tracking data collection are system latency, defined
the accuracy and spatial precision of the system.
as the average end-to-end delay from an actual
Accuracy is defined as the average angular offset
movement of the tracked eye until the recording
(in degrees of visual angle) between the locations
computer signals that a movement has occurred
of fixations and the corresponding locations of the
(COGAIN, 2014), and temporal precision, which is
fixation targets, whereas spatial precision is the root-
the standard deviation of the system latency. These
mean-square of the angular distance (in degrees of
factors are important because low temporal preci-
visual angle) between successive samples (Com-
sion results in variability in the latency, making it
munication by Gaze Interaction [COGAIN], 2014).
difficult to draw firm conclusions regarding the
Essentially, accuracy refers to how well the eye
relationships between events and eye location. That
tracker is showing the location of the gaze in rela-
is, if the system has a large degree of variability in
tion to where the person is actually looking, which
its measurement of the time at which the eye has
in social science is often referred to as the validity
moved (temporal precision), then it will affect the
of the measure. In contrast, precision refers to the
ability to determine the location of eye and on what
reliability of multiple measures of viewing the same
it is fixating because there will be variable delay
location, which in social science means the same
between when the eye moves and when the system
thing as reliability of the measure. The relationship
records it (system latency).
between these two elements is shown in Figure 20.6.
Both are important, but if the interest is primarily Sampling rate.  A final factor for consideration is
in the consistency of measurements irrespective of the system sampling rate, which is how many times

529
Frank M. Marchak
per second the eye tracker measures the person’s
gaze location (measured in hertz). There is a wide
range of sampling rates available from different eye-
tracking systems, ranging from 30 Hz to 2000 Hz.
However, faster is not necessarily better, as it
depends on the experimental purpose and the types
of measures to be calculated. If the intent is only
to measure where a participant is looking, a lower
speed eye tracker will suffice. In addition, very high-
speed sampling rates can limit the degree to which
the participant must be constrained, as any head or
body movement will result in artifacts, such as loss
of tracking and dropped data samples. If the intent is
to measure when an event, such as a fixation, began
and its duration, a higher sampling rate will provide
greater precision. The majority of current systems
sample in the 30 Hz to 250 Hz range, which is more
than adequate for most studies related to nonverbal
communication, as it balances the danger of too
many artifacts with the precision to capture the
actual movement of eyes.
Calibration
Video-based, pupil–corneal eye trackers determine
eye position by calculating the change in the angle
between the center of the pupil and the corneal
reflection (Hammoud, 2008). To map the location
of the eyes to the location being viewed, some sort
of calibration procedure is required. This typically
involves having the participant fixate on a number of
known locations in the environment while the sys-
tem determines the specific pupil–corneal reflection
relationship for each location. This information is
then used to extrapolate to other locations. Although
procedures differ somewhat among manufacturers,
most provide software that conducts the calibration
and provides output of its accuracy. Points that are
not adequately captured can be repeated until an
adequate level of accuracy is achieved. Several fac-
tors can contribute to inaccuracies in eye tracking
and are discussed in the next section.
Other Considerations
Although enormous strides have been made in
recent years in the accuracy, reliability, and usability
of eye trackers, certain factors can affect performance
due primarily to the nature of the camera optics and
530
the ability of the systems to robustly track across
a variety of situations. The use of mascara by a
participant can greatly decrease tracking accuracy,
sometimes to the point of preventing the ability to
track at all. Mascara tends to absorb the infrared sig-
nal, scattering the light, which results in numerous
reflections that prevent the eye-tracking algorithms
from identifying the corneal reflection. In general,
it is preferable to require participants to remove eye
makeup before participating.
Long eyelashes and drooping eyelids can also
cause difficulty in tracking. Because the systems
need to have a clear view of the pupil, if it is
occluded by eyelashes or the eyelid, then its perfor-
mance will be degraded because it will lose track
of the pupil and drop data collection points. One
possible solution is to vary the position of the eye-
tracking cameras to view the eyes at a steeper angle
that avoids the occlusion, but this ability varies
across systems.
Contact lenses seem to not affect the ability to
track the eye movements, although on occasion air
bubbles between soft contact lenses and the eyeball
can cause difficulties, as under infrared illumination
they appear as multiple corneal reflections. These
can then confuse the system (does not know which
to focus upon), thus disrupting accuracy (Holmqvist
et al., 2011). This problem can often be remedied by
slightly unfocusing the camera, which would tend
to cause the bubbles to fade before the actual focus
spot, thus bringing that spot into relief.
Certain types of eyeglasses, such as bifocals
and trifocals, can affect the ability to eye track.
The solid border between the lens sections results
in numerous reflections of the infrared light and,
much like the situation with mascara, prevents the
eye-tracking system from locating a single corneal
reflection in the eye image. Progressive lens glasses
do not present this problem, as there are no physi-
cal discontinuities between magnification areas of
the lenses. The eyeglass frames for the most part do
not pose a problem, as they are normally outside the
area that the algorithms use to find the pupil and
corneal reflection.
A final consideration is the ambient illumina-
tion in the area. Direct sunlight is composed of
infrared light at an intensity far greater than that

of the eye-tracking illuminators. Although this is
often more of a problem in outdoor situations, such
as driving, it can also occur from sunlight shining
through a window. Thus, experimenters should be
careful in how they construct the ambient lighting
in a room, as natural sunlight can disrupt the track-
ing of infrared-based systems. Most indoor lighting,
such as overhead fluorescent light fixtures, does not
adversely affect eye-tracking performance. However,
care should be taken with bright incandescent light
sources in the field of view of the participant, as
these can result in decreased pupil diameter as well
as spurious changes in pupil diameter resulting from
pupil contraction if the participant fixates on the
light source.
OCULAR MEASURES
The preceding section described the basic func-
tioning of eye-tracking systems and factors that
can affect their accuracy and performance. When
operating properly, eye-tracking systems essentially
measure three parameters: x and y eye location in
the two-dimensional plane in front of the observer
and pupil diameter. Although these are the only
aspects that are directly measured, numerous other
measures may be derived from these data. Hol-
mqvist et al. (2011) have described more than 120
measures that can be calculated from eye-movement
data. They also provided a method for conceptual-
izing and creating new measures. Clearly a detailed
examination of all these measures is beyond the
scope of the current work, but a subset of those
most relevant to issues related to nonverbal com-
munication is reviewed. These measures are divided
into measures of eye movements, pupil diameter,
and blinks. (For a detailed exposition on eye-
tracking experimental design, conduct, and analysis,
see Holmqvist et al., 2011.)
Eye-Movement Measures
For the current purposes, the basic eye-movement-
measuring elements can be divided into fixations
and saccades. Fixations refer to those times when
the eye stops moving and focuses on a given loca-
tion, whereas saccades refer to the movement of
the eyes between fixations. A fixation is a relatively
Measuring Eye Behavior
stable eye-in-head position defined by a spatial
dispersion of less than 2°, a minimal duration of
100–200 ms, and a threshold velocity—that is, the
speed at which the movement of the eye is consid-
ered a saccade—of less than 15°–100° per ms
(Ciuffreda & Tannen, 1995).
Most eye-tracking systems software provides
output of fixations based on three general methods
(Salvucci & Goldberg, 2000). Velocity-based
methods use point-to-point velocities over some
threshold to separate fixations from saccades. This
is important because the eye is never completely sta-
tionary and is characterized by miniature eye move-
ments of less than 1° in size, including flicks, drifts,
and high-frequency tremors. As described earlier,
the fovea of the eye is directed to the point of inter-
est. These miniature eye movements shift the image
a small amount across the fovea so that the fovea
is constantly stimulated. This is necessary because
the visual receptors fatigue with stimulation, and
if the image did not shift it would fade (Steinman,
Haddad, Skavenski, & Wyman, 1973). Because the
eye is constantly in motion, velocity-based methods
characterize low-velocity movements as fixations
and high-velocity movements as saccades.
Dispersion-based methods are based on groups
of consecutive points within a maximum separa-
tion. The assumption here is that fixations are rep-
resented by points that are near one another. This
approach measures the distance between successive
points and groups all points that are within some
criterion distance apart as a fixation.
Temporal-based methods use duration informa-
tion, informed by the fact that fixations are seldom
less than 100 ms and are primarily in the range of
200–400 ms (Holmqvist et al., 2011). This approach
relies solely on time analysis of the data to differen-
tiate fixations and saccades.
The selection of a fixation calculation method
is usually constrained by the particular software
used by the eye-tracking manufacturer. In terms
of robustness, all three approaches are roughly
equivalent. In terms of accuracy, velocity-based
and dispersion-based methods perform best
(Salvucci & Goldberg, 2000).
Saccades are involuntary, abrupt, very rapid
movements of both eyes simultaneously in changing
531
Frank M. Marchak
the point of fixation. They are characterized by a
duration of 30–80 ms, a spatial extent of 4°–20°,
and a velocity of 30°–500° per second (Holmqvist
et al., 2011). Several methods are used to calculate
saccades, including those using fixed or adaptive-
threshold models as described earlier as well as the
eye-movement acceleration signal (Behrens, MacKe-
ben, & Schröder-Preikschat, 2010). In general, most
applications focus on the detection of fixations using
the methods mentioned earlier and define saccades
as the periods between fixations.
Pupil-Diameter Measures
Most pupil–corneal reflection eye-tracking systems
provide a measure of pupil diameter because it is
necessary to calculate the center of the pupil to per-
form eye tracking. As with fixations and saccades,
each manufacturer uses a slightly different method
to carry out this calculation, the details of which
are usually proprietary. Most systems use head
position to make a correction to account for dis-
tance changes between the participant and the cam-
era, because as the participant gets closer to the eye
tracker, the size of the pupil increases. Further, the
output of pupil diameter can be provided in a vari-
ety of units, from number of pixels, which cover
the area defined as the pupil, to millimeters to
meters, which take into account the distance of the
pupil from the eye tracker and provide an absolute
measure of diameter. It is important to be certain of
the measurement scale before engaging in further
analyses. The preferred reporting is always in milli-
meters of either absolute diameter or change in the
diameter. Reporting of percentage change often is
distorting if groups or conditions began with differ-
ent initial/baseline diameters. For example, if there
is a large age difference among participants —older
participants tend to have smaller pupil sizes—or
differences in ambient illumination, percentage
change measurements would not give an accurate
characterization of any effects. Further, report-
ing of pixels provides no replicable information,
because pixel size differs depending on the specif-
ics of the camera and the viewing distance; it works
for analyses within a participant but not necessar-
ily across participants and never across different
researchers.
532
In addition, because the eye-tracking software
represents real numbers with 8-bit precision regard-
less of the capabilities of the sensor, pupil diameter is
often reported to six or seven decimal places, despite
the fact the actual precision of the systems is never
at that level (S. Steinhauer, personal communica-
tion, September 14, 2010). Restricting interpretation
of pupil diameter precision to two decimal places
is usually a prudent approach, as most eye-tracking
manufacturer technical specifications indicate that
degree best reflects the precision of their sensors.
Blink Measures
Blinks are inferred from the eye-movement data as
samples in which the x/y position of the eye goes to
zero. Because the eye is closed, the eye tracker can-
not calculate position, so this lack of information
(represented as a zero in most systems) is used to
determine blinks. As with the eye-movement and
pupil measures, most eye-tracking software systems
can provide blink data. The specifics of calculating
differ slightly among manufactures but, in general,
are based on a succession of zero-values in the data
stream. However, care must be taken to ensure that
the measure truly indicates a blink and not just a
data-sampling error or signal loss, both of which
will report an x/y location of 0. For example, if
because of system latency issues the eye tracker fails
to sample the data in a timely fashion, or if the par-
ticipant moves out of the range of the eye tracker,
the result would be a measured value of zero but
would not indicate a blink.
EYE-BEHAVIOR DATA ANALYSIS
The previous section outlined the basic ocular
measures that can be calculated from eye-tracking
data. This section provides an overview of analysis
techniques that can be used to analyze eye behavior
once you measure them. As before, these techniques
are divided into analyses of eye movements, pupil
diameter, and blinks (see also Chapter 13, this
handbook).
Eye-Movement Analyses
According to Goldberg and Kotval (1999), eye-
movement data can provide accurate insights into

human cognitive activity. These eye-movement
data can be analyzed into three types of measures:
measures of cognitive processing, measures of men-
tal search, and measures of scanpaths (which are a
measure of the patterns of examining a fixed object).
Measures of cognitive processing include factors
such as number of fixations, locations of fixations,
fixation duration, cumulative fixation time, and
cluster analysis of areas of interest. Each of these
measures have been shown to provide information
regarding cognitive processes, not just in nonverbal
communication but in areas such as reading and
problem solving (Underwood, 2005). For example,
it has been found that consistent and universal
eye-movement patterns are found that reflect maxi-
mizing information intake by regulating where the
eyes move when reading (Vitu, 2011). Others (e.g.,
Grant & Spivey, 2003) have shown that perceptual
properties of objects guide eye movements and
attention that aid in problem solving and reasoning.
Measures of search can include number of sac-
cades, saccade amplitude, saccade velocity, and sac-
cade length. These characteristics can lend insight
into factors such as what elements are attended,
what processes are used in examining elements, and
which features contribute to the detection of the
search item (Findlay & Gilchrist, 1998). Schmid,
Mast, Bombari, Mast, and Lobmaier (2011) exam-
ined the effect of mood on information processing
styles in an emotion recognition task. Using the
interfeatural saccade ratio—comparing the num-
ber of interfeatural saccades to the total number of
saccades—they determined that participants in a
happy mood processed information more globally,
as higher interfeatural saccade ratios are indicative
of the use of more information integration in overall
impression formation. Bannerman and colleagues
(Bannerman, Milders, de Gelder, & Sahraie, 2009;
Bannerman, Milders, & Sahraie, 2010) found faster
saccadic orienting to both fearful body and face
stimuli, indicating faster localization of threat by the
oculomotor system. Similar findings of differences
in saccade latency have been shown for differences
in emotional content of facial stimuli (Bannerman,
Hibbard, Chalmers, & Sahraie, 2012).
Scanpaths refer to the sequence of the visual pat-
tern employed when viewing a particular stimulus.
Measuring Eye Behavior
In its strongest form, Noton and Stark (1971) sug-
gested that when a visual pattern is viewed, a par-
ticular sequence of eye movements is executed, and
this sequence is important in accessing the visual
memory for that pattern. Although their theory is
disputed, the methodology provides evidence that
viewers attend to informative details and not neces-
sarily the entire stimulus (Ellis & Stark, 1986).
There are at least two methods of quantifying
scanpaths. One involves modeling patterns as Mar-
kovian processes, that is, a stochastic process where
any transition between states depends only on the
current state (Stark & Ellis, 1981). By examin-
ing one- and two-step transitions, it is possible to
describe the statistical constraint in a viewing pat-
tern. For example, Althoff and Cohen (1999) used
transitions as a measure of prior knowledge of faces.
They defined regions on the face around each eye,
the nose, the mouth, and all other areas. They then
calculated the probabilities of transitioning between
any two areas—say, left eye to right eye, nose to
mouth—as well as the probabilities of transitioning
across three areas—for example, left eye to nose to
right eye. These probabilities were used to construct
transition matrices that characterized the probability
of making one-step or two-step transitions among
the defined regions from which the eye leaves to the
region into which the eye lands. If the scanpaths
were random, the probabilities in a matrix would all
be equal, because each region would be as likely to
be fixated as the next.
Departures from a completely random pattern
of scanning indicate a level of constraint in how an
image is viewed. Following Ellis and Stark (1986),
it is possible to calculate the conditional informa-
tion in the matrices as entropy, where the higher
the entropy, the higher the degree of randomness.
The entropy measure serves to describe informa-
tion in that it characterizes the degree of association
between a fixation and the fixation that preceded it.
As the entropy increases, more information is being
transmitted, and as it decreases, the scanning pat-
tern is less constrained.
Another quantification method is based on Lev-
enshtein distance, a metric for measuring the amount
of difference between two sequences (Yujian & Bo,
2007). This approach is defined as the minimum
533
Frank M. Marchak
number of edits needed to transform one string into
the other, with allowable edit operations being inser-
tion, deletion, or substitution of a single element.
If fixation locations are treated as the individual ele-
ments, it is possible to compare different viewing
sequences. Josephson and Holmes (2008) used such
an approach to study eye-path sequences to exam-
ine cross-race recognition deficit or own-race bias
in cross-racial eyewitness identification. Each of six
photos in a lineup array was designated as an area
of interest, and the scanpath for each participant’s
viewing was characterized by a sequence of fixations.
For example, a single fixation of suspect image “B”
followed by four fixations of suspect “D” then two
fixations of suspect “E” would be characterized by
the sequence “BDDDDEE.” Applying the Levensh-
tein distance metric, it was possible to compare the
sequences using optimal matching analysis and to
generate a matrix of distances for every possible pair
of sequences for each stimulus. Similar sequences
will have smaller distances. In this way, it was pos-
sible to examine differences in viewing pattern of
own-race versus cross-race conditions.
Pupil-Diameter Analyses
The pupil is unique in that it is innervated by the
autonomic (Steinhauer, Siegle, Condray, & Pless,
2004) and central nervous (Koss, 1986) systems.
Thus, changes in pupil diameter can be related
to both cognitive workload as well as emotional
arousal. The basic analysis of pupil data involves
determining changes in diameter in relation to an
external stimulus or event. Observing change over
time in relation to multiple events can provide
insight into underlying processes such as attentional
effort and emotional arousal. For example, Kahne-
man and Beatty (1966) found a dilation of the pupil
with the presentation of each additional digit in a
short-term memory task. The degree of dilation was
a function of the number of digits presented and
thus the degree of attentional effort. Further, Brad-
ley, Miccoli, Escrig, and Lang (2008) showed an
increase in pupil diameter during affective picture
viewing indicating an increase in emotional arousal.
One caution in collecting pupil data is that the
pupillary reflex is sensitive to changes in stimu-
lus and background illumination, as its primary
534
function is to regulate the amount of light that
enters the eye (Lowenstein & Lowenfeld, 1962).
As discussed earlier, when illumination is constant,
pupil size varies systematically in relation to psycho-
logical factors such as mental effort and emotional
arousal. Care must be taken during data collection
to ensure any differences are due to the underly-
ing psychological processing as opposed to lighting
effects. For example, visual stimuli presented on a
computer screen must be controlled for equal lumi-
nance, and ambient illumination must be constant
with no bright point sources of light.
Blink Analyses
Analysis of eye blink data has shown its useful-
ness in a variety of domains, including determining
workload, as those engaged in a higher cognitive
workload blinked more (Stern & Skelly, 1984); pro-
viding information during psychiatric interviews,
as blink rates were affected differently by different
topics (Kanfer, 1960); and as a measure of deceptive
intent (Marchak, 2013). For example, in a series of
studies with increasing ecological validity, Marchak
(2010a, 2010b, 2010c, 2011, 2013) examined
changes in ocular measures when participants with
true and false intent answered questions related to a
mock crime, an unrelated crime, and neutral issues.
Those with false intent showed a suppression of blink-
ing during the relevant questions about the mock
crime when compared to the 10-s period after the end
of the questions. They also showed a lower number of
blinks and shorter maximum blink duration.
Analyses of eye blink data include blink onset,
duration, latency, rate, and number. As with pupil-
diameter data, each of these measures is calculated
in relation to the onset of a stimulus or event. For
example, reduction in blink rate is greater in more
demanding tasks (Drew, 1951); duration and fre-
quency of eye blinks are sensitive to the require-
ments of monitoring specific sensory channels, with
visual presentation exhibiting fewer and briefer
closures than auditory presentation (R. Goldstein,
Walrath, & Stern, 1982); and latency serves as a
sensitive measure of the efficiency of response pro-
cessing (Stern, Walrath, & Goldstein, 1984). Taken
together, analysis of blink parameters can provide
insight into underlying cognitive processes.

General Notes on Analyses
Although the majority of eye-tracking systems pro-
vide software that automates data analysis, atten-
tion to the parameters employed by the algorithms
and the need to screen data are still important. As
described earlier, the primary fixation calculation
algorithms use either velocity- or dispersion-based
calculation methods. Regardless of the method, it is
important that the type of algorithm and settings are
kept constant for all participants in a given study.
In terms of settings, by increasing the dispersion
threshold or increasing the maximum velocity, the
total number of fixations and the average fixation
duration will increase. This is the result of fixations
that would be otherwise identified as separate and
individual being combined into a single larger fixa-
tion. The most important factor is to ensure consis-
tency in settings if results across different studies are
to be compared.
A second important consideration involves
adequate screening of the data. Data loss can occur
for a variety of reasons, including poor calibration,
excessive participant movement, or movement of
the eye-tracking system. It is desirable to set a cri-
terion of acceptable missing data, which is usually
between 10% and 30% (Holmqvist et al., 2011).
Similarly, elimination of outliers that are more than
three standard deviations on either side of the mean
is also desirable. Lastly, any data that may have been
the result of extraneous circumstances, such as the
participant sneezing during data collection, should
also be removed. In general, although advances in
software have helped eliminate many of the issues
involved in data analysis, it is still important to be
familiar with the details of your data.
ADVANCES IN EYE-TRACKING
TECHNOLOGY AND FUTURE DIRECTIONS
As described earlier, eye-tracking methods can pro-
vide a number of ocular measures that can be used
to characterize a wide variety of aspects of nonverbal
behavior. The technology has changed rapidly over
the past decade, resulting in systems that are more
robust under different environments and applica-
tions as well as being far easier to use. Many early
eye-tracking systems required making a dental
Measuring Eye Behavior
impression of each participant that was used in a
bite bar to stabilize the head. Even the more flex-
ible systems required at least a chin rest. Advances
in camera technology and software algorithms now
support the ability to allow a larger degree of partici-
pant head movement. Usability of the data collection
and analysis software has also improved, removing
the need for a dedicated technician and permitting
users with general computer experience the abil-
ity to collect accurate data and perform complex
analyses.
Manufacturers are focusing on increasing the
performance of system response in terms of accu-
racy (validity) and precision (reliability of measure-
ment) as well as accommodating a wider variety of
participant types, such as those with droopy eyelids
or eye makeup. Further, remote eye-tracking sys-
tems now support data collection from children and
infants by having a wider field of view and simplify-
ing the calibration process.
Other approaches are examining the ability to
gather gaze data in terms of number of viewers
from a large number of participants simultaneously
and with no calibration (e.g., using the Eyebox2).
While this system does not provide eye position
data or pupil diameter, it permits determining when
and how many individuals are viewing a particular
target over a period of time. For example, such a
system can be installed above a sign or billboard
in a public location and can provide information
on the number of individuals who viewed the
advertisement.
Although collecting measures of eye behav-
ior have become easier, more work is needed in
expanding the off-the-shelf analysis software that
accompanies most systems, as this is the process by
which the measures become the data with which
scientists test their hypotheses. Eye tracking has
moved beyond its original research roots and has
found its way into a wide range of application areas,
such as advertising, marketing, and usability testing.
In advertising and marketing, eye tracking is used
to test everything from the location of products on
store shelves to different product label designs to the
effectiveness of websites in generating sales. These
domains often only require basic analysis meth-
ods, such as the order in which items are viewed
535
Frank M. Marchak
and which features first attract attention; therefore,
researchers in traditional areas are required to
develop their own methods for more involved types
of data analysis, such as scanpath analysis, pupillary
measures, and blink analysis. The Open Source Soft-
ware movement has provided many resources for
obtaining access to advanced analysis methods, but
these resources still require a level of programming
knowledge on the part of the user to employ them.
For example, ILAB (Gitelman, 2002), which is soft-
ware containing a set of routines for performing
eye-movement analysis, provides advanced analysis
capabilities not available in the commercial software
distributed by most eye-tracker manufacturers, but
it requires knowledge of MATLAB software (Math-
Works, n.d.) to employ its capabilities.
Measuring eye behavior offers a rich source of
information across a variety of nonverbal commu-
nication situations that can provide insight into the
cognitive and emotional states of mind of individu-
als. Advances in technology have made the process
noninvasive and straightforward for the practitioner.
The applications are limited only by the ability to
develop innovative experimental designs and by
scientists’ own comprehension of the processes that
underlie these measures.
References
Althoff, R. R., & Cohen, N. J. (1999). Eye-movement-
based memory effect: A reprocessing effect in face
perception. Journal of Experimental Psychology:
Learning, Memory, and Cognition, 25, 997–1010.
http://dx.doi.org/10.1037/0278-7393.25.4.997
Bannerman, R. L., Hibbard, P. B., Chalmers, K., &
Sahraie, A. (2012). Saccadic latency is modulated by
emotional content of spatially filtered face stimuli.
Emotion, 12, 1384–1392. http://dx.doi.org/10.1037/
a0028677
Bannerman, R. L., Milders, M., de Gelder, B., & Sahraie,
A. (2009). Orienting to threat: Faster localization
of fearful facial expressions and body postures
revealed by saccadic eye movements. Proceedings of
the Royal Society of London B: Biological Sciences, 276,
1635–1641. http://dx.doi.org/10.1098/rspb.2008.1744
Bannerman, R. L., Milders, M., & Sahraie, A. (2010).
Attentional cueing: Fearful body postures capture
attention with saccades. Journal of Vision, 10, 23.
http://dx.doi.org/10.1167/10.5.23
Beatty, J., & Lucero-Wagoner, B. (2000). The pupillary
system. In J. T. Cacioppo, L. G. Tassinary, &
536
G. G. Berntson (Eds.), Handbook of psychophysiology
(2nd ed., pp. 142–162). Cambridge, England:
Cambridge University Press.
Behrens, F., MacKeben, M., & Schröder-Preikschat, W.
(2010). An improved algorithm for automatic detection
of saccades in eye movement data and for calculating
saccade parameters. Behavior Research Methods, 42,
701–708. http://dx.doi.org/10.3758/BRM.42.3.701
Boyton, R. M. (1979). Human color vision. New York, NY:
Holt, Rinehart & Winston.
Bradley, M. M., Miccoli, L., Escrig, M. A., & Lang, P. J.
(2008). The pupil as a measure of emotional
arousal and autonomic activation. Psychophysiology,
45, 602–607. http://dx.doi.org/10.1111/j.1469-
8986.2008.00654.x
Ciuffreda, K., & Tannen, B. (1995). Eye movement basics
for the clinician. Maryland Heights, MO: Mosby.
Communication by Gaze Interaction. (2014). Definition of
terms. Retrieved from http://www.cogain.org/system/
files/Definition%20of%20terms.pdf
Drew, G. E. (1951). Variations in reflex blink-rate during
visual-motor tasks. Quarterly Journal of Experimental
Psychology, 3, 73–88. http://dx.doi.org/10.1080/
17470215108416776
Ellis, S. R., & Stark, L. (1986). Statistical dependency in
visual scanning. Human Factors, 28, 421–438.
Findlay, J. M., & Gilchrist, I. D. (1998). Eye guidance and
visual search. In G. Underwood (Ed.), Eye guidance
in reading and scene perception (pp. 295–312). http://
dx.doi.org/10.1016/B978-008043361-5/50014-6
Gitelman, D. R. (2002). ILAB: A program for
postexperimental eye movement analysis. Behavior
Research Methods, Instruments, and Computers, 34,
605–612. http://dx.doi.org/10.3758/BF03195488
Goldberg, J. H., & Kotval, X. P. (1999). Computer
interface evaluation using eye movements: Methods
and constructs. International Journal of Industrial
Ergonomics, 24, 631–645. http://dx.doi.org/10.1016/
S0169-8141(98)00068-7
Goldstein, E. (2013). Sensation and perception. Belmont,
CA: Wadsworth/Cengage Learning.
Goldstein, R., Walrath, L. E., & Stern, J. A. (1982). Blink
activity as a function of stimulus modality and schedule
of presentation in a discrimination task. St. Louis,
MO: Washington University Behavior Research
Laboratory.
Grant, E. R., & Spivey, M. J. (2003). Eye movements and
problem solving: Guiding attention guides thought.
Psychological Science, 14, 462–466. http://dx.doi.
org/10.1111/1467-9280.02454
Hammoud, R. I. (Ed.). (2008). Passive eye monitoring:
Algorithms, applications and experiments. http://
dx.doi.org/10.1007/978-3-540-75412-1

Holmqvist, K., Nystrom, A., Andersson, R., Dewhurst, R.,
Jarodzka, H., & Van De Weijer, J. (2011). Eye
tracking: A comprehensive guide to methods and
measures. Oxford, England: Oxford University Press.
Josephson, S., & Holmes, M. E. (2008, March). Cross-race
recognition deficit and visual attention: Do they all look
(at faces) alike? Paper presented at the Eye Tracking
Research and Application Symposium, Savannah, GA.
Kahneman, D., & Beatty, J. (1966). Pupil diameter and
load on memory. Science, 154, 1583–1585. http://
dx.doi.org/10.1126/science.154.3756.1583
Kanfer, F. H. (1960). Verbal rate, eyeblink, and content
in structured psychiatric interviews. Journal of
Abnormal and Social Psychology, 61, 341–347. http://
dx.doi.org/10.1037/h0038933
Koss, M. C. (1986). Pupillary dilation as an index of
central nervous system α2-adrenoceptor activation.
Journal of Pharmacological Methods, 15, 1–19. http://
dx.doi.org/10.1016/0160-5402(86)90002-1
Leigh, R. J., & Zee, D. S. (2006). The neurology of
eye movements (4th ed.). New York, NY: Oxford
University Press.
Lin, X., Craig, J., Dean, S., Klette, G., & Klette, R. (2003).
Accurately measuring the size of the pupil of the eye.
Auckland, New Zealand: University of Auckland,
CITR Tamaki and Medical School.
Lowenstein, O., & Lowenfeld, I. (1962). The pupil.
In H. Davson (Ed.), The eye: Vol. 3. Muscular mechanisms
(pp. 256–329). New York, NY: Academic Press.
Marchak, F. M. (2010a). Task 1 report. OBSERVER:
Ocular-Based Screening Evaluation and Real-Time
Verification Examination Remotely. Bozeman, MT:
Veridical Research and Design.
Marchak, F. M. (2010b). Task 2 report. OBSERVER:
Ocular-Based Screening Evaluation and Real-Time
Verification Examination Remotely. Bozeman, MT:
Veridical Research and Design.
Marchak, F. M. (2010c). Task 3 report. OBSERVER:
Ocular-Based Screening Evaluation and Real-Time
Verification Examination Remotely. Bozeman, MT:
Veridical Research and Design.
Marchak, F. M. (2011). Task 4 report. OBSERVER: Ocular-
Based Screening Evaluation and Real-Time Verification
Examination Remotely. Bozeman, MT: Veridical
Research and Design.
Marchak, F. M. (2013). Detecting false intent using eye
blink measures. Frontiers in Psychology, 4, 736.
http://dx.doi.org/10.3389/fpsyg.2013.00736
MathWorks. (n.d.). MATLAB [Computer software].
Retrieved from http://www.mathworks.com/
products/matlab
Noton, D., & Stark, L. (1971). Scanpaths in eye
movements during pattern perception. Science,
Measuring Eye Behavior
171, 308–311. http://dx.doi.org/10.1126/science.
171.3968.308
Oyster, C. W. (1999). The human eye. Sunderland, MA:
Sinauer Associates.
Purves, D., Augustine, G. J., Fitzpatrick, D., Katz, L. C.,
LaMantia, A.-S., McNamara, J. O., & Williams, S. M.
(2001). Neuroscience. Sunderland, MA: Sinauer
Associates.
Salvucci, D. D., & Goldberg, J. H. (2000, November).
Identifying fixations and saccades in eye-tracking
protocols. Paper presented at the Eye Tracking
Research and Application Symposium, Palm Beach
Gardens, FL.
Schmid, P. C., Mast, M. S., Bombari, D., Mast, F. W., &
Lobmaier, J. S. (2011). How mood states affect
information processing during facial emotion
recognition: An eye tracking study. Swiss Journal
of Psychology, 70, 223–231. http://dx.doi.org/
10.1024/1421-0185/a000060
Stark, L. W., & Ellis, S. R. (1981). Scanpaths revisited:
Cognitive models direct active looking. In D. F.
Fisher, R. A. Monty, & J. W. Senders (Eds.), Eye
movements: Cognition and visual perception
(pp. 193–226). Hillsdale, NJ: Erlbaum.
Steinhauer, S. R., Siegle, G. J., Condray, R., & Pless,
M. (2004). Sympathetic and parasympathetic
innervation of pupillary dilation during sustained
processing. International Journal of Psychophysiology,
52, 77–86. http://dx.doi.org/10.1016/j.ijpsycho.
2003.12.005
Steinman, R. M., Haddad, G. M., Skavenski, A. A., &
Wyman, D. (1973). Miniature eye movement.
Science, 181, 810–819. http://dx.doi.org/10.1126/
science.181.4102.810
Stern, J. A., & Skelly, J. J. (1984, October). The eye blink
and workload considerations. Paper presented at the
Human Factors and Ergonomics Society 28th Annual
Meeting, Orlando, FL.
Stern, J. A., Walrath, L. C., & Goldstein, R. (1984).
The endogenous eyeblink. Psychophysiology, 21,
22–33. http://dx.doi.org/10.1111/j.1469-8986.1984.
tb02312.x
Tobii Technology. (2010). An introduction to eye tracking
and Tobii Eye Trackers. Retrieved from http://www.
tobii.com/en/eye-tracking-research/global/library/
white-papers/tobii-eye-tracking-white-paper
Underwood, G. (Ed.). (2005). Cognitive processes in
eye guidance. http://dx.doi.org/10.1093/acprof:
oso/9780198566816.001.0001
Vitu, F. (2011). On the role of visual and oculomotor
processes in reading. In S. P. Liversedge, I. D.
Gilchrist, & S. Everling (Eds.), The Oxford handbook
of eye movements (pp. 731–749). http://dx.doi.
org/10.1093/oxfordhb/9780199539789.013.0040
537
Frank M. Marchak

Wade, N. J., & Tatler, B. W. (2005). The moving tablet of Yujian, L., & Bo, L. (2007). A normalized Levenshtein
the eye: The origins of modern eye movement research. distance metric. IEEE Transactions on Pattern
http://dx.doi.org/10.1093/acprof:oso/97801985 Analysis and Machine Intelligence, 29, 1091–1095.
66175.001.0001
Wandell, B. A. (2014). Foundations of vision. Retrieved
from https://foundationsofvision.stanford.edu/
chapter-3-the-photoreceptor-mosaic

538
 Chapter 21
Methods in Olfactory Research
Robin Freyberg, Patricia Wilson, and Jeannette Haviland-Jones
As readers of a handbook of nonverbal communica-
tion, an immediate question most likely comes to
mind; that is, “why is there a chapter about research
methods in olfaction within a foundation text in
nonverbal behavior?” After all, about a decade ago,
nonverbal research in emotion concentrated on
facial and body movement; olfaction or olfactics did
not enter into the picture. There are good reviews of
methods in olfaction but not specifically for olfactics
or nonverbal behavior using the olfactory system
(e.g., for a classic review of olfaction technology, see
Cain, Cometto-Muñiz, & de Wijk, 1992). The scien-
tific community that focused on human interaction
and behavior held that visual information had vastly
outpaced olfactory information across the evolution-
ary spectrum among humans, and therefore, smell
was not an important human sense (see Claussen,
Howes, & Synott, 1994; for a related discussion, see
also Chapter 14, this handbook).
However, this general disregard for the impor-
tance of the olfactory system is changing rapidly.
Researchers now know that more of the human
genome is dedicated to olfaction than to any other
sensory system (Buck & Axel, 1991). Additionally,
new analyses of the relative merits of different con-
tributors to olfactory acuity and detection are giving
rise to new hypotheses and new approaches because
the olfactory system is one of the earliest to develop,
and it may be the most integrated and complex of
the sensory systems in brain development. The chal-
lenge now is to discover what this means for human
social behavior, and in our case, what it means for
olfactics—nonverbal communication using airborne
http://dx.doi.org/10.1037/14669-021
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
compounds—which includes odors, scents, and
semiochemicals (for more detailed definitions, see
Chapter 14, this handbook). In other words, social
behavioral research in olfaction is an area ripe for
innovation with numerous unanswered questions
of particular relevance to those studying nonverbal
behavior.
For the purposes of this review, we examine
odor research from two perspectives. First, we
describe how researchers work with odors and
the variables that must be considered to choose,
prepare, and administer the various stimuli. Sec-
ond, we describe various methods for determining
whether the odor stimuli maintain psychological
and physiological effects. Lastly, we conclude with
some ideas for future research directions and a dis-
cussion of how the psychological impact of odors is
an emerging area for researchers to answer a num-
ber of essential questions that remain unexplored.
THE STIMULI
In the sections that follow, we discuss how research-
ers work with odors, including how to choose an
odor for research, determine the appropriate inten-
sity, select the delivery method, and use human
body odors in research.
Choosing an Appropriate Stimulus
Depending on the research question, researchers
will start by determining what stimulus should
be selected. In this realm, scientists could either use
human body odors (e.g., the putative pheromone
539
Freyberg, Wilson, and Haviland-Jones
androstadienone [AND]; this somewhat controver-
sial designation as a pheromone is beyond the scope
of this chapter; see Wysocki & Preti, 2004), natural
odors (e.g., plant odors), or artificial odorants
(e.g., fine fragrance). Of course, such categories may
be somewhat misleading, as commercially available
fragrances often contain natural elements as well.
One additional issue may be how to obtain the par-
ticular odorant of interest. Some researchers choose
to obtain odor samples from the fragrance producers
directly, such as Firmenich, International Flavors
and Fragrances, and Symrise. Alternatively, natural
body odors may sometimes be produced in the labo-
ratory context itself, whereas some artificial odors
can be bought directly in stores.
Practically, how might an emerging odor
researcher decide what kind of odor to choose?
Choosing a fragrance to use can be a little like
choosing an appropriate questionnaire. Introduc-
tory-level textbooks in research methods often
strongly urge researchers to choose questionnaires
that have been used in prior studies before tackling
the challenging job of constructing a novel scale
(Smith & Davis, 2013). Similar suggestions apply to
the new odor researcher, such that examining what
odors have been used previously by other research-
ers is a good first step. For example, researchers
interested in attention should consider the literature
on peppermint, as previous research has repeatedly
demonstrated that peppermint can have a profound
impact on alertness (Raudenbush, Meyer, & Eppich,
2002). Alternatively, emotion researchers may
want to consider using human body odors. Recent
research by Haviland-Jones and McGuire (1999) and
Chen, Katdare, and Lucas (2006) has shown strik-
ing mood effects when individuals were exposed to
underarm pads worn during mood induction films.
However, once a particular odor is chosen, many
questions remain regarding the particular stimulus.
Usually very little information exists regarding the
composition of various odorants and why exactly they
maintain their particular effects on human behav-
ior. It could be one identifiable specific compound
that impacts behavior or it might be the combina-
tion of multiple components that ultimately main-
tains the impact. Within fragrance development,
perfumers refer to accords in the same way that
540
musicians describe chords of music. An accord
refers to a combination of individual perfume
essences that create a new unified fragrance that
is distinct from the individual components in the
same way that a combination of notes in a chord is
distinct from the individual notes (Jellinek, 1996).
It is difficult to identify the individual fragrance ele-
ments of an accord, as the accord has created a new
harmonious fragrance; thus, it is often unclear how
the accord’s effects differ from their isolated indi-
vidual elements.
Similarly, when using human body odors, it
can be difficult to pinpoint one specific component
or perhaps multiple specific components within
body odors that affect emotion and behavior. Fur-
ther, there may be a genetic component that can
be isolated for its ultimate impact as well. Previous
research has demonstrated that mothers and infants
can identify each other through smell (Kaitz, Good,
Rokem, & Eidelman, 1987; Macfarlane, 1975).
Further, strangers can match samples of genetically
related people, such as father and son, but not of
socially related people, such as husband and wife
(Porter, Balogh, Cernoch, & Franchi, 1986). Such
findings suggest that unique genetic information
is embedded within each individual’s body odor.
In terms of fragrances, research has indicated that
genetic background contributes to fragrance selec-
tion (Milinski & Wedekind, 2001). The full extent
of the genetic contribution to olfaction is still
unclear and should be explored further.
Perhaps it is possible to think about both human
and artificial odorants as being composed of both
active and inactive ingredients. In other words,
using pharmaceuticals as an analogy, they often
contain both active ingredients that treat particular
illnesses along with inactive ingredients that do
not. Researchers currently do not know whether
odorants similarly contain active ingredients that
affect behavior and inactive ingredients that can be
detected and often co-occur but do not affect behav-
ior. The assumption made by scientists has mainly
been to consider the entirety of the particular olfac-
tory stimulus; however, it is certainly possible that
effects are due to exposure of limited number of
components. Future research should isolate par-
ticular olfactory ingredients to determine what

components specifically impact behavior. Further-
more, future researchers must be aware that what
they have measured or detected may not have been
the active ingredient.
In sum, there are many necessary considerations
when scientists decide to pursue odor research. Spe-
cifically, they need to consider what kind of olfac-
tory stimulus is appropriate based on the research
question and where it should be obtained. Further,
they need to consider whether they are examin-
ing several olfactory components or a specific iso-
lated one and how that might ultimately impact
the results. Similarly, they must consider possible
genetic contributions to the odor of interest that
may interact with the findings. Lastly, they must
consider both inactive and active ingredients in the
chosen stimulus and what their roles might be.
Determining Intensity
Once a particular stimulus has been chosen, the
question of how much stimulus to present to partici-
pants should be addressed. Some researchers choose
odors that are above conscious detection. Alterna-
tively, some have suggested that the actual effect of
fragrance generally may work best at the low or peri-
threshold (generally below the level of conscious
detection) level (Degel & Köster, 1999; Degel,
Piper, & Köster, 2001; Jellinek, 2003). For example,
Phillips and Cupchik (2004) reported that accuracy
for recalling story details was lower when fragrance
intensity was greater. Such findings suggest that
fragrance awareness may undermine specific effects.
If attention is drawn to the fragrance so that it is
identified or an attempt is made by the participant to
identify it, then associations to the semantic aspect
of its name begin to take precedence. If the perfume
is identified, correctly or incorrectly, people are
more likely to search their memories for associations
to the particular odorant. However, there may be
cases when the fragrance and its associations are
of interest and part of the research question
(e.g., Herz, Schankler, & Beland, 2004). In these
instances, studying the fragrance at higher intensi-
ties would be important.
Often when researchers are studying a variety
of odors, they will determine whether perceived
intensities or strengths are similar. In other words,
Methods in Olfactory Research
are the various odors similar in terms of how
strong or weak they are perceived by individu-
als? This intensity matching process may be done
by asking participants to report on the intensity
of particular odorants. Alternatively, fragrance
producers may determine the intensities of the
odorants in question through measurement of
chemical concentrations. However, as of now, it
is unclear what kind of difference this intensity
matching between different odors makes. It is also
unclear whether regular participants can detect
intensity differences among fragrances as well as
a trained fragrance professional. Alternatively, is
it always necessary to make sure that various fra-
grances are similar in intensity? After all, in the
real world, odorants are rarely empirically matched
for intensity. Furthermore, there are often large
individual differences in perceptions of intensity.
Most researchers use average intensity ratings
when examining differences in intensity; however,
those averages often do not reflect the wide range
of scores surrounding those average ratings. What
impact does it ultimately have if the odorants are of
differing intensities? As of now, we do not know.
Taken as a whole, researchers need to consider
a number of variables once they have chosen the
appropriate stimulus. Specifically, they must con-
sider what level of intensity is appropriate for the
research. In some cases, it might be useful to use a
stimulus at levels below conscious detection (peri-
threshold), whereas in other research designs, a
conscious approach may be preferred. Similarly,
they must consider whether it is necessary to incor-
porate some kind of intensity matching across
stimuli when multiple odors are incorporated.
Researchers should consider what the presence
or absence of similar intensities across multiple
stimuli might mean for the data that are ultimately
collected.
Method of Delivery
Once a particular odorant and its intensity have
been identified, the next challenge is to determine
how participants should be exposed to it. Broadly,
participants can be exposed to odorants directly
via the body or indirectly via the external envi-
ronment. Typically, direct bodily exposure might
541
Freyberg, Wilson, and Haviland-Jones
involve a scented stick held near the nose (Ebihara
et al., 2012), an odor-infused cotton ball taped
between the nose and upper lip (Kiecolt-Glaser
et al., 2008), an odorous patch attached on the
body (Y. Sakamoto et al., 2012), an odorant deliv-
ered via nasal cannula (or two-pronged tube placed
in the nostrils) from which fragranced air flows
(Raudenbush et al., 2002), or odors applied directly
to the participant (Freyberg & Ahren, 2011; Hum-
mer & McClintock, 2009; Schiffman, Sattely-Miller,
Suggs, & Graham, 1995).
Indirect exposure is typically done within the
external environment. No consensus currently
exists regarding the optimum method of delivery.
Researchers have used a variety of methods, includ-
ing fragrance diffusers (Guéguen & Petr, 2006;
R. Sakamoto, Minoura, Usui, Ishizuka, & Kanba,
2005; Schifferstein, Talke, & Oudshoorn, 2011),
fragrance pellets that aired the room using a fan
(Haviland-Jones, Hudson, Wilson, Freyberg, &
McGuire, 2013), carefully controlled odor chambers
(Fiedler et al., 2008), or olfactory stimuli placed in
jars near the participant (Herz, 2003). Currently,
it is unclear whether such differences in delivery
method ultimately contribute to the various psycho-
logical effects reported.
Aside from differences in type of delivery, a
more subtle difference exists in the philosophy
regarding control of the olfactory stimulus during
exposure. Some researchers have chosen delivery
systems that carefully control the exact amount of
exposure along with other potential environmental
factors (e.g., humidity), whereas other researchers
have taken a more naturalistic and less controlled
approach and have exposed participants to odorants
in methods that closely mirror real-world contexts.
Previous research has not documented how find-
ings may be different between carefully controlled
olfactory exposure and exposure that is less con-
trolled but more representative of daily experience.
However, if the experimental question is one that
involves emotion and nonverbal communication,
we are concerned about the nature of findings that
may be obtained in certain heavily controlled envi-
ronments that place participants in particularly
uncomfortable situations (e.g., nasal cannulae) that
may affect the ultimate emotions experienced or
542
the ability to communicate. Kirk-Smith and Booth
(1987) have similarly suggested that it is time to
expand research beyond the “traditional, ‘sterilized’
types of laboratory-based experimentation” (p. 164).
In conclusion, the final step involved in stimulus
choice includes issues in how the stimulus ulti-
mately will be delivered to the participant. Broadly,
the researcher should consider whether odor deliv-
ery will occur directly on the participant’s body or
through exposure in the experimental room. Lastly,
researchers need to think about the balance between
internal and external validity. Specifically, although
carefully controlled odor delivery systems may
remove a number of potential confounding variables
in the stimulus presentation, they may contribute to
confounds in the study design itself through loss of
external validity.
Odor as a Dependent Variable: Human
Body Odor Collection
The mood odor research is based on the premise
that humans emit nonverbal olfactory signs that can
be interpreted by others (see Chapter 14, this hand-
book). Previous research has established that such
odors can be discriminated by detectors (Chen &
Haviland-Jones, 2000).
Human body odors are obtained by collecting
axillary (underarm) secretions. The researchers thus
far have generally limited the collection of axillary
secretions to adult male donors. This controls for
possible variations in body odor due to fluctuat-
ing hormonal cycles of adult females. The extent
of control that researchers feel may be required
can vary. If one is concerned about controls, there
are parameters that can be met. Typically, males of
approximately the same age are recruited based on
the assumption that hormone production is likely
to be similar because hormone production changes
with age. Body odor is affected by other factors as
well, for example, illness, medications, diet, and
chemicals applied to the skin. To control for these
influences, individuals who take medications for
chronic illness or who are currently unwell would
not be considered appropriate donors. To control
for the effects of diet, the donors would be asked
to refrain from eating spicy or odorous foods for
several days prior to collection and for the duration

of the collection. Because chemicals applied to the
skin (i.e., fragranced body-care products, including
lotions, shampoos, and deodorants) will be part of
the collected secretions, donors would be required
to refrain from their use for several days prior to
and for the duration of the collection. Research-
ers would then supply unfragranced products to
donors for use during the collection. Deodorant
is supplied rather than an antiperspirant because
antiperspirant builds up on the skin and can remain
for several days without reapplication. On the other
hand, some researchers are more concerned with
naturalistic mood odor production and would not
apply such controls. As we presently do not know
which organic compounds in axillary secretions are
active for mood communication, the effect of other
compounds present in axillary sections is unknown
as well.
When researchers specifically want to study
human mood odors, some method of mood induc-
tion is employed. For example, mood-inducting vid-
eos have been used by many researchers (e.g., Chen
et al., 2006; Haviland-Jones & McGuire, 1999), and
their reliability is widely accepted (Gross & Leven-
son, 1995). On the naturalistic side, Mujica-Parodi
et al. (2009) used sky-diving to produce anxiety.
Because more than one type of mood odor is often
desired, the same donors are used in each type of
collection (thus making the odors within-subject).
The axillary secretions are obtained by placing a
sterile gauze pad into the armpit prior to the mood
induction procedure. Then, the collected gauze
pads must be frozen to control for bacterial growth.
The gauze pads must be brought back to room or
body temperature for odor delivery. Typically, the
samples of several donors are pooled to create a
“group of people” representation, that is, what one
might expect to be exposed to in the natural course
of social activity.
In conclusion, presently to collect human mood
body odors underarm secretions are used. These col-
lections are from a homogeneous group and are fro-
zen immediately after collection. The odors may or
may not be collected from donors who maintained
a restricted diet or who controlled use of fragranced
body care products. The samples from several
donors are pooled.
Methods in Olfactory Research
MEASURING EFFECTS IN OLFACTORY
RESEARCH
In the sections that follow, we describe various
methods for measuring the effects of odors in
research. Specifically, we discuss self-report, narra-
tive, psychophysiology, neuroimaging, and behav-
ioral applications. Lastly, we will consider various
cross-cultural applications.
Self-Report
Questionnaire measures are still considered by many
to be the gold standard for studying psychological
experience. Within olfaction research, self-reports
are one of the most common methods for studying
how odors impact emotional and social communi-
cation. Although self-reports are often used because
they are easy, inexpensive, and quick to implement,
within the fragrance industry self-reports are also
seen as a valid reflection of an individual’s experi-
ence. However, several problems remain when odor
researchers solely rely on questionnaires (for a related
discussion about self-report measures of nonverbal
sensitivity, see also Chapter 23, this handbook).
First, social desirability effects can make ques-
tionnaires particularly unreliable. Although people
report that their emotions are fairly stable and not
susceptible to manipulation (White, 1975), Brody
and Hall (1993, 2000) have observed that men typi-
cally report less emotion than women except in
cases of anger where the pattern is reversed. How-
ever, behavioral studies often do not show such
gender differences. For example, in our work exam-
ining emotional and behavioral responses to flow-
ers, the cultural expectation would be that women
would be more influenced than men. However,
when given a flower, men were as likely to smile,
socially approach, and initiate a conversation as
were women (Haviland-Jones, Rosario, Wilson, &
McGuire, 2005). Although most people believe
women are more influenced by flowers or even a
floral scent, it does not seem to be true at the behav-
ioral level. In fact, men’s behavior showed even
more change from baseline than women’s behavior.
Importantly, although self-reports can be a very
good source of information if and when participants
are aware of their behavior or feelings, their use
543
Freyberg, Wilson, and Haviland-Jones
becomes more problematic when experience hap-
pens quickly. Often, emotional expressions travel
fast and outside of conscious awareness. The use of
questionnaires assumes that the questionnaires can
be properly used to describe a participant’s experi-
ence. It is possible that some experiences are too
complex to be captured through specific questions
that have been defined by the researcher. In such
cases, questionnaires would not completely capture
the full extent of individual experience.
Similarly, self-report measures may be too
imprecise for studying effects due to odor expo-
sure because such effects are often quite subtle.
We observed that when individuals were exposed
to perithreshold odors, they often did not report
emotional changes. Instead, participants incorpo-
rated more emotion words in their narratives when
asked to write about recent events (Haviland-Jones
et al., 2013). Similarly, when adolescent girls were
exposed to an unfamiliar fragrance, there were no
direct self-report effects for any of the measures.
Instead, indirect effects were observed for social
behavior and narrative content when they wore their
favorite fragrance compared to other fragrances
(Freyberg & Ahren, 2011).
As a result, we have proposed that fragrance can
serve as a biological search engine (see Chapter 14,
this handbook). When a particular odor is in the
air, people will search for events or people that cor-
respond to the particular odorant. When matches
occur between the fragrance and environment, a
search engine organizes people’s attention and expe-
rience. Such processes occur rapidly and indirectly
and, therefore, are not reported and are usually out-
side of conscious awareness. For example, if an indi-
vidual is exposed to a sensual fragrance, we propose
that a sensual behavior, memory, semantic network,
or emotion will more likely emerge than if there is
no sensual fragrance. As a result, researchers need
to look beyond self-reports to psychophysiology,
narrative, or nonverbal behavior to detect the search
engine effects, because the person having the experi-
ence may not be aware of the shift in attention and
particularly may not attribute it to a fragrance.
In sum, researchers should remain cautious
when incorporating self-report within their designs,
especially if it is the sole method of capturing
544
participant experience. As with much psychological
research, social desirability effects are still a consid-
erable concern, such that participants may not be
willing to honestly report their experience due to
individual or societal factors. In addition, question-
naires may not be subtle enough to fully capture the
cognitions, emotions, or behaviors of participants
due to the fact that psychological phenomena are
often too fast, out of conscious awareness, or too
complex for the instrument.
Narrative Methods
Because traditional methods of analysis, such as
questionnaires, have often been limited in measur-
ing more subtle processes of behavior, we incorpo-
rated narrative methods in several of our studies to
more fully capture experience during odor expo-
sure. An example of instructions for a narrative
might be as follows:
Think about a personal experience. It can
be any type of personal experience but
focus on one in which you actually par-
ticipated and one that you have a good
recollection of. Write down a descrip-
tion of the experience, including all of
the details about the experience that you
can recall. Include your feelings about
the experience at the time. (Freyberg &
Ahren, 2011, p. 239)
The process of working with narrative is different
than working with questionnaires—each yields dif-
ferent information and is analyzed differently (see
Tausczik & Pennebaker, 2010, for a complete discus-
sion). Although narrative methods have been used
extensively in other areas of psychological research
when conscious methods, such as self-reports, may
not be helpful (see Pennebaker, Mehl, & Niederhof-
fer, 2003, for a review), their use within olfaction
research is only starting to emerge, but they have
already been demonstrated to be a useful method.
For example, work from our lab demonstrated that
although self-reports indicated that participants
reported no differences in pleasantness or intensity
across fresh air, floral odor, or fragrance conditions,
differences emerged in narrative content (Castel-
lanos, Hudson, Haviland-Jones, & Wilson, 2010;

Haviland-Jones et al., 2013). Specifically, using
software designed to analyze narrative content, we
observed that participants exposed to floral odors
at perithreshold levels incorporated more words
indicative of enjoyment. This is akin to the work on
the embodiment phenomenon where general cat-
egories, even metaphorical categories, are associated
with automatic perceptual sensory experiences (see
Bargh, Schwader, Hailey, Dyer, & Boothby, 2012).
This makes narrative analyses far more central than
self-report and far more informative.
In other contexts, narratives can serve to extend
findings from questionnaires, especially when odors
are delivered at levels above conscious detection.
For example, Freyberg (2009) demonstrated that
adolescent participants reported that social activi-
ties were the most enjoyable and time-consuming
activities of the day, and their narratives also incor-
porated more themes of intimacy and closeness than
other types of themes. However, finding connec-
tions between narratives and self-reports may not
be direct. Research by Freyberg and Ahren (2011)
examined narratives and questionnaires of adoles-
cents when they wore their favorite fragrance or an
unfamiliar one provided by the researchers. Analyses
demonstrated that reports of social enjoyment and
words indicative of intimacy within the narratives
decreased when they wore the unfamiliar fragrance
compared to their favorite but only when they also
found the unfamiliar fragrance to be less pleasant
than their favorite. Such findings provide additional
evidence that effects of odor exposure are often sub-
tle and require analyzing the data with greater com-
plexity. Taken as a whole, incorporating narrative in
odor research can be particularly advantageous over
relying on questionnaires because narratives often
can confirm, extend, or delve deeper than self-report
methods are typically capable of doing.
Psychophysiology
Psychophysiological methods—such as heart rate,
skin conductance, and others—have been incorpo-
rated in olfactory research as a way of understanding
how odors affect participants on a physical level.
For example, Bensafi et al. (2002) observed that
heart rate was positively associated with pleas-
antness ratings, whereas skin conductance was
Methods in Olfactory Research
positively correlated with how arousing the particu-
lar odor was perceived by participants. Similarly,
Delplanque et al. (2009) observed that heart rate
and facial muscles reacted to novelty and pleasant-
ness evaluations of various odors. Although such
findings provide useful information about how odor
perception affects psychophysiology, they do not tell
how such odors affect the individual participant’s
emotions or nonverbal communication.
Further, psychophysiology does not always coin-
cide with responses on questionnaires. Graham,
Janssen, and Sanders (2000) investigated the pos-
sible impact of male and female fragrances on sexual
arousal and mood during the menstrual cycle. Par-
ticipants watched either an erotic video or engaged
in a private sexual fantasy. They found a heightened
sexual response by females to a male fragrance dur-
ing their follicular phase (prior to the periovulatory
phase) but only when they engaged in a personal
sexual fantasy. The researchers did not observe any
changes in self-report. Such findings provide further
evidence that a disconnect between questionnaires
and psychophysiology happens, and thus research-
ers should exercise caution.
Looking at affect in general, several studies have
incorporated psychophysiology as a means of under-
standing how odors impact moods. For example,
several studies involving the semiochemical AND
have reported a modulating effect, whereby reported
negative mood and heart rate response and/or gal-
vanic skin response decreased compared to the
control odor (Jacob & McClintock, 2000). Kiecolt-
Glaser et al. (2008) observed enhanced positive
mood via self-report for lemon oil compared to lav-
ender; however, they observed no changes in heart
rate, blood pressure, or salivary cortisol. Such find-
ings contrast with others who have observed notable
changes in heart rate due to odor exposure (Bensafi
et al., 2002).
In our lab, one of the difficulties in incorporat-
ing psychophysiology when studying behavioral
responses to odors remains the interpretation of
the physical responses. As a result, we have typi-
cally measured psychophysiological responses to
odors to confirm and extend findings derived from
other methods. For example, in a study involv-
ing pairs of close friends who engaged in a 10-min
545
Freyberg, Wilson, and Haviland-Jones
discussion while exposed to different odors, we
found heart rate response paralleled some of the
findings that were observed using self-reports or rat-
ings of video data (Freyberg & Bart, 2012). Because
behavioral responses due to odor exposure are often
quite subtle, psychophysiology may be too gross or
obtrusive on its own. Regardless, researchers are
encouraged to exercise caution when incorporating
psychophysiology when examining odor exposure.
Psychophysiology can be an incredibly useful way to
determine how the body is physically responding to
chemosignals, yet interpretations can sometimes be
problematic.
Neuroimaging Methods
Advances in brain imaging have enabled access to
brain structures that are involved in the processing
of olfactory stimuli. Measurement of cerebral blood
flow as an indicator of brain activation is obtained
via positron emission topography (PET) or func-
tional magnetic resonance imaging (fMRI). Such
techniques can be used to elucidate the brain region
activations associated with olfactory stimuli. Most
important, such brain imaging studies can demon-
strate that an olfactory stimulus impacts not only
areas specifically dedicated to olfactory processing
but other brain areas as well. This enables associa-
tions between brain region function and stimuli,
thus enhancing understanding of the behaviors that
olfactory stimuli may elicit or affect. For example,
Jacob, Kinnunen, Metz, Cooper, and McClintock
(2001) used PET to investigate modulation of neu-
ral activity associated with exposure to AND (see
Chapter 14, this handbook). They found sustained
brain changes in areas not only specifically associ-
ated with olfactory processing but also in areas
considered to be associated with attention and
mood. Further, these effects were found without
conscious exposure to AND. A similar study found
increased glucose utilization in brain areas associ-
ated with social cognition when exposed to AND
(Gulyás, Kéri, O’Sullivan, Decety, & Roland, 2004).
Specificity of responses to olfactory stimuli has been
demonstrated as well. Specifically, using PET analy-
ses, Lundström, Boyle, Zatorre, and Jones-Gotman
(2008) found that different neural pathways
process body odors compared to nonbody odors.
546
Further discrimination was evident in that body
odors from friends as opposed to strangers activated
different brain areas. Such information allows for a
more nuanced interpretation of behavioral effects.
Understanding the different brain activation
areas in olfactory processing can ultimately lead to
inferences regarding function. For example, Herz,
Eliassen, Beland, and Souza (2004) asked partici-
pants to identify a commercial fragrance that evoked
pleasant memories for them. The participants were
then exposed to the scent of the identified fragrance
as well as pictures of the fragrance; the control was
the scent and picture of an unmarketed fragrance.
Using fMRI analyses, Herz, Eliassen, et al. found
that the pleasant memory odor cue was significantly
more likely to activate the amygdala and hippocam-
pus, suggesting that the personally meaningful cue
had demonstrable effects on brain areas associated
with emotion and memory.
A fascinating new direction has been discussed
by Aglioti and Pazzaglia (2011). In their review dis-
cussing the possible impact of auditory and olfactory
stimuli on interactions with others, they have raised
the issue of whether olfactory stimuli may have the
sort of impact on the motor system (i.e., mirror neu-
rons) that visual stimuli has. Although they speci-
fied that the available evidence at this point is scant,
they also offered research evidence that suggests that
this is an important research direction both in terms
of multimodal effects and of specific sensory effects.
One such study using fMRI demonstrated that
specific brain regions were activated when observ-
ing an action directed toward a smelled object and
that these brain regions were different from those
activated when observing an action directed toward
an object that could be seen (Tubaldi et al., 2011).
Others have implicated the mirror neuron system as
important to the understanding of nonverbal inter-
actions; it is noteworthy that the research direction
here is implicating olfaction both specifically and as
part of a multimodal representation.
In conclusion, various neuroimaging techniques
can be particularly useful to explore odor effects
through allowing researchers to explore brain func-
tion and activation. However, it is essential to ensure
that using such methods of examination and stimuli
are valid so that appropriate conclusions can be made.

Behavioral Methods
Because self-reports may not always be the best
method for determining the effects of odors particu-
larly at the perithreshold level, behavioral methods
can provide useful insight into psychological effects
when there is a supportive context. Work in our lab
demonstrated that participants exposed to floral air
were three times more likely to approach and touch
an experimenter who dressed and acted as a mime
than those exposed to fine fragrance or fresh air
(Haviland-Jones et al., 2013). Similarly, we observed
socially supportive behavior, such as smiling and
initiating of conversations, when individuals were
presented with flowers compared to those who
received a candle, fruit basket, or pen (Haviland-
Jones et al., 2005). Such findings suggest that given
the appropriate context, odors can produce socially
productive behaviors even when the odors are at
levels below conscious detection.
Behavioral methods have also been useful when
fragrances are presented at conscious levels of detec-
tion given the appropriate context. Baron (1997)
was one of the first to apply research methodologies
from social psychology to olfaction and demon-
strated that participants in a shopping mall were
more likely to help a confederate in the presence of
pleasant odors. More recent extensions of this work
have observed that pleasant ambient odors also
encouraged helping behaviors that were spontane-
ous (Guéguen, 2012b) as well as social approach
behaviors defined as providing a phone number to a
potential suitor (Guéguen, 2012a).
Ebihara et al. (2012) incorporated lavender
and grapefruit odors to explore potential effects
in movement for older adults. They observed that
odor exposure improved gait speed. The authors
suggested that such effects may ultimately result in
reduced fall rates for older adults. Such research as
a whole provides evidence of the wide range of pos-
sible behavioral methods in odor research.
Aside from the more overt types of behavioral
methods that have been discussed so far, there is
additional evidence that individuals may not always
be aware of whether and how odors ultimately
impact their behavior. For example, participants
exposed to odors from a citrus-scented cleanser
engaged in more cleaning behaviors, such as
Methods in Olfactory Research
removing snack crumbs from their work environ-
ment (Holland, Hendriks, & Aarts, 2005). Addi-
tionally, Baron (1983) observed that participants
maintained differing perceptions of a confederate
during an interview scenario based on the interac-
tion between clothing style and presence of per-
fume. Specifically, participants positively perceived
the confederate if she wore either perfume with
casual clothing or dressy clothing with no perfume.
Confederates who wore fragrance with dressy cloth-
ing or casual clothes with no perfume were per-
ceived more negatively. Such findings suggest that
the behavioral effects of fragrance are often out of
people’s conscious awareness and depend on the
overall environmental context.
Taken as a whole, there are a wide variety of
behavioral methods that can provide useful insights
into the psychological effects of odor exposure.
Behavioral studies are useful in a number of con-
texts, particularly when odors are delivered at peri-
threshold levels, especially given that the behavioral
effects are often out of conscious awareness. Ulti-
mately, behavioral approaches may demonstrate a
complex depth of participant experience that may be
unattainable through self-reports.
Cross-Cultural Applications
Currently, as the psychological study of odors is
still somewhat new, many large gaps remain in the
research literature, particularly in regard to cross-
cultural approaches. One of the most remarkable
gaps, given the consensus among those in the indus-
try or among historians, with regard to differences
is the lack of documentation of differences across
ethnic, cultural, or geographic groups (see Claussen
et al., 1994, for historical and sociological reviews).
As an example from industry, in 2012 the European
Cleaning Journal published interviews from spokes-
persons in several companies and concluded that
people prefer scents that are geographically endoge-
nous, with the British having preferences for grasses
and fruits; those in the Arabian peninsula preferring
strong, sweet scents; and those in southern Europe
preferring more citrus (Laffeaty, 2012). In Asia, aro-
matherapies tend to favor spices, such as clove and
cinnamon. However, no published academic stud-
ies to our knowledge have examined cross-cultural
547
Freyberg, Wilson, and Haviland-Jones
or geographic differences in odor exposure. In a
related field of individual differences, geneticists
note that the olfactory system is probably the largest
source of genetic variation among humans, provid-
ing ample opportunity for the development of global
differences. The expectation is that large-scale func-
tional genetic screens combined with sensory and
emotional testing will uncover substantial differ-
ences across ethnic groups and global areas (Hasin-
Brumshtein, Lancet, & Olender, 2009).
DIRECTIONS FOR FUTURE RESEARCH
Generally, questions in social psychology (such as
attitude formation and social influence) and in the
sensory sciences (such as taste and odor discrimi-
nation) historically have been examined indepen-
dently. Given that olfactory chemical compounds
surround our daily experience and influence our
moods and behaviors, we must consider their pos-
sible effects on us. For example, we can consider
whether odors can impact individuals’ prejudiced
beliefs and use of stereotypes. We can consider
whether ambient environmental stimuli contribute
toward cooperation, competition, and empathy.
They seem to; pleasant odors favorably impact
the work environment, increasing the efficiency
and reducing competition of workers in general
and increasing the self-efficacy of male workers in
particular (Baron, 1990). These issues have drawn
some research attention, but this is not sufficient to
understand fully the odor effects. The mechanisms
under which certain phenomena exist need to be
elucidated, and the proper measurements and meth-
ods are the place to start.
References
Aglioti, S. M., & Pazzaglia, M. (2011). Sounds and scents
in (social) action. Trends in Cognitive Sciences, 15,
47–55. http://dx.doi.org/10.1016/j.tics.2010.12.003
Bargh, J. A., Schwader, K. L., Hailey, S. E., Dyer, R. L., &
Boothby, E. J. (2012). Automaticity in social-
cognitive processes. Trends in Cognitive Sciences, 16,
593–605. http://dx.doi.org/10.1016/j.tics.2012.
10.002
Baron, R. A. (1983). “Sweet smell of success”? The
impact of pleasant artificial scents on evaluations
of job applicants. Journal of Applied Psychology, 68,
548
709–713. http://dx.doi.org/10.1037/0021-9010.
68.4.709
Baron, R. A. (1990). Environmentally induced positive
affect: Its impact on self-efficacy, task performance,
negotiation, and conflict. Journal of Applied
Social Psychology, 20, 368–384. http://dx.doi.
org/10.1111/j.1559-1816.1990.tb00417.x
Baron, R. A. (1997). The sweet smell of . . . helping:
Effects of pleasant ambient fragrance on prosocial
behavior in shopping malls. Personality and Social
Psychology Bulletin, 23, 498–503. http://dx.doi.
org/10.1177/0146167297235005
Bensafi, M., Rouby, C., Farget, V., Bertrand, B.,
Vigouroux, M., & Holley, A. (2002). Autonomic
nervous system responses to odours: The role of
pleasantness and arousal. Chemical Senses, 27,
703–709. http://dx.doi.org/10.1093/chemse/27.8.703
Brody, L. R., & Hall, J. A. (1993). Gender and emotion.
In M. Lewis & J. M. Haviland-Jones (Eds.), The
handbook of emotion (pp. 447–460). New York, NY:
Guilford Press.
Brody, L. R., & Hall, J. A. (2000). Gender, emotion,
and expression. In M. Lewis & J. M. Haviland-Jones
(Eds.), The handbook of emotion (2nd ed.,
pp. 338–349). New York, NY: Guilford Press.
Buck, L., & Axel, R. (1991). A novel multigene family
may encode odorant receptors: A molecular basis for
odor recognition. Cell, 65, 175–187. http://dx.doi.
org/10.1016/0092-8674(91)90418-X
Cain, W. S., Cometto-Muñiz, J. E., & de Wijk, R. A.
(1992). Techniques in the quantitative study of
human olfaction. In M. J. Serby & K. L. Chobor
(Eds.), Science of olfaction (pp. 279–308). http://
dx.doi.org/10.1007/978-1-4612-2836-3_9
Castellanos, K. M., Hudson, J. A., Haviland-Jones, J., &
Wilson, P. J. (2010). Does exposure to ambient
odors influence the emotional content of memories?
American Journal of Psychology, 123, 269–279. http://
dx.doi.org/10.5406/amerjpsyc.123.3.0269
Chen, D., & Haviland-Jones, J. (2000). Human olfactory
communication of emotion. Perceptual and Motor
Skills, 91, 771–781. http://dx.doi.org/10.2466/
pms.2000.91.3.771
Chen, D., Katdare, A., & Lucas, N. (2006). Chemosignals
of fear enhance cognitive performance in humans.
Chemical Senses, 31, 415–423. http://dx.doi.
org/10.1093/chemse/bjj046
Claussen, C., Howes, D., & Synott, A. (1994). Aroma: A
cultural history of smell. London, England: Routledge.
Degel, J., & Köster, E. P. (1999). Odors: Implicit memory
and performance effects. Chemical Senses, 24,
317–325. http://dx.doi.org/10.1093/chemse/24.3.317
Degel, J., Piper, D., & Köster, E. P. (2001). Implicit
learning and implicit memory for odors:

The influence of odor identification and retention
time. Chemical Senses, 26, 267–280. http://dx.doi.org/
10.1093/chemse/26.3.267
Delplanque, S., Grandjean, D., Chrea, C., Coppin, G.,
Aymard, L., Cayeux, I., . . . Scherer, K. R. (2009).
Sequential unfolding of novelty and pleasantness
appraisals of odors: Evidence from facial
electromyography and autonomic reactions. Emotion,
9, 316–328. http://dx.doi.org/10.1037/a0015369
Ebihara, S., Nikkuni, E., Ebihara, T., Sakamoto, Y.,
Freeman, S., & Kohzuki, M. (2012). Effects of
olfactory stimulation on gait performance in frail
older adults. Geriatrics and Gerontology International,
12, 567–568. http://dx.doi.org/10.1111/j.1447-
0594.2011.00801.x
Fiedler, N., Kelly-McNeil, K., Ohman-Strickland, P.,
Zhang, J., Ottenweller, J., & Kipen, H. M. (2008).
Negative affect and chemical intolerance as risk
factors for building-related symptoms: A controlled
exposure study. Psychosomatic Medicine, 70, 254–262.
http://dx.doi.org/10.1097/PSY.0b013e31816074f4
Freyberg, R. (2009). Quantitative and qualitative
measures of behavior in adolescent girls. Adolescence,
44, 33–54.
Freyberg, R., & Ahren, M.-P. (2011). A preliminary trial
exploring perfume preferences in adolescent girls.
Journal of Sensory Studies, 26, 237–243. http://dx.doi.
org/10.1111/j.1745-459X.2011.00339.x
Freyberg, R., & Bart, M. (2012, April). Olfactory
environment influences close relationships through
multiple methods of measurement. Poster presented
at the 33rd Annual Meeting of the Association of
Chemoreception Sciences Society, Huntington
Beach, CA.
Graham, C. A., Janssen, E., & Sanders, S. A. (2000). Effects
of fragrance on female sexual arousal and mood across
the menstrual cycle. Psychophysiology, 37, 76–84.
http://dx.doi.org/10.1111/1469-8986.3710076
Gross, J. J., & Levenson, R. W. (1995). Emotion elicitation
using films. Cognition and Emotion, 9, 87–108. http://
dx.doi.org/10.1080/02699939508408966
Guéguen, N. (2012a). The sweet smell of . . . courtship:
Effects of pleasant ambient fragrance on women’s
receptivity to man’s courtship request. Journal of
Environmental Psychology, 32, 123–125. http://dx.doi.
org/10.1016/j.jenvp.2012.01.004
Guéguen, N. (2012b). The sweet smell of . . . implicit
helping: Effects of pleasant ambient fragrance on
spontaneous help in shopping malls. Journal of Social
Psychology, 152, 397–400. http://dx.doi.org/10.1080/
00224545.2011.630434
Guéguen, N., & Petr, C. (2006). Odors and consumer
behavior in a restaurant. International Journal of
Hospitality Management, 25, 335–339. http://dx.doi.
org/10.1016/j.ijhm.2005.04.007
Methods in Olfactory Research
Gulyás, B., Kéri, S., O’Sullivan, B. T., Decety, J., &
Roland, P. E. (2004). The putative pheromone
androstadienone activates cortical fields in
the human brain related to social cognition.
Neurochemistry International, 44, 595–600. http://
dx.doi.org/10.1016/j.neuint.2003.10.003
Hasin-Brumshtein, Y., Lancet, D., & Olender, T.
(2009). Human olfaction: From genomic variation
to phenotypic diversity. Trends in Genetics, 25,
178–184. http://dx.doi.org/10.1016/j.tig.2009.02.002
Haviland-Jones, J., Hudson, J. A., Wilson, P., Freyberg, R., &
McGuire, T. (2013). The emotional air in your space:
Scrubbed, wild or cultivated? Emotion, Space and
Society, 6, 91–99. http://dx.doi.org/10.1016/
j.emospa.2011.10.002
Haviland-Jones, J., & McGuire, D. (1999). The scents of
fear and funny. Aroma-Chology Review, 8, 11.
Haviland-Jones, J. M., Rosario, H. H., Wilson, P., &
McGuire, T. R. (2005). An environmental approach
to positive emotion: Flowers. Evolutionary
Psychology, 3, 104–132.
Herz, R. S. (2003). The effect of verbal context on
olfactory perception. Journal of Experimental
Psychology: General, 132, 595–606. http://dx.doi.org/
10.1037/0096-3445.132.4.595
Herz, R. S., Eliassen, J., Beland, S., & Souza, T. (2004).
Neuroimaging evidence for the emotional potency
of odor-evoked memory. Neuropsychologia,
42, 371–378. http://dx.doi.org/10.1016/
j.neuropsychologia.2003.08.009
Herz, R. S., Schankler, C., & Beland, S. (2004). Olfaction,
emotion and associative learning: Effects on motivated
behavior. Motivation and Emotion, 28, 363–383. http://
dx.doi.org/10.1007/s11031-004-2389-x
Holland, R. W., Hendriks, M., & Aarts, H. (2005). Smells
like clean spirit: Nonconscious effects of scent on
cognition and behavior. Psychological Science, 16,
689–693. http://dx.doi.org/10.1111/j.1467-9280.
2005.01597.x
Hummer, T. A., & McClintock, M. K. (2009). Putative
human pheromone androstadienone attunes
the mind specifically to emotional information.
Hormones and Behavior, 55, 548–559. http://dx.doi.
org/10.1016/j.yhbeh.2009.01.002
Jacob, S., Kinnunen, L. H., Metz, J., Cooper,
M., & McClintock, M. K. (2001). Sustained human
chemosignal unconsciously alters brain function.
NeuroReport, 12, 2391–2394. http://dx.doi.
org/10.1097/00001756-200108080-00021
Jacob, S., & McClintock, M. K. (2000). Psychological
state and mood effects of steroidal chemosignals in
women and men. Hormones and Behavior, 37, 57–78.
http://dx.doi.org/10.1006/hbeh.1999.1559
Jellinek, P. (1996). The psychological basis of perfumery.
In J. S. Jellinek (Ed.), The psychological basis of
549
Freyberg, Wilson, and Haviland-Jones
perfumery (4th ed., pp. 1–162). http://dx.doi.
org/10.1007/978-94-009-1567-1
Jellinek, P. (2003). Implicit olfaction: Effect on use of
fragrance in functional products. Cosmetics and
Toiletries, 118, 47–64.
Kaitz, M., Good, A., Rokem, A. M., & Eidelman, A. I.
(1987). Mothers’ recognition of their newborns by
olfactory cues. Developmental Psychobiology, 20,
587–591. http://dx.doi.org/10.1002/dev.420200604
Kiecolt-Glaser, J. K., Graham, J. E., Malarkey, W. B.,
Porter, K., Lemeshow, S., & Glaser, R. (2008).
Olfactory influences on mood and autonomic,
endocrine, and immune function. Psychoneuro­
endocrinology, 33, 328–339. http://dx.doi.org/
10.1016/j.psyneuen.2007.11.015
Kirk-Smith, M. D., & Booth, D. A. (1987). Chemoreception
in human behaviour: Experimental analysis of the
social effects of fragrances. Chemical Senses, 12,
159–166. http://dx.doi.org/10.1093/chemse/12.1.159
Laffeaty, A. (2012, October 31). World of fragrance.
European Cleaning Journal. Retrieved from http://
www.europeancleaningjournal.com/magazine/
articles/latest-news/world-of-fragrance
Lundström, J. N., Boyle, J. A., Zatorre, R. J., & Jones-
Gotman, M. (2008). Functional neuronal processing
of body odors differs from that of similar common
odors. Cerebral Cortex, 18, 1466–1474. http://dx.doi.
org/10.1093/cercor/bhm178
Macfarlane, A. (1975). Olfaction in the development
of social preferences in the human neonate.
Parent–Infant Interaction, 33, 103–17.
Milinski, M., & Wedekind, C. (2001). Evidence for
MHC-correlated perfume preferences in humans.
Behavioral Ecology, 12, 140–149. http://dx.doi.org/
10.1093/beheco/12.2.140
Mujica-Parodi, L. R., Strey, H. H., Frederick, B., Savoy, R.,
Cox, D., Botanov, Y., . . . Weber, J. (2009).
Chemosensory cues to conspecific emotional stress
activate amygdala in humans. PLoS ONE, 4, e6415.
http://dx.doi.org/10.1371/journal.pone.0006415
Pennebaker, J. W., Mehl, M. R., & Niederhoffer, K. G.
(2003). Psychological aspects of natural language
use: Our words, our selves. Annual Review of
Psychology, 54, 547–577. http://dx.doi.org/10.1146/
annurev.psych.54.101601.145041
Phillips, K., & Cupchik, G. C. (2004). Scented memories
of literature. Memory, 12, 366–375. http://dx.doi.
org/10.1080/09658210344000053
550
Porter, R. H., Balogh, R. D., Cernoch, J. M., & Franchi,
C. (1986). Recognition of kin through characteristic
body odors. Chemical Senses, 11, 389–395. http://
dx.doi.org/10.1093/chemse/11.3.389
Raudenbush, B., Meyer, B., & Eppich, B. (2002). The effects
of odors on objective and subjective measures of athletic
performance. International Sports Journal, 6, 14–27.
Sakamoto, R., Minoura, K., Usui, A., Ishizuka, Y., &
Kanba, S. (2005). Effectiveness of aroma on work
efficiency: Lavender aroma during recesses prevents
deterioration of work performance. Chemical Senses,
30, 683–691. http://dx.doi.org/10.1093/chemse/bji061
Sakamoto, Y., Ebihara, S., Ebihara, T., Tomita, N.,
Toba, K., Freeman, S., . . . Kohzuki, M. (2012).
Fall prevention using olfactory stimulation with
lavender odor in elderly nursing home residents:
A randomized controlled trial. Journal of the
American Geriatrics Society, 60, 1005–1011. http://
dx.doi.org/10.1111/j.1532-5415.2012.03977.x
Schifferstein, H. N. J., Talke, K. S. S., & Oudshoorn, D. J.
(2011). Can ambient scent enhance the nightlife
experience? Chemosensory Perception, 4, 55–64.
http://dx.doi.org/10.1007/s12078-011-9088-2
Schiffman, S. S., Sattely-Miller, E. A., Suggs, M. S., &
Graham, B. G. (1995). The effect of pleasant odors
and hormone status on mood of women at midlife.
Brain Research Bulletin, 36, 19–29. http://dx.doi.
org/10.1016/0361-9230(94)00133-L
Smith, R. A., & Davis, S. F. (2013). The psychologist
as detective: An introduction to conducting research
in psychology (6th ed.). Saddle River, NJ: Pearson
Education.
Tausczik, Y. R., & Pennebaker, J. W. (2010). The
psychological meaning of words: LIWC and
computerized text analysis methods. Journal of
Language and Social Psychology, 29, 24–54. http://
dx.doi.org/10.1177/0261927X09351676
Tubaldi, F., Turella, L., Pierno, A. C., Grodd, W.,
Tirindelli, R., & Castiello, U. (2011). Smelling odors,
understanding actions. Social Neuroscience, 6, 31–47.
http://dx.doi.org/10.1080/17470911003691089
White, R. S. (1975). Lives in progress: A study of the
natural growth of personality (3rd ed.). Oxford,
England: Holt, Rinehart & Winston.
Wysocki, C. J., & Preti, G. (2004). Facts, fallacies,
fears, and frustrations with human pheromones.
Anatomical Record, Part A: Discoveries in Molecular,
Cellular, and Evolutionary Biology, 281, 1201–1211.
http://dx.doi.org/10.1002/ar.a.20125
 Chapter 22
Measuring Body Movement:
Current and Future Directions
in Proxemics and Kinesics
Nele Dael, Nadia Bianchi-Berthouze, Andrea Kleinsmith, and Christine Mohr
Methodological development in nonverbal behavior
research has two major driving forces. One is the need
for a pragmatic solution to investigate a priori theoreti-
cal questions. For example, multimodal integration
has become a pressing issue after decades of research
focusing on single channels, such as facial or vocal
expression. The current surge of multimodal accounts
of emotion expression and perception in particular
demands time-based descriptions of body movement
that can be integrated with other time-based accounts
from vocal or facial behavior (e.g., Dael, Mortillaro, &
Scherer, 2012a). Further, interests in dynamic behavior
coordination as in social interaction studies fostered
the use of sequential behavior analyses (e.g., Yale,
Messinger, Cobo-Lewis, & Delgado, 2003). Another
driving force is technological advancement creating
new affordances. The 21st century is experiencing an
exponential growth of computational sciences, where
increasing storage and processing speeds along with
high-resolution recording devices enable sophisticated
micro- (e.g., frame-by-frame tracking of small body
parts) as well as macroanalyses of nonverbal behavior
(e.g., navigation patterns of entire populations based on
cell phone signals). Given that new methods in body
movement measurement are often inspired by (or inspi-
rational to) methods developed in related fields, we
dedicate a large section in this chapter to the introduc-
tion of computational analysis of body movement.
A LONG HISTORY OF DIVERGING APPROACHES
Western interest in labeling or describing body
movement goes back for centuries, when Greek and
The writing of this chapter was supported by the Institute of Psychology, University of Lausanne, Lausanne, Switzerland.
http://dx.doi.org/10.1037/14669-022
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
Roman philosophers (e.g., Aristotle, 330 BC/1924;
Quintilian, 1856/2006) analyzed the role of body
movement during rhetoric communication. Some of
the earliest works dedicated exclusively to the mean-
ings of body actions date from the 17th century. Bon-
ifacio (1616) and Bulwer (1644a, 1644b) emphasized
the role and importance of body actions as a natural
human language (see illustration of hand shapes and
supposed meaning in Figure 22.1). Later in the 19th
century, Darwin (1872) was the first to focus on the
functional role of body movement in emotion com-
munication in humans and animals. He treated body
actions and postures as signs of internal emotional
processing that later evolved to carry an additional
signaling value following the increased human vol-
untary control over motor expression and the selec-
tive advantage of recognizing emotions in others.
The same diverse picture can still be drawn
of today’s scientific landscape, where human and
animal behavior researchers investigate several
functions of body movement in nonverbal commu-
nication. Clinical psychology focuses on the diag-
nostic and therapeutic value of nonverbal behavior
and, in particular, on behaviors that foster empathy
and rapport during clinician–patient interaction
(J. A. Hall, Harrigan, & Rosenthal, 1995; Harrigan &
Rosenthal, 1986). Emotion and personality research
investigates mechanisms of expression and percep-
tion of internal traits and states and how they can
be conveyed from changes in body movement (e.g.,
Koppensteiner & Grammer, 2010; Wallbott, 1998).
Studies on the coordination of social interaction
551
Dael et al.
Figure 22.1.  Plate reprinted from Chirologia: Or,
the naturall language of the hand (p. 151), by J. Bulwer,
1644, London, England: Thomas Harper. In the public
domain.
may focus on the role of body movement in turn-
taking, for example (Bavelas, Chovil, Coates, & Roe,
1995), or on marking interpersonal relationships,
such as status or intimacy (J. A. Hall, Coats, &
LeBeau, 2005). Human–computer interaction stud-
ies are then again interested in tracking and model-
ing body movement for its natural reproduction in
embodied agents (e.g., Hyniewska, Niewiadomski,
Mancini, & Pelachaud, 2010) and, more generally,
in building automatic recognition systems of human
affect, for instance, for enhancing media experience
(for reviews, see Kleinsmith & Bianchi-Berthouze,
552
2013; Scherer, Bänziger, & Roesch, 2010). The list
continues almost endlessly to include ethology,
developmental psychology, and linguistic research
on speech–gesture integration.
All of the earlier listed topics touch each other in
their common interest in describing body movement
and its functions, from different perspectives, and in
using different methods. The multifunctionality, in
addition to the anatomical complexity, of the skel-
etal body system logically leads to a high diversity
in descriptive methods. Scherer and Ekman’s (2005)
historical review highlights how different approaches
depend on research (epistemological) traditions.
Researchers tend to focus either on the factors that
determine nonverbal behavior at the individual level
or microlevel, or on interactions at the macrolevel as
influenced by cultural and social dynamic factors.
In addition, even within a research tradition or
discipline, there is no consensus on a conceptual
framework of body movement from which methods
can be derived (a shared observation with Harrigan,
2005). Two scientific “virtues” seem to stand in
opposition to each other: generality and selectivity.
A research field advances when a common meth-
odology is applied that make results from different
studies comparable. We have witnessed such devel-
opments in the facial and vocal domain with the
development of, for instance, the Facial Action Cod-
ing System (FACS; Ekman & Friesen, 1978) and of
standardized vocal extraction techniques (Scherer,
1979). On the other hand, a method is also success-
ful if it produces relevant information to answer a
specific research question. It has been argued that
generic methods lack sufficient specificity and are
therefore not picked up by the research community.
However, standardized methods could be of common
use if they can be flexibly used to target the specific
range of behaviors under investigation. Such evolu-
tion has occurred in the field of facial expression
research, for example. No such crystallization toward
golden standards in measuring body movement has
yet occurred, as new tools are still emerging.
OVERVIEW OF THIS CHAPTER
The previous sections showed that methodologi-
cal approaches and coding systems differ widely

depending on research topics. Instead of attempting
to review methods used for specific topics, we intro-
duce two major areas in body movement research
that have followed a rather separate methodological
tradition. Earlier surveys of methods in nonverbal
behavior research (e.g., Harrigan, 2005; Rosenfeld,
1982) provide a cornerstone for this work, in which
we first introduce research methods and practices
used in proxemics (the study of interpersonal and
environmental space) with a special focus on inter-
personal distance and touch and, second, in kinesics
(the study of body movement), in which we include
gait. Research in proxemics is mainly driven by an
interest in sociocultural and environmental factors
influencing nonverbal interaction, whereas kinesic
research is focused on the study and description of
body movement in communication (see Chapter 15,
this handbook).
These separate traditions do not need to imply
isolated measurement tools, as evidenced by
several studies adopting a hybrid of proxemic and
kinesic variables (e.g., Harrigan & Carney, 2005;
Harrigan & Rosenthal, 1986). In the following
sections of this chapter, we therefore more glob-
ally present common methodological issues and
highlight examples from various fields in body
movement research. In one section, we introduce
different techniques for sampling and measuring
behavior. In a following section, we discuss the
correspondence between the measured behavior
and the psychological variables of interest to the
researcher. We describe the links between encoded
and decoded variables with the theoretical model
known as probabilistic functionalism. We conclude
with a number of suggestions for the next genera-
tion of method development, which would benefit
from more refined behavior patterning and applica-
tions to study behavior in real-life settings.
PROXEMICS
Proxemics refer to the perception, use, and structur-
ing of interpersonal and environmental space and
how it marks relationships between individuals.
E. T. Hall (1959, 1963) has pioneered this research
domain, and his methodological approach can still
be regarded as a major contribution to the field.
Measuring Body Movement
To systematize his observation that cultures hold
different implicit norms regarding acceptable ranges
of distance between two interlocutors, E. T. Hall
proposed coding rules for eight types of proxemic
variables ranging from basic postures to thermal and
olfactive contact. An important feature of his coding
approach is that these variables are coded from the
perspective of the interlocutor. For instance, body
orientations vary from each other depending on
whether bodies face each other (sociopetal) or are
averted from each other (sociofugal). Also,
interpersonal distance is measured as a function
of ranges of reaching and touching the interlocutor
(with the head, forearms, extended arms or legs,
and so forth; see Figure 22.2). Since its develop-
ment, this coding system has undergone little
change (E. T. Hall, 1974; Watson & Graves, 1966),
but it has inspired many studies on the expression
and perception of status and affiliative attitudes
in social interaction from spatial arrangements
(Sommer, 1959; Sommer & Becker, 1969) or from
posture, orientation, and distance (J. A. Hall et al.,
2005; Mehrabian, 1968; Mehrabian & Friar, 1969).
The concept of immediacy (proposed by Mehrabian,
1967) is closely related to proxemics in that greater
immediacy (more touching, forward lean, eye con-
tact, directness of orientation, and smaller distances)
indicates a more positive speaker attitude toward an
addressee. Proxemic variables have also been stud-
ied in clinician–patient interaction. For example,
maintaining close distance from the patient, adopt-
ing a forward body orientation, and leaning toward
the patient correlate with higher patient satisfaction
and ratings of clinician empathy (see, e.g., J. A. Hall
et al., 1995; Harrigan & Rosenthal, 1986).
Techniques for Assessing Proxemic
Variables
Usually several raters independently judge prox-
emic behaviors on a number of criteria. Mehrabian
(1968) reported interrater correlations ranging from
.55 (eye contact) to .99 (total distance). Remland,
Jones, and Brinkman (1995) coded distance between
interactants’ torsos (Pearson’s r = .98) and heads
(Pearson’s r = .95) in inches estimated from video
recordings as well as from the coding system devel-
oped by Jones and Aiello (1973), which adjusts for
553
Dael et al.

CHART SHOWING INTERPLAY OF THE DISTANT & IMMEDIATE RECEPTORS

IN PROXEMIC PERCEPTION
FEET 0 1 2 3 4 5 6 7 8 10 12 14 16 18 20 22 30
INFORMAL DISTANCE INTIMATE PERSONAL SOCIAL–CONSULTIVE PUBLIC
CLASSIFICATION CLOSE NOT CLOSE CLOSE NOT CLOSE CLOSE NOT CLOSE
MANDATORY RECOGNITION DISTANCE BEGINS HERE NOT CLOSE BEGINS AT 30’–40’

KINESTHESIA HEAD, PELVIS, THIGHS TRUNK CAN BE BROUGHT INTO CONTACT OR MEMBERS CAN ACCIDENTLY TOUCH. HANDS CAN REACH & MANIPULATE ANY PART OF TRUNK EASILY
HANDS CAN REACH AND HOLD EXTREMITIES EASILY BUT WITH MUCH LESS FACILITY THAN ABOVE. SEATED CAN REACH AROUND & TOUCH OTHER SIDE OF TRUNK. NOT SO CLOSE AS TO RESULT IN ACCIDENTAL TOUCHING
ONE PERSON HAS ELBOW ROOM.

2 PEOPLE BARELY HAVE ELBOW ROOM. ONE CAN REACH OUT AND GRASP AN EXTREMITY.

JUST OUTSIDE TOUCHING DISTANCE.

2 PEOPLE WHO’S HEADS ARE 8’–9’ APART CAN PASS AN OBJECT BACK & FORTH BY BOTH STRETCHING.
OUT OF INTERFERENCE DISTANCE BY REACHING ONE CAN JUST TOUCH THE OTHER.

Figure 22.2.  Interpersonal distance measured as a function of ranges of reaching and touching the interlocutor.
From Current Anthropology (p. 93), by S. Tax, 1968, Chicago, IL: University of Chicago Press. Copyright 1968 the
University of Chicago Press. Reprinted with permission.

height differences (Cohen’s κ = .76 for heads and for system, including adaptor or manipulator actions,
torsos). They coded six types of interpersonal touch described later in the Kinesics section). It becomes
depending on the body parts used and duration apparent in the ensuing review that reliability mea-
of touching (intercoder reliability not indicated). sures differ considerably for every study, and its
Based on prior work on touch (Jones & Yarbrough, calculation is often not specified. Thus, we caution
1985; Weiss, 1992), Hertenstein and colleagues readers that this can render direct comparisons
(Hertenstein, Holmes, McCullough, & Keltner, misleading.
2009; Hertenstein, Keltner, App, Bulleit, & Jaskolka, Hayduk (1983) and Aiello (1987) have offered
2006) developed a coding system that grouped spe- extensive reviews of proxemic research with a focus
cific types of touch (such as squeezing, stroking, on the methodology for assessing spatial behavior
rubbing, poking, hitting, tickling, etc.) to investi- still used today (Sommer, 2002). A distinction is
gate their effectiveness in communicating specific generally made between projective and real-life
emotions to others. Reported intercoder agreement techniques. Projective techniques involve paper and
(statistical test not specified) has ranged from .83 pencil drawing tests or placing miniature figures
to .99. Touching as a proxemic variable as in these representing humans in various imagined interac-
studies is other-focused, though it should be noted tion settings. Real-life techniques involve asking
that touching behavior occurring on one’s own body people to move or position themselves in some way
(self-focused) or on objects has equally been studied and monitoring their response. For example, one
extensively in the field of nonverbal communica- technique involves asking participants to approach
tion, notably with regard to the emotional state of another person until the point where one starts to
the person performing the touching behavior (see feel uncomfortable (indicating violation of appropri-
Ekman and Friesen’s, 1969, 1972, functional coding ate distance). Another real-life paradigm involves

554

a confederate who approaches the participant until
the participant asks the confederate to stop when
closeness becomes uncomfortable. Other real inter-
action scenarios involve spatial arrangements in
which participants are asked to select or displace a
seat. Finally, qualitative (e.g., E. T. Hall, 1974) or
quantitative (Costa, 2010; Remland et al., 1995)
observations can be made from unobtrusive video
recordings. Because correlations between projective
and real-life measures are found to be low, Hayduk
and Aiello both argued for the use of real-life mea-
sures. They ascribed the lack of correspondence
between the two types of techniques to distance dis-
tortion in projective techniques; Subjects may have
difficulty adopting an external visual perspective and
mentally rescaling the size of the interaction setting.
Proxemic variables have also been studied from
body positions other than sitting. For example,
Costa (2010) analyzed group walking formations in
city streets. Remland et al. (1995) investigated dif-
ferences in distance, orientation, and touch between
standing dyads in cities across different countries.
Variations in proxemics have been explained as
a function of personal factors—such as sex, age,
personality, and cultural background—but also as
a function of situational context—such as conversa-
tion topic, environmental setting, as well as acquain-
tance to and affective judgment of the interaction
partner (see the general overview in Knapp, Hall, &
Horgan, 2014; see also Chapter 8, this handbook).
Human Interaction in Virtual Reality
Human interaction moves more and more into vir-
tual space through the use of cell phones, Internet,
and human–computer interfaces such as virtual real-
ities or games using agents or avatars that embody
the interactants. This opens up a new area of investi-
gation of how proxemic behavior is affected by these
new media (Sommer, 2002). It also offers exciting
new ways to experimentally control and manipulate
several nonverbal behavior variables that would oth-
erwise be hard to achieve even in the laboratory (see
Blascovich et al., 2002, for a review). To investigate
the effect of mimicry on impression formation in a
virtual touch interaction, for example, Bailenson,
Yee, Brave, Merget, and Koslow (2007) used a force
feedback joystick to objectively describe and mimic
Measuring Body Movement
participants’ handshakes. In a study of interpersonal
distance implemented in virtual reality, Bailenson,
Blascovich, Beall, and Loomis (2003) investigated
the effect of a virtual man and woman and his or
her gaze on the shape of the personal space bubble
around the virtual human drawn by participants’
walking patterns. Besides being able to measure
the exact distance and orientation from the motion
patterns (see Figure 22.3), such studies are able to
disentangle particular confounds such as the typical
one between gender and size, namely, that greater
interpersonal distance between men compared to
women could be due to the fact that men have larger
body sizes than women.
The increasing quality of immersive virtual
environments could reduce the traditional trad-
eoff between ecological validity and experimental
control (Blascovich et al., 2002). Considerable
research efforts have been allocated to identify-
ing the elements that a virtual environment must
include to create “presence” in the participant.
The concept of presence is closely related to
ecological validity following its definition as the
Distance (m)
Distance (m)
Figure 22.3.  Illustration of participants’ walking
patterns around a virtual human. From “Interpersonal
Distance in Immersive Virtual Environments,” by J. N.
Bailenson, J. Blascovich, A. C. Beall, and J. M. Loomis,
2003, Personality and Social Psychology Bulletin, 29, p. 825.
Copyright 2003 by Sage. Reprinted with permission.
555
Dael et al.
sense of being in a virtual environment rather
than the place in which the person’s body is actu-
ally located (Sanchez-Vives & Slater, 2005). Even
though the person knows that the events he or
she sees, hears, and feels are not physically real,
the person nevertheless thinks, feels, and behaves
as if they were really happening. According to
Sanchez-Vives and Slater (2005), “the phenom-
enon of presence is based on the transportation
of consciousness into an alternative, virtual real-
ity so that, in a sense, presence is consciousness
within that virtual reality” (p. 338). Elements such
as virtual body representation, display param-
eters such as frame-rate and head tracking, and
also other nonvisual channels, such as spatialized
sound and haptics, induce feelings of presence
and are thus maximized in experiments. Research-
ers typically measure presence by questionnaires
as well as from its behavioral and physiological
effects (e.g., increased heart rate or postural sway
when participants look over a virtual precipice).
The feeling of presence can be very strong, to the
extent where it can create an out-of-body-like illu-
sion (Lenggenhager, Tadi, Metzinger, & Blanke,
2007). Further, research on the effectiveness of
virtual reality therapy for anxiety disorders (e.g.,
fear of flying, heights, public speaking) suggests
that exposure treatment in virtual environments
can be equivalent to exposure “in vivo” in terms
of clinical improvement (Emmelkamp et al., 2002;
Parsons & Rizzo, 2008; Powers & Emmelkamp,
2008; Rothbaum, Hodges, Anderson, Price, &
Smith, 2002).
Though virtual environment is a relatively recent
and not yet widely available technique due to the
need of technical know-how and necessary fund-
ing, published reviews and the accumulation of
successful experiments do indicate its recognition
as a valid technique for behavior and communica-
tion research. Researchers studying proxemics in
particular can thus achieve fairly realistic percep-
tion and, importantly, an increasingly flexible use
and structuring of interpersonal and environmental
space. Accurate real-time physical movement track-
ing, indispensable for adequate rendering in virtual
environments, provides another methodological
advantage for researchers interested in the study of
556
body movement (see the computational techniques
described later).
KINESICS
Kinesics refers to the study of skeletal body move-
ment, that is, movements of the head, torso, and
limbs, excluding gaze and facial expressions. Bird-
whistell (1952, 1970), who originally coined this
term, developed one of the first coding systems
aimed at exhaustive symbolic transcription of body
movement, which he assumed to be hierarchically
structured following linguistic principles. Body
movement is segmented into kinemes, the most
elementary unit of behavior, much like phonemes
in speech. Kinemes combine into kinemorphs (ana-
lagous to morphemes), and further into larger units
of kinemorphic constructions. The descriptive units
mostly refer to static configurations of body parts
and regions that correspond to the anatomy of the
skeletal system. Birdwhistell’s system is a major
precursor for anatomically based coding systems,
but the system itself has not been widely adopted
in research, potentially because the linguistic anal-
ogy of the descriptors has been criticized (see a
more detailed survey in Rosenfeld, 1982). Another
historical theoretical framework and descriptive
system of body movement originates in dance
expression. Laban (1956, 1988) developed a system
of dance notation later referred to as Labanotation
(Hutchinson, 1961). Modeled after musical nota-
tion, Labanotation uses a complex collection of
symbols that provide highly detailed descriptions of
upper and lower body actions and positions. Little
used in its original format, the theoretical approach
underlying this descriptive system still drives mod-
ern methods developed in the fields of dance and
dance therapy (Lausberg, 1998), emotion research
(Gross, Crane, & Fredrickson, 2010), as well as
human–computer interfaces (Camurri, Mazzarino, &
Volpe, 2004; Chi, Costa, Zhao, & Badler, 2000).
Body Posture, Action, and Gesture
In kinesics, a distinction is usually made between
actions and postures. A posture represents the gen-
eral alignment of the body to a particular position or
configuration. It includes positions of the arms and

legs such as crossed, of the trunk such as leaning
backward, and whole-body positions such as lying
down. Postural movement shows periodic changes
known as posture shifts. An action is a local excur-
sion of one or a set of articulators (head, trunk,
arm, hand) outside a resting configuration with a
very discrete onset, a relatively short duration, and
a distinct offset where the articulator returns to a
resting configuration. Body actions are performed by
the head, the trunk, as well as the upper and lower
limbs and involve activities such as nodding, shrug-
ging, pointing, scratching, kicking, and so forth.
Gesture forms a specific group of body actions
performed mostly by the hands and arms. They usu-
ally but not exclusively occur during spoken interac-
tion. Usually a distinction is made between gestures
and other types of body movement based on the
defining feature that gestures are voluntary actions
that arise from the deliberative attempt to convey
semantic and pragmatic information, and they are
recognized as such by the viewer (for related dis-
cussions of gestures and their measurement, see
Chapters 12 and 19, this handbook). However, the
distinction between speech-related and speech-
unrelated actions is not strictly demarcated. Gesture
is rarely treated as an autonomous expressive system
in the field of kinesics, but it is usually included in a
broad range of communicative and regulatory body
movements with varying degrees of communicative
intention and connection to the verbal utterance
(see Harrigan, 2005).
Coding systems may focus more or less on pos-
ture, gesture, or action, or just body movement with-
out making a distinction between these movement
types. Body posture has been investigated extensively
in both kinesics and proxemics (discussed earlier;
e.g., Mehrabian, 1968) but has been defined in almost
as many ways as there are studies (see Harrigan,
2005). In addition, coding systems rarely provide
criteria for distinguishing movement as either part of
a posture or action (for notable exceptions, see Bull,
1987; Dael et al., 2012a; Harrigan & Carney, 2005;
see also Table 4.2 in Harrigan, 2005, p. 181).
Levels of Description
Researchers in kinesics have adopted various lev-
els of describing body movement (Harrigan, 2005;
Measuring Body Movement
Rosenfeld, 1982). In the next sections, we distin-
guish three large approaches and highlight recent
developments according to each approach. Cod-
ing systems may be focused on the description of
movement at a sociofunctional level based on an
empirical or conceptual framework (e.g., pointing),
at the form level based on an anatomical model of
the body (e.g., extended index finger), or noncat-
egorically, that is, from its spatio-temporal char-
acteristics (e.g., velocity). Some coding systems
integrate behavior descriptions based on different
levels of description. Recent examples include
(a) Bressem (2008), who proposed a hand move-
ment coding system that is mainly form based but
includes spatial coding and qualitative descriptors
related to the spatiotemporal flow of the move-
ment; (b) Lausberg and Sloetjes (2009), who com-
bined functional descriptors with Laban-inspired
spatiotemporal ones for describing cospeech
hand gestures; (c) Wallbott (1998), whose coding
system offers a mix of both functional and form
related categories of movement and even a few
movement qualities (e.g., activity, expansiveness,
energy); and (d) Dael et al. (2012a), who followed
a structured approach providing a parallel descrip-
tion of movement in terms of its function, anatomi-
cal articulation, and form.
Functional descriptions.  Actions and postures
can be assigned a functional label or code, such as
scratching or pointing. Based on the seminal work
of David Efron (1941) on conversational hand
movement, Ekman and Friesen (1969, 1972) devel-
oped a typology of classes and subtypes of hand
actions depending on their socioaffective function.
It distinguishes between illustrators (actions that
accompany, illustrate, or accentuate the content,
rhythm, flow, or intonation of verbal utterances),
emblems (conventionalized actions with a cultur-
ally shared and fixed meaning), manipulators or
adaptors (actions occurring on or against the body
that are first learned as part of an effort to satisfy
self needs or body needs and other functions such
as managing emotions), and regulators (actions
that regulate turn-taking of speaking and listening
between interlocutors). Their typology has been
developed for the study of hand movement, but
557
Dael et al.
their definition is not confined to hand movement,
as these functions can be served in principle by any
body part (e.g., biting one’s lip, illustrative head
movement, deictic eye gaze). Friesen, Ekman, and
Wallbott (1979) reported high intercoder agreement
of three of these major classes of actions, namely,
emblems, illustrators, and manipulators. The aver-
age kappa coefficient of hand action classification
was .70, and the average rank-order correlation on
the frequency and duration of a class of hand activ-
ity ranged between .85 and .98. This typology has
been widely adopted in body movement research
(see Harrigan, 2005; Scherer & Wallbott, 1985), and
it has provided a platform for studies on cultural dif-
ferences in emblematic gestures (Calbris & Doyle,
1990; Ekman, 1976; D. Morris, 1979; to list just a
few), emotional expression (e.g., Dael, Mortillaro, &
Scherer, 2012b; Wallbott, 1998), deception (e.g.,
Caso, Maricchiolo, Bonaiuto, Vrij, & Mann, 2006;
see DePaulo et al., 2003, for a meta-analysis), psy-
chotherapy (Harrigan & Rosenthal, 1986), and
clinical diagnosis (Ekman & Friesen, 1974), as
well as language production and comprehension
(Alibali, Kita, & Young, 2000; Goldin-Meadow,
2003; Gullberg & Kita, 2009; Rauscher, Krauss, &
Chen, 1996). More detailed typologies have evolved
from this approach that make explicit assumptions
on the dependency of the action function to spoken
language (Duncan, McNeill, & McCullough, 1995;
Kendon, 2004; McNeill, 2005; see also Chapter 12,
this handbook).
Functional movement categories are generally
assumed to be mutually exclusive and binary; the
movement does or does not fulfill that function. In
some cases, however, explicit claims are made that
actions can be multifunctional (e.g., iconicity com-
bined with deixis) and can thus belong to several
categories in different degrees (Dael et al., 2012a;
McNeill, 2005). Functional coding systems involve
a higher level of inference (about the function and
the purpose of a behavior) than pure descriptive cod-
ing systems (capturing form-related aspects of the
movement). For example, coding an extended index
finger as a deictic or pointing movement (subtype
of illustrator) assumes a referencing function to an
object, person, event, or location in real or imagined
space. Given their physical foundation, form-based
558
descriptions tend to be more reliable than functional
ones (e.g., Dael et al., 2012a). Irrespective of the level
of description, however, reliability decreases with
lower frequency of occurrence, increasing complex-
ity of the observed behavior
and with the level of detail or resolution required
(temporal and spatial; see Dael et al., 2012a).
Anatomically based form descriptions. 
Descriptions of movement form such as direction
and orientation are founded in anatomical kinesiol-
ogy (Hinson, 1977; Neumann, 2002). The moving
body part is described with respect to three orthogo-
nal, Cartesian axes (sagittal, vertical, and transver-
sal/horizontal) that are defined relative to a baseline
position of the body. Birdwhistell (1952) and Laban
(1975) pioneered the development of body move-
ment description based on common anatomical
models. These early low-level coding systems pro-
vide highly detailed descriptions but are known to
be very lengthy considering the large number of
movable body parts and interdependencies between
related joints. For coding to be nevertheless feasible,
contemporary anatomically based coding systems
tend to focus on selected body parts.
Several form-based coding systems have been
proposed for describing upper body movement.
These include the trunk and upper arms, while
lower limb movement such as leg and feet move-
ment are often little or not considered. The recently
developed Body Action and Posture coding system
(Dael et al., 2012a) provides 141 codes for the
time-aligned microdescription of body movement,
which is segmented in posture or action units
(see illustration of the graphical coder interface
in Figure 22.4). Although some of these codes
refer to functional categories and gaze behavior,
most of them refer to visible articulations of the
neck (head), trunk (spine and chest), upper limbs
(shoulders, elbows, wrists, and fingers), and lower
limbs (knees). The direction of the articulated body
part is defined as moving along one or more of the
three orthogonal axes relative to the anatomical
standard position (body upright standing). Defini-
tions are designed to be physically straightforward;
for example, the action code “right head tilt” is
defined as a rightward rotation of the head around
 Measuring Body Movement

Figure 22.4.  Illustration of the graphical coder interface of the Body Action and Posture (BAP) coding system.
From “The Body Action and Posture Coding System (BAP): Development and Reliability,” by N. Dael, M. Mortillaro,
and K. R. Scherer, 2012, Journal of Nonverbal Behavior, 36, p. 109. Copyright 2012 by Springer. Reprinted with
permission.

its sagittal axis. Single codes refer to minimal units
of action that can overlap. Orientation of the head
and trunk is coded continuously along an external
reference frame—as either facing or averted from
the (imagined) interlocutor standing in front of
the person. Intercoder reliability for each category
was established using Cohen’s kappa coefficient at
the occurrence level and at three levels of temporal
precision (within matching time-windows of 40
ms, 200 ms, and 440 ms). The latter approach has
also been used to assess the reliability of timing
FACS-coded action units (Sayette, Cohn, Wertz,
Perrott, & Parrott, 2001), and it has generally been
suggested as a solution to the problem of assess-
ing temporal agreement on differently segmented
behavioral streams (Bakeman & Gottman, 1987).
The Body Action and Posture coding system has
been used for identifying the bodily correlates of
various expressed emotions but is not limited to the
study of emotion, as the description is aimed to be
inherently atheoretical (similar to FACS).
Other systems target the description of hand
and finger movement. For example, Bressem (2008;
Ladewig & Bressem, 2013) reviewed methods for
describing cospeech hand movements from a lin-
guistic-semiotic perspective but argued to separate
the description of gesture from the accompanying
speech analysis to have independent accounts of
the gestural form and its functional meaning. This
allows applications beyond speech interactions (see
Chapter 12, this handbook) and is thus discussed
here. The proposed coding system offers a detailed
and purely form-based description of hand move-
ment in terms of the four parameters originally
proposed in sign language (Battison, 1974; Stokoe,
1960): hand shape, orientation, spatial position,
and movement. Hand shape coding falls back on the
HamNoSys standard German sign language coding
system (Prillwitz, 1989), differentiating four basic
categories: fist, flat hand, single fingers, and combi-
nations of fingers. After identifying the most promi-
nent feature of the hand shape, further distinctions

559
Dael et al.

are made regarding the number and shape of the
fingers. The orientation of the hand is coded as
originally proposed by McNeill (1992) with only a
few further differentiations, taking into account the
angle of the palm (e.g., up) in relation to the ges-
ture space of the speaker (e.g., toward the center).
Also the position of the hand is described following
McNeill’s (1992, 2005) notational system. Gesture
space is depicted as a two-dimensional (2D) grid of
concentric squares centered on the speaker’s upper
body in a sitting position (see Figure 22.5). The
hand on the lap, for example, is coded as located
in the “lower periphery (left/right).” Bressem adds
four broad divisions to capture the hand’s distance
from the own body along the depth dimension.
Hand movement coding is performed at three levels,
namely, type of movement, direction of movement,
and character of movement. Type of movement
identifies different movement shapes drawn by
either the arm and shoulder, wrist, and fingers, such
as arced arm movement or wrist rotation. The direc-
tion of these motion patterns can be coded along the
three major body axes (left/right, up/down, away
from/toward the body), with a few additional spe-
cific directions, such as circular. A limited number
of qualitative aspects, called character, of the move-
ment can be described with regard to speed (e.g.,
accelerated), size (e.g., enlarged), and flow (e.g.,
accentuated ending). This system has been used
in two internal studies showing recurrent gesture
structures across speakers and contexts, though
no indication of its reliability has been reported
(Bressem, 2008; Ladewig & Bressem, 2013).
By separating physically based form descriptions
from speech-dependent functional descriptions, this
approach could well be applied outside conversa-
tional settings, though no study has yet taken up
this challenge.

EXTREME
upper
PERIPHERY

upper right PERIPHERY upper left

CENTER

CENTER-
right left
CENTER

lower right lower left

Figure 22.5.  Gesture space is depicted as a two-dimensional grid of concentric squares

centered on the speaker’s upper body in a sitting position. From Hand and Mind: What Gestures
Reveal About Thought (p. 89), by D. McNeill, 1992, Chicago, IL: University of Chicago
Press. Copyright 1992 by the University of Chicago Press. Reprinted with permission.

560

Comprehensive, generic coding systems, such as
those mentioned earlier, tend to be labor-intensive
when integrally applied to a given material set.
They are, however, designed so that researchers
can select the behavioral categories and descrip-
tive levels according to their research interest. It
allows comparison across different studies that use
the same set of common behavior codes. Although
ideal for the time-aligned microdescription of body
movement, these tools lend themselves less well to
a more coarse, macrodescription of long behavior
sequences. More narrowly defined systems have
generally been proposed, tailored to a specific
research topic of type of body movement (see over-
view in Harrigan, 2005). A recent example is the
one developed by Tracy and Robins (2007) aimed at
identifying the most diagnostic cues for the recogni-
tion of pride. Their coding scheme lists a reduced
number of head, arm, and body codes assumed to
describe important cues related to pride and shame
(such as hands on hips, head tilted up, moving
hands to cover face, etc.; see Tracy & Matsumoto,
2008, for reported intercoder reliabilities).
While functional codes already establish mean-
ingful units of analysis, the researcher doing form-
based coding may not be interested in the raw
measurements per se but how the measures correlate
with other (higher order) variables such as personal
traits or states, or changes in the social environment.
Behavior described at the anatomical level thus needs
further processing (variable pooling, data reduc-
tion, clustering) to build psychologically meaningful
constructs. Although this presents an extra burden
to the researcher, it limits unmeasured subjective
interpretation at the data level. This is exactly the
approach used in the FACS. FACS coding produces
action units such as Action Unit 12 (AU12; lip corner
puller), which can be related to what observers may
label as a smile or to the expresser’s emotional inten-
tions (Ekman & Friesen, 1982). FACS describes
facial behavior beyond emotion, and according to
Ekman, Friesen, and Hager (2002), serves the goal
“to build the system free of any theoretical bias about
the possible meaning of facial behaviors” (p. 2).
FACS is a good example of an anatomically based
coding system by linking facial muscle movement to
visible change in facial appearance.
Measuring Body Movement
Spatio-temporal approaches.  Alternative to clas-
sifying movements according to a typology of forms
or functions, body movements can be described
along a number of dynamically changing character-
istics, that is, the way in which a movement is
performed with respect to time and space.
Labanotation (Hutchinson, 1961; Laban, 1956)
puts a particular emphasis on the description of the
temporal dynamics of body movement as it unfolds.
Movement qualities are referred to as aspects of
effort and shape. Effort generally describes how
exertion is concentrated during movement; it can
for example be forceful or light (“energy”), tense
or relaxed (“flow”). Shape refers to how the body
changes shape during the movement in terms of
the directional path, for instance, or the form of the
body itself (e.g., growing or shrinking of the torso).
Intercoder reliability of Laban-derived descrip-
tions has in the past either not been reported (see
Rosenfeld, 1982) or been found difficult to achieve,
perhaps due to the relatively high level of subjec-
tive interpretation involved. In one study (Lausberg,
1998), skilled therapists rated movement charac-
teristics of anorexic and other patient groups dur-
ing dance sessions using Laban movement analysis
rating scales (e.g., kinesphere, flow, strength). The
average Cohen’s kappa reached only .67, indicat-
ing that many of the scales caused considerable
disagreement. The recently developed Neuropsy-
chological Gesture Coding System (Lausberg &
Sloetjes, 2009) comprises a module for kinetic cod-
ing (related to the trajectory and dynamics of hand
movement) that is partly based on Laban’s theory of
effort. Interrater agreement was calculated for over-
lapping segments of irregular, phasic, or repetitive
movements and produced modified Cohen’s kappa
values ranging between .48 and .68 (see Holle &
Rein, 2013). Further, Gross and Crane (Crane &
Gross, 2013; Gross et al., 2010) reported an average
Cronbach’s alpha of more than .80 from nonexpert
ratings of Laban-derived effort features and shape
features of whole body emotional expressions.
These authors as well as others have translated
Laban’s effort–shape movement qualities to motion
captured kinematic parameters (Camurri, Lagerlöf, &
Volpe, 2003; Van Dyck, Maes, Hargreaves, Lesaffre, &
Leman, 2013).
561
Dael et al.
Descriptions of spatial and spatiotemporal move-
ment characteristics can also be obtained from
trained spatial coding or from motion tracking. The
spatial location of a hand movement relative to one’s
own body can be categorically coded using McNeill’s
(1992, 2005) gesture space grid (see Figure 22.5). A
movement located below the trunk on the right, for
example, would be coded as “periphery lower right.”
This coding system provides only spatial informa-
tion. By contrast, in the Bernese system (Frey, Hirs-
brunner, Florin, Daw, & Crawford, 1983; Frey &
Pool, 1976), the position of any body part is coded
at regular time intervals in numerical terms denot-
ing the amount of deviation of the body part from its
standard position (i.e., “upright and facing ahead”)
along the three Cartesian axes. On the basis of these
position–time series data, the number and detail of
spatiotemporal qualities that can be obtained with
the Bernese system are large and provide objective
indexes of what other researchers identify through
verbal labels, such as expansion, imitation, reaching
out, and so forth. Hand openness is, for example,
objectively measured by the lateral distances of the
hands from the trunk of the body. However, train-
ing and coding are considered very time-consuming,
which might explain why this system has little been
used in body movement studies. This approach has
been modernized by the computational analysis of x,
y, z coordinates based on motion tracking techniques
(Bente, Krämer, Petersen, & de Ruiter, 2001).
Modern techniques for motion tracking or motion
capture have provided an efficient way for the com-
putational extraction of various spatiotemporal
parameters from the movement trajectories, such as
speed and distance. These techniques allow for the
body to be represented as three-dimensional (3D)
according to Cartesian axes and/or Euler angles (i.e.,
the orientation of the body represented as rotations
around the x, y, z axes). Representing the body in
this way makes computing feature sets reasonably
straightforward. For example, in a study where this
technique was used to create point-light displays of
arm movement, people categorized the expressed
emotions based on temporal movement informa-
tion rendered by the point-light stimuli (Pollick,
Paterson, Bruderlin, & Sanford, 2001). Another
study on whole body kinematic analysis showed that
562
emotional expressions tended to be associated with
parameters such as elbow range of motion, elbow
extensor velocity, shoulder flexion, and torso exten-
sion (Gross et al., 2010).
Gait research in particular has much advanced
since the application of refined kinematic analyses
derived from motion tracking. Given the high dimen-
sionality of gait patterns, dimension reduction tech-
niques are typically applied to the data to model gait,
for example, principal component analysis (PCA;
Troje, 2002) and independent component analysis
(Ivanenko, Cappellini, Dominici, Poppele, &
Lacquaniti, 2005). Using a motion captured database
of gait patterns, Karg, Kühnlenz, and Buss (2010)
compared unsupervised techniques (PCA and kernel
PCA) with supervised techniques (linear discrimi-
nant analysis and general discriminant analysis) for
extracting relevant features based on joint angles
from gait patterns. Using the extracted feature set,
machine-learning algorithms are then used to clas-
sify the gait patterns according to affective state.
Results show that shoulder angle has a significant
influence on the recognized affective state. Moreover,
individuals could be automatically recognized by the
machine learning algorithms according to both affect
and walking style. Roether, Omlor, Christensen, and
Giese (2009) also used motion capture data of emo-
tionally expressive gaits in an effort to validate the
minimum set of spatiotemporal motor primitives that
drive the perception of particular emotions in gait.
A novel unsupervised learning algorithm was used to
build parameterized models with the goal of minimiz-
ing redundancy parameters. Spatiotemporal changes
in gait due to emotion could then be determined by
analyzing the changes in the parameters across mod-
els. In sum, various refined spatiotemporal movement
parameters have been identified related to meaningful
psychological variables such as emotions, gender, and
personalities (see Troje, 2002, for a review).
TECHNIQUES FOR MEASURING BODY
MOVEMENT
Different techniques for sampling and describing
body movement exist common to both kinesics and
proxemics. The decision should be guided by the
research question, that is, it should be sensitive

to and provide the most direct assessment of the
variable under investigation.
We first offer a few references for selecting an
appropriate sampling technique. We then distin-
guish two main approaches used in human obser-
vation studies, namely, expert coding and human
judgment. Next, following the rapid increase in
technologically assisted coding and analysis of body
movement, we introduce computational techniques
that have been used to track and extract body move-
ment features, which present a noteworthy contribu-
tion to the nonverbal communication research field.
Sampling Techniques
Several sampling techniques exist that are suitable
for different kinds of data handling and analyses.
Altmann’s (1974) widely cited review presents a
summary of the characteristics of seven major types
of behavior samples and their recommended uses
from mere exploration to estimation of frequency
and duration of behaviors. Bakeman and Gottman
(1987, 1997) explained the use of four basic strate-
gies for recording data with examples mainly from
infancy studies. Event coding is probably the most
common way of recording observational data allow-
ing flexible data analysis. The observer detects cod-
able events whenever they occur during the stream
of behavior and classifies the event with or without
time-stamps. The observer can also cross-classify
events on multiple dimensions defined in the cod-
ing scheme. For example, the observer interested
in conflict behavior within a group will not only
code the occurrence of conflicts to analyze their
frequency but may also code who was the initiator
and then classify the type of conflict according
to a predefined category set. As an alternative to
(cross-) classifying events, recording can be trig-
gered by predetermined times. In interval coding,
time is divided into a number of successive inter-
vals (usually a number of seconds). The observer
classifies each interval or codes whether certain
events occur during the interval. In time sampling,
a number of nonsuccessive time intervals are
categorized, and the percentage of time intervals
assigned a particular code is used to estimate the
proportion of time an individual devotes to that
kind of activity.
Measuring Body Movement
In the interest of reducing time spent coding,
Murphy (2005) tested the use of short behavior-
coded segments (“thin slices”) to predict the same
behavior across longer interactions. Correlations
between the short and longer segments of a few
general behavior categories were high except for
self-touch (see Table 22.1), demonstrating that thin
slices can be used to predict behavior in longer seg-
ments but strongly depend on the slice length and
type of interaction (where circumstances do not
fluctuate much). This paradigm was tested for five
types of behavior so far (see Table 22.1), and it can
be expected that certain behaviors, such as those
forming a pattern, will be more predictable than
others from short segments. The thin slice paradigm
has been used to predict various interpersonal out-
comes with moderate success in terms of reliability
(Ambady & Rosenthal, 1992). Since then, more
and more research is proving its effectiveness, and
more methodological considerations (such as the
location of the slice in the interaction sequence) are
improving the paradigm (e.g., Carney, Colvin, &
Hall, 2007).
Human Observation
Direct or human observation is still an important
“tool” in behavior research despite the growing
availability of computational techniques. Observa-
tions can differ hugely in terms of detail (spatial,
temporal, anatomical), focus (e.g., on the individual
or the interaction, encoding or decoding), standards
(behavior definitions), and amount of interpretation
allowed (the observer can be naive or trained).
Human judgment.  The judgment method is per-
haps the most commonly used method in studies of
nonverbal behavior (Rosenthal, 2005). Skilled (e.g.,
clinicians) or naive (e.g., students) judges view non-
verbal behaviors and are asked to rate that behavior
on a particular behavior attribute (e.g., movement
fluency, openness) or psychological construct (e.g.,
joy, warmth) based on a verbal label defined by
the experimenter. In a perception study on what
makes bodily expressions “emotionally salient” to
lay observers, perceptual ratings of spatio-temporal
characteristics of gesture, such as gesturing speed
and force, were found to significantly differentiate
563
Dael et al.

Table 22.1

Correlations Between Short and Longer Segments of Behavior Categories

Behavior 1-min slice Two 1-min slices Three 1-min slices

Slice(s) contained in 15-min interaction
Gaze .73** .81** .91**
Gesture .95 **
.91** .92**
Nod .79 **
.75** .69**
Self-touch .41 .76** .83**
Smile .62* .68* .76**
Mean r .79 .80 .84
Slice(s) removed from 15-min interaction
Gaze .64* .72** .85**
Gesture .93 **
.80** .77**
Nod .70** .60* .37
Self-touch .32 .68** .73**
Smile .52* .52* .56*
Mean r .68 .67 .68

Note. All participant ns = 15, except gaze (n = 13). Two 1-min slices represent the sum of behavior for two randomly
selected 1-min slices from the full 15-min interaction. Three 1-min slices represent the sum of behavior for three randomly
selected 1-min slices from the full 15-min interaction. Slices removed from the 15-min interaction refer to a part–whole
correlation between (J) and (W–J), where (J) = given slice length behavior total, and (W) = the sum of behavior in the
15-min interaction including J. Slices contained in the 15-min interaction refer to a correlation between (J) and (W).
Mean rs are weighted. From “Using Thin Slices for Behavioral Coding,” by N. A. Murphy, 2005, Journal of Nonverbal
Behavior, 29, p. 242. Copyright 2005 by Springer. Reprinted with permission.
*p ≤ .05. **p ≤ .01.

expressed emotions (Dael, Goudbeek, & Scherer,
2013). The aim of the study was not to identify the
objective or direct cues indicating emotion (as could
be obtained from kinematic analysis or from mea-
suring muscle activity) but rather whether such cues
can be accurately perceived, making them available
cues for emotion inference. One could alternatively
correlate behavior measurements (coded, rated, or
extracted from kinematic analysis; see the Automatic
Body Movement Processing section) to perceived
emotions, that is, attributions of the emotional
state to the expressor. Here, lay judges rate emo-
tional constructs based on assorted types of stimuli
often carefully constructed to differ along certain
variables of interest (posed expressions, point light
displays, inverted bodies, etc.). In using exactly this
approach, Pollick et al. (2001) found that attribu-
tions of emotional activation to instrumental arm
movement (knocking and drinking actions) were
positively related to temporal movement parameters
(velocity, acceleration, and jerk). Further, in an
experiment on emotion recognition from dynamic
bodily expressions under various visually deterio-
rated conditions (body segments or silhouette detail;
see Visch, Goudbeek, & Mortillaro, 2014), anger
was more robustly recognized than other emotions,
especially from the arm movement. This study sup-
ports the notion that perceivers have particularly
good recognition ability for threatening information
such as angry expressions in visually poor condi-
tions (e.g., in the dark).
Expert coding.  The distinction between judgment
and coding is not strictly demarcated, but the differ-
ence can be understood as the degree of subjective
inference required from the observer, being higher
in judgment and restricted in coding through the
use of standardized coding definitions.
Coding schemes provide explicit definitions
of the behaviors the investigator believes are rel-
evant for the topic of study. Bakeman and Gottman
(1987, 1997) systematically listed desirable features
of different types of coding schemes, such as the
use of mutually exclusive and exhaustive codes. In

564

addition, to improve transparency when compar-
ing behaviors across different studies, the behavior
definitions should ideally include discrimina-
tive inclusion criteria (e.g., what is considered an
action, posture, gesture) to avoid confusion with
other behavior categories within and across coding
schemes. Especially when timing or durations are to
be analyzed, definitions should include criteria for
behavior onset and offset times. In order to analyze
or exclude behaviors that may prove difficult to
code, it can help to include an option “not codable”
or to use a gradient coding style such as an ordinal
score for behavior salience or confidence rating.
Visual illustrations in addition to verbal behavior
descriptions seems to also greatly improve reliability
(e.g., Porter & ten Brinke, 2008).
A typical coding process consists of one or more
trained observers who follow a standardized cod-
ing protocol that is composed of fixed guidelines of
viewing, order of coding, and a clear set of defini-
tions of movement categories. Before actual data
gathering, observers go through an initial period
of training, after which they independently code a
common part of the material. This is followed by an
intercoder agreement test. The coders discuss areas
of disagreement and adjust or remove the coding
definitions where the reliability is too low. When
finally the reliability is considered sufficient, usually
one coder continues to code the target material and
provides the descriptive data for subsequent analysis.
Expert coding is increasingly facilitated by the
appearance of manual video annotation software
guiding standardized and precise time-locked cod-
ing, such as ANVIL (Kipp, 2001, 2014; e.g., Dael
et al., 2012a), ELAN (developed at the Max Planck
Institute for Psycholinguistics in Nijmegen; e.g.,
Lausberg & Sloetjes, 2009), and The Observer
(Noldus, Trienes, Hendriksen, Jansen, & Jansen,
2000). The coding scheme and manual can be speci-
fied within the software module or separately in
XML format, which is then implemented into an
intuitive graphical user interface and browsable
coding manual guiding systematic observation.
Furthermore, the XML format for data storage gives
maximum design flexibility to the investigator. Such
software often includes tools to search annotations
and to calculate reliability as well as various import
Measuring Body Movement
(e.g., speech transcriptions) and export (element
or frame-based text output) options for subsequent
statistical analysis.
Two examples illustrate how expert coding has
been used to investigate emotional expression. First,
after validation of the Body Action and Posture
coding system (Dael et al., 2012a), the coding sys-
tem was used to identify common patterns of body
movements shown by professional actors to express
a range of emotions (Dael et al., 2012b). Single
behavior categories clustered in higher order co-
occurring behavior compounds, which differentiated
emotions and also grouped similar emotions such as
pride and elated joy (characterized by symmetrical
up and down arm movement), pointed to an over-
all flexible, nonprototypical response mechanism
of emotion. Second, Tracy and Matsumoto (2008)
studied whether spontaneous pride and shame
expressions are cross-culturally displayed in response
to success and failure. Prototypical pride- and shame-
related behaviors were coded reliably based on a cod-
ing system developed for these particular emotions
(Tracy & Robins, 2007). Both sighted and (congeni-
tally) blind individuals shared these pride and shame
expressive behaviors, suggesting that these behaviors
may well be an innate biological behavioral response
to success and failure. Culture was found to moderate
the shame response, indicating emotion regulation in
accordance with cultural norms.
Reliability.  These types of observation imply not
only a different level of interpretation allowed from
the observer and hence a different explanatory
approach of the measured behavior (focus on decod-
ing or encoding; see the Correspondence Between
Behavior and Psychological Variables section) but
also different consequences in terms of reliability.
In judgment studies, usually several observers
rate the behavior. Adding more judges increases the
mean reliability, providing a more stable, that is,
reliable, estimation of the rating scale (Rosenthal,
2005). However, high interrater agreement may
indicate that a behavior or construct can be reliably
perceived across observers, but it does not imply
that observers correctly perceive this behavior or
construct (i.e., validity). Both correct and incorrect
judgments can be informative in decoding studies,
565
Dael et al.
as they inform researchers how people subjectively
perceive or interpret others’ behaviors.
Studies based on expert coding only include a
few observers who go through a period of training
or “calibration” to code behaviors as they are pre-
defined in a standardized coding scheme. Essential
here is replicability and maximized objectivity of
the obtained measurements, as ultimately they need
to provide a reliable and unbiased account of the
behavior as it was actually produced.
A study by Wallbott (1985) exemplifies this
nontrivial distinction between coding and judgment.
He compared parameters obtained from spatial
coding with judgments of hand movement quality
during admission and discharge interviews of psy-
chiatric patients diagnosed as primarily depressed.
The results indicated that lay judgments were reli-
ably but incorrectly used in the evaluation of a
patient’s psychopathological state (as “before” or
“after” successful therapy), whereas the coded hand
movement parameters did not differ significantly
between factual therapy discharge or admission.
The most commonly used statistics for estab-
lishing reliability of behavior coding are Cohen’s
kappa coefficient (Cohen, 1960; Fleiss, 1981) for
categorical data and correlation coefficients for
data measured at an ordinal, interval, or ratio scale.
Cohen’s kappa deals with the major drawback of
simple percentage of agreement because the calcula-
tion corrects for chance agreement, thus yielding a
more accurate measure of reliability. Moreover, the
agreement matrix can be useful for detecting sources
of disagreement such as differences in observer
sensitivity (see also Bakeman & Gottman, 1987).
We further refer the interested reader to Rosenthal
(2005), who has offered a comprehensive overview
of reliability assessment in observational studies
using judgment or expert coding, including an
in-depth discussion on parameters influencing the
kappa coefficient.
Automatic Body Movement Processing
A major methodological contribution to the study
of human behavior and communication consists
of the increasing availability of systems for auto-
matic detection of body parts and tracking their
movements across space and time. Similar to the
566
spatio-temporal descriptive approach in kinesics,
various movement parameters can be extracted from
the resulting 2D or 3D movement trajectories. Here,
the “observer” no longer labels behavior directly by
visual comparison with a set of coding rules (cod-
ing scheme) but explicates the coding rules in a
set of algorithms for (semi-) automatic movement
detection. These tools alleviate the manual labor of
coding yet provide a more refined and reliable analy-
sis of body parts across space and time. Observer
interpretation is removed at the level of data gath-
ering and moves up at the level of evaluating the
algorithm or analyzing the resulting data. The level
of accuracy depends greatly on the type of sensors,
from the tracking and maintaining of the location of
each body joint to the tracking and maintaining of a
coarse area corresponding to the whole body. Their
calibration relies on comparative performance with
other tools or with human evaluation.
The field of automatic detection and recognition
of body movement is increasingly receiving atten-
tion as technology becomes computationally faster
and cheaper. A set of steps is necessary to accom-
plish this task: body tracking, segmentation, feature
extraction, learning, and classification. Although
this is an increasingly active field of research, each
of these steps is quite complex, and these steps
are still far from being fully solved, especially
in nonconstraint settings and in the case of
computer-vision-based systems. In this section, we
review first the different types of sensors used to
track body movement, and then we discuss some of
the approaches and challenges used to accomplish
the steps mentioned earlier.
Automatic tracking of body movements.  When
determining how to automatically track and rec-
ognize body movements, there are a number of
systems and methods that can be used. The most
notable are optical motion capture systems, elec-
tromechanical/electromagnetic motion capture sys-
tems, and markerless vision-based systems. Many
optical motion capture systems (e.g., Vicon and
OptiTrack) make use of infrared cameras (gener-
ally 8–12) to track the movement of retro-reflective
markers placed on the body. Optical systems provide
the 3D positions of each marker as output. The more

markers used, the more accurate the description of
the configuration. This technique is quite popular
in emotion perception research (e.g., Atkinson,
Tunstall, & Dittrich, 2007; Kleinsmith & Bianchi-
Berthouze, 2013; Pollick et al., 2001) and gait
research in particular (Roether et al., 2009; Troje,
2002). Electromechanical or electromagnetic motion
capture systems (e.g., XSens and Animazoo’s IGS
range) instead require the person to wear active sen-
sors (e.g., inertial sensors, accelerometers, or mag-
netometers) on various parts of the body. In the case
of full body capture, these sensors are generally inte-
grated in a suit for easy wearing. These active sen-
sors detect rotational or acceleration information of
the body segments on which the sensors are placed.
Less obtrusive, markerless vision-based systems use
video or web cameras to record movement, after
which the data are processed with image-processing
techniques to determine the position and orienta-
tion of the person’s body and also to extract body
parts. This technique has been used to detect human
actions (e.g., Camurri et al., 2000; Glowinski et al.,
2011) and to differentiate between real and fake
pain (Bartlett, Littlewort, Frank, & Lee, 2014). Troje
(2002) reviewed spatio-temporal pattern recognition
techniques to describe complex human motion such
as gait, with the aim to understand how information
about emotion, gender, personality, and so forth is
encoded in body movement. For the purpose of this
chapter, namely, the application of motion tracking
techniques in human interaction, we focus here on
the major advantages and disadvantages to using
each type of system.
Optical and electromechanical motion capture
systems offer the advantage of accuracy. Using these
systems, a precise numeric representation of the
body in a 3D space can be easily obtained—either
in terms of x, y, z coordinates or Euler rotations.
This allows for the human body to be represented
in varying degrees of detail, such as point-light dis-
plays or skeletons (Ma, Paterson, & Pollick, 2006).
According to Thomas, McGinley, Carruth, and
Blackledge (2007), the data from optical systems
are more accurate than electromechanical systems.
However, optical systems do require more manual
intervention in the reconstruction of the human
body due to marker occlusion problems. Hence,
Measuring Body Movement
they are less suitable for online recognition of body
actions or expressions. Another disadvantage of
optical motion capture systems is mobility. Typi-
cally, once these systems are in place, they are not
moved due to camera transportation and placement
difficulties as well as calibration issues, making it
difficult to capture body movement in real-life set-
tings. On the other hand, electromechanical and
electromagnetic motion capture systems are highly
portable, which makes them easily used in almost
any setting, indoors and outdoors.
Although mobility issues are not as significant
a problem for vision-based systems, environmental
conditions can pose some challenges. Often, varia-
tions in lighting conditions, skin color, clothing,
body part occlusion, touching, and so forth can
cause difficulties in automatically extracting, seg-
menting, and recognizing body movement from
complex scenes with vision-based systems. Marker
occlusion is also an issue with optical motion cap-
ture systems when the line of sight is interrupted
(e.g., a hand or an object is between the marker and
at least one of the cameras) or not easily discrim-
inable from other markers because of their close
position. In the case of electromagnetic motion cap-
ture systems, a drawback is that they can be sensi-
tive to magnetic fields.
Low cost and readily available new vision-based
motion capture systems have started to emerge over
the last few years, such as Microsoft’s Kinect, which
is a markerless vision-based sensor. These types
of motion capture systems make use of cameras
that project infrared patterns for depth recovery of
motion. An advantage of this latter type of system
together with traditional vision-based systems is
that they are not intrusive, that is, participants are
not required to wear suits or sensors, which makes
it a more naturalistic experience. Therefore, vision-
based systems lend themselves well to certain areas
of research, such as security and surveillance (Moes-
lund & Granum, 2001). Software libraries that allow
the tracking and extraction of the body skeleton (3D
joint positions) are available, which make the use of
this tracking system a popular choice even among
those who are not experts in computer vision. How-
ever, the tracking is still quite poor, and for a more
reliable performance, expertise in computer vision
567
Dael et al.
is needed to analyze the depth map captured by this
system.
Although these types of motion capture sys-
tems allow the tracking of the major body regions,
smaller articulations such as detailed hand and
finger positions are typically not possible to record.
Although there are motion-tracking gloves that can
be worn, they can be more intrusive and/or some-
times cannot be used in conjunction with full body
systems. A recent study by Oikonomidis, Kyriazis,
and Argyros (2011) solved some of these issues
by proposing a model-based approach to 3D hand
tracking using a Kinect sensor. Some commercial
products are also beginning to appear, such as Leap
Motion (https://leapmotion.com).
So far, we have highlighted systems aimed at
capturing full body movement data. However, the
use of single accelerometer-based systems—either
in handheld devices or dedicated, commercially
available systems placed directly on the body—may
also be used to capture kinematic information
about a person (e.g., Fitbit: http://www.fitbit.com/
uk; Empatica bracelet: https://www.empatica.com).
Various studies in the field of ubiquitous computing
(e.g., quantified self-movement; Kwapisz, Weiss, &
Moore, 2010; Long, Yin, & Aarts, 2009; Romera-
Paredes, Aung, & Bianchi-Berthouze, 2013) are
now exploring the possibility of using this informa-
tion, combined with other contextual information
(e.g., location through a global positioning system,
or daily schedule) to interpret the type of physi-
cal activity or emotional state of the person on the
move and to provide support to behavioral changes
(e.g., fitness, sustainable society) and physical
rehabilitation.
Body movement recognition: Feature extrac-
tion, segmentation, and classification.  Various
approaches have been investigated to perform
feature extraction. They can be divided into two
main categories—spatial action representation and
temporal action representation—and are gener-
ally used in different phases of the process (for an
in depth review, see Weinland, Ronfard, & Boyer,
2011). Spatial representation methods focus mainly
on extracting features related to body configuration
without taking into account the overall temporal
568
information of the action. Temporal representation
methods are generally used to model the tempo-
ral relations between spatial information, such as
the duration of an arm movement or the interde-
pendences between submovements composing an
action.
A first set of approaches falling into the spatial
representation methods attempts to reconstruct
from an image of a body its 2D or a 3D representa-
tion. Given the high degrees of freedom that the
body presents and the variability in shape and size,
these approaches are generally based on parametric
models calibrated to the context of use. The initial
approaches were based on a simplistic representa-
tion of the body (Marr & Nishihara, 1978; Rohr,
1994). Although simplistic representations are suf-
ficient for recognition of gross actions, they may
not always provide the necessary level of detail
(e.g., the way the trunk is bent characterizes dif-
ferent emotions). More recent approaches have
been proposed (Gavrila & Davis, 1996; Green &
Guan, 2004; Ikizler & Forsyth, 2007; Ramanan &
Forsyth, 2003) that make use of more complex
mathematical representations of the body. Other
techniques aim instead at tracking 3D trajectory
of important joints. More coarse representations
aim instead to just track body regions of interest
(Starner & Pentland, 1997).
Image-based model approaches have instead
focused on detecting the body area (e.g., its silhou-
ette; Wang & Suter, 2007) rather than identifying
the different parts of the body. The extraction of
the features to recognize the action performed is
based on the image features within this extracted
area. Although following a simpler approach, these
methods have shown similar results to the body
model-based methods. However, these methods are
of less interest if the aim is to support the annotation
of fine body parts for posthuman analysis, as it still
requires a lot of human intervention in splitting the
overall silhouette in its component before coding
can be automatically extracted. In addition, some
of the features extracted by these methods are not
always easily interpretable by humans (e.g., Motion
History Image and Motion Energy Image, where the
color of a 2D image pixel is used to summarize the
amount of movement or energy of the movement

that occurred over the entire video in the spatial
location represented by the pixel). Other types of
descriptors describing the shapes of the silhouette
have been also proposed (Carlsson & Sullivan, 2001;
Ogale, Karapurkar, Guerra-Filho, & Aloimonos,
2004), in particular for the recognition of emotional
expressions (Camurri et al., 2003). In this latter
case, the gross description of the body shape is pow-
erful information, especially in contexts in which
emotions are powerfully or stereotypically expressed
(e.g., theatre).
Most of these approaches work well when the
background is clearly different from the human
body (e.g., different from the person’s clothes;
Darrell & Pentland, 1993). However, the extrac-
tion of the silhouette from more complex or noisy
backgrounds is also a highly explored research
question. A promising approach is to use shape-
context models (Sminchisescu, Kanaujia, Zhiguo, &
Metaxas, 2005), but unfortunately, they may fail
in the case of partial view of the body (e.g., occlu-
sion due to an object or self-occlusions). Other
approaches use optical flow extracted from con-
secutive images (Laptev & Perez, 2007; Polana &
Nelson, 1992; Robertson & Reid, 2005). The limi-
tation here is that these methods are based on the
assumptions that changes in consecutive images
are mainly due to body movement where instead
they could be due to other factors such as lighting,
camera movements, or appearance of objects. Some
of these drawbacks are addressed by using hybrid
approaches—for example, by combing optical flow
with Gabor filters (e.g., Jhuang, Serre, Wolf, &
Poggio, 2007).
Even if these methods are simpler, they are still
affected by variability in body shape and size and
orientation of the body. One way to overcome this
problem with the earlier described approaches is
to make use of a large set of silhouette models to
account for this variability. A different set of meth-
ods instead remove this dependence from a body
model by using statistical methods to detect points
of interest that can be then used to infer the type
of movement. The main approach is to divide the
images in smaller regions independently of the body
parts or image coordinates. The recognition of the
action is based on the overall statistics computed
Measuring Body Movement
for each region. The advantage of these methods
is that they are independent of body detection.
These approaches generally use a bottom-up strat-
egy aimed at detecting first the points of interests
and then assigning each region to a selected set
of categories of interest according to the statistics
computed on the basis of the detected points. These
categories of interest can vary from high-level
categories indicating, for example, body parts to
less interpretable low-level categories describing
aspects of the image (Joshi, Goecke, Parker, &
Breakspear, 2013). The classification of the move-
ment is then based on the so-called bag of features,
for example, histogram of occurrence of these cat-
egories. For example, Laptev and Lindeberg (2003)
proposed an algorithm to detect points of interest
that are characterized by high-variation values in
space and nonconstant motion in time. By com-
bining their algorithm with a model of particular
actions (e.g., modeling the periodicity of the walk-
ing movement), they show how their algorithm
indeed detects meaningful events without strong
constraints of body detection. Some approaches add
structural information to maximize the relevance
of the extracted features. For example, Gilbert, Ill-
ingworth, and Bowden (2008) proposed the use of
compound features to exploit the relative position
between features. These approaches are becoming
very trendy, as they appear quite powerful and
can be applied to quite complex scenes (Weinland
et al., 2011).
Once the body has been reconstructed or body
features have been extracted, temporal approaches
are generally used to classify the type of movement.
A typical method used to recognize an action is to
model the temporal sequence as a graphical model.
Given the interesting results obtained in speech rec-
ognition, the most used approach is the hidden
Markov model (HMM). For example, the original
work by Yamato, Ohya, and Ishii (1992) used an
HMM to model the sequence of a particular action
by using binned silhouette features. The problem
with HMMs is that they are very good at modeling
sequential aspects of an overall movement but not at
modeling the movement of independent body parts
(Weinland et al., 2011). Although multiple HMMs
for different body parts could be used to address
569
Dael et al.
this limitation, more general approaches have been
proposed, such as dynamic Bayesian networks or
conditional random fields, with promising results
(Sminchisescu et al., 2005).
The approaches presented earlier require the
use of presegmented sets of data from a video or
a motion capture data sequence, that is, the set of
images or motion capture data that constitute an
instance of action or of expression to be classified.
The most common method used for segmen-
tation is to detect temporal discontinuities. For
example, Marr and Vaina (1982) suggested detect-
ing temporal discontinuities as local minima, that
is, where the movement primitives present clear
changes of directions in their trends. The inves-
tigation of segmentation by humans (e.g., Zacks,
Kumar, Abrams, & Mehta, 2009) has shown that
humans make use of temporal motion information
from the image, whereas coarse-grain segmentation
is more strongly based on top-down and contextual
analysis of the action. Given their low computa-
tional cost, various examples of temporal disconti-
nuity-based approaches have been proposed (Rui &
Anandan, 2000; Ogale et al., 2004).
A different technique is the sliding window. In
this case, the data are split into overlapping multiple
windows, and each window is separately classified.
The aim of the window classification is to determine in
which part of the sequence the action occurs. Hence,
the segmentation is dependent on the classification pro-
cess, and it is strongly based on the segmentation sam-
ples provided in the training (Weinland et al., 2011).
These methods are computationally expensive because
of the analysis and classification being performed for
each window. In addition, to increase performances,
various methods propose the use of different size win-
dows making the computation even heavier. The posi-
tive side of these methods is that they do not depend on
action models (Weinland et al., 2011).
As often higher level phenomena (e.g., discrimi-
nating between different types of laughter) present a
certain level of fuzziness, model methods have been
proposed not just to categorize the expressions or
the actions according to a certain class but to model
the probability distribution of how a body move-
ment is recognized from human observers (Griffin
et al., 2013).
570
Although the problem of automatic classifica-
tion of human actions is still far from being fully
solved, automatic classification schemes are starting
to be proposed as a way to substitute and improve
existing scales for the measurement of higher order
phenomena. For example, the work by Joshi et al.
(2013) aims at providing a tool to detect level of
depression by continuously tracking and analyz-
ing a patient’s body behavior. Similarly, computa-
tional tools have been proposed (Aung et al., 2013;
Olugbade, Aung, Bianchi-Berthouze, Marquardt, &
Williams, 2014) to detect protective behavior and
perceived pain level in self-directed physiotherapy
in chronic pain rehabilitation. Finally, by just using
minimalistic sensors (e.g., smartphone), classifica-
tion of activity is carried out by simply exploiting
kinematic features (e.g., amount of movement,
changes in accelerations) gathered by a gyroscope or
one accelerometer within the smartphone or small
portable gadgets (e.g., Fitbit) that people easily carry
with them (Kwapisz et al., 2010; Long et al., 2009;
Romera-Paredes et al., 2013), without having to
wear sensors on all body parts. Similarly, Singh et al.
(2014) uses the gyroscope integrated in the smart-
phone to detect the amount and quality of move-
ment of the body part to which the smartphone is
attached (e.g., amount and fluidity of trunk move-
ments) during physical rehabilitation. The detected
movements are fed back to the person in the form of
pleasurable sounds to provide an enhanced experi-
ence of one’s exertion with the aims of increasing
self-efficacy, compensating for altered propriocep-
tion, and improving awareness of compensatory
movements. Nijhar, Bianchi-Berthouze, and
Boguslawski (2012) found that the simple measure-
ment of arm movement made it possible to predict
the type of motivation (relaxing vs. competing) and
level of expertise of a person playing a full-body
tennis computer game.
Correspondence Between Behavior and
Psychological Variables
Unlike spoken language, body movement does
not form a “language” that is out there waiting
to be deciphered. With the exception of emblems
that have an invariant and distinct meaning (e.g.,
waving goodbye), body movement cannot be

translated as directly as verbal behavior because
its meaning or diagnostic value is best described
as probabilistic. Whereas the anatomical complex-
ity of the body affords a differentiated expressive
and signaling system, a functional meaning can
be expressed in different ways, and the same body
movement can convey different meanings depend-
ing on other factors within the same or other
modalities (co-occurring facial or vocal behavior,
language, culture, etc.).
The theoretical perspective of probabilistic
functionalism accounts for this variability in inter-
personal perception. According to this view, the
environment perceived is uncertain or probabilistic
regardless of how deterministic it may be in terms
of physical principles. To adapt, the individual has
to learn to employ probabilistic evidence to achieve
communicative goals and to make inferences about
the environment. Brunswik (1956) proposed a theo-
retical model of visual perception and formulated
several methodological principles that stem from
this basic notion of functional probabilism. His lens
model has been further adapted to model social per-
ception processes (Funder, 1995; Hammond &
Stewart, 2001; Scherer, 1978) and has been applied
to investigate the perception of intelligence and
personality traits (Gifford, 1994; Hirschmüller,
Egloff, Nestler, & Back, 2013; Scherer, 1978) as
well as transient emotional states from nonverbal
auditory cues (Juslin & Laukka, 2001). In one case,
the model has been applied in the domain of bodily
communication to investigate judgment accuracy
of patients’ psychopathological state on the basis of
hand movement cues (Wallbott, 1985). Common to
these studies is the aim to model how perceivers try
to infer a covert variable from a set of overt behav-
ioral cues that are uncertain and partly redundant.
The methodological approach here aims for a high
ecological validity by modeling natural (noisy, mul-
tidetermined) environments as opposed to the clas-
sic experimental approach of targeting a few isolated
variables in tightly controlled experimental settings.
The process of perception or social judgment is
illustrated in Figure 22.6 and can be formulated by
probabilistic relations between a social object in the
environment (encoded variable or criterion, “C,” on
the left hand side of Figure 22.6; e.g., the emotional
Measuring Body Movement
state of joy), a limited number of indicators (in the
center of Figure 22.6), and the perception or rep-
resentation of this object (attribution, “A,” on the
right hand side of Figure 22.6). In the example of
the nonverbal communication of joy, this emotion
is externalized, intentionally or purely reactive, in
the form of distal (objectively measurable) indicator
cues in the behavior of an organism (“D” in Figure
22.6; e.g., high velocity movement, arms stretched
sideways or symmetrical up-down arm movement).
These distal cues are perceived by an observer and
are subjectively represented as proximal indicator
cues or percepts (“P” in Figure 22.6; impressions of
motion activity, energy, etc.). Finally, these proxi-
mal cues are combined in the impression formation
process to result in an inference or attribution of the
sender’s state (emotional judgment of joy).
This conceptual framework highlights an essen-
tial aspect of nonverbal communication, namely,
that effective communication requires both encod-
ing and appropriate decoding and, thus, a “shared
code” between expressor and perceiver. Mismatches
between encoding and decoding can present valu-
able insights about communication mechanisms.
Encoding may take place without decoding, inform-
ing researchers about sources of information not
available to the unskilled observer. Decoding can
also be inaccurate, in which case researchers are
studying social stereotypes, poor detection skills,
or erroneous beliefs (e.g., Wallbott, 1985). The lens
model illustrates the crucial distinction between
expression (encoding) and perception (decoding)
processes by separating distal and proximal cues,
with distal cues being directly and proximal cues
indirectly related to the encoded variable, as the lat-
ter has passed through a potentially noisy channel in
the environment and through the perceiver’s recep-
tive and interpretative systems. Correlation coef-
ficients describe the measure of association between
(a) the social object in the environment and the
perceiver’s estimate of the social object (functional
validity operationalized as the accuracy coefficient;
see Figure 22.6), (b) the probabilistic diagnosticity
of the distal or proximal cues with either the social
object (association coefficients) or its estimate (utili-
zation coefficients), and (c) the distal and proximal
cues (representation coefficients).
571
Dael et al.

Functional validity

Perceptual Inferential
Externalization representation utilization
Phenomenal
level
Distal
Proximal
Trait/state indicator Attribution
percepts
cues

D1 P1

D2 P2
C A

Di Pj

Indicator Perceptual Attributional

Criterion value judgments
values judgments
Operational
level Association Representation Utilization
coefficients coefficients coefficients

Accuracy coefficient

Figure 22.6.  Modified version of the lens model (originally developed by Brunswik, 1956) show-
ing a probabilistic functional approach to nonverbal communication. From “Personality Inference
From Voice Quality: The Loud Voice of Extroversion,” by K. R. Scherer, 1978, European Journal of
Social Psychology, 8, p. 469. Copyright 1978 by Wiley. Adapted with permission.

The model also has some methodological impli-
cations with regard to the recommended technique
for measuring body movement, for instance. The
type of data gathered from judgment studies as
described earlier informs us about people’s percep-
tual representation of and inferences based on the
observed behavior. When the interest lies in describ-
ing how the actual behavior is produced, then distal
variables provide a direct assessment. The descrip-
tion of encoding processes in body movement thus
necessitates observer independent measurement,
either by trained expert coders focusing on purely
visible aspects of the movement (form, anatomi-
cal, spatial) and/or by computational techniques for
feature extraction. As such, body movement can be
regarded as a dependent variable, reflecting changes
in the encoder’s state, personality, status, and so
forth, or an independent variable to investigate its
effect on others’ behavior or inferences.
The challenge of assigning psychological
meaning to behavior is a burning topic in the
computational field, where automatic detection
performances are evaluated against an established
ground truth. In laboratory-controlled settings, the
encoded (independent) variable can be induced or
simulated using actors, for instance, and variations
in the behavior measurements are attributed to
different levels of this variable. However, in many
(especially field) studies, the encoded variable (e.g.,
expressed emotion) cannot be measured directly or
be experimentally controlled. With no “objective
ground truth” available, self-report or observer rat-
ings often provide the best, though likely erroneous,
approximation at hand. When the aim is to model
the emotion recognition process, observer ratings
provide an excellent “subjective ground truth” of
the decoded variable (e.g., attributed emotion).
Kleinsmith and Bianchi-Berthouze (2013) proposed
a number of ways to benchmark computational
models of affect recognition that deal with observer
disagreement, such as the use of multilabeling tech-
niques (Liscombe, Venditti, & Hirschberg, 2003;

572

Yannakakis, 2009) and crowdsourcing (Sheng,
Provost, & Ipeirotis, 2008).
FUTURE CHALLENGES
Moving forward means to step out of the single-
channel approach and into real-life settings. In the
sections that follow, we elaborate on these two areas
of method development and implementation. Con-
cretely, a first proposition is that body movement
research would benefit from more refined behavior
patterning based on reliable segmentation. Whereas
most accounts in the field of nonverbal communi-
cation are unimodal, sequential analyses of body
movement allow dynamic integration with other
behavior streams of the same or another person.
A second proposition is the continued development
and use of applications to study human interac-
tion in real-life settings. We here report on current
efforts in adapting computational techniques to
track and classify body movement and touch.
Segmentation of the Behavior Stream
Nonverbal behaviors rarely occur in isolation. Often
compounds of behaviors co-occur and are interde-
pendent within and across modalities (face, voice,
body), calling for integrative methods allowing the
identification of behavior compounds or patterns.
An increase of interest in multimodal integration
and coordination of behavior, driven by the need to
understand mechanisms of expression and dynamic
social interaction, has further pushed for time-
locked measurements instead of global frequency
analyses.
Several methods exist for analyzing behavior
coordination. Principal component analyses
and cluster analyses allow the detection of co-
occurring events, grouped as higher order behavior
components or clusters. For example, Scherer and
Ellgring (2007) combined facial, vocal, and bodily
measures to identify major unimodal- and multi-
modal clusters of behavior recurrent in the process
of emotion expression. These patterns of broad cat-
egories of co-occurring actions (e.g., arm stretched
sideways + mouth stretch) were defined on the level
of occurrence. Refined principal component and
cluster analyses have shown to be useful in detecting
Measuring Body Movement
higher order behavior organizations based on micro-
coded body movements (Dael et al., 2012b).
Defining the unit of measurement and its tem-
poral boundaries becomes of increasing importance
when frequency and timing are at stake, especially
in relation to other co-occurring internal and
external events, such as facial configurations or a
response from the interaction partner. FACS, for
instance, not only includes coding rules for deter-
mining the anatomical origin of visible movement in
the face but the definition of each action unit entails
a guided decision on its boundaries and its three
phases: onset, apex, and offset (Ekman & Friesen,
1978). This proves to be important in communica-
tion, as, for example, in distinguishing expressed
and perceived enjoyment from nonenjoyment smiles
(Frank, Ekman, & Friesen, 1993; Krumhuber &
Kappas, 2005).
Though body movement segmentation was a
major concern in early days (Birdwhistell, 1970),
segmentation rules have rarely been specified in
contemporary body movement research. Perhaps
with the help of graphical coding interfaces showing
a detailed timeline, time-locked coding can become
more feasible (Kipp, 2001; Lausberg & Sloetjes,
2009). For instance, all units coded with the Body
Action and Posture coding system (e.g., Dael et al.,
2012a) represent temporal behavioral segments.
On the basis of criteria for onset and offset times, a
behavioral event is described through one or mul-
tiple units that can overlap in time (e.g., right arm
action upward + forward). Because actions and pos-
tures have a different temporal structure, different
rules apply for segmenting action and posture units.
Once segmentation rules have been defined
within a coding system, whether they are based on
anatomical, formal, or functional features of the
movement, the reliability of detecting the tempo-
ral boundaries needs to be assessed. The practical
difficulty this presents in terms of calculation has
already been discussed by Bakeman and Gottman
(1987, 1997). Computing the kappa on time inter-
vals using increasing tolerance windows is one solu-
tion that has now been successfully adopted in facial
(Sayette et al., 2001) and body (Dael et al., 2012a)
coding. A more indirect measure that has been
adopted in body coding is the intercoder correlation
573
Dael et al.
of unit frequency within a longer total time segment
(Dael et al., 2012a). Further implementation of mea-
sures that compare the time-points of the segment
boundaries constitutes an important prerequisite
for advancing body movement segmentation. A reli-
able observation-based segmentation of meaningful
behavior units could further provide a basis for test-
ing convergent validity of computational algorithms
for boundary detection.
Behavioral Patterning
A next challenge following reliable segmentation of
body movement is the analysis of temporal patterns
of behaviors. Body movement unfolds in a coordi-
nated way, and this unfolding can be described in
terms of order (sequences) or timing of events.
Two closely related research areas have provided
leading methodological contributions. First,
Bakeman and Gottman (Bakeman & Gottman,
1997; Bakeman & Quera, 2011; Gottman & Bake-
man, 1979) have offered encompassing introduc-
tions to the analysis of sequential data originating
from any type of time-based interactions. Research
examples mainly stem from social interaction
studies (such as adult–child interaction) using
observational data. Second, Van Hooff’s (1982) fun-
damental survey of methods of behavior patterning
and sequential analyses shows that also ethologists
have long sought to analyze the complex organiza-
tion of behavior in several animal species. We here
offer a mere taste of the possibilities available to the
nonverbal researcher interested in behavior pattern-
ing within and across modalities. To our knowledge,
there is yet little body movement research adopting
such tools even though they would seem to be well
adapted to the analyses of movement coordination
or synchronization. Future research could thus ben-
efit from advancements made in these related areas.
A basic statistic to describe sequences is the tran-
sitional probability, which is a type of conditional
probability, namely, the probability with which a
particular target event occurred relative to another
given event with both events occurring at differ-
ent times. Different time lags are used to indicate
the displacement in time between target and given
event (target event occurring immediately after the
given event or after an intervening event, and so on).
574
Transitional probabilities for pairs of events can be
presented in transitional frequency or probability
matrices or diagrams that give a simple but straight-
forward picture of how events are sequenced in time.
How should one interpret probabilities of event
sequences? A pair of behaviors are said to be related
if their co-occurrence exceeds what can be expected
from chance. Standard chi-square analysis tests an
observed distribution of co-occurrences to the dis-
tribution that is expected under the null-hypothesis,
that is, that each behavior is independently and
randomly distributed with its observed average fre-
quency. However, often the assumptions for using
the chi-square are violated in behavior research
(low numbers of observations; behaviors are not
independent from each other). Instead, various types
of z scores can be used to test whether the observed
transitional probabilities deviate significantly from
their expected values (Bakeman & Gottman, 1997),
but their computation generally requires sufficient
amounts of data. Recently, bootstrapping analy-
ses have been successfully used to determine the
frequencies expected by chance based on repeated
computer simulations of the observed distributions
(Yale et al., 2003; Yale, Messinger, Cobo-Lewis,
Oller, & Eilers, 1999). The authors adopted this
type of analysis to investigate the temporal coordina-
tion of infants’ vocalizations, facial expressions, and
gaze directions at their mothers’ faces based on the
observed versus expected frequencies of temporal
overlap between each pair of these behaviors. Four
types of overlap were considered, where one behav-
ior occurs entirely within the time span of the other,
or one behavior starts and ends before the other
behavior. For instance, the most frequently occurring
coordinated pattern in this study was a vocalization
beginning and ending within a smile or frown. Such
analyses could of course apply to any temporally
coded behavior including body movement. This
approach does, however, become cumbersome when
the aim is to analyze multiple behavior events and
their joint coordination as well as when examining
more complex types of temporal patterns.
The lag-sequential method is also used to
detect coordinated sequences but is a more flex-
ible approach to investigate sequences comprising
more than two events. Log-linear analysis promotes

a whole-table view compared to the narrow focus
of lag-sequential analysis on individual transitions,
thus avoiding the inflation of Type I error. It pro-
vides an integrated method for determining whether
there are effects at various lags, and is especially
useful when the number of transitions of interests
is high, such as in exploratory studies. Details on
the logic, computation, and restrictions of these and
other statistics for sequence detection depending on
the type of data and research aims can be found in
Bakeman and Gottman (1997).
Recently, Magnusson (2000) proposed a flexible
method for the bottom-up detection of temporal pat-
terns in behavior. A computer program called Theme
(Noldus Information Technology, n.d.) implements a
detection algorithm for different types of relationships
between pairs of events in a time series. So-called
T-patterns (see, e.g., Figure 22.7) are hierarchically
organized patterns of intra- or interindividual recur-
rent behaviors that not need to immediately follow
each other, thus allowing the detection of incomplete
patterns. Essential for the detection of T-patterns is
the search for the critical interval, that is, the smallest
possible time interval that is significantly greater than
expectation. This approach has been used in vari-
ous studies on social interaction (Castañer, Torrents,
Anguera, Dinusová, & Jonsson, 2009; Grammer,
Kruck, & Magnusson, 1998; Jonsson et al., 2006;
Kerepesi et al., 2005) and can be particularly useful
when the time-lag window is unknown.
Automatic Measurement of Body
Movement in Real-Life Settings
Automated motion tracking techniques gener-
ally require a high level of environmental control
(equipment, clothing, background, etc.) as well as
considerable postprocessing of the signal (noise
reduction). Therefore, a significant proportion of
research studies have focused on detecting body
movement in laboratory settings (De Silva &
Bianchi-Berthouze, 2004; Gunes, Piccardi, & Pantic,
2008; Kleinsmith, De Silva, & Bianchi-Berthouze,
2006; Shibata, Michishita, & Bianchi-Berthouze,
2013). Currently, efforts are being made for these
techniques to become feasible for the study of
human interaction in less constrained or real-
life settings. Both the miniaturization of existing
Measuring Body Movement
technologies and the advent of new types of cap-
turing technologies have helped to move motion
capture out of laboratory acting settings and into
real-world applications (Afzal & Robinson, 2009;
Ashraf et al., 2009; Landry & Pares, 2014; Savva,
Scarinzi, & Bianchi-Berthouze, 2012).
How can researchers capture body movements
in real life more easily and reliably, that is, what
types of sensors are feasible? Some of the sensor
technologies that have been successfully used in
the laboratory under controlled conditions may no
longer be feasible. For instance, optical motion cap-
ture systems are not practical outside of a controlled
environment due to set up constraints and camera
placement requirements. Instead, the focus needs to
be on unobtrusive and lightweight sensors that can
be placed directly on the body without requiring
cameras or significant amounts of additional equip-
ment. By placing the sensors directly on the body,
the wearer has the ability to move naturally any-
where and in any environment without constraints
(type and/or variations of the movements, emotion
portrayed, capturing environment, etc.).
Electromyography (EMG) is one type of data that
can be used to capture body movements. Although
muscle activation affects the way a movement is
performed, unfortunately these effects may not
always be easily detected through motion capture
systems and/or video cameras. The appearance of
ka b m n c p d q e r t a u b vc d s e w
Time
a b c d e a b c d e
Time
Figure 22.7.  Illustration of a hierarchically organized
pattern of recurrent behavior events (indicated by letters
positioned on a timeline). From a behavior sequence
depicted in the upper part of the figure, the computer
program Theme detects T-patterns, as depicted in the
lower part of the figure. Pairs of events form patterns
(a b) and (c d) on Level 1 when the interval between
the paired events occurs more often than expected by
chance (cf. the critical interval in Theme). These pat-
terns can combine with other patterns or events at
higher levels. Here, (a b) and (c d) form a pattern on
Level 2, which, in turn, forms a pattern with an event on
Level 3: {[(a b) (c d)] e}.
575
Dael et al.
fully wireless EMG systems (e.g., Noraxon and BTS
Bioengineering systems) offers less obtrusive ways
to measure such information, which can make the
recording more natural. The electrodes are small
and individually placed on the person’s body with-
out requiring cables. Although these systems have
generally been used for biomechanics, they have
been shown useful in the field of affective body
expressions, for example (Romera-Paredes et al.,
2013). Wireless EMG systems also provide a way to
measure affect on the move. Romera-Paredes et al.
(2013) explored the possibility of using machine
learning algorithms to predict EMG activity from
body movement during physical activity in people
with chronic pain to limit the types of sensors to be
worn, that is, lightweight, small, and unobtrusive
sensors.
Inertial sensor systems comprising accelerom-
eters and/or gyroscopes are also very useful as wear-
able sensors for body movement detection (Ermes,
Pärkka, Mantyjarvi, & Korhonen, 2008) in real-life
settings. According to Godfrey, Conway, Meagher,
and ÓLaighin (2008), “accelerometers have become
the preferred choice for continuous, unobtrusive
and reliable method in human movement detec-
tion and monitoring” (p. 1369). Accelerometers in
particular have been used to capture body move-
ments for decades (J. R. Morris, 1973). However, the
majority of body movement capturing with accel-
erometers took place in a laboratory setting due to
the bulkiness and high cost of the sensors (Godfrey
et al., 2008). Technological advances in recent years
have made accelerometers even more usable and
advantageous in real-world, full body movement
tracking without encumbering the wearer or being
too obtrusive. For instance, size, cost, and power
usage have been reduced, making them more acces-
sible than ever (Godfrey et al., 2008; Mathie, Coster,
Lovell, & Celler, 2004; Zhou & Hu, 2008).
Though full body tracking systems using multi-
ple accelerometers and/or gyroscopes have improved
and have become smaller and more compact in the
last decade (e.g., Animazoo’s IGS range of systems),
they still may be uncomfortable and may restrict
free movement due to either cabling connecting the
sensors (Bao & Intille, 2004) or the requirement of
a special suit to be worn within which the sensors
576
are embedded. Therefore, a lot of researchers are
now focusing on systems based on a single or a
minimal number of sensors for real-life settings (for
an in-depth review on activity recognition based on
wearable sensors, see Chen, Hoey, Nugent, Cook, &
Yu, 2012; Lara & Labrador, 2013). For instance,
the activPAL Professional from PAL Technologies
in particular has been used a lot in the medical field
to track physical activity levels of people from many
different populations, such as people with back
pain (Ryan, Gray, Newton, & Granat, 2007), older
adults (Egerton, 2004; Granat, Grant, & Dall, 2007;
Grant, Dall, Ryan, & Granat, 2007), and prosthetics
and lower limb amputees (Dumbleton et al., 2007;
Mendes, Simpson, Correia, & Macfayden, 2007;
Tang, Stansfield, Spence, Simpson, & Maxwell,
2007), to name a few.
Ideally, the move should be toward sensor sys-
tems embedded in clothing and accessories, for
example, belts, jewelry, and so forth. By embedding
sensors within items worn by users, several issues
could be solved. Clothing embedded with sensors
means multiple sensors could be used, providing a
more complete description of the body movements
performed and higher recognition performance.
Moreover, cabling between the sensors also would
be embedded and thus would not hinder the wearer
(Tognetti, Bartalesi, Lorussi, & De Rossi, 2007).
Embedding also allows for long-term monitoring
and solves the problem of obtrusiveness of wearing
a capturing system (Bonato, 2005). Johnson, Rog-
ers, van der Linden, and Bianchi-Berthouze (2012)
designed an Arduino-based jacket endowed with
accelerometers and a tactile actuator to help violin-
ist students to maintain a correct posture (elbow
and trunk position and arm extension) during
real-life music classes. Johnson, Bianchi-Berthouze,
Rogers, and van der Linden (2013) also explored
how making the self-calibration of the movement
detection sensors a responsibility of the user leads
to better learning achievements. This shows how
the decreasing cost of these sensors and their easier
integration into people’s clothes and devices are
leading toward being able to study behavior in
real-life, complex, uncontrolled settings with the
concomitant possible emergence of new research
questions.

Automatic Detection of Touch Behavior
Touch behavior has an important role in
human–human interaction such as the commu-
nication of affect (see the Proxemics section; e.g.,
Hertenstein et al., 2009). With the increased num-
ber of touch-based devices used in every aspect of
our lives, automatic detection of touch behavior
(e.g., for the purpose of affect recognition) has thus
started to attract increasing interest.
Early work in computing has explored touch as
a way to facilitate social interaction or as a way to
automatically recognize the emotional state of a per-
son. Following the seminal work of Clynes (1973),
research has been devoted to developing technology
that is able to capture and classify affective touch
behavior. Most of the work has been carried out
on purposely designed devices or keyboards and
on acted affective touch behavior (Bailenson et al.,
2007). Khanna and Sasikumar (2010), for example,
analyzed keyboard typing behavior and found that
the emotional state differently affected the frequency
of selection of certain keys (e.g., backspace) and
that in a positive mood their typing speed tended
to increase. Matsuda and Isomura (2010) investi-
gated finger behavior of deaf people to automati-
cally recognize their emotional state when using a
Braille device. Duration and acceleration of finger
dotting (a tactile communication media utilized
by deaf-blind individuals) appeared to discrimi-
nate between neutral, joy, sadness, and anger with
shorter duration in the joy condition and longer
duration in the sadness condition. Gao, Bianchi-
Berthouze, and Meng (2012) investigated natural-
istic touch behavior in touch-based game devices.
Their study showed that length, pressure, direc-
tion, and speed of finger swipe in game-play could
automatically discriminate between four reported
emotions. The position of the finger over the touch-
based devices was continuously tracked by software
purposely developed for the game, and the detection
of pressure was simulated by measuring the area of
contact of the finger.
FINAL WORD
In this chapter, we have illustrated the high diver-
sity in past and current methods for measuring
Measuring Body Movement
body movement as tools to study nonverbal com-
munication. Methods in proxemics and kinesics
are often used together even though they have
different historical origins and research foci. Body
movement research benefits from this nondogmatic
flexibility by allowing the researcher to choose
among many methods or to create new methods
tailored to answer a specific question. Yet, a com-
mon framework is necessary so that results from
different studies can be compared and can form a
coherent pool of knowledge. Methods and sampling
techniques have different strengths and shortcom-
ings, but the choice of method should be guided by
the research question, as, for instance, expert cod-
ing of body movement to study its diagnostic role in
expressing emotions. Perhaps as a consequence of
the large number of existing methods, measurement
reliability has become an established and indispens-
able criterion of evaluation and should thus be
specified in every study where a not yet validated
method is used.
The immergence of computational methods for
tracking and classifying body movement has produced
new ways of studying nonverbal communication. In
proxemics, virtual environments allow the precise
manipulation of interpersonal and spatial parameters
that cannot or can hardly be achieved in real environ-
ments. In kinesics, technologically assisted expert
coding is rendered more feasible enabling multimodal
and microcoding, for instance. Movement dynamics
have been historically hard to measure, but the current
surge of kinematic analyses is revealing meaningful
information regarding how and when the body moves,
walks, or even touches in different communication
settings. This creates new challenges, such as the
automatic detection of touch, but also allows exist-
ing challenges to be met more easily, such as accurate
behavior segmentation and patterning as well as the
study of human communication in nonobtrusive, real-
life settings. As sound methods are the motor of good
research, the study of body movement in communica-
tion is moving on rapidly, answering former questions
and bringing many more within reach.
References
Afzal, S., & Robinson, P. (2009). Natural affect
data—Collection and annotation in a learning
577
Dael et al.
context. In Proceedings of the Third International
Conference on Affective Computing and Intelligent
Interaction (pp. 1–7). http://dx.doi.org/10.1109/
ACII.2009.5349537
Aiello, J. R. (1987). Human spatial behavior. In D. Stokols &
I. Altman (Eds.), Handbook of environmental psychology
(Vol. 1, pp. 389–504). New York, NY: Wiley.
Alibali, M. W., Kita, S., & Young, A. J. (2000). Gesture
and the process of speech production: We think,
therefore we gesture. Language and Cognitive
Processes, 15, 593–613. http://dx.doi.org/
10.1080/016909600750040571
Altmann, J. (1974). Observational study of behavior:
Sampling methods. Behaviour, 49, 227–266. http://
dx.doi.org/10.1163/156853974X00534
Ambady, N., & Rosenthal, R. (1992). Thin slices
of expressive behavior as predictors of
interpersonal consequences: A meta-analysis.
Psychological Bulletin, 111, 256–274. http://dx.doi.
org/10.1037/0033-2909.111.2.256
Aristotle. (1924). Rhetorica (W. R. Roberts, Trans.).
In W. D. Ross (Ed.), The works of Aristotle
(pp. 1354–1420). Oxford, England: Clarendon Press.
(Original work published ca. 330 BC)
Ashraf, A. B., Lucey, S., Cohn, J. F., Chen, T., Ambadar,
Z., Prkachin, K. M., & Solomon, P. E. (2009).
The painful face—Pain expression recognition
using active appearance models. Image and
Vision Computing, 27, 1788–1796. http://dx.doi.
org/10.1016/j.imavis.2009.05.007
Atkinson, A. P., Tunstall, M. L., & Dittrich, W. H.
(2007). Evidence for distinct contributions of
form and motion information to the recognition of
emotions from body gestures. Cognition, 104, 59–72.
http://dx.doi.org/10.1016/j.cognition.2006.05.005
Aung, M. S. H., Romera-Paredes, B., Singh, A., Lim,
S., Kanakam, N., Williams, A. C., & Bianchi-
Berthouze, N. (2013). Getting rid of pain-related
behaviour to improve social and self perception:
A technology-based perspective. In Proceedings of
the 14th International Workshop on Image Analysis
for Multimedia Interactive Services (pp. 1–4). http://
dx.doi.org/10.1109/WIAMIS.2013.6616167
Bailenson, J. N., Blascovich, J., Beall, A. C., & Loomis,
J. M. (2003). Interpersonal distance in immersive
virtual environments. Personality and Social
Psychology Bulletin, 29, 819–833. http://dx.doi.
org/10.1177/0146167203029007002
Bailenson, J. N., Yee, N., Brave, S., Merget, D., & Koslow,
D. (2007). Virtual interpersonal touch: Expressing
and recognizing emotions through haptic devices.
Human–Computer Interaction, 22, 325–353.
Bakeman, R., & Gottman, J. M. (1987). Applying
observational methods: A systematic view.
578
In J. D. Osofsky (Ed.), Handbook of infant
development (pp. 818–854). New York, NY: Wiley.
Bakeman, R., & Gottman, J. M. (1997). Observing
interaction: An introduction to sequential analysis (2nd
ed.). Cambridge, England: Cambridge University
Press.
Bakeman, R., & Quera, V. (2011). Sequential analysis
and observational methods for the behavioral sciences.
http://dx.doi.org/10.1017/CBO9781139017343
Bao, L., & Intille, S. S. (2004). Activity recognition
from user-annotated acceleration data.
Pervasive Computing, 3001, 1–17. http://dx.doi.
org/10.1007/978-3-540-24646-6_1
Bartlett, M. S., Littlewort, G. C., Frank, M. G., & Lee, K.
(2014). Automatic decoding of facial movements
reveals deceptive pain expressions. Current
Biology, 24, 738–743. http://dx.doi.org/10.1016/
j.cub.2014.02.009
Battison, R. (1974). Phonological deletion in American
Sign Language. Sign Language Studies, 5, 1–19. http://
dx.doi.org/10.1353/sls.1974.0005
Bavelas, J. B., Chovil, N., Coates, L., & Roe, L. (1995).
Gestures specialized for dialogue. Personality and
Social Psychology Bulletin, 21, 394–405. http://dx.doi.
org/10.1177/0146167295214010
Bente, G., Krämer, N. C., Petersen, A., & de Ruiter,
J. P. (2001). Computer animated movement and
person perception: Methodological advances
in nonverbal behavior research. Journal of
Nonverbal Behavior, 25, 151–166. http://dx.doi.
org/10.1023/A:1010690525717
Birdwhistell, R. L. (1952). Introduction to kinesics.
Louisville, KY: University of Louisville.
Birdwhistell, R. L. (1970). Kinesics and context: Essays on
body motion communication. Philadelphia: University
of Pennsylvania Press.
Blascovich, J., Loomis, J., Beall, A. C., Swinth, K. R.,
Hoyt, C. L., & Bailenson, J. N. (2002). Immersive
virtual environment technology as a methodological
tool for social psychology. Psychological
Inquiry, 13, 103–124. http://dx.doi.org/10.1207/
S15327965PLI1302_01
Bonato, P. (2005). Advances in wearable technology and
applications in physical medicine and rehabilitation.
Journal of Neuroengineering and Rehabilitation, 2, 2.
http://dx.doi.org/10.1186/1743-0003-2-2
Bonifacio, G. (1616). L’arte de’ cenni [The art of signs].
Vicenza, Italy: F. Grossi.
Bressem, J. (2008). Notating gestures—Proposal for a
form based notation system of coverbal gestures.
Unpublished manuscript.
Brunswik, E. (1956). Perception and the representative
design of psychological experiments. Berkeley:
University of California Press.

Bull, P. (1987). Posture and gesture. Oxford, England:
Pergamon Press.
Bulwer, J. (1644a). Chirologia: Or, the naturall language of
the hand. London, England: Thomas Harper.
Bulwer, J. (1644b). Chironomia: Or, the art of manuall
rhetoricke. London, England: Thomas Harper.
Calbris, G., & Doyle, O. (1990). The semiotics of French
gestures. Bloomington: Indiana University Press.
Camurri, A., Hashimoto, S., Ricchetti, M., Ricci, A., Suzuki,
K., Trocca, R., & Volpe, G. (2000). EyesWeb: Toward
gesture and affect recognition in interactive dance and
music systems. Computer Music Journal, 24, 57–69.
http://dx.doi.org/10.1162/014892600559182
Camurri, A., Lagerlöf, I., & Volpe, G. (2003). Recognizing
emotion from dance movement: Comparison of spectator
recognition and automated techniques. International
Journal of Human–Computer Studies, 59, 213–225.
http://dx.doi.org/10.1016/S1071-5819(03)00050-8
Camurri, A., Mazzarino, B., & Volpe, G. (2004). Analysis
of expressive gesture: The EyesWeb Expressive
Gesture Processing Library. In A. Camurri & G.
Volpe (Eds.), Gesture-based communication in
human–computer action (pp. 460–467). Berlin,
Germany: Springer-Verlag.
Carlsson, S., & Sullivan, J. (2001). Action recognition by
shape matching to key frames. Retrieved from http://ftp.
nada.kth.se/CVAP/reports/ws_mod_ex_sull_car.pdf
Carney, D. R., Colvin, C. R., & Hall, J. A. (2007). A thin
slice perspective on the accuracy of first impressions.
Journal of Research in Personality, 41, 1054–1072.
http://dx.doi.org/10.1016/j.jrp.2007.01.004
Caso, L., Maricchiolo, F., Bonaiuto, M., Vrij, A., & Mann,
S. (2006). The impact of deception and suspicion
on different hand movements. Journal of Nonverbal
Behavior, 30, 1–19. http://dx.doi.org/10.1007/
s10919-005-0001-z
Castañer, M., Torrents, C., Anguera, M. T., Dinusová,
M., & Jonsson, G. K. (2009). Identifying and
analyzing motor skill responses in body movement
and dance. Behavior Research Methods, 41, 857–867.
http://dx.doi.org/10.3758/BRM.41.3.857
Chen, L., Hoey, J., Nugent, C. D., Cook, D. J., & Yu,
Z. (2012). Sensor-based activity recognition. IEEE
Transactions on Systems, Man, and Cybernetics, Part
C: Applications and Reviews, 42, 790–808. http://
dx.doi.org/10.1109/TSMCC.2012.2198883
Chi, D., Costa, M., Zhao, L., & Badler, N. (2000). The
EMOTE model for effort and shape. In Proceedings of
the 27th Annual Conference on Computer Graphics and
Interactive Techniques (pp. 173–182). http://dx.doi.
org/10.1145/344779.352172
Clynes, M. (1973). Sentography: Dynamic forms of
communication of emotion and qualities. Computers
Measuring Body Movement
in Biology and Medicine, 3, 119–130. http://dx.doi.
org/10.1016/0010-4825(73)90058-9
Cohen, J. A. (1960). A coefficient of agreement for
nominal scales. Educational and Psychological
Measurement, 20, 37–46. http://dx.doi.org/10.1177/
001316446002000104
Costa, M. (2010). Interpersonal distances in group
walking. Journal of Nonverbal Behavior, 34, 15–26.
http://dx.doi.org/10.1007/s10919-009-0077-y
Crane, E. A., & Gross, M. M. (2013). Effort-shape
characteristics of emotion-related body movement.
Journal of Nonverbal Behavior, 37, 91–105. http://
dx.doi.org/10.1007/s10919-013-0144-2
Dael, N., Goudbeek, M., & Scherer, K. R. (2013).
Perceived gesture dynamics in nonverbal expression
of emotion. Perception, 42, 642–657. http://dx.doi.
org/10.1068/p7364
Dael, N., Mortillaro, M., & Scherer, K. R. (2012a). The
Body Action and Posture coding system (BAP):
Development and reliability. Journal of Nonverbal
Behavior, 36, 97–121. http://dx.doi.org/10.1007/
s10919-012-0130-0
Dael, N., Mortillaro, M., & Scherer, K. R. (2012b).
Emotion expression in body action and posture.
Emotion, 12, 1085–1101. http://dx.doi.org/10.1037/
a0025737
Darrell, T., & Pentland, A. (1993). Space-time gestures.
In Proceedings of the 1993 IEEE Computer Society
Conference on Computer Vision and Pattern
Recognition (pp. 335–340). http://dx.doi.org/10.1109/
CVPR.1993.341109
Darwin, C. (1872). The expression of the emotions in man
and animals. http://dx.doi.org/10.1037/10001-000
DePaulo, B. M., Lindsay, J. J., Malone, B. E., Muhlenbruck,
L., Charlton, K., & Cooper, H. (2003). Cues to
deception. Psychological Bulletin, 129, 74–118. http://
dx.doi.org/10.1037/0033-2909.129.1.74
De Silva, P. R., & Bianchi-Berthouze, N. (2004).
Modeling human affective postures: An information
theoretic characterization of posture features.
Computer Animation and Virtual Worlds, 15, 269–276.
http://dx.doi.org/10.1002/cav.29
Dumbleton, T., Buis, A., McHugh, B., McKay, G., Murray,
K., & Sexton, S. (2007). Trans-tibial prosthetic
system design and benefits for the amputee, service
providers and society: An evidence based clinical
study. In Proceedings of the 12th World Congress of
the International Society for Prosthetics and Orthotics
(p. 108). Vancouver, British Columbia, Canada:
International Society for Prosthetics and Orthotics.
Duncan, S., McNeill, D., & McCullough, K.-E. (1995).
How to transcribe the invisible—And what we see.
In D. O’Connell, S. Kowal, & R. Posner (Eds.),
Zeichen für Zeit: Zur Notation und Transkription von
579
Dael et al.
Bewegungsablaüfen [Signs for time: Notation and
transcription of movements] (Vol. 18, pp. 75–94).
Tübingen, Germany: Gunter Narr Verlag.
Efron, D. (1941). Gesture and environment: A tentative
study of some of the spatio-temporal and “linguistic”
aspects of the gestural behavior of Eastern Jews and
Southern Italians in New York City, living under
similar as well as different environmental conditions.
New York, NY: King’s Crown Press.
Egerton, T. (2004). Physical activity of older adults—A
pilot study investigating actual upright mobility.
Australian Journal on Ageing, 23, A34–A36.
Ekman, P. (1976). Movements with precise meanings.
Journal of Communication, 26, 14–26. http://dx.doi.
org/10.1111/j.1460-2466.1976.tb01898.x
Ekman, P., & Friesen, W. V. (1969). The repertoire of
nonverbal behavior: Categories, origins, usage, and
coding. Semiotica, 1, 49–98.
Ekman, P., & Friesen, W. V. (1972). Hand movements.
Journal of Communication, 22, 353–374. http://dx.doi.
org/10.1111/j.1460-2466.1972.tb00163.x
Ekman, P., & Friesen, W. V. (1974). Nonverbal
behavior and psychopathology. In R. J. Friedman &
M. M. Katz (Eds.), The psychology of depression:
Contemporary theory and research (pp. 203–232).
Washington, DC: Winston & Sons.
Ekman, P., & Friesen, W. V. (1978). Facial Action Coding
System: A technique for the measurement of facial
movement. Palo Alto, CA: Consulting Psychologists
Press.
Ekman, P., & Friesen, W. V. (1982). Felt, false, and
miserable smiles. Journal of Nonverbal Behavior, 6,
238–252. http://dx.doi.org/10.1007/BF00987191
Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial
Action Coding System: Investigator’s guide (2nd ed.).
Salt Lake City, UT: Research Nexus.
Emmelkamp, P. M. G., Krijn, M., Hulsbosch, A. M., de
Vries, S., Schuemie, M. J., & van der Mast, C. A. P. G.
(2002). Virtual reality treatment versus exposure
in vivo: A comparative evaluation in acrophobia.
Behaviour Research and Therapy, 40, 509–516. http://
dx.doi.org/10.1016/S0005-7967(01)00023-7
Ermes, M., Pärkka, J., Mantyjarvi, J., & Korhonen, I.
(2008). Detection of daily activities and sports with
wearable sensors in controlled and uncontrolled
conditions. IEEE Transactions on Information
Technology in Biomedicine, 12, 20–26. http://dx.doi.
org/10.1109/TITB.2007.899496
Fleiss, J. L. (1981). Statistical methods for rates and
proportions. New York, NY: Wiley.
Frank, M. G., Ekman, P., & Friesen, W. V. (1993).
Behavioral markers and recognizability of the smile
of enjoyment. Journal of Personality and Social
580
Psychology, 64, 83–93. http://dx.doi.org/10.1037/
0022-3514.64.1.83
Frey, S., Hirsbrunner, H. P., Florin, A., Daw, W., &
Crawford, R. (1983). A unified approach to the
investigation of nonverbal and verbal behavior
in communication research. In W. Doise &
S. Moscovici (Eds.), Current issues in European
social psychology (Vol. 1, pp. 143–199). Cambridge,
England: Cambridge University Press.
Frey, S., & Pool, J. (1976). A new approach to the analyses
of visible behavior. Bern, Switzerland: University of
Bern, Department of Psychology.
Friesen, W. V., Ekman, P., & Wallbott, H. (1979). Measuring
hand movements. Journal of Nonverbal Behavior, 4,
97–112. http://dx.doi.org/10.1007/BF01006354
Funder, D. C. (1995). On the accuracy of personality
judgment: A realistic approach. Psychological Review,
102, 652–670. http://dx.doi.org/10.1037/0033-
295X.102.4.652
Gao, Y., Bianchi-Berthouze, N., & Meng, H. (2012).
What does touch tell us about emotions in
touchscreen-based gameplay? ACM Transactions on
Computer–Human Interaction, 19, 1–30. http://dx.doi.
org/10.1145/2395131.2395138
Gavrila, D. M., & Davis, L. S. (1996). 3-D model-
based tracking of human movement: A multi-view
approach. In Proceedings of the 1996 IEEE Computer
Society Conference on Computer Vision and Pattern
Recognition (pp. 73–80). http://dx.doi.org/10.1109/
CVPR.1996.517056
Gifford, R. (1994). A lens-mapping framework for
understanding the encoding and decoding of
interpersonal dispositions in nonverbal behavior.
Journal of Personality and Social Psychology, 66,
398–412. http://dx.doi.org/10.1037/0022-3514.66.2.398
Gilbert, A., Illingworth, J., & Bowden, R. (2008). Scale
invariant action recognition using compound
features mined from dense spatio-temporal corners.
In D. Forsyth, P. Torr, & A. Zisserman (Eds.),
Computer Vision—ECCV 2008 (pp. 222–233). http://
dx.doi.org/10.1007/978-3-540-88682-2_18
Glowinski, D., Dael, N., Camurri, A., Volpe, G.,
Mortillaro, M., & Scherer, K. R. (2011). Towards a
minimal representation of affective gestures. IEEE
Transactions on Affective Computing, 2, 106–118.
http://dx.doi.org/10.1109/T-AFFC.2011.7
Godfrey, A., Conway, R., Meagher, D., & ÓLaighin, G.
(2008). Direct measurement of human movement
by accelerometry. Medical Engineering and Physics,
30, 1364–1386. http://dx.doi.org/10.1016/
j.medengphy.2008.09.005
Goldin-Meadow, S. (2003). Hearing gesture: How our
hands help us think. Cambridge, MA: Harvard
University Press.

Gottman, J. M., & Bakeman, R. (1979). The sequential
analysis of observational data. In M. E. Lamb, S. J.
Suomi, & G. R. Stephenson (Eds.), Social interaction
analysis: Methodological issues (pp. 185–206).
Madison: University of Wisconsin Press.
Grammer, K., Kruck, K. B., & Magnusson, M. S.
(1998). The courtship dance: Patterns of nonverbal
synchronization in opposite-sex encounters. Journal
of Nonverbal Behavior, 22, 3–29. http://dx.doi.
org/10.1023/A:1022986608835
Granat, M. H., Grant, M., & Dall, P. M. (2007).
Quantification of activity patterns of the elderly
population in different care settings. In Proceedings of
the 4th World Congress of the International Society of
Physical Medicine and Rehabilitation
(p. 143). Seoul, Korea: International Society of
Physical and Rehabilitation Medicine.
Grant, M., Dall, P. M., Ryan, C. G., & Granat, M. H.
(2007). Validity of a novel activity monitor as
an objective measure of walking in older adults.
In Proceedings of the 4th World Congress of the
International Society of Physical Medicine and
Rehabilitation (p. 341). Seoul, Korea: International
Society of Physical and Rehabilitation Medicine.
Green, R. D., & Guan, L. G. L. (2004). Quantifying
and recognizing human movement patterns
from monocular video images—Part I: A new
framework for modeling human motion. IEEE
Transactions on Circuits and Systems for Video
Technology, 14, 179–190. http://dx.doi.org/10.1109/
TCSVT.2003.821976
Griffin, H. J., Aung, M. S. H., Romera-Paredes,
B., McKeown, G., Curran, W., McLoughlin,
C., & Bianchi-Berthouze, N. (2013). Laughter type
recognition from whole body motion. In Proceedings
of the 2013 HUMAINE Association Conference on
Affective Computing and Intelligent Interaction (pp.
349–355). http://dx.doi.org/10.1109/ACII.2013.64
Gross, M. M., Crane, E. A., & Fredrickson, B. L. (2010).
Methodology for assessing bodily expression of
emotion. Journal of Nonverbal Behavior, 34, 223–248.
http://dx.doi.org/10.1007/s10919-010-0094-x
Gullberg, M., & Kita, S. (2009). Attention to speech-
accompanying gestures: Eye movements and
information uptake. Journal of Nonverbal
Behavior, 33, 251–277. http://dx.doi.org/10.1007/
s10919-009-0073-2
Gunes, H., Piccardi, M., & Pantic, M. (2008). From
the lab to the real world: Affect recognition using
multiple cues and modalities. In J. Or (Ed.), Affective
computing: Focus on emotion expression, synthesis, and
recognition (pp. 185–218). http://dx.doi.org/
10.5772/6180
Hall, E. T. (1959). The silent language. New York, NY:
Doubleday.
Measuring Body Movement
Hall, E. T. (1963). A system for the notation of proxemic
behavior. American Anthropologist, 65, 1003–1026.
http://dx.doi.org/10.1525/aa.1963.65.5.02a00020
Hall, E. T. (1968). Proxemics. In S. Tax (Ed.), Current
anthropology (Vol. 9, pp. 83–108). Chicago, IL:
University of Chicago Press.
Hall, E. T. (1974). Handbook for proxemic research.
Washington, DC: Society for the Anthropology of
Visual Communication.
Hall, J. A., Coats, E. J., & LeBeau, L. S. (2005). Nonverbal
behavior and the vertical dimension of social relations:
A meta-analysis. Psychological Bulletin, 131, 898–924.
http://dx.doi.org/10.1037/0033-2909.131.6.898
Hall, J. A., Harrigan, J. A., & Rosenthal, R. (1995).
Nonverbal behavior in clinician–patient interaction.
Applied and Preventive Psychology, 4, 21–37. http://
dx.doi.org/10.1016/S0962-1849(05)80049-6
Hammond, K. R., & Stewart, T. R. (Eds.). (2001).
The essential Brunswik: Beginnings, explications,
applications. New York, NY: Oxford University Press.
Harrigan, J. A. (2005). Proxemics, kinesics, and gaze. In
J. A. Harrigan, R. Rosenthal, & K. R. Scherer (Eds.),
The new handbook of methods in nonverbal behavior
research (pp. 137–198). New York, NY: Oxford
University Press.
Harrigan, J. A., & Carney, D. R. (2005). A coding method
for body/head positions and actions: Laboratory
manual. Fullerton: California State University.
Harrigan, J. A., & Rosenthal, R. (1986). Nonverbal
aspects of empathy and rapport in physician–patient
interaction. In P. D. Blanck, R. Buck, & R. Rosenthal
(Eds.), Nonverbal communication in the clinical
context (pp. 36–73). University Park: Pennsylvania
State University Press.
Hayduk, L. A. (1983). Personal space: Where we now
stand. Psychological Bulletin, 94, 293–335. http://
dx.doi.org/10.1037/0033-2909.94.2.293
Hertenstein, M. J., Holmes, R., McCullough, M., &
Keltner, D. (2009). The communication of emotion
via touch. Emotion, 9, 566–573. http://dx.doi.
org/10.1037/a0016108
Hertenstein, M. J., Keltner, D., App, B., Bulleit, B. A., &
Jaskolka, A. R. (2006). Touch communicates distinct
emotions. Emotion, 6, 528–533. http://dx.doi.
org/10.1037/1528-3542.6.3.528
Hinson, M. M. (1977). Kinesiology. Dubuque, IA: W. C.
Brown.
Hirschmüller, S., Egloff, B., Nestler, S., & Back, M. D.
(2013). The dual lens model: A comprehensive
framework for understanding self–other agreement
of personality judgments at zero acquaintance.
Journal of Personality and Social Psychology, 104,
335–353. http://dx.doi.org/10.1037/a0030383
581
Dael et al.
Holle, H., & Rein, R. (2013). The modified Cohen’s
kappa: Calculating interrater agreement for
segmentation and annotation. In H. Lausberg (Ed.),
Understanding body movement: A guide to empirical
research on nonverbal behaviour (pp. 261–277).
Frankfurt, Germany: Peter Lang.
Hutchinson, A. (1961). Labanotation. New York, NY:
New Directions.
Hyniewska, S., Niewiadomski, R., Mancini, M., &
Pelachaud, C. (2010). Expression of affects in
embodied conversational agents. In K. R. Scherer,
T. Bänziger, & E. B. Roesch (Eds.), A blueprint for
affective computing: A sourcebook and manual (pp.
213–221). New York, NY: Oxford University Press.
Ikizler, N., & Forsyth, D. (2007). Searching video for
complex activities with finite state models. In
Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition (pp. 1–8). http://
dx.doi.org/10.1109/CVPR.2007.383168
Ivanenko, Y. P., Cappellini, G., Dominici, N., Poppele,
R. E., & Lacquaniti, F. (2005). Coordination of
locomotion with voluntary movements in humans.
Journal of Neuroscience, 25, 7238–7253. http://dx.doi.
org/10.1523/JNEUROSCI.1327-05.2005
Jhuang, H., Serre, T., Wolf, L., & Poggio, T. (2007). A
biologically inspired system for action recognition.
In Proceedings of the 11th IEEE International
Conference on Computer Vision (pp. 1–8). http://
dx.doi.org/10.1109/ICCV.2007.4408988
Johnson, R., Bianchi-Berthouze, N., Rogers, Y., & van
der Linden, J. (2013). Embracing calibration in body
sensing: Using self- tweaking to enhance ownership
and performance. In Proceedings of the 2013 ACM
International Joint Conference on Pervasive and
Ubiquitous Computing (pp. 811–820). http://dx.doi.
org/10.1145/2493432.2493457
Johnson, R., Rogers, Y., van der Linden, J., & Bianchi-
Berthouze, N. (2012). Being in the thick of
in-the-wild studies: The challenges and insights
of researcher participation. In Proceedings of
the SIGCHI Conference on Human Factors in
Computing Systems (pp. 1135–1144). http://dx.doi.
org/10.1145/2207676.2208561
Jones, S. E., & Aiello, J. R. (1973). Proxemic behavior
of Black and White first-, third-, and fifth-grade
children. Journal of Personality and Social Psychology,
25, 21–27. http://dx.doi.org/10.1037/h0034267
Jones, S. E., & Yarbrough, A. E. (1985). A naturalistic
study of the meanings of touch. Communication
Monographs, 52, 19–56. http://dx.doi.
org/10.1080/03637758509376094
Jonsson, G. K., Anguera, M. T., Blanco-Villaseñor, Á.,
Losada, J. L., Hernández-Mendo, A., Ardá, T.,
. . . Castellano, J. (2006). Hidden patterns of play
interaction in soccer using SOF-CODER. Behavior
582
Research Methods, 38, 372–381. http://dx.doi.
org/10.3758/BF03192790
Joshi, J., Goecke, R., Parker, G., & Breakspear, M. (2013).
Can body expressions contribute to automatic
depression analysis? In Proceedings of the 10th IEEE
International Conference and Workshops on Automatic
Face and Gesture Recognition (pp. 1–7). http://dx.doi.
org/10.1109/FG.2013.6553796
Juslin, P. N., & Laukka, P. (2001). Impact of intended
emotion intensity on cue utilization and decoding
accuracy in vocal expression of emotion. Emotion, 1,
381–412. http://dx.doi.org/10.1037/1528-3542.
1.4.381
Karg, M., Kühnlenz, K., & Buss, M. (2010). Recognition
of affect based on gait patterns. IEEE Transactions on
Systems, Man, and Cybernetics, Part B: Cybernetics,
40, 1050–1061. http://dx.doi.org/10.1109/
TSMCB.2010.2044040
Kendon, A. (2004). Gesture: Visible action as utterance.
Cambridge, England: Cambridge University Press.
Kerepesi, A., Jonsson, G. K., Miklósi, A., Topál, J.,
Csányi, V., & Magnusson, M. S. (2005). Detection
of temporal patterns in dog–human interaction.
Behavioural Processes, 70, 69–79. http://dx.doi.org/
10.1016/j.beproc.2005.04.006
Khanna, P., & Sasikumar, M. (2010). Recognising
emotions from keyboard stroke pattern. International
Journal of Computers and Applications, 11,
8887–8975.
Kipp, M. (2001). ANVIL—A generic annotation tool
for multimodal dialogue. In Proceedings of
INTERSPEECH (pp. 1367–1370). Geneva, Switzerland:
International Speech Communication Association.
Kipp, M. (2014). ANVIL: The video annotation research
tool. In J. Durand, U. Gut, & G. Kristofferson
(Eds.), The Oxford handbook of corpus phonology
(pp. 420–436). Oxford, England: Oxford University
Press.
Kleinsmith, A., & Bianchi-Berthouze, N. (2013). Affective
body expression perception and recognition: A
survey. IEEE Transactions on Affective Computing, 4,
15–33. http://dx.doi.org/10.1109/T-AFFC.2012.16
Kleinsmith, A., De Silva, P. R., & Bianchi-Berthouze, N.
(2006). Cross-cultural differences in recognizing
affect from body posture. Interacting With Computers,
18, 1371–1389. http://dx.doi.org/10.1016/
j.intcom.2006.04.003
Knapp, M. L., Hall, J. A., & Horgan, T. G. (2014).
Nonverbal communication in human interaction (8th
ed.). Boston, MA: Cengage Learning.
Koppensteiner, M., & Grammer, K. (2010). Motion patterns
in political speech and their influence on personality
ratings. Journal of Research in Personality, 44, 374–379.
http://dx.doi.org/10.1016/j.jrp.2010.04.002

Krumhuber, E., & Kappas, A. (2005). Moving smiles: The
role of dynamic components for the perception of the
genuineness of smiles. Journal of Nonverbal Behavior,
29, 3–24. http://dx.doi.org/10.1007/s10919-004-
0887-x
Kwapisz, J. R., Weiss, G. M., & Moore, S. A. (2010).
Activity recognition using cell phone accelerometers.
ACM SIGKDD Explorations Newsletter, 12, 74–82.
http://dx.doi.org/10.1145/1964897.1964918
Laban, R. (1956). Principles of dance and movement
notation. London, England: MacDonald & Evans.
Laban, R. (1975). Principles of dance and movement notation
(Vol. 2). London, England: MacDonald & Evans.
Laban, R. (1988). The mastery of movement. Plymouth,
England: Northcote House.
Ladewig, S. H., & Bressem, J. (2013). New insights into
the medium hand: Discovering recurrent structures
in gestures. Semiotica, 2013, 203–231. http://dx.doi.
org/10.1515/sem-2013-0088
Landry, P., & Pares, N. (2014). Controlling and
modulating physical activity through interaction
tempo in exergames: A quantitative empirical
analysis. Journal of Ambient Intelligence and Smart
Environments, 6, 277–294. http://dx.doi.org/
10.3233/AIS-140258
Laptev, I., & Lindeberg, T. (2003). Space-time
interest points. In Proceedings of the Ninth IEEE
International Conference on Computer Vision
(Vol. 1, pp. 432–439). http://dx.doi.org/10.1109/
ICCV.2003.1238378
Laptev, I., & Perez, P. (2007). Retrieving actions in
movies. In Proceedings of the IEEE 11th
International Conference on Computer Vision
(pp. 1–8). http://dx.doi.org/10.1109/ICCV.
2007.4409105
Lara, O. D., & Labrador, M. A. (2013). A survey on
human activity recognition using wearable sensors.
IEEE Communications Surveys and Tutorials, 15,
1192–1209. http://dx.doi.org/10.1109/SURV.
2012.110112.00192
Lausberg, H. (1998). Does movement behavior have
differential diagnostic potential? Discussion of a
controlled study on patients with anorexia nervosa
and bulimia. American Journal of Dance
Therapy, 20, 85–99. http://dx.doi.org/10.1023/
A:1022148427586
Lausberg, H., & Sloetjes, H. (2009). Coding gestural
behavior with the NEUROGES–ELAN system.
Behavior Research Methods, 41, 841–849. http://
dx.doi.org/10.3758/BRM.41.3.841
Lenggenhager, B., Tadi, T., Metzinger, T., & Blanke,
O. (2007). Video ergo sum: Manipulating bodily
self-consciousness. Science, 317, 1096–1099. http://
dx.doi.org/10.1126/science.1143439
Measuring Body Movement
Liscombe, J., Venditti, J., & Hirschberg, J. B. (2003).
Classifying subject ratings of emotional speech using
acoustic features. In Proceedings of EUROSPEECH
(pp. 725–728). Geneva, Switzerland: International
Speech Communication Association.
Long, X., Yin, B., & Aarts, R. M. (2009). Single-
accelerometer-based daily physical activity
classification. In Proceedings of the Annual
International Conference of the IEEE Engineering in
Medicine and Biology Society (pp. 6107–6110).
http://dx.doi.org/10.1109/IEMBS.2009.5334925
Ma, Y., Paterson, H. M., & Pollick, F. E. (2006). A motion
capture library for the study of identity, gender, and
emotion perception from biological motion. Behavior
Research Methods, 38, 134–141. http://dx.doi.org/
10.3758/BF03192758
Magnusson, M. S. (2000). Discovering hidden time
patterns in behavior: T-patterns and their detection.
Behavior Research Methods, Instruments, and
Computers, 32, 93–110. http://dx.doi.org/10.3758/
BF03200792
Marr, D., & Nishihara, H. K. (1978). Representation
and recognition of the spatial organization of
three-dimensional shapes. Proceedings of the
Royal Society of London B: Biological Sciences,
200, 269–294. http://dx.doi.org/10.1098/
rspb.1978.0020
Marr, D., & Vaina, L. (1982). Representation and
recognition of the movements of shapes. Proceedings
of the Royal Society of London B: Biological
Sciences, 214, 501–524. http://dx.doi.org/10.1098/
rspb.1982.0024
Mathie, M. J., Coster, A. C. F., Lovell, N. H., &
Celler, B. G. (2004). Accelerometry: Providing
an integrated, practical method for long-term,
ambulatory monitoring of human movement.
Physiological Measurement, 25, R1–R20. http://dx.doi.
org/10.1088/0967-3334/25/2/R01
Matsuda, Y., & Isomura, T. (2010). Teaching of
emotional expression using finger braille. In
Proceedings of the 2010 Sixth International Conference
on Intelligent Information Hiding and Multimedia
Signal Processing (pp. 368–371). http://dx.doi.org/
10.1109/IIHMSP.2010.96
McNeill, D. (1992). Hand and mind: What gestures reveal
about thought. Chicago, IL: University of Chicago
Press.
McNeill, D. (2005). Gesture and thought. http://dx.doi.
org/10.7208/chicago/9780226514642.001.0001
Mehrabian, A. (1967). Orientation behaviors and
nonverbal attitude communication. Journal of
Communication, 17, 324–332. http://dx.doi.org/
10.1111/j.1460-2466.1967.tb01190.x
Mehrabian, A. (1968). Inference of attitudes from the
posture, orientation, and distance of a communicator.
583
Dael et al.
Journal of Consulting and Clinical Psychology, 32,
296–308. http://dx.doi.org/10.1037/h0025906
Mehrabian, A., & Friar, J. T. (1969). Encoding of attitude
by a seated communicator via posture and position
cues. Journal of Consulting and Clinical Psychology,
33, 330–336. http://dx.doi.org/10.1037/h0027576
Mendes, E., Simpson, D., Correia, M., & Macfayden, A.
(2007). The assessment of lower limb transfemoral
amputees’ mobility and locomotion. In Proceedings
of the 12th World Congress of the International Society
for Prosthetics and Orthotics (p. 321). Vancouver,
British Columbia, Canada: International Society for
Prosthetics and Orthotics.
Moeslund, T. B., & Granum, E. (2001). A survey of
computer vision-based human motion capture.
Computer Vision and Image Understanding, 81,
231–268. http://dx.doi.org/10.1006/cviu.2000.0897
Morris, D. (1979). Gestures: Their origins and distribution.
London, England: J. Cape.
Morris, J. R. (1973). Accelerometry—A technique for
the measurement of human body movements.
Journal of Biomechanics, 6, 729–736. http://dx.doi.
org/10.1016/0021-9290(73)90029-8
Murphy, N. A. (2005). Using thin slices for behavioral
coding. Journal of Nonverbal Behavior, 29, 235–246.
http://dx.doi.org/10.1007/s10919-005-7722-x
Neumann, D. A. (2002). Kinesiology of the musculoskeletal
system: Foundations for physical rehabilitation.
Philadelphia, PA: Mosby.
Nijhar, J., Bianchi-Berthouze, N., & Boguslawski, G.
(2012). Does movement recognition precision affect
the player experience in exertion games? In A.
Camurri & C. Costa (Eds.), Intelligent technologies
for interactive entertainment (pp. 73–82). http://
dx.doi.org/10.1007/978-3-642-30214-5_9
Noldus, L. P. J. J., Trienes, R. J. H., Hendriksen, A.
H. M., Jansen, H., & Jansen, R. G. (2000). The
Observer Video-Pro: New software for the collection,
management, and presentation of time-structured
data from videotapes and digital media files. Behavior
Research Methods, Instruments, and Computers, 32,
197–206. http://dx.doi.org/10.3758/BF03200802
Noldus Information Technology. (n.d.). Theme [Computer
software]. Retrieved from http://www.noldus.com/
human-behavior-research/products/theme
Ogale, A., Karapurkar, A., Guerra-Filho, G., &
Aloimonos, Y. (2004). View-invariant identification of
pose sequences for action recognition. Retrieved from
http://citeseerx.ist.psu.edu/viewdoc/download?doi=
10.1.1.117.6884&rep=rep1&type=pdf
Oikonomidis, I., Kyriazis, N., & Argyros, A. (2011).
Efficient model-based 3D tracking of hand
articulations using Kinect. In Proceedings of
the British Machine Vision Conference (pp.
101.1–101.11). http://dx.doi.org/10.5244/C.25.101
584
Olugbade, T. A., Aung, M. S. H., Bianchi-Berthouze, N.,
Marquardt, N., & Williams, A. C. (2014). Bi-modal
detection of painful reaching for chronic pain
rehabilitation systems. In Proceedings of the 16th
International Conference on Multimodal Interaction
(pp. 455–458). http://dx.doi.org/10.1145/2663204.
2663261
Parsons, T. D., & Rizzo, A. A. (2008). Affective outcomes
of virtual reality exposure therapy for anxiety and
specific phobias: A meta-analysis. Journal of Behavior
Therapy and Experimental Psychiatry, 39, 250–261.
http://dx.doi.org/10.1016/j.jbtep.2007.07.007
Polana, R., & Nelson, R. C. (1992). Recognition of
motion from temporal texture. In Proceedings of the
IEEE Computer Society Conference on Computer Vision
and Pattern Recognition (pp. 129–134). http://dx.doi.
org/10.1109/CVPR.1992.223216
Pollick, F. E., Paterson, H. M., Bruderlin, A., & Sanford,
A. J. (2001). Perceiving affect from arm movement.
Cognition, 82, B51–B61. http://dx.doi.org/10.1016/
S0010-0277(01)00147-0
Porter, S., & ten Brinke, L. (2008). Reading between the
lies: Identifying concealed and falsified emotions in
universal facial expressions. Psychological Science, 19,
508–514. http://dx.doi.org/10.1111/j.1467-9280.
2008.02116.x
Powers, M. B., & Emmelkamp, P. M. G. (2008).
Virtual reality exposure therapy for anxiety
disorders: A meta-analysis. Journal of Anxiety
Disorders, 22, 561–569. http://dx.doi.org/10.1016/j.
janxdis.2007.04.006
Prillwitz, S. (1989). HamNoSys: Version 2.0: Hamburg
Notation System for Sign Language: An introductory
guide (Vol. 5). Hamburg, Germany: Signum.
Quintilian, M. F. (2006). Institutes of oratory (L.
Honeycutt, Ed., & J. S. Watson, Trans.). Retrieved
from http://rhetoric.eserver.org/quintilian (Original
work published 1856)
Ramanan, D., & Forsyth, D. A. (2003). Automatic
annotation of everyday movements. In S. Thrun,
L. K. Saul, & B. Schölkopf (Eds.), Advances in
Neural Information Processing Systems 16: Proceedings
from the Neural Information Processing Systems 2003
Conference (pp. 1547–1554). Cambridge, MA: MIT
Press.
Rauscher, F. H., Krauss, R. M., & Chen, Y. (1996).
Gesture, speech, and lexical access: The role
of lexical movements in speech production.
Psychological Science, 7, 226–231. http://dx.doi.
org/10.1111/j.1467-9280.1996.tb00364.x
Remland, M. S., Jones, T. S., & Brinkman, H. (1995).
Interpersonal distance, body orientation, and touch:
Effects of culture, gender, and age. Journal of Social
Psychology, 135, 281–297. http://dx.doi.org/10.1080/
00224545.1995.9713958

Robertson, N., & Reid, I. (2005). Behaviour
understanding in video: A combined method. In
Proceedings of the Tenth IEEE International Conference
on Computer Vision (Vol. 1, 808–815). http://dx.doi.
org/10.1109/ICCV.2005.47
Roether, C. L., Omlor, L., Christensen, A., & Giese, M. A.
(2009). Critical features for the perception of
emotion from gait. Journal of Vision, 9, 15. http://
dx.doi.org/10.1167/9.6.15
Rohr, K. (1994). Towards model-based recognition of
human movements in image sequences. CVGIP:
Image Understanding, 59, 94–115. http://dx.doi.
org/10.1006/ciun.1994.1006
Romera-Paredes, B., Aung, H., & Bianchi-Berthouze,
N. (2013). A one-vs-one classifier ensemble
with majority voting for activity recognition. In
Proceedings of the European Symposium on Artificial
Neural Networks, Computational Intelligence and
Machine Learning (pp. 443–448). Retrieved from
https://www.elen.ucl.ac.be/Proceedings/esann/
esannpdf/es2013-124.pdf
Rosenfeld, H. M. (1982). Measurement of body motion
and orientation. In K. R. Scherer & P. Ekman (Eds.),
Handbook of methods in nonverbal behavior research
(pp. 199–286). Cambridge, England: Cambridge
University Press.
Rosenthal, R. (2005). Conducting judgment studies:
Some methodological issues. In J. A. Harrigan, R.
Rosenthal, & K. R. Scherer (Eds.), The new
handbook of methods in nonverbal behavior research
(pp. 199–234). New York, NY: Oxford University
Press.
Rothbaum, B. O., Hodges, L., Anderson, P. L., Price, L., &
Smith, S. (2002). Twelve-month follow-up of virtual
reality and standard exposure therapies for the fear of
flying. Journal of Consulting and Clinical Psychology,
70, 428–432. http://dx.doi.org/10.1037/0022-
006X.70.2.428
Rui, Y., & Anandan, P. (2000). Segmenting visual
actions based on spatio-temporal motion patterns.
In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition (Vol. 1, pp. 111–118).
http://dx.doi.org/10.1109/CVPR.2000.855807
Ryan, C. G., Gray, H., Newton, M., & Granat, M.
H. (2007). An investigation of the effects of
psychological distress on physical activity levels
in individuals with chronic lower back pain.
In Proceedings of the 4th World Congress of the
International Society of Physical and Rehabilitation
Medicine (p. 81). Seoul, Korea: International Society
of Physical and Rehabilitation Medicine.
Sanchez-Vives, M. V., & Slater, M. (2005). From presence
to consciousness through virtual reality. Nature
Reviews Neuroscience, 6, 332–339. http://dx.doi.
org/10.1038/nrn1651
Measuring Body Movement
Savva, N., Scarinzi, A., & Bianchi-Berthouze, N. (2012).
Continuous recognition of player’s affective
body expression as dynamic quality of aesthetic
experience. IEEE Transactions on Computational
Intelligence and AI in Games, 4, 199–212. http://
dx.doi.org/10.1109/TCIAIG.2012.2202663
Sayette, M. A., Cohn, J. F., Wertz, J. M., Perrott,
M. A., & Parrott, D. J. (2001). A psychometric
evaluation of the Facial Action Coding System
for assessing spontaneous expression. Journal of
Nonverbal Behavior, 25, 167–185. http://dx.doi.
org/10.1023/A:1010671109788
Scherer, K. R. (1978). Personality inference from voice
quality: The loud voice of extroversion. European
Journal of Social Psychology, 8, 467–487. http://
dx.doi.org/10.1002/ejsp.2420080405
Scherer, K. R. (1979). Nonlinguistic vocal indicators of
emotion and psychopathology. In C. E. Izard (Ed.),
Emotions in personality and psychopathology
(pp. 493–529). http://dx.doi.org/10.1007/978-
1-4613-2892-6_18
Scherer, K. R., Bänziger, T., & Roesch, E. B. (Eds.).
(2010). A blueprint for affective computing: A
sourcebook and manual. New York, NY: Oxford
University Press.
Scherer, K. R., & Ekman, P. (2005). Methodological issues
in studying nonverbal behavior. In J. A. Harrigan,
R. Rosenthal, & K. R. Scherer (Eds.), The new hand­
book of methods in nonverbal behavior research (pp.
471–512). New York, NY: Oxford University Press.
Scherer, K. R., & Ellgring, H. (2007). Multimodal
expression of emotion: Affect programs or
componential appraisal patterns? Emotion, 7, 158–171.
http://dx.doi.org/10.1037/1528-3542.7.1.158
Scherer, K. R., & Wallbott, H. G. (1985). Analysis
of nonverbal behavior. In T. A. van Dijk (Ed.),
Handbook of discourse analysis: Vol. 2. Dimensions
of discourse (pp. 199–230). London, England:
Academic Press.
Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008).
Get another label? Improving data quality and data
mining using multiple, noisy labelers. In Proceedings
of the 14th ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining (pp. 614–622).
http://dx.doi.org/10.1145/1401890.1401965
Shibata, T., Michishita, A., & Bianchi-Berthouze, N.
(2013). Analysis and modelling of affective Japanese
sitting postures by Japanese and British observers.
In Proceedings of the 2013 HUMAINE Association
Conference on Affective Computing and Intelligent
Interaction (pp. 91–96). http://dx.doi.org/10.1109/
ACII.2013.22
Singh, A., Klapper, A., Jia, J., Fidalgo, A., Tajadura-
Jimenez, A., Kanakam, N., Bianchi-Berthouze,
N., & Williams, A. (2014). Motivating people
585
Dael et al.
with chronic pain to do physical activity:
Opportunities for technology design. In Proceedings
of the 32nd ACM Conference on Human Factors in
Computing Systems (pp. 2803–2812). http://dx.doi.
org/10.1145/2556288.2557268
Sminchisescu, C., Kanaujia, A., Zhiguo, L., & Metaxas,
D. (2005). Conditional models for contextual human
motion recognition. In Proceedings of the 10th IEEE
International Conference on Computer Vision (Vol.
2, pp. 1808–1815). http://dx.doi.org/10.1109/
ICCV.2005.59
Sommer, R. (1959). Studies in personal space. Sociometry,
22, 247–260. http://dx.doi.org/10.2307/2785668
Sommer, R. (2002). Personal space in a digital age. In
R. B. Bechtel & A. Churchman (Eds.), Handbook of
environmental psychology (pp. 647–660). New York,
NY: Wiley.
Sommer, R., & Becker, F. D. (1969). Territorial defense
and the good neighbor. Journal of Personality
and Social Psychology, 11, 85–92. http://dx.doi.
org/10.1037/h0027046
Starner, T., & Pentland, A. (1997). Real-time American
Sign Language recognition from video using hidden
Markov models. In M. Shah & R. Jain (Eds.), Motion-
based recognition (pp. 227–243). http://dx.doi.
org/10.1007/978-94-015-8935-2_10
Stokoe, W. C. (1960). Sign language structure: An outline
of the visual communication systems of the American
deaf. Silver Spring, MD: Linstok Press.
Tang, K., Stansfield, B. W., Spence, W. D., Simpson,
D., & Maxwell, D. (2007). The quantification of
lower limb prosthetic wearing times and amputees’
activity levels when using suction sockets. In
Proceedings of the 12th World Congress of the
International Society for Prosthetics and Orthotics
(p. 304). Vancouver, British Columbia, Canada:
International Society for Prosthetics and Orthotics.
Thomas, M. D., McGinley, J. A., Carruth, D. W., &
Blackledge, C. (2007). Cross-validation of an infrared
motion capture system and an electromechanical
motion capture device. In Proceedings of the Society
of Automotive Engineers Digital 2007 Human Modeling
Conference (SAE Technical Paper No. 2007-01-
2475). http://dx.doi.org/10.4271/2007-01-2475
Tognetti, A., Bartalesi, R., Lorussi, F., & De Rossi, D.
(2007). Body segment position reconstruction
and posture classification by smart textiles. Transactions
of the Institute of Measurement and Control, 29, 215–253.
http://dx.doi.org/10.1177/0142331207069487
Tracy, J. L., & Matsumoto, D. (2008). The spontaneous
expression of pride and shame: Evidence for biologically
innate nonverbal displays. Proceedings of the National
Academy of Sciences, USA, 105, 11655–11660. http://
dx.doi.org/10.1073/pnas.0802686105
586
Tracy, J. L., & Robins, R. W. (2007). The prototypical
pride expression: Development of a nonverbal
behavior coding system. Emotion, 7, 789–801. http://
dx.doi.org/10.1037/1528-3542.7.4.789
Troje, N. F. (2002). Decomposing biological motion: A
framework for analysis and synthesis of human gait
patterns. Journal of Vision, 2, 371–87. http://dx.doi.
org/10.1167/2.5.2
Van Dyck, E., Maes, P.-J., Hargreaves, J., Lesaffre, M., &
Leman, M. (2013). Expressing induced emotions
through free dance movement. Journal of Nonverbal
Behavior, 37, 175–190. http://dx.doi.org/10.1007/
s10919-013-0153-1
Van Hooff, J. A. R. A. M. (1982). Categories and
sequences of behavior: Methods of description
and analysis. In K. R. Scherer & P. Ekman (Eds.),
Handbook of methods in nonverbal behavior research
(pp. 362–439). Cambridge, England: Cambridge
University Press.
Visch, V. T., Goudbeek, M. B., & Mortillaro, M. (2014).
Robust anger: Recognition of deteriorated dynamic
bodily emotion expressions. Cognition and Emotion,
28, 936–946. http://dx.doi.org/10.1080/02699931.
2013.865595
Wallbott, H. G. (1985). Hand movement quality: A
neglected aspect of nonverbal behavior in clinical
judgment and person perception. Journal of
Clinical Psychology, 41, 345–359. http://dx.doi.
org/10.1002/1097-4679(198505)41:3<345::AID-
JCLP2270410307>3.0.CO;2-9
Wallbott, H. G. (1998). Bodily expression of
emotion. European Journal of Social Psychology,
28, 879–896. http://dx.doi.org/10.1002/
(SICI)1099-0992(1998110)28:6<879::AID-
EJSP901>3.0.CO;2-W
Wang, L. W. L., & Suter, D. (2007). Recognizing
human activities from silhouettes: Motion subspace
and factorial discriminative graphical model. In
Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition (pp. 1–8). http://
dx.doi.org/10.1109/CVPR.2007.383298
Watson, O. M., & Graves, T. D. (1966). Quantitative
research in proxemic behavior. American
Anthropologist, 68, 971–985. http://dx.doi.org/
10.1525/aa.1966.68.4.02a00070
Weinland, D., Ronfard, R., & Boyer, E. (2011). A survey
of vision-based methods for action representation,
segmentation and recognition. Computer Vision and
Image Understanding, 115, 224–241. http://dx.doi.
org/10.1016/j.cviu.2010.10.002
Weiss, S. J. (1992). Measurement of the sensory qualities in
tactile interaction. Nursing Research, 41, 82–86. http://
dx.doi.org/10.1097/00006199-199203000-00005

Yale, M. E., Messinger, D. S., Cobo-Lewis, A. B., &
Delgado, C. F. (2003). The temporal coordination
of early infant communication. Developmental
Psychology, 39, 815–824. http://dx.doi.
org/10.1037/0012-1649.39.5.815
Yale, M. E., Messinger, D. S., Cobo-Lewis, A. B., Oller, D.
K., & Eilers, R. E. (1999). An event-based analysis
of the coordination of early infant vocalizations
and facial actions. Developmental Psychology,
35, 505–513. http://dx.doi.org/10.1037/0012-
1649.35.2.505
Yamato, J., Ohya, J., & Ishii, K. (1992). Recognizing
human action in time-sequential images using
hidden Markov model. In Proceedings of the IEEE
Computer Society Conference on Computer Vision and
Measuring Body Movement
Pattern Recognition (pp. 379–385). http://dx.doi.
org/10.1109/CVPR.1992.223161
Yannakakis, G. N. (2009). Preference learning for
affective modeling. In Proceedings of the 3rd
International Conference on Affective Computing and
Intelligent Interaction and Workshops (pp. 1–6).
http://dx.doi.org/10.1109/ACII.2009.5349491
Zacks, J. M., Kumar, S., Abrams, R. A., & Mehta,
R. (2009). Using movement and intentions to
understand human activity. Cognition, 112, 201–216.
http://dx.doi.org/10.1016/j.cognition.2009.03.007
Zhou, H., & Hu, H. (2008). Human motion tracking
for rehabilitation—A survey. Biomedical Signal
Processing and Control, 3, 1–18. http://dx.doi.
org/10.1016/j.bspc.2007.09.001
587
 Chapter 23
Measuring Nonverbal
Sensitivity
Ronald E. Riggio and Annick Darioly
In this chapter, we review the various means that
nonverbal communication researchers have used to
assess nonverbal sensitivity, which involves abilities
to read and decode nonverbal cues in others. The
review of methods is divided into several sections.
In the first section, we focus on performance-based
measures of nonverbal sensitivity. In the second
section, we review self-report measures of nonver-
bal sensitivity. We also review research methods
designed to assess nonverbal cues of personality
and attitudes as well as the construct of empathic
accuracy. We discuss the strengths and limita-
tions of each methodology and briefly review key
research findings. Finally, we discuss the core issues
in assessing nonverbal sensitivity and discuss future
directions for this line of research, including the use
of automated decoding systems.
DEFINING NONVERBAL SENSITIVITY
Nonverbal sensitivity refers to the ability of individu-
als to read and decode nonverbal cues in others
and, importantly, the ability to correctly interpret
the meaning of those cues. Most often, nonverbal
cues involve cues of emotion; cues of dominance
or status; or cues about attitudes, intentions, and
interpersonal orientation (likes and dislikes, desire
to terminate an interaction, etc.). Emotional sensitiv-
ity is the term that is typically used when focusing
solely on nonverbal cues of emotion. Nonverbal sen-
sitivity, and emotional sensitivity, can be seen as a
subset of the broader category of interpersonal sensi-
tivity, which deals with the ability to read a person’s
http://dx.doi.org/10.1037/14669-023
APA Handbook of Nonverbal Communication, D. Matsumoto, H. C. Hwang, and M. G. Frank (Editors-in-Chief)
Copyright © 2016 by the American Psychological Association. All rights reserved.
traits, the ability to detect deception, recognition of
adherence to social norms, and sensitivity to subtle
cues of appearance (Bernieri, 2001). It is often
impossible to “pull apart” the nonverbal and verbal
elements of communication; therefore, much of the
research that involves nonverbal sensitivity, such as
the accurate decoding of nonverbal cues of decep-
tion, must also take into account the verbal elements
of the message.
Nonverbal sensitivity is conceptualized as an
individual difference variable (Rosenthal, 1979).
It is assumed that nonverbal sensitivity is an impor-
tant contributor to the concept of empathy, with
ability to read others’ emotions critical for empathic
understanding (Riggio, Tucker, & Coffaro, 1989),
and nonverbal sensitivity is seen as one of the build-
ing blocks of good interpersonal relationships. More
recently, sensitivity to nonverbal cues, particularly
nonverbal cues of emotion, is seen as a core com-
ponent of the construct of emotional intelligence
(Cherniss, 2010; Mayer, Salovey, & Caruso, 2000;
see also Chapter 10, this handbook). Indeed, ability
to read nonverbal cues (termed identifying emotions)
is one of the core components in the abilities
model of emotional intelligence (along with abilities
to express emotions accurately and to manage
emotions; Mayer & Salovey, 1997).
PERFORMANCE-BASED MEASURES OF
NONVERBAL SENSITIVITY
The earliest measures of nonverbal sensitivity were
performance-based. Performance-based measures
589
Riggio and Darioly
require the individual being assessed to accurately
read and decode nonverbal cues. This is a different
methodology than the self-report measures of non-
verbal sensitivity that are introduced later. From a
measurement standpoint, performance measures are
superior to other methods because there is an objec-
tively correct answer. We chronologically review
some of the most widely used performance measures
of nonverbal sensitivity.
Brief Affect Recognition Test
Building on research that attempted to assess ability to
recognize brief, “microexpressions” of emotion (e.g.,
Garwood, Guiora, & Kalter, 1970; Taylor, Guiora,
Catford, & Lane, 1969), one of the earliest measures
of nonverbal sensitivity was the Brief Affect Recogni-
tion Test (BART; Ekman & Friesen, 1974). The BART
focuses on accurate decoding of six basic emotions
(happiness, sadness, fear, surprise, disgust, and anger)
identified in the research of Ekman and Friesen (1974).
Using a tachistoscope, test-takers view a brief (often
1/25th of a second, although exposure times may
vary) photograph of faces showing basic emotional
expressions. The actors (eight females, six males) were
trained to ensure that the facial expressions were cor-
rectly and clearly displayed. The test-takers choose
from the six basic expressions the one that they believe
is portrayed in each photo. The BART consists of 110
photos/items, which are all in black and white.
Aside from Ekman and Friesen’s (1974) use of
the BART, there has been little research with it,
although Mufson and Nowicki (1991) found a now
well-established difference, with women receiving
significantly higher scores on emotion recognition
than men on the BART. There has been some criti-
cism of the BART methodology, particularly the
very brief presentation of facial expressions
(Matsumoto et al., 2000; Nowicki & Duke, 1994;
see also Chapter 10, this handbook) and the use
of still photographs. In addition, it is important to
emphasize that the BART uses posed expressions
that are enacted in the absence of any felt emotion.
Matsumoto et al. (2000) used the BART idea as the
basis for the development of a more sophisticated
emotion recognition test, the Japanese and Cau-
casian Brief Affect Recognition Test (JACBART),
which is discussed later.
590
Communication of Affect Receiving
Ability Test
Buck (1976) developed a slide-viewing technique
that was designed to elicit felt emotional expressions
for the purpose of assessing the accuracy/ability to
express spontaneous facial expressions of emotion
(see Chapter 17, this handbook). In this paradigm,
participants are shown a series of emotion-inducing
slides (e.g., pleasant vs. unpleasant; sexual, unusual)
while their faces are videotaped. The videotaped
expressions are then shown to individuals who
receive an accuracy score on their ability to “read”
the displayed subtle facial cues of emotion and
determine which type of slide the sender is viewing.
Buck (1976) reported some validity evidence
for the Communication of Affect Receiving Ability
Test (CARAT), as did Hall, Halberstadt, and O’Brien
(1997). The CARAT itself, however, has not been
widely used in research. The slide-viewing para-
digm, however, has been used in nonverbal com-
munication research to assess individual differences
in the sending and receiving of emotional cues (e.g.,
Fujita, Harper, & Wiens, 1980; Tucker & Riggio,
1988). One advantage of the CARAT over the BART
is that video clips are used so that the dynamic
movement of the face is visible, as opposed to
simply using static facial expressions as stimuli
(see Chapter 17, this handbook).
The Profile of Nonverbal Sensitivity
The Profile of Nonverbal Sensitivity (PONS; Rosenthal,
Hall, DiMatteo, Rogers, & Archer, 1979) is one of
the most widely researched performance measures
of nonverbal sensitivity. In its full version, the
PONS contains 220, 2-s video or audio-only clips
of a single actress expressing various emotional
scenarios. Visual channels are presented with and
without sound, using the two verbal elimination
methods, resulting in 11 separate channels: Emotions
are expressed with (a) only the face visible, (b) the
entire face and body of the sender, or (c) only the
body (from neck to knees), and there are two audio-
only channels, with the verbal content eliminated
through either (d) electronic filtering, creating a
muffled sound so that the words are unintelligible,
but the emotional tone remains, or (e) random-
ized splicing, which makes the verbal content

unrecognizable but also disrupts the order of the
nonverbal vocal cues. The six other channels are the
combinations of the three visual recordings (a, b, c)
with the two manipulated voice/speech recordings (d, e).
A total of 20 different emotionally laden
scenes are presented in the PONS, which try
to represent four affective quadrants, using
the dimensions of positivity–negativity and
dominance–submissiveness. Examples of
positive–submissive scenes involve the actress
asking for a favor or ordering food in a restaurant.
Negative–submissive scenes include asking for
forgiveness or talking about the death of a friend.
Negative–dominant scenes involve threatening or
criticizing another, whereas positive–dominant
scenes include talking about a wedding. A fuller
description of the PONS and its structure is
presented in Hall (2001).
Research with the PONS shows that married
women with toddler-age children score higher than
married women without children and that per-
formance on the PONS improved with age, from
middle childhood to college years (Hall, Andrze-
jewski, & Yopchick, 2009). Moreover, college stu-
dents scoring high on the PONS learned more in an
interpersonal learning situation (Bernieri, 1991).
The PONS has consistently shown that women
score on average higher than men, and this is con-
sistent across many studies (of both the PONS and
nonverbal sensitivity in general; see Chapter 6, this
handbook). Research on physician–patient commu-
nication showed that doctors scoring higher on the
PONS had more satisfied patients (DiMatteo,
Friedman, & Taranta, 1979).
The PONS demonstrates good reliability and
validity, and it has been used frequently in research
focusing on individual differences in nonverbal
sensitivity. Despite low interitem intercorrela-
tions, the PONS has high internal consistency
(Kuder–Richardson Formula 20 of .86; Rosen-
thal et al., 1979). Moreover, the PONS has shown
remarkable predictive validity. Rosenthal et al.
(1979) reported several expected significant rela-
tionships. For example, high scorers had healthier,
well-adjusted personalities, and they were assessed
as more interpersonally sensitive by peers and supe-
riors as well as better in their job performance (as
Measuring Nonverbal Sensitivity
clinicians and teachers). The PONS was tested in
many different cultures, and accuracy rates were
the same across cultures depending on the degree
of similarity in terms of modernization. One obvi-
ous drawback is the length of the PONS, so brief
versions (64 items; 40 items) have been created,
but these fail to show good psychometric proper-
ties (Bänziger, Scherer, Hall, & Rosenthal, 2011;
Hall, 2001). Although the PONS is considered one
of the best performance measures of nonverbal sen-
sitivity, there are limitations to the methodology.
First, there is only a single actress represented on
the entire test, which limits ability to assess skill
at decoding across senders. Second, the actress is
posing the emotional scenarios rather than actually
involved in emotional interactions with others. On
the positive side, the PONS gives a very detailed
assessment of an individual’s ability to decode non-
verbal cues across channels. Moreover, the emo-
tional scenarios are more complicated than simply
the portrayal of basic emotions that are depicted in
the BART and CARAT.
Recently, Bänziger et al. (2011) created the Mini-
PONS, which is a short, multichannel version of the
PONS designed to measure the ability to recognize
emotions, interpersonal attitudes, and communica-
tive intentions in different nonverbal channels. It
consists of 64 items that are highly correlated with
the full version of the PONS and that are positively
correlated with other tests of emotion recognition.
The MiniPONS showed acceptable construct valid-
ity. It showed a respectable correlation of r = .70
with the full version of PONS and ranging from
r = .24 to r = .39 with other measures of emotion
recognition (see Bänziger et al., 2011). The reduc-
tion in the length of the test (64 items vs. 220 items
for the full version of the PONS), however, resulted
in lower reliability/internal consistency. The inter-
nal consistency of the MiniPONS was measured
by intraclass correlation coefficients (ICCs). The
single-items ICC was .02, and the combined-items
ICC was .57. Single-items ICCs are an approxima-
tion of the average correlation between items (which
was .03). Combined-items ICCs are similar to Cron-
bach’s alpha. Regarding the test–retest of the Mini-
PONS, the correlation was r = .64 (see Bänziger
et al., 2011).
591
Riggio and Darioly
The Diagnostic Analysis of Nonverbal
Accuracy
The Diagnostic Analysis of Nonverbal Accuracy
(DANVA; Nowicki & Duke, 1994) was designed to
assess the nonverbal sensitivity of children, but it
was later expanded for use with adults, resulting in
the Diagnostic Analysis of Nonverbal Accuracy—2
(DANVA–2; Nowicki & Duke, 2001). The DANVA
and DANVA–2 consist of three subtests (Decoding
of Facial Expressions, Postures, and Paralanguage;
see Chapters 10, 11, and 15, this handbook). The
measure presents a series of photographs of child/
adult faces with high- and low-intensity portrayals
of the four basic emotions of happiness, anger, sad-
ness, and fear. The emotional expressions are both
natural/spontaneous and posed, and each subtest
uses more than one actor as well as mixed genders
(as opposed to the single actress in the PONS). Two
additional subtests have audio segments where
actors say content neutral sentences (e.g., “I am
going out of the room now, but I’ll be back later”)
while portraying the four basic emotions. Respon-
dents simply choose among the four basic emotions.
The DANVA has good psychometric properties
and has been used, and continues to be used, in
research with both children and adults. It is inter-
esting that in recent years, nonverbal sensitivity,
particularly as measured by the DANVA, has been
used by researchers as a proxy for emotional intel-
ligence, as noted earlier (Elfenbein & Ambady,
2002; Rubin, Munz, & Bommer, 2005). Using the
DANVA, Nowicki and his colleagues have found
that children in preschool and elementary school
who score higher at decoding nonverbal behaviors
are more socially competent, more popular, less
aggressive, less anxious, less depressed, and higher
on internal locus of control (Baum & Nowicki, 1998;
Lancelot & Nowicki, 1997; Nowicki & Carton, 1997;
Nowicki & Duke, 1994; Nowicki & Mitchell, 1998).
Furthermore, research on transformational leadership
showed that leader emotion recognition ability using
the DANVA is positively associated with leader trans-
formational leadership behavior (Rubin et al., 2005).
The test’s relative brevity (48 items; 24 in each
of the visual and auditory subtests) and ease of
administration are advantages for researchers. The
DANVA, however, suffers from some of the same
592
weaknesses mentioned in some of the other sensitiv-
ity measures, such as the use of static facial expres-
sions and the posed nature of the expressions.
To overcome the shortcomings of the posture
subtest from the DANVA–2 (i.e., a single actress
portrayed the postural stimuli, which did not
include seated postures), Pitterman and Nowicki
(2004) created the Diagnostic Analysis of Nonverbal
Accuracy—2 Posture Test (DANVA2–POS). This test
measures an individual’s ability to identify emotion
in human standing and sitting postures. A shortcom-
ing, however, is that it uses static and posed photo-
graphs rather than video presentation of postures.
The authors reported some evidence for validity and
internal consistency for the DANVA2–POS. They
found that DANVA2–POS accuracy scores increased
with age and were both internally consistent and reli-
able over time. In terms of validity, they found that
DANVA2–POS accuracy scores were linked to per-
sonality traits, such as self-esteem, locus of control,
social anxiety, and loneliness. The scores were more
accurate in participants expected to be more accurate
(e.g., dancers). They were shown to be predictive
in sports or business performance. Finally, as with
other nonverbal skills, women were more accurate
than men in identifying emotion in postures (Hall,
1990; see also Chapter 6, this handbook). However,
the measure has not been widely used in research.
Japanese and Caucasian Brief Affect
Recognition Test
Matsumoto et al. (2000) sought to develop an
improved version of the BART, and one that could
be used to explore cross-cultural differences in
emotion recognition. Like the BART, the JACBART
attempts to assess recognition of very brief presen-
tations of emotionally expressive faces. To control
for after-images created by the brief presentation of
a facial emotion, the emotional expression is pre-
sented briefly in the middle of a 1-s presentation
of the same sender’s neutral facial expression. This
allows for a quick comparison of the sender’s neutral
face, with the emotional expression. The JACBART
includes the basic emotions of anger, contempt,
disgust, fear, happiness, sadness, and surprise, and
it contains 56 items. Test-takers rate the presence
or absence of the seven emotions by rating their

intensity on 9-point scales. An alternative (and eas-
ier) format requires the test-takers to simply choose
from the seven emotions (forced-choice format).
The emotions measured in the JACBART generally
correlated well with each other in the various stud-
ies done by Matsumoto et al. (2000). Happiness was
the only emotion that did not correlate consistently
with the others due to a ceiling effect. On the whole,
the JACBART has good internal convergent validity.
Moreover, Cronbach alphas were acceptable (.40–
.85), supporting internal reliability. The test–retest
total score was significant (r = .78). Finally, in terms
of personality traits, the measure was significantly
correlated with Openness (from r = .21 to r = .38)
and Conscientiousness (from r = .11 to r = .35)
using the Big Five Inventory (John, 1989);
furthermore, Extraversion was positively correlated
with the measure (r = .47), and Neuroticism was
significantly negatively correlated with it (r = −.45)
using the Eysenck Personality Inventory (Eysenck &
Eysenck, 1968). This shows concurrent validity for
the JACBART. People who score high on Openness
are generally curious, are attentive to inner feelings,
and have an active imagination. Thus, they are more
attentive and receptive to their environment and the
people around them. They are more in tune with
reading emotions of others. People who score high on
Conscientiousness are generally reliable and perse-
verant. They may see details better than others, such
as facial expressions, which would explain the signifi-
cant correlations between this trait and the JACBART.
People who score high on Extraversion are generally
sociable and outgoing, so they are more in tune with
the emotions of others and can better read and inter-
pret them. People who are low on Neuroticism are
emotionally labile and more in tune with their own
emotional states, so they cannot focus on others.
The JACBART is an improvement of the BART,
and it has particular usefulness for exploring cross-
cultural issues in nonverbal sensitivity. So far, how-
ever, most of the research with the JACBART has
been conducted by Matsumoto and colleagues. For
example, Hall and Matsumoto (2004) demonstrated
that women are more accurate than men in judging
emotional meaning from nonverbal cues using the
JACBART. In terms of limitations, the JACBART
focuses only on the decoding of basic emotions from
Measuring Nonverbal Sensitivity
posed, static expressions—unlike the PONS and
CARAT (which assess decoding of dynamic non-
verbal cues) as well as the PONS (which takes into
account elements of the social situation).
The Contextual and Affective
Sensitivity Test
The Contextual and Affective Sensitivity Test
(CAST) was created to overcome many of the per-
ceived shortcomings in the existing measures of
emotional decoding skill (Trimboli & Walker,
1993). The CAST uses clips from unscripted televi-
sion programs (news segments, interviews) that
involve actual interactions and felt emotional states
rather than posed expressions. Unlike many of the
other nonverbal sensitivity tests, the CAST includes
the context in which the target is interacting.
The CAST contains 32 items. Sixteen of the items
are consistent messages where the correct emotion/
affect needs to be correctly identified. Eight items
contain inconsistency in either the verbal or non-
verbal cues, so that one must be disregarded to cor-
rectly identify the affect. The remaining eight items
(two sets of four items each) are ambiguous mes-
sages that are arranged so that additional contextual
information is given sequentially, four times. In
other words, these test items allow the respondent
to accurately identify the correct affect with only
minimal information (the first item in each set) or in
continual trials with each segment adding additional
information (items 2–4 of each set). Trimboli and
Walker (1993) stated that the CAST assesses three
aspects of nonverbal sensitivity: “the ability to iden-
tify a feeling or emotion, the ability to attend to the
verbal or nonverbal components as appropriate, and
the ability to interpret ambiguous information with
respect to context” (p. 62).
The authors reported some validity evidence for
the CAST, and this measure is relatively brief and
easy to use. It is surprising that the CAST has gener-
ated little research, but it does offer a broader alter-
native to the more narrow performance measures of
nonverbal sensitivity.
Multimodal Emotion Recognition Test
The Multimodal Emotion Recognition Test (MERT;
Bänziger, Grandjean, & Scherer, 2009) measures
593
Riggio and Darioly
the perception of dynamic vocal, facial, and bodily
expressions. The MERT uses 10 actor-portrayed
emotions, which include the following: anxiety, cold
anger, hot anger, contempt, despair, disgust, elation,
happiness, sadness, and panic fear, with two variants
for each. The emotions are presented with two dif-
ferent levels of intensity, or arousal. Three film clips
illustrate each emotion in four different channels:
video only (facial cues), audio only (vocal cues),
audio/video (using both facial and vocal cues), and
still photographs extracted from the film clips. The
test takes about
45 min to administer, and the film clips are black
and white. Results of research with the MERT show
that emotion recognition is better when emotional
cues are present in the voice and face at the same
time as opposed to cues from the voice only (see
Chapters 10 and 11, this handbook). Despite this
finding, it is suggested that it is not necessarily the
case that multimodal information leads to better rec-
ognition of emotional expression.
There is some evidence of the reliability and
construct validity of the MERT. To assess interrater
agreement for the MERT, Bänziger et al. (2009) com-
puted the average interparticipant correlation. The
values obtained were as follows: audio/video,
r = .35; audio, r = .38; video, r = .40; and still pic-
ture, r = .39—yielding an average of .38 across all
modes. The correlations of the test–retest reliability
were as follows: audio/video, r = .55; audio, r = .56;
video, r = .46; and still picture, r = .56. The correla-
tion for the total scores reached r = .78. Regarding
construct validity, Bänziger et al. reported correlations
of .22 with DANVA, .51 with PONS, and .28 with the
Emotion Recognition Index (ERI; Scherer & Scherer,
2011). However, there has not been good evidence
of predictive validity for the MERT. This needs to be
established in examining the relationships of recog-
nition accuracy tested with other skills, task-oriented
behavior, and interpersonal interaction, or long-term
relationships, as has been done with other instru-
ments (Hall et al., 2009; Hall & Bernieri, 2001).
The MERT uses video and static cues, as opposed
to the other measures of nonverbal decoding skill,
which typically use only one or the other, and there
is a wider range of emotions. Because this measure
has been recently developed, there is little published
594
research using the MERT. Given that the MERT is
freely accessible on the Internet (http://www.affec-
tivesciences.org/MERT) and that individuals can
take the test and receive their accuracy scores, there
could be a great deal of research forthcoming with
the MERT.
Emotion Recognition Index
The ERI (Scherer & Scherer, 2011) is a brief test of
emotional decoding skill, which consists of two
subtests: the Index of Facial Emotion Recognition
and the Index of Vocal Emotion Recognition.
The Index of Facial Emotion Recognition contains
30 items with five types of emotions (anger, fear,
joy, sadness, disgust) and uses pictures from Ekman
and Friesen’s (1975) Pictures of Facial Affect. The
Index of Vocal Emotion Recognition has 30 items
with five emotional categories (sadness, fear, anger,
joy, neutral) and uses vocal emotion portrayals from
four German professional radio actors (see Scherer,
Banse, Wallbott, & Goldbeck, 1991). Stimuli are
presented automatically on a computer screen, and
there is no time limit to the test. The ERI facial sub-
score correlates significantly with the corresponding
MERT scores (r = .29) but not with the DANVA
scores (r = .15). The ERI vocal subscore correlates
significantly with the corresponding MERT score
(r = .27) but not with the DANVA (r = .01) and
PONS (r = .03) scores.
The ERI has been validated with a larger num-
ber of managers in international companies (see
Scherer & Scherer, 2011). ERI score correlates with
a number of personality factors, such as Extraver-
sion, Need for Social Support, and Emotional Stabil-
ity. Extraverted individuals and those who privilege
internal attribution of responsibility and search for
empathy and social support are more competent in
recognizing the emotional state of interaction partners
than are individuals characterized by introversion or
affect repression. Moreover, self-assured, emotionally
stable individuals might be more competent in recog-
nizing emotions than are anxious, neurotic, and emo-
tionally labile people (Scherer & Scherer, 2011).
Interpersonal Perception Task
The Interpersonal Perception Task (IPT; Archer &
Costanzo, 1987; Costanzo & Archer, 1989) and

its predecessor, the Situational Interpretations
Task (Archer & Akert, 1977), take a much broader
approach to nonverbal sensitivity than the other
performance-based assessment tools. Instead of
focusing mainly on emotional expression, the IPT
assesses individual differences in the ability to read
and decode nonverbal, verbal, and social/situational
cues. The test consists of 30 videotaped scenes (15
scenes are used in a brief version: the Interpersonal
Perception Task—15; Costanzo & Archer, 1989).
Each of the scenes is of a naturally occurring inter-
action, such as a telephone call, an interaction
among people, or an interview after an event. The
test-taker watches each scene, and using a multiple-
choice format, he or she tries to choose the correct
answer. The questions ask about the relationship
between the interactants (“Who is the child of the
man?” “What is the relationship between these
people?”) or about truth-telling or deception. Rather
than focusing on just emotions, the IPT assesses
ability to decode status and relationship cues, com-
petition, deception, and liking/familiarity.
The IPT represents an important expansion of
research on measuring nonverbal sensitivity, as it
goes far beyond mere decoding of emotional expres-
sions, and it includes real-world interactions requir-
ing decoding of verbal, emotional, relationship, and
situational cues—and combining all of this infor-
mation to reach a conclusion. Although the IPT is
psychometrically sound, it does not have the high
levels of internal consistency that are found in the
emotional decoding measures, primarily because of
the complexity and breadth of the videotape items
(see Archer, Costanzo, & Akert, 2001).
Despite its breadth of assessing sensitivity to both
emotional and social cues, as well as knowledge of
social situations, research with the IPT consistently
shows a sex difference, with women tending to score
higher than men (e.g., Costanzo & Archer, 1989;
Horgan & Smith, 2006), which is consistent with
other measures of nonverbal sensitivity.
Although measures of emotional decoding assess
some of the underlying components of emotional
intelligence, the IPT, the CAST, and (to some
extent) the PONS assess dimensions that might be
better labeled social intelligence (see Archer, 1980).
Although some researchers have used emotional
Measuring Nonverbal Sensitivity
decoding as a proxy for emotional intelligence, the
IPT might be an indicator of social intelligence.
Indeed, research has found that scores on the IPT
correlate with the level of appropriate social func-
tioning in psychiatric patients (Cusi, MacQueen, &
McKinnon, 2012).
Strengths and Limitations of Performance-
Based Measures of Nonverbal Sensitivity
As suggested earlier, there is a vast array of per-
formance-based measures of nonverbal sensitivity.
Each measures a very specific domain of decoding
skill, from visual facial expressions of primary emo-
tions (e.g., BART, JACBART), to decoding
of emotions in both visual and vocal channels
(e.g., DANVA–2, ERI), to more complex emotional
expressions associated with specific contexts (e.g.,
PONS, IPT). Emotional stimuli can range from
static photographed expressions to video or audio
segments, and they can be presented in multiple
channels (visual, vocal, combined, face-only, body-
only, etc.). Yet, all of these varied measurement
tools purport to measure the general construct of
nonverbal sensitivity. For example, a nonverbal
decoding instrument that focuses on only static
facial cues, such as the BART or JACBART, is assess-
ing a different aspect of nonverbal sensitivity than
the IPT, which is measuring sensitivity to emotional
cues, cues of dominance, and situational elements,
and which sometimes has multiple individuals
in a videotaped interaction. Given the variety of
stimuli used in these different assessment tools, it
is no wonder that intercorrelations among the vari-
ous measures are rather low. A researcher should
carefully consider which domains of nonverbal
sensitivity are of interest in selecting a measure for
nonverbal sensitivity.
When researchers attempt to measure nonverbal
or emotional sensitivity, they rely primarily on per-
formance-based assessments. Their great strength is
the measurement of actual performance on objective
tasks. For example, there is an agreed-upon “cor-
rect” answer to most items in a performance-based
test. On the negative side, the samples of nonverbal
behavior are necessarily limited to specific chan-
nels (most often facial expressions of emotions),
are more often posed (rather than spontaneous, or
595
Riggio and Darioly
actual, emotional expressions), and involve brief
exposures; furthermore, the context is included in
only some of the measures (e.g., CAST, IPT). In
addition, respondents are tested at only one point
in time, and the assessment focuses on ability to
decode nonverbal behaviors in a limited domain
(i.e., specific channels, specific individuals, brief
segments).
SELF-REPORT MEASURES OF NONVERBAL
SENSITIVITY
Although performance measures of nonverbal sen-
sitivity/decoding are the gold standard, there have
been some attempts to measure nonverbal sensitiv-
ity using self-report techniques. We review some
of the self-report measures that have been used to
assess nonverbal sensitivity as well as some instru-
ments that measure constructs that are theoretically
or conceptually linked to nonverbal sensitivity.
The Perceived Decoding Ability Scale
The first attempt to assess nonverbal sensitivity
through self-report means was done by Zuckerman
and Larrance (1979), who constructed the Perceived
Decoding Ability (PDA) scale (and the correspond-
ing Perceived Encoding Ability scale). The PDA
scale contains 32 self-report items designed to assess
decoding skill in both visual and auditory channels.
Respondents indicate their agreement/disagree-
ment with each statement using a 7-point response
scale. Following the model used for the PONS,
the four quadrants created by the intersection of
the positive–negative and dominant–submissive
dimensions are used. Items include “I can usu-
ally tell when someone is angry from his or her
tone of voice,” which is designed to measure vocal
sensitivity to negative–dominant affect. The item
“When someone tries to please me, I can usually
tell from his or her facial expression” is intended
to measure visual sensitivity to facial cues of
positive–submissive affect. To protect against acqui-
escence response bias, some of the items are reverse
coded (“I am usually unaware of other
people’s feelings”).
The initial validation studies of the PDA scale
were somewhat discouraging, with only small,
596
and often nonsignificant, correlations between the
PDA scale and performance measures of nonverbal
decoding (e.g., PONS). Part of the problem was that
many of the PDA scale items asked, as the title of
the scale suggests, about the individual’s perceived
abilities. It may be that people are unaware of their
decoding skills and nonverbal sensitivity unless the
items are grounded in behavior upon which respon-
dents might be able to give accurate information
(e.g., “Other people have told me that I am sensitive
to their feelings”). In any case, because of the lack
of validity evidence, the PDA and Perceived Encod-
ing Ability scales have not been used in subsequent
research. The efforts of Zuckerman and Larrance
(1979), however, were the inspiration for develop-
ing another self-report measure of nonverbal/
emotional sensitivity as a component of the Social
Skills Inventory (SSI; Riggio, 1986, 1989).
The Social Skills Inventory: Subscales of
Emotional and Social Sensitivity
The SSI (Riggio, 1986, 1989) was created to measure
basic skills in nonverbal, as well as verbal/social,
communication. The primary emphasis was on non-
verbal/emotional encoding, decoding, and control
over the expression of emotional and nonverbal
messages. The other half of the SSI assesses skills
in verbal expression, sensitivity, and social role-
playing skill, as well as elements associated with
social competence/intelligence. Of particular interest
for nonverbal sensitivity are the Emotional Sensitiv-
ity (ES) and Social Sensitivity (SS) subscales of the
SSI. Emotional sensitivity, as measured by the SSI,
assesses skill in decoding emotional and other non-
verbal messages in others. Social sensitivity is verbal
listening skill but also the ability to decode the sub-
tleties of social situations, including knowledge of
social norms and scripts (Riggio & Carney, 2003).
It is more closely aligned with some of the decoding
skills that are required in the IPT.
Each of the SSI subscales consists of 15 self-
report items, with a 5-point response scale of how
much each item is “like me” (ranging from not at all
like me to exactly like me). Every attempt was made
to avoid having respondents report directly about
their possession of emotional sensitivity/decoding
skills. Instead, items try to focus on feedback that

the respondent might have received from others
concerning his/her nonverbal sensitivity (e.g.,
“I am often told that I am a sensitive, understanding
person”; “When my friends are angry or upset, they
seek me out to help calm them down”). The overall
SSI and the ES and SS subscales have been well vali-
dated and are psychometrically sound (Riggio, 1986;
Riggio & Carney, 2003).
The ES subscale of the SSI correlates significantly
with performance measures of nonverbal decoding
from the DANVA–2 and from the emotional decod-
ing task from the abilities measure of emotional
intelligence (the Multifactor Emotional Intelligence
Scale; Mayer, Salovey, & Caruso, 1997). In addi-
tion, there is a positive correlation between the SS
subscale of the SSI and the IPT, suggesting that the
SS subscale measures more sophisticated nonverbal
(and social) sensitivity. Moreover, there are strong
positive correlations between the ES subscale of the
SSI and other self-report measures of emotional/
nonverbal sensitivity, including scales of emotional
contagion, affective intensity, and empathy
(Riggio & Carney, 2003).
The SSI has been used regularly in research,
although it is most often used as a measure of
overall social competence. Some researchers have
focused primarily on the ES subscale as a measure of
nonverbal sensitivity (e.g., Bloise & Johnson, 2007),
but there are still concerns among researchers about
the use of self-report measures of nonverbal skill,
with preference given to performance-based
measures, such as the DANVA–2.
Strengths and Limitations of Self-Report
Measures of Nonverbal Sensitivity
The obvious advantage is the ease of administration
that self-report assessments afford. Another advan-
tage of self-report methodology is that it can be
used to assess a wide range of skills across a variety
of channels and situations as well as with different
interactants. For example, a respondent can report
on success in reading cues in interpersonal relation-
ships, in formal and informal group settings, and in
structured work situations. Self-reports can focus
on the respondents’ perceptions of their abilities
(i.e., metaknowledge of communication skills and
success), or they can ask about success in specific
Measuring Nonverbal Sensitivity
communication events/interactions or about the
frequency with which they are able to communicate
successfully (i.e., decode accurately) in social situa-
tions. In sum, there is greater breadth and flexibility
in assessing the domain of nonverbal sensitivity
using self-report methods.
The primary limitation to self-report methodol-
ogy is whether individuals have insight into their
ability to accurately decode nonverbal cues. It has
been suggested, however, that relying on self-reports
of decoding success/failure might be a more valid
approach. For example, good self-report items of
nonverbal sensitivity should focus on past or typical
behaviors in which the person might have engaged
(e.g., “I must admit I rarely notice the irritation in
other people’s voices”), or they should focus on
what others have reported to the respondent about
their nonverbal sensitivity-related behaviors (e.g.,
“Other people have told me that I am a sensitive
person”; see Riggio & Riggio, 2001). An obvious
concern is socially desirable responding, as being
nonverbally sensitive is typically considered a
valued skill, particularly for women.
It is interesting that the same issue of perfor-
mance-based versus self-report methods prevails
in research on emotional intelligence. The two
different approaches are represented by the abili-
ties model of emotional intelligence compared to
the mixed model (see Mayer et al., 2000; Riggio,
2010b). The abilities model of emotional intel-
ligence is most frequently assessed through
performance-based methods that require, in part,
decoding skill. The mixed model measures are self-
reports and look more like personality measures
and attitude questionnaires. Research suggests that
abilities models of emotional intelligence are more
valid (Livingstone & Day, 2005; Van Rooy, Viswes-
varan, & Pluta, 2005), but self-report methods
are often used in studies of emotional intelligence
because of their ease of use.
SELF-REPORT MEASURES OF
CONSTRUCTS RELATED TO NONVERBAL
SENSITIVITY
There are several psychological constructs that are
related to nonverbal sensitivity (Losoya & Eisenberg,
597
Riggio and Darioly
2001). For example, the personality trait of empa-
thy, which involves the ability to recognize emo-
tions and feelings in others and to show concern for
them (Davis, 1983; Mehrabian & Epstein, 1972), is
closely related to nonverbal sensitivity. Yet, empathy
is a complex, multidimensional construct, involving
both the ability to recognize emotions in others and
to be able to take another’s perspective (e.g., Davis,
1983; Hogan, 1969).
Two specific empathy measures are interesting
because they focus on nonverbal sensitivity. The
first is Mehrabian and Epstein’s (1972) Question-
naire Measure of Emotional Empathy, which is a
33-item, self-report scale that measures recognition
and responsiveness to emotions in others. Mehra-
bian and Epstein suggested that emotional empathy
involves a susceptibility to others’ emotions, emo-
tional responsiveness/sympathy, and being affected
by others’ emotions. The Questionnaire Measure of
Emotional Empathy has been validated in a num-
ber of studies and has been shown to correlate with
empathy-related behaviors, such as helping and
aggression. Moreover, men were consistently found
to be less empathic than women (for a review, see
Mehrabian, Young, & Sato, 1988).
A very different perspective on empathy was
taken by Hogan (1969), who viewed empathy as a
sort of cognitive perspective-taking—understanding
another’s perspective and recognizing their emo-
tions/emotional states. Hogan developed a 64-item,
self-report scale: the Hogan Empathy Scale. The
items were selected from the California Psychologi-
cal Inventory (Gough, 1964) and the Minnesota
Multiphasic Personality Inventory (Hathaway &
McKinley, 1943) based on their ability to discrimi-
nate between previously assessed high- and
low-empathic groups.
Davis (1983) constructed a multidimensional
empathy scale, the Interpersonal Reactivity Index
(IRI), which consists of subscales designed to mea-
sure both the cognitive aspects of empathy (i.e., an
IRI subscale termed Perspective-Taking) and the
emotional elements. The emotional form of empa-
thy is further divided into two types by Davis. One
of the IRI subscales, which Davis termed Empathic
Concern, deals with decoding others’ emotions and
responding to them. A second emotional empathy
598
subscale termed Personal Distress measures empa-
thy as vicariously feeling another’s emotional state.
There are important connections between these
different types of empathy and nonverbal sensitiv-
ity, as each form of empathy requires the ability to
decode emotions in others. Although empathy is
distinct, and perhaps broader, than emotional/
nonverbal sensitivity, it is interesting that the sen-
sitivity subscales of the SSI (ES and SS) correlate
significantly and positively with all of the various
empathy scales and subscales (Riggio et al., 1989).
An additional construct that is closely aligned
with nonverbal sensitivity is the notion of emo-
tional contagion. Emotional contagion is closely
associated with the Personal Distress subscale of
the IRI. Emotional contagion occurs when indi-
viduals unconsciously attend to others and mimic
their emotional and nonverbal behavior. It is seen
as a more “primitive” form of empathy (Hatfield,
Cacioppo, & Rapson, 1992). Although viewed as a
process by which emotions are nonverbally commu-
nicated, and shared, from one individual to another
(Hatfield, Cacioppo, & Rapson, 1994), there has
been an attempt to conceptualize susceptibility to
emotional contagion as an individual difference
variable (Doherty, 1997). Doherty (1997) created
the 15-item, self-report Emotional Contagion Scale,
which assesses a respondent’s susceptibility to basic
emotions (happiness, love, anger, sadness, fear) in
others (e.g., “Being with a happy person picks me
up when I’m feeling down”; “It irritates me to be
around angry people”). The Emotional Contagion
Scale correlates significantly and positively with the
ES subscale of the SSI, and it correlates positively
with measures of empathy (Doherty, 1997; Riggio &
Carney, 2003).
As suggested by the social skills model (Riggio,
1986; Riggio & Carney, 2003), sensitivity (both
emotional/nonverbal and social sensitivity) is part
of a larger construct of social competence. Like-
wise, emotional/nonverbal sensitivity is a part of the
constellation of abilities that make up emotional
intelligence. In that vein, there are additional mea-
sures of social competence that contain elements
of nonverbal sensitivity. One such measure is the
self-report Interpersonal Competence Questionnaire
(Buhrmester, Furman, Wittenberg, & Reis, 1988).

One of the subscales is called Emotional Support
and includes elements of nonverbal/emotional sensi-
tivity. This subscale correlates strongly with the ES
subscale of the SSI, and it predicts emotional social
support from friends and social popularity. The
Emotional Support subscale from the Interpersonal
Competence Questionnaire does not, however,
correlate with performance measures of emotional
decoding skill (Buhrmester et al., 1988).
Measuring Empathic Accuracy
Ickes and colleagues (e.g., Ickes, 2001, 2009;
Ickes & Simpson, 2004) have studied the construct
of empathic accuracy. In short, empathic accuracy
is “everyday mind reading” that involves accurately
decoding other people’s thoughts and feelings
(Ickes, 1997, 2003). Unlike emotional sensitivity
that involves recognition of nonverbal emotional
cues, empathic accuracy is inferring the specific con-
tent of another person’s thoughts and feelings.
The typical procedure for assessing empathic
accuracy uses one of two formats: an unstructured
dyadic interaction that is videotaped when two par-
ticipants are left alone together, or the standard
stimulus paradigm where individual participants
view a standard set of videotaped interactions and
attempt to infer the thoughts and feelings of the
target persons (Gesn & Ickes, 1999; Ickes, 2001;
Marangoni, Garcia, Ickes, & Teng, 1995). In the
unstructured dyadic interaction, the participants
are later shown their videotape and asked to record
their own thoughts and feelings at various points
in the video and note the time. The participants
then watch the video a second time, and it is paused
at each point where the other participant noted a
thought or feeling, and the first participant is asked
to infer the other’s thought/feeling at each paused
point in the interaction. The standard stimulus para-
digm uses prerecorded tapes of two interactants who
have previously noted their thoughts and feelings at
various points in the videotape. The video is stopped
at these points, and the participant is asked to infer
the thoughts and feelings of the two interactants. In
each paradigm, trained coders assess the degree of
agreement between the interactant’s thoughts/feel-
ings and those inferred by the participant, and they
provide an empathic accuracy score.
Measuring Nonverbal Sensitivity
Research on empathic accuracy has centered
primarily on its role in interpersonal relationships.
For example, Simpson and colleagues (Simpson,
Ickes, & Oriña, 2001; Simpson, Oriña, & Ickes,
2003) found evidence that accurately reading your
partner to anticipate a need, avert a conflict, or keep
a small problem from escalating into a large one is
likely to be adaptive and healthy. Crosby (2002)
found that mothers who were accurate in inferring
their own child’s thoughts and feelings had children
with more positive self-concepts as family members.
Because several external raters assess the degree
of similarity between the perceiver’s empathic infer-
ences and the corresponding thoughts or feelings
that the target individual actually reported, one can
assess interrater reliability. Across all of the studies
conducted, the average interrater reliability has been
about .90 (Ickes, 2001). Establishing the convergent
and discriminant validity of the empathic accuracy
measure has proven to be difficult and complicated
(Davis & Kraus, 1997; Mortimer, 1996). Therefore,
it is possible to distinguish people who are empathi-
cally skilled from those who are empathically
challenged, and to then use this information as a
selection criterion, for instance, to select future
leaders, counselors, or psychotherapists.
Over the past 2 decades, vast numbers of
empathic accuracy studies have been reported.
Empathic accuracy research is now well established
in labs throughout the Belgium, England,
New Zealand, Switzerland, and the United States.
Moreover, researchers have demonstrated the poten-
tial of empathic accuracy research to bridge the major
areas of psychology in which the study of empathy is
now concentrated: clinical, cognitive, developmental,
social, and physiological psychology (for a review of
the more noteworthy findings in each of these areas,
see Ickes, 2001). One of the most important practical
implications of the research on empathic accuracy is
that it provides a reliable and objective measure of
individuals’ performance as everyday mind readers
(e.g., Marangoni et al., 1995).
Measuring Sensitivity to Deception
Another form of nonverbal sensitivity—and one
that also incorporates elements of sensitivity to
verbal communication, situational cues, and basic
599
Riggio and Darioly
knowledge of social norms and conventions—is
sensitivity to lying and truth-telling. There has been
a long history of trying to detect deception through
nonverbal cues. Most notable is the use of physi-
ological signs of arousal that are measured through
the polygraph. Polygraphs typically record nonverbal
cues associated with arousal that are recorded while a
person is being questioned about truths and lies (see
Grubin & Madsen, 2005, for an historical overview).
The polygraph usually measures heart and respira-
tory rate and/or palmar sweating via galvanic skin
response electrodes placed on the finger/hand. There
is a great deal of controversy over both the validity
and the reliability of the polygraph, with ranges of
its accuracy estimated to range from slightly above
chance to 90% or better accuracy (with the higher
estimates usually coming from persons associated
with the polygraph industry; Iacono, 1991, 2008).
Another long line of research has attempted to
detect deception purely through observation of
nonverbal cues (e.g., DePaulo, Rosenthal, Rosen-
krantz, & Green, 1982; Klaver, Lee, & Hart, 2007;
Miller & Stiff, 1993; Rockwell, Buller, & Burgoon,
1997; Vrij, Akehurst, Soukara, & Bull, 2004). Yet,
nonverbal sensitivity to deception is not very accu-
rate. Meta-analyses of accuracy at detecting decep-
tion have found that the typical rates of lie–truth
discrimination are only slightly more than 50%
(where 50% is chance; Bond & DePaulo, 2006;
Kraut, 1980; Vrij, 2000). Some factors facilitate
lie–truth discrimination, and others inhibit it. For
instance, lies are more detectable when they can
be heard. By contrast, it has been suggested that
facial behaviors provide no indication of a speaker’s
veracity (Bond & DePaulo, 2006; Ekman & Friesen,
1969; but see Chapter 10, this handbook). Ekman
(2001) attributed failures at lie detection to many
factors, such as socialization to overlook lies, poor
evolutionary preparation, the benefits of trust, and
inadequate feedback concerning accuracy/errors.
Interestingly, professionals—such as police officers,
robbery investigators, judges, customs inspectors,
federal law enforcement officers, federal polyg-
raphers, and psychiatrists—are no more capable
at detecting deception than the average person
(Aamodt & Custer, 2006; Bond & DePaulo, 2006;
Ekman & O’Sullivan, 1991).
600
Ekman, O’Sullivan, and colleagues were able to
study a rare group of individuals who were particu-
larly good at accurately decoding honest and decep-
tive communications (Ekman, O’Sullivan, & Frank,
1999; O’Sullivan & Ekman, 2004). Termed the
“wizards,” these individuals are significantly above
chance at detecting deception and do it consistently.
Representing less than 10% of the population, these
individuals seem to be particularly sensitive to sub-
tle nonverbal cues and discrepancies between verbal
and nonverbal behavior. In addition, the wizards
use their knowledge of lying behavior, the situations
in which people might lie, and their knowledge of
emotions in detecting deception—consistent with
the broader categorization of this form of nonverbal
sensitivity as a larger social competence/intelligence
construct (O’Sullivan, 2005).
THE FUTURE OF ASSESSING NONVERBAL
SENSITIVITY
There has been renewed interest in measuring non-
verbal sensitivity as a core construct of emotional
intelligence and interpersonal competence. Emo-
tional decoding is a part of the most commonly used
performance measures of emotional intelligence
(e.g., Mayer, Salovey, & Caruso, 2002), and it has
been used as a proxy for emotional intelligence
(e.g., Rubin et al., 2005). Yet, the obvious connec-
tion between nonverbal sensitivity (and nonverbal
encoding, for that matter) and emotional intelli-
gence is not well known among researchers other
than those who investigate nonverbal communica-
tion (Riggio, 2010a).
One reason why there is not more research
on nonverbal sensitivity is the time-consuming
nature of the performance-based measures and the
lack of solid validity for self-report measures of
nonverbal sensitivity. Another concern is that the
readily available and better researched measures
(i.e., PONS, DANVA–2, IPT) are rather narrow in
their focus. Researchers interested in nonverbal/
emotional competencies are more likely to turn to
measures of emotional intelligence, which offer
assessment of a broader range of emotional skills.
Given the popularity of the construct of emotional
intelligence, it would make sense for these two

worlds to come together. Nonverbal communica-
tion researchers offer precision in the assessment
of nonverbal sensitivity (and other nonverbal/
emotional abilities)—making nonverbal measures
easily available, and relevant, to researchers inter-
ested in investigating the broader construct of
emotional intelligence—and an awareness of the
different streams of research could be fruitful.
Moreover, there is growing use of computer-
generated technology in research on nonverbal com-
munication, particularly the use of avatars to simu-
late emotional expressions (e.g., using immersive
virtual environment technology; Blascovich et al.,
2002). To date, various projects are being conducted
in the United States and Switzerland using new com-
putational methods for automatic extraction of non-
verbal behavior from face-to-face interaction from
audio, video, and mobile sensors (e.g., the Sinergia
interdisciplinary program of the Swiss National
Science Foundation).
It will also be interesting to watch the develop-
ment of alternative measurement approaches. More
implicit measures of nonverbal sensitivity would be
options. Moreover, studies that have the potential
to reveal real-life interpersonal consequences of
nonverbal sensitivity are needed. If resources allow,
multiple methods should be used. Meta-analyses
that examine results for different methods are par-
ticularly valuable (Elfenbein & Ambady, 2002).
NONVERBAL SENSITIVITY: IMPLICATIONS
AND APPLICATIONS
It is clear that there is untapped potential in the
construct of nonverbal sensitivity and its assess-
ment. Nonverbal sensitivity is an important variable
that is clearly related to the popular construct of
emotional intelligence and to interpersonal skills/
competencies. It is related to core psychological
constructs such as empathy, sympathy, compassion,
and nurturance, and it is negatively related to social
anxiety (Riggio & Carney, 2003). This suggests that
nonverbal sensitivity plays an important role in the
development and maintenance of interpersonal rela-
tionships and psychosocial adjustment (Riggio &
Zimmerman, 1991). Historically, there has been a
great deal of interest in the role that interpersonal
Measuring Nonverbal Sensitivity
and social skills play in mental health and psycho-
social adjustment (e.g., Trower, Bryant, & Argyle,
1978). For example, deficits in nonverbal sensi-
tivity have been associated with impaired social
relationships (Feldman, Philippot, & Custrini,
1991; Nowicki & Duke, 2001; Vosk, Forehand, &
Figueroa, 1983), with autism (McGee & Morrier,
2003), with schizophrenia (see Perez & Riggio,
2003, for a review), and with problems associated
with alcoholism (Philippot, Kornreich, & Blairy,
2003). Therefore, performance-based measures of
nonverbal sensitivity have been used as diagnostic
tools in clinical work and to measure progress
following clinical interventions.
Another interesting application of nonverbal
sensitivity measures involves examining the role
that nonverbal sensitivity plays in the workplace.
One line of research has examined nonverbal
“eavesdropping,” operationalized as sensitivity to
nonverbal cues of emotions. Increased sensitivity
to positive emotions in the workplace is associ-
ated with more positive evaluations of the decoder.
However, the ability to accurately decode negative
emotions in coworkers or to detect subtle cues that
the sender might be trying to hide is associated with
lower evaluations of the nonverbal “eavesdropper,”
suggesting that too much sensitivity may make
coworkers uncomfortable (Elfenbein & Ambady,
2002; Puccinelli & Tickle-Degnen, 2004).
A workplace context that seems particularly
relevant for sensitivity to nonverbal behavior is the
employment interview. The interviewer’s task is to
gather as much information as possible concern-
ing the applicant’s suitability for a job. Ability to
accurately decode nonverbal cues should relate
to interviewer effectiveness. Interviewer success
is somewhat akin to detecting truth–deception
through nonverbal cues (see Riggio, 2005, for a
discussion of nonverbal behavior/sensitivity in the
workplace).
There has also been some research examining
the role that nonverbal sensitivity plays in effective
leadership. According to many leadership theo-
ries, responding to followers’ feelings, needs, and
concerns is critical for effective leadership (Bass &
Riggio, 2006; Riggio, 2014). Therefore, it makes
sense that more effective leaders should possess
601
Riggio and Darioly
high levels of nonverbal sensitivity, and research has
indeed found this relationship (Rubin et al., 2005).
Leadership research, for the most part, has focused
on the broader construct of emotional intelligence
as a key to leadership success (e.g., Goleman, 1998),
with nonverbal sensitivity being a key component of
emotional intelligence.
In summary, there is growing interest in the
role that interpersonal skills play in success in
social life, occupations, and the like. Yet, there
is little consensus on just what the interpersonal
skills necessary for success entail. There is, how-
ever, agreement that nonverbal skills, in general,
and nonverbal sensitivity, in particular, are criti-
cal. However, nonverbal skills are elusive. They
are hard to precisely define and even more difficult
to measure. There have been some advancements
in the measurement of nonverbal sensitivity, but
there is a great need for assessment tools that are
valid, reliable, and easy to use. Hopefully, research
on this very interesting construct will grow as
scholars realize its importance.
References
Aamodt, M. G., & Custer, H. (2006). Who can best catch
a liar? A meta-analysis of individual differences in
detecting deception. Forensic Examiner, 15, 6–11.
Archer, D. (1980). How to expand your social intelligence
quotient. New York, NY: M. Evans.
Archer, D., & Akert, R. M. (1977). Words and
everything else: Verbal and nonverbal cues in social
interpretation. Journal of Personality and Social
Psychology, 35, 443–449. http://dx.doi.org/10.1037/
0022-3514.35.6.443
Archer, D., & Costanzo, M. (1987). The Interpersonal
Perception Task [DVD]. Retrieved from http://www.
berkeleymedia.com/catalog/berkeleymedia/films/
communication/nonverbal_communication/the_
interpersonal_perception_task
Archer, D., Costanzo, M., & Akert, R. (2001). The
Interpersonal Perception Task (IPT): Alternative
approaches to problems of theory and design. In J. A.
Hall & F. J. Bernieri (Eds.), Interpersonal sensitivity:
Theory and measurement (pp. 161–182). Mahwah,
NJ: Erlbaum.
Bänziger, T., Grandjean, D., & Scherer, K. R. (2009).
Emotion recognition from expressions in face, voice,
and body: The Multimodal Emotion Recognition
Test (MERT). Emotion, 9, 691–704. http://dx.doi.
org/10.1037/a0017088
602
Bänziger, T., Scherer, K. R., Hall, J. A., & Rosenthal,
R. (2011). Introducing the MiniPONS: A short
multichannel version of the Profile of Nonverbal
Sensitivity. Journal of Nonverbal Behavior, 35,
189–204. http://dx.doi.org/10.1007/s10919-011-
0108-3
Bass, B. M., & Riggio, R. E. (2006). Transformational
leadership (2nd ed.). Mahwah, NJ: Erlbaum.
Baum, K. M., & Nowicki, S., Jr. (1998). Perception
of emotion: Measuring decoding accuracy of
adult prosodic cues varying in intensity. Journal
of Nonverbal Behavior, 22, 89–107. http://dx.doi.
org/10.1023/A:1022954014365
Bernieri, F. J. (1991). Interpersonal sensitivity in teaching
interactions. Personality and Social Psychology
Bulletin, 17, 98–103. http://dx.doi.org/10.1177/
0146167291171015
Bernieri, F. J. (2001). Toward a taxonomy of
interpersonal sensitivity. In J. A. Hall & F. J.
Bernieri (Eds.), Interpersonal sensitivity: Theory and
measurement (pp. 3–20). Mahwah, NJ: Erlbaum.
Blascovich, J., Loomis, J., Beall, A. C., Swinth, K. R.,
Hoyt, C. L., & Bailenson, J. N. (2002). Immersive
virtual environment technology as a methodological
tool for social psychology. Psychological
Inquiry, 13, 103–124. http://dx.doi.org/10.1207/
S15327965PLI1302_01
Bloise, S. M., & Johnson, M. K. (2007). Memory for
emotional and neutral information: Gender and
individual differences in emotional sensitivity.
Memory, 15, 192–204. http://dx.doi.org/10.1080/
09658210701204456
Bond, C. F., Jr., & DePaulo, B. M. (2006). Accuracy
of deception judgments. Personality and Social
Psychology Review, 10, 214–234. http://dx.doi.
org/10.1207/s15327957pspr1003_2
Buck, R. (1976). A test of nonverbal receiving ability:
Preliminary studies. Human Communication
Research, 2, 162–171. http://dx.doi.
org/10.1111/j.1468-2958.1976.tb00708.x
Buhrmester, D., Furman, W., Wittenberg, M. T., &
Reis, H. T. (1988). Five domains of interpersonal
competence in peer relationships. Journal of
Personality and Social Psychology, 55, 991–1008.
http://dx.doi.org/10.1037/0022-3514.55.6.991
Cherniss, C. (2010). Emotional intelligence:
Toward clarification of a concept. Industrial and
Organizational Psychology: Perspectives on Science
and Practice, 3, 110–126. http://dx.doi.org/10.1111/
j.1754-9434.2010.01231.x
Costanzo, M., & Archer, D. (1989). Interpreting the
expressive behavior of others: The Interpersonal
Perception Task. Journal of Nonverbal Behavior, 13,
225–245. http://dx.doi.org/10.1007/BF00990295

Crosby, L. (2002). The relation of maternal empathic
accuracy to the development of self-concept
(Unpublished doctoral dissertation). Fielding
Institute, Santa Barbara, CA.
Cusi, A. M., MacQueen, G. M., & McKinnon, M. C.
(2012). Patients with bipolar disorder show impaired
performance on complex tests of social cognition.
Psychiatry Research, 200, 258–264. http://dx.doi.
org/10.1016/j.psychres.2012.06.021
Davis, M. H. (1983). Measuring individual differences in
empathy: Evidence for a multidimensional approach.
Journal of Personality and Social Psychology, 44, 113–126.
http://dx.doi.org/10.1037/0022-3514.44.1.113
Davis, M. H., & Kraus, L. A. (1997). Personality and
empathic accuracy. In W. Ickes (Ed.), Empathic
accuracy (pp. 144–168). New York, NY: Guilford
Press.
DePaulo, B. M., Rosenthal, R., Rosenkrantz, J., & Green,
C. R. (1982). Actual and perceived cues to deception:
A closer look at speech. Basic and Applied Social
Psychology, 3, 291–312. http://dx.doi.org/10.1207/
s15324834basp0304_6
DiMatteo, M. R., Friedman, H. S., & Taranta, A. (1979).
Sensitivity to bodily nonverbal communication as
a factor in practitioner–patient rapport. Journal of
Nonverbal Behavior, 4, 18–26. http://dx.doi.org/
10.1007/BF00986909
Doherty, R. W. (1997). The Emotional Contagion
Scale: A measure of individual difference. Journal
of Nonverbal Behavior, 21, 131–154. http://dx.doi.
org/10.1023/A:1024956003661
Ekman, P. (2001). Telling lies: Clues to deceit in the
marketplace, politics, and marriage (3rd ed.). New
York, NY: Norton.
Ekman, P., & Friesen, W. V. (1969). Nonverbal leakage
and clues to deception. Psychiatry, 32, 88–106.
Ekman, P., & Friesen, W. V. (1974). Nonverbal behavior
and psychopathology. In R. J. Friedman & M. Katz
(Eds.), The psychology of depression: Contemporary
theory and research (pp. 3–31). Washington, DC:
Winston & Sons.
Ekman, P., & Friesen, W. V. (1975). Unmasking the face.
Englewood Cliffs, NJ: Prentice Hall.
Ekman, P., & O’Sullivan, M. (1991). Who can catch a
liar? American Psychologist, 46, 913–920. http://
dx.doi.org/10.1037/0003-066X.46.9.913
Ekman, P., O’Sullivan, M., & Frank, M. G. (1999). A few
can catch a liar. Psychological Science, 10, 263–266.
http://dx.doi.org/10.1111/1467-9280.00147
Elfenbein, H. A., & Ambady, N. (2002). Predicting
workplace outcomes from the ability to eavesdrop on
feelings. Journal of Applied Psychology, 87, 963–971.
http://dx.doi.org/10.1037/0021-9010.87.5.963
Measuring Nonverbal Sensitivity
Eysenck, S. B., & Eysenck, H. J. (1968). The
measurement of psychoticism: A study of factor
stability and reliability. British Journal of Social
and Clinical Psychology, 7, 286–294. http://dx.doi.
org/10.1111/j.2044-8260.1968.tb00571.x
Feldman, R. S., Philippot, P., & Custrini, R. J. (1991).
Social competence and nonverbal behavior. In
R. S. Feldman & B. Rime (Eds.), Fundamentals of
nonverbal behavior (pp. 329–350). New York, NY:
Cambridge University Press.
Fujita, B. N., Harper, R. G., & Wiens, A. N. (1980).
Encoding–decoding of nonverbal emotional
messages: Sex differences in spontaneous and
enacted expressions. Journal of Nonverbal Behavior, 4,
131–145. http://dx.doi.org/10.1007/BF00986815
Garwood, R., Guiora, A. Z., & Kalter, N. (1970).
Manifest anxiety and perception of micromomentary
expression. Comprehensive Psychiatry, 11, 576–580.
http://dx.doi.org/10.1016/0010-440X(70)90021-0
Gesn, P. R., & Ickes, W. (1999). The development of
meaning contexts for empathic accuracy: Channel
and sequence effects. Journal of Personality and
Social Psychology, 77, 746–761. http://dx.doi.
org/10.1037/0022-3514.77.4.746
Goleman, D. (1998). Working with emotional intelligence.
New York, NY: Bantam.
Gough, H. G. (1964). Manual for the California
Psychological Inventory (Rev. ed.). Palo Alto, CA:
Consulting Psychologists Press.
Grubin, D., & Madsen, L. (2005). Lie detection and the
polygraph: A historical review. Journal of Forensic
Psychiatry and Psychology, 16, 357–369. http://
dx.doi.org/10.1080/14789940412331337353
Hall, J. (1990). Nonverbal sex differences: Communication
accuracy and expressive style. Baltimore, MD: Johns
Hopkins University Press.
Hall, J. A. (2001). The PONS test and the psychometric
approach to measuring interpersonal sensitivity.
In J. A. Hall & F. J. Bernieri (Eds.), Interpersonal
sensitivity: Theory and measurement (pp. 143–160).
Mahwah, NJ: Erlbaum.
Hall, J. A., Andrzejewski, S. A., & Yopchick, J. E. (2009).
Psychosocial correlates of interpersonal sensitivity:
A meta-analysis. Journal of Nonverbal Behavior, 33,
149–180. http://dx.doi.org/10.1007/s10919-009-0070-5
Hall, J. A., & Bernieri, F. J. (2001). Interpersonal
sensitivity: Theory and measurement. Mahwah, NJ:
Erlbaum.
Hall, J. A., Halberstadt, A. G., & O’Brien, C. E. (1997).
Subordination and nonverbal sensitivity: A
study and synthesis of findings based on trait
measures. Sex Roles, 37, 295–317. http://dx.doi.
org/10.1023/A:1025608105284
603
Riggio and Darioly
Hall, J. A., & Matsumoto, D. (2004). Gender differences
in judgments of multiple emotions from facial
expressions. Emotion, 4, 201–206. http://dx.doi.
org/10.1037/1528-3542.4.2.201
Hatfield, E., Cacioppo, J. T., & Rapson, R. L. (1992).
Primitive emotional contagion. In M. S. Clark (Ed.),
Emotion and social behavior (pp. 151–177). Thousand
Oaks, CA: Sage.
Hatfield, E., Cacioppo, J. T., & Rapson, R. L. (1994).
Emotional contagion. New York, NY: Cambridge
University Press.
Hathaway, S. R., & McKinley, J. C. (1943). Manual for
the Minnesota Multiphasic Personality Inventory. New
York, NY: Psychological Corporation.
Hogan, R. (1969). Development of an empathy scale.
Journal of Consulting and Clinical Psychology, 33,
307–316. http://dx.doi.org/10.1037/h0027580
Horgan, T. G., & Smith, J. L. (2006). Interpersonal reasons
for interpersonal perceptions: Gender-incongruent
purpose goals and nonverbal judgment accuracy.
Journal of Nonverbal Behavior, 30, 127–140. http://
dx.doi.org/10.1007/s10919-006-0012-4
Iacono, W. G. (1991). Can we determine the accuracy of
polygraph tests? In J. R. Jennings, P. K. Ackles, &
M. G. H. Coles (Eds.), Advances in psychophysiology:
A research annual (Vol. 4, pp. 201–207). London,
England: Jessica Kingsley.
Iacono, W. G. (2008). Accuracy of polygraph techniques:
Problems using confessions to determine ground
truth. Physiology and Behavior, 95, 24–26. http://
dx.doi.org/10.1016/j.physbeh.2008.06.001
Ickes, W. (Ed.). (1997). Empathic accuracy. New York,
NY: Guilford Press.
Ickes, W. (2001). Measuring empathic accuracy. In J. A.
Hall & F. Bernieri (Eds.), Interpersonal sensitivity:
Theory and measurement (pp. 219–241). Mahwah,
NJ: Erlbaum.
Ickes, W. (2003). Everyday mind reading: Understanding
what other people think and feel. Amherst, NY:
Prometheus Books.
Ickes, W. (2009). Empathic accuracy: Its links to clinical,
cognitive, developmental, social, and physiological
psychology. In J. Decety & W. Ickes (Eds.), The
social neuroscience of empathy (pp. 57–70). http://
dx.doi.org/10.7551/mitpress/9780262012973.
003.0006
Ickes, W., & Simpson, J. A. (2004). Managing empathic
accuracy in close relationships. In H. T. Reis & C. E.
Rusbult (Eds.), Close relationships: Key readings (pp.
345–363). Philadelphia, PA: Taylor & Francis.
John, O. (1989). The BFI–54. Unpublished test, Institute
of Personality and Social Research, Department of
Psychology, University of California, Berkeley.
604
Klaver, J. R., Lee, Z., & Hart, S. D. (2007). Psychopathy
and nonverbal indicators of deception in offenders.
Law and Human Behavior, 31, 337–351. http://dx.doi.
org/10.1007/s10979-006-9063-7
Kraut, R. (1980). Humans as lie detectors: Some second
thoughts. Journal of Communication, 30, 209–218.
http://dx.doi.org/10.1111/j.1460-2466.1980.tb02030.x
Lancelot, C., & Nowicki, S., Jr. (1997). The association
between receptive nonverbal processing abilities and
internalizing/externalizing problems in girls and
boys. Journal of Genetic Psychology, 158, 297–302.
http://dx.doi.org/10.1080/00221329709596669
Livingstone, H. A., & Day, A. L. (2005). Comparing the
construct and criterion-related validity of ability-
based and mixed-model measures of emotional
intelligence. Educational and Psychological
Measurement, 65, 757–779. http://dx.doi.org/
10.1177/0013164405275663
Losoya, S. H., & Eisenberg, N. (2001). Affective empathy.
In J. A. Hall & F. Bernieri (Eds.), Interpersonal
sensitivity: Theory and measurement (pp. 21–43).
Mahwah, NJ: Erlbaum.
Marangoni, C., Garcia, S., Ickes, W., & Teng, G. (1995).
Empathic accuracy in a clinically relevant setting.
Journal of Personality and Social Psychology, 68,
854–869. http://dx.doi.org/10.1037/0022-3514.
68.5.854
Matsumoto, D., LeRoux, J., Wilson-Cohn, C., Raroque,
J., Kooken, K., Ekman, P., . . . Goh, A. (2000). A
new test to measure emotion recognition ability:
Matsumoto and Ekman’s Japanese and Caucasian
Brief Affect Recognition Test (JACBART). Journal
of Nonverbal Behavior, 24, 179–209. http://dx.doi.
org/10.1023/A:1006668120583
Mayer, J. D., & Salovey, P. (1997). What is emotional
intelligence? In P. Salovey & D. J. Sluyter (Eds.),
Emotional development and emotional intelligence (pp.
3–34). New York, NY: Basic Books.
Mayer, J. D., Salovey, P., & Caruso, D. R. (1997).
Multifactor Emotional Intelligence Scale (Pencil-and-
Paper Version). Needham, MA: Virtual Knowledge.
Mayer, J. D., Salovey, P., & Caruso, D. R. (2000). Models
of emotional intelligence. In R. J. Sternberg (Ed.),
Handbook of intelligence (2nd ed., pp. 396–420).
http://dx.doi.org/10.1017/CBO9780511807947.019
Mayer, J. D., Salovey, P., & Caruso, D. R. (2002).
Mayer–Salovey–Caruso Emotional Intelligence Test
user’s manual. Toronto, Ontario, Canada: Multi-
Health Systems.
McGee, G., & Morrier, M. (2003). Clinical implications
of research in nonverbal behavior of children with
autism. In P. Philippot, E. J. Coats, & R. S. Feldman
(Eds.), Nonverbal behavior in clinical settings (pp.
287–317). New York, NY: Oxford University Press.
 Measuring Nonverbal Sensitivity

Mehrabian, A., & Epstein, N. (1972). A measure of Philippot, P., Kornreich, C., & Blairy, S. (2003).
emotional empathy. Journal of Personality, 40, Nonverbal deficits and interpersonal regulation in
525–543. http://dx.doi.org/10.1111/j.1467- alcoholics. In P. Philippot, E. J. Coats, & R. S. Feldman
6494.1972.tb00078.x (Eds.), Nonverbal behavior in clinical settings (pp.
209–231). New York, NY: Oxford University Press.
Mehrabian, A., Young, A. L., & Sato, S. (1988).
Emotional empathy and associated individual Pitterman, H., & Nowicki, S., Jr. (2004). A test of the
differences. Current Psychology, 7, 221–240. http:// ability to identify emotion in human standing
dx.doi.org/10.1007/BF02686670 and sitting postures: The Diagnostic Analysis
of Nonverbal Accuracy—2 Posture Test
Miller, G. R., & Stiff, J. B. (1993). Deceptive
(DANVA2–POS). Genetic, Social, and General
communication. Newbury Park, CA: Sage.
Psychology Monographs, 130, 146–162. http://dx.doi.
Mortimer, D. C. (1996). “Reading” ourselves “reading” org/10.3200/MONO.130.2.146-162
others: Actual versus self-estimated empathic accuracy
Puccinelli, N. M., & Tickle-Degnen, L. (2004). Knowing
(Unpublished master’s thesis). University of Texas at
too much about others: Moderators of the relationship
Arlington.
between eavesdropping and rapport in social inter­
Mufson, L., & Nowicki, S., Jr. (1991). Factors affecting action. Journal of Nonverbal Behavior, 28, 223–243.
the accuracy of facial affect recognition. Journal of http://dx.doi.org/10.1007/s10919-004-4157-8
Social Psychology, 131, 815–822. http://dx.doi.org/
Riggio, R. E. (1986). Assessment of basic social skills.
10.1080/00224545.1991.9924668
Journal of Personality and Social Psychology,
Nowicki, S., Jr., & Carton, E. (1997). The relation of 51, 649–660. http://dx.doi.org/10.1037/0022-
nonverbal processing ability of faces and voices and 3514.51.3.649
children’s feelings of depression and competence.
Riggio, R. E. (1989). Manual for the Social Skills Inventory.
Journal of Genetic Psychology, 158, 357–364. http://
Palo Alto, CA: Consulting Psychologists Press.
dx.doi.org/10.1080/00221329709596674
Riggio, R. E. (2005). Business applications of nonverbal
Nowicki, S., Jr., & Duke, M. P. (1994). Individual
communication. In R. E. Riggio & R. S. Feldman
differences in the nonverbal communication of affect:
(Eds.), Applications of nonverbal communication
The Diagnostic Analysis of Nonverbal Accuracy
research (pp. 119–138). Mahwah, NJ: Erlbaum.
scale. Journal of Nonverbal Behavior, 18, 9–35. http://
dx.doi.org/10.1007/BF02169077 Riggio, R. E. (2010a). Before emotional intelligence:
Research on nonverbal, emotional, and social
Nowicki, S., Jr., & Duke, M. P. (2001). Nonverbal
competencies. Industrial and Organizational
receptivity: The Diagnostic Analysis of Nonverbal
Psychology: Perspectives on Science and Practice, 3,
Accuracy (DANVA). In J. A. Hall & F. J. Bernieri
178–182. http://dx.doi.org/10.1111/j.1754-9434.
(Eds.), Interpersonal sensitivity: Theory and
2010.01221.x
measurement (pp. 183–198). Mahwah, NJ: Erlbaum.
Riggio, R. E. (2010b). Emotional intelligence and
Nowicki, S., Jr., & Mitchell, J. (1998). Accuracy in
interpersonal competencies. In M. G. Rothstein &
identifying affect in child and adult faces and voices
R. J. Burke (Eds.), Self-management and leadership
and social competence in preschool children.
development (pp. 160–182). http://dx.doi.
Genetic, Social, and General Psychology Monographs,
org/10.4337/9781849805551.00013
124, 39–59.
Riggio, R. E. (2014). A social skills model for
O’Sullivan, M. (2005). Emotional intelligence and
understanding the foundations of leader
deception detection: Why most people can’t
communication. In R. E. Riggio & S. J. Tan (Eds.),
“read” others, but a few can. In R. E. Riggio &
Leader interpersonal and influence skills: The soft skills
R. S. Feldman (Eds.), Applications of nonverbal
of leadership (pp. 31–49). New York, NY: Routledge/
communication (pp. 215–253). Mahwah, NJ:
Psychology Press.
Erlbaum.
Riggio, R. E., & Carney, D. C. (2003). Social Skills
O’Sullivan, M., & Ekman, P. (2004). The wizards of
Inventory manual (2nd ed.). Redwood City, CA: Mind
deception detection. In P. Granhag & L. Stromwall
Garden.
(Eds.), The detection of deception in forensic
contexts (pp. 269–286). http://dx.doi.org/10.1017/ Riggio, R. E., & Riggio, H. R. (2001). Self-report
CBO9780511490071.012 measurement of interpersonal sensitivity. In J. A.
Hall & F. Bernieri (Eds.), Interpersonal sensitivity:
Perez, J. E., & Riggio, R. E. (2003). Nonverbal social
Theory and measurement (pp. 127–142). Mahwah,
skills and psychopathology. In P. Philippot,
NJ: Erlbaum.
E. J. Coats, & R. S. Feldman (Eds.), Nonverbal
behavior in clinical settings (pp. 17–44). New York, Riggio, R. E., Tucker, J. S., & Coffaro, D. (1989). Social
NY: Oxford University Press. skills and empathy. Personality and Individual

605
Riggio and Darioly
Differences, 10, 93–99. http://dx.doi.org/10.1016/
0191-8869(89)90184-0
Riggio, R. E., & Zimmerman, J. A. (1991). Social skills
and interpersonal relationships: Influences on social
support and support seeking. In W. H. Jones & D.
Perlman (Eds.), Advances in personal relationships:
Vol. 2. A research annual (pp. 133–155). Oxford,
England: Jessica Kingsley.
Rockwell, P., Buller, D. B., & Burgoon, J. K. (1997). The
voice of deceit: Refining and expanding vocal cues to
deception. Communication Research Reports,
14, 451–459. http://dx.doi.org/10.1080/
08824099709388688
Rosenthal, R. (Ed.). (1979). Skill in nonverbal
communication: Individual differences. Cambridge,
MA: Oelgeschlager, Gunn, & Hain.
Rosenthal, R., Hall, J. A., DiMatteo, M. R., Rogers, P. L.,
& Archer, D. (1979). Sensitivity to nonverbal
communications: The PONS test. Baltimore, MD: Johns
Hopkins University Press.
Rubin, R. S., Munz, D. C., & Bommer, W. H. (2005).
Leading from within: The effects of emotion
recognition and personality on transformational
leadership behavior. Academy of Management
Journal, 48, 845–858. http://dx.doi.org/10.5465/
AMJ.2005.18803926
Scherer, K. R., Banse, R., Wallbott, H. G., & Goldbeck,
T. (1991). Vocal cues in emotion encoding and
decoding. Motivation and Emotion, 15, 123–148.
http://dx.doi.org/10.1007/BF00995674
Scherer, K. R., & Scherer, U. (2011). Assessing the
ability to recognize facial and vocal expressions of
emotion: Construction and validation of the Emotion
Recognition Index (ERI). Journal of Nonverbal
Behavior, 35, 305–326. http://dx.doi.org/10.1007/
s10919-011-0115-4
Simpson, J. A., Ickes, W., & Oriña, M. (2001). Empathic
accuracy and preemptive relationship maintenance.
In J. Harvey & A. Wenzel (Eds.), Close romantic
relationships: Maintenance and enhancement
(pp. 27–46). Mahwah, NJ: Erlbaum.
Simpson, J. A., Oriña, M. M., & Ickes, W. (2003). When
accuracy hurts, and when it helps: A test of the
606
empathic accuracy model in marital interactions.
Journal of Personality and Social Psychology, 85,
881–893. http://dx.doi.org/10.1037/0022-3514.
85.5.881
Taylor, L. L., Guiora, A. Z., Catford, J. C., & Lane, H. L.
(1969). The role of personality variables in second
language behavior. Comprehensive Psychiatry, 10,
463–474. http://dx.doi.org/10.1016/0010-440X
(69)90082-0
Trimboli, A., & Walker, M. (1993). The CAST test of
nonverbal sensitivity. Journal of Language and Social
Psychology, 12, 49–65. http://dx.doi.org/10.1177/
0261927X93121004
Trower, P., Bryant, B., & Argyle, M. (1978). Social skills
and mental health. Pittsburgh, PA: University of
Pittsburgh Press.
Tucker, J. S., & Riggio, R. E. (1988). The role of social
skills in encoding posed and spontaneous facial
expressions. Journal of Nonverbal Behavior, 12,
87–97. http://dx.doi.org/10.1007/BF00986927
Van Rooy, D. L., Viswesvaran, C., & Pluta, P. (2005).
An evaluation of construct validity: What is
this thing called emotional intelligence? Human
Performance, 18, 445–462. http://dx.doi.org/10.1207/
s15327043hup1804_9
Vosk, B. N., Forehand, R., & Figueroa, R. (1983).
Perception of emotions by accepted and rejected
children. Journal of Behavioral Assessment, 5,
151–160. http://dx.doi.org/10.1007/BF01321447
Vrij, A. (2000). Detecting lies and deceit: The psychology
of lying and the implications for professional practice.
New York, NY: Wiley.
Vrij, A., Akehurst, L., Soukara, S., & Bull, R. (2004).
Detecting deceit via analyses of verbal and
nonverbal behavior in children and adults.
Human Communication Research, 30, 8–41.
http://dx.doi.org/10.1111/j.1468-2958.2004.
tb00723.x
Zuckerman, M., & Larrance, D. (1979). Individual
differences in perceived encoding and decoding
abilities. In R. Rosenthal (Ed.), Skill in nonverbal
communication: Individual differences (pp. 171–203).
Cambridge, MA: Oelgeschlager, Gunn, & Hain.
 Index

Aarde, S. M., 60 Ahn, S. J., 436 Anxiety, 83. See also Fear; Neuroticism
Aarts, H., 377 Ahren, M.-P., 371, 545 gaze behavior and, 351
Abe, J. A. A., 274 Ahrens, R., 336 nonverbal decoding, 175–177
Abney, S., 484 Aiello, J. R., 553–555 odor and, 366–367
About Face (Cole), 5 Akert, R. M., 177 recognition of fear faces, 275
Acoustic analog expression, 321 Al Dujaili, E. A. S., 231 Aphasias, 297, 299, 315, 439–440
Acta Psychologica, 18 Alexithymia, 339, 450–452 Appraisal processes, 30, 266, 268
ActivPAL Professional system, 576 Allometry Architectural elements, effect of on non-
Adams, R. B., Jr., 31, 144, 338, 350–351 dominance, 64 verbal communication, 201–202
Adaptive value of nonverbal cues, 9–10 facial mobility, 406 Argyle, M., 348–350, 352
Adaptors (self-adaptors), 308–309, 394, Allport, G. W., 164 Argyros, A., 568
503, 515, 554, 557 Althoff, R. R., 533 Aristotle, 10, 19, 232
Adiposity, facial, 226–228 Altman, I., 83, 189, 206–207, 392 Artificial facial signs, 257
Adobe Audition software, 488 Altmann, J., 563 Atkinson, A. P., 387–388
AEDS. See Automated emotion detection Altun, Y., 474 Atterer, M., 485
systems Alvarado, N., 266 Attractiveness halo effect, 221, 238
Affect, Imagery, and Consciousness Ambady, N., 173–174, 237–238, 242–243 Attunement, 31, 116, 124
(Tomkins), 260 Anandan, P., 570 Autism, 275, 326, 337, 342, 351–352
Affect displays, 24, 308 Ananthakrishnan, S., 471–472, 477, 487 AuToBI software, 488–489
AFFEX (System for Identifying Affect Ancoli, S., 266 Automated emotion detection systems
Expressions by Holistic Andersen, P. A., 10, 32 (AEDS), 430–437
Judgments), 426–427 Anderson, K. J., 150–151 challenges for, 434–435
Affiliative equilibrium model, 349–350 Andrews, S. L., 388 competitions, 433–434
Affordances perspective, 30–33 Anger, 261–262 detecting emotion categories, 431–433
Age bodily communication of, 51 history of, 431
individual differences in gaze disgust versus, 25 Autonomic nervous system (ANS), 48,
behavior and perception, 352 facial communication of, 50 50–51, 269
information gathering, 346 interpersonal effects of facial Averageness hypothesis, 224–225
individual differences in pupillary expression, 271–272
behavior, 342 paralinguistic cues, 92 Babyfacedness, 242
individual differences in reading posture and, 387 Bachenko, J., 483
facial expressions, 337 serviceable habits principle, 271 Back-channel responses, 149–151,
individual differences in smiling, 145 survival value of, 54 155, 277
individual differences in tears, 340 temperature and, 23 Baig, K., 60
individual differences in touch, 147 Tomkins’s theory of facial Bailenson, J. N., 555
lifetime changes in voice, 294–295 expression, 426 Bakeman, R., 563–564, 574–575
recognizing facial expressions of voice and, 297 Baldaro, B., 176
emotion later in life, 117–118 ANS (autonomic nervous system), 48, Bannerman, R. L., 533
skin condition and, 222 50–51, 269 Bänziger, T., 591, 594
Aglioti, S. M., 546 Antithesis principle, 50–51 BAP (Body Action and Posture) coding
Agreeableness, 167, 170–171, 233–234 ANVIL annotation software, 565 system, 558–559, 565, 573

607
APA Handbook of Nonverbal Communication, edited by D. Matsumoto, H. C. Hwang,
and M. G. Frank
Copyright © 2016 American Psychological Association. All rights reserved.
Index

Bard, K. A., 412 openness to experience, 167, automatic processing, 566–570

Bared-teeth display, 401–402, 408–409, 171–172, 233–234 body movement recognition,
412–418 physiognomy and, 233–234 568–570
Bargh, J. A., 379 Biological preparedness, 104 types of systems, 566–568
Baron, R., 447, 547 Birdwhistell, R. L., 5–6, 18, 27, 428–430, correspondence between behavior
Baron-Cohen, S., 344–345 556, 558 and psychological variables,
Barrette, D., 447 Bixler, N. R., 5 571–573
BART (Brief Affect Recognition Blackledge, C., 567 future research directions, 573–577
Test), 590 Blair, C., 4 automatic detection of touch
Bartels, C., 486 Blascovich, J., 555 behavior, 577
Barton, R. A., 65 Blind individuals, 122 automatic measurement in real-
Basal tears, 338 body movement, 389 life settings, 575–577
Bateson, G., 27, 429 gestures, 324 behavioral patterning, 574–575
Batliner, A., 480 smiling, 104 segmentation of behavior stream,
Baumeister, R. F., 80–81 study of universal facial expressions 573–574
Bavelas, J. B., 4 of emotions, 262–263 history of research into, 551–552
Bayliss, A. P., 351 Blinking human observation, 563–566
Beall, A. C., 555 analysis, 534 coding schemes, 564–565
Beards, 64, 237 measuring, 532 judgment method, 563–564
Beat gestures, 311, 323, 502 Block, J. H., 172 reliability of, 565–566
Beatty, J., 534 Blushing, 22, 62, 152, 155, 435 kinesics, 556–562
Beatty, M. J., 167 Bocharov, A. V., 176–177 levels of description, 557–562
Becker, F. D., 207 Body Action and Posture (BAP) coding posture, action, and gesture,
Behavioral Approach System, 178 system, 558–559, 565, 573 556–557
Behavioral ecology approaches, 27–30, 33 Body Hot Spots (Guthrie), 18 proxemics, 553–556
Fridlund’s behavioral ecology view, 28 Body language, 6, 429 interaction in virtual reality,
social goals versus emotions, 29 Body Language (Fast), 18 555–556
status cues perspective, 28–30 Body movement and expressions, techniques for assessing, 553–555
Being watched, 83, 141, 145, 348 387–395. See also names of specific sampling techniques, 563
Beisner, B. A., 417 body movements and expressions Body orientation, 141, 149–150, 155, 387
Beland, S., 546 Big Five framework of personality Body rhetoric, 5
Bell, C., 259 traits, 167 Body size, as sign of dominance, 64–65
Bennett, D. S., 119 communication of dominance, Body symmetry, 59–60
Bensafi, M., 545 64–65 Body synchrony (posture mirroring),
Benus, S., 460 communication of emotions 149, 388
Berkeley Expressivity Questionnaire, 170 signals, 52–54 Boersma, P., 488
Berliner, D. A., 369 signs, 51 Boguslawski, G., 570
Bernese system, 562 symbols, 54 Bois, J., 447
Bernieri, F. J., 7, 165, 179 communication of fertility, 61–62 Bolinger, D., 473
Berrien, F. K., 342 communication of health, 59–60 Bombari, D., 533
Berry, D. S., 168, 171–172 communication of sexual receptivity/ Bonanno, G. A., 266, 273
Berry, J. W., 80–81 availability, 62–63 Bonar, C. J., 265
Berry, M., 8 cultural norms for, 94 Bond, M. H., 83
Bethell, E. J., 418 developmental arc of, 122–125 Bonifacio, G., 551
Bettadapura, V., 432, 437, 439, 449 emotion and body movements, Book, A., 57
Bettens, F., 370 123–125 Booth, D. A., 542
Bianchi-Berthouze, N., 570, 573, 576–577 head movements, 123 Borkenau, P., 169–172, 174
Biau, E., 517 pointing gestures, 123 Borod, J. C., 440–441
Biehl, M., 260 movements and gait, 389–390 Boundary tone, 479
Big Five framework of personality traits, postures, 387–388 Bowden, R., 569
165–172 by primates, 403 Boyle, J. A., 546
agreeableness, 167, 170–171, 233–234 proxemics, 389–392 Bradley, M. M., 534
conscientiousness, 167, 171, 233–234 touch, 392–394 Brainstorms (Dennett), 19
extraversion, 166–170, 233–234 Body movement coding and analysis, Brandt, A., 164
neuroticism, 167, 170, 233–234 551–578 Braunschweiler, N., 473

608
 Index

Brave, S., 555 Cartwright-Mills, J., 444–445 of facial skin, 222–223

Breathy voice, 292 CAST (Contextual and Affective Communication of Affect Receiving Ability
Brenier, J., 472 Sensitivity Test), 593 Test (CARAT), 447–450, 590
Bressem, J., 557, 559 Catchments, 510 Communication process approaches, 21,
Bricker, P. D., 293 Chameleon effect, 388 26–32
Brief Affect Recognition Test (BART), 590 Chan, J., 231 affordances, 30–31
Brinkman, H., 553 Character viewpoint, 506 behavioral ecology approaches, 27–30
Briton, N. J., 139 Chemosensory cues, 363–364. See also lens model approaches, 30
Broaders, S. C., 326 Olfactics and olfaction social-cognitive perspectives, 31–32
Brody, L. R., 543 Chen, B. F., 483 status cues perspective, 28–30
Brosin, H., 429 Chen, D., 111, 366, 540 “Communication Without Words”
Brothers, L., 344 Chen, F. R., 470 (Mehrabian), 17
Brow movements, 50, 121, 237, Chen, K., 480 Complementary gesture–speech
258–259, 271, 276–277, 336, 426 Chimeras, 226 relationship, 311–313, 317
Brown, C. E., 146, 173 Chimeric faces, 59 Confucius, 19
Brown, M., 467 Chirologia: Or, the naturall language of the Conkie, A., 462, 470
Bruce, G., 467 hand (Bulwer), 551 Conner, B., 173
Brunswik, E., 30, 189, 571 Chomsky, N., 473 Conscientiousness, 167, 171, 233–234
Buck, R., 10, 179, 436, 440–441, Chong, S. C. F., 116 Contact and noncontact (cultural
444–445, 447–448, 590 Chovil, N., 4 dimension), 8, 92, 94
Buhmann, J., 468 Christensen, A., 562 Contact comfort, 107
Bulwer, J., 551–552 Chronemics, 6 Contempt, 23, 261–262
Burdick, C. A., 388 Church, R. B., 510–512 flirting and, 28
Burgoon, J. K., 7, 9, 19 Cicero, 19 survival value of, 54
Burke, K., 5 Civil inattention, 193 Tomkins’s theory of facial
Burrows, A. M., 265 Clark, J., 468 expression, 426
Bush, G. W., 335 Claussen, C., 375–376 Context (cultural dimension), 8, 82–84
Bushdid, C., 365 Clean odors, 377–378 Context analysis, 429
Buss, D. M., 63, 202 Clothing Contextual and Affective Sensitivity Test
Buss, M., 562 as sign of dominance, 65 (CAST), 593
as sign of health, 60 Conversation analysis, 6–7
Calder, A., 351 as sign of sexual receptivity/ Conway, R., 576
California Adult Q-Sort, 179 availability, 62–63 Cook, J., 374
Call, J., 106 Clynes, M., 577 Cook, M., 352
Camilleri, J. A., 57 Coarticulation, 290 Cook, S. W., 325
Campbell, A., 169–170 Codispoti, M., 176 Cooper, M., 546
Campos, J. J., 272 Coetzee, V., 223, 226–227, 241 Cooper, R. M., 176
Camras, L. A., 119 Coffield, C. N., 368 Cornwell, R. E., 232
Candid Camera (television program), 18 Cognitive approach to gesture analysis, Corson, R. L., 173
Canon, L. K., 200 507, 510–513 Cosmides, L., 49
Canon of delivery, 4 coding gestures, 510 Cospeech gestures, 309–311, 501–507.
Caponigro, J. M., 443 effects on learning, 511–512 See also Gesture
Cappella, J. N., 7 effects on problem solving, 510–511 beat gestures, 311
Captured attention, 113 effects on speech processing, 512–513 cognitive approach to measuring, 507
CARAT (Communication of Affect Cohen, A., 467 deictic gestures, 310–311, 314
Receiving Ability Test), 447–450, Cohen, J. A., 566 describing form and meaning,
590 Cohen, N. J., 533 504–506
Carbaugh, D., 8 Cole, J., 5, 468, 470 hand shape, 505
Carmen, B., 140, 145–146, 150 Coleman, J., 468 location, 504–505
Carney, C. C., 270 Collett, P., 90 movement, 505–506
Carney, D. R., 145, 171–172 Collier, R., 467 orientation of hand and fingers, 505
Carotenoids, 223 Collins, M., 235, 484 use of one or two hands, 505
Carré, J. M., 64 Color viewpoint, 506
Carroll, J. M., 116 effect on nonverbal communication, differentiating from other
Carruth, D. W., 567 203–204 behaviors, 502

609
Index

from nonlanguage movement, for emotions and emotional Dawkins, R., 19

502–503 expressions, 86–88 Dean, J., 348–350
from other language associated for gaze, 92 De Boulogne, D., 45
gestures, 503 for gestures, 89–92 Deception. See Lying and deception
gesture words, 503–504 for interpersonal space and touch, Declination, 479–480
iconic gestures, 309–310, 314 93–94 DeFrank, R. S., 143
in Kendon’s continuum, 501 for perception of other’s emotional De Groot, J. H. B., 367
linguistic approach to analyzing, expressions, 88–89 Deictic (indexical) gestures, 310–311,
506–507 for touch, 94 314, 323, 502
metaphoric gestures, 310–311, 315 for vocal behavior, 92–93 Delaunay-El Allam, M., 368
Costa, M., 555 for whole body movements, 94 Delplanque, S., 545
Costa, P. T., 165, 168 olfactics, 371–373, 547–548 Dennett, D. C., 19
Costanza, R., 393 origins of, 77–80 Density, 207–208
Costello, K., 57 ecology, 78 DePaulo, B. M., 348
Coulson, M., 387 evolved human mind and brain, Derlega, V. J., 393
Coupland, J., 7 79–80 Developmental niche, 105–106
Coupland, N., 7 group life, 78–79 emotion recognition from faces,
Cox, S., 476, 484 resources, 78 116–117
Crane, E. A., 561 posture, 388 touch in relationships, 108–109
Creaky voice (glottal fry), 292 proxemics, 391–392 Devine, P. G., 144
Crosby, L., 599 pupillary responses and perception, 342 De Waal, F. B. M., 22, 264, 417
Crowding, 207–208 sexual dimorphism, 229, 232 Dewsbury, D. A., 259
Crowley, S., 5 signaling emotions that one may not Diagnostic Analysis of Nonverbal
Crying, 52, 105 feel, 53–54 Accuracy (DANVA), 178, 592
gender and, 152, 155 touch, 108–109, 393–394 Diagnostic Analysis of Nonverbal
tears, 338–340 unfocused interactions in public Accuracy—2 Posture Test
Cultural display rules, 86–88, 273 settings, 194 (DANVA2–POS), 592
Culture, 77–95 universality of human emotions, 23 Dick, A. S., 515, 518
context and, 82–84 voice Differentiation, 105
interactants and social roles, 84 geographical factors, 294 in emotional expression
settings, 83–84 lifetime changes in, 295 production, 119
definition of, 80–81 signs of emotion, 52 in responses to emotional facial
developmental niche, 105–106 social roles and, 295 expressions, 115–116
elements of, 81–82 speaker identity and, 294 Dilley, L., 467
facial adiposity, 227 whole body movements, 389 Dimberg, U., 428
facial attractiveness, 222 Cunningham, M. R., 175, 177, 179 Ding, J. Y. C., 174
facial expressions, 120, 263 Cupchik, G. C., 541 Diplophonia (vocal fry), 460
around eyes, 337 Current Anthropology, 19 Directional asymmetry, 225
social signals from, 122 Disambiguating gesture–speech
universality of, 260–261, Dael, N., 389, 557 relationship, 311–313
273–274 Daelmans, W., 474 Disgust, 261–262
gaze and eye contact, 147, 346–347, Dai, L.-R., 483 anger versus, 25
352 Dancing, 60, 124, 389, 556, 561 facial communication of, 50
gesture–speech integration during DANVA (Diagnostic Analysis of olfactic signs, 367
language production, 315 Nonverbal Accuracy), 178, 592 paralinguistic cues, 92
head movements, 123 DANVA2–POS (Diagnostic Analysis of serviceable habits principle, 271
interpersonal space and environment, Nonverbal Accuracy—2 Posture survival value of, 25, 54–55
202 Test), 592 Tomkins’s theory of facial
inventing versus restricting Darwin, C. R., 9, 17, 22–23, 46, 50–52, 56, expression, 426
behavior, 83 257, 259–260, 265, 271, 276–277, Display Rule Assessment Inventory, 87
judging sexual orientation from 296, 340, 401, 426, 428, 449, 551 Display rules, 10, 86–88
faces, 242 Davidson, R. J., 349 Dittmann, A. T., 18
judging social group membership Davila Ross, M., 402 Dittrich, W. H., 387–388
from faces, 243–244 Davis, M. H., 598 Dobson, S. D., 409
norms, 82, 84–94 Davitz, J. R., 24 Doherty, R. W., 598

610
 Index

Dominance, 63 studying facial expression, 428 display rule norms, 86–88

body openness and, 149 Eliassen, J., 546 perception of other’s emotional
communicating through body Ellgring, H., 441, 573 expressions, 88–89
expressions, 64–65, 124 Ellis, S. R., 533 questions and controversies
communicating through face, 63–64, Ellsworth, P. C., 347 regarding, 24–26
235–237 Ellyson, S. L., 146, 173 survival value, 54–56
gaze and, 348–349 Emblems (symbolic gestures; EmotiW challenge, 433–434
personality traits, 172–173 emblematic gestures), 276 Empathy, 598–599
Dougherty, L. M., 427–428 cultural norms for, 90–92 Energy of speech. See Intensity of speech
Dovidio, J. F., 146, 173 defined, 89 Engquist, G., 447
Duchenne smiles, 261, 266, 336, 402 as gestures, 308–309 Environmental features, 195–204
Duckett, P., 374 in Kendon’s continuum, 501, 503 architectural elements, 201–202
Duffy, J. R., 440 referencing shared meanings, 314, 323 color, 203–204
Duffy, R. J., 440–441 Embodied cognition, 89 lighting, 196–197
Duke, M., 11, 178 Emde, J. J., 272 mobile technology, 204
Dunbar, R. I. M., 108 EMFACS (Emotion Facial Action Coding objects and personal belongings,
Duncan, S., Jr., 18 System), 428 202–203
Duration of speech, 462–463 EMG. See Electromyography odor, 199–200
Emmorey, K., 293 sounds, 200–201
Early attachment Emotional and behavioral disorders temperature, 197–199
biological preparedness, 104 (EBDs), 441–443 Environmental Psychology and Nonverbal
olfaction and, 110–111 Emotional contagion, 597–598 Behavior, 189
touch and, 107–108 Emotional Contagion Scale, 598 The Environment and Social Behavior:
contact comfort, 107 Emotional gestures, 500 Privacy, Personal Space, Territory,
kangaroo care, 107–108 Emotional sensitivity, 589 Crowding (Altman), 189
skin-to-skin contact, 107 Emotional tears, 338 EPI (Eysenck Personality Inventory),
Earnst, K. S., 443 Emotion communication platform, 21 176–177
Easton, C., 444, 446 Emotion Facial Action Coding System Epstein, N., 598
EBDs (emotional and behavioral (EMFACS), 428 Equilibrium theory, 27, 348–349, 392
disorders), 441–443 Emotion Recognition Index (ERI), 594 ERI (Emotion Recognition Index), 594
Ebihara, S., 547 Emotions, 20–26, 48 ERPs. See Event-related potentials
Ecological contamination hypothesis, 193 basic categories of, 9, 23–25, 49 Error management theory, 144
Ecological psychology, 189. See also communicating through body Esch, M., 517
Physical environment expressions, 123–125 Escrig, M. A., 534
Ecological systems model, 208–210 posture, 387–388 Essa, I. A., 432
Ecological theory of social perception, signals, 52–54 Essays on Physiognomy (Lavater), 233
30–31 signs, 51 Estrogen, 61
EEG. See Electroencephalographic symbols, 54 Etiquette, 82
measurement of gesture communicating through facial Evanini, K., 468
Efron, D., 17–18, 90, 557 expressions Event-related potentials (ERPs)
Eibl-Eibesfeldt, I., 19 gaze, 350 measurement of gesture, 516–518
Eisner, F., 293 producing emotional expressions, learning, 517
Ekman, P., 4, 6, 9, 18–19, 23–24, 49, 118–121 speech processing, 516–517
112, 169, 176, 260, 266, 268, 270, recognition of emotional recognizing and interpreting facial
273–274, 276, 297, 308, 364, expressions, 115–118 expressions of emotion,
426–428, 430–431, 435, 437, 501, signals, 52–54 115–116, 118
552, 554, 557–558, 590, 594, 600 signs, 49–51 Evolution and cultural development,
ELAN annotation software, 175, symbols, 54 79–80
507–508, 520, 565 communicating through voice Evolutionary approach to nonverbal
Electroencephalographic (EEG) signals, 52–54 research, 9
measurement of gesture, 516–517 signs, 51–52 Evolutionary influence on nonverbal
learning, 517 symbols, 54 communication, 45–66
speech processing, 516–517 cultural development and, 80 reproductive fitness, 46, 57–66
Electromyography (EMG) cultural norms for, 86–88 dominance, 63–65
studying body movement, 576 culturally moderated emotions, 86 fertility, 61–62

611
Index
health, 58–60
role of nonverbal communication,
46–47
sexual receptivity/availability, 62–63
survival to age of reproduction, 46,
48–57, 66
emotion, 48–56
identity, 56–57
role of nonverbal communication,
46–47
voluntariness of signs, signals, and
symbols, 47–48
Exline, R. V., 18, 350
Expectations for behavior, 83
The Expression of the Emotions in Man and
Animals (Darwin), 9, 17, 22, 259,
426
Expressiveness and reservedness (cul-
tural dimension), 85–87, 90, 93
External variable approach, 27
Extraversion, 166–170
nonverbal decoding, 177–178
voice and personality traits, 295
Eyben, F., 488
Eye behavior, 335–352
attraction to eye region, 335–336
facial actions around eyes, 336–338
individual differences in reading,
337–338
social dynamics of, 336–337
gaze behavior, 343–352
as channel for information
gathering, 343–345
emotional expression and, 350–351
individual differences in,
345–347, 351–352
as signal of approach and
avoidance, 349–350
as signal of self-relevance, 347–348
as signal of social affiliation, 348
as signal of social dominance,
348–349
social dynamics of, 343
pupillary behavior, 340–342
individual differences in, 341–342
social dynamics of, 341–342
tears, 338–340
individual differences in, 339–340
social dynamics of, 339
Eye behavior coding and analysis, 525–536
analysis, 532–535
blinking, 534
movement, 532–534
pupil diameter, 534
anatomy, 525–526
612
eye-tracking, 527–531
calibration, 530
configurations, 528–529
considerations, 529–531
techniques, 527–528
future research directions, 535–536
measurement, 531–532
blinking, 532
movement, 531–532
pupil diameter, 532
movements, 526–527
Eyebrow movements, 50, 121, 237,
258–259, 271, 276–277, 336, 426
Eye contact
culture and, 147, 346–347, 352
gender and, 146–147, 155
as sign of sexual receptivity/­
availability, 61
Eye Direction Detector mechanism,
344–345
Eye-tracking systems
calibration, 530
configurations, 528–529
chin rest systems, 528
desk-mounted systems, 528–529
eyeglass-mounted systems, 528
head-mounted systems, 528
performance issues, 529–531
ambient illumination, 530–531
contact lenses, 530
drooping eyelids, 530
eyeglasses, 530
long eyelashes, 530
mascara, 530
sampling rate, 529–530
system latency, 529
validity and reliability, 529
Eysenck, H. J., 176
Eysenck, S. B. G., 176
Eysenck Personality Inventory (EPI),
176–177
Eysenck Personality Questionnaire, 169,
179
Face and Body PONS, 178
The Face of Emotion (Izard), 19
Face-to-face dialogue, 27
FACET software, 434
Fach, M., 477
Facial Action Coding System (FACS), 6,
23, 25, 113, 258, 265, 405–407,
411–412, 427–428, 431, 435,
440–441, 552, 559, 573
Facial Action Scoring Technique (FAST),
427–428
Facial appearance, 221–244
attractiveness, 221–232
adiposity, 226–228
averageness, 224–225
sexual dimorphism, 228–232
skin condition, 222–224
symmetry, 225–226
dynamic and malleable facial cues,
239–241
2D versus 3D, 240–241
static versus dynamic stimuli,
239–240
new directions in contemporary
physiognomy, 241–244
detecting success from faces,
241–242
judging sexual orientation from
faces, 242–243
judging social group membership
from faces, 243–244
personality attributions from,
232–239
ability to read faces, 237–238
causal relationships, 235–237
modern physiognomy, 233–234
origins of physiognomy, 232–233
overgeneralization, 238–239
Facial coloration, 59
Facial expression, 257–278
around eyes, 336–338
individual differences in reading,
337–338
social dynamics of, 336–337
of basic emotion, 259–274
evolutionary origins of study,
259–260
influence of culture, 273–274
interpersonal effects, 271–272
intrapersonal effects, 271
links with appraisal processes, 268
links with physiological
responses, 268–269
links with subjective experience,
266–268
links with subsequent behaviors,
269–270
mechanics, 270–271
research documenting universality,
260–265
sociocultural effects, 272–273
Big Five framework of personality
traits, 167
biological preparedness, 104
broad individual differences in,
274–275

cognitive and physical states and, 276
communication of affect, 114–121
changing attunements to, 116
gender and recognition, 117
later-life recognition, 117–118
niche elements in recognition,
116–117
producing emotional expressions,
118–121
producing facial expressions of
emotion, 118–119
recognition of emotional
expressions, 115–118
recognizing and interpreting, 115
signals, 52–54
signs, 49–51
symbols, 54
communication of dominance, 63–64
communication of fertility, 61
communication of health, 58–59
communication of sexual receptivity/
availability, 62
developmental arc of, 112–122
communicating emotions, 114–121
sensation and facial expression,
112–114
social signals from facial
expressions, 121–122
differentiation, 105
facial anatomy, 258–259
future research directions, 277–278
mental and physical health and,
275–276
of other emotions and affective states,
274
in primates, 401–405, 411–418
context dependence/meaning,
417–418
identification of major categories,
411–415
multimodal components, 415–417
as reliable leakage clues to deception,
276
role in conversation, 276–277
Facial expression coding and analysis,
425–453
approaches to, 426–430
Ekman, 427–428
facial EMG techniques, 428
Izard, 426–427
structural approaches, 428–430
Tomkins, 426
automated emotion detection systems
(AEDS), 430–435
challenges for, 434–435
competitions, 433–434
detecting emotion categories,
431–433
history of, 431
future research directions, 452–453
measuring dynamic and
spontaneous facial emotion
communication, 437–452
CARAT as standard database, 449
expressiveness versus communi-
cation accuracy, 449–452
in patients with brain damage,
439–441
in psychopathology, 441–446
segmentation technique, 446–449
slide-viewing technique, 437–439
unresolved issues in, 435–437
bottom-up facial expression
detection, 435–436
emotion detection versus
communication, 436
facial expression databases,
436–437
spontaneous versus intentional
displays of emotion, 437
Facial Expression Recognition
and Analysis (FERA) challenge,
433, 437
Facial feedback hypothesis, 271
Facial ratio, 63–64
Facial recognition technology, 431
Facial symmetry, 58–60
FACS. See Facial Action Coding System
FACS Investigator’s Guide (Ekman and
Friesen), 428
Factors of influence
culture, 77–95
context and, 82–84
definition of, 80–81
elements of, 81–82
inventing versus restricting
behavior, 83
norms, 82, 84–94
origins of, 77–80
developmental arc of nonverbal
communication, 103–126
biological preparedness, 104
developmental niche, 105–106
differentiation, 105
involuntary to voluntary
control, 104
olfaction, 110–112
touch, 107–110
vision and body expressions,
122–125
Index
vision and facial expressions,
112–122
evolution, 45–66
reproductive fitness, 46–47, 57–66
survival to age of reproduction,
46–57, 66
voluntariness of signs, signals,
and symbols, 47–48
gender, 139–155
blushing, 152, 155
body orientation, 149–150, 155
context, 141–142
crying, 152, 155
decoding accuracy, 143–145, 155
deconstructing, 139–141
encoding accuracy, 142–143, 155
flirtation, 152–153
gait, 152, 155
gaze and eye contact, 146–147, 155
gesture, 149–150, 155
interpersonal distance, 149–150,
155
posture, 149–150, 155
smiling, 145–146, 155
touch, 147–149, 155
voice, 150–151, 155
personality traits, 163–180
Big Five framework, 166–172
future research directions, 179–180
historical controversies and meth-
odological issues, 163–165
intelligence, 174–175
nonverbal decoding, 175–179
nonverbal encoding, 179
organizational frameworks,
165–166
sexual orientation, 173–174
status and dominance, 172–173
Fakotakis, N., 483
Fan, M., 483
Farris, C., 144
Fasel, B., 433
FAST (Facial Action Scoring Technique),
427–428
Fast, J., 18
Fear, 261–262
adaptive, 47
bodily communication of, 51
communicating through touch, 110
facial communication of, 50
interpersonal effects of facial
expression, 271
olfactic signs, 365–367
serviceable habits principle, 271
survival value of, 49, 55
613
Index

Tomkins’s theory of facial Fundamental frequency of speech. See Gemmell, A. J., 387–388
expression, 426 Pitch of speech Gender, 139–155
voice and emotion, 297 Funder, D. C., 168, 171–172 blushing, 152, 155
Fehr, B. J., 350 body orientation, 149–150, 155
Feinberg, D. R., 241 Gait, 390 context, 141–142
Feldman, R., 108 analysis methods, 562 crying, 152, 155
FERA (Facial Expression Recognition cultural norms for, 94 decoding accuracy, 143–145, 155
and Analysis) challenge, 433, 437 gender and, 152, 155 deconstructing, 139–141
Ferber, S. G., 108 health and, 390 emotional expression production,
Ferstl, R., 367 identifying potential targets of 120–121
Fertility aggression by, 57, 390 emotion recognition from facial
communicating through body, 61–62 as sign of health, 60 expressions, 117
communicating through face, 61 Galati, D., 263 encoding accuracy, 142–143, 155
Field theory approach, 189 GALE system, 518 facial adiposity, 226–228
Fischer, A. H., 339 Galinsky, A. D., 377 flirtation, 152–153
Fischer, J. L., 143 Gallup, G. G., Jr., 63 gait, 152, 155
Fishy smell, 364–365, 376–377 Galton, F., 224 gaze and eye contact, 146–147, 155
Fitzpatrick, E., 483 Game theory, 27–28 gaze as information gathering, 346
Five-Factor Model of personality, 165 Gangestad, S. W., 231–232 gaze behavior and perception, 351–352
Fixations, 531, 535 Gao, Y., 169–170, 577 gesture, 149–150, 155
Flack, J. C., 417 Gaulin, S. J. C., 235 interpersonal distance, 149–150, 155
Flirtation Gaze avoidance, 114, 194 judging sexual orientation from faces,
gender and, 152–153 Gaze behavior, 343–352 242–243
gender and decoding accuracy, as channel for information gathering, olfaction, 200
144–145 343–345 personal objects, 203
Floral scents, 374 communication of dominance, 173 posture, 149–150, 155
Floyd, K., 9 cultural norms for, 92 proxemics, 392
Fluctuating asymmetry, 59–60, 225–226 emotional expression and, 350–351 pupillary responses and
fMRI. See Functional magnetic facial expressions responding to, perception, 342
resonance imaging 113–114 reading facial actions around eyes, 337
Focused interactions, 190 feelings of crowding and, 208 sexual dimorphism, 228–232
Four-Factor Test of Social gender and, 146–147, 155 femininity in women’s faces,
Intelligence, 178 individual differences in behavior and 228–229
Fox, E., 351 perception, 351–352 masculinity in men’s faces, 230–232
Fox, R., 123 age, 352 smiling, 145–146, 155
Fraccaro, P. J., 241 context and culture, 352 static versus dynamic stimuli and
Freeman, J. B., 174 gender, 141, 351–352 perception of attractiveness,
Freyberg, R., 47, 371, 545 traits, 351 240
Friday, A. E., 63 individual differences in gaze as tears, 339–340
Fridlund, A. J., 10, 22, 28 information gathering, 345–347 touch, 147–149, 155, 393–394
Friederici, A. D., 515–516 age, 346 voice, 150–151, 155
Friedman, H. S., 179 context and culture, 346–347 anatomical differences, 291
Friesen, W. V., 6, 19, 86–88, 169, 176, gender, 346 social roles, 295
260, 266, 273–274, 276, 308, 428, traits, 345–346 George-Falvy, J., 388
430–431, 435, 437, 501, 554, in public settings, 194 Gesture, 307–327. See also names of
557–558, 590, 594 sexual orientation and, 174 specific gestures
Fromm-Reichmann, F., 429 as signal of approach and avoidance, categories of, 308–309
Fujisaki, H., 467, 477, 482 349–350 compared to action, 321–322
Fulcher, J. S., 262 as signal of self-relevance, 347–348 compared to other body
Functional magnetic resonance imaging as signal of social affiliation, 348 movements, 557
(fMRI) as signal of social dominance, 348–349 compared to voice, 321–322
measurement of gesture, 513–518 social dynamics of, 343 cospeech gestures, 309–311
learning, 515–516 types of movements, 526 cultural norms for, 89–92
speech processing, 514–515 Gaze cueing, 343–344, 351 emblems, 90–92
measurement of olfactics, 546 Geertz, C., 80–81 speech illustrators, 89–90

614

as dominant linguistic system, 318–320
cospeech gestures versus silent
gesture, 319–320
homesign and emerging sign
languages, 318–319
future research directions, 326–327
gender and, 149–150, 155
human versus animal use of, 322–323
hybrid, 90
information communicated through,
313–315
abstract thoughts, 315
environment, 314
objects, actions, and events, 314
shared meanings, 314
integration of speech and, 315–318
during language comprehension,
315–317
during language development,
317–318
during language production, 315
involvement of hands, 308
less digital than speech, 320–321
made by primates, 403
noncommunicative uses for, 323–325
conceptualization, 324
lexical access, 324
new knowledge, 325
thinking, 324
working memory, 325
relationship to speech, 311–313
complementary, 311–313
disambiguating, 311–313
gesture-speech mismatches, 313
supplementary, 311–312
research applications, 325–326
sign language versus, 307–308
Gesture coding and analysis, 499–520
applications for, 508
cognitive research applications for,
510–513
coding gesture in, 510
learning, 511–512
problem solving, 510–511
speech processing, 512–513
cospeech gestures, 501–507
cognitive approach, 507
describing form and meaning,
504–506
differentiating from other
behaviors, 502–503
gesture words, 503–504
linguistic approach, 506–507
defining gestures, 499
future research directions, 517–519
Index
language-associated gestures, 500–501 Guerrero, L. K., 9
linguistic research applications for, Guilt. See Shame, guilt, and humiliation
509–510 Gunnery, S. D., 145–146
coding gesture in, 509–510 Gur, R. C., 446
story narration, 510 Guthrie, R. D., 18
neurological applications for, 513–517
ERP measurement of gesture, Haan, N., 270
516–517 Hage, J., 83
fMRI measurement of gesture, Haggard, E. A., 276
513–516 Hagoort, P., 514–515, 517
tools for, 507–508 Halberstadt, A. G., 590
interrater reliability, 508 Hale, J. L., 7
video recording, 507–508 Hall, E. T., 6, 8, 18, 85, 93, 148, 189,
Gesture-speech mismatches, 313 376, 390–391, 553
Gesture utterances, 511 Hall, J. A., 20, 139, 145–147, 149, 393,
Ghazanfar, A. A., 416 543, 590–591, 593
Gibson, J. J., 189 Hallahan, M., 173
Giese, M. A., 562 Halle, M., 473
Gifford, R., 10–12, 30, 168, 171 Hallett, K. C., 174
Gilbert, A., 569 Hämäläinen, H., 344
Giles, H., 7 Hamilton, W. D., 230
Glottal fry (creaky voice), 292 Hamm, J., 446
Godfrey, A., 576 Hämmerli, A., 371
Goffman, E., 5–8, 18, 190, 194 HamNoSys standard coding system,
Goldberg, J. H., 532 559–560
Goldberg, L. R., 165 Hands-digit ratio, 64–65
Goldin-Meadow, S., 325–326, 510–512, Hansen, J. S., 168, 171–172
515 Happiness and joy, 261–262
Goldman, C., 442, 444–445 facial communication of, 50
Goodall, J., 18 flirting and, 28
Goodman, N., 447 olfactic signs, 365–368
Gosling, S. D., 171 posture, 387
Gottman, J. M., 6, 563–564, 574–575 survival value of, 55
Gould, S. J., 19 Tomkins’s theory of facial
Grabe, E., 468 expression, 426
Graham, C. A., 545 voice and emotion, 297
Graham, J., 189 Haptics, 6, 392–394. See also Touch
Grammer, K., 144, 153 Hare, B., 265
Gratitude, 24, 110, 393 Hargreaves, J., 389
Gravano, A., 460 Harker, L., 274
Greenberg, S., 468, 475 Harkness, S., 105
Gregory, M., 474 Harlow, H. F., 109–110
Grief, 266, 275, 296 Harris, C. R., 266
Gross, J. J., 170–171, 266 Harrison, S., 393
Gross, M. M., 561 Hasegawa-Johnson, M., 470, 475
Grosz, B. J., 473 Hasson, O., 339
Group affiliation recognition, 56–57 Haviland-Jones, J. M., 47, 122, 366,
Group behavior, 56 368, 540
Group life Hawhee, D., 5
cultural development and, 78–80 Haxby, J. V., 344–345
gaze behavior as signal of social Hayduk, L. A., 554–555
affiliation, 348 Health, 58–60
judging social group membership communicating through body, 59–60
from faces, 242–243 communicating through face, 58–59
Guerin, B., 83 Heintz, M., 414–415
615
Index

Heisel, A. D., 167 Hymes, D., 8 Interpersonal space. See Proxemics

Heldner, M., 482 Hypothetical thinking, 80 Interpersonal synchrony, 430
Helminen, T. M., 348 Interruptions
Heltman, K., 146, 173 Ickes, W., 149, 599 agreeableness and, 171
Hendriks, M., 377 Iconic gestures, 309–310, 314, 323, 501, gender and, 141, 150–151
Henley, N. M., 11, 147, 393 503, 511 intrusive, 150
Henning, R. A., 436 Identity, 56–57 In the Shadow of Man (Goodall), 18
Henry VII, 233, 238 group behavior, 56 Intimacy equilibrium hypothesis, 392
Herbert, G., 335 individual recognition through Intonational phrasing, 478–488
Herrmann, E., 406 olfaction, 110–111 automatic classification of phrase
Hertenstein, M. J., 110, 554 olfactic signs, 369–372 boundary types, 485–488
Herz, R. S., 375, 546 person recognition, 56–57 automatic detection of phrase
Hess, E. H., 340–341 speaker identity and voice, 294 boundaries, 480–485
Hess, U., 144 Iizuka, Y., 169 in non-English languages, 482–483
Hewes, G. W., 19 ILAB software, 536 using lexical information, 483–485
Hietanen, J. K., 344, 348 Illingworth, J., 569 Intonational prominence. See Pitch accent
High-context cultures, 8 Illustrators, 308–309, 501, 511 Intonational Variation in English (IViE)
Hill, R. A., 65 cultural norms for, 89–90 system, 467
Hill, S. E., 63 defined, 89 Intrusive interruptions, 150
Hinde, R. A., 19 functional descriptions, 557–558 Involuntary control of behavior, 104
Hirose, K., 477 Imhof, M., 173 IPT (Interpersonal Perception Task),
Hirschberg, J., 460, 466, 471, 473–474, Immediacy concept, 349–350, 553 594–595
476, 481, 484–487 Immediacy cues, 8, 348 IRI (Interpersonal Reactivity Index), 598
Hirst, D., 467 Immunocompetence handicap Isaacs, K. S., 276
Hjortsjö, C.-H., 427–428 hypothesis, 58, 223, 230 Ishi, C. T., 487
Hockett, C., 429 Indexical (deictic) gestures, 310–311, Ishii, K., 570
Hoffman, E. A., 344–345 314, 323, 502 Ishizuka, Y., 374
Hogan, R., 598 Individualism and collectivism (cultural Isomura, T., 577
Hogan Empathy Scale, 598 dimension), 8–9, 87–88, 274, 346 IViE (Intonational Variation in English)
Holland, R. W., 377 Inherited nature of nonverbal cues, 10 system, 467
Holle, H., 517 Institute for Perception Research, 467 Iwano, K., 477
Holmes, A., 418 Intensity (energy) of speech, 460–462 Izard, C. E., 19, 23, 114, 274, 426–428
Holmes, M. E., 534 reset and stylization, 461
Holmes, R., 110 shape modeling of contours, JACBART (Japanese and Caucasian Brief
Holmqvist, K., 531 461–462 Affect Recognition Test), 592–593
Homesign, 318–320 Intentionality, 47, 79, 403–404, 437 Jacob, S., 546
Homogeneity of behavior, 193 Intentionality Detector mechanism, 344 Jahoda, G., 80–81
Hopkins, M., 516 Interaction adaptation theory, 27 James, K. H., 518
Horgan, T. G., 20 Interaction management, 7 James, T. W., 518
Horn, W., 174–175 Interest, 23, 49, 114, 272, 508 Janssen, E., 545
Hoste, V., 474 coding facial expressions, 426–428 Japanese and Caucasian Brief Affect
Howes, D., 373 pupil diameter and, 341 Recognition Test (JACBART),
“How Well Do You Read Body Language?” sexual and romantic, 62, 144–145, 592–593
(Archer and Akert), 17 152–153 John, H., 144
Hu, G.-P., 483 International Personality Items Pool, 238 John, O. P., 168, 170–171
Huber, E., 18 International Transcription System for Johnson, K. L., 174
Hugdahl, K., 349 Intonation, 467 Johnson, R., 576–577
Hugenberg, K., 32, 122 Interpersonal Adjective Scales, 168 Jones, B., 19
Humiliation. See Shame, guilt, and Interpersonal Competence Jones, S. E., 553
humiliation Questionnaire, 598–599 Jones, T. S., 553
Huntington, G. H., 342 Interpersonal mimicry, 165 Jones-Gotman, M., 546
Hwang, H. C., 48, 125, 270 Interpersonal Perception Task (IPT), Josephson, S., 534
Hwang, H. S., 89, 93 594–595 Joshi, J., 570
Hybrid gestures, 90 Interpersonal Reactivity Index (IRI), 598 Jost, J. T., 171
Hyde, J. S., 144 Interpersonal sensitivity, 589 Journal of Communication, 18

616
 Index

Journal of Environmental Psychology, 190 Knöfler, T., 173 Lesaffre, M., 389
Journal of Nonverbal Behavior, 190 Knyazev, G. G., 176–178 Lesbian, gay, bisexual, transgender, and
Journal of Personality and Social Kochanski, G., 468, 475 questioning (LGBTQ)
Psychology, 18 Koehler, K., 467 decoding accuracy, 144
Joy. See Happiness and joy Koehn, P., 484 flirting, 153
Koff, E., 440 gait, 152
Kaasinen, S. M., 348 Kohler, C. G., 446 gaze and eye contact, 146
Kaegi, M., 371 Kokkinakis, G., 483 gestures, 150
Kahlbaugh, P. E., 122 Kole, C., 377 interpersonal space, 150
Kahneman, D., 534 Konner, M. J., 19 judging sexual orientation from
Kahou, S. E., 433–434 Koopmann, B., 266 faces, 242–243
Kanade, T., 431 Koralewski, M. A., 388 personality traits, 173–174
Kanazawa, S., 412, 415 Koslow, D., 555 pupillary responses, 341–342
Kanba, S., 374 Kotval, X. P., 532 same sex touch, 147–149
Kangaroo care, 107–108 Kouroupetroglou, G., 477 smiling, 146
Kant, I., 339 Kravitz, C., 516 stereotypes, 140
Karg, M., 562 Kreiman, J., 293 Levenson, R. W., 266, 268
Karlson, P., 368 Kring, A. M., 443 Levine, R. V., 8
Kashima, E. S., 83 Kroeber, A. L., 80–81 Levitan, R., 460
Katdare, A., 540 Kruger, A. C., 79 Levow, G.-A., 473
Keating, C. F., 146, 173 Kühnlenz, K., 562 Lewin, K., 189
Keller, A., 365 Kumar, V., 462 Lewis, M., 105
Kelly, S. D., 316, 516–517 Kupperbusch, C., 9, 266 Lewis, R. J., 393
Keltner, D., 110, 144, 266, 270, 273–274 Kymalainen, J. A., 446 LGBTQ. See Lesbian, gay, bisexual,
Kendon, A., 18, 500, 506, 510, 519 Kyriazis, N., 568 transgender, and questioning
Kendon’s continuum, 500–501 Li, C., 471
Khanna, P., 577 Laban, R., 556, 558, 561 Li, J.-F., 483
Kiecolt-Glaser, J. K., 375, 545 Labanotation, 556, 561 Liebal, K., 106
Kim, S.-S., 470 Ladd, D. R., 470 Lieberman, M. D., 177
Kim, Y.-J., 483 La France, B. H., 167–168 Liebert, R. M., 270
Kinemes, 429 LaFrance, M., 140, 145–146, 149–150 Liebler, A., 169–172, 174
Kinemorphs, 429 Lakin, J. L., 10 Lie to Me (television program), 24
Kinesics, 5–6, 27, 429, 556–562 Lammers, V. M., 203 Lighting, effect of on nonverbal
body actions, 557 Lang, P. J., 534 communication, 196–197
body posture, 556–557 Langlois, J. H., 224 Liljenquist, K., 377
defined, 556 Language-associated gestures, 500–501 Lilley, S., 375
gestures, 557 Laptev, I., 569 Lindeberg, T., 569
levels of description, 557–562 Larrance, D. T., 143, 596 Linguistic approach to gesture analysis,
anatomically-based form Larynx, 289–291, 294 506–507, 509–510
descriptions, 558–561 Laughter, 24–25, 32, 52, 264–266, coding gestures, 509–510
functional descriptions, 557–558 296–297, 402–403, 436 effects on story narration, 510
spatio-temporal approaches, Lausberg, H., 557 Linton, R., 80
561–562 Lavater, J. C., 233 Liò, P., 63
Kinesics and Context: Essays on Body Lavender, 374–375 Lippa, R., 168
Motion Communication Law Smith, M. J., 229 Lipsmacking, 410
(Birdwhistell), 5 Leakage, 176, 259–260, 276 Little, A. C., 233
Kinnunen, L. H., 546 Leaper, C., 150–151 Liu, J., 471
Kin recognition, 57 Leap Motion, 568 Llewellyn, L. G., 18
Kirk-Smith, M. D., 542 Lee, S. W., 376 Lobmaier, J. S., 533
Kleck, R. E., 31, 144, 350 Leistungsprüfsystem, 174–175 Loewenfeld, I. E., 341
Kleinsmith, A., 573 Leman, M., 389 Logothetis, N. K., 416
Klinnert, M. D., 272 Lens model approaches, 30, 32, Loomis, J. M., 555
Kluckhohn, C., 80–81 571–572 Lorch, M. P., 440
Knapp, M. L., 3–4, 20 Leppänen, J. M., 115 Lordosis, 62
Knight, M., 346 Lerner, J. S., 268–269 Lorenz, K., 19

617
Index

Lucas, N., 540 Maximally Discriminative Facial kinesics, 556–562

Luettin, J., 433 Movement Coding System proxemics, 553–556
Lundström, J. N., 546 (MAX), 426–427, 435 sampling techniques, 563
Luscher, M., 368 Mayhew, E. M., 368 eye behavior coding and analysis,
Lying and deception, 24 McAuley, P. C., 83 525–536
difficulty of fabricating emotions, McCarter, L., 266 analysis, 532–535
52–53 McCarter, R., 260, 426 anatomy, 525–526
diplophonia, 460 McClintock, M. K., 369, 546 blinking, 532, 534
gaze and, 92, 348 McCormick, C. M., 64 eye-tracking, 527–531
measuring sensitivity to deception, McCowan, B., 417 future research directions,
599–600 McCrae, R. R., 165, 168 535–536
pupillary behavior, 342 McCullough, M., 110 measuring, 531–532
McDaniel, C., 123 movement, 526–527, 531–534
Macedonia, M., 515–516 McDevitt, T., 517 pupil diameter, 532, 534
Macfarlane, A., 368 McGinley, J. A., 567 facial expression coding and analysis,
MacLarnon, A., 418 McGrew, W. C., 19 425–453
Maddux, W. W., 347 McGuire, D., 540 approaches to, 426–430
Maes, P.-J., 389 McGuire, W., 32 automated emotion detection
Magnasco, M. O., 365 McKeown, K., 485 systems, 430–435
Magnusson, M. S., 575 McNeill, D., 500, 506, 509–510, 517, future research directions, 452–453
Mahrt, T., 467–468 560–561 measuring dynamic and spontane-
Major, B., 147, 393 McQueen, J. M., 293 ous facial emotion commu-
Makeup and cosmetics, 59, 62–63 McQuown, N., 429 nication, 437–452
Makhoul, I. R., 108 Mead, G. H., 7 unresolved issues in, 435–437
Male warrior hypothesis, 56 Meagher, D., 576 gesture coding and analysis, 499–520
Mallet, P., 370 Megyesi, B., 482 applications for, 508
Mammucari, A., 441 Mehrabian, A., 18, 349, 553, 598 cognitive approach, 507
Man’s Face and the Mimic Language Mehu, M., 402 cognitive research applications
(Hjortsjö), 427–428 Meier, L. E., 436 for, 510–513
Manual for the FACS (Ekman and Meirick, J., 152 cospeech gestures, 501–507
Friesen), 428 Melanin, 223 defining gestures, 499
Manusov, V., 9 Memory describing gesture form and
Maragoudakis, M., 483 cultural development and, 80 meaning, 504–506
Marchak, F. M., 534 gesture and working memory, 80 future research directions, 517–519
Marler, P., 410 Men. See Gender gesture words, 503–504
Marlier, L., 370 Meng, H., 577 language-associated gestures,
Marr, D., 570 Menstrual synchrony, 369 500–501
Marsh, A. A., 31 Merget, D., 555 linguistic approach, 506–507
Marsh, P., 90 Mermelstein, P., 462 linguistic research applications
Marsi, E., 474 MERT (Multimodal Emotion for, 509–510
Marwell, G., 83 Recognition Test), 593–594 neurological applications for,
Mast, F. W., 533 Metaphoric gestures, 310–311, 315, 323, 513–517
Mast, M. S., 533 501–502 tools for, 507–508
Masuda, T., 89, 347 Methodology nonverbal sensitivity, 589–602
Material rhetoric, 4 body movement coding and analysis, defined, 589
Mather, M., 346 551–578 future research directions, 600–601
Mathews, A., 351 automatic body movement performance-based measures,
Mathews, K. E., Jr., 200 processing, 566–570 589–596
Matlab software, 488, 536 correspondence between self-report measures, 596–597
Matsuda, Y., 577 behavior and psychological self-report measures of constructs
Matsumoto, D., 9, 48, 87, 89, 93, 125, variables, 571–573 related to, 597–600
176–178, 263, 266, 269–270, 565, future research directions, olfactic coding and analysis, 539–548
590, 593 573–577 cross-cultural applications, 547–548
Maudsley Personality Inventory, 169 history of research into, 551–552 future research directions, 548
Mauss, I. B., 266 human observation, 563–566 measurement methods, 543–547

618
 Index

stimuli for, 539–543 gestures, 315 Nonverbal sensitivity, 589–602

voice coding and analysis, 459–490 in primates, 404, 409–411, 415–417 defined, 589
higher level prosodic features, search engine hypothesis, 378 future research directions, 600–601
464–488 Multimodal Emotion Recognition Test performance-based measures of,
low-level acoustic features, 459–464 (MERT), 593–594 589–596
tools for prosodic analysis, 488–489 Murphy, N. A., 145, 563 Brief Affect Recognition Test, 590
Metz, J., 546 Murray, K., 487 Communication of Affect
Miccoli, L., 534 Mutual gaze, 113 Receiving Ability Test, 590
Microexpressions, 276 Contextual and Affective
Microsoft Kinect, 567–568 Nakai, M., 482 Sensitivity Test, 593
Miller, R. E., 437 Nakatani, C., 481 Diagnostic Analysis of Nonverbal
Mind, Self, and Society (Mead), 7 The Naked Ape (Morris), 18 Accuracy, 592
Mind mechanism theory, 344 Narayanan, S., 471–472, 477, 487 Emotion Recognition Index, 594
Minoura, K., 374 Narus, L. R., Jr., 143 Interpersonal Perception Task,
Misch, A., 406 The Natural History of an Interview 594–595
Mishra, T., 462 (McQuown et al.), 429 Japanese and Caucasian Brief
Mitchell, Z. A., 325 Nebraska Symposium on Motivation, 18 Affect Recognition Test,
Mobile technology, effect of on nonverbal Nelson, C. A., 115 592–593
communication, 204 Nemiah, J. C., 450 Multimodal Emotion Recognition
Möbius, B., 467 Nenkova, A., 475 Test, 593–594
“Modal” voice, 292 NEO Five-Factor Inventory, 168, 176, 179 Profile of Nonverbal Sensitivity,
Moffitt, T. E., 270 Nerve force principle, 50–51 590–591
Molfese, D. L., 518 Neuropsychological Gesture Coding strengths and limitations of,
Montagu, A., 18 System, 175, 561 595–596
Montebarocci, O., 176 Neuroticism, 167, 170, 233–234 self-report measures of, 596–597
Montepare, J. M., 31, 94 nonverbal decoding, 175–177 Perceived Decoding Ability
Moore, F. R., 231–232 voice and personality traits, 295 scale, 596
Morphemes, 79, 429, 504 “The New Truth Machine: Does Your Social Skills Inventory, 596–597
Morrin, M., 374–375 Voice Give You Away?” (Rice), 17 strengths and limitations of, 597
Morris, D., 18, 90–91 Newtson, D., 447 self-report measures of constructs
Mortillaro, M., 389 Nezlek, J. B., 266 related to, 597–600
Morton, R., 470 Nicholas, C. L., 174 empathic accuracy, 599
Moss, M., 374 Nicholas, M., 440 sensitivity to deception, 599–600
Motion tracking and capture systems, Nijhar, J., 570 Norman, W. T., 169
432, 562, 566–570, 575–576 Nodding, 123, 149, 168, 430 Noton, D., 533
movement recognition, 568–570 Nolan, F., 291 Nowicki, S. J., Jr., 11, 178, 590, 592
classification, 570 “Nonverbal Communication” Nummenmaa, L., 344
feature extraction, 568–570 (Duncan), 18 Nurmikari-Berry, M., 8
segmentation, 570 Nonverbal Communication (Hinde), 19 Nusbaum, H. C., 515
types of, 566–568 Nonverbal communication research and Nyman, M. J., 344
electromechanical/­ theory, 17–34
electromagnetic, 567 approaches to, 20 Objects and personal belongings, effect
markerless vision-based, 567 emphasizing communication of on nonverbal communication,
optical, 566–567 processes, 21, 26–32 202–203
single accelerometer-based, 568 emphasizing emotion and O’Brien, C. E., 590
Motley, M., 10 expression, 20–26 The Observer annotation software, 565
Movement synchrony, 389–390 future research directions, 32–33 Observer viewpoint, 506
Moving Bodies: Kenneth Burke at the history of, 3–12, 17–19 O’Connor, J. J. M., 241
Edges of Language (Hawhee), 5 cultural heritage, 8–9 Ogale, A., 570
Moving Pictures Experts Group, 432 ethological heritage, 9–10 OGI Speech Tools software, 488
Muehlenhard, C. L., 388 linguistic heritage, 5–6 Oh, Y.-H., 483
Mufson, L., 590 psychological heritage, 10–12 Ohrt, A., 367
Mujica-Parodi, L. R., 543 rhetorical heritage, 3–5 Ohya, J., 570
Müller, K., 515–516 sociological heritage, 6–7 Oikonomidis, I., 568
Multimodal communication, 551 Nonverbal involvement, 191 Oishi, S., 189

619
Index
ÓLaighin, G., 576
Olfactic coding and analysis, 539–548
cross-cultural applications, 547–548
future research directions, 548
measurement, 543–547
behavioral methods, 547
narrative methods, 544–545
neuroimaging methods, 546
psychophysiological methods,
545–546
self-report methods, 543–544
stimuli for, 539–543
choosing, 539–541
human body odor collection,
542–543
intensity, 541
method of delivery, 541–542
Olfactics and olfaction, 363–381
developmental arc of, 110–112
communication of affect and,
111–112
early attachment and, 110–111
relationships and, 111
effect of odor on nonverbal
communication, 199–200
future research directions, 380
human signs, 365–372
identity signs, 369–372
mood signs, 366–368
sexual attraction signs, 368–369
nonhuman signs and/or signals,
372–376
florals, 374
general aroma signals, 375–376
lavender, 374–375
peppermint, 375
nonhuman symbols, 376–377
clean and virtuous, 377
fishy and foul, 376–377
odor as sign of fertility, 61
odor as sign of health, 60
search engine hypothesis, 377–380
signals, 363–364
signs, 363
symbols, 364–365
Oliver, D., 477
Omlor, L., 562
Oosterhof, N. N., 235
Openness to experience, 167, 171–172,
233–234
OpenSMILE software, 488
O’Shaughnessy, M., 90
Ostendorf, M., 471–474, 476, 480, 483, 487
O’Sullivan, M., 600
Owren, M. J., 404
620
Paepke, A. J., 370
Pain, 25–26, 108
body movement and, 570, 576
facial expressions of, 113, 275
odor and, 375
Pan, S., 485
Panter, A. T., 177
Pantic, M., 432–433, 436
Pantomimes, 500, 503
Paralanguage, 27
Paralinguistic cues, 92–93
Parallel process model, 32
Parkinson’s disease, 60, 296, 390, 440
Parkkola, R., 344
Parr, L. A., 265, 412, 414–415
Partan, S. R., 410
Patterson, M. L., 32, 203, 208, 348
Pattnaik, S., 377
Pause, B. M., 367
Pauses, 150–151, 170, 460, 463–464
Pazzaglia, M., 546
PCA (principal component
analysis), 562
PDA (Perceived Decoding Ability)
scale, 596
Pearson, K. L., 440
Pelto, G. H., 80–81
Pelto, P. J., 80–82
Pentland, A. P., 432
Penton-Voak, I. S., 176, 233, 238
Peppermint, 375, 540
Perceived Decoding Ability (PDA)
scale, 596
Perrett, D. I., 63–64, 226, 229, 233
Personality Inventory for Children, 178
Personality traits, 163–180
attributions from facial appearance,
232–239
ability to read faces, 237–238
causal relationships between faces
and personality, 235–237
modern physiognomy, 233–234
origins of physiognomy, 232–233
overgeneralization, 238–239
Big Five framework, 166–172
agreeableness, 170–171
conscientiousness, 171
extraversion, 166–170
neuroticism, 170
openness to experience, 171–172
cultural development and, 80
future research directions, 179–180
historical controversies and
methodological issues, 163–165
individual differences
in gaze as information gathering,
345–346
in gaze behavior and perception,
351
in pupillary behavior, 342
in reading facial expressions, 337
in tears, 339
intelligence, 174–175
nonverbal decoding, 175–179
introversion/extroversion, 177–178
neuroticism/anxiety, 175–177
nonverbal encoding, 179
organizational frameworks, 165–166
sexual orientation and, 173–174
status and dominance, 172–173
voice and, 295–296
Personal Space (Sommer), 18
Person-situation debate, 163–164
Pfau, T., 462
Pharynx, 291–292
Pheromones, 368–369, 539–540
Philipsen, G., 8
Phillips, K., 541
PhonEdit software, 488
Phonemes, 290, 429
Physical environment, 189–210
ecological systems model, 208–210
environmental features, 195
architectural elements, 201–202
color, 203–204
lighting, 196–197
mobile technology, 204
objects and personal belongings,
202–203
odor, 199–200
sounds, 200–201
temperature, 197–199
historical context, 189–190
settings, 192–195
social goals, 204
regulation of appropriateness, 205
regulation of belonging, 205–206
regulation of controlling, 205–206
regulation of efficiency, 205
regulation of privacy, 206–208
regulation of self-enhancement,
205–206
regulation of understanding,
205–206
utility of nonverbal communication,
190–192
Physiognomica (Aristotle), 232, 241
Physiognomy. See also Facial appearance
detecting success from
faces, 241–242
 Index

judging sexual orientation from faces, The Presentation of Self in Everyday Life regions of analysis, 464
242–243 (Goffman), 7 speaker normalization, 463
judging social group membership Preuschoft, S., 409 tools for analysis, 488–489
from faces, 243–244 Price, P., 487 Proust, M., 376, 380
modern, 233–234 Pride, 30, 51, 122, 389, 561, 565 Provine, R. R., 110
origins of, 232–233 Prieto, P., 484–485 Proxemics (interpersonal space), 6, 27,
PICA (Porch Index of Communicative Primate nonverbal communication, 389–392, 553–556
Ability), 440 401–418 cultural norms for, 93–94
Pierrehumbert, J., 466, 470, 486 anatomy of voice, 290 defined, 553
Piliavin, I., 193 communicating sexual receptivity/ environment and, 202
Pipitone, R. N., 63 availability, 62–63 gender and, 149–150, 155
Pisanski, K., 241 compared to human communication, interaction in virtual reality, 555–556
Pitch (fundamental frequency) of 22, 27, 405–406 placement of personal objects and, 203
speech, 460–462 discrimination of facial expressions, techniques for assessing, 553–555
reset and stylization, 461 411–418 unfocused interactions in public
shape modeling of contours, 461–462 context dependence/meaning, settings, 195
Pitch accent (intonational prominence), 417–418 Proyer, R. T., 436
468–478 identification of major categories, Pruzansky, S., 293
automatic classification of, 475–478 411–415 Pseudospontaneous communication, 440
automatic detection of, 470–478 multimodal components, 415–417 Psychological Bulletin, 18
impact of speaking style on, 475 dynamic and multimodal signals, Psychological gender (sex-role
using lexical information, 409–411 identification), 140, 151–153
473–475 eye behavior and gaze, 343–345, Psychology Today, 17
types of, 469–470 348–349 Puccinelli, N. M., 176
Pitrelli, J., 476 facial, 401–403 Pupillary behavior, 340–342
Pitterman, H., 592 facial expressions, 264–265 dilation, 62
Plant, E. A., 144 gestures, 323 individual differences in, 341–342
Platt, T., 436 influences on, 406–409 age, 342
Play face, 402, 410, 412–414, 416 ecological, 407–408 context and culture, 342
“Plus face,” 122 physical, 406–407 gender, 342
Pointing gestures, 123, 323 social, 408–409 traits, 342
Pollen, M., 374 lessons learned from, 403–405 pupil diameter
Pollick, F. E., 564 nonfacial, 403 analysis, 534
Polt, J. M., 340 onset of expressive behaviors, 106 measuring, 532
Polygraphs, 600 physical confrontation, 235 social dynamics of, 341–342
PONS (Profile of Nonverbal Sensitivity), skin condition, 223 Purkinje images, 527
176, 178, 237, 590–591 smiling, 19, 401–402 Putin, V., 335
Population density, 78 social signals from facial expressions,
Porch Index of Communicative Ability 121 Questionnaire Measure of Emotional
(PICA), 440 touch, 108–109 Empathy, 598
Posture, 124–125, 387–388 Principal component analysis (PCA), 562 Quintilian, M. F., 4
communication of dominance, 172–173 Prkachin, K. M., 113
defined, 387 Probabilistic functionalism, 30, 189, 571 Race and ethnicity. See also Culture
gender and, 149–150, 155 Profile of Nonverbal Sensitivity (PONS), environments perceived as
posture mirroring, 149, 388 176, 178, 237, 590–591 unreceptive, 203
sexual orientation and, 174 Prosody gaze as a signal of self-relevance, 348
Posture-gesture mergers, 388 higher level features, 464–488 individual differences in gaze
Potter, J., 171 conventions for describing behavior and perception, 352
Pound, N., 233 prosody, 464–468 judging sexual orientation from
Powell, E., 172 intonational phrasing, 478–488 faces, 242
Power asymmetry hypothesis of pitch accent, 468–478 Rambow, O., 474, 484, 487
motivational emancipation, 409 low-level features, 459–464 RaP (Rhythm and Pitch) labeling
Powers, S. R., 436 duration and rhythm, 462–463 system, 467
Praat software, 488 pauses, 463–464 Rapid facial signs, 257
Prehn, A., 367 pitch and intensity, 460–462 Rapport, 31

621
Index

mirroring of posture, 388 Robinson, J., 6–7 Schaeffer, G. H., 208

whole body movements, 389–390 Roese, N. J., 147 Scheflen, A. E., 6, 18, 27
Ratchet effect, 79 Roether, C. L., 562 Scherer, K. R., 30, 297–298, 389, 552, 573
Ratner, H. H., 79 Rogers, Y., 576–577 Schizophrenia, 337, 346
Ratneshwar, S., 374 Roggman, L. A., 224 inappropriate affect, 443–444
Raudenbush, B., 375 Rohner, R. P., 80–81 SVT studies, 444–446
Rauh, C., 436 Romera-Paredes, B., 576 Schmid, H., 485
Raven Advanced Progressive Matrices, 175 Rosati, A. G., 265 Schmid, P. C., 533
Re, D. E., 58–59, 241 Rose, R., 470 Schmidlin, A. M., 147, 393
Read, I., 476, 484 Rosenberg, A., 462, 475, 477–478, 481, Schneider, C., 106
Reading the Mind in the Eyes Test 485, 487–488 Schwartz, B., 172
(RME), 337–338 Rosenberg, E. L., 268 Schwarz, N., 376
Redican, W. K., 264 Rosenfeld, H. M., 31 Schweisgut, C., 371
Reflexive tears, 338 Rosenthal, R., 7, 11, 177, 388, 566, 591 Science, 19
Regulating gestures, 500 Rosner, B., 468 Search engine hypothesis, 377–380, 544
Relational messages, 7 Ross, E., 444–445 Seated Kinesic Activity Notation
Remland, M. S., 553, 555 Ross, K., 472–474, 476 System, 168
Ren, Y., 470 Rossi, N., 176 Seebeck, T., 370
Rendall, D., 404 Rothkrantz, L. J. M., 432–433, 436 Segmentation
Renninger, L. A., 153 Rothstein, S., 28 of behavior stream, 573–574
Reproductive fitness, 57–66 Rowe, A. C., 176 body movement recognition, 570
dominance and, 63–65 Rubenstein, A. J., 239 facial expression coding and analysis,
communicating through body, Ruch, W., 436 446–449
64–65 Ruffman, T., 118 Self-adaptors. See Adaptors
communicating through face, Rui, Y., 570 Self-conscious emotions, 29–30
63–64 Rule, N. O., 58–59, 174, 242–243, 338 Self-touch, 109–110, 148, 394, 554
fertility and, 61–62 Rushton, J. P., 169–170 Sell, A., 235
communicating through Ruske, G., 462 Selzer, J., 5
body, 61–62 Russell, J. A., 10, 22, 28, 116 Semiotica, 18
communicating through face, 61 Russell, R. L., 178 Semple, S., 418
health and, 58–60 Ryabichenko, T. I., 176–177 Serviceable associated habits, 259
communicating through Ryan, M. J., 404 Serviceable habits principle, 50, 265, 271
body, 59–60 Settings, behavioral, 192–195
communicating through Saccadic eye movements, 526, 531–533 Settings, cultural meaning of, 83–84
face, 58–59 Sacco, D. F., 32, 122 Sex differences. See Gender
sexual receptivity/availability Sadness, 261–262 Sex-role identification (psychological
and, 62–63 culture and, 52 gender), 140, 151–153
communicating through facial communication of, 50 Sexual dimorphism, 228–232
body, 62–63 paralinguistic cues, 92 femininity in women’s faces, 228–229
communicating through face, 62 posture, 387 masculinity in men’s faces, 230–232
Response linkage, 88 pupillary behavior, 341–342 Sexual orientation
Reynaert, M., 474 serviceable habits principle, 271 decoding accuracy, 144
Rhodes, G., 60, 231–232 shame and, 25 flirting, 153
Rhythm and Pitch (RaP) labeling system, survival value of, 55 gait, 152
467 Sagisaka, Y., 482 gaze and eye contact, 146
Rhythm of speech, 462–463 Sailer, L. D., 235 gestures, 150
Riccardi, G., 470 Saine, T. J., 19 interpersonal space, 150
Riggio, H. R., 11, 170, 179 Sakamoto, R., 374 judging from faces, 242–243
Riggio, R. E., 11, 170, 179 Salleh, L. M., 8 personality traits, 173–174
Ring, B., 344 Samson, A. V. M., 402 pupillary responses, 341–342
Rituals, 82 Sanchez-Vives, M. V., 556 same sex touch, 147–149
Riverside Behavioral Q-Sort, 168 Sanders, S. A., 545 smiling, 146
RME (Reading the Mind in the Eyes Santana, S. E., 407 stereotypes, 140
Test), 337–338 Sasikumar, M., 577 Sexual pleasure, 26
Roberts, S. C., 239 Schaal, B., 370 Sexual receptivity/availability, 62–63

622

communicating through body, 62–63
communicating through face, 62
gender and decoding accuracy, 144
olfactic signs, 368–369, 373
Seyfarth, R. M., 404
Shame, guilt, and humiliation, 22, 25,
51–52, 122, 426–427, 565
blushing, 152
body movement and, 389
facial expressions of, 426
Shapiro, B., 447
Shared Attention mechanism, 344–345
Shared intentionality, 79
Shared signal hypothesis, 350
Shearn, D., 152
Sheehan, M., 444–445
Sheldon, H., 18
Shimodaira, H., 482
Shutter, J. M., 119
Sidner, C. L., 473
Sifneos, P. E., 450
Signals
bodily, 52–54
differentiation from signs and
symbols, 47
facial, 52–54
olfactic, 363–364, 372–376
vocal, 52–54
voluntariness of, 47–48
Sign languages, 520
coding hand signs, 505
compositionality of, 308
emerging, 318–319
gestures versus, 307–308
homesign, 318–320
in Kendon’s continuum, 500, 503
Signs
bodily
communicating dominance, 64–65
communicating emotions, 51
communicating fertility, 61–62
communicating health, 59–60
communicating sexual receptivity/
availability, 62–63
differentiation from signals and
symbols, 47
facial
communicating dominance, 63–64
communicating emotions, 49–51
communicating fertility, 61
communicating health, 58–59
communicating sexual receptivity/
availability, 62
olfactic, 363
human, 365–372
Index
nonhuman, 372–376 Social intelligence hypothesis, 408
vocal, 51–52 Social referencing, 272
voluntariness of, 47–48 Social roles
The Silent Language (Hall), 8 cultural meaning of, 84
Silipo, R., 468, 475 voice and, 295
Simmons, L. W., 60, 231 Social Skills Inventory (SSI), 596–597
Simpson, J. A., 599 Sociobiology (Wilson), 19
Singer, H., 482 Socrates, 19–20
Singh, A., 570 Sommer, R., 18, 27, 189
Singh, D., 61 Sorce, J. F., 272
Situational Interpretations Task, 594–595 Sorenson, E. R., 19
16 Personality Factor Questionnaire, 169 Soto-Faraco, S., 517
Skin condition, 222–224 Sound eXchange (SoX) software, 488
Skipper, J. I., 515 Sounds, effect of noise on nonverbal
Slater, M., 556 communication, 200–201
Slide-viewing technique (SVT), 437–439, Source/filter theory of speech
452 production, 289–290
studies in patients with brain Sources of messages
damage, 439–440 body expressions, 387–395
studies in patients with body movement, 389–390
schizophrenia, 444–446 gait, 389–390
studies in preschool children, 438–439 postures, 387–388
studies in young adults, 438 proxemics, 389–392
Slobodskaya, H. R., 176–177 touch, 392–394
Sloetjes, H., 557 eye behavior and gaze, 335
Slow facial signs, 257 attraction to eye region, 335–336
Small, S. L., 515 facial actions around eyes, 336–338
Smiling gaze behavior, 343–352
eye region of face and, 336 pupils, 340–342
friendly contact in public settings, 193 tears, 338–340
gender and, 141, 145–146, 155 facial appearance and physiognomy,
interpersonal effects of facial 221–244
expression, 272 attractiveness, 221–232
neonatal, 264 dynamic and malleable facial
subjective experience and, 266 cues, 239–241
universality of facial expression of new directions in, 241–244
emotions, 261 personality attributions from,
Smooth pursuit eye movements, 526 232–239
Sneed, C. D., 168, 171–172 facial expressions, 257–278
Snyder, D. K., 178 of basic emotion, 259–274
Sobel, N., 112 broad individual differences,
Social and moral emotions, 29–30 274–275
Social biofeedback, 449–450 cognitive and physical states and,
Social-cognitive perspectives, 31–32 276
Social communication competence, 107 facial anatomy, 258–259
Social complexity hypothesis, 408–409 future research directions, 277–278
Social goals, 204–208 mental and physical health and,
regulation of appropriateness, 205 275–276
regulation of belonging, 205–206 of other emotions and affective
regulation of controlling, 205–206 states, 274
regulation of efficiency, 205 as reliable leakage clues to
regulation of privacy, 206–208 deception, 276
regulation of self-enhancement, role in conversation, 276–277
205–206 gestures, 307–327
regulation of understanding, 205–206 categories of, 308–309
623
Index
cospeech gestures, 309–311
as dominant linguistic system,
318–320
future research directions, 326–327
information communicated
through, 313–315
integration of speech and, 315–318
involvement of hands, 308
non-communicative uses for,
323–325
relationship to speech, 311–313
research applications, 325–326
sign language versus, 307–308
unique nature of, 320–323
olfactics, 363–381
future research directions, 380
human signs, 365–372
nonhuman signs and signals,
372–376
nonhuman symbols, 376–377
search engine hypothesis, 377–380
signals, 363–364
signs, 363
symbols, 364–365
physical environment, 189–210
ecological systems model, 208–210
environmental features, 195–204
historical context, 189–190
settings, 192–195
social goals, 204–208
utility of nonverbal communication,
190–192
primate nonverbal communication,
401–418
compared to human communica-
tion, 405–406
discrimination of facial
expressions, 411–418
dynamic and multimodal signals,
409–411
facial, 401–403
influences on, 406–409
lessons learned from, 403–405
nonfacial, 403
voice, 289–300
adaptations and interactions,
298–300
anatomical and mechanical
factors affecting, 289–294
biological events affecting, 296–298
cultural factors affecting, 294–295
evolution of, 290–291
life-span factors affecting, 294–295
personal factors affecting, 295–296
Soussignan, R., 370
624
Souza, T., 546 emotions and, 48
SoX (Sound eXchange) software, 488 communicating through body,
Specific Action Coding System, 6 51–54
Spectral balance, 468, 470 communicating through face,
Spectral tilt, 468 49–54
Speech illustrators. See Illustrators communicating through voice,
Spellman, L., 152 51–54
Spiegel, N. H., 143 survival value, 54–56
Spiliotopoulos, D., 477 identity and, 56–57
Sridhar, R., 462 group behavior, 56
Sridhar, V. R., 471 person recognition, 56–57
SSI (Social Skills Inventory), 596–597 SVT. See Slide-viewing technique
Stark, L., 533 Swann’s Way (Proust), 376
State-Trait Anxiety Inventory, 176–177 Symbolic gestures. See Emblems
Static facial signs, 257 Symbols
Status cues perspective, 28–30 bodily, 54
Status-signaling lens, 121 differentiation from signs and
Steele, S., 470 signals, 47
Stephen, I. D., 223 facial, 54
Stern, D. N., 349 olfactic, 364–365, 376–377
Stier, D. S., 393 vocal, 54
Stigma: Notes on the Management of voluntariness of, 47–48
Spoiled Identity (Goffman), 7 Symmetry of facial features, 225–226, 243
Still-face paradigm, 112 Sympathy, 55–56, 110, 148, 272, 275
Stirrat, M., 63–64 System for Identifying Affect Expressions
Stoddart, D. M., 373 by Holistic Judgments (AFFEX),
Stokes, J. M., 178 426–427
Stokoe, W. C., 504, 510–511
Stouthamer-Loeber, M., 270 Tacesics, 6
Straley, B., 152 TAIS (Test of Attentional and
Street, R. L., 7 Interpersonal Style), 169–170
Stress timing, 294 Tamburini, F., 473
Structural approaches, 27, 428–430 Taste, facial expressions in reaction to, 113
Stryker, K., 152 Tatler, B. W., 527
Submissiveness, 28–29, 121–122, 124, Tattoos, 64–65
339, 389, 409, 591 Taylor, P., 461, 469–470, 477
Subordination hypothesis, 11 TCoG (Tonal Center of Gravity), 462
Subramanyam, V. R., 377 TDRNN (Time Delay Recursive Neural
Success. See Triumph and success Network), 470–471
Sullivan, M. W., 105 Tears, 338–340
Sullivan, T. E., 375 individual differences in, 339
Sun, X., 472–473, 476 age, 340
Super, C. M., 105 context and culture, 340
Supplementary gesture-speech gender, 339–340
relationship, 311–312, 317–318 traits, 339
Surcinelli, P., 176 social dynamics of, 339
Surprise, 261–262 Temperature, effect of on nonverbal
flirting and, 28 communication, 197–199
serviceable habits principle, 271 Tepperman, J., 487
startle facial expression, 112–113 Territoriality, 207
survival value of, 55 Terzopoulos, D., 431
temperature and, 23 Tesser, A., 172
Tomkins’s theory of facial Test of Attentional and Interpersonal
expression, 426 Style (TAIS), 169–170
Survival to age of reproduction, 48–57, 66 Testosterone, 58
 Index

dominance and, 63 Tucker, J. S., 179 biological events affecting, 296–298

facial appearance and personality, Tylor, E. B., 80 clinical changes, 296
230–232, 236 emotion, 296–298
’t Hart, J., 467 Ueno, A., 264 pharmacological changes, 296
Thatcher, M., 298 Ueno, Y., 264 brain physiology and, 299–300
Theodorou, P., 270 Uncertainty, 83, 463–464 calls made by primates, 403–404,
Thomas, M. D., 567 Unfocused interactions, 190 406, 408
Thompson, J., 262 The Unspoken Dialogue (Burgoon and communicating emotions
Thornhill, R., 231–232 Saine), 19 signals, 52–54
Threat assessment, 57 Usui, A., 374 signs, 51–52
Tickle-Degnen, L., 11, 176, 388 symbols, 54
Tickling, 110 Vaina, L., 570 conscious changes to, 298–299
Tightness and looseness (cultural Valstar, M. F., 433 conversational interaction and, 298
dimension), 82 Van den Bosch, A., 474 cultural norms for, 92–93
Tigue, C. C., 241 Van der Linden, J., 576–577 evolution of, 290–291
Tilt model, 469, 477 Van Dyck, E., 389 gender and, 150–151, 155
Time Delay Recursive Neural Network Vanger, P., 441 geographical origins and, 294
(TDRNN), 470–471 Van Hooff, J. A. R. A. M., 19, 264, 409, 574 lifetime changes to, 294–295
Timing of expressions and gestures, 27 Van Lancker, D., 293 personality and, 295–296
Tipples, J., 351 Van Lear, C. A., 10, 440 social roles and, 295
ToBI (Tones and Break Indices), The Varieties of Human Physique unconscious changes to, 298
465–467, 489–490 (Sheldon), 18 Voice coding and measurement, 459–490
Todd, E. R., 179 Vasen, A., 105 higher level prosodic features, 464–488
Todorov, A., 235 Veilleux, N., 483 conventions for describing
Tomasello, M., 79 Vergence eye movements, 526 prosody, 464–468
Tomkins, S. S., 19, 52, 260, 426, 435 Verma, R., 446 intonational phrasing, 478–488
Tomonaga, M., 264 Vernon, P. E., 164 pitch accent, 468–478
Tonal Center of Gravity (TCoG), 462 Vick, S. J., 412 low-level acoustic-prosodic features,
Tones and Break Indices (ToBI), Vinicur, D. L., 173 459–464
465–467, 489–490 Vision, 112–125 duration and rhythm, 462–463
Tooby, J., 49 communication of affect, 114–121 pauses, 463–464
To Tell the Truth (television program), 18 recognizing and producing body pitch and intensity, 460–462
Touch (haptics), 392–394 movement, 123–125 regions of analysis, 464
automatic detection of touch sensation and facial expression, speaker normalization, 463
behavior, 577 112–114 tools for prosodic analysis, 488–489
developmental arc of, 107–110 social signals from facial expressions, Von Clef, J., 375
communication of affect and, 121–122 Vosshall, L. B., 365
109–110 Visual rhetoric, 4
early attachment and, 107–108 Vocal fry (diplophonia), 460 Wade, N. J., 527
relationships and, 108–109 Voice, 289–300 Wade, T. J., 153
gender and, 141, 147–149, 155 acoustic context and, 298 Waist-to-hip ratio (WHR), 61
self-touch, 109–110, 148, 394, 554 adaptations and interactions, 298 Wakefield, E. M., 518
techniques for assessing, 553–554 anatomical and mechanical factors Walker, M., 593
Touch avoidance, 393–394 affecting, 289–294 Walker-Andrews, A. S., 368
Touching (Montagu), 18 acoustic cues to vocal identity, Wallbott, H. G., 557–558, 566
Tracy, J. L., 565 292–293 Waller, B. M., 265, 402, 406, 412
Trade-off theory, 230 contextual and interactive factors, Wang, M. Q., 484
Transcriber software, 488 293–294 Wang, R., 483
Trémeau, F., 443 facial features, 292 Wartenburger, I., 175
Trimboli, A., 593 individual features, 291 Waters, K., 431
Triumph and success, 94, 241–244, 389 larynx, 289 Watson, O. M., 18, 85, 92, 94, 394
Troje, N. F., 567 pharyngeal and hard palate Wavesurfer software, 488
Trommsdorff, G., 144 morphology, 291–292 Wedekind, C., 370
Truitt, G., 388 Big Five framework of personality Weenink, D., 488
Tubbs, M. E., 203 traits, 167 Weisman de Mamani, A. G., 446

625
Index

Werker, J. F., 116 WinPitch software, 488 Yoon, T.-J., 481

Werthman, C., 193 Winstead, B. A., 393 Young, A. W., 387–388
Wesnes, K., 374 Winter, D. D. N., 388 Young, R. K., 61
West, T. V., 436 Withgott, M., 470 Yuki, M., 347
Weston, E. M., 63 Wokurek, W., 477
Whalen, P. J., 350 Women. See Gender Zaidel, D. W., 60
Whitehouse, J., 406 Worchel, S., 391 Zatorre, R. J., 546
WHR (waist-to-hip ratio), 61 Wundt, W. M., 25 Zebrowitz, L. A., 31, 94, 231, 235
Widaman, K. F., 179 Zechner, K., 468
Widell, C., 388 Xia, S., 471 Zeng, Z., 433
Widen, S. C., 116 Xydas, G., 477, 483 Zervas, P., 483
Wiggins, J. S., 168 Zhong, C. B., 377
Wightman, C., 471, 474, 480, 483, 487 Yabrove, G., 270 Zhou, W., 111
Wilhelm, F. H., 266 Yallop, C., 468 Zivin, G., 122
Willems, R. M., 514–515, 517 Yamada, H., 89 Zoladz, P., 375
Williams, L., 147, 393 Yamato, J., 570 Zuckerman, M., 143, 596
Willingham, B., 263 Yee, N., 555 Zuk, M., 230
Wilson, E. O., 19 Yeshurun, Y., 112
Wilson, P., 47 Yiend, J., 351

626