Situated Interactions: Agency in Context and Affordances

Trevor Hess
University of North Carolina at Charlotte
9201 University City Blvd, Charlotte, NC 28223
ahess5@uncc.edu
ABSTRACT

This paper analyzes the use of affordances in prevailing

methods of contextual representation. In doing so, it identifies a number of key deficiencies in their description. First
a number of authors cited do not conform to the original
definition of affordances proposed by James J. Gibson in
his original writings. Secondly, their proposed use in representation of context and the literature cited does not provide a complete account of agency that allows affordances
to describe intention and motivation. Through this analysis,
the paper constrains the ability of affordances to describe
context, and seeks to redefine the logic of contextual description.
Author Keywords

Context-awareness, Affordances, Representation

General Terms

Design, Theory
INTRODUCTION

As computational devices pervade the world we live in

today, we should consider the ability for computers to be
intelligence agents that can be interacted with. While the
dream of an artificial intelligence humanoid no longer exists as a research goal, the need to understand people still
exists.
Manifest today in the field of context-aware computing,
considerable effort has been committed to accurately representing and interpreting the world to imitate human understanding. Arising through the observation of people and
their actions, context serves as the goal. Defined as our
position relative to the physical world, the social realm and
historical lineage, the ability to computationally understand
context would mean the ability to understand human behavior.
This paper takes a critical look at the prevailing solution to
contextual interpretation, embodied interaction. Proposed
by Paul Dourish in 2001, it responds to interpretive methods proposed by ubiquitous computing researchers. To integrate contextual interpretation into applications, ubiquitous computing researchers have taken a static approach to
context. Based on location, identity, time and activity, context does not change without one of these parameters
changing [1]. Instead Dourish proposes an interactive model of context [4]. In this model context is driven by activity,
specifically interaction with the physical and social envi-

ronment. The result is a definition of context that allows it

to by dynamic and occasioned based on represented behaviors. By negotiating between multiple people in the same
environment, this can also achieve an interpretation of context that extends beyond the individual.
While Dourishs model solves a number of problems regarding the representation and interpretation of context, his
solution, embodied interaction [5] does not consider a
number of critical writings that fundamentally alter embodied interactions to accurately describe context.
THE REPRESENTATION PROBLEM

In 1969, John McCarthy and Patrick J. Hayes identified the

need for formal representations of the world as a philosophical problem for computational intelligence [9]. Alone, this
is not a difficult challenge. A Turing machine, at its core,
satisfies the requirements for general intelligence since its
representation, 0 and 1, is not tailored to a specific problem. An issue arises, however, when computers are required to imitate human understandings of the world. When
imitation is required, a computer must create a numeric
representation of the environment. Once created, the representation must be interpreted to construct an understanding that is analogous to human perception of the environment.
Today the issue of representation directly affects the field
of Human Computer Interaction (HCI) through ubiquitous
computing [18]. While the original paper has inspired a
number of research directions, a key component to the
ubiquitous computing paradigm is context-awareness. Context in this case refers to the meaning of the relationship
between the individual and their environment. Thus, when
computers attempt to represent and interpret human context, it is an analogous to the problem raised by McCarthy
and Hayes. In both cases, the computer must be able to
represent and interpret the world in the same way that people understand it.
From a computational perspective, context must be approached through a quantitative view of the environment.
Information to be interpreted must be collected through
sensors, leading to a mathematical, model driven view on
context and human behavior [4]. Since sensors will never
provide a complete description of the environment due to
inherent inaccuracies, research has focused on the devel-

opment of a set of environmental properties that can be

interpreted as context.
In the proposal for context-aware computing, Bill Schilit
proposed location, the identity of surrounding people, and
environmental objects as the identifying characteristics of
context [12]. As his proposal focused specifically on the
automatic reconfiguration of computer settings, such as
automatically connecting to networks, these properties
proved sufficient for achieving his goals. When tailored to
specific goals, limited, quantitative measurements of the
environment can be sufficient, but fall short of a general
approach to context.
Gregory Abowd and Anind Dey provide a general approach
by formally defining context and context-aware applications [1]. Instead of identifying the goals of awareness, they
define context as any information that can be used to identify the situation of a person, place or object. With this information, context-aware applications are those that use
context to provide relevant information and/or services to
the user. To represent context in the computer they identify location, identity, activity and time as primary descriptors.
While primary descriptors of context may be useful when
trying to achieve specific goals, they directly contradict the
opposing, human understanding of context. This understanding, which must be described through observation
results from activity. As individuals interact with their surroundings and each other, they create meaning, which is
ultimately described as context.
In response to the inability to accurately represent context,
two prevailing methods emerged from the computational
community. The first, situated actions was proposed by
Lucy Suchman [17] responds to the rationalist interpretation of human behavior prevalent in early artificial intelligence systems. She critiques the computational understanding of human behavior that people create rational plans and
then perform a set of action steps to achieve goals. Instead
of plans, she states that human behavior must be considered
in context, adapting to the changing relationship between
the individual and their environment. For application designers this means that the situation provides a framework
for interpretation, but creating this framework requires prior knowledge of the environment and the social group that
will use the application.
Paul Dourish expands on Suchmans proposal of situated
actions with embodied interaction. In this, Dourish addresses the understanding that context is a dynamic and
occasioned property created through interaction between
the individual and their environment [4]. His proposal
states that social and physical interactions have meaning,
and by negotiating these meanings, computers can interpret
context arising from behavior. In his definition, Dourish
defines context in two parts, being in the physical world

and being in the social world. Being, in this case refers to

Heideggers phenomenological perspective on being-inthe-world [5], which relates to the relationship between the
individual and their environment. In his interpretation of
the philosophical tradition, Dourish interprets something as
phenomenological if it has be ability to create a phenomenon, in this case context. To account for being in the social
world, Dourish cites Suchman, whose situated perspective
allows the meaning of social interaction to be presumed
based on the relationships between people.
To account for being in the physical world, Dourish introduces the idea of affordances. In his writings on embodied
interaction affordances are defined through three authors.
James J. Gibson provides the initial definition of affordances as opportunities to interact with the environment [8].
Coming from psychology, Gibson conceived affordances as
a way to explain explain perception of the world. Then
Dourish cites Donald Norman, prevalent in the field of industrial design. In Normans definition, affordances are
perceived opportunities for interaction with the environment and the objects in it [10]. Finally, William Gaver describes the affordances of technology [6]. Through these
definitions affordances allow embodied interactions to account for the qualitative understanding of context is observed and described.
QUANTITATIVE DESCRIPTIONS OF AFFORDANCES

With the identification of representation as a key problem

in computational interpretation of context, the ability to
quantitatively describe situated actions and embodied interaction should be assessed. The situated perspective of
Suchman and social being in embodied interaction requires
the presupposition of the situation so it is not available for
translation from qualitative to quantitative representation.
Embodied interactions, however, describe a relationship
between the individual and the environment through affordances, both of which can be measured.
Anthony Chemero provides a holistic perspective on affordances and whether they are properties of the individual,
the environment or both [2]. Affordances describe opportunities for interaction they require both an environment
whose had properties that can be utilized, and an individual
who has the ability to utilize the properties. Since both environment and individual are required, affordances can be
seeing as properties of the relationship between the two. To
take Gibsons [8] example flat surface does not afford
standing without someone to stand on it.
This relationship, and the ability to quantify affordances are
highlighted in a series of studies on the perception that
stairs afford climbing. The studies showed that as the
height of risers increase, people show a reduced perception
of climb-ability. Since the results produced quantifiable
results, the studies showed that affordances and perception
could be translated into computational representations.

The first of these studies conducted in 1984 [19] presented

participants of varying heights a set of stairs and were
asked whether or not they thought they could climb them.
Results revealed a split in perception between short and tall
participants. Intuitively, the perception of climb-ability
stopped for shorter participants at a lower riser height than
it did for taller participants.

William Gaver is cited by Dourish to describe the affordances of technology. While Gibson describe affordances explicitly in the physical world, Gaver responds to a
need in HCI to study perception in virtual and technologic
space [6]. When perceiving an affordance in technology, he
states that affordances exist relative to the needs of the individual and are perceived once need arises.

The second of these experiments conducted in 1992 [15]

explored the relationship between muscular strength and
perceived climb-ability. The study was conducted by presenting young and old adults with a set of stairs, asking
them which stair they perceived to be the tallest they could
climb. The participants then attempted to climb the stair
without assistance moving up an increment if successful or
down if not successful. The findings produced two results.
First they supported the findings of the earlier study, since
a large gap existed between the abilities of tall and short
participants. Secondly, they extended the findings by showing that perception and ability changed with age and
strength.

To conclude the discussion about affordances and context,

we turn to a paper published by Phil Turner stating that
affordances can be used as context. In the paper he talks
about two types of affordances: simple and complex. Simple affordances refer to those defined by Gibson. Contrary
to Gibsons definition, however, Turner states that affordances both invite and demand interaction based on a
gestalt understanding of affordances [13]. Complex affordances then account for the social and historical components of context. In Turners definition, affordances create
interaction. When these interactions are observed, a human
description of context can be created.

The findings of these studies mean that perception can be

quantified. This allows for a direct translation of qualitative
descriptions of affordances to the quantitative descriptions
needed to integrate embodied interaction with computational systems. Unlike writings in ubiquitous computing,
these studies show that given the same environment, perceptions between individuals will vary. From the standpoint
of embodied interaction, this means that our interactions
with the environment are constrained by our perception of
affordances. Since affordances still exists without perception [2], they still stand as a viable description of the environment and our relationship to it.

While Dourish cites Donald Normans work in support of

his view on affordances, his misses a key publication of
Normans in 1999. In this paper Norman retracts his original position on affordances, which does not account for a
situation where they are not perceived [11]. When not perceived, affordances cannot suggest certain types, or methods of interaction. By reconsidering the perception of affordances, Normans description of affordances is realigned
with Gibson. While affordances suggest a way to interact,
they do not encourage behavior.

AFFORDANCES AS CONTEXT
Expanding Gibsonian Affordances

Affordances arose with the publication of The Ecological

Approach to Perception by James J. Gibson. Rather than
approach perception of the environment through the study
of vision, prevalent in psychology at the time, affordances
look at behavior and perception of the environment by the
opportunities offered to the individual [8]. Essentially, Gibson defines affordances as mere opportunities; they do not
invite or encourage behavior. Without the ability to create
activity the Gibsonian definition of affordances is not sufficient for describing the observed phenomenon of context.
With the publication of The Psychology of Everyday
Things, Donald Norman extends Gibsons original definition of affordances by stating that affordances encourage
interaction [10]. In his description, the presence of affordances strongly suggests the operation of objects. When
looking at a door, a doorknob suggests that we turn it,
while a push plate suggests that we push it. By encouraging
interaction, affordances gain the ability to create activity.

Problems of Interpretation

In the same way that Normans pre-retraction position on

affordances incorrectly expands Gibsons original definition so to does Gaver and Turner [14]. Gaver states that
affordances present relevant and desirable actions to the
individual. This creates a scenario where affordances only
exist when they are needed. While this supports Dourishs
use of affordances in the description of context, affordances
that only exist when the user needs them have the same
invitation quality that Gibson rejects. By stating that affordances invite behavior, Turner is subject to the same
criticism. Since affordances exist as opportunities for interaction whether the individual uses them or not [2], positions justified through invitation or automatic perception
should be rejected. Without the ability to create or invite
observable behavior, the ability for affordances to describe
context is severely limited.
Redefined Affordances as Descriptors of Context

The incorrect expansion of Gibsons definition of affordances should be sufficient to prove that alone they are
not descriptors of context. Affordances cannot account for
the social or historical aspects of context, nor do they possess the invitational qualities that would allow them to describe or encourage interaction.

While the inability to encourage behavior damages Dourishs description of embodied interaction, the relationship
between the individual and environment still exists. Since
context is described by Dourish as emerging from activity,
the interaction, or the usage of affordances can still describe context. The primary change to Dourishs position is
the conflation of perception and practice. Without the ability to invite interaction, perception cannot be used to describe context. It is purely a mental activity that does not
manifest itself physically or socially.
The mere presence of an affordance does not guarantee a
certain behavior. This restricts Dourishs position on affordances, but does not eliminate it. Through interaction,
affordances still describe behavior that can be observed and
described as context.
THE PROBLEM OF AGENCY

Lying at the core of this misinterpretation of affordances is

agency. Defined as the freedom or ability for an individual
to act and affect the world around them, agency is a key
property that Gibson did not describe [3]. Without providing a theory of agency, affordances cannot describe the
intention or motivation behind interaction.
A theory of agency is necessary for affordances to explain
why people use certain affordances instead of others. Since
affordances do not invite behavior, the opportunity they
provide cannot account for the meaning that underlies interaction. Simply interacting with an object or environment
is not inherently meaningful. Consider the affordances of a
piece of paper:
A piece of paper affords equally writing gibberish and
sonnets; it affords writing a shopping list or a note to a
colleague; it affords making a map; it affords writing nothing upon; it affords wadding up and throwing away; it affords making paper airplanes; it affords shredding, cutting
into pieces, making paper dolls, making a montage, making
Mobius strips; it affords pasting on the wall; if affords coloring green or fingerpainting upon; if affords photocopying
and photocopying onto; if affords stapling to other pieces
of paper or clipping them into an album; if affords bookmarking; it affords wrapping a package or making into an
envelope; it affords cleaning the gaps between teeth; it affords cleaning up after your dog or wiping your hands; it
affords burning; it affords filtering particulates; it quite
simply affords all the possible things I can do with it. My
behavior is virtually unconstrained by its affordances. [3]
Since affordances cannot constrain behavior, agency must
act as an interpretive mechanism to explain meaning and
choice. To resolve the issue of agency, Edward Reed externalizes the meaning of interaction in the same way the
Dourish presents meaning through embodiment. He does
this by equating affordances with values of the world and
meaning with information [16] Through interaction, or the

use of affordances, individuals gain information or meaning

from the world.
To explain how individuals chose between the affordances,
Reed refers to selective retention theory [7], a process similar to natural selection. As individuals interact with the
world they learn which affordances provide the most meaning through a process of variation and selection. In this
goal-oriented approach to interaction, intention is measured
through the selection of an affordance based on prior experience and perception of the most useful affordance.
Keeping Reeds theory of agency in mind, the ability to
invite behavior should be reconsidered. Since affordances
are simply opportunities they will not all invite interaction.
Given a piece of paper it is impossible to act on all of its
affordances. Likewise, the consideration of agency does not
guarantee invitation [21]. Rather, an affordance invites behavior if the individual perceives its importance in achieving a goal.
WHAT IS LEFT OF EMBODIED INTERACTION?

Thus far, embodied interaction has been critiqued for extending the definition of affordances without respecting
Gibsons original definition [8] and Reeds theory of agency [16] has been considered. By dismissing the ability of
affordances to describe context, it should be asked, where
does this leave embodied interaction?
In the dismissal of Norman and Gaver, affordances lost
their ability to invite behavior through presence or perception. While perception alone could be construed as type of
interaction with the environment it does not result in an
observable interaction that would be needed to describe
context.
The consideration of agency, however, is problematic for
embodied interaction. When individuals gain the ability to
choose between affordances, the meaning of interaction
cannot be given a fixed definition. While still embodied,
meaning cannot be ascribed without being defined by the
individual. In any application, this reliance on the individual would be cumbersome, and should not be considered as a
viable solution.
Without a reliable way to have the individual assign meaning to their interactions we must rely on the situated perspective provided in the definition of situated actions. In
this configuration, the application designer must presume
the meaning of interaction. To preserve agency and respect
the dynamic quality of context [4], this situated perspective
on interaction must not assume it is correct. While the situated perspective may address the general meaning of interaction with the environment, the possibility always exists
that the interpretation of meaning could be wrong.
Consider a person who has the opportunity to interact with
a pencil and a piece of paper. While interaction with either
of these objects can be detected, the meaning of interaction

cannot be determined. Consider now that we now that person is a novel writer. From a situated perspective, where
personal history is considered, we can make an assumption
about the meaning of interaction, that the person is writing
or planning a novel. With the inclusion of agency, however,
we cannot be sure that this assumption is correct. Having
the opportunity to fold it, write a poem or note on it means
that attention of the application designer must turn towards
recognizing whether their assumptions are correct. or making an assumption and reconfiguring functionality in the
periphery.
The issue of situated interactions can be seen in the actions
of Microsoft Words feature, Clippy [20]. While it knew
that you were interacting with a virtual piece of paper, it
did not have the ability to assume any meaning from simple
interaction. Instead it would make a situated assumption
based on continued interaction. When an assumption was
made, it would appear and ask a question, It looks like
your writing a letter. Would you like help? In doing so it
respected the individuals ability to confirm or deny the
meaning of interaction, but it did not consider that its assumption would probably be wrong. Instead it concluded
that its assumption was correct, leading to frustration on the
part of the user.
CONCLUSION

This paper has critiqued the use and definition of affordances in Paul Dourishs definition of embodied interaction. While he is correct that context is described through
interaction, he does not consider agency. This means that
meaning cannot be understood as a static property of interaction. Instead, the individual determines meaning.
This paper has also described how affordances and the perception of affordances can be quantified. This allows for a
qualitative and quantitative view of affordances and means
that affordances can be described and represented in the
computer. Ultimately, this strengthens Dourishs use of
affordances as a way of describing peoples perception of
the world. Thus, interaction goes a long way towards making true context-awareness possible for applications designers.
While seeing embodied interaction as a viable direction for
computational interpretation and negotiation of context,
this paper addresses a critical hole in the definition of affordances embedded in Dourishs description of embodied
interaction. The authors cited incorrectly expand upon Gibsons original definition of affordances, and do not account
for an individuals freedom of choice in the environment.
While this proposal should not be considered a final or holistic solution to contextual representation it serves as a
way to create context-aware applications. This relies on
situated observations of context, and the recognition that
assumptions will not always be correct. The solution is not
ideal for application designers, but considering the re-

strictions placed on embodied interaction, it may need to

serve as a solution until a viable alternative can be proposed.
FUTURE WORK

While this paper has presented an example of situated interaction in Microsoft Word, this example does not engage
the physical environment. Future work, therefore, will test
the idea of situated interactions through the conceptual
analysis of a physical space. The space will be chosen
based on the rigor behind its layout, with ideal candidates
designed to encourage certain behaviors.
REFERENCES

1. Abowd, Gregory D., Anind K. Dey, Peter J. Brown,

Nigel Davies, Mark Smith, and Pete Steggles. "Towards
a better understanding of context and contextawareness." In Handheld and ubiquitous computing, pp.
304-307. Springer Berlin Heidelberg, 1999.
2. Chemero, Anthony. "An outline of a theory of affordances." Ecological psychology 15, no. 2 (2003):
181-195.
3. Cutting, James E. "Two ecological perspectives: Gibson
vs. Shaw and Turvey." The American Journal of Psychology (1982): 199-222.
4. Dourish, Paul. "What we talk about when we talk about
context." Personal and ubiquitous computing 8, no. 1
(2004): 19-30.
5. Dourish, Paul. 2001. Where the action is: the foundations of embodied interaction. Cambridge, Mass: MIT
Press.
6. Gaver, William W. "Technology affordances." In Proceedings of the SIGCHI conference on Human factors
in computing systems, pp. 79-84. ACM, 1991.
7. Ghiselin, Michael T. The triumph of the Darwinian
method. Courier Dover Publications, 2003.
8. Gibson, James J. 1979. The ecological approach to visual perception. Boston: Houghton Mifflin.
9. McCarthy, John, and Patrick Hayes. Some philosophical
problems from the standpoint of artificial intelligence.
USA: Stanford University, 1968.
10. Norman, Donald A. The psychology of everyday things.
Basic books, 1988.
11. Norman, Donald A. "Affordance, conventions, and design." interactions 6, no. 3 (1999): 38-43.
12. Schilit, Bill, Norman Adams, and Roy Want. "Contextaware computing applications." In Mobile Computing
Systems and Applications, 1994. WMCSA 1994. First
Workshop on, pp. 85-90. IEEE, 1994.
13. Turner, Phil. "Affordance as context." Interacting with
computers 17, no. 6 (2005): 787-800.
14. Kaptelinin, Victor, and Bonnie Nardi. "Affordances in
HCI: toward a mediated action perspective." In Proceedings of the SIGCHI Conference on Human Factors
in Computing Systems, pp. 967-976. ACM, 2012.
15. Konczak, Jrgen, Harry J. Meeuwsen, and Marie E.
Cress. "Changing affordances in stair climbing: the per-

ception of maximum climbability in young and older

adults." Journal of Experimental Psychology: Human
Perception and Performance 18, no. 3 (1992): 691.
16. Reed, Edward. 1996. Encountering the world: toward an
ecological psychology. New York: Oxford University
Press.
17. Suchman, Lucille Alice. 1987. Plans and situated actions: the problem of human-machine communication.
Cambridge [Cambridgeshire]: Cambridge University
Press.
18. Weiser, Mark. "The computer for the 21st century."
Scientific american 265, no. 3 (1991): 94-104.
19. Warren, William H. "Perceiving affordances: visual
guidance of stair climbing." Journal of Experimental
Psychology: Human Perception and Performance 10,
no. 5 (1984): 683.
20. Whitworth, Brian. "Polite computing." Behaviour &
Information Technology 24, no. 5 (2005): 353-363.
21. Withagen, Rob, Harjo J. de Poel, Duarte Arajo, and
Gert-Jan Pepping. "Affordances can invite behavior:
Reconsidering the relationship between affordances and
agency." New Ideas in Psychology 30, no. 2 (2012):
250-258.