From Cognitive Amplifiers To Cognitive Prostheses

INTERDISCIPLINARY SCIENCE REVIEWS, Vol. 37 No.
1, March, 2012, 418
From Cognitive Amplifiers to Cognitive Prostheses: Understandings of the Material Basis of Cognition in Visual Analytics
Published by Maney Publishing (c) IOM Communications Ltd
Richard Arias-Hernandez, Tera M Green and Brian Fisher

School of Interactive Arts and Technology, Simon Fraser University
The most salient ways in which data visualization and interactive techniques have been understood as the material basis of cognition in the emergent field of visual analytics are discussed. Three main dominant understandings have captured the imagination and theorizations of researchers and technicians in this field: data visualizations and interactive techniques as cognitive amplifiers, cognitive prostheses, and cognitive mediators. The analysis of this treatment of materiality in cognition provides an up-to-date report on whether remarks on the situated character of cognition and the active role of human agents have, in effect, been incorporated in this field or not. We argue that even though visual analytic researchers have incorporated some of the ideas of situated cognition and tempered traditional arguments of information processing from cognitive science, understandings of the role of materiality in cognition are still marked by universalisms and ascriptions of exacerbated agency to visual representations.
keywords cognitive studies of technology, distributed cognition, action theory, visual analytics
Introduction
Defined as the science of analytical reasoning supported by interactive visual interfaces (Thomas and Cook 2005), visual analytics (VA) is a post-9/11 technoscientific endeavour. Its research agenda was drafted in 2004 by visualization researchers from academia, industry, and government laboratories on the request of the US Department of Homeland Security. The initial objective of VA was to address analytical challenges in intelligence analysis and public safety posed by the threat of terrorism and natural disasters. As dramatically illustrated by the events of 9/11, intelligence analysis failed to connect the dots among dispersed yet available information,
Institute of Materials, Minerals and Mining 2012 Published by Maney on behalf of the Institute DOI 10.1179/0308018812Z.0000000001
FROM COGNITIVE AMPLIFIERS TO COGNITIVE PROSTHESES
which, if appropriately considered, could have prevented those attacks in the US. In retrospect, this weakness in intelligence analysis was explained in terms of the human and technological inability to cope with the information overload produced by enormous amounts of constantly generated, intelligence-related data. During the first years of development of VA, researchers concentrated their efforts in the development of computationally based, interactive, visual representations as well as on the development of several assumptions about analytical cognition during visual analysis (Pirolli and Card 2005). The foundational assumption in VA is that, in contrast to computation alone, visualization can harness the human minds innate visual intelligence to gain novel insights into situations characterized by complex data that may contain uncertainty in fact or relevance to the problem, or time and location of occurrence. Two basic types of computer-based artefacts have been the focus of design and development in VA to harness this visual intelligence: visual representations and interaction techniques. By developing computer-based visual representations of data that are designed to trigger instinctive perceptual responses, VA is expected to create systems that allow the human analyst to offload cognitive processes, such as comparing values or detecting outliers, into the more basic pre-cognitive, sensorial system (Card et al. 1999). For example, an analyst may visually mark outliers with different colours, shapes or sizes. This kind of offloading of cognition emphasizes the use of visualizations as external representations, which are coupled with the analysts internal knowledge representations. By designing interactions with these visualizations, VA aims at extending these systems to allow the analyst to offload some other smaller cognitive processes, such as juggling the memory of multiple search results or annotating the strength of a relationship between two visualized concepts into what are essentially outputs of the motor system. There are, for example, the outputs of saving the state of an analytical process or creating annotations about the reasoning process. These artefacts of interaction can be both realtime, such as mousing behaviours and menu interaction, as well as asynchronous, such as when an analyst creates notes for future use by herself or by other analysts who will use the visualization afterwards. In this study, we discuss how these two kinds of artefacts, visualizations and interaction techniques, have been understood as the material basis of cognition in VA. Our main argument is that these understandings still rely on traditional cognitive models that focus on universalisms and assumptions of humans as passive cognitive agents while downplaying recent models that emphasize the situatedness and active role of humans in tight couplings with external representations-processes. For example, mainstream cognitive models in VA assume a homogeneous, universal, and rather passive cognitive agent that couples with visual analytic systems to augment her cognitive skills (Liu and Stasko 2010). More recently, however, research in the cognition of visual and spatial representations has challenged this view showing that cognitive agents display a wide variety of cognitive behaviours when interacting with visual analytic systems (Keehner et al. 2008). This expression of heterogeneous, situated and active cognitive agents that negotiate the terms of their coupling to external representations problematizes the mainstream cognitive models used in VA. Our discussion centres around what we consider are the three more dominant understandings of the materiality of cognition in VA: cognitive amplifiers, cognitive prostheses, and cognitive
INTERDISCIPLINARY SCIENCE REVIEWS, Vol. 37 No. 1, March, 2012
RICHARD ARIAS-HERNANDEZ, TERA M GREEN and BRIAN FISHER
mediators. The first two understandings are related to theories of distributed and external cognition and the third understanding is related to Activity Theory. The following sections present these understandings in details, highlight similarities and differences, and discuss their implications for changing notions of cognition. We conclude this study with an analysis of potential implications for revitalizing these understandings with more assertive notions of human agency and situatedness.
Living in a material world: theories and understandings of the material basis of cognition in InfoVis and visual analytics
VA is an outgrowth of two closely related disciplines: information visualization (InfoVis) and scientific visualization (Wong and Thomas 2004), and most of its current understandings about the material basis of cognition have been taken from psychology and cognitive paradigms adopted in these fields. In this section, we review the uptake of two of these paradigms: distributed cognition and Activity Theory, and the way they have shaped understandings of the materiality of cognition in VA.
Cognitive amplifiers: InfoVis and visual analytics take on distributed cognition

The distributed cognition paradigm has been developed in cognitive sciences (Norman 1993; Zhang and Norman 1994; Larkin and Simon 1987) and in cognitive anthropology (Lave 1988; Hutchins 1995) since the mid 1980s. Its main tenet is that human cognition is a phenomenon that is not bounded to internal states located in the human brain. On the contrary, human cognitive processes emerge as a result of the tight coupling of internal (i.e. mental) representations-processes with external representations-processes (Scaife and Rogers 1996). The external representations are normally embodied in physical form in a material object, or artefact (e.g. maps, charts, compasses, and stars used in naval navigation). The external processes can be implemented either by artefacts (e.g. calculators, computers) or by other human agents. According to this paradigm, an understanding of cognition cannot be limited to internal, mental representations but it should also include interactions among humans, and interactions between humans and artefacts. In other words, cognition is not the exclusive result of processes occurring inside the human brain but the result of interactions across individuals of a social group and across individuals and their physical environment. Within the theories and models of distributed cognition, two perspectives can be distinguished. First, consider cognitive anthropological perspectives of distributed cognition, such as Hutchins and Laves. These perspectives take as their unit of analysis a sociotechnical system, in which cognitive processes, such as memory or problem solving, are distributed in a network of collaborative agents (humans and nonhumans). These approaches emphasize as much the interaction between social agents as they emphasize the interaction between individuals and artefacts. In other words, these analyses do not privilege individuals over social groups, or individuals over artefacts. It is rather the system as a holistic unit that captures the attention of the researchers. Methodologically, these approaches are also characterized by their use of ethnography.
A second perspective, also known as external cognition (Scaife and Rogers 1996), corresponds to cognitive science approaches to distributed cognition. Donald Norman, J. H. Laird, and Herbert Simon, among other cognitive scientists, have heralded this approach. Central to external cognition theories is the concept proposed by Norman of cognitive artefacts (Norman 1993). Cognitive artefacts are external aids invented by humans for the purpose of overcoming limits imposed in memory, thought, and reasoning. One class of cognitive artefacts is graphical inventions of all sorts, such as maps or data charts. Before the 1990s, traditional cognitive modelling relied heavily on internal representations, such as mental models (Johnson-Laird 1986), to explain cognitive phenomena. However, external cognition approaches have been instrumental in shifting the attention of researchers towards the interaction, or coupling, between internal and external representations (i.e. cognitive artefacts) in the production of naturalistic cognitive behaviour. Different from cognitive anthropological perspectives, external cognition perspectives do not emphasize social interaction or cultural influences in their models. The unit of analysis is not a sociotechnical system including social structures, culture, individuals and tools, but rather a more limited interaction between the internal representations of one individual and external representations (i.e. cognitive artefacts). External cognitive approaches also tend to assume universal properties of individuals and artefacts, instead of accounting for situated variability. In other words, the system in external cognition approaches is a more reductionist interaction between a standard individual interacting with one more or less deterministic artefact. External cognitive perspectives tend to privilege individuals over social groups, and cognitive artefacts over individuals. From these two approaches to distributed cognition, the external cognition approach developed by cognitive scientists has been the one adopted by InfoVis and VA communities, and the one approach that has mostly shaped their research and design agenda (Dykes et al. 2005, Campbell et al. 2008, Liu et al. 2008). This perspective has also been the strongest influence to sediment understandings of interactive, visual representations as cognitive amplifiers. For example, the very definition of InfoVis states that its purpose is the use of computer-supported, interactive, visual representations of abstract data to amplify cognition (our italics, Card et al. 1999, 7). This strong emphasis on interpreting interactive, visual representations, as amplifiers of cognition was grounded by Card et al. in their seminal book Readings in Information Visualisation: Using Vision to Think (1999). Card et al. presented external cognition as the way in which internal and external representations and processing weave together in thought (Card et al. 1999, 1). They argued that external, visual representations are aids that extend human cognitive abilities (e.g. working memory and computation) and amplify cognitive performance. To illustrate this point, Card et al. compared how long it takes for individuals to multiply a pair of two-digit numbers in their heads versus doing the multiplication in longhand using pencil and paper. They argued that the reduction in time by a factor of five produced in the pencilpaper condition results from the aid of the visual representation. Card et al. argued that one of the challenges of mental multiplication is to hold partial results in working memory, until they can be used. By using the visual representation, individuals are not required to hold partial results in their memory. Instead, an individual would simply write those numbers down. Additionally, the visual structure produced by neatly aligning numbers
in columns speeds up the retrieval of information when partial results are required. In so doing, the visual representation extends a persons working memory (Card et al. 1999, 2). Moreover, the authors claimed that some artefacts such as slide rules, calculators or computers, which outperform the unaided mind in computational processing, could easily externalize the necessary calculations or transformations required by complex multiplications, augmenting an individuals computational abilities. Further, Card et al. argued that visualizations could amplify cognition in six major ways: (1) by increasing the memory and processing resources available to users, (2) by reducing the search for information, (3) by using visual representations to enhance the detection of patterns, (4) by enabling perceptual inference operations, (5) by using perceptual attention mechanisms for monitoring, and (6) by encoding information in a manipulable medium (Card et al. 1999, 16). This argument was consistent with research done by Larkin and Simon (1987), Norman (1993), Kirsh and Maglio (1994), Kirsh (1995), and Scaife and Rogers (1996). Indeed, Larkin and Simons (1987) research on using diagrams to solve physics problems argued that diagrams help an analyst with searching, recognition, and inference in three ways: (1) by grouping together information that is used together, large amounts of search are avoided, (2) by grouping data about an object, visualizations can avoid symbolic labels, leading to reductions in search and working memory, and (3) by offloading cognitive inferences done symbolically into inferences done with simple perceptual operations. In a similar vein, Normans influential book Things that make us smart (Norman 1993) advanced the concept of cognitive artefacts and their use in experiential and reflective thought. He also supported the argument of using visualizations as aids to expand the working memory available for solving a problem. According to Norman, visual aids facilitate the reflective process by acting as external memory storage, allowing deeper chains of reasoning over longer periods of time than it would not be possible without the aids (Norman 1993, 25). Kirshs research on the Tetris game (1995) also highlighted the potential of using interactive visual interfaces to offload mental processes such as rotations of spatial icons, which are computationally expensive, into perceptual and motor processes with are faster and more accurate. Kirsh (1995, 65) argued that using visualspatial arrangements (1) reduce memory load of tasks; (2) reduce the number of steps involved in internal computation; and (3) simplify visual search and categorization. Scaife and Rogers (1996) followed on Normans argument by supporting the idea that visual representations could be used as external repositories of information to free up working memory for other aspects of thinking. Internal representations when using visual aids, they argued, do not need to be complete but rather can allow working memory to keep a minimum amount of information about the physical location of relevant points on the visual display and pointers to other equally important locations. In other words, the representation is distributed in internal dynamic pointers and external and detailed, visual information (Scaife and Rogers 1996, Pylyshyn 2003). Since VA is an outgrowth of InfoVis, understandings of the role of interactive, visual representations as cognitive amplifiers were also adopted in VA, and these ideas have become mainstream within the discipline. The most representative example of this point is found in the research agenda for VA drafted in 2004 and synthesized in the book Illuminating the Path (Thomas and Cook 2005). This research agenda starts by endorsing Card
et al.s perspective of interactive, visual representations of data as amplifiers of cognition (Thomas and Cook, 2005, 4647), and throughout the agenda there are consistent interpretations of visual aids as augmenting cognition or as amplifying cognition, such as:
Information visualisation amplifies human cognitive capabilities. (Thomas and Cook 2005, 46) Augmenting the cognitive reasoning process with perceptual reasoning through visual representations permits the analytical reasoning process to become faster and more focused. (Thomas and Cook 2005, 69) Visual representations are the equivalent of power tools for analytical reasoning. (Thomas and Cook 2005, 70)
This treatment of interactive, visual representations as cognitive amplifiers in InfoVis and VA has implications. Here we discuss two. First, even though theories of external cognition emphasize the system created by the coupling of internal and external representations as the unit of analysis, the overemphasis of external representations in VA and InfoVis has systematically underplayed internal representations and interaction/coupling in understandings of external cognition (Yi et al. 2007, Liu, Nersessian, and Stasko 2008). The treatment of other components of the distributed cognitive system, such as interaction techniques and mental models, has remained underdeveloped in these fields. More importantly, by neglecting internal cognitive processes, interaction and coupling with external visual representations has resulted in the taken-for-granted assumption of agency granted to visualizations in processes of amplification of cognition. In other words, assuming that visualizations per se are cognitive amplifiers automatically grants them agency and control of the cognitive process over other components of the system internal representations, motor interaction, and coupling of internal/external representations. This philosophical view assumes that, with visualizations, humans automatically and unconsciously, offload internal mental processes into external perceptual and motor processes. However, this may not be so. More nuanced theories of external cognition have challenged this view by arguing that human agency drives motor processes that make changes to external representations in order to save computations in the head (Kirsh 1995); conventions of visual representation have to be internalized for their effective use (Hegarty 2011); and internal, cognitive control of interactive behaviour minimizes effort by using a least effort combination of the mechanisms available to it rather than automatically expending perceptual-motor efforts to conserve lesser amounts of cognitive efforts (Gray and Fu 2004; Gray et al. 2006). What these cognitive scientists have highlighted is that practitioners of InfoVis and VA have privileged external representations as amplifiers of cognition, when research has shown that there is little reason to think that (1) people automatically offload internal cognitive processes into visualizations or that (2) visual representations have a privileged status in relation to other components of external cognition (e.g. mental models, perception, motor interaction, coupling of internal/external representations, etc.). A second aspect is that cognitive science researchers in external cognition generally acknowledge that cognitive artefacts in fact do not amplify cognition (Larkin and Simon 1987, Norman 1993, Hutchins 1995, Kirsh 1995, Gray and Fu 2004). For them, the amplification of cognition is a misleading notion.
10
Rather, what happens in distributed cognitive systems is that cognitive tasks change once external aids are incorporated (i.e. cognitive tasks change to perceptual and/or motor tasks). When we use cognitive artefacts such as pencil and paper, maps or other visual representations, our internal working memory does not increase, nor does the speed of our internal computational processes. Rather, it is the nature of the task at hand that changes. For example, in the case of the previous illustration of the multiplication of a pair of two-digit numbers, mental multiplication usually involves the following cognitive tasks: computation of intermediary numbers, holding partial results and carriers in working memory, and retrieval of partial results when needed for further calculation (e.g. addition). When a cognitive artefact such as pencil and paper is used, the same goal is achieved by performing different tasks: perception and motor systems (i.e. vision and use of the hand) are used to write down the numbers on the paper and align the numbers in columns; long term memory is used to recall results of multiplications of single digits; perception-motor systems are used to write down temporary results and carriers; visual perception is used to retrieve partial results from the paper; and mental computation is used for simple addition of single digits. As this example illustrates, it is not that working memory amplifies when pencil and paper are used, rather unaided cognitive tasks that required intense use of working memory get simplified or are no longer necessary, once external aids are available. Hutchins puts it this way: it is not that the cognitive properties of individual minds get amplified; it is rather the effective coupling of individuals and cognitive artefacts that produces cognitive properties of a system that are more effective than those produced by the individual minds alone (Hutchins 1995). Even though this more reserved view of augmentation of cognition seems to be widespread in the cognitive science community, practitioners of InfoVis and VA still seem to subscribe to the idea that interactive visualizations in fact amplify cognition (Liu et al. 2008).
Cognitive prosthetics in visual analytics: from universal cognitive abilities to individual differences
Another understanding of the role of interactive visualizations in cognition in InfoVis and VA is that of cognitive prostheses. Understandings of interactive visualizations as cognitive prostheses are directly derived from those of cognitive amplifiers. An interactive visualization is a cognitive prosthesis when it is designed and used to supplement a cognitive limitation or to reconstitute a cognitive ability that is considered to be impaired. As is the case with prosthetic devices that are designed to replace or supplement body parts, a cognitive prosthesis requires a standard of normal faculties, skills or abilities to compensate for. For example a standard of vision of 20/20, which could be measured to discriminate among individuals state of vision, is used to determine if someones vision is falling behind what should be expected to be the ideal condition. The purpose of glasses, a prosthetic device, is then to correct for this limitation so that artefact-aided vision gets closer to the ideal standard. Similarly, for cognitive prostheses to compensate for limitations in cognitive faculties or abilities, some standards have to be determined so that gaps in individual capabilities can be found and diagnosed. As problematic as these standards or normalizations are in their work in society as regulatory devices (Foucault 1995, Bowker and Star 1999), they are central to the work of modern science. Cognitive science, for example, has advanced several metrics to determine standards for cognitive skills and
11
abilities that are hard wired in humans and are assumed to be universally distributed. Some examples of these are: preattentive processing rates for form, colour, motion and spatial position (Ware 2004); Millers famous magic number: 7 2 chunks that can be held in working memory (Miller 1956); subitizing ranges that capture a feeling of immediately knowing how many items lie within a visual scene (Kaufman et al., 1949); and the small number of visual elements or objects (46) that can be indexed (FINSTed) in early stages of visual perception (Pylyshyn 1989). Characteristic of these standards is the tendency to configure a prototypical human cognitive agent, very similar to the standard individual that it is assumed in understandings of cognitive amplifiers. Another strategy to determine differences in cognitive abilities is using statistical analyses to discriminate among groups that respond to cognitive task better (i.e. faster or more accurate) than others. For example, using tests, such as Guay and McDaniels visualization of viewpoints (Keehner et al. 2008), for scoring spatial ability the ability to mentally store and manipulate visualspatial representations accurately. These sorts of tests allow researchers to score individuals and statistically group them as having high or low abilities. Understandings of computer-generated visual representations of data being used as cognitive prostheses are less common in InfoVis than those of cognitive amplifiers, and are normally found under the label of assistive technologies (Lamming et al. 1994). For example, Nugent et al. (2008) and Alm et al. (2007) have developed stationary and mobile devices that include visual and interactive interfaces as test beds to validate their use as memory aids for people suffering from mild dementia. Czerswinski et al. (2004) redesigned the Windows XP bar to support automatic generation of visual reminders to resume tasks after interruptions and reduce the difficulty of multitasking in some users. In addition to these examples, InfoVis and humancomputer interaction (HCI) has had a long tradition of developing customizable visual interfaces for dyslexic people, shortsighted people, and the elderly. In VA, a few applications have also been designed as cognitive prostheses. Arnott et al. (2006), for example, designed interactive galaxy views in the user interface of an augmentative and alternative communication (AAC) system as a cognitive aid for non-speaking individuals. The visualization was designed to support individuals with severe communicative impairments in retrieving information from a biographical AAC database. Another line of development in VA that intersects with understanding of cognitive prostheses has clearly come from the domain of intelligence analysis. Researchers of intelligence analysis (Heuer 1999, Cooper 2005) have proposed a set of analytic pathologies that could be overcome with the help of VA tools. Among these pathologies are: ignoring high-profit documents due to data overload (Patterson et al. 2001); confirmation bias (Johnston 2005); oversensitivity to consistency, persistence of impressions based on discredited evidence, illusory correlations (Heuer 1999); and not considering levels of trust and certainty in sources of data (Johnston 2005). Several of the visual analytic tools developed under the sponsorship of US Department of Homeland Security have been designed to provide computer-support to analysts in order to overcome these so-called analytic pathologies. For example, the Scalable Reasoning System (Pike et al. 2009) is a visual analytic tool that provides features to ascribe levels of certainty to pieces of evidence in order to discriminate them during processes of generation and falsification
12
of hypotheses. Some other tools have been tailored to specific methods that help overcome some other cognitive biases. For example, the ACH tool developed by the Palo Alto Research Center provides a quick visual aid to help on the analysis of competing hypotheses and in the reduction of confirmation bias.1 More recently, Green et al. (2009) have proposed using artificial intelligence (AI) agents in mixed-initiative VA systems to help human analysts neutralize confirmation bias and provide an automatic recommendation system for consideration of alternative hypotheses:
Through observation of what interests the human collaborator, the computer can suggest information that is semantically related, but up to this point, has not been considered ... The human is free to explore or to reject suggestions. But by making the effort in ensuring that nothing important is overlooked, the computer works to counteract human cognitive biases that can interfere with complete mental modelling. (Green et al. 2009, 3)
Most of these understandings of artefacts as cognitive prostheses in InfoVis and VA have relied on similar assumptions to those discussed in the section of cognitive amplifiers. The reason for this, as mentioned before, is that understandings of cognitive prostheses are corollary to theories of augmentation of cognition. Therefore, a similarly strong emphasis has also been placed in granting agency to the visual artefacts. As long as an adequate prosthesis is matched to an individual with the cognitive limitation it is supposed to overcome, then the artefact should do the trick. These assumptions again are not fully supported. Contradictory evidence about using interactive visual aids as cognitive prostheses has also come from the cognitive sciences. For example, studying individual differences in spatial abilities, Keehner et al. (2008) tested the effect of the spatial ability of a group of individuals on the effects of external, interactive visualizations. They wanted to test if provision of an interactive visualization could work as a cognitive prosthesis for low-spatial individuals who had poor internal abilities to mentally rotate and imagine cross-sections of 3D objects. Their results showed that individuals with low-spatial ability got lower accuracy scores in cross-section tasks than individuals with highspatial ability, but all of these scores were independent of participants use or not of interactive visualizations as aids. In other words, there was no evidence that, at least in this particular case, the external visualization could act as a cognitive prosthetic that could compensate for low internal mental abilities.
Cognitive mediators: activity theory in InfoVis and visual analytics

Activity Theory has its roots in Leontievs (1978) psychological analyses of activity and Vygotskys (1978) work on child development, from which he derived his insights on the zone of proximal development. In the 1980s, Engestrm reformulated the theory and it is in large part this strand of Activity Theory that has been drawn upon by humancomputer interaction (Nardi, 1996), and more recently by information visualization and VA researchers (Zhao et al., 2008; Gotz and Zhou, 2009). The aim of Activity Theory is to understand the unity of consciousness and activity (Kaptelinin and Nardi, 2009). Activity Theory is as much a theory of consciousness as it is a theory of activity. Human experience is seen as mediated through signs and
13
artefacts within an activity system, yet the system is given motive, intentionality and meaning through consciousness, which is uniquely human. In Activity Theory, the main unit of analysis is the activity itself. An activity is composed of a subject, object, actions, operations and mediators. A subject is an individual or social group engaged in an activity. An object is what motivates the activity; the motif or need that shapes action, and it is intrinsic to the subject. An object, in Activity Theory, is not to be confused with a physical artefact. An object is better understood as a goal or, as Kaptelinin and Nardi (2009) suggest, the object of the game. Actions are processes undertaken in pursuit of the object or goal within the activity. Actions are conscious formulations aimed at fulfilling the goal. For example, where the goal may be to get a coffee from your favourite caf, an action might be to find your coat to go outside or ask if someone else is going for coffee and offer to buy them a coffee in exchange for going to the caf for you. Operations are processes that have become routine and are unconscious. For example, for an experienced driver, changing gears is a routine process. Operations are related to actions, and in fact, the same process of changing gears is an action to a novice driver, i.e. the inexperienced driver is conscious of the goal and the process required to shift from one gear to another. Mediators are external elements that facilitate actions or operations. Mediators can be human or nonhuman. Both, people and tools can act as mediators of actions and operations. Activity Theory holds that the elements of an activity system are not fixed but can change as conditions change. The object is not immutable and can be transformed in the course of an activity, yet it is not the moment-by-moment dynamism of situated actions (Kaptelinin and Nardi 2009). The notion of mediation through artefacts is a central concept in Activity Theory (Kaptelinin and Nardi 2009). From an InfoViz or VA perspective informed by this theory, interactive visualizations are understood as mediators of activity, and interactions of users subjects with interactive visualizations are understood as actions or operations in a computermediated activity (Kaptelinin and Nardi 2009). Several InfoVis researchers have incorporated Activity Theory or some conceptual elements of it in their research and design practices. For example, Zhao et al. (2008) have mapped movement of population in geovisualizations using activity as the transformational dimension, along with time and space. Matthews (2006) also incorporated Activity Theory in her design of glanceable peripheral displays. She conceptualized peripheral displays as mediators of operations (i.e. unconscious processes). These displays had visual information about secondary tasks during multitasking and could remain at the unconscious level of peripheral vision monitoring (Matthews 2006). The mediation of activity by tools has also been used in VA to capture actions (i.e. conscious processes) that are mediated by humancomputer interactions with VA tools. These actions are captured in time sequences and feed visual representations to provide a history of analytical reasoning (Gotz and Zhou 2009; Shrinivasan and van Wijk, 2009). Green et al. (2011) have also proposed to understand VA tools as cognitive mediators of analytical activity to guide the interactive design and evaluation of VA tools. There are several differences and similarities between Activity Theory and external cognition approaches in their treatment of physical artefacts, such as interactive visualizations. Both approaches include material artefacts as part of cognitive activity and ascribe a vital role to them. In other words, they
14
make technology visible in cognitive theory. However, their treatment of materiality in cognition is different. For Activity Theory, the locus of cognition is human activity, not the isolated individual mind or the material artefacts. On the other hand, for external cognition, the locus of cognition is the systemic interactions between internal and external representations. Systemic cognition (i.e. augmented cognition) is an emergent property that cannot be ascribed to only one of the components of the system, whether human or nonhuman. This conceptual difference impacts on the level of attention and importance granted to humans, activity, artefacts, and interactions. Activity Theory privileges human activity over artefacts as the source of cognition, treating external artefacts as tools that humans choose to use to support or mediate their activities. Activity Theory only treats humans as cognitive agents. External cognition, on the other hand, does not grant any special privileges to any of the components of the system, but rather to the system as a whole as the source of systemic cognition, privileging the system over human and nonhuman components. External cognition distinguishes between systemic cognition and cognition that occurs at the components level as two different levels of cognitive phenomena. By insisting in symmetric treatment of the components of the system, external cognition theories cannot avoid granting artefacts status as cognitive agents. Another related difference that Activity Theory places on the treatment of artefacts is related to agency. As mentioned in the section of cognitive amplifiers, InfoViz and VA take on external cognition shifted ascriptions of agency from the system user-interacting-with-a-visualization towards the computer-based visual artefact, and skewed ascriptions of agency from the system to the artefact. On the contrary, Activity Theory has a clear and strong position that only ascribes agency to human subjects. The theorys understanding of agency is grounded on the idea of a purposeful subject that orients her activity towards the satisfaction of her needs. The theory clarifies that only living things have needs, and excludes any non-living thing, or artefact, from any sort of claim over agency. From an action theory perspective, a computer-based, interactive visualization cannot be considered as having agency of any sort. Activity Theory also re-introduces the element of social interaction that is present in anthropological perspectives of distributed cognition but that is absent from external cognition perspectives adopted in InfoVis and VA. Since activity is mediated not only by tools but also by other humans (e.g. Vygotskys zone of proximal development), Activity Theory gets closer to anthropological versions of distributed cognition, such as Hutchins, in their inclusion of social groups and culture as constitutive of consciousness and human activity. This is an element that has been underplayed in adaptations of Activity Theory in InfoVis and VA, which tend to focus on partial aspects of the theory, but one that could be relevant for information visualizations scenarios that emphasize collaboration, such as collaborative VA or uses of visualizations in CSCW. It also promises to be a rich and useful approach for what is being currently developed as knowledge visualization (Wang and Mu, 2009, Zhang et al. 2010).
Can we bring agency and situatedness to visual analytics?

So far in this paper, we have discussed three understandings of the material basis of cognition in VA: cognitive amplifiers, cognitive prostheses and
15
cognitive mediators. We highlighted how these understandings usually have found their way first in information visualization and how they have found their way into VA, owing to the big overlap between InfoVis and VA communities. We also explored how some defining theories from cognitive science provided the theoretical underpinnings of these understandings. However, it is necessary to emphasize here again that the translation of these theories from cognitive science to information visualization and VA has not been without modification, adaptation, appropriations and opportunistic selection of partial aspects of these theories. One example that we offered was the treatment of augmentation of cognition by InfoVis and VA, something that is not heralded by cognitive scientists who work on external cognitive theories (Gray et al. 2006, Hegarty 2011). This incomplete translation has had as a consequence an overemphasis in ascriptions of agency to external, visual representations and interactive techniques, and a consequent underplay of the active, human cognitive agent. This explains why most of the developments in this field have focused on the design of computer-based visualizations and interactive techniques rather than on studies of human cognition in its interaction with visual analytic tools. Another aspect that we have found underplayed in VA and that is evident in anthropological perspectives of distributed cognition, action theory, and some cognitive science theories, is situatedness. The concept of situatedness, when applied to cognition (Suchman 1987), refers to the opportunistic character of cognitive activity as produced by the resources of the immediate situation. Situated perspectives of cognition are characterized for their acknowledgement of the particular, their emphasis on culture as shaper of cognition, and their resistance for developing universal models of cognition or theories that abstract cognition from phenomenological experience. Two examples of these perspectives are situated cognition (Suchman 1987) and embodied cognition (Dourish 2001). Cognitive anthropological perspectives of distributed cognition (Hutchins 1995, Lave 1988), Activity Theory (Kaptelinin and Nardi 2009), and some contemporary versions of extended cognition (Gray and Fu 2004, Gray et al. 2006) also share the same interest for situatedness of cognition. However, this is one of the aspects that have been lost in the translation of these theories to InfoVis and VA, which tend to overemphasize universal models of human cognition and standard generic conceptions of the human agent. Even adaptations of Activity Theory, such as those of Gotz and Zhou (2009), seem to ignore the aspects of situated activity and ascribe to a fixed set of universal actions to characterize the interactions of users with visual analytic tools. It would seem that information visualization and visual analytic communities cannot deal with the contingency that characterizes human activity and persist in pursuing the quest for standard models and universals to characterize the role of visualizations and interaction in cognition. The abundant ecology of materiality in cognition in VA, as discussed here,2 shows how central artefacts are in the mainstream cognitive narratives in this field. From this initial exploration and analysis, it is evident for us that some elements of these narratives of the material basis of cognition have departed from traditional cognitive theories that demarcated binaries between mind and brain and that excluded technology from playing an important role in cognition. However, it is also evident that these same narratives have also been subsumed by deeply grounded universalisms in order to fit into the dominant scientific orders of information visualization and VA. Some
16
consequences of this have been the intense search for a universal prototype or standards of human cognitive abilities and the quest for a holy grail of general principles for the design of information visualization and VA technology. Another consequence of this commitment to universalism is that other cognitive-related approaches that emphasize situatedness and human agency have received much less attention, development and visibility in the information visualization and the VA community. This may partially explain why some theories from sociology and anthropology, which could be relevant to the understanding of the relations between human cognition and the material world (e.g. symbolic interactionism, situated cognition, and actornetwork theory) have not permeated in the theoretical frameworks currently used in VA. It may also explain why the whole field of VA has not been able to attract more social scientists to help shape its research and design agenda. Published by Maney Publishing (c) IOM Communications Ltd
Notes
1
Analysis of Competing Hypothesis, Palo Alto Research Center, accessed November 18, 2011, http://www2.parc.com/istl/projects/ach/ach. html. We need to clarify that there are more understandings that inhabit the imaginations of researchers
and designers in VA, such as: cognitive affordances, cognitive scaffolds, cognitive extractors, and cognitive armatures. However, we decided to limit this paper only to amplifiers, prostheses, and mediators, and expand our analysis later to encompass other minor understandings.
Bibliography
Alm, Norman, Arlene Astell, Gary Gowans, Richard Dye, Maggie Ellis, Phillip Vaughan, and Alan F. Newell. 2007. An Interactive Entertainment System Usable by Elderly People with Dementia. Proc. 4th Int. Conf. on Universal Access in HumanComputer Interaction. Arnott, John L., Leishi Zhang, David OMara, Norman A. Alm, and Andrew Taylor. 2006. Information Visualisation in the User Interface for Augmentative and Alternative Communication. Technology & Disability 18(3): 147161. Bowker, Georey C., and Susan L. Star. 1999. Sorting Things Out: Classication and Its Consequences. Cambridge: The MIT Press. Campbell, Bruce D., Huseyin O. Mete, Tom Furness, Suzanne Weghosrt, and Zelda Zabinsky. 2008. Emergency Response Planning and Training through Interactive Simulation and Visualisation with Decision Support. IEEE Proc. Conference on Technologies for Homeland Security 176180. Card, Stuart K., Jock D. Mackinlay, and Ben Schneiderman, eds. 1999. Readings in Information Visualisation: Using Vision to Think. San Francisco: Morgan Kaufmann Publishers Inc. Cooper, Jerey R. 2005. Curing Analytic Pathologies: Pathways to Improved Intelligence Analysis. Washington, DC: Center for the Study of Intelligence. Czerwinski, Mary, Eric Horvitz, and Susan Wilhite. 2004. A Diary Study of Task Switching and Interruptions. Proc. CHI 2004 6(1): 175182. Dourish, Paul. 2001. Where the Action Is: The Foundations of Embodied Interaction. Cambridge: The MIT Press. Dykes, Jason, Alan M. MacEachren, and Menno-Jan Kraak. 2005. Advancing Geovisualisation. In Exploring Geovisualisation, edited by Jason Dykes, Alan M. MacEachren and Menno-Jan Kraak, 693703. Kidlington: Elsevier Ltd. Foucault, Michel. 1995. Discipline and Punish: The Birth of the Prison. New York: Vintage. Gotz, David and Michelle X. Zhou. 2009. Characterizing Users Visual Analytic Activity for Insight Provenance. Information Visualisation 8(1): 4255. Gray, Wayne D., and Wai-Tat Fu. 2004. Soft Constraints in Interactive Behavior: The Case of Ignoring Perfect Knowledge In-The-World for Imperfect Knowledge In-The-Head. Cognitive Science 28(3): 359 382. Gray, Wayne D., Chris R. Sims, Wai-Tai Fu, Michael J. Schoelles. 2006. The Soft Constraints Hypothesis: A Rational Analysis Approach to Resource Allocation for Interactive Behavior. Psychological Review 113(3): 461482.
17
Green, Tera M., William Ribarsky, and Brian Fisher. 2009. Building and Applying a Human Cognition Model for Visual Analytics. Information Visualisation 8(1), 113. Green, Tera M., Ron Wakkary, and Richard Arias-Hernandez. 2011. Expanding the Scope: Interaction Design Perspectives for Visual Analytics. Proc. of HICSS-44. Computer Society Press. Hegarty, Mary. 2011. The Cognitive Science of VisualSpatial Displays: Implications for Design. Topics in Cognitive Science 3(3), 446474. Heuer, Richards J. 1999. The Psychology of Intelligence Analysis. Washington, DC: Center for the Study of Intelligence. Hutchins, Edwin. 1995. Cognition in the Wild. Cambridge: The MIT Press. Janney, Richard W. 1997. The Prosthesis as Partner: Pragmatics and the HumanComputer Interface. Proc. IEEE International Conference on Cognitive Technology: 8287. Johnson-Laird, Philip N. 1986. Mental Models. Cambridge: Harvard University Press. Johnston, Rob. 2005. Analytic Culture in the US Intelligence Community: an Ethnographic Study. Washington, DC: Center for the Study of Intelligence. Kaufman, E.L., M.W. Lord, T.W., Reese, and J. Volkmann. 1949. The Discrimination of Visual Number. The American Journal of Psychology 62(4): 498525. Kaptelinin, Victor and Bonnie A. Nardi. 2009. Acting with Technology: Activity Theory and Interaction Design. Cambridge: The MIT Press. Keehner, Madeleine, Mary Hegarty, Cheryl Cohen, Peter Khooshabeh, and Daniel R. Montello. 2008. Spatial Reasoning With External Visualisations: What Matters Is What You See, Not Whether You Interact. Cognitive Science 32(7): 10991132. Kirsh, David and Paul Maglio. 1994. On Distinguishing Epistemic from Pragmatic Actions. Cognitive Science 18(4), 513549. Kirsh, David. 1995. The Intelligent Use of Space. Articial Intelligence 73(12): 3168. Lamming, Mik, Peter Brown, Kathleen Carter, Margery Eldridge, Mike Flynn, Giord Louie, Peter Robinson, and Abigail Sellen. 1994. The Design of a Human Memory Prosthesis. The Computer Journal 37(3): 153163 Larkin, Jill H., and Herbert A. Simon. 1987. Why a Diagram Is (Sometimes) Worth Ten Thousand Words. Cognitive Science 11(1): 65100. Lave, Jean. 1988. Cognition in Practice: Mind, Mathematics and Culture in Everyday Life. Cambridge: Cambridge University Press. Leontiev, A. 1978. Activity, Consciousness, and Personality. Englewood Clis, N.J.: Prentice Hall. Liu, Zhicheng, Nancy J. Nersessian, and John T. Stasko. 2008. Distributed Cognition as a Theoretical Framework for Information Visualisation. IEEE Transactions on Visualisation and Computer Graphics 14(6): 11731180. Liu, Zhicheng and John T. Stasko. 2010. Mental Models, Visual Reasoning and Interaction in Information Visualisation: A Top-down Perspective. IEEE Transactions on Visualisation and Computer Graphics 16(6): 9991008. Matthews, Tara. 2006. Designing and Evaluating Glanceable Peripheral Displays. Proc. DIS 2006: 343345. Miller, George A. 1956. The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information. Psychological Review 63(2): 8197. Nardi, Bonnie A. (ed.). 1996. Context and Consciousness: Activity Theory and Human-computer Interaction. Cambridge, MA: MIT Press. Norman, Donald A. 1993. Things That Makes Us Smart: Defending Human Attributes in the Age of the Machine. New York: AddisonWesley. Nugent, Chris D., Richard J. Davies, Mark P. Donnelly, Josef Hallberg, Mossaab Hariz, David Craig, Franka Meiland, Ferial Moelaert, Johan E. Bengtsson, Stefan Savenstedt, Maurice Mulvenna, and Rose-Marie Dres. 2008. The Development of Personalized Cognitive Prosthetics. Proc. IEEE EMBS Conference: 787790. Patterson, E. S., E. M. Roth, and D. D. Woods. 2001. Predicting Vulnerabilities in Computer-Supported Inferential Analysis under Data Overload. Cognition, Technology & Work 3(4): 224237. Pike, William, Joe Bruce, Bob Baddeley, Daniel Best, Lyndsey Franklin, Richard May, Douglas Rice, Rick Riensche, and Katarina Younkin. 2009. The Scalable Reasoning System: Lightweight Visualisation for Distributed Analytics. Information Visualisation 8(1): 7184. Pirolli, Peter and Stuart K. Card. 2005. The Sensemaking Process and Leverage Points for Analyst Technology as Identied Through Cognitive Task Analysis. Proceedings of International Conference on Intelligence Analysis. McLean, VA: Mitre.
18
Pylyshyn, Zenon W. 2003. Return of the mental image: are there really pictures in the brain?. Trends in Cognitive Science 7(3): 113118. Pylyshyn, Zenon W. 1989. The Role of Location Indexes in Spatial Perception: A Sketch of the FINST Spatial-Index Model. Cognition 32(1): 6597. Thomas, James J. and Kristin A. Cook, eds. 2005. Illuminating the Path: the Research and Development Agenda for Visual Analytics. Los Alamitos: IEEE. Scaife, Mike and Yvonne Rogers. 1996. External Cognition: How Do Graphical Representations Work? International Journal of Human-Computer Studies 45(2): 185213. Shrinivasan, Yedendra B., and Jarke J. van Wk. 2009. Supporting Exploration Awareness in Information Visualisation. IEEE Computer Graphics and Applications 29(5): 3443. Suchman, Lucy. 1987. Plans and Situated Actions. Cambridge, Mass.: Cambridge University Press. Vygotsky, L. 1978. Mind in Society: The Development of Higher Psychological Processes. Cambridge, Mass.: Harvard University Press. Wang, Xiao-yue, and Yan Mu. 2009. Visualisation Based on Concept Maps: An Ecient Way to Knowledge Sharing and Knowledge Discovery in E-science Environment. Proc. IEEE FSKD 2009: 144147 Ware, Colin. 2004. Information Visualisation: Perception for Design. San Francisco: Morgan Kaufmann. Wong, Pak Chung and Jim Thomas. 2004. Visual Analytics. IEEE Computer Graphics and Applications 24(5): 2021. Yi, Ji Soo, Youn ah Kang, John T. Stasko. 2007. Toward a Deeper Understanding of the Role of Interaction in Information Visualisation. IEEE Transactions on Visualisation and Computer Graphics 13(6): 12241231. Zhang, Jianping, Da Zhong, and Jiahua Zhang. 2010. Knowledge Visualisation: An Eective Way of Improving Learning. Proc. IEEE 2nd Int. Workshop on Education Technology and Computer Science 1: 598601. Zhang, Jiaje and Norman, Donald A. 1994. Representations in Distributed Cognitive Tasks. Cognitive Science 18 1): 87122. Zhao, J., Forer, P. and Harvey, A.S. 2008. Activities, ringmaps and geovisualisation of large human movement elds. Information Visualisation 7: 198209.
Notes on contributors
Richard Arias-Hernandez is a postdoctoral fellow at the School of Interactive Arts and Technology, Simon Fraser University. His current research focuses on bridging symbolic interactionism and applied psycholinguistics to solve coordination problems in collaborative visual analytics. Correspondence to: Richard Arias-Hernandez: ariasher@sfu.ca. Tera Marie Green is a PhD student and graduate research assistant at the School of Interactive Arts and Technology, Simon Fraser University. Her research involves applied cognitive science for visual interfaces. Brian Fisher is an associate professor of Interactive Arts and Technology and Cognitive Science at Simon Fraser University and associate director of the Media and Graphics Interdisciplinary Centre at University of British Columbia.

From Cognitive Amplifiers To Cognitive Prostheses - Understandings of The Material Basis of Cognition in Visual Analytics

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

From Cognitive Amplifiers To Cognitive Prostheses - Understandings of The Material Basis of Cognition in Visual Analytics

Загружено:

Авторское право:

Доступные форматы

INTERDISCIPLINARY SCIENCE REVIEWS, Vol. 37 No.

1, March, 2012, 418

Richard Arias-Hernandez, Tera M Green and Brian Fisher

Published by Maney Publishing (c) IOM Communications Ltd

RICHARD ARIAS-HERNANDEZ, TERA M GREEN and BRIAN FISHER

Published by Maney Publishing (c) IOM Communications Ltd

Cognitive amplifiers: InfoVis and visual analytics take on distributed cognition