About the author

Clarisse Sieckenius de Souza

Clarisse Sieckenius de Souza is Full Professor of Computer Science at the Departamento de Informtica, PUC-Rio, where she has also served as the director of Graduate Studies Program for two terms (2003-2005 and 2007-2009). In 2010, Clarisse was a co-winner of the prestigious ACM SIGDOC Rigo Award (see pictures and more in SERG Website). In 1996 she founded SERG (the Semiotic Engineeri...

25. Semiotics
and Human-Computer Interaction
by Clarisse Sieckenius de Souza. How to cite in your report.

This chapter covers why and how Semiotics can help advance some of the major goals of Human-Computer Interaction (HCI) and be useful when designing interactive products. It begins with brief definitions and explanations of a few central concepts in Semiotics. This is followed by a discussion of harder challenges involved in bringing Semiotics into the domain of HCI research and the consequences of viewing computers as media. Following Semiotic Engineering concepts, which we have been developing and using for two decades now, we then revisit computer-mediated communication in view of 21st century literacy issues. First, we show that basic computing skills exhibited by contemporary users are in fact semiotic engineering abilities of the same sort as required from professional designers. Then we show how these skills can leverage an individuals participation in a variety of social processes. In conclusion, the chapter presents our personal answer to the question that most readers certainly have in mind: So, whats in it for me?

Table of contents
25.1 Introduction

25.2 Some preliminary definitions 25.2.1 Saussure 25.2.2 Peirce 25.2.3 Culture, signification and communication 25.3 Semiotics and HCI: Two Disciplines, Two Cultures 25.4 Computers as Media 25.4.1 Semiotic Engineering 25.4.2 Contrast between MS Word and the OO Writer interfaces 25.5 A closer look at computer-mediated communication 25.5.1 Some contemporary examples 25.5.1.1 Example 1: Programming via parameter setting 25.5.1.2 Example 2: Programming via macro recording 25.5.1.3 Example 3: Programming on wheels 25.5.2 Going a few steps further 25.6 So, what's in it for me? 25.7 Acknowledgments 25.8 Commentary by Alan Blackwell 25.9 User-contributed notes 25.10 References

25.1 Introduction
Semiotics and HCI have more in common than is usually acknowledged in either side of the cultural divide that has been keeping them apart. Researchers and professional practitioners working with these disciplines have different interests and perspectives when selecting, applying and building knowledge. As a result, mutual understanding has not only been rare, but usually perceived as more cost than benefit. This chapter is an essay on why and how Semiotics can help advance some of the major goals in HCI. It begins with a definition of Semiotics and a brief explanation of a few central concepts that will be used throughout the chapter. Next, it discusses some deeper challenges of using Semiotics in the context of HCI research. Then it explores the notion of computers as media, the hallmark of all semiotic approaches to HCI proposed to date. It highlights the interdisciplinary work of pioneers like Mihai Nadin and Peter Bgh Andersen and ends with a description of Semiotic Engineering, a comprehensive semiotic theory of HCI which we have been developing and using at SERG, the Semiotic Engineering Research Group in Rio de Janeiro, since 1990. In subsequent sections, we take a closer look at computermediated communication in view of contemporary opinions that having basic

programming skills is as important for citizens of the 21st century as reading, writing and counting have been in the 20th century and before. Some examples of basic computing literacy skills exhibited by contemporary users are framed as semiotic engineering abilities of the same sort as required from professional HCI designers. This helps us to show why computers are indeed the most pervasive media used by contemporary societies, and it points at computing literacy issues with which HCI is not only necessarily involved, but also in which it has clear vested interests. The chapter then provides our answer to the provocative question implied by its title, proposed by the editors of the Encyclopedia of HCI: So, whats in it for me? The answer, I hope, will attract more attention to this fascinating discipline in times when computers so obviously coalesce and transform societys means of communication and participation.

25.2 Some preliminary definitions

Semiotics is the study of signs. Although strictly correct, this definition is not helpful for those who do not know what signs are and how they can be studied. So, let us begin with additional definitions. The selected material will also give the reader a flavor of the cultural divide between Semiotics and HCI as expressed by terminology and conceptual framing. In the Encyclopedia of Semiotics (Bouissac 1998), the entry for sign explores numerous variations in the way philosophers and semioticians have defined the central object of interest in this discipline over the centuries. Here are two of them:

a. In the tradition inaugurated by Ferdinand de Saussure: [...] the sign can be

understood as a correlation of differences. (Bouissac 1998: p. 573)

b. In the tradition inaugurated by Charles Sanders Peirce: [...] the defining

characteristic of signs is their capacity to determine additional signs [in the mind]. (Bouissac 1998: p. 574) Saussure and Peirce are the founding fathers of Semiotics. Although they were contemporaries (Saussure died in 1913 and Peirce in 1914), they lived in different continents (Europe and North America, respectively) and followed completely independent paths. Saussure was a linguist, interested in a formal characterization of natural languages. Peirce was a logician, interested in meaning and knowledge discovery processes. The definitions above are both very true to their respective theories, but they are also mysterious for non-semioticians. I picked them up

because they also illustrate rhetorical differences between disciplines, which we have to understand in order to reap the benefits of interdisciplinary research.

25.2.1 Saussure
Figure 25.1: Oppositions in form that do not correspond to oppositions in meaning. Copyright: See page copyright notice. No higher resolution available Keeping in mind that Saussure (Saussure 1972) was focused almost exclusively on linguistic signs, the correlates with which he defined the sign were: an acoustic image (the signifiant, or signifier); and a concept (the signifi, or signified). The former is a physical entity, whereas the latter is an abstract one. Differences determine what constitutes a sign at various levels and dimensions. Just as a brief illustration, take the word privacy in English. Its acoustic image in spoken language is not unique: some say |'prvsi|, while others say |'pravsi|. Although in English the long vowel |a| is different from the short vowel || (e. g. |bat| and | bt| are not the same word), this difference is neutralized in words like privacy: The concept, no matter the pronunciation, is the same. So, what is a sign? In phonology, |a| and || are signs because they have significant differences. One is associated with the concept of long vowel, and the other with short vowel. However, in syntax or semantics |'prvsi| and |'pravsi| are not distinct signs. Difference, as the reader can infer, is a very powerful principle in meaning making and sense making. In natural languages, differences and oppositions are systemic. That is, certain opposing combinations are recurrent and meaningful in the language, whereas others arent. Therefore, a language can be described as an inventory of signs and sign combinations (or structures), for which the correlation between the signifiant and the signifi can be formally established by systemic differences. Here is a simple example of how this principle can be applied in the context of HCI.

Figure 25.2: Well-formed sign structures in HCI. Copyright: See page copyright notice. No higher resolution available

Figure 25.3: Ill-formed sign structures in HCI. Copyright: See page copyright notice. No higher resolution available In Figure 25.1, we see different renditions of a familiar window control element. The signifiants constitute different signs in the equivalent of HCI phonology. Nevertheless, in HCI syntax and semantics, the whole collection corresponds to a single lexical item (a single sign for close window). In Figure 25.2 and Figure 25.3, we see the parallel with Saussures patterns of structures. In one case, the close window sign is combined with other signs that are systemically used in conjunction with it, whereas in the other case it is combined with signs that should never cooccur with it. Therefore, in this interface language, only one of the two structures is well-formed. Note that signifiant/signifi correlations in interface languages are much weaker than in natural languages. Semiotic correlations are programmed into software. Compared to natural language, this can be a blessing and a curse. The blessing is that in interface languages, unlike in natural languages, the conceptual correlate of signs can (and must) be fully specified and instantiated by a mechanical process. Thus, we always have access to the origin of meaning. The curse is that mechanical instantiation of the signifis as correlates of interface signifiants is arbitrarily

programmed by human minds. That is, nothing can prevent a mischievous programmer from specifying that |x| causes a window to freeze, rather than close. This is not the case in natural languages: neither you nor I can decide and establish that from now on privacy means something else in English. The correlation between parts of signs in formal grammatical descriptions of natural languages is not established by individuals. This example gives us a chance to mention another fundamental dichotomy in Saussurean theory, that between langue (language) and parole (speech). This significant opposition has been posed to account for the fact that individual variations in language use do not affect the system. It is easy to collect numerous examples of individual linguistic behavior contradicting the general rules and conventions of a given language system (e. g. we make occasional grammatical mistakes as we speak, which may be due to heavy cognitive loads). However, individual and sporadic variations do not change or affect the language system. It is only if and when such variations make their way into a formal abstract and general sign system, extending over large spans of space and time, that they actually cause a change in langue. Other variations, within smaller spatial and temporal coverage, correspond to changes in parole. Saussures interest was in a theory of langue, his formal theory of signs. This refers us to a silent Cartesian tradition running under the surface of non-Cartesian concepts, such as the determining forces of social structures and conventions. The Cartesian gene in the theory is the assumption that linguists can describe langue and that individual minds can have access to supra-individual concepts and formulate them in abstract, free of psychological, social, spatial and temporal contingencies. In other words, a linguists description of langue objects should be warranted by this individuals ability to capture abstract signifis that exist independently of his or her mind and to preserve them from contamination of individual interpretation when building formal theories of language. Part of Descartes much more complex philosophy, not discussed here, was that individual minds could have access to ultimate meanings by exercising methodical doubt and analysis (Descartes 2004). Eventually, Descartes believed, the mind would reach an unquestionable state, some primary cognition that constitutes the true meaning of the object(s) under consideration. In the remainder of this chapter, whenever we use the word Cartesian we are referring to this specific aspect of

Descartes theory, an important one for discussing alternative semiotic theories and their application to HCI.

25.2.2 Peirce
Peirce (Peirce 1992), as already mentioned, was interested in logic and the origin of meaning. His theory can actually be framed as a long and deeply elaborate rebuttal to the Cartesian canons we just mentioned (Santaella 2004). He defined signs as a three-part structure consisting of a representation (the representamen), an object and a mediating element (the interpretant) that binds the object to its representation in somebodys mind. In other words, unless there is a mental mediation, nothing is a representation of anything. This will probably remind the reader of the proverbial saying that meaning is in the mind of the beholder. But, as the next paragraphs will show, the theory goes far beyond intuitive interpretations of the old adage. The definitional feature of signs mentioned in (b) on page 2 that they can generate other signs refers to the fact that, according to Peirce, the mediating interpretant, just like the representamen, stands for the object that it binds to a particular representation. In other words, the interpretant is a second-order representation of the object itself. The sign has, in itself, a generative recursive seed that produces other signs, ad infinitum. For example, let us go back to the word privacy. This is a sign where |'prvsi| is the representamen and something outthere-in-the-world is its object because in some minds (like yours and mine) there is an interpretant that binds them together. Is this interpretant unique? Is it the same for everyone? Is it the same for you or me throughout our entire lives? The answer is no. From the very moment that we inspect this interpretant (i. e. that we take it to mean the object out-there-in-the-world which we call |'pravsi|), it becomes a second-order representation of its very object. A newly generated interpretant (e. g. an explanation or further elaboration referring to the object) is instantiated, which generates yet another sign instantiation as soon as our mind engages in further interpretation. In order to illustrate how these concepts can be used in a practical HCI context description, let us go back to the |x| window control (see Figure 25.4). In a Peircean account of this sign, |x| is the representamen of the whole sign, whereas the specified computational procedure that causes a well-determined subsequent state of the system (in which the Street video window is no longer visible) is its object. All of the interesting things have to do with the interpretant. What

mediating sign (which will generate other signs) is binding representation and object together? To answer the question, we would have to inspect minds, like the programmers and the users, for example. An exhaustive account for all interpretants all possible reasons and related facts that can cause |x| to stand for whatever it is that makes the window disappear when we click on it in programmers and users minds, over large spans of time and space, is beyond the ability of individual minds, caught in the contingencies of momentary interpretation. We can already see how Peirces theory differs in focus and essence from Saussures, and also why, in the context of this chapter, a semiotic theorys positioning relative to certain aspects of Cartesian tradition can extensively determine its fate in HCI. For instance, the tension between a Peircean perspective on meaning and the algorithmic nature of interface sign interpretation and generation processes in computer systems cannot be missed. Neither can the apparently smooth compatibility between Saussures theory and the basic tenets of formal language and automata theory in Computer Science (Hopcroft and Ullman 1979) be ignored. Nevertheless, there is more promise in Peircean theory for HCI than meets the eye. Let us briefly introduce a couple of additional concepts and definitions.

Figure 25.4: How simple interface signs can generate unlimited semiosis.

Copyright: See page copyright

notice. No higher resolution available The first is Peirces most widely known and used definition of sign: A sign [...] is something which stands to somebody for something in some respect or capacity. It [...] creates in the mind of that person an equivalent sign, or perhaps a more developed sign. (Buchler 1955: p. 99) Note that the mediation of the interpretant is done in some respect or capacity (i. e. for some potentially partial and arbitrary reason). Thus, although most users will tend to generate an interpretation for |x| that is at least superficially appropriate for the immediate context of interaction sketched in Figure 25.4, subsequently generated signs are likely to be very different. For example, thanks to some unfortunate previous experience, one user may accept that |x| means close the window, but develop further interpretations according to which the presence of |x| in that particular window is actually the result of negligent programming. She may believe that if she clicks on it and closes that window, the browser will freeze. In her mind, to prevent the unfortunate situation, the |x| control should not be active, especially because the Close & Return button is there to control the closing operation safely. Some other user, however, thanks to luckier interactions, may develop the interpretation that the |x| control and the Close & Return button mean exactly the same thing. The difference, he may think, is a practical one. The generic close a window can be communicated with different signs: to click on |x| or press some specific key combination. The latter can be faster than the point-and-click alternative. Thus, whereas advanced users may interpret the presence of |x| as meaning that the use of keyboard input to control the window on Figure 25.4 is allowed, novice users may miss this signification of | x| altogether and interpret that to close the window they must click on the Close & Return button. The next steps in each users theories of what |x| means in different situations are virtually impossible to predict since they depend substantially on the users experience with software, their level of technical awareness, and virtually all other meanings that populate their minds. The second concept of Peircean theory that this example helps to illustrate is the process of abductive reasoning (also known as hypothetical reasoning or abduction), which underlies according to the theory all meaning-making activities. Very simply put, this kind of reasoning consists of generating hypotheses to explain (i. e. to interpret) significant elements of reality around us. Once a hypothesis is generated, it is tested against ready-to-hand evidence. If the evidence contradicts the hypothesis, a new one is generated and tested. If not, the hypothesis is accepted as a general principle capable of binding not only the tested instances of

representations to the tested objects, but also and more importantly future instances and yet-to-be-met objects as well. The principle is true until further evidence is found that calls for revision of beliefs. This indefinitely long process of sign generation is named semiosis. It can, for example, explain Carroll and Rossons observations reported in an influential early work in HCI, The paradox of the active user (Carroll & Rosson 1987). The authors remark that users hastily assemble ad hoc theories of how the system works and refuse to learn more even though they hold important misconceptions about the interface language semantics, which can lead to inefficient or ineffective interaction. A Peircean account of this phenomenon tells us that, in the process of abductive reasoning, users come upon mediating signs that do not require or motivate further abductive effort. They are content with their inferencing and fixate a generative belief (i. e. an interpretant that will be used in other sense-making situations). Even if from a programmers point of view the users reasoning and interpretations are totally wrong, for as long as they work, the users believe that they are right. The meaning of work, and not so much the meaning of interface elements themselves, is the key to understanding interaction patterns and choices in this case. The third and last additional concept that we will discuss with respect to Peircean Semiotics is that of a/the true meaning of signs. If human minds can engage in widely (and wildly) different interpretive directions, how is it that two people can understand each other, or at least truly believe that they do so? And, much more importantly, is there such a thing as a/the true meaning of signs? In a Peircean perspective, meaning emerges in a constantly changing chain of equivalent [...] or perhaps more developed signs. Subsequent interpretants are connected, corrected, adjusted, expanded, stabilized and destabilized by the insistence of the real. Without going into the very complex details of Peirces theory of truth and true meanings, it is worth highlighting a few points about his particular view of reality and realitys role in sign-making processes:

I.

Sign making (which for the purposes of this chapter is the same as sense making) is a natural disposition of the human species, just like flying is a natural disposition of birds. Humans cannot help but assign meaning to whatever they sense in the world around them. Therefore, reality is always mediated by signs.

II.

Because it is an abductive inferencing process, human instinctive sign making is fundamentally prone to error. We constantly make mistakes while

interpreting reality. However, our innate interpretive apparatus is such that it can correct interpretations in the presence of new evidence. This mode of mental operation constitutes the first and more essential step in any sort of human mental activity, from the mundane daily inferences about the freshness of products on market stalls to the scholarly debate about the validity of scientific theories.

III.

The principle that makes human knowledge and understanding gravitate towards truth is the insistence of reality, the inexorable massive set of evidence provided by the world around us. This is also an indication that, in this theory, truth is not the result of an individuals introspective activity (as Descartes would wish), but a collective ongoing semiotic process, in which abductive corrections made and expressed by other minds penetrate culture through social processes that eventually affect individual sense-making. Hence, as a social, cultural and historical process, human interpretation gravitates toward truth.

Back to HCI, these three points allow for a reinterpretation of heated controversies. Take for example Jared Spools criticism of User-Centered Design. In a CHI Panel in 2005 (Spool & Schaffer 2005), among other things, Spool claimed that beyond small teams with simple issues, formalized UCD doesnt seem to work and that UCD pretends to act like an engineering discipline (formalized methods that have repeatable results independent of practitioner), but actually behaves as if its a craft (dependent completely on skills and talents of practitioners with no repeatable results). (p. 1174) Spools position is a good example of how the insistence of reality is related to, but not the same as, repeatable results. A Peircean account of meaning tells us that predictions of how users will interpret interface signs must be taken very cautiously. When making sense about the natural and the cultural world, a users abductive reasoning paths will eventually be corrected and refined by the insistence of a massive volume of facts that penetrate the mind as signs contingently related to each other in some respect or capacity. Contingency the craft that lies at the opposite of repeatable results expected by Spool is actually the essence of human sense making. Computer encoded interface signs, however, are different. They are, and must be, uniquely defined and engineered into software artifacts that continually repeat exactly the same mechanical interpretation and generation for each and every

representation in the program. These representations are produced by human minds (the software designers and developers) and meant to affect other human minds (the users). The inexorable repetition of these representations in computer systems interfaces is ontologically very different from what Peirce refers to as the insistence of the real. Actually, computer interface signs are no more than the expression of a particular moment in someones (or some group of peoples) abductive reasoning path, which is as prone to interpretive error as any users abductions while trying to make sense of the interface. We thus see that Spools vigorous criticism of UCD echoes some of the Cartesian notions we discussed above. He, just like UCD adopters, is looking for repeatable semiosis independent of practitioner, meaning independent of mind, which should nevertheless be mentally achieved by an individuals use of rigorous methods of inspection and analysis. The interest of a semiotic perspective on this debate is to show that there are theoretical choices beyond Cartesianism, and that depending on whether the focus is on human semiosis or computer semiosis, some choices are better than others. For illustration, we comment on two alternative paths. Louis Hjelmslev (Hjelmslev 1961) followed Saussures ideas, but gave them a more axiomatic interpretation. In his view, signs could be studied as the correlation between any two autonomous systems. Neither one had to (although they might) be anchored in psychological reality. Charles Morris (Morris 1971), in turn, followed Peirces ideas, but gave them a radically behaviorist interpretation. In his view, signs should be defined in terms of stimulus and response. Influenced by Peirces notion that habit was a powerful mediating force in sign-making processes, Morris developed a semiotic model that explained all sign processes in terms of reactions caused by the presence of signs under certain conditions. Hjelmslev was the theorist chosen by Peter Bgh Andersen (Andersen 1990), when writing his Theory of Computer Semiotics, and Morris the one chosen by Heinz Zemanek (Zemanek 1966), when writing his pioneering Semiotics and Programming Languages.

25.2.3 Culture, signification and communication

Before we proceed to the next sections, let us revisit the saying that meaning is in the mind of the beholder for final remarks about conditions for human mutual understanding. We will borrow Umberto Ecos Theory of Semiotics (Eco 1976), which brings together Peircean and Saussurean elements in interesting ways. On the one hand, Eco adopts the notion that signs trigger an indefinitely long chain of

other signs in the mind (he speaks of unlimited semiosis). On the other, he defines signs as content-expression correlates. The apparently diverging perspectives are reconciled with the introduction of culture in two fundamental semiotic processes: signification and communication. Signification is a correlation between content and expression. Signification systems are collections of discrete units [...] or vast portions of discourse, provided that the correlation has been previously posited by a social convention. Communication is a process in which the possibilities provided by a signification system are exploited in order to physically produce expressions for many practical purposes. (Eco 1976: p. 4) Notice that Eco does not say that in communication one repeats the conventions encoded in signification systems. He says that communicators exploit the possibilities provided by a signification system. In other words, communication may include sign productions that deviate from the system, as well as others that dont. In fact, Ecos theory of sign production defines communication as a process that manipulates signs, considering, or disregarding, the existing [socially conventionalized] codes (Eco 1976: p. 152). The primacy of social conventions in communication, even when stepping out of existing signification systems, places culture at the center of Ecos Semiotics, which he actually characterizes as the logic of culture. Semiotics, in his view, is thus a comprehensive study of culturally-determined codes and sign productions at all levels of human experience.

25.3 Semiotics and HCI: Two Disciplines, Two Cultures

The work of Hirschheim, Klein and Lyytinen (Hirschheim et al 1995) on the conceptual and philosophical foundations of Information Systems Development (ISD) can further our understanding of why Semiotics and HCI have been keeping apart from each other. Following a still widely-accepted view that information systems are technical implementations of social systems, the authors trace the differences among ISD approaches, concepts, models, methods and even tools back to their proponents interpretation of reality. Building on Burrell and Morgans work in organizational analysis (Burrell & Morgan 1979), they list four paradigms in ISD research and professional practice. The paradigms are simplified world-view models produced by combining the end points of two axes: objectivity-subjectivity and order-conflict (see Table 25.1).

Paradigm

Choice between Objectivity and Subjectivit y

Choice between Order and Conflict

Assumptions about Reality

Functionalism

Objectivity

Order

There is an objective order in social reality, which can be known and described independently of subjective interpretation.

Social Relativism

Subjectivity

Order

Reality is a social construction, which is the result of interpretations determined by continuous cultural changes.

Radical Structuralism

Objectivity

Conflict

Reality is determined by objective super-structures that are in constant power conflicts.

Neohumanism

Subjectivity

Conflict

Reality is continually transformed by multiple objective and subjective epistemologies that coexist and contribute to historical evolution.

Table 25.1: Hirschheim and colleagues paradigms at the base of information systems development. Different assumptions about reality, shown in the last column of Table 25.1, can be aligned with distinctions between Saussurean and Peircean perspectives on meaning. This is not to say, however, that the same set of paradigms can be generated by semiotic theories alone, and much less that Saussure or Peirce (would) have embraced the principles of functionalism, social relativism, radical structuralism or neo-humanism. We must only observe the fact that paradigms that preclude subjectivity have higher compatibility with definitions of meaning that do not speak about mental mediation than those that include it, and vice versa. Likewise, speaking only of the paradigms where order is important, we should

observe that the defining characteristics of functionalism in Table 25.1 are more compatible with Cartesian canons than is the case with social-constructivism. Hirschheim and co-authors note that most of the work in ISD published until 1995 tacitly falls into the functionalist perspective, according to which subjective interpretation doesnt play a role when systems requirements are elicited and data models are built, for example. Testing hypotheses against empirical data can, in this view, prevent subjective intuitions, beliefs and aspirations from contaminating scientific knowledge of an objective reality. Consequently, this paradigm has long supported a feedback loop in which information and communication technologies (ICT) are viewed and produced as the impersonal (i. e. non-subjective) result of applying empirically-tested facts and principles to solve problems. Computer scientists have thus been increasingly led into empirical research practices looking for more facts and principles, but speaking very little of their interpretations and valued perspectives regarding what these facts and principles mean, why, and under what kinds of conditions. As a multi-disciplinary field, HCI encompasses almost as much diversity in terms of scientific traditions as the number of disciplines that can be listed among its contributors (e. g. Psychology, Anthropology, Sociology, Design, Ergonomics and Computer Science, to name but only a few). However, most of its success and prestige is associated to ever more sophisticated and pervasive kinds of interactive products coming from the ICT industry. In this context, predictive knowledge, based on sound empirical experimentation, is in high demand. The industry is avidly looking for tested facts and principles to guide its engineering processes and found the establishment of norms and standards. Thus, the pressure of industrial demands on HCI research has been reinforcing functional perspectives on science and restating the belief that scientists and technologists do their work impersonally, backed by statistical inferencing methods applied to objectively collected empirical data. Semiotics has been successfully applied in marketing and advertising (UmikerSebeok 1988; Floch 2001), for example, which suggests that HCI and Semiotics could work together to respond to industrial demands. Nonetheless, the power of opposing scientific traditions and practices has been keeping them on separate tracks. Semioticians of Peircean extraction, for instance, are interested in the rich variety of meanings that emerge in sign production and sign interpretation contexts. They may use extensive interpretive and hermeneutic methods to

generate in-depth understanding of situated meaning-related processes, before they engage in statistical inferencing methods leading to general signification facts and principles. In addition, they must not dismiss the fact that in order to establish the very object of their investigation (a sign or sign process) they need the mediation of their own minds. That is, in order to take anything as a sign (or evidence) of other signs or sign processes of interest, investigators must carry out interpretation. The process and product of their investigation will depend totally on other signs: their inferences, their validation criteria, their conclusions, and last but not least on the way they signify their knowledge discovery process and communicate it to the scientific community. The main criteria for scientific validity, which must be made apparent in this specific situation, is that newly discovered signs can: be framed into coherent inferential discourse that consistently relates to previously existing knowledge; correct, expand or innovate aspects of previous scientific findings; be exposed to the evaluation of other researchers in the scientific community; and produce, as a result of such exposure, the advancement of collective scientific knowledge. This view contrasts in important ways with the one that credits scientific validity only to research results obtained by an empirical testing of hypotheses and subsequent statistical inferencing. Firstly, although a semiotic view acknowledges the scientific validity of knowledge produced with these methods, it requires that the choice of hypotheses be accounted for as a signification process. Why have the hypotheses been chosen? Why are they significant per se, or more significant than other possible hypotheses? Secondly, and by consequence, it brings into the territory of legitimate scientific activity the very elaboration of candidate theories and hypotheses that will be tested in the course of empirical knowledge discovery processes. By means of rigorous interpretive methods, Semiotics is ready to inspect and validate the processes by which scientists assign meaning to (i. e. interpret) their hypotheses, their testing and their results, against the backdrop of theories that they propose to be building. These contrastive characteristics can explain why semiotic knowledge is usually viewed by the HCI community as speculative and subjective, an impractical philosophic discussion that cannot usefully inform the design and development of ICT products. By the same token, they can also explain why researchers with conscious or unconscious semiotic awareness tend to be skeptical of HCI theories built upon hypotheses whose selection is not carefully and consistently accounted

for. Moreover, motivated by the huge diversity of human meaning-assigning strategies, these researchers may be far more attracted to the possible variations of users interpretive behavior in view of even small contextual changes than to regularities found under highly controlled tests in laboratory. Given the above-mentioned differences in purpose, practice, beliefs and values, the most difficult obstacle for bringing together Semiotics and HCI is probably not the fact that Semiotics requires extensive learning of unfamiliar concepts and methods. This has always been the case in the multidisciplinary research practices that characterize HCI. The real challenge for the two disciplines seems to be how to combine different epistemologies and rise above scientific validity disputes. Only then can Semiotics seed progress in HCI. The rest of this chapter is my personal narrative of a successful case, the theoretical construction of Semiotic Engineering (de Souza 2005; de Souza and Leito 2009). I hope many others are on their way to correct and refine our thinking, partaking in the ongoing scientific semiosis.

25.4 Computers as Media

Today, viewing computers as media is like viewing cars as vehicles. Of course, computers enable, support and enrich individual communication, group communication, and mass communication. We can even put it the other way around: All contemporary media involve computation. But, how obvious was this connection when HCI emerged as a discipline? And, how obvious right now are the consequences of this view for current and future computation? Viewing computers as media is the hallmark of all semiotic approaches to computing and human-computer interaction. In 1988, Kammersgaard proposed that human-computer interaction could be characterized in substantially different ways (Kammersgaard 1988). One of them was called the media perspective, which involved not only the fact that computers could be used by people to communicate with each other, but also the fact that computer application programmers could communicate with users through systems interfaces. As the author acknowledges, the latter was influenced by illustrious predecessors:

I will not go into further detail about this last type of communication, except to

mention that Oberquelle, Kupka & Maass (1983) talk about delegation of communicating behaviour from the designer to the machine and then treat the

situation as seen from a dialogue partner perspective, whereas Andersen (1985) treats the designer as having the role of one sender in a collective of senders, who makes a contribution to each message sent through the medium.

-- Kammersgaard 1988: p. 356 Only a few years before, Winograd and Flores had published a notorious book, in which they proposed to shift the focus in computing from pursuing ever-increasing artificial cognitive capacities to supporting pervasive human communication and coordination processes (Winograd & Flores 1986). And in 1990, Andersen would publish the first comprehensive account of computers as media in his Theory of Computer Semiotics (Andersen 1990). From the late 1980s to the mid 1990s, a number of researchers, in different parts of the world, started thinking about how to articulate Semiotics and Computing. In 1988, in a 55-page chapter in Hartsons Advances in Human-Computer Interaction (Hartson 1988), Nadin (Nadin 1988a) explored the semiotic implications of interface design and evaluation. In the same year, he would also publish an article in Semiotica, about a semiotic paradigm to systems interface design (Nadin 1988b). In 1993, Andersen edited a book explicitly called The Computer as Medium (Andersen et al 1993), and we published our first paper on the semiotic engineering of human-computer interfaces (de Souza 1993). All of this early work explored the power of viewing computer programs and systems and especially their interfaces as signs of a very specific kind. They are designed and engineered according to the laws and limits of computation, but what they ultimately communicate is a message meant by humans (systems designers and developers) and to humans (systems users). This is the rationale for viewing human-computer interaction as computer-mediated human communication, the powerful idea underlying Winograd and Floress manifesto for a language-action perspective (LAP). LAP attracted many semiotically-inspired researchers in Computer Science (Communications of the ACM, May 2006). Most of them turned towards Organizational Semiotics (Liu 2000), which in general is closer to information systems development than to human-computer interaction design. In Brazil, however, a group of researchers has been exploring the use of Organizational Semiotics concepts in the design of human-computer interaction (Baranauskas et al 2003; Bonacin et al 2004; Neris et al 2011).

Semiotics was also implicit or explicit in the work of other researchers working in HCI. For example, Mullet and Sano (Mullet & Sano 1995) explored semiotic features of representations used in the design of visual interfaces. Their emphasis was on how to communicate the design intent to users through interface signs, an idea that was gaining momentum at the time. In fact, computer systems interfaces provided an excellent context to revisit the concept that there are languages of/in design. The chapter by Rheinfrank and Evenson in Winograds Bringing design to software (Winograd 1996) is a great illustration. The authors discuss design as communication, whether intentional or not, and advance the idea that when it is intentional, the use of design languages can increase the effectiveness of communication. As computers became more pervasive and took over a larger portion of social processes, the computers as media perspective gained the interest of researchers investigating the users response to intentional communication by service providers (Light 2001) and information appliances in general (Fogg 2002), for example. Foggs work, for one, developed into an elaborate study of persuasion through technology (Fogg 2003), which has been interestingly explored in the context of games and culture-sensitive communication (Khaled et al 2006). In a recent work, ONeill (O'Neill 2005) theorizes about interactive media in view of contemporary technologies that provide new mediation opportunities for human semiotic processes. Research on the convergence of Semiotics, interactive technology and literacy is also noteworthy. Marion Waltons work (Walton 2008) in South Africa provides insightful elements to the study of computers as media, enriched by empirical evidence collected in African school childrens encounters with computers and the Internet (Prinsloo & Walton 2008). Inspired by Andersens pioneering work in Computer Semiotics, we developed an extensive semiotic account of human-computer interaction, Semiotic Engineering (de Souza 2005). Having started as a semiotic approach to user interface language design (de Souza 1993), over the years Semiotic Engineering evolved into a semiotic theory of HCI with its own ontology and specifically-designed methods of investigation (de Souza and Leito 2009). This is the theory I will use in the rest of this chapter to illustrate, in practice, the kinds of connections existing between Semiotics and HCI.

25.4.1 Semiotic Engineering

Semiotic Engineering picked up the early view that human-computer interaction is in fact computer-mediated human communication. It then defined and articulated a number of fundamental concepts, their relations with each other and their implications not only for the development of the theory itself (the internal motivation), but as a contribution to the advancement of HCI (the external motivation). The most striking distinction proposed by Semiotic Engineering compared to other theories and conceptual frameworks in HCI (Carroll 2003) is to develop the early suggestion put forth by Kammersgaard (Kammersgaard 1988) and postulate that designers of interactive software are active participants in the communication that takes place through user interfaces. They communicate their design vision to users by means of interface signs like words, icons, graphical layout, sounds, and interface controls like buttons, links, and dropdown lists. Users unfold and interpret this message as they interact with the system. In other words, the communication between users and systems is in fact part of a metacommunication process; part of the communication process initiated by designers about how, when, where and why to communicate with the system they have designed. Since designers cannot be personally present when a user interacts with software, they have to represent themselves in the interface, using a specifically designed signification system, and subsequently tell the users what the software does, how it can be used, why, and so on. The choice of representations is actually wide. For instance, designers may represent themselves as humanoids (e. g. systems interfaces with human characteristics like affect and natural language abilities), as machines (e. g. systems interfaces with press buttons, slide controls, dials, and the like), or even as spaces (e. g. systems interfaces with virtual worlds that users explore to achieve various kinds of effects). Depending on the message that they have to communicate, some representations will work better than others. Humanoids, for example, are likely to communicate explanations and instructions more easily than virtual spaces. Machines, in turn, convey physical affordances more efficiently than the natural language discourse of humanoids. No matter their self-representation, through systems interfaces designers are telling users:

what they know about the users (who they are, what they know, what they wish or need to do, in which preferred ways, and why);

how they have responded to the users needs or aspirations (what system they have built and how it works) what values are encoded in their response (why and how does the system improve the users lives).

Here is a very simple sketched example of how the elements above are communicated by designers through the interface. We contrast Microsoft Word (Figure 25.5) and Open Office Writer (Figure 25.6) tools menu, showing how the designers message comes across.

Figure 25.5: A sketched version of Microsofts Word interface unfolding the Tools menu. Copyright: See page copyright notice. No higher resolution available

Figure 25.6: A sketched version of Open Offices Writer interface unfolding the Tools menu. Copyright: See page copyright notice. No higher resolution available The contrast of both designs shows that they communicate different views on users needs and opportunities. The following is a summary of the most striking differences between the two messages communicated through Microsoft Word and Open Office Writer interfaces.

25.4.2 Contrast between MS Word and the OO Writer interfaces

Copyright: See page copyright

notice. No higher resolution available

Microsoft Word MS Word Tools can be grouped into 4 categories (list items separated by a line): the first with tools related to document content; the second to document manipulation and authoring; the third to mail preparation; and the fourth to interface customization and extensions.

Copyright: See page copyright

Open Office Writer

notice. No higher resolution available

OO Writer Tools can be grouped into 7 categories (list items separated by a line): the first with tools related to document content; the second to document structure; the third to inclusion of media files; the fourth to bibliography handling; the fifth to mail preparation; the sixth to data field manipulation; and the seventh to interface customization and extension. The designers of both applications are telling their users that tools applying to document content are top of the list, and that Spelling and Grammar tools are the most important ones. By comparison, we (like users) get the message that MS Word provides more resources than OO Writer in this particular category. In fact, OO Writer tools are a subset of MS Words. Similarities in shortcuts are remarkable, suggesting that OO Writers designers expect their users to have (or to have had) experience with MS Word at some point in their lives.

Similarities extend also to other categories. In both applications, there are tools to handle mail preparation and to support customization and extensions. The tools in these categories, however, may have different names, and OO Writer provides an additional tool, compared to MS Word: it lets the users customize XML Filters Settings. At a closer look, we see that differences in names actually communicate a different perspective on the tools themselves. Whereas in MS Word names tend to express objects (with the exception of customize), in OO Writer the designers communication, when different from MS Words, alludes to agents: Mail Merge Wizard and Extension Manager. In these two cases, the communication is clearer (more precise) than in MS Word, which may achieve the effect of encouraging novice users into experimenting new features (since a wizard and a manager will be there to help them). Microsoft Word MS Words designers communicate very clearly their emphasis on group collaboration. They are telling users that document preparation is or can be a group activity, requiring such things as shared workspaces and online communication, in addition to tracking changes and comparing, merging and protecting documents. The communication of their design vision at this point clearly locates users in a computer-supported organizational context. Open Office Writer OO Writers designers, by contrast, do not talk about computer-supported collaboration. The communication of their design vision at this point portrays an individual user, working on the details of his/her document. The designers give special emphasis to document structure (numbering and notes) and bibliographic reference management, communicating that they want to support users in preparing complex extensive documents. By comparison with OO Writers designers communication, MS Words designers communicate their concern with usability in different ways. Five of their tools come with icon representations that help users identify their presence in toolbars (a more concise and more easily accessible interface control than menus). Three menu options have associated keyboard shortcuts, and four options unfold into submenus, rather than invoking dialog windows (communicated by ...). Together

these messages tell the users that MS Words designers are concerned with providing fast access and fast learning support for their users. In contrast, OO Writers designers favor longer dialogs (...), which actually is in line with longer processes that can also be expected from interaction with a wizard and a manager. Furthermore, by communicating that they expect their users to need a bibliographic database, OO Writers designers give us the impression that they have worked for meticulous users engaged in longer-term tasks. By contrast, MS Word designers give us the impression that they have worked for busy users, engaged in broader-context activities of which document preparation is only a part. In both cases, however, the complete message about which tools are available and how valuable they are expected to be for the targeted users is not likely to be understood unless users engage into interacting with the application. In MS Word, for example, in order to understand what research really means in context, users will probably click on the option (i. e. interact with the message itself) and see what comes next. If they do, theyll find out that the designers are offering the possibility to connect to and search online reference books and other resources. Likewise, in order to understand what bibliography database is about in OO Writer, users will also probably click on the option. If they do, they will find out that this option connects them to an existing bibliography that is installed with OO Writer, and which can be extended or modified to accommodate the users reference items. In both applications, the users can resort to online help resources to learn more about these and other parts of the designers message conveyed in the menus above. Thus, we see that communication is not limited to visible signs in isolated screens communication is also conveyed in the process of subsequent communication with the application and its resources (i.e. online help and documentation). The summative contrast between communication expressed by the MS Word and the OO Writer interfaces is a good sampler of the essence of Semiotic Engineering, whose main tenets are the following:

1. HCI is an instance of metacommunication communication about how,

when, where and why to communicate with computer systems. The metacommunication message content can be summarized as follows (note the presence of the designers using 1st person pronouns like I, my, etc.)

Here is my understanding of who you are, what Ive learned you want or need

to do, in which preferred ways, and why. This is the system that I have therefore designed for you, and this is the way you can or should use it in order to fulfill a range of purposes that fall within this vision.

2. Both designers and users are engaged in metacommunication; the systems interface represents its designers at interaction time, it is the designers deputy. The interface enables all and only the designed types of user-system conversations encoded in the underlying computer programs at development time. The metacommunication message from the designers is unfolded and received as users interact with it and learn what the system means.

3. There are three classes of metacommunication signs that designers can use
to communicate their message to users. Static signs communicate the essence of their meaning with instant time-independent representations (typically representations that can be correctly interpreted in static screen shots). Dynamic signs communicate the essence of their meaning with a series of time-dependent representations (typically representations that can only be correctly interpreted over a number of subsequent screens or states of the system). Metalinguistic signs are static or dynamic signs that differ from either the former or the latter because the essence of their meaning is an explanation, a description, an illustration, a demonstration or an indication of other [interface] signs (typically textual or video material referring to the meaning of some other static or dynamic sign).

4. Metacommunication signs in HCI must be produced computationally.

Consequently, this specific kind of computer-mediated human communication introduces critical constraints in sign production processes. Systems designers must create representations that by necessity have a single definitive encoded meaning no matter if the designers (and the users) can easily produce evolved meanings for these representations in natural sign-exchange situations. The algorithmic nature of the medium in which metacommunication takes place mechanizes human semiosis, in both

directions (designers signs produced by the system interface and users signs produced with the system interface).

5. The quality required for metacommunication to be efficient and effective is

communicability, a systems ability to signify and communicate the designers intent (which is ultimately to satisfy the users). The evaluation of communicability involves a methodical analysis of how the designers message is emitted (composed and sent through the interface) and of how it is received (interpreted and followed by physical and/or mental signmediated action) by the users.

6. Because the users response to the designers metacommunication must be

mediated by interface signs, the users must learn the interface language, a unique signification system, in which the designers message is fully encoded. Users learn this language in the very process of using it in interaction. This process is similar to natural language acquisition, which humans are fully equipped to do, except that the users immediate interlocutor (unlike humans) does not reason abductively. Therefore, some of the users communicative intent may persistently fail to be interpreted simply because the designers have not anticipated the users sign-making strategies. Because this can cause disorientation in the process of interface language acquisition, designers should also explicitly signify the communication principles that they have chosen to encode their message. Semiotic Engineering has been gradually making its way into HCI, especially after Don Norman, one of the leading figures in early UCD (Norman 1986), wrote about advantages that he sees in a semiotic engineering perspective on HCI (Norman 2004; Norman 2007; Norman 2009). In the next section, we begin to conclude this chapter with very brief concrete examples of how this perspective influences the analysis and design of computer-mediated communication in HCI.

25.5 A closer look at computer-mediated communication

As mentioned earlier in this chapter, saying that computers are media has become a clich in the turn of the century. Computers are everywhere, computing is embedded in hundreds of things that we carry and encounter in daily life, and digital information exchanged through computers, with or without our control, affects almost every step we take as individuals and as a society. A relevant question

for HCI is: How well are HCI theories prepared to inform and equip HCI designers in this context? The question is actually not new. In 1986, Winograd and Flores asked it from the Artificial Intelligence community, mainly, hitting additional targets in HCI and in Computer-Supported Collaborative Work (Winograd & Flores 1986). In spite of this history, however, most of the existing HCI theories are dominated by cognitive perspectives. They can and do inform the design of interaction with respect to facilitating the learning, memorization and retrieval of productive interactive patterns, for instance. And nobody disputes that this is a fundamental requirement in HCI design. However, the user-centered approach, as its name so clearly signifies, intentionally or unintentionally has treated design meanings as if they were mind-independent entities that are somehow elicited from users and then reified in design models and prototypes before they eventually take the shape of a systems interface. The designers interpretation and signification processes are not accounted for in original UCD (Norman 1986), whose theorists have in general tended to follow the Cartesian tradition of postulating the existence of primary cognitions. These are shared by all humans (and thus by all users), which stimulates research seeking for universal primitives and, based on their consequences, making predictions about users behavior. The problem of UCD in the computer as medium paradigm is that it cannot accommodate computer-mediated communication between systems designers and users. Designers, quite plainly, do not belong in UCD interaction models. Neither of the two most influential historical sources of UCD the seven-step theory of action (Norman 1986) and the human information processing model (Card et al 1983) account for the designers cognitive processes and meaning-making activity. This part of the story has been covered by methods (e. g. ethnography (Bouissac 1998) and theories (e. g. Activity Theory (Kaptelinin & Nardi 2006)), which have nonetheless failed to form with UCD a seamless body of knowledge that can satisfactorily account for the whole design process in accordance with the computer as medium perspective. The final ad hoc combination of uncohesive parts in actual design processes seems to be what Spool refers to as the skills and talents of practitioners (p. 8, above). Although the new media perspective instantly reminds us of Web applications design and of interaction with or through mobile devices, there is yet more to it. Digital literacy, which is undoubtfully a requirement for the achievement of full

citizenship in the 21st century, has evolved to include computational thinking skills without which users (citizens) are not likely to be able to engage in the new cultures of participation (Fischer 2011). In the abstract of a thoughtful and provocative article, Jeannette Wing (Wing 2008) expresses her perception of the extent to which changes in current computing practices and possibilities affect science, technology and society:

Computational thinking will influence everyone in every field of endeavour.

This vision poses a new educational challenge for our society, especially for our children. In thinking about computing, we need to be attuned to the three drivers of our field: science, technology and society. Accelerating technological advances and monumental societal demands force us to revisit the most basic scientific questions of computing.

-- Wing 2008: p. 3717

In the following, I will present three contemporary examples of different levels of computational thinking involved in new kinds of social interactions. In each case, I will underline the potential advantages of a semiotic framing for the illustrated phenomenon. Then, in a separate sub-section, I will talk about research we are doing at SERG, the Semiotic Engineering Research Group of Pontifcia Universidade Catlica do Rio de Janeiro (PUC-Rio), partnering with colleagues in Colorado University at Boulder and Universidade Federal Fluminense (UFF), a public university in the State of Rio de Janeiro, Brazil.

25.5.1 Some contemporary examples

In the illustrations below, we will see how computer-mediated human communication may involve explicit computational representations of self and message at different levels of complexity. In each case, the purpose of communication is very prosaic to help someone use his email account. The innovation lies in the chosen form of communication and the wealth of new meanings that it introduces in human experience. We will also see that computing literacy is the key factor for reaping the benefits of this new means of selfexpression and social participation. Taking a semiotic perspective on literacy the ability to interpret and produce signs of socially-valued signification systems in order to achieve social participation and full citizenship the point of the illustrated cases is to show that being able to program computers has become as

important in the 21st century as reading, writing and counting has been in previous centuries.

25.5.1.1 Example 1: Programming via parameter setting

Figure 25.7: Internet Buttons entry page. Copyright: See page copyright notice. Download or view: Full resolution The first example is Internet Buttons, a tool that allows you to create a personalized page with buttons that direct you straight to your favorite web sites, services and applications. Internet Buttons was created by a not-for-profit company in the UK to get people from different generations talking more, sharing more and spending more time together. In Figure 25.7, we show their home page. Notice how they get their message across with different kinds of signs like text (Create a page of buttons for someone you know [...] and Hey Mum, call me if you get stuck [...]) and image (colorful buttons and personal picture) and think of how obviously the software designers not only participate actively in interaction with the user, but also of how they would completely fail to achieve their intent if they did not have the ability (a personal talent?) to express themselves through software.

The beauty of Internet Buttons lies, however, in the recursive nature of the designers message. Through their program, they are inviting Internet users to program super easy Internet interaction for someone they know. The programming paradigm is also an extremely simple form of parametric procedure where all that the users have to do is provide the correct values for pre-selected parameters. Therefore, the kind of computational thinking required is very basic. In Figure 25.8, we show a step of the programming required to access a Gmail account.

Figure 25.8: Creating an Internet Button to access Googles Gmail login page. Copyright: See page copyright notice. Download or view: Full resolution As the Internet Buttons entry page shows, once the buttons are programmed they go into a personalized web page where the user can place any number of buttons especially created for someone else. The personalized page template allows the user-programmer to add a personal picture and a message directly addressed to whoever he are talking to through software that speaks for him. Therefore, the software carries a representation of self in addition to a representation of the message to be communicated.

25.5.1.2 Example 2: Programming via macro recording Our next example is IBMs CoScripter, a macro recorder for the Web. Unlike
Internet Buttons, CoScripter does not focus on personal communication and representations of self. However, just like Internet Buttons, the purpose of CoScripter is to make Internet processes easier for the user himself/herself and for whoever needs help with Web interaction. The system allows the user to record and playback interaction steps, optionally using the values of variables stored in a Personal Database in the users machine. Recorded CoScripts are stored in an Internet server and can be shared with other users. In Figure 25.9, the arrows that we added to the snapshot show how, in this case, the CoScript instructions (on the left) use the information in the Personal Database (at the bottom left) to command interaction with the Gmail login page.

Figure 25.9: The execution of a CoScript for accessing a Gmail account. Copyright: See page copyright notice. Download or view: Full resolution Although the scripters are not explicitly represented in the CoScript (an interesting sign of the effect of impersonal and implied communication between script designer and script user), the purpose of CoScripter creators is also interpersonal communication through software. In their home page, visited in December of 2011, we read the following message:

CoScripter is a system for recording, automating, and sharing processes

performed in a web browser such as printing photos online, requesting a vacation hold for postal mail, or checking flight arrival times. Instructions for processes are recorded and stored in easy-to-read text here on the CoScripter web site, so anyone can make use of them. If you are having trouble with a web-based process, check to see if someone has written a CoScript for it!

-- http: //coscripter.researchlabs.ibm.com/coscripter The programming paradigm in CoScripter is more sophisticated than in Internet Buttons. One of its creators, Allen Cypher, is a leading figure in programming by demonstration (Cypher 1993). The user-programmer in this case is dealing with a simple programming language, in which demonstrated interactions are automatically encoded. Optionally, the user may manipulate variables to make CoScripts more general and reusable in similar, but not identical, contexts. Mainly because of variable manipulations, the level of computational thinking required to use CoScripter is intermediary. There are a number of interesting semiotic issues to explore with CoScripter. In this chapter, I will only briefly remark that, unlike Internet Buttons, whose messages are computationally encoded with fixed token-level semantics (each button means a single Internet address), CoScripter messages can be computationally encoded with fixed type-level semantics (each script can be executed with different parameters specified in the end users Personal Database). Each CoScript means a range of possible interactions with the same web page, web service or web application. All of the interactions are predicated by the same interactive steps, but there can be variations in contextual parameters. Thus, the communication content that can be expressed with CoScripter is considerably more complex (and powerful) than in the previous example. The interested reader can see a deeper semiotic analysis of CoScripter in one of our previous publications (de Souza & Cypher 2008).

Figure 25.10: Programming with "Customize Your Web" Copyright: See page copyright notice. Download or view: Full resolution

25.5.1.3 Example 3: Programming on wheels Our last example is Customize Your Web, a personalization tool that allows
users to change the appearance of web pages and add new functionality to them without having to dive too deeply into JavaScript programming if they dont want or dont know how to do it. Customize Your Web is an extension to Firefox designed by Rudolf Noe. In Figure 25.10, we show the end user programming environment offered by this tool. The user can select elements of a Web page (in this example, the Google Search button is selected) and make changes to its appearance and behavior (see the menu from which the user is about to select

between Insert JavaScript, Insert HTML or Insert CSS). Interface elements can be deleted, inserted and relocated as desired, as long as they can be uniquely identified on the existing page whenever it is loaded. Compared to the previous example, Customize Your Web opens even more powerful possibilities for computer-mediated communication. In Figure 25.11 and Figure 25.12, we show how facilitating interaction with Gmail is substantially different in Customize Your Web, compared to CoScripter and Internet Buttons. Because the programming paradigm is close to programming on wheels, the possibilities for representation of self and message are limited only by technicalities of the tool (like problems with unique identification of elements on certain Web pages) and the resources of the user. The level of computational thinking required to use this tool effectively is advanced.

Figure 25.11: Gmail login page modified with Customize your Web Copyright: See page copyright notice. Download or view: Full resolution In Figure 25.11 (see customized text starting with Hi, Dad on the left), we can see that the user can script his own presence and participation in someone elses experience with Gmail. The complexity of self-representation and message in this case is considerably high. Notice that in Figure 25.12, the conversation between the scripter and the targeted user (his Dad) extends over whole interaction spans with Gmail. In other words, a parallel communication about communication with Gmail (i. e. genuine metacommunication about Gmail) is explicitly and intentionally in place. This is one of the best examples of why the computer as media perspective calls for a different breed of HCI theories in order to help end users take the best

out of the virtually unlimited possibilities of social interaction and participation now available for them.

Figure 25.12: Gmail inbox page modified with Customize your Web Copyright: See page copyright notice. Download or view: Full resolution Designing cohesive and consistent dialog turns throughout a Web process, like shown in the Customize Your Web example, is end user HCI design not only end user programming and development. However, theories that do not include the designers in the communication process that they are designing have very little to say about how (and why) the current interface of Customize Your Web (see Figure 25.10) must be improved. For example, UCD is not going to help the user design all the conversational paths that his parallel communication with Dad might take in his absence. Dad can decide to use any of the available interface controls during interaction with Gmail. So, how does the user build a representation of

conversational context that is consistent with what he is telling Dad at each step of the programmed interaction? The answer to these questions can not only show the complexity of programming that end users can engage in at this stage of technological development, but also and more importantly it can show what kinds of new dimensions a semiotic perspective can bring to HCI research and practice. In the next sub-section, we will very quickly mention how we are exploring computer-mediated communication issues with Semiotic Engineering.

25.5.2 Going a few steps further

Ones level of computing literacy will determine how one will be able to participate in new kinds of social interactions made possible by new ICT. As the examples above have shown, higher literacy levels raise the transformative power of communication to unprecedented standards. The challenge is in the air: If computational thinking will be used everywhere, then it will touch everyone directly or indirectly. [...] If [it] is added to the repertoire of thinking abilities, then how and when should people learn this kind of thinking and how and when should we teach it? (Wing 2008: p. 3720) One of the most successful responses to-date is AgentSheets, a visual programming environment designed mainly for teaching computational thinking at schools (Repenning & Ioannidou 2004). The Scalable Game Design Project, carried out by Alexander Repenning and his group at Colorado University at Boulder has been educating school teachers and students in computational thinking for many years now. In 2010, we brought the project to Brazil and started working with a public school in the city of Niteri. Given our interest in semiotic theories of HCI, we have approached the project with a dual perspective. On the one hand, we want to educate teachers and students in computational thinking. On the other, we want to show them that computational thinking gives them a completely new language for self-expression and communication, with which they can do whatever they can mean (or signify, in semiotic terms). Our strategy is to work constantly with the semiotic relations that hold between three distinct signification systems (see Figure 25.13): natural language, game play (i. e. executable games programmed by AgentSheets users) and game code (i. e. the various levels of programming language encodings that make

the games run). In AgentSheets, the users can program in Visual AgenTalk, a visual programming language with a textual XML counterpart.

Figure 25.13: Semiotic varieties in a visual programming environment. Copyright: See page copyright notice. No higher resolution available Empirical evidence collected in game design sessions with a group of students in 2010 has shown elaborate and intriguing signification patterns when we compared, for each game, the natural language descriptions provided by the programmers, the Visual AgenTalk code, and the resulting executable game representations (de Souza et al 2011). In particular, we traced interesting entity-naming strategies, token/type relations in representation choices, and curious transitive structure changes when contrasting natural language and computer program representations. Subsequent research steps carried out in 2011 with another group of students suggest that semiotic relations between signification systems involved in using AgentSheets can be used to raise the level of computing awareness among learners. In Figure 25.14, we show successive screen shot snippets from a semiotic exploration of relations between game play signs, game code signs and natural language signs in a very abstract environment used by CS graduate students. The game representation (an executable simulation) has only four static signs: an orange circle; three colored squares; and a black background. The dynamic signs

are also very few and very simple: if the user presses arrow keys the orange circles moves in the corresponding direction; if the circle moves next to the boundaries of any of the three squares, it is trapped by the square and the game is reset. The fascinating aspects of the exploration were: to examine what meanings the participating students assigned to the simulation; and to examine how the apparently identical behaviors of the three squares had been intentionally encoded by the experimenters as completely different representations.

Figure 25.14: Screen shots from a semiotic exploration of simulation signs with AgentSheets. Copyright: See page copyright notice. Download or view: Full resolution When asked to write down their explanation (Explicao in Figure 25.14) for the orange circle behavior, inferred exclusively from the participants interaction with the game, the students used signs referenced to the protagonist of the game. For example, the explanation depicted in Figure 25.14 says that the orange circle cannot collide with any of the obstacles represented by the squares. A totally different story, however, would be told if the student looked at the underlying program. The three squares are actually programmed as very different agents. The green square traps the circle as it comes near it. The white square is like an attractive area into which the orange circle throws itself as soon as it comes near it. And the white square is no more than a hole in the ground, an agent-free space in the worksheet, into which the circle may fall propelled by the ground. Thus, the

designers story about this simulation is a totally different one if looked at from the inside or the outside. From the inside, it might go like this, for instance: The orange circle loves the grey square and jumps into it as soon as it sees it; it must escape from the green square that traps it when the circle comes near it; and it must beware of a square-shaped hole in the ground, into which it must not fall. An HCI-oriented mind will probably say that for the designer to communicate his story appropriately to the user, the abstract squares should be replaced with icons that adequately represent trap, attractive area and hole in the ground. However, this is actually telling only part of the computational thinking story, of course. The fact that similar effects can be computationally represented by very different data structures and algorithms is one of the most fundamental and powerful principles of Computer Science. Therefore, we should be telling school teachers and students that computational signs are like plastic semiotic material, and stimulating them to play with it in as many ways as they can imagine. This is what will really empower them to express themselves computationally. In Figure 25.15, we show a sketched version of Exploratorium, a semiotically-rich prototype plug-in to AgentSheets, which we are developing for its users to be able to explore how different game play representations are encoded in Visual AgenTalk. Our intent is to raise the students and teachers awareness of the various possibilities in computer representations, and to stimulate them to play with these possibilities at their will. The links on the illustrated page show that users can ask a number of things about the representations they see (how they are produced, where they are located, what they stand for and why). The answers are partly generated automatically from Visual AgenTalk representations and partly retrieved from annotations (secondary notations) elicited from the programmers in dialogs about what they meant.

Figure 25.15: The Scalable Game Design Brasil Project: A semiotically-rich environment to explore signs in computing. Copyright: See page copyright notice. Download or view: Full resolution In Figure 25.16, we show a sketched version of the Web Navigation Helper (Intrator and de Souza 2009; Intrator & de Souza 2008; Monteiro & de Souza 2011), a user agent to increase Web accessibility that we have been expanding to bridge cultural barriers of various sorts (language barriers, literacy barriers, etc.). As shown in this illustration, the user agent actually rephrases the designers communication originally expressed in a Web page. In Figure 25.16, the user agent is an interpreter for a Brazilian user that is trying to navigate a Web page produced in the USA. Technically, the user agent is a Firefox extension that works in combination with CoScripter. For every interaction that it mediates, the agent requires a recorded CoScript and a full specification of the mediating dialog that will take place at each step of the recorded interaction. The illustrated dialog, in Portuguese, is telling the Brazilian user how he or she can interact with the Scalable Game Design Arcade page. Notice that the dialog is not a verbatim translation of the page (although this would also be a kind of mediation). The agent is actually guiding a foreigners interaction with material produced in and by another culture.

Figure 25.16: An interaction mediator helping a Brazilian user navigate through a website in English. Copyright: See page copyright notice. Download or view: Full resolution Guidance is designed using conceptual viewpoint metaphors proposed by Salgado (Salgado et al 2011; Salgado et al 2012). Very briefly, the metaphors allow interaction designers to organize cross-cultural communication from different perspectives, ranging from the strongest form of mediation (where the foreign culture is invisible to the user thanks to communication carried out in his or her own native language, with complete neutralization of all references to foreign cultural practices and values) to the weakest, actually non-existing, form of mediation (where the foreign culture is fully exposed, with no trace of the users culture communication is carried out in a foreign language, with reference to foreign cultural practices and values).

Semiotic Engineering research, as the examples above can show, expands the limits of user-centered HCI and establishes connections with other specialized areas of Computer Science and Information Systems Development, like Programming (through the semiotics of computer representations) and Information Architecture (through the organization of explicit cultural references to be used in computational signification systems).

25.6 So, what's in it for me?

In his preface to the Encyclopedia of Semiotics Paul Bouissac writes:

The twentieth century has witnessed an increasing fragmentation of knowledge

into a multitude of disciplines and specialties. At the same time, integrative visions have arisen in an effort to make sense of the flood of information generated by the modern intensification of formal and empirical research. Semiotics represents one of the main attempts perhaps the most enduring one at conceiving a transdisciplinary framework through which interfaces can be constructed between distinct domains of inquiry.

-- Bouissac 1998: p. ix

Only a few years after Bouissac, John Carroll writes in the introduction to one of his books:

An ironic downside of the inclusive multidisciplinarity of HCI is fragmentation.

[...] There are too many theories, too many methods, too many application domains, too many systems. Indeed, fragmentation may be a bit worse than it has to be. Some HCI researchers, faced with the huge intellectual scope of concepts and approaches, deliberately insulate themselves from some portion of the fields activity and knowledge. This tension between depth and breadth in scientific expertise is not unique to HCI but it clearly undermines the opportunity for multidisciplinary progress.

-- Card et al 1983: p. 6

So, whats in Semiotics for you? I hope that this chapter has provided multiple signs of my personal answer to the question. Semiotics can provide the foundations for new integrative theories of HCI. Now, why should we need integrative theories? One of the answers is explicitly

given by John Carroll in the citation above. But this chapter has touched on some of the implications of Carrolls answers, as well as on some other possible answers to the challenging question. A semiotic perspective on HCI proposes that meaning is the most fundamental concept in this discipline, and that our inquiry should account for how meaning is communicated through computer systems interfaces, in as many guises as they now occur. It follows from this that HCI must not only care for meaning takers, but also for meaning makers: communication implies an exchange of signs produced by at least two communicating minds. Therefore, to escape the embarrassment of having to postulate that systems have minds of their own, HCI will benefit from theories that can account for meaning-making processes in which HCI designers clearly engage while in the process of building systems interfaces to support productive user interaction with computing artifacts. In fact, computer meaning-making process can engage not only designers, but also programmers and a whole host of developers that encode meaning elements (signs) in software tools with which modern computer artifacts are built. Therefore, a semiotic perspective can potentially track human signification throughout long chains of software instances, in space and time, just like semiotic theories trace meaning in society, culture and history. Because they push the designers into representing self-images and thinking of their communicative intent and strategies, HCI theories of semiotic extraction can also stimulate the development of a new ethics in HCI design. Although the UCD tradition has clearly established an ethics commitment with the users needs and aspirations, it has not contributed much to an ethics of communication since UCD theories only talk about relations and interactions between users and systems not about users and systems designers. It is remarkable, however, that in his most recent book (Norman 2010), Don Norman, who inspired UCD with his Cognitive Engineering and seven-step theory of action in the 1980s (Norman 1986) speaks openly of the importance of communication to live with complexity, and goes as far as dedicating a full chapter to social signifiers a semiotic concept. This is itself the sign of that an ethics of communication begins to make its way into design and HCI. This new ethics may play a relevant role in the wake of the 21 st century when global awareness of the need to respect and preserve cultural diversity is greater than ever before. A semiotic perspective can help technologists in trying to prevent that their

products inadvertently extinguish valued diversity of meanings from user communities (Salgado et al 2012). By bringing designers and users together in the process of HCI, a semiotic perspective can naturally promote more reflection in action (Schn 1983). When designers view themselves as talking to users, they can naturally tap on their instinctive abilities to signify reality and prompt communicative reaction from people around them. They are as native in the human side of human-computer interaction as the users. Therefore, we need as many good theories about them as we have about users. Reflective theories will help designers to produce computermediated communication in which they can consciously model their own signification and communication A semiotic outlook can also bring about some clarifications at meta-theoretical levels that scientific research should attend to. In this chapter, we have only highlighted aspects of Cartesian influence in the thinking of many HCI researchers and practitioners. We commented on why ultimate primary cognitions, which human minds can access by means of a rigorous inspection method, are not only appealing to computer language theorists, but often tacitly subscribed by members of the HCI community. This notion allows for framing HCI as a formal linguistic exchange between humans and computers. Semiotic theories that are philosophically aligned with this specific Cartesian view will probably reinforce beliefs that there are primitive conceptual meanings that can be known in the process of investigation. However, even semiotic theories that do not upset the applecart in CS, as nonCartesian theories tend to do, lead researchers into making explicit commitments with the ontological origins of meaning and its variations. For example, a semiotic foundation requires that researchers provide a theoretical explanation of why and how people (in our case users and designers) assign different meanings to identical representational stimuli. This explanation will guide their choices in designing computer-mediated communication between software producers and software users, showing that theoretically-informed design is actually a lot more than a craft. Researchers will also have to deal with hard methodological and epistemological questions. If there are such things as conceptual meaning primitives, what are they? How can they tell true primitives from false ones? How can they explain that sometimes our meanings are right, and other times these very meanings are wrong? How can they explain why, even when some of their

meanings are wrong, people occasionally make the right inferences using them? All of these questions are fundamentally important for scientific research in HCI and, as we have seen, Semiotics can play an important role in helping us to formulate them, to understand what they mean, and then to investigate possible answers. To conclude, it is my belief that, grounded on more explicit philosophical commitments, the most important contribution of Semiotics to HCI is to rehabilitate human talent and intuition from the dark corners where it has been hiding, and to bring it to light as one of the most powerful elements for effective, efficient, creative and enjoyable computer-mediated communication in the 21st century. Our task, as HCI researchers and practitioners, is to generate and use the knowledge that is necessary to turn computers into the richest medium for human signification and communication.

25.7 Acknowledgments
I have to thank my sponsors CNPq and FAPERJ and many people that have supported, stimulated and developed research in Semiotic Engineering over the last twenty years. In particular, I would like to thank namely those that have contributed more directly to SERG projects mentioned in this chapter: Carla Leito, Luciana Salgado, Cristina Garcia, Ingrid Monteiro, Juliana Ferreira, Marcelle Mota, Cleyton Slaviero, Eduardo Tolmasquim, Leonardo Faria, Alexander Repenning, Nadia Repenning, Allen Cypher, Gerhard Fischer, CoScripter developers and AgentSheets developers. I would also like to thank Mads Soegaard and Allan Holstein-Rathlou for first-class editorial work. I owe special thanks to Mads for having invited me to write this chapter and for his constant encouragement. It has been a much longer project than I first thought it would be, but also a much more rewarding thinking and learning adventure. For more information about Semiotic Engineering, please visit our website.


25.8 Commentary by Alan Blackwell
How to cite this commentary in your report Andersen, Peter B. (1997): A Theory of Computer Semiotics : Semiotic Approaches to Construction and Assessment of Computer Systems. Cambridge University Press Andersen, Peter B. (1990): A Theory of Computer Semiotics. Cambridge, UK, Cambridge University Press Andersen, Peter Bgh, Holmqvist, Berit and Jensen, Jens F. (eds.) (1994): The Computer as Medium (Learning in Doing: Social, Cognitive and Computational Perspectives). Cambridge University Press

Computers are developing into a powerful medium integrating film, pictures,

text, and sound, and the use of computers for communication and information is rapidly expanding. The Computer as Medium brings insights from art, literature, and theater to bear on computers and discusses the communicative and organizational nature of computer networks within a historical perspective. The book consists of three parts: The first part characterizes the semiotic nature of computers and discusses semiotic approaches to programming and interface design. The second section discusses narrative and aesthetic issues of interactive fiction, information systems, and hypertext. The final part contains papers on the cultural, organizational, and historical impact of computers.

All rights reserved Andersen et al. and/or Cambridge University Press Andersen, Peter Bgh, Holmqvist, Berit and Jensen, Jens F. (eds.) (1993): The Computer as Medium (Learning in Doing: Social, Cognitive and Computational Perspectives). Cambridge University Press

Computers are developing into a powerful medium integrating film, pictures,

text, and sound, and the use of computers for communication and information is rapidly expanding. The Computer as Medium brings insights from art, literature, and theater to bear on computers and discusses the communicative and

organizational nature of computer networks within a historical perspective. The book consists of three parts: The first part characterizes the semiotic nature of computers and discusses semiotic approaches to programming and interface design. The second section discusses narrative and aesthetic issues of interactive fiction, information systems, and hypertext. The final part contains papers on the cultural, organizational, and historical impact of computers.

All rights reserved Andersen et al. and/or Cambridge University Press Baranauskas, M. Cecilia M., Liu, Kecheng and Chong, Samuel (2003): Website Interfaces as Representamina of Organizational Behaviour. In: Gazendam, H. W., Jorna, Ren and Cijsouw, R. S. (eds.). "Dynamics and Change in Organizations: Studies in Organizational Semiotics (Studies in Organisational Semiotics, 3.)". Springer Beyer, Hugh R. and Holtzblatt, Karen (1998): Contextual Design: Defining CustomerCentered Systems. San Francisco, Morgan Kaufmann Publishers Bonacin, Rodrigo, Baranauskas, M. Cecili C. and Liu, Kecheng (2004): Interface Design for the Changing Organisation: an Organisational Semiotics Approach. In: Liu, Kecheng (ed.). "Virtual, Distributed and Flexible Organisations: Studies in Organisational Semiotics". Springer

Within the Organisational Semiotics (OS) perspective, software systems should

be designed as part of the whole organisation in which it will be embedded. As organisations are in continuous change, the technical information systems should also be changing in line with their organisations informal and formal information systems. From the Software Engineering perspective, the difficulties of solving this problem are well known: changes in information systems are usually associated with high costs and its maintenance may cost more than the initial development. While literature in Computer Supported Co-operative Work (CSCW) acknowledges the importance of proposing approaches to deal with this issue, it also acknowledges that we are far from having this problem solved. In this paper we propose a norm driven environment for the system interface configuration, as a way of dealing with the complexity of allowing changes in the system as the organisational norms change.

All rights reserved Bonacin et al. and/or Springer Bouissac, Paul (ed.) (1998): Encyclopedia of Semiotics. Oxford University Press, USA

Three hundred entries by leading scholars in a variety of fields--from

anthropology and literary theory to linguistics and philosophy--survey the study of signs and symbols in human culture in this new work. The articles cover key concepts, theories, theorists, schools, and issues in communications, cognition, and cultural theory. From introductions to Barthes and Bakhtin to analyses of gossip and myth, this is a valuable reference for students and scholars.

All rights reserved Bouissac and/or Oxford University Press, USA Buchler, Justus (ed.) (1955): Philosophical Writings of Peirce. Dover Publications

Arranged and integrated to reveal epistemology, phenomenology, theory of

signs, other major topics. Includes "The Fixation of Beliefs," "How to Make Our Ideas Clear," "The Scientific Attitude and Fallibilism," "Philosophy and the Sciences: A Classification," " The Principles of Phenomenology," " Logic as Semiotic: The Theory of Signs," and "The Criterion of Validity in Reasoning." All rights reserved Buchler and/or Dover Publications Burrell, Gibson and Morgan, Gareth (1979): Sociological Paradigms and Organisational Analysis: Elements of the Sociology of Corporate Life. Ashgate Publishing

The absolute classic of theory of organization. The book reveals hidden

assumptions made by the organizations' members, the beliefs of 'consultants' or researchers - and divides the presumptions into four separate categories (named 'paradigms' after Kuhn's specification). The authors brilliantly show, how the paradigms influence our perceptions and the ways we look at the organizations. All the approaches are fairly exemplified. The pros and cons of every paradigm become evident. A different light is shed on many famous theories and on the root methodology itself. The book should definitely be an obligatory lecture of management and sociology students - the managers and consultants can pick something for the as well.

All rights reserved Burrell and Morgan and/or Ashgate Publishing

Burrell, Gibson and Morgan, Gareth (2007): Sociological Paradigms and Organisational Analysis: Elements of the Sociology of Corporate Life. Ashgate Publishing

The absolute classic of theory of organization. The book reveals hidden

assumptions made by the organizations' members, the beliefs of 'consultants' or researchers - and divides the presumptions into four separate categories (named 'paradigms' after Kuhn's specification). The authors brilliantly show, how the paradigms influence our perceptions and the ways we look at the organizations. All the approaches are fairly exemplified. The pros and cons of every paradigm become evident. A different light is shed on many famous theories and on the root methodology itself. The book should definitely be an obligatory lecture of management and sociology students - the managers and consultants can pick something for the as well.

All rights reserved Burrell and Morgan and/or Ashgate Publishing Card, Stuart K., Moran, Thomas P. and Newell, Allen (1983): The Psychology of Human-Computer Interaction. Hillsdale, NJ, Lawrence Erlbaum Associates Carroll, John M. (ed.) (2003): HCI Models, Theories, and Frameworks: Toward a multidisciplinary science. San Francisco, Morgan Kaufmann Publishers Carroll, John M. and Rosson, Mary Beth (1987): The paradox of the active user. In: Carroll, John M. (ed.). "Interfacing Thought: Cognitive Aspects of HumanComputer Interaction (Bradford Books)". The MIT Presspp. 80-111 Cypher, Allen (1993): Watch What I Do: Programming by Demonstration. Cambridge, MA, The MIT Press de Souza, Clarisse Sieckenius (1993): The Semiotic Engineering of User Interface Languages. In International Journal of Man-Machine Studies, 39 (5) pp. 753-773

Semiotic approaches to design have recently shown that systems are messages

sent from designers to users. In this paper we examine the nature of such messages and show that systems are messages that can send and receive other messages -they are metacommunication artefacts that should be engineered according to explicit semiotic principles. User interface languages are the primary expressive resource for such complex communication environments. Existing cognitivelybased research has provided results which set the target interface designers should hit, but little is said about how to make successful decisions during the process of

design itself. In an attempt to give theoretical support to the elaboration of user interface languages, we explore Eco's Theory of Sign Production (U. Eco, A Theory of Semiotics, Bloomington, IN: Indiana University Press, 1976) and build a semiotic framework within which many design issues can be explained and predicted. All rights reserved de Souza and/or Academic Press de Souza, Clarisse Sieckenius and Cypher, Allen (2008): Semiotic engineering in practice: redesigning the CoScripter interface. In: Levialdi, Stefano (ed.) AVI 2008 Proceedings of the working conference on Advanced Visual Interfaces May 28-30, 2008, Napoli, Italy. pp. 165-172 de Souza, Clarisse Sieckenius and Leito, Carla Faria (eds.) (2009): Semiotic Engineering Methods for Scientific Research in HCI (Synthesis Lectures on Human-Centered Informatics). Morgan and Claypool Publishers

Semiotic engineering was originally proposed as a semiotic approach to

designing user interface languages. Over the years, with research done at the Department of Informatics of the Pontifical Catholic University of Rio de Janeiro, it evolved into a semiotic theory of human-computer interaction (HCI). It views HCI as computer-mediated communication between designers and users at interaction time. The system speaks for its designers in various types of conversations specified at design time. These conversations communicate the designers' understanding of who the users are, what they know the users want or need to do, in which preferred ways, and why. The designers' message to users includes even the interactive language in which users will have to communicate back with the system in order to achieve their specific goals. Hence, the process is, in fact, one of communication about communication, or metacommunication. Semiotic engineering has two methods to evaluate the quality of metacommunication in HCI: the semiotic inspection method (SIM) and the communicability evaluation method (CEM). Up to now, they have been mainly used and discussed in technical contexts, focusing on how to detect problems and how to improve the metacommunication of specific systems. In this book, Clarisse de Souza and Carla Leito discuss how SIM and CEM, which are both qualitative methods, can also be used in scientific contexts to generate new knowledge about HCI. The discussion goes into deep considerations about scientific methodology, calling the reader's attention to the essence of qualitative methods in research and the kinds of results they can produce. To illustrate their points, the authors present an extensive case study with a free opensource digital audio editor called Audacity. They show how the results obtained with a triangulation of SIM and CEM point at new research avenues not only for semiotic engineering and HCI but also for other areas of computer science such as software engineering and programming. Table of Contents: Introduction / Essence of Semiotic Engineering / Semiotic Engineering Methods / Case Study with

Audacity / Lessons Learned with Semiotic Engineering Methods / The Near Future of Semiotic Engineering Publishers Descartes, Rene (2004): Discours de la Methode. Project Gutenberg

All rights reserved de Souza and Leito and/or Morgan and Claypool

http://www.gutenberg.org/ebooks/13846 French-language version of

"Discours de la methode" by Rene Descartes.

All rights reserved Descartes and/or Project Gutenberg Eco, Umberto (1978): A Theory of Semiotics. Indiana University Press

"... the greatest contribution to [semiotics] since the pioneering work of C. S.

Peirce and Charles Morris." Journal of Aesthetics and Art Criticism"... draws on philosophy, linguistics, sociology, anthropology and aesthetics and refers to a wide range of scholarship... raises many fascinating questions." Language in Society"... a major contribution to the field of semiotic studies." Robert Scholes, Journal of Aesthetics and Art Criticism"... the most significant text on the subject published in the English language that I know of." Arthur Asa Berger, Journal of CommunicationEco's treatment demonstrates his mastery of the field of semiotics. It focuses on the twin problems of the doctrine of signscommunication and significationand offers a highly original theory of sign production, including a carefully wrought typology of signs and modes of production. All rights reserved Eco and/or Indiana University Press Eco, Umberto (1976): A Theory of Semiotics. Indiana University Press

"... the greatest contribution to [semiotics] since the pioneering work of C. S.

Peirce and Charles Morris." Journal of Aesthetics and Art Criticism"... draws on philosophy, linguistics, sociology, anthropology and aesthetics and refers to a wide range of scholarship... raises many fascinating questions." Language in Society"... a major contribution to the field of semiotic studies." Robert Scholes, Journal of Aesthetics and Art Criticism"... the most significant text on the subject published in the English language that I know of." Arthur Asa Berger, Journal of

CommunicationEco's treatment demonstrates his mastery of the field of semiotics. It focuses on the twin problems of the doctrine of signscommunication and significationand offers a highly original theory of sign production, including a carefully wrought typology of signs and modes of production. All rights reserved Eco and/or Indiana University Press Fischer, Gerhard (2011): Understanding, fostering, and supporting cultures of participation. In Interactions, 18 (3) pp. 42-53 Floch, Jean-Marie (ed.) (2001): Semiotics, Marketing and Communication: Beneath the Signs, the Strategies (International Marketing Series). Palgrave Macmillan

Semiotics, or the study of signs, plays an increasingly important role within

marketing as a guide to psychological and social aspects of communication. JeanMarie Floch provides an introduction to the potential offered by a semiotic approach to a variety of marketing and communication problems or situations. Key semiotic concepts and principles are gradually introduced using real life studies.

All rights reserved Floch and/or Palgrave Macmillan Fogg, B. J. (2002): Persuasive Technology: Using Computers to Change What We Think and Do. Morgan Kaufmann

Can computers change what you think and do? Can they motivate you to stop

smoking, persuade you to buy insurance, or convince you to join the Army? "Yes, they can," says Dr. B.J. Fogg, director of the Persuasive Technology Lab at Stanford University. Fogg has coined the phrase "Captology"(an acronym for computers as persuasive technologies) to capture the domain of research, design, and applications of persuasive computers.In this thought-provoking book, based on nine years of research in captology, Dr. Fogg reveals how Web sites, software applications, and mobile devices can be used to change people's attitudes and behavior. Technology designers, marketers, researchers, consumers-anyone who wants to leverage or simply understand the persuasive power of interactive technology-will appreciate the compelling insights and illuminating examples found inside. Persuasive technology can be controversial-and it should be. Who will wield this power of digital influence? And to what end? Now is the time to survey the issues and explore the principles of persuasive technology, and B.J. Fogg has written this book to be your guide. * Filled with key term definitions in persuasive

computing*Provides frameworks for understanding this domain*Describes real examples of persuasive technologies

All rights reserved Fogg and/or Morgan Kaufmann Hirschheim, Rudy, Klein, Heinz K. and Lyytinen, Kalle (1995): Information Systems Development and Data Modeling: Conceptual and Philosophical Foundations. Cambridge University Press

Data modeling was hypothesized to be the "salvation" of an organization's data

problems. This book aims to analyze the problems encountered and to present a comparative philosophical study of the various approaches. On the philosophical level, the authors explore the epistemology, ontology, and rationality of each modeling approach. While on the theoretical computer science level, a systematic study of the history and development of three major strands of data modeling is presented. This book will be of great interest to all computer scientists using information systems as well as philosophers with an interest in computing applications.

All rights reserved Hirschheim et al. and/or Cambridge University Press Hix, Deborah (ed.) (1988): Advances in Human-Computer Interaction. Norwood, NJ, Intellect Hjelmslev, Louis (1961): Prolegomena to a Theory of Language. Univ of Wisconsin Press Hopcroft, John E., Motwani, Rajeev and Ullman, Jeffrey D. (1979): Introduction to Automata Theory, Languages, and Computation. Addison Wesley

This classic book on formal languages, automata theory, and computational

complexity has been updated to present theoretical concepts in a concise and straightforward manner with the increase of hands-on, practical applications. This new edition comes with Gradiance, an online assessment tool developed for computer science. Please note, Gradiance is no longer available with this book, as we no longer support this product.

All rights reserved Hopcroft et al. and/or Addison Wesley

Hopcroft, John E., Motwani, Rajeev and Ullman, Jeffrey D. (2006): Introduction to Automata Theory, Languages, and Computation (3rd Edition). Addison Wesley

This classic book on formal languages, automata theory, and computational

complexity has been updated to present theoretical concepts in a concise and straightforward manner with the increase of hands-on, practical applications. This new edition comes with Gradiance, an online assessment tool developed for computer science. Please note, Gradiance is no longer available with this book, as we no longer support this product.

All rights reserved Hopcroft et al. and/or Addison Wesley Hopcroft, John E., Motwani, Rajeev and Ullman, Jeffrey D. (2000): Introduction to Automata Theory, Languages, and Computation (2nd Edition). Addison Wesley

It has been more than 20 years since this classic book on formal languages,

automata theory, and computational complexity was first published. With this longawaited revision, the authors continue to present the theory in a concise and straightforward manner, now with an eye out for the practical applications. They have revised this book to make it more accessible to today's students, including the addition of more material on writing proofs, more figures and pictures to convey ideas, side-boxes to highlight other interesting material, and a less formal writing style. Exercises at the end of each chapter, including some new, easier exercises, help readers confirm and enhance their understanding of the material. All rights reserved Hopcroft et al. and/or Addison Wesley Houser, Nathan and Kloesel, Christian J.W. (eds.) (1992): The Essential Peirce, Volume 1: Selected Philosophical Writings (1867-1893). Indiana University Press

"... a first-rate edition, which supersedes all other portable Peirces.... all the

Peirce most people will ever need." Louis Menand, The New York Review of Books"The Monist essays are included in the first volume of the compact and welcome Essential Peirce; they are by Peirce's standards quite accessible and splendid in their cosmic scope and assertiveness." London Review of BooksA convenient two-volume reader's edition makes accessible to students and scholars the most important philosophical papers of the brilliant American thinker Charles Sanders Peirce. This first volume presents twenty-five key texts from the first

quarter century of his writing, with a clear introduction and informative headnotes. Volume 2 will highlight the development of Peirce's system of signs and his mature pragmatism.

All rights reserved Houser and Kloesel and/or Indiana University Press Intrator, Chantal and de Souza, Clarisse Sieckenius (2008): Using web scripts to improve accessibility. In: Proceedings of the VIII Brazilian Symposium on Human Factors in Computing Systems 2008. pp. 292-295 Kammersgaard, John (1988): Four Different Perspectives on Human-Computer Interaction. In International Journal of Man-Machine Studies, 28 (4) pp. 343-362

This paper will stress the value of a multi-perspective view on the use of

computers. It will argue that the ability to apply more than one perspective is valuable to designers of computer applications, to researchers dealing with humancomputer interactions, as well as to users of a particular computer application. As a means for that the paper will present the systems perspective, the dialogue partner perspective, the tool perspective, and the media perspective. All four perspectives will primarily be characterized in relation to human-computer interaction, and the characterizations will be based on a common set of concepts presented in the beginning of the paper. The last section of the paper will, with the help of a few examples, illustrate the value of applying multiple perspectives. All rights reserved Kammersgaard and/or Academic Press Kaptelinin, Victor and Nardi, Bonnie A. (2006): Acting with Technology: Activity Theory and Interaction Design. The MIT Press Khaled, Rilla, Barr, Pippin, Fischer, Ronald, Noble, James and Biddle, Robert (2006): Factoring culture into the design of a persuasive game. In: Kjeldskov, Jesper and Paay, Jane (eds.) Proceedings of OZCHI06, the CHISIG Annual Conference on Human-Computer Interaction 2006. pp. 213-220

Preliminary studies indicate that games can be effective vehicles for persuasion.

In order to have a better chance at persuading target audiences, however, we claim that it is best to design with the background culture of the intended audience in mind. In this paper, we share our insights into the differences of perception between New Zealand (NZ) Europeans and Maori (the indigenous people of NZ), regarding smoking, smoking cessation, and social marketing. Based on our

findings, we discuss how we have designed two different versions of culturally relevant persuasive game about smoking cessation, one aimed at a NZ European audience, the other aimed at a Maori audience.

All rights reserved Khaled et al. and/or their publisher Light, Ann (2001): Interactivity and User Commitment - Relationship Building through Interaction on Websites. In: Proceedings of the HCI01 Conference on People and Computers XV 2001. pp. 459-474 Liu, Kecheng (2000): Semiotics in Information Systems Engineering. Cambridge, Cambridge University Press

Semiotics, a well established discipline of signs and their use in human and

computer communications, is increasingly recognized as important to understanding information systems and computing in general. This important new resource examines a set of semiotic methods for information systems development. Kecheng Liu offers well balanced coverage of recent theoretical investigations and practical applications. He introduces the MEASUR approach for requirements elicitation, analysis, and representation and illustrates the methods in three major case studies. In these cases he demonstrates how information systems can be developed to meet business requirements and to support business objectives. All rights reserved Liu and/or Cambridge University Press Monteiro, Ingrid Teixeira and de Souza, Clarisse Sieckenius (2011): Embedded cultural features in the design of an accessibility agent for the web. In: Proceedings of the 6th international conference on Universal access in human-computer interaction design for all and eInclusion - Volume Part I 2011. pp. 295-304

This paper presents the Web Navigation Helper (WNH), an interface agent for

users with special needs originally developed for Brazilian users. WNH mediates scripted interaction with web sites, by providing alternative dialogs with appropriate style, structure, etc. The paper reports the results of qualitative empirical studies done at the early design stages. In particular, it shows how our design vision changed when findings from initial studies revealed that the technology we were about to develop was implicitly guided by a sociability model that was not prevalent in the Brazilian culture. The main contributions of the paper are to expose the process by which we became aware of cultural factors affecting the design of accessibility agents, and to propose a kind of technology that may be

adopted in cultures whose sociability models are based on personal relations with friends and family members.

All rights reserved Monteiro and de Souza and/or their publisher Morris, Charles (1971): Writings on the General Theory of Signs. Mouton De Gruyter Mullet, Kevin and Sano, Darrell (1995): Designing Visual Interfaces: Communication Oriented Techniques. Sunsoft Press Nadin, Mihai (1988a): Interface design: A semiotic paradigm. In Semiotica, 69 (3) pp. 269-302 Nadin, Mihai (1988b): Interface design and evaluation - Semiotic implications. In: Hix, Deborah (ed.). "Advances in Human-Computer Interaction". Norwood, NJ: Intellectpp. 45-100 Norman, Donald A. (2009): Systems thinking: a product is more than the product. In Interactions, 16 (5) pp. 52-54 Norman, Donald A. (1986): Cognitive engineering. In: Norman, Donald A. and Draper, Stephen W. (eds.). "User Centered System Design: New Perspectives on HumanComputer Interaction". Hillsdale, NJ: Lawrence Erlbaum Associatespp. 31--61 Norman, Donald A. (2010): Living with Complexity. The MIT Press

If only today's technology were simpler! It's the universal lament, but it's

wrong. We don't want simplicity. Simple tools are not up to the task. The world is complex; our tools need to match that complexity. Simplicity turns out to be more complex than we thought. In this provocative and informative book, Don Norman writes that the complexity of our technology must mirror the complexity and richness of our lives. It's not complexity that's the problem, it's bad design. Bad design complicates things unnecessarily and confuses us. Good design can tame complexity.Norman gives us a crash course in the virtues of complexity. But even such simple things as salt and pepper shakers, doors, and light switches become complicated when we have to deal with many of them, each somewhat different. Managing complexity, says Norman, is a partnership. Designers have to produce things that tame complexity. But we too have to do our part: we have to take the time to learn the structure and practice the skills. This is how we mastered reading and writing, driving a car, and playing sports, and this is how we can master our complex tools. Complexity is good. Simplicity is misleading. The good life is complex, rich, and rewarding--but only if it is understandable, sensible, and meaningful.

 All rights reserved Norman and/or The MIT Press Norman, Donald A. (2007): The Design of Future Things. Basic Books

Donald A. Norman, a popular design consultant to car manufacturers,

computer companies, and other industrial and design outfits, has seen the future and is worried. In this long-awaited follow-up to The Design of Everyday Things, he points out whats going wrong with the wave of products just coming on the market and some that are on drawing boards everywhere-from smart cars and homes that seek to anticipate a users every need, to the latest automatic navigational systems. Norman builds on this critique to offer a consumer-oriented theory of natural human-machine interaction that can be put into practice by the engineers and industrial designers of tomorrows thinking machines. This is a consumer-oriented look at the perils and promise of the smart objects of the future, and a cautionary tale for designers of these objects-many of which are already in use or development.

All rights reserved Norman and/or Basic Books Norman, Donald A. (2004). Design as communication. Retrieved 19 March 2012 from JND.org: http://jnd.org/dn.mss/design_as_communication.html Norman, Donald A. and Draper, Stephen W. (eds.) (1986): User Centered System Design: New Perspectives on Human-Computer Interaction. Hillsdale, NJ, Lawrence Erlbaum Associates Nris, Vnia Paula de Almeida, Almeida, Leonelo Dell Anhol, Miranda, Leonardo Cunha de, Hayashi, Elaine Cristina Saito and Baranauskas, M. Ceclia C. (2011): Collective Construction of Meaning and System for an Inclusive Social Network. In International Journal of Information Systems and Social Change (IJISSC), 2 (3) pp. 16-35

TopInformation and Communication Technology has the potential of

benefiting citizens, allowing access to knowledge, communication and collaboration, and thus promoting the process of constitution of a fairer society. The design of systems that make sense to the users community and that respect their diversity demands socio-technical views and an in-depth analysis of the involved parties. The authors have adopted Organizational Semiotics and Participatory Design as theoretical and methodological frames of reference to face this challenge in the design of an Inclusive Social Network System for the Brazilian context. This paper presents the use of some artifacts adapted from Problem

Articulation Method to clarify concepts and prospect solutions. Results of this clarification fed the Semantic Analysis Method from which this paper presents and discusses an Ontology Chart for the domain and the first signs of the inclusive social network system.

All rights reserved Nris et al. and/or their publisher O'Neill, Shaleph (2008): Interactive Media: The Semiotics of Embodied Interaction. Springer

The author discusses the existing theoretical approaches of semiotically

informed research in HCI, what is useful and the limitations. He proposes a radical rethink to this approach through a re-evaluation of important semiotic concepts and applied semiotic methods. Using a semiotic model of interaction he explores this concept through several studies that help to develop his argument. He concludes that this semiotics of interaction is more appropriate than other versions because it focuses on the characteristics of interactive media as they are experienced and the way in which users make sense of them rather than thinking about interface design or usability issues.

All rights reserved O'Neill and/or Springer Prinsloo, Mastin and Walton, Marion (2008): Situated Responses to the Digital Literacies of Electronic Communication in Marginal School Settings. In Yearbook 2008 African Media African Children, pp. 99-116

In this chapter we examine examples of young childrens encounters with

computers and the Internet in poorly resourced schools in an African setting. We argue that computerized and networked media resources operate in these settings in specific ways that are sometimes ignored in the discussion of digital divides and the call for the expansion of physical access to computers and the Internet. These local ways of using the digital resources of the media do not always fit with common assumptions about the value of such technology for enhancing learning in otherwise deprived or poorly resourced educational settings Universitet Repenning, Alexander and Ioannidou, Andri (2004): Agent-based end-user development. In Communications of the ACM, 47 (9) pp. 43-46

All rights reserved Prinsloo and Walton and/or Nordicom, Geborgs

Saussure, Ferdinand de (1972): Cours de linguistique gnrale. Payot Schn, Donald A. (1988): Educating the Reflective Practitioner: Toward A New Design for Teaching and Learning in the Professions. San Francisco, Jossey-Bass Schn, Donald A. (1983): The Reflective Practitioner: How Professionals Think In Action. Basic Books

A leading M.I.T. social scientist and consultant examines five professions

engineering, architecture, management, psychotherapy, and town planningto show how professionals really go about solving problems.The best professionals, Donald Schn maintains, know more than they can put into words. To meet the challenges of their work, they rely less on formulas learned in graduate school than on the kind of improvisation learned in practice. This unarticulated, largely unexamined process is the subject of Schns provocatively original book, an effort to show precisely how reflection-in-action works and how this vital creativity might be fostered in future professionals. All rights reserved Schn and/or Basic Books Souza, Clarisse Sieckenius de de (2005): The Semiotic Engineering of HumanComputer Interaction (Acting with Technology). The MIT Press

In The Semiotic Engineering of Human-Computer Interaction, Clarisse

Sieckenius de Souza proposes an account of HCI that draws on concepts from semiotics and computer science to investigate the relationship between user and designer. Semiotics is the study of signs, and the essence of semiotic engineering is the communication between designers and users at interaction time; designers must somehow be present in the interface to tell users how to use the signs that make up a system or program. This approach, which builds on -- but goes further than -- the currently dominant user-centered approach, allows designers to communicate their overall vision and therefore helps users understand designs -rather than simply which icon to click.According to de Souza's account, both designers and users are interlocutors in an overall communication process that takes place through an interface of words, graphics, and behavior. Designers must tell users what they mean by the artifact they have created, and users must understand and respond to what they are being told. By coupling semiotic theory and engineering, de Souza's approach to HCI design encompasses the principles, the materials, the processes, and the possibilities for producing meaningful interactive computer system discourse and achieves a broader perspective than cognitive, ethnographic, or ergonomic approaches.De Souza begins with a theoretical overview and detailed exposition of the semiotic engineering account of HCI. She then shows how this approach can be applied specifically to HCI evaluation and design of online help systems, customization and end-user

programming, and multiuser applications. Finally, she reflects on the potential and opportunities for research in semiotic engineering.

All rights reserved Souza and/or The MIT Press Spool, Jared M. and Schaffer, Eric M. (2005): The great debate: can usability scale up?. In: Proceedings of ACM CHI 2005 Conference on Human Factors in Computing Systems 2005. pp. 1174-1175 Umiker-Sebeok, Jean (ed.) (1988): Marketing and Semiotics: New Directions in the Study of Signs for Sale. Mouton De Gruyter Wing, Jeannette M (2008): Computational thinking and thinking about computing. In Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 366 (1881) pp. 3717-3725

Computational thinking will influence everyone in every field of endeavour.

This vision poses a new educational challenge for our society, especially for our children. In thinking about computing, we need to be attuned to the three drivers of our field: science, technology and society. Accelerating technological advances and monumental societal demands force us to revisit the most basic scientific questions of computing.

All rights reserved Wing and/or his/her publisher Winograd, Terry (1996): Bringing Design to Software. ACM Press

This book aims to illuminate and stimulate the discipline of software design.

Collecting insights and experience from experts in diverse fields, it addresses the growing demand that the software industry produce software that really workssoftware that fits people and situations far better than the examples we see today. With Terry Winograd's introductory framework to guide readers through thoughtful essays, perceptive interviews, and instructive profiles of successful projects and programs, the book explores the issues and concerns that most directly influence the functionality, usability, and significance of software. Contributors include some of the most prominent names in the computing and design fields. Programming Languages Survey/Compilers

All rights reserved Winograd and/or ACM Press

Winograd, Terry and Flores, Fernando (1986): Understanding Computers and Cognition. Norwood, NJ, Intellect Zemanek, Heinz (1966): Semiotics and programming languages. In Communications of the ACM, 9 (3) pp. 139-143

Changes to this chapter

24 Apr 2012: Added

About this chapter Author(s): Clarisse Sieckenius de Souza

Commentaries by: Alan Blackwell This encyclopedia chapter has undergone double-blinded peer-review by two reviewers based

on the reviewing

guidelines, language copy-editing, typesetting, and reference validation. How to cite/reference this page
URL: http://www.interaction-design.org/encyclopedia/semiotics_and_human-computer_interaction.html Open Access We believe in Open Access and the democratization of knowledge. Unfortunately, world class educational materials are normally hidden behind payment systems or in expensive textbooks. If you want this to change, you should help

us out! Kind thoughts are not enough - you need to act!