11. Philosophy of Interaction
Over the last two decades, interaction design has emerged as a design discipline alongside traditional design disciplines such as graphics design and furniture design. While it is almost tautological that furniture designers design furniture, it is less obvious what the end product of interaction design is. Löwgren's answer is "interactive products and services" (Löwgren 2008). This narrows it down, but leaves open the question of what it means for something to be interactive.
Interactive systems have been studied within the field of Human-Computer Interaction since the early 1980s. This research has given us valuable knowledge about users, systems and design methodology, but few have asked "philosophical" questions about the very nature of interactivity and the interactive user experience.
I will approach the question of interactivity from a number of angles, in the belief that a multi-paradigmatic analysis is necessary to give justice to the complexity of the phenomenon. I will start by defining the scope through some examples of interactive products and services. Next, I will analyse interactivity and the interactive user experience from a number of perspectives, including formal logic, cognitive science, phenomenology, and media and art studies. A number of other perspectives, e.g. ethnomethodology, semiotics, and activity theory, are highly relevant, but are not included here. (For an analysis that includes these perspectives, see (Svanaes 2000)).
The Merriam-Webster dictionary defines interaction as "mutual or reciprocal action or influence". Taking this definition as a starting point, what is the meaning of interactive and interactivity? A product or service is interactive if it allows for interaction. An artefact's interactivity is its interactive behaviour as experienced by a human user. Or to be more precise, it is the potential for such experiences. Its interactivity is a property of that artefact; alongside other properties like its visual appearance. Interactivity can also be used as a noun to signify everything interactive, similar to how radioactivity refers to everything radioactive.
Many definitions exist for "the user experience". I prefer this one: "a person's perceptions and responses that result from the use or anticipated use of a product, system or service" (ISO 2009)
11.2 The scope: Interactive products and services
What makes a product or service interactive? One of the simplest interactive products imaginable is a touch-sensitive light switch like the one in Figure 1A. You touch it once to turn the light on, and again to turn the light off. At the other end of the complexity scale you find interactive products like the cockpit of a modern aeroplane (Figure 1B); allowing trained pilots to fly the plane through a number of input devices and visual displays.
Examples of interactive services include internet banking, online shopping, and social media, all made possible through networked digital devices like PCs (1C) and mobile phones (1D). All above examples are interactive. Are there digital products that are not interactive?
If you solder a light bulb to a battery and leave it on your desk until the battery is drained (Figure 2A), this digital product can hardly be called interactive. You can of course turn it off by cutting one of its wires, but that would not be an intended interactivity of the product. The light bulb could be substituted with something far more complex, like a digital photo frame that was programmed to generate random fractals on a screen (Figure 2B). With no buttons, handles or other means for interacting, despite its complex behaviour, neither that product would be interactive. It would be like the 1957 Sputnik 1 satellite (Figure 2C), which contained a "transmit only" radio beacon that transmitted beeps from space for 20 days until its batteries ran out.
From the above examples it becomes clear that what makes a product or service interactive is not its complexity, nor the fact that it is digital, but whether it is designed to respond to actions by a user.
11.3 Formal descriptions of interactive behaviour
One way of describing the interactive behaviour of a product or a service is through a formal representation. A number of such formalisms exist, the simplest being state diagrams. A state diagram is a visual representation of a Finite State Machine.
Figure 3A shows a state diagram for the touch-sensitive light switch in Figure 1A. It contains two states, "Light off" and "Light on", and two user-initiated transitions between the states ("Touch switch"). The black dot leading in to the "Light off" state tells us that this is the initial state, i.e. the light starts out in the "off" state.
In Figure 3B, we see the state diagram for the non-interactive "Bulb-with-battery" device. It starts with the light being on and stays in that state until the battery is drained. The black dot in a circle is the "game over" symbol. When the battery is drained, the device stops being what it was intended to be.
A number of more sophisticated formalisms have been used for describing interactive behaviour, including Harel's hierarchical state diagrams (Harel 1987), temporal logic (Hartson and Gray 1992), Petri nets (Elkoutbi and Keller 2000) and algebra (Thimbleby 2004).
Formal representations of interactive behaviour are well suited to describe the technical side of interactivity, but say little of the human side. They are of little value in answering questions like: "How is the interaction experienced?", "What does the interaction mean to the user?" To be able to answer such questions about the interactive user experience, we have to leave formal logic and the natural sciences and turn to the humanities and the social sciences.
11.4 Cognitive science: Interaction as information processing
Since the birth of Human-Computer Interaction (HCI) as a scientific discipline in the 1980s, cognitive science has been the dominant paradigm for describing the human side of the equation. "The Psychology of Human-Computer Interaction" by Stuart K. Card, Thomas P. Moran and Allen Newell (Card et al 1983) presented a model of the user based on an information processing metaphor (Figure 4). Here, the interaction is modelled as information flowing from the artefact to the user, where it is processed by the user's "cognitive processor", leading to actions like pushing a button. Their model sees interaction as the sum of stimuli reception and user actions.
Imagine the user in Figure 4A operating the light switch in Figure 1A. The act of turning on the light switch would be modeled as information about the state of the light reaching the perceptual processor through the user's eyes, where it would flow to the working memory, and be processed by the cognitive processor. A command would then be sent to the motor processor, leading to the hand pushing the switch.
In "The Design of Everyday Things" (Norman 1988), Don Norman elaborates the details of what is going on as a seven step "action cycle" (Figure 4B). Returning to our user in front of the light switch in Figure 1A, Norman would describe this as the user having the goal of turning the light on (step 1). This goal would lead to an intention to act (step 2), leading to a sequence of actions being sent to the motor processor (step 3), where it would trigger a hand movement (step 4). In "the world", the light would turn on, and this would be perceived (step 5) and interpreted (step 6) by the user. Finally, the user would evaluate the new state of the light as a fulfillment of the goal (step 7), and be ready for a new action cycle. The action cycle is described by Norman in the following video.
Based on their model of the user, Card, Moran and Newell devised a framework for predicting user behaviour called GOMS (Goals, Operators, Methods, and Selection Rules). A number of GOMS-inspired cognitive frameworks have since been developed to model the behaviour of the user, all based on the same basic assumptions of the human information processing model.
GOMS-like models have been successful in predicting key-level human behaviour for routine tasks, but have shown little explanatory and predictive power when it comes to more open tasks, like updating your Facebook profile. Further, they are of little help in understanding the interactive user experience.
In "The Design of Everyday Things", Don Norman introduced the concept of affordance that had been developed by the psychologist J.J. Gibson. Norman defines affordance: "...the perceived and actual properties of the thing, primarily those fundamental properties that determine just how the thing could possibly be used". While Gibson's ecological approach to human cognition and perception in many respects is incommensurate with the information-processing approach, the affordance concept has mainly been interpreted within HCI to describe what functions an object allows for, and how this is "signalled" through its visual appearance. Norman illustrates the affordance concept in the video below. Adding the concept of affordance to the framework, the light switch in Figure 1A would appear to the user as an object that affords turning the light on and off.
A number of researchers in HCI have argued that the information-processing model reduces the user to a mechanical symbol-processing machine, leaving out important aspects of what defines us as human. One of the earliest criticisms of the information processing approach to human-computer interaction was voiced by Stanford professor Terry Winograd and Fernando Flores in their influential book "Computers and Cognition" (Winograd and Flores 1986). The book was primarily written as a criticism of artificial intelligence and cognitive science, but has strong relevance for a discussion of interactivity.
Winograd and Flores presented three alternatives to cognitive science, of which the phenomenology of the German philosopher Martin Heidegger (1889-1976) is the most relevant here.
11.5 Heidegger: Interaction as tool use
Winograd and Flores argue in "Computers and Cognition" that cognitive science takes for granted that human cognition and communication are symbolic, and that symbols like "cat" refer in a one-to-one manner to objects in the world. Heidegger's philosophy of being (Heidegger 1996; original version is Heidegger 1927) rejects this view and starts out with our factual existence in the world and the way in which we cope with our physical and social environment. His philosophy spans a wide range of topics, of which Winograd and Flores mainly use his analysis of tools. Heidegger used a carpenter and his hammer as an example (Figure 5B). Winograd and Flores argue that a computer can be viewed as a tool: For skilled users of computers, the computer is transparent in use - it is ready-to-hand. When I write a document in a text editor, my focus is on the text and not on the text editor. If my text editor crashes, my focus is moved from the text that I am working on to the text editor itself. It is only when we have a breakdown situation and the computer stops working as a tool that it emerges as an object in the world - it becomes unready-to-hand. If we are not able to fix the problem that causes the breakdown, it becomes present-at-hand.
Heidegger would describe the light switch in Figure 1A as a tool for controlling the light. As part of our everyday life, a light switch is an integral part of our background of readiness-to-hand, and the interaction with the switch is to some extent invisible to us. It is only when the switch stops working as expected, or when we consciously chose to reflect on it, that it emerges from the background as an object.
Heidegger does not deny the fact that the light switch exists in the world as an object to be viewed, touched and manipulated. His point is that the essence of the switch only emerges through use. Its "switchness" is hidden for us until we put it into use. An important aspect of its "switchness" is that it allows for a certain kind of interaction. When the ape in Kubrick's 2001: A Space Odyssey (Figure 5C) realises that the piece of bone in front of him can be used to crack things, the "hammerness" of the bone emerges to him - and bones forever stop only being bones. The bone's "hammerness" had been there all the time, but it needed to be put into practice to emerge. Similarly with the interactivity of a light switch - its "switchness" emerges through use.
From a Heideggerian perspective, the specific meaning of the interaction with the light switch depends on the use situation and the user's intention. Turning the light on as part of my everyday action of entering a room is different from turning the light on to see if the switch works. In the first case the interaction is part of a wider goal, while in the second case it would be a goal in itself. Cognitive science would miss this subtle difference, as it would model both interactions as the same goal-seeking information processing behaviour. Heidegger would also argue that to be able to understand how an interaction is meaningful for a specific user, we would have to understand the lifeworld of that user, i.e. the cultural and personal background that serves as a frame of reference and context for every experience of that person.
Heidegger further argues that tools exist in the shared practice of a culture as part of an equipmental nexus , e.g. hammers with nails and wood. The hammer gets it significance through its relation to nails and wood, as the nail get its significance through its relation to hammer and wood. The elements form a whole, and each element gets its significance from its role in this whole.
11.6 Merleau-Ponty: Interaction as perception
Maurice Merleau-Ponty (1908-1961) was, besides Jean Paul Sartre, the most influential French philosopher of the 1940s and 1950s. Inspired by Heidegger, Merleau-Ponty stressed that every analysis of the human condition must start with the fact that the subject is in the world. This being-in-the-world is prior to both object perception and self-reflection. To Merleau-Ponty, we are not Cartesian self-knowing entities detached from external reality, but subjects already existing in the world and becoming aware of ourselves through interaction with our physical environment and with other subjects.
In his major work, "The Phenomenology of Perception" (original: Merleau-Ponty 1945; Translated: Merleau-Ponty 1962), Merleau-Ponty performs a phenomenological analysis of human perception. His purpose is to study the "precognitive" and embodied basis of human existence. He ends up rejecting most of the prevailing theories of perception at his time. In all his writing there is a focus on the first-person experience. Merleau-Ponty rejected the idea of perception as a passive reception of stimuli. When we perceive objects with our eyes, this is not a passive process of stimuli reception, but an active movement of the eyeballs in search of familiar patterns. This view is in total opposition to the popular view in "information-processing" HCI that sees perception as sense data being passively received by the brain. To Merleau-Ponty there is no perception without action; perception requires action.
Perception hides for us this complex and rapid process going on "closer to the world" in "the pre-objective realm". Modern eye trackers allow us to see these rapid perceptual interactions unfold in vision. The optics of the human eye is such that we see the world through a rapidly moving peephole. In Figure 6 we see the rapid eye movements of two different persons viewing the same painting, a non-artist (6A) and a trained artist (6B). We see how the artist rapidly scans the whole painting (Figure 6B), while the layperson mostly focuses on the face of the girl (Figure 6A) (from (Vogt and Magnussen 2007)). The result of their different viewing styles is that they actually see different paintings. Merleau-Ponty uses the term phenomenal field to denote the personal background of experiences, training, and habits that shapes the way in which we perceive the world.
Merleau-Ponty saw perception as an active process of meaning construction involving large portions of the body. The body is, a priori, the means by which we are intentionally directed towards the world. When I hold an unknown object in my hand and turn it over to view it from different angles, my intentionality is directed toward that object. My hands are automatically coordinated with the rest of my body and take part in the perception in a natural way. Any theory that locates visual perception to the eyes alone does injustice to the phenomenon. To Merleau-Ponty, the body is an undivided unity, and it is meaningless to talk about the perceptual process of seeing without reference to all the senses, to the total physical environment in which the body is situated, and to the "embodied" intentionality we always have toward the world.
The body has an ability to adapt and extend itself through external devices. Merleau-Ponty used the example of a blind man's stick to illustrate this. When I have learned the skill of perceiving the world through the stick, the stick has ceased to exist for me as a stick and has become part of "me". It has become part of my body and at the same time changed it.
Applied to an analysis of interactivity, Merleau-Ponty invites us to see interaction as perception. If I test out the light switch in Figure 1A to see if it works, this interaction can be seen as a perceptual act involving both eyes and hand. I move my hand to the switch as part of the process of perceiving its behaviour, in the same way as my eyes make rapid eye movements when I see a painting. The hand movements towards the switch result from my directedness towards the object of perception, i.e. the behaviour of the switch.
In more complex interactions, like when an experienced computer user plays World-of-Warcraft, the perceiving body extends into the game. When the gameplayer tries out a new sword that she has acquired for her game character, she perceives its working through the mouse and the part of the software that let her control her character. Playing World-of-Warcraft is similar to riding a bicycle or driving a car in that the technology becomes a tool, but it differs in that the world is computer generated.
The integrated view of action and perception makes Merleau-Ponty an interesting starting point for a discussion of meaningful interactive experiences. A consequence of his theory is that it should be possible to lead users into interactions with the computer that are meaningful at a very basic level. The interactions themselves can be meaningful.
The interactive artefact in Figure 7 exemplifies this. Try it by clicking on the "Mr. Peters" button!
The button has a script that makes it jump when the cursor is moved over it. The user tries to click on the button, but experiences that "Mr. Peters" always "escapes". Most users understand the intended meaning of the example and describe Mr. Peters as a person who always avoids you, a person you should not trust. The interaction itself works as a metaphor for Mr. Peters' personality. How does the philosophy of Merleau-Ponty shed light on this example?
11.6.1 Key points
220.127.116.11 Perception requires action
- Perception of the "Mr. Peters" button requires action. The button as interactive experience is the integrated sum of its visual appearance and its behaviour. Without action, we are left with the visual appearance of the button, not the actual button as it emerges to us through interaction.
18.104.22.168 Perception is an acquired skill
- One of the necessary conditions for the Mr. Peters example to work is that the user has acquired the skill of moving the mouse cursor around. This skill (Merleau-Ponty: habit) is part of being a computer user. Without this skill, the only perception of the Mr. Peters button would be its visual appearance.
22.214.171.124 Tool integration and bodily space
- For the trained computer user, the mouse has similarities with the blind man's stick. The physical mouse and the corresponding software in the computer are integrated into the experienced body of the user. The computer technology, and the skills to make use of it, changes the actual bodily space of the user by adding to the potentials for action in the physical world also the potentials for action presented by the computer. The world of objects is in a similar manner extended to include also the "objects" in the computer.
126.96.36.199 Perception is embodied
- Experiencing the "Mr. Peters" button requires not only the eye, but also arm and hand. Mouse movements and eye movements are integrated parts of the perceptual process that lead up to the perception of the button's behaviour. The interactive experience is both created by and mediated through the body.
188.8.131.52 Intentionality towards-the-world
- As a skilled computer user, I have a certain "directedness" towards the computer. Because of this intentionality, the Mr. Peters button presents itself to me not only as a form to be seen, but also as a potential for action with an expectation for possible reactions. From seeing the button to moving the cursor towards it, there is no need for a "mental representation" of its position and meaning. The act of trying to click on the button is part of the perceptual process of exploring the example. When the button jumps away, I follow it without having to think.
184.108.40.206 The phenomenal field
- In the above example, the context of the button is given by the leading text and by the user's past experiences with graphical user interfaces. It is important to notice that this example only works with users who are used to clicking on buttons to find more information. This is the horizon of the user, i.e. the phenomenal field that all interaction happens within. The Mr. Peters button emerges as a meaningful entity because the appearance of a button on a computer screen leads to a certain expectation and a corresponding action. The action is interrupted in a way that creates an interactive experience that is similar to that of interacting with a person who always escapes you.
With Merleau-Ponty it becomes meaningless to talk about interaction as the sum of stimuli reception and action as cognitive science tells us. Interaction is better described as a kind of perception. I perceive the behaviour of the "Mr. Peters" button through interaction. This perception involves both hand and eye in an integrated manner. Interaction-perception is immediate and "close to the world".
11.7 A media and art perspective
While phenomenology can help us understand the interactive user experience for a specific product, and might help us choose between two or more alternative designs, it gives us little guidance on what designs are possible. To be able to fully utilize the potential of interactive media, it is important to have a deep understanding of the medium itself. There is a tradition in Media and Art Studies for asking questions concerning the nature of the medium being studied. However, compared to the vast literature on the social and cultural impact of new media, media studies with a focus on the properties of the medium itself are rare. The most prominent author on this subject is Rudolf Arnheim (1904-2007). Arnheim dealt with non-interactive media like film, painting, drawing, sculpture, and architecture, and he analysed their media-specific properties from an artistic and psychological perspective. In the introduction of "Art and Visual Perception: a Psychology of the Creative Eye" (Arnheim 1974), he states explicitly that he is not concerned with the cognitive, social, or motivational aspects. Nor is he concerned with "the psychology of the consumer". By ignoring all elements of social function and meaning in a traditional sense, he was free to discuss issues such as balance, space, shape, form, and movement in relation to the different media. Arnheim draws heavily on examples from art and gestalt psychology. What is relevant for the current study is not his results, but his approach to the study of a new medium.
We find a similar approach to studying a medium in artist and Bauhaus teacher Johannes Itten's theory of colours.
In Figure 8 we see some of the coloured squares that he drew to illustrate how the perception of colour changes with the background (Itten 1974).
Itten brought to the Bauhaus School of Design the idea that students of design should develop a deep knowledge of their media and materials through explorations of their properties. Seen as a medium and a material, the modern computer can be viewed as a display of pixels that each can have only one colour at any given time. Through some input device(s), the user can interact with this matrix of pixels. In Figure 9 we see a white and a black pixel enlarged.
The two pixels in Figure 9 are static in the sense that they do not respond to input from the user. An interactive pixel is a pixel that responds to user actions. The simplest "artefact" of this kind is a pixel that changes colour when clicked on. The pixel in Figure 10B has what we call "push button" behaviour. Figure 10A shows its state diagram.
The two interactive artefacts in Figure 11 look the same, but differ in behaviour. The "user experience" of an interactive artefact is the sum of its visual appearance and its interactive behaviour. The behaviour can only be experienced through interaction, and requires an active user. The fact that the pixel in Figure 11A is a "push button" and the pixel in Figure 11B is a "toggle" can only be perceived through interaction.
Borrowing from gestalt psychology, I use the term interaction gestalts for these kinds of basic interactive user experiences (Svanaes 1993).
If Figure 11A were placed in an art gallery, such an artefact would require the interactive behaviour to be perceived through use. Figure 12 illustrated this. A detached observer would miss the essence of this piece of minimalist interactive abstract art.
The artefacts in Figure 13 each consist of two pixels. Their behaviours are so simple that we can take them in as wholes, i.e. as interaction gestalts.
The previous artefacts all had the response located to the pixel you click on. By allowing pixels to affect each other, we get more complex artefacts. The artefacts in Figure 14 illustrate this. When clicked on, Figure 14A creates a foreground and a background; a black square moving on a white background. In Figure 14B the white square is active. This creates the illusion of a white square moving on a black background.
With three pixels like in Figure 15, the spatialisation becomes clearer. This simple swapping of colours is what happens at the pixel level when objects move on the computer screen.
The three-pixel artefact in Figure 16B has three states. As the artefact does not create the illusion of foreground/background, we perceive the behaviour as a rotation between states. It starts out all white, and after two states we are back "home" where we started. Figure 16A shows its state diagram. This kind of "state space" is the perceptual basis for the World-Wide-Web metaphor of moving between web pages, each web page being a state of the screen's pixels.
Real computers do not have two pixels, but millions. Figure 17 shows an example of how a matrix of elements with simple behaviour can become something potentially useful. Even with as few as 5x5 pixels you can use it to make basic shapes, i.e. an icon editor.
These kinds of explorations of interactive media can be extended in different directions. All the above examples have been with only two colours and discrete states. If we include analogue input, full colour, hidden states, delays, animations, sound, algorithms and communication through Internet, we get enough complexity to justify a whole new profession: Interaction Design.
11.8 Implications for interaction designers
What are the implications of this for interaction designers? We have gone through four perspectives on interactivity: interaction as information processing (Cognitive Science), interaction as tool use (Heidegger), interaction as perception (Merleau-Ponty), and interaction from a media and art perspective (Arnheim/Itten). The focus of the current analysis has mostly been on the interactivity of digital products. The implications for design will consequently mainly be related to the design of interactivity:
11.8.1 Interactivity is an important part of the user experience
- Combining the perspectives of Merleau-Ponty and Arnheim, it becomes evident that the interactivity of digital artefacts is perceived as interaction gestalts at a very immediate level, similar to that of visual perception. Interactivity is not simply the behaviour necessary to implement a certain functionality, but an important quality of a digital product. We often talk about the "look and feel" of a digital product. Users perceive "the feel" of the product through interaction, and this thus becomes an important part of the resulting user experience. Users care about "the feel".
11.8.2 A product's "feel" should be designed with care to detail
- If you want to design interactive products that stand out, you must be conscious of the interactive qualities of your products. "The feel" should consequently be designed; not only engineered. As with everything designed, God is in the details also concerning interactivity.
11.8.3 Perception of "the feel" requires action: Invite the user in
- Perception of the interactive dimension of a product, its “feel”, requires user actions. It is therefore important that the product signals its potentials for interaction, its “affordances”. The user experience is created through interaction. Interactivity that is hidden away is like a tree falling in the forest with no one watching.
11.8.4 Design for unforeseen use
- From a phenomenological perspective, objects get their meaning through use and social interaction. As Norman pointed out in his video on affordances, a good product should be designed for use situations other than the one intended.
11.8.5 Design for "the lived body"
- When designing for interaction beyond mouse and keyboard (e.g. mobile and whole-body interaction), design for the experienced body. This requires a focus on the bodily feeling of using the product.
11.8.6 Take responsibility for the feel of the total user experience
- In some cases, the user experience is a result of the user’s interaction with a number of interconnected products and services, e.g. the combination of earplugs, an MP3 player, and an online music store. In those cases it is important that the "feel" of these products and services are designed in such a way that the sum give rise to a good user experience.
11.8.7 Interaction designers need to learn basic programming skills
- Designing interactivity requires the ability to make rapid "sketches" of interactive behaviour. This is important to be able to explore different behaviours, and to have running specification to hand over to the programmers. Despite numerous attempts to make the process of designing interactivity non-technical, interaction designers who want to add an extra quality to the interactive user experience still need to learn basic programming skills. Programming is the tool for shaping interactivity. A lot can be done with simple programming environments like Processing and Arduino.
11.8.8 The perceptual field is personal: Test with real users and listen carefully
- Make numerous sketches of interactive behaviour, but always test them on real users before you make important design decisions. Different users can perceive the same interactive behaviour in surprisingly different ways; and often very differently from you. This is not only because they interpret and experience their interactions differently, but also because their ways of interacting differ. Test for more than usability; ask them how it feels in use and listen carefully to what they tell you.
11.8.9 Intentionality and context matters: Make the tests realistic
- The perception of a product's interactivity is to a large extent coloured by the user's intention. Trying out "the feel" of a product in a controlled setting is very different from using it in a real context and for a real purpose. Ideally, tests should be done with real tasks and in real contexts. If that is not possible, you should be aware of the difference between real use and your test setting, and how this colours the user experience.
11.8.10 Interaction designers should be skilled in kinesthetic thinking
- The interactivity of digital products should be designed by interaction designers with a special sensitivity for "the feel" of a product. "The feel" is about action-reaction; it involves the whole body and it is about timing and rhythm. This requires the interaction designer to develop skills in kinesthetic thinking and bodily intelligence. While drawing classes are excellent for designing "the look", interaction designers should also consider classes in dance, drama or martial art to develop their sensitivity for things interactive.
Getting the feel right is of course not sufficient to make successful interactive products, but it is my belief that in a competitive market, products with a well designed feel will always stand out; interactivity matters.
11.9 Where to learn more
11.9.2 Relevant papers
Moen, Jin (2005): Towards people based movement interaction and kinaesthetic interaction experiences. In: Bertelsen, Olav W., Bouvin, Niels Olof, Krogh, Peter Gall and Kyng, Morten (eds.) Proceedings of the 4th Decennial Conference on Critical Computing 2005 August 20-24, 2005, Aarhus, Denmark. pp. 121-124
Ozenc, Fatih Kursat, Kim, Miso, Zimmerman, John, Oney, Stephen and Myers, Brad A. (2010): How to support designers in getting hold of the immaterial material of software. In: Proceedings of ACM CHI 2010 Conference on Human Factors in Computing Systems 2010. pp. 2513-2522
Schiphorst, Thecla (2007): Really, really small: the palpability of the invisible. In: Proceedings of the 2007 Conference on Creativity and Cognition 2007, Washington DC, USA. pp. 7-16
Sundstrom, Petra and Hook, Kristina (2010): Hand in hand with the material: designing for suppleness. In: Proceedings of ACM CHI 2010 Conference on Human Factors in Computing Systems 2010. pp. 463-472
11.10 Commentary by Donald A. Norman
It is clear that our understanding of the nature of interaction has been evolving in a practical sense, in terms of our scientific understanding, and in terms of the underlying philosophy. Dag Svanaes has done an excellent job of describing the changes in philosophical approaches to the study of interaction that have occurred in the last few decades. But the review feels as if it stopped at the end of the 20th century, but today we are well into the 21st century. So the changes described in the article have continued: we are still evolving our understanding, still making progress.
From my point of view, the study of interaction is far ranging, including all the ways by which we interact with the environment. As Svanaes points out, interaction means the interplay between the state of the environment and the actions of agents (which are usually people), where the actions change the state of the world. Interaction can be accidental or purposeful, goal-directed or deliberately non-intentional.
Although the definition provided by Svanaes is appropriately broad, the chapter and its examples are surprisingly limited to discussions of the way by which individuals interact with information systems. The chapter primarily reflects single individuals interacting with computational systems. Pity: interaction is far broader. Interaction designers must deal with groups of people, sometimes separated by time and or distance, with non-computational objects and with the natural world. Designers who are designing services, rules, and procedures must be concerned with how systems and services interact with employees, technical staff, and of course customers and others who make use of the services. Some interaction designers have to be concerned with interactions among machines. The principles of interaction design are far more important to be limited to single individuals interacting with information systems and displays.
Interaction design is critically important for lots of non-computer systems. Thus, skis are a tool for interaction, as are baseball and cricket bats, the rules of engagement of soccer (football), and the way by which we steer and control our automobiles. Interaction designers must work with a wide variety of technologies, not just those of information and computer systems.
Our philosophical basis for the understanding of interaction has been evolving rather dramatically. As Svanaes points out, in the early days of personal computers, which is where interaction design started, we took a strict, internal-processing, information processing point of view to describe the system and to develop rules for designers. This approach has been seminal in teaching us about affordances (and what I now call “signifiers”), conceptual models, and the role of feedback, appropriate mapping, and all the many other formal, theoretical principles that have resulted from these studies. However, this approach is also very narrow, limiting, and now widely considered either to be too restrictive or, by some, simply wrong.
Today, our understanding of human cognition is undergoing major changes. We no longer look at pure information processing. Instead, we know that emotion and cognition interact in fundamental ways, that the entire human body interacts with the environment, and that as a result we must come to understand the whole system. It is not enough to use limited input and output mechanism. Hence, the development of mobile systems that are location, position, and orientation sensitive, where the entire body is involved, where the gestures, posture, and movements of the entire body are taken as inputs to our systems and where the results are not simply changes in some image on a screen, but changes that envelop the body, surrounding it with sights, sounds, and feelings, exciting the entire panoply of sensory systems. Haptics and gestures are missing from this article as are simulators that move in space to engage the proprioceptive systems. Three-dimensional sound is as important as three-dimensional sight. Haptics is increasingly important.
The review of affordances in the article needs to be expanded. Affordances are about possibilities: an affordance is a relationship between a person and the environment. But it is not affordances that are critical to interaction, it is our perception of those affordances. After all, an affordance that is not known about might as well not exist. To make this distinction more clear, in my recent work I call the perceptual signal of an affordance a “signifier.” (D. A. Norman, 2011: Living with Complexity. MIT Press.) Signifiers are the clues for people about possible actions, hence possible interactions.
I don’t know where Svanaes got the notion that the popular view of information processing in HCI is that it “sees perception as sense data being passively received by the brain.” (In his section on Merleau-Ponty). The notion that perception is an active, constructive process is an old one in psychological theories and it is certainly dominant today. We don’t follow Merleau-Ponty: we follow where the data have taken us. But Gibson, for example, was a strong advocate of the active nature of exception, and that was 50 years ago.
The view of HCI and of cognitive science held by Svanaes seems rooted in the last century.
Today, we study interaction as a fundamental property of human and social behavior, rooted in the entire body, where the person – or people – is inside a sensory field that is rich in information. All the senses are involved, not just the visual sense that is the focus of this chapter: haptics, proprioception, smell, sight, and sound. That is why the development of rich sensors and novel haptics are so encouraging. This is why we have surround sound, motor and solenoid-driven hairs and other objects to grasp, fondle, and manipulate. This is why medical simulators involve the entire body, allowing the novice surgeon to put the hand inside the simulated body, feeling the beating heart, the throbbing veins, and the different pressures and feelings associated with cutting through different layers of skin. Interaction today is a rich, vibrant filed.
Some of these topics are still in their infancy, so the definitive articles within this Encyclopaedia will have to wait a few years. I am disappointed, however, that these new developments and approaches are not discussed within the article – they aren’t even hinted at. So stay tuned for further news.
11.11 Commentary by Eva Hornecker
What is Interaction or Interactivity? Most HCI and Interaction Design textbooks do not really define (beyond more then one sentence) one of the most often used terms in our vocabulary. What seems so straightforward turns elusive when we try to dig deeper. We are in this situation probably because it is not evident where to turn to for definitions and theoretical underpinnings.
Different disciplines offer a large variety of definitions of interaction, ranging from phenomena of intersubjectivity and human sociality in social philosophy to system-theoretical approaches that analyze and model observable reciprocal action-effect relations and patterns. The word ‘interaction’ thus allows both for ‘human as information processor’ models, the kinds of formal state diagrams exemplified in this encyclopedia entry with a light switch, and 3d generation HCI approaches investigating the user experience of interaction. It even allows for anthropomorphic interpretations of human-computer interaction. Dag Svanaes provides a very good example of how we ascribe intentionality and agency to an animated object (the Mr. Peters button is evasive), even though the button itself merely acts on preprogrammed stimuli-response schemata. We perceive its behaviour – and our mind is basically desperately trying to see meaning in whatever we perceive and experience.
Svanaes’ article traces the history of HCI’s understanding of what happens on the user side of HCI. He steers the reader towards Phenomenology, starting with the Heideggerian view introduced by Winograd and Flores. The videos go into far more detail here – an important thought that seems missing from the written version is that ecologies of object derive meaning through each other.
The unique view that Svanaes contributes combines ‘interaction as perception-action’ (Merleau-Ponty) with elements of gestalt-perception, using very simple but powerful examples. He emphasizes how experiencing the ‘feel’ of interaction requires active engagement – the encyclopedia entry itself has interactive components that provide a first-hand experience. To experience this process, users must enter it; they need to actively interact in order to conceive the “dynamic gestalt” and the feel of interactive products. Feel is about dynamics, timing, and how the users’ body is part of the interaction. One of the unusual recommendations that arises from this thought is that interaction designers should learn some form of sport or exercise that trains kinesthetic thinking and bodily awareness.
Svaneas beautifully explains (and illustrates) the part of interactivity that results from the perception-action loop – but it is hard to scale this up from his little colour-shifting disobedient buttons to large-scale behaving, intelligent, ambient objects and environments, with interactions that extend over long periods of time (and/or space). It is a bit like we smashed a mirror and now inspect a small crystal, which in miniature allows us to experiment with reflections, but that we find hard to put together so we can see the whole picture. I don’t really have any recommendations on how to proceed from here – I think we are still only just beginning to understand what it means to have artificial interactive objects (that are not really living entities), and there may be further branches of philosophy to look at.
The view that Svanaes shares with many other contemporary interaction design protagonists is that the dynamics of interaction should be central in design. The experience of use is created through the interrelation of system behaviour and user activity. It is a dialogical and time-based, temporal process. I want to point out some of the literature in interaction design that I have found helpful for a better understanding of the longer-term process of interaction. These take a far more pragmatic and practical approach, without aiming to provide a philosophy of interaction, that may complement the approach taken in this article.
Crawford (2002) uses human conversation as a metaphor. It does not fully work, but turns out to be useful – the metaphor carried much further than I initially thought it was able to. It emphasizes how interactivity depends on having two active participants that contribute to the interaction and how the quality of interaction depends on what they are able to contribute, how well they understand each other, and what they can perceive of each other. The metaphor highlights that interaction can be skewed and dominated by one partner, or that one partner might be severely handicapped in their expressivity or ability to perceive and understand the other. Toni Robertson and her coworkers have done a similar analysis from a situated action perspective, analyzing what is available to the machine in an interaction process. Shedroff (2000) sees Interactivity as a continuum, between passive and interactive experiences. He points out aspects that make a system interactive, the most elemental components being system feedback and user control. Of more interest is that it often is the nature of the user activity that makes something interactive, e.g. in creativity tools or communication media. This means that a system that merely mediates messages between 2 points may be perceived as interactive, because it is an open system where the interactive component is ‘donated’ by the outside world, or, in the case of e.g. a drawing tool, by the user. Interaction transforms closed to open systems (cf. Wegner 1997) – even more so if interaction contributes content. Svanaes’ article highlights how even within the seemingly mundane level where we only have user control and system feedback (the Mr. Peters button) interactional meaning emerges. Crawford and Shedroff motivate and might help us to analyse longer and more complex processes – does Mr Peters always react the same way, can we engage in a dialogue with him and what else can we do?