This was an interesting chapter for me to try to understand and there is a banquet here for discussion, although I only have time to address one of the main dishes.

First, I want to say that I greatly appreciate the work of Kia Höök and others she cites to develop technologies for enhancing people’s awareness of affect and helping people better reflect on and understand emotions, of self and others. I also deeply appreciate the work of designers to address holistic situations and design for people, including their feelings but also never only their feelings. These goals – creating interactions and designs that enhance affective understanding and that respond to the richness of human needs – are truly significant for improving much of what it means to be human. That said, I would like to correct an important misconception. Let me start with a story.

It was 1999 and Joe LeDoux, Antonio Damasio, and I had been invited to give talks on Emotion & Knowledge for the Barcelona Museum. The talks were simultaneously translated into multiple languages, giving me time to speak carefully and slowly in English, relying on the hard work of people more talented than I to translate into Catalan, Spanish, French, and more. It was a great experience overall – meeting fascinating people and engaging deeply in topics that were new and stimulating. But, there was one negative part that stands out in my memory.  At the reception, a dark, trim, middle-aged man came striding in my direction, red-faced, furrowed brow, gesturing sharply, and having a hard time speaking. I’ve never seen somebody so angry in a museum. I glanced around me, thinking he was targeting somebody nearby who tried to steal his wife, after all, I was just nibbling on a canape. But his anger was at me. I swallowed, listened carefully, and gradually came to understand that in the language he was hearing my talk translated, he heard me claim something to the effect of “We have built or could now build human emotion into computers.” I was actually extremely careful to NOT say that, but in his mind, I was denying the special feelings and experience we have that accompany human emotion, and reducing the great riches of our emotional experience entirely down to something like a text editor or game app. Listening to him, I realized that my careful choice of words in English, to say what we were doing precisely, and what I thought could be done, was translated inaccurately from my engineering culture, to his culture, which was social psychology.

In that reception, I learned, painfully, that what I meant by “modeling” was very different than what he heard when I said that word. I learned it was not enough to just be very careful with my words stating what we’re doing. I needed to also anticipate how people from different fields could misinterpret what I said. I needed to learn to make additional clarifying remarks of what I did not mean.  I should have said not only, “These are some of the mechanisms related to emotion that we are able to implement,” but also, “These are not all of what emotion is.” I should have said not only “by ‘mechanisms of’ I mean ‘Attempts to represent’,” but also, “Representing is not the same as reproducing.” I did not realize he would otherwise be led to the wrong conclusion.
 
Why do I bring up this story? Höök’s article refers to the Affective Computing approach as cognitivistic and reductionist, which is quite similar to the misunderstanding that happened in Barcelona in 1999.

When I speak or write of mechanisms of emotion, or models of emotion, I speak as an engineer trying to represent a complex phenomenon as best we can with tools we have: I do not confuse these representations with emotion itself. I am not a reductionist and Affective Computing is not reductionistic. I do not believe that emotion can be reduced to these representations, nor does Affective Computing claim this. I do not believe that emotion is “nothing but” the mechanisms we identify and build. The mechanisms we implement are not equivalent to the riches of human emotional experience, nor have I ever said that they will be: We have no evidence to make such claims. If people want to believe that emotions are entirely reducible to logical computation and bits, then that belief is based on faith, not science.

While people can write about any concept using information and bits, including emotion, I do not see evidence supporting the view that emotion can be fully reduced to bits and information. When I wrote Affective Computing, I knew many readers would be from AI, and would want to know how emotion might be implemented in machines, and so I described the parts of that process that I could envision. I was also very careful in my wording to not promote that such a method would be sufficient. However, I had not yet encountered the man in Barcelona, so one has to read my words carefully.

Unfortunately, if a person’s views are multi-dimensional, people will try to reduce them to one dimension, and conveniently peg them on one hook or the other. The process is rather like tidying up the foyer by hanging each jacket on whatever hook happens to be available and strong enough to hold it up. Cognitivism is a handy hook, promoting the belief that thought can be fully reduced to rules and algorithms.

Cognitivism was a strong influence for AI pioneers like my friend and colleague, Marvin Minsky, who kept telling me “Emotions are just a special kind of thought”, a sentence I disagreed with him on regularly and once get him to at least compromise by removing the word “just”. Marvin believed bodies were irrelevant, except during infancy when people needed to be touched, else (studies showed) they withered and died. I have met other pioneers in AI who thought similarly. I am not of their camp, and my writings in Affective Computing talk about the body and about aspects of conscious experience that we haven’t a clue how to implement in information and bits. There are some researchers who work in Affective Computing who hold a cognitivist view, but Affective Computing is not cognitivist.

Yes, Affective Computing includes some models and some researchers whose work might fit on a cognitivist hook, e.g. the cognitive rule-based models like OCC’s could hang on a cognitivist hook for people who believe that approach could fully account for emotion (I don’t). The stochastic signal-representing models of affect in speech or facial expression dynamics might hang on a different hook, and there are other hooks as well. The closet can be better organized than I have taken time to write about, especially as new garments keep arriving.  But don’t confuse the hooks with the house.

For some supportive examples, see “Chapter 1: Emotions are Physical and Cognitive” in Affective Computing (1997), containing some of my earliest writings on the need for a combined body-mind view in emotion research. That clearly does not fit on the cognitivist hook. Similarly, readers might be interested in the emphasis I placed on machines continusly co-creating interactions with people, taking into account not only emotion but also context and more, which resonates with the other areas Kia Höök’s article attempts to delineate (more examples are in “Chapter 8: Affective Wearables, see sections such as “Out of the lab and into the world.”) An affective technology does not have to use a formal AI model of emotion, or use discrete emotion recognition or a pattern classifier to fall under the area of affective computing.

But enough about organizing. I think the splitting and naming of pieces of a pie – whether it is an “affective computing pie” or some other kind of pie, is not as interesting as another question I see lurking behind the drive of some designers to separate themselves from a more objective engineering approach: Are emotions fully describable or are they ineffable?

In our work we have described emotion computationally and semantically, in numerous ways – discrete, dimensioned, numeric, semantic, as well as by quantifying creative behaviors, facial expressions, signal measurements of physiology and more. In no case do I think that we have “fully captured” human emotion with our models, methods, or descriptions. Something remains undescribed.

Affective computing often (but not always) tries to describe, objectively, more about emotion than has ever been described subjectively. Much of my work has pushed to make concrete, precise, in an engineering sense, measures of things that previously had only been addressed with words, self-report, questionnaires, whether applied to internal feelings or to outwardly observed behaviors. I am bothered by the way all subjective measurement methods are themselves influenced by emotion, and I want something more objective. Objective measures, however, do not imply reductionism any more than subjective measures imply reductionism. Both approaches “reduce” emotion to something – words, numbers, pictures, “blobby objects.” Using a representation is not being reductionistic. Reductionism is when people take an additional leap and say our emotions are “nothing but” what the computer is representing. The latter leap is one I have never promoted (except through mistranslated remarks).

When I closed my conversation with the man in Barcelona, we realized we were both deeply interested in better understanding emotion, and we realized that our perceived differences were actually not differences at all. Efforts to model do not imply a view of reductionism. Working to build representations that imitate some functions of emotions based on rules and categories does not mean cognitivism. Implementation of affective measures in bits does not mean emotion is only information. Affective computing creates tools toward greater goals – toward greater understanding of what makes us human. The man and I exchanged a hearty handshake and a smile before he departed.

I still have a lot to learn about communicating what we are trying to do with emotion – it’s a big topic, and it’s not one that just an engineering approach can conquer. I’m thrilled, as an engineer, to be sharing the journey with people from social psychology, design, neuroscience, AI, as well as many other arts and sciences.  Together we’ll figure out much more than if we set up different camps.

The original definition I gave of affective computing is broader than the one Kia paraphrases: Computing (includes machines, robots, phones, sensors, smart clothing, anything that can do computation) that relates to, arises from, or deliberately influences emotion or other affective phenomena. This was never just about AI or HCI, or about making intelligent machines, although those were the largest communities I was trying to convince to work on emotion at the time.

Perhaps I can be permitted to close, using the opening I wrote in 1997, which still rings true today:

Affective computing is an exiting discipline and Kristina Höök offers us some nice examples of what the field can bring. Wouldn't it be great if intelligent machines could somehow 'sense' what we feel when interacting with them and then adjust their actions accordingly? This is actually what the pioneers of affective computing saw as their challenge and this is what they have been after:

1. Is it possible to recognize people's emotional responses/states from their behavior, physiological responses or facial expressions, and
2. Can we make the system (e.g. computer, product, mobile device) take this information into account in appropriate responses? Rosalind Picard will correct me if I am wrong that these were and still are the main challenges of the AC discipline.

Is this reductionist? Sure it is, you can only measure a few indicators of people's emotional responses and each and every indicator (e.g. pressure exerted, skin conductance, heart rate variability, facial muscles) only tells a little part of the story. There are many behavioral, physiological and psychological sides to an emotion and we simply cannot tap them all. But what is the alternative? We want our measurements to be as non-invasive as possible. If we end up affecting people's behavior or even change their emotional states because of the way we are measuring, the whole purpose is lost. Preferably, we measure user's emotional responses without them being aware of it. The question is what each and every indicator of an emotion actually tells us (its validity) and how accurately – and unobtrusively – we can measure it. A lot of the work in AC has been put into these questions.

As an alternative, Kristina Höök proposes the "Interactional Approach" where the system allows users to reflect on their emotional experiences. This, however, does not eliminate the measurement problem as we can for example see in the "Affective Diary". This application registers movement and arousal as indicators of people's emotional state and transfers these data into shapes and colors as a form of feedback. If you aim to give people feedback on their feelings this is the 'appropriate response' you decided on and you have moved to the second challenge of AC: how to respond? And of course, the response you aim for very much depends on the type and function of the system. When I am typing a document in Word – as I am doing now – I do not want the system to give me continuous feedback on my emotional states, nor do I want to see these reflected in the words I type. But I may want the system to recognize that I am in a hurry, or impatient, or stressed, and subtly 'make me' slow down, without me being aware of it. Miguel Bruns Alonso recently explored this idea in the design of a pen that senses implicit behaviors related to restlessness and responds by providing inherent feedback to lower the stress level (Bruns Alonso et al., 2011).

So, is there an alternative to the measurement problem? In another example from Kristina Höök's chapter, she describes eMoto, an extended SMS-service that allows people to communicate their feelings in colorful and animated shapes. This type of 'measurement' – making users express their own emotions in words or images – only works when we are dealing with communication devices and is problematic for different reasons. First of all, it is obtrusive and not very recommendable if communicating emotions is not the design goal. But also, there are validity problems in people's verbal or non-verbal reports of their own emotions. All kind of social rules, demand characteristics (of the device?), and response styles may interfere with a valid report of your own feelings (see e.g. Mauss and Robinson, 2009 for a review of emotion measures).

And yes, I fully agree with Kristina that the role of our body is relatively unexplored in the AC field and offers a lot of potential, both to the recognition and response challenge. Given recent advances in cognitive science (see e.g. Johnson, 2007), where bodily experiences are increasingly recognized as being at the roots of our thinking and feeling, we may expect more and more studies – like Bruns Alonso's – in which our body is the main mediator. How else can we "grasp" the affective domain?

References

• Bruns Alonso, M., Hummels, C.C.M., Keyson, D.V., & Hekkert, P. (Conditionally accepted). Measuring and adapting behavior during product interaction to influence affect. Personal and Ubiquitous Computing.
• Johnson, M. (2007). The meaning of the body: Aesthetics of human understanding. Chicago: The University of Chicago Press.
• Mauss, I.B. & Robinson, M.D. (2009). Measures of emotion: A review. Cognition and Emotion, 23, 209-237.

On the bodily expressions of emotion, be aware: More than a century of research!

When thinking about this commentary, ideas popped up and emotions emerged. What to comment on? Kia Höök delivered an excellent chapter. She mentions three angles to approach emotion in technology from (cf. Van den Broek, 2011), namely: affective computing, affective interaction, and technology as experience. In this commentary, I will narrow the focus to affective computing solely. Furthermore, I have also chosen to take a step back and be so bold as to take a methodological perspective with a historical flavor. Why? Well, throughout the years I have discovered more and more literature that touches the core of affective computing but appears to be unknown (e.g., Arnold, 1968; Candland, 1962; Dunbar, 1954). This commentary is founded on two books from a time long before the term affective computing was coined, the 50s and 60s of the previous century. Both books are taken from completely distinct branches of science. Knowledge on science’s history can prevent us, both practitioners and scientists, from repeating mistakes. As such, this commentary touches upon the essence of science itself.

Kia Höök provides a concise overview of emotion in technology. She embraced affective interaction instead of affective computing. In contrast, in this commentary, I have taken the affective computing standpoint. Moreover, Kia Höök has taken a design perspective, where this commentary touches upon and questions the fundaments of emotions in technology. Lessons had been learned but have already been forgotten (Arnold, 1968; Candland, 1962; Dunbar, 1954). Consequently, affective computing tends to reinvent the wheel, at least to some extent. Yes, this is a bold claim, a very bold claim but I hope that after reading this commentary, you as a reader may share my concerns.

In 1954, 5 years before her death, Flanders Dunbar delivered the fourth edition of “Emotions and bodily changes: A survey of literature on psychosomatic interrelationships 1910-1953”. With this impressive volume, she provides an exhaustive and structured review of scientific literature of (roughly) the first half of the previous century on emotions and bodily changes. The volume's title is well chosen and reflects its content nicely. This makes this book undoubtedly valuable for the community of affective computing. However, as far as I know, outside my own work (e.g., Van den Broek, 2011), not a single reference is made to this book in any affective computing article, report, or book. I can only hope that I have missed quite a few ...

Flanders Dunbar starts her book (1954) with:

Where the work of Dunbar illustrates that the origins of affective computing can be traced back to more than a century ago, this quote illustrates that knowledge on the interaction between body and mind was already known more than 25 centuries ago! Let us now identify some core concepts as mentioned in the quote from Dunbar (1954), which are crucial for affective computing.

The old Greek already noted that “the body could not be cured without the mind” (cf. Kia Höök’s chapter). So, both are indisputably related and, hence, in principle, the measurement of emotions should be feasible. This is well illustrated by the remark that “the cure of many diseases is unknown to the physicians of Hellas”, as the Greek culture was devoted to the body and not to the mind. Recent work confirmed this relation. For example, when chronic stress is experienced, similar physiological responses emerge as were present during the stressful events from which the stress originates. If such physiological responses persist, they can cause pervasive and structural chemical imbalances in people’s physiological systems, including their autonomic and central nervous system, their neuroendocrine system, their immune system, and even in their brain (Brosschot, 2010). This brings us to the need for the “knowledge of the whole of things”, a holistic view, perhaps closely related to what Kia Höök denotes as Technology as Experience. Although the previous enumeration of people’s physiological systems can give the impression that we are close to a holistic model, it should be noted that this is in sharp contrast with the current level of science. For example, with (chronic) stress, a thorough understanding is still missing. This can be explained by the complexity of human’s physiological systems, the continuous interaction of all systems, and their integral dynamic nature. However, Brosschot (2010) considers emotions as if these can be isolated and attributed to bodily processes only. I firmly agree with Kia Höök that dynamics beyond the body should also be taken into account. Moreover, as Kia Höök also notes, in relation to computing entities, the interaction consists of much more than emotions; however, the same is true when no computing is involved at all.

25 centuries ago scientists did not apply modern statistics; however, 1 century ago, scientists did already apply statistics; for example, Fisher invented the ANOVA class of statistical models in 1918. This provided the means to test and generalize findings on emotions and bodily changes and boosted the development of behavioral sciences in general (Dunbar, 1954). Moreover, this work fits into Rosalind W. Picard’s definition of affective computing: “… a set of ideas on what I call “affective computing,” computing that relates to, arises from, or influences emotions.” (Picard, 1995, p. 1) At least it fits when taken as the traditional interpretation of computing (i.e., to determine by mathematical means). However, the added value of affective computing would be its engineering component, in particular, signal processing and pattern recognition (Van den Broek, 2011). This would enable machines to sense emotions, reason about them, and perhaps develop them themselves. This would mark a new era of computing.

With the invention of computing machinery, shortly after World War II, a new type of statistics was developed: pattern recognition. In his edited volume “Methodologies of Pattern Recognition” (1969), Satosi Watanabe collected a set of papers that were presented or meant to be presented at the International Conference on Methodologies of Pattern Recognition in 1968. Watanabe started his book with defining pattern recognition:

Watanabe denotes pattern recognition by computers as the “mechanization of the most fundamental human function of (i) perception and (ii) concept formation.”  Up to this date human pattern recognition in general is largely unsolved. We do not understand how we, as humans, process affective signals (Van den Broek, 2011). Moreover, the perception of signals and, subsequently, patterns is one thing; their interpretation in terms of emotions is something completely different. This issue refers to content validity; that is, (i) the agreement among experts on the domain of emotions; (ii) the degree to which a (low level) percept adequately represents an emotion; and (iii) the degree to which (a set of) percepts adequately represents all aspects of the emotions under investigation.

The issue of concept formation relates to the process of construct validation, which aims to develop a ground truth (or an ontology or semantic network), constructed around the emotions investigated. Such a framework requires theoretically grounded, observable, operational definitions of all constructs and the relations between them. Such a network aims to provide a verifiable theoretical framework. The lack of such a network is one of the most pregnant problems affective computing is coping with. Kia Höök describes emotions as if we can pinpoint them. Although intuitively this is indeed the case, in practice it proves to be very hard to define emotions (Duffy, 1941; Kleinginna & Kleinginna, 1981).

Par excellence, humans can recognize patterns in noisy environments. Moreover, the ease with which humans adapt to new situations, to new patterns remains striking. Moreover, this is in sharp contrast with the performance of signal processing and pattern recognition algorithms. Often, these perform well in a controlled environment; however, in the “real world” their performance deteriorates (Healey, 2008). This problem refers to the influence of the context on measurements, which is also denoted as ecological validity. Due to a lack of real world research, in general, the ecological validity of research on affective computing is limited and its use often still has to be shown in “real world” practice. However, as Kia Höök illustrates, some nice exceptions to this statement have been presented throughout the last decade.

In 1941, Elizabeth Duffy published her article “An explanation of ‘emotional’ phenomena without the use of the concept ‘emotion’” in which she starts by stating that she considers “… ‘emotion’, as a scientific concept, is worse than useless. … ‘Emotion’ apparently did not represent a separate and distinguishable condition.”  Although this statement is 60 years old it is still (or, again) up to date, perhaps even more than ever (cf. Kleinginna & Kleinginna, 1981). Almost fifty years later, in 1990, John T. Cacioppo and Louis G. Tassinary expressed a similar concern; however, they more generally addressed the complexity of psychophysiological relations. These “are conceptualized in terms of their specificity (e.g., one-to-one versus many-to-one) and their generality (e.g., situation or person specific versus cross-situational and pancultural).” (Cacioppo & Tassinary, 1990). They proposed a model, which “yields four classes of psychophysiological relations: (a) outcomes, (b) concomitants, (c) markers, and (d) invariants.” Although Cacioppo and Tassinary (1990) discuss the influence of context, they do not operationalize it; hence, this discussion’s value for affective computing is limited. Nevertheless, articles such as this are food for thought. Regrettably, attempts such as this are rare in the community of affective computing; consequently, the field’s research methods are fragile and a solid theoretical framework is missing (Van den Broek, 2011).

To ensure sufficient advancement, it has been proposed to develop computing entities that respond on their user(s) physiological response(s), without the use of any interpretation of them in terms of emotions or cognitive processes (Tractinsky, 2004). This approach has been shown to be feasible for several areas of application. However, this approach also undermines the position of affective computing itself as a field of research. It suggests that emotion research has to mature further before affective computing can be brought to practice. This would be an honest conclusion but a crude one for the field of affective computing. It implies that affective computing should take a few steps back before making its leap forward. A good starting point for this process would be the hot topics on emotion research that Gross (2010, p. 215) summarized in his article “The future's so bright, I gotta wear shades” (see also Van den Broek, 2011).

Taken together, Kia Höök should be acknowledged for her concise overview of emotion in technology. In her chapter she takes the affective interaction standpoint. In contrast, with this commentary, I have taken an affective computing standpoint. Moreover, Kia Höök has taken a design perspective, where this commentary touches upon the fundaments of emotions in technology. I pose that if anything, affective computing has to learn more about its roots (e.g., Arnold, 1968; Candland, 1962; Dunbar, 1954); then, affective computing can and probably will have a bright future!

References

• Arnold, M.B. (1968). The nature of emotion: Selected readings. Harmondsworth, Middlesex, England: Penguin Books Ltd.
• Brosschot, J.F. (2010). Markers of chronic stress: Prolonged physiological activation and (un)conscious perseverative cognition. Neuroscience & Biobehavioral Reviews, 35(1), 46-50.
• Cacioppo, J.T. and L. Tassinary, L.G. (1990). Inferring psychological significance from physiological signals. American Psychologist, 45(1), 16-28.
• Candland, D.K. (1962). Emotion: Bodily change - An enduring problem in psychology, Selected readings. Princeton, NJ, USA: D. van Nostrand Company, Inc.
• Duffy, E. (1941). An explanation of "emotional" phenomena without the use of the concept “emotion”. Journal of General Psychology, 25, 283-293.
• Dunbar, F. (1954). Emotions and bodily changes: A survey of literature on psychosomatic interrelationships 1910—1953 (4th ed.). New York, NY, USA: Columbia University Press.
• Fisher, R.A. (1918). The correlation between relatives on the supposition of Mendelian inheritance. Transactions of the Royal Society of Edinburgh, 52(2), 399-433.
• Gross, J.J. (2010). The future's so bright, I gotta wear shades. Emotion Review, 2(3), 212-216.
• Healey, J.A. (2008). Sensing affective experience. In J.H.D.M. Westerink, M. Ouwerkerk, T. Overbeek, W.F. Pasveer, and B. de Ruyter (Eds.), Probing Experience: From Academic Research to Commercial Propositions (Part II: Probing in order to feed back), Chapter 8, p. 91-100. Series: Philips Research Book Series, Vol. 8. Dordrecht, The Netherlands: Springer Science + Business Media B.V.
• Kleinginna, P.R. and Kleinginna, A.M. (1981). A categorized list of emotion definitions, with a suggestion for a consensual definition. Motivation and Emotion, 5(4), 345-379.
• Picard, R.W. (1995). Affective Computing. Technical Report No. 321. Perceptual Computing Section, M.I.T. Media Laboratory, Cambridge, MA, USA.
• Tractinsky, N. (2004). Tools over solutions? Comments on Interacting with Computers special issue on affective computing. Interacting with Computers, 16(4), 751-757.
• Van den Broek, E.L. (2011). Affective Signal Processing (ASP): Unraveling the mystery of emotions. PhD-thesis, Human Media Interaction (HMI), Faculty of Electrical Engineering, Mathematics, and Computer Science, University of Twente, Enschede, The Netherlands.
• Watanabe, S. (1969). Methodologies of Pattern Recognition. New York, NY, USA: Academic Press, Inc.

Kristina Höök has given us an inspiring view of three directions of research targeted at the crossroads of technology and affect, namely (traditional) Affective Computing, Affective Interaction, and Technology as Experience. She emphasizes that each line of research has contributed to the development of applications for various types of users, since they are complementary in their approach. I can only underline this conclusion from my experiences in industrial research. A few aspects in particular I'd like to single out for further discussion.

Let me start with an assumption that is contained in this and many other texts and views on Affective Computing, but never stated explicitly, namely that for any viable application in this domain, you need a measurement of an emotion-relevant signal. This could be a camera signal, as in Affector, movement signals as in eMoto, or physiological signals as in the Affective Diary. Also much of our own effort has been spent in the pursuit of unobtrusive measurement techniques for emotion-related signals, like our skin conductance wristband (Ouwerkerk, 2011). However, to reach the goal of 'making a machine that deliberately ... influences emotion or other affective phenomena', measurement is not strictly needed. A case in point is any TV-set or MP3 player: we use them all the time to change our mood with music, or have a TV-show experience that propels us through a series of emotions. That it works is because people are similar in their reactions to a certain extent and because TV-show directors and music composers are very skilled in creating emotional experiences for the general audience, or for specific target groups. Nevertheless, in our domain of research, everyone tacitly assumes that measurement of emotion-related signals is necessary, and indeed it allows for a further refinement of the affective influencing. Especially for changes away from the average of the crowd, in the direction of adaptation to individuals. This means that ultimately, individual models are not only necessary in the Affective Interactional approach, as Kia Höök proposes, but also in the Affective Computing paradigm. With the emotion-related measurements aboard, we also immediately enter the domain of closed-loop applications (see Van Gerven et al., 2009, Van den Broek, 2011): the emotion-related measurements are interpreted in terms of affect, then a decision is made what actions are applicable (based on present and previous measurements), and these actions are executed, after which a new measurement is done to check the new situation, etc. etc. (see Figure 1). The closed-loop model basically describes that whenever there are measurement data available, they are used to try and achieve a better situation. In this way, one's (affective) state can be guided in a targeted direction. Our Affective Music Player (Janssen et al., 2011, Van der Zwaag et al., 2009), constructed in the best Affective-Computing tradition, can serve as an example: it measures my personal reactions to music, and uses this information to adapt the playlist to direct me (not others) to a certain chosen target mood. All in all, I conclude that emotion-related measurements, individual models, and closed-loop applications are tightly interlinked in any research line in our domain.

Figure 12.1: Emotional Closed Loop.

Copyright: See page copyright notice. No higher resolution available

The affective closed loop in Figure 1 reserves a substantial part for interpretation of the emotion-related signal. This interpretation can be done by a human, as Kia Höök advocates along the lines of the Affective Interaction paradigm, and this human can either be the person that is measured (e.g. Affective Diary) or someone else (e.g. Affector). In both cases, the measurement information will be used to reflect on the situation measured, and if needed, to take action to change it (making it a closed loop indeed). If the raw emotion-related signals are presented, we will not stand the chance to lose information that is of value to the user, that is true. But on the other hand, this information might also be overwhelming (at least at first) and a user could benefit from help in the form of an interpretation made by an algorithm (in the Affecting Computing tradition). There is no need to make a choice between the two alternatives, we could think of implementing both. For instance, our electronic wristband does show the raw skin conductance/arousal patterns over the course of a day or week, but we can also give the user a discreet buzz (vibration alarm) whenever an algorithm interprets that tension has risen considerably. Of course, Kia Höök points out that it is difficult to make the correct interpretation as context is varying in many applications, and this is underlined by the fact that much of the research effort in affective computing has gone into algorithms deriving affective states from emotion-related signals. Nevertheless, there are options to try and overcome this: a technological approach is by adding additional sensors to monitor the context, like the accelerometer in our wristband that helps us estimate the activity level of the wearer and with that interpret the skin conductance signal. And another way out is by averaging over multiple measurements in varying circumstances to distil an overall effect. This is for instance done in our Affective Music Player, where the mood impact of a single song is modeled by taking the average affective effect (corrected for the Law of Initial Values) of multiple presentations, and this is proven to be good enough to select songs capable of directing one's mood to a certain state. Moreover, neither the raw emotion-related signal, nor its interpretation is presented to the user of our Affective Music Player: (s)he doesn't want to bother and only experiences that (s)he is brought into a different mood. Concluding, we find that both human and algorithm interpretation of emotion-related signals are important ingredients of future applications, and both are capable to deal with context to some extent.

Kia Höök argues that in normal life, emotions are always part of a larger experience, and that it is this larger experience that we need to support with our affective technology, in line with the Technology-as-Experience direction of research. This will certainly broaden the field of applications to include related fields in which emotions play a role. For example, emotions are important in communication, and building up relationships, and it is foreseeable that affective technologies can help (Janssen et al., 2010). It relates to the 'decide on actions' part of the closed loop in Figure 1: what do we want to do with the information gained? Nevertheless, the broadness of possible goals does not preclude that there are also applications that do have the goal to impact affect itself. A case in point is the Affective Music Player described before, which is exactly intended to direct affect, namely the mood. We have also shown (Van der Zwaag et al., 2011) that the optimal, individually selected, music can indeed help to prevent the emotion (or affective state) of anger in the frustrating traffic situations Kia Höök describes. On the other hand, I am not so sure whether consumers are interested in knowing or influencing their emotions. Despite the abundance of emotion-overloaded reality shows on TV, and despite the fact that emotion as a research topic has become fashionable in recent years, the general public still maintains a 'nice for others, not for me' attitude. In my view, this is related to the emotion/female versus rationality/male distinction Kia Höök mentions: The average male continues to see emotions as a female sign of weakness, of which they do not want to be reminded, not even if our measurement technology gives it a more masculine twist. For the females, it is the other way round: They do feel (more) comfortable with mood and emotions and acknowledge their impact on our everyday life, but they are less inclined to deploy masculine technology to alter them. For this reason also, I agree with Kia Höök , that our affective technologies are most likely to be used in applications that target a broader experience than that of affect alone.

To wrap up, let me highlight what I think is the most important message in Kia Höök 's story: That affective technologies will benefit from individual models (not only for human, but also for algorithm interpretation of the emotion-relate signals measured), and that they can be deployed in a wide range of applications extending far beyond the original domain of measuring and influencing affect. I am looking forward to see them appear incorporated in products and applications in the world around us....

References

• Janssen, Bailenson, IJsselsteijn, & Westerink (2010), Intimate heartbeats: Opportunities for affective communication technology. IEEE Transactions on Affective Computing, Vol. 1, No 2, 72-80.
• Janssen, Van den Broek, Westerink, Tune in to your emotions: A robust personalized affective music player, User Modeling and User-Adaptive Interaction, In Press.
• Ouwerkerk (2011), Unobtrusive Emotions Sensing in Daily Life, in: Sensing Emotions. The Impact of Context on Experience Measurements (2011), Westerink, Krans, Ouwerkerk (eds.). Philips Research Book Series, volume 12, Springer, Dordrecht, The Netherlands, pages 21-39.
• Van den Broek (2011), Affective Signal Processing, Unraveling the mystery of emotions, Ph.D. Thesis University of Twente.
• Van der Zwaag, Westerink, Van den Broek (2009), Deploying music characteristics for an affective music player, Proceedings VOLUME I, 2009 International Conference on Affective Computing & Intelligent Interaction, ACII 2009, September 10-12, 2009, Amsterdam, The Netherlands, pages 459-465.
• Van der Zwaag, Fairclough, Spiridon, Westerink (2011). The impact of music on affect during anger inducing drives. In S. Dmello et al. (Eds.), 4th international conference on affective computing and intelligent interaction. Part I, LNCS 6974 (pp. 407-416). Memphis, Tennessee, USA: Springer, Heidelberg.
• Van Gerven, Farquhar, Schaefer, Vlek, Geuze, Nijholt, Ramsey, Haselager, Vuurpijl, Gielen and Desain (2009). The brain–computer interface cycle. Journal of Neural Engineering, Vol.6, No.4, 1-10.