Experience and the Emotion Commotion

Podcast transcripts brought to you by the Opera Developer Network

August:

Ok, so my name is August and people always ask me—yes, I was born in the month of August on the 17th. I’m not going to tell the joke about my twenty-ninth birthday! I’m a designer and I’m from Boston, but I live in Seattle where, yes, I work for Microsoft in a group called Surface and as you could see…. well actually, since you’ve had an opportunity to experience Surface yourself I won’t go deeply into what it’s about but it’s essentially a vision-enabled, multi-touch, multi-user gestural system. And it’s kind of cool, or to use Robin’s word, it’s ‘juicy’!

I wasn’t always at Surface. I used to work for Windows but not on the software side. I worked in a small strategic industrial design team and our role was to try and influence all the PC manufacturers in the world, and we released good little ideas. For example, my boss on that team—he’s the guy responsible for the scroll wheel on the mouse and my buddy, Karl, who sat next to me, he helped pick the colours of the connectors on the back of PCs. So one of my projects on that team was actually designing the spec for the Windows key, so when my boss told me, ‘Ok, I want you to go design the spec for this’, I was like, ‘All right, I’ll knock it out in a day’. But it actually took six months and part of that is because you have to understand, especially for me as a designer, you don’t really understand the scale at which Microsoft tries to solve problems. So I could tell you that I studied intensely the optimum reflection factor according to German health standards that have to come off a keyboard—you can’t make your keys too shiny—so it’s like, all these things, when you’re designing for the world. And another funny story about this which is actually pretty fresh is whenever I go through passport control, now and then one of the border guards always says, ‘What kind of designer are you?’ And a couple of times recently I was able to point to the keyboard and point to that key and I said ‘I designed that’. No, I don’t!

So this was another project of mine on that team, and it’s a sketch for a touch-based interface, and we prototyped it and made it really great, and we partnered with Hewlett Packard, and we’ve put it on some of their prototype hardware and a flavour of this eventually got released on the market as the first version of the HP Touch Smart. This is where I first got into Touch and it kind of led me to Surface, but it wasn’t a direct path. In fact, I had a little side journey when I was dealing with and studying Emotion, and so that’s what the talk is about today, it’s not going to be so much about Surface. In fact, when my projects at Windows were kind of dying down, and it was the eve before Vista graced the world with its presence, I was wondering what am I going to do next? What kind of projects do I want next? And one thing that has always fascinated me is how experiences of products, objects, can elicit such strong emotional responses out of people, good or bad. And also what kind of drives people to do seemingly irrational things. And there’s a lot of work in the discourse about emotional design, and most of it addresses the ‘why’, but for me—being a practical sort—I was more curious in the ‘how’.

So, like a good Microsoft employee, I decided to formalise my research and through a stroke of luck, I got accepted into the Graduate School of Design at Harvard. And what’s interesting about this image is you can see how I felt on my first day at Microsoft compared to how I felt on my first day at school! And so the Graduate School of Design at Harvard is a small… it’s the smallest school at Harvard, but it has a tremendous legacy. Included in that legacy are folks like Le Corbusier, Walter Gropius, Michael Graves and (unintelligible 04.38) Peare are both graduates, on faculty there is Vern Klophaus and many, many more. This is the school building, which is at the edge of Harvard yard, and it’s pretty much what you’d expect out of a design school, and it stands in stark contrast to the rest of Harvard, which looks more like this. This is the dining hall, this is Annenberg Hall and it’s the dining hall for first-year students, and yes, it bears an uncanny resemblance to the dining hall in the Harry Potter movies, and that’s how people refer to it!

So I found myself in this unusual place where I was straddling both the corporate world and the academic world, and I started to understand some of the misperceptions on both sides and with my academic hat on, I could see how corporate work could seem distasteful and self-serving, and commercial to the point of being intellectually bankrupt. And with my corporate hat on, I could see how the perceptions of academic work seemed hermetic and arcane, and impractical to the point of being utterly useless. And all of us, on both sides, know that neither of that is entirely true, so what I hope to present today is something both scholarly and practical.

And one more note about my research—I found something that challenged my understanding of emotion and my presumptions about emotion, and it was so challenging, so counter-intuitive to what I thought, I would say that it’s the opposite of what I assumed about emotion. And in all the history of art and poetry and philosophy, I can’t think of a better example to describe this than out of the annals of Seinfeld, so please take a look at this.

Seinfeld soundtrack:

Voice 1: Why did it all turn out like this for me? I had so much promise, I was personable, I was bright—oh, maybe not academically speaking but I was perceptive. I always know when someone’s uncomfortable at a party. It all became very clear to me sitting out there today that every decision I’ve ever made in my entire life has been wrong. My life is the complete opposite of everything I wanted to be. Every instinct I have in every aspect of life, be it something to wear, something to eat—it’s all been wrong. No, now wait a minute. I always have tuna on toast, nothing’s ever worked out for me with tuna on toast. I want the complete opposite of tuna on toast. Chicken salad on rye, untoasted with a side of potato salad and a cup of tea.

Voice 2: Well, there’s no telling what can happen from this.

Voice 3: You know, chicken salad’s not the opposite of tuna, salmon’s the opposite of tuna because salmon swim against the current and the tuna swim with it.

Voice 1: Good for the tuna.

Voice 2: Er, George, you know that woman just looked at you.

Voice 1: So what? What am I supposed to do?

Voice 2: Go talk to her.

Voice 1: Elaine, bald men with no jobs and no money who live with their parents don’t approach strange women.

Voice 3: Well, here’s your chance to try the opposite, instead of tuna salad and being intimidated by women, chicken salad and going right up to her.

Voice 1: Yeah, that should do the opposite. I should.

Voice 3: If every instinct you have is wrong, then the opposite would have to be right.

Voice 1: Yes, I will do the opposite. I used to sit here and do nothing, and regret it for the rest of the day, so now I will do the opposite and I will do something. (goes up to woman) Excuse me, I couldn’t help but notice that you were looking in my direction.

Woman’s voice: Oh yes, I was, you just ordered the same exact lunch as me.

Voice 1: My name is George. I’m unemployed and I live with my parents.

Woman’s voice: I’m Victoria, hi.

Audgust:

So, yeah, we’ll get to more about the opposite and emotion in a bit. And this kind of culminates in what I call a two-factor design theory and we’ll get to all of that later.

But let me start the story at the very beginning and this is back in 2001, and I was living in the Netherlands in Amsterdam, and I was working for Phillips Design, and like all Americans living in the Netherlands, well all foreigners for that matter, I had to register with this organisation and it’s the Police for foreigners, and in Amsterdam the precinct for the Police for foreigners was a street with this name, and it’s easier to read if you break it up and that last word means ‘lane’. But basically it’s a street named after this guy, and so like a good new arrival to a foreign country I decided to look up who Jan Huizinga was and it turns out that Huizinga is considered the greatest Dutch historian of the twentieth century. And one of his seminal works is this book called ‘Homo Ludens’ which means ‘human at play’ or the ‘being at play’ and in it he looks into the play element within culture. And Huizinga suggests that things like ritualised combat and theatre and religious rituals are all borne out of this element of play. And some scholars recently suggest that we are living up to Huizinga’s ‘Homo Ludens’, that we’ve realised it today because we’ve networked communications and digital technology.

The behaviours that we elicit in our communications patterns in everyday life match those of how children communicate with each other in states of play, like in a playground and so forth. In fact, play is one of six aptitudes that Daniel Pink in his book, ‘A whole New Mind’, talk about that will be important as we shift out of the information age into what he calls the conceptual age. And he points out three reasons for this—Asia, Abundance and Automation—but the one that’s relevant to the talk today is that around Abundance. And Pink says that since World War II, generations of druckers—information workers—have created a standard of living that has been unmatched in all of human history. A small example of this is the typical American dream consist of owning your home and a car, and he cites…. well, I guess until the recent economic events… that two-thirds of Americans own the home in which they live and some thirteen per cent actually own a second home. And as for cars, in the United States, there are more cars than there are licensed drivers, implying that anyone who knows how to drive owns the means by which to do so.

Well, the sum of this is that as we get more and more choice available to us, we make our choices, our decisions based on less rational sensibilities—things like beauty, spirituality and emotion—or what Virginia Postrel in her book, ‘The Substance of Style’ calls the ‘aesthetic imperative’. Well, to put this in a broader context, as humans start worrying less and less about things like where our next meals are coming from, or finding shelter, or running away from dinosaurs, this creates free time or playtime and in this free time we start becoming more introspective and we start asking ourselves more metaphysical questions like ‘Why am I here? What’s the meaning of life? What is true happiness?’ and so forth. And as technology accelerates, we find that it has a broader effect on society as more and more people have this playtime. And so we find ourselves shifting from a material society towards an immaterial society, or as Pine and Gilmore call it in their now cliched ‘Experience Economy’, the people now value experiences over things. In Pine and Gilmore, they talk about the role of products in this experience economy, and there are kind of props in this kind of consumerist theatre where people actually pay for the memories and experiences of their interactions with their products. And to put a more scholarly bent on it, I pose the question—‘Well, what is the role of objects in this immaterial society?’

Well, this brings up a kind of controversial figure in academic circles—this is a philosopher, Peter Sloterdijk out of the University of Karlsruhe and one of the reasons why he’s a bit controversial is he rejects this notion of dualisms, and dualisms are these pairs of concepts or pairs of ideas that we traditionally think of as being two separate or discrete things—things like body and soul, culture and nature, subject and object. Rather than these being dual or separate entities, Sloterdijk suggests that they persist in these faces of co-existence. An example of this would work in play, because if we look at this image of the carpenter in the background, it could either be a carpenter working at his daily vocation, or it could be a web developer who likes to do carpentry on the weekends and build furniture and so forth. I mean a traditional act of work has now become a form of play.

To help explain this whole notion of dualisms, the creators of South Park actually animated a short discussion of this by Alan Watts, and I’ll just get right to it: Alan Watts: You see, in the history of philosophy and poetry and art, we always find the interchange of two personality types which I call prickles and goo. The prickly people are advocates of intellectual porcupinism. They want a rigor, they want precise statistics and they have a certain clipped attitude in their voices, and you know this very well in academic circles where there are people who are always edgy like that, and they accuse other people of being disgustingly vague and miasmic and mystical. But the vague, miasmic and mystical people accuse the prickly people of being mere skeletons with no flesh on their bones. And they say to you, ‘You just rattle, you’re not really a human being. You know the words but you don’t know the music.’ And so therefore, if you belong to the prickly type, you hope that the ultimate constituent of matter is particles. If you belong to the gooey type, you hope it’s waves. If you are prickly, you’re a catharsist and if you’re gooey, you’re a romanticist, and going back into mediaeval philosophy, if you’re prickly you’re a nominalist, if you’re gooey you’re a realist. But we know very well that this natural universe is neither prickles nor goo exclusively, it’s gooey prickles and prickly goo! And, you see, it all depends on your level of magnification. If you’ve got your magnification on something so that the focus is clear, you’ve got a prickly point of view. You’ve got structure, shape, clearly outlined, sharply defined. A little out of focus and it goes blllllllrrrr…and you’ve got goo. But we’re always playing with the two.

So, yeah, the fancy word for this that you could toss around at cocktail parties is the dialectic, and if you want to be really prickly about it—and this is a concept that’s pre-Socratic, Parmenides… ok, so I’m going to introduce a couple of dialectics and talk about people’s relationships with objects, and one of the first people to articulate the human/object relationship is John Lark—it’s not John Lost Lark but John Social Contractor… And so Lark says that as humans go through the environment, we encounter raw materials and we start imbuing these materials with labour, and the more labour we put into the materials—and eventually we create an object—we create the first articulated notion of our relationship with objects, and Lark called this possession, in the legal sense. If we jump ahead to 2006, there is a linguist out of California who wrote this book called ‘Smart World’, and in it he describes a scene from the Ron Howard movie, ‘Apollo 13’, in which Tom Hanks playing the astronaut is up in space and he’s trying to figure out all these calculations to save the lives of him and his crew. And as he’s performing these calculations, there is a roomful of engineers on the ground who are performing the exact same calculations to confirm the outcome of the math…. and the only difference is the engineers on the ground are using this tool, the slide rule, to verify the calculations, and in Argol’s argument, he says that the intelligence that they were using had actually been embedded into this object of the slide rule, so there’s this whole notion of embedded intelligence. And Argol goes on to talk about this notion of idea spaces where collections of objects actually create a kind of ambient intelligence in spaces.

Right, so I took this idea and I was thinking, ‘All right, so I’m going to go back to Microsoft and talk about this, and it’s not going to go over very well.’ So while I was at Harvard, I decided to find a very nearby population of engineers and so I ran down the street to MIT with all of these ideas, and I actually studied with someone in my team named Marvin Minsky who, aside from being a pioneer in artificial intelligence, also happened to be Stanley Kubrick’s science adviser for the film ‘2001: A Space Odyssey’. In his class, we read his latest book called ‘The Emotion Machine’ and in it, he too creates a dialectic around rational thought versus emotional thought, and he suggests that those aren’t two separate things, those aren’t two modes or different ways of thinking, rather that they’re all… at the mechanical level, they’re the same. And I’m not doing it justice, you have to read the whole book to get the argument, but it’s a great argument. And so taking this, if objects can have a kind of embedded intelligence, well at the same time one could argue that objects could have embedded emotion as well. And this would explain the sort of sentimental ties we have to things like souvenirs that we pick up on trips, or a lucky shirt, or whatever. The image on the right is a robotic vacuum cleaner from the company, iRobot in the United States, and the owners of iRobot were being interviewed on national public radio in the US about their company, and they were telling the story of this one family whose Rumba robotic vacuum broke down, and they wanted to send it in to get it fixed, and the folks at iRobot said, ‘Well, you know, we’ll just send you a brand new one and we’ll take the old one and fix it, and sell it as refurbished.’ And the family protested because they had already come to think of it as another member of the family, and they said, quote, ‘It’s been living with us for years now’ so they wanted the specific Rumba to come back to them.

Well, given this kind of relationship, it kind of makes sense to raise the question, ‘What are emotions?’ And Minsky always told me the simplest questions have the most complex answers and vice versa. For example, all of physics could be boiled down to a few rules. The field of psychology has kind of suffered from a bit of envy of physics in that there have been attempts to try and boil psychology down to a manageable number of rules. I think the best attempt was, like, six thousand six hundred rules, and within those are there many theories of emotion, but the one that I’m going to talk about today is one that’s called the James Lange Theory of Emotion. And it was developed by William James and the Danish physiologist, Karl Lange, and I only bring that up because when I first heard about it, I said, ‘Who’s James Lange?’ and there’s no such person as James Lange!

Well, what the James Lange theory suggests is that emotion is a label that we apply after we experience something and this is where it starts to get into the opposite. So the common sense theory of emotion could go something like this—so I’m walking through the woods and I see a bear, so I become scared, I experience fear and as a result my adrenalin shoots up, my blood pressure rises, my heart starts beating and I want to run, and here’s where the opposite comes in. And this is where the James Lange theory gets a bit controversial, and this is where I get to play the designer’s card too. The whole point of this research is I’m not a psychologist, I’m not trying to prove or disprove any sort of psychological model, rather I’m trying to find a tool that’s useful for designers to think about emotion in creating or designing their projects. So in the James Lange Theory of Emotion it suggests that our autonomic nervous systems are constantly monitoring the environment and as a result we experience slight deltas in our physiology, in our blood pressure, adrenalin, heart rate and so forth.

And so to apply this to our bear example, I’m walking through the woods and my nervous system is monitoring the environment, and it senses a large menacing object and as a result my body reacts and it increases my adrenalin, it increases my blood pressure, I have that flight factor and this is all happening within a split second. And the moment that I recognise that that large menacing object is a bear and I not being a bear trainer at the circus, draw the conclusion that this change in my physiology is fear, and I know it’s… I’m going to give a couple of examples to explain this further. The kind of hallmark example of this theory is The High Bridge Study conducted in 1974 and in this study there were two groups of young male graduate students, and they were each asked to cross a rope bridge that was suspended high over a gorge. One by one. And there was also an actress who was playing the role of the lab assistant, and in the first condition of the experiment the lab assistant was on the far side of the bridge so the student, the volunteer would cross the bridge and the lab assistant would then proceed to ask him questions about his experience crossing the bridge. And unbeknownst to the subject, she would then start to flirt with him and eventually give him her telephone number, suggesting that he call for a date. And in the second condition, everything was exactly the same except the lab assistant was halfway across the bridge and the results were pretty interesting. They found that less than a quarter of the students in the first group actually called her back, only to get the office of the Department of Psychology, but also they described their experience crossing the bridge as fearful, anxiety causing, really scary. But in the second group, more than two-thirds of the participants actually called her back and they described the experience as thrilling, exciting, adventurous and so forth. So it was the same physical stimulii but with different conclusions drawn from their experience.

Another anecdotal example around this is there was a race car driver fitted with biometric equipment and he was asked to run a race… drive a race, and the results of the biometrics were shown to a team of doctors and without knowing anything else about the patient, and the doctors’ conclusion was that this patient was suffering from a panic attack, when in fact the driver was fine and had a great time driving in the race. I bring these examples up because there’s one field that actually does this all the time and it’s the design of video games. And so I have to make here…. I’m going to talk about the MDA model… all right, I’m both excited and really intimidated because one of the pivotal reference papers of my presentation is co-authored by Robin who just presented before me. So in this part of the presentation I have to answer some of the questions too because in my own research I actually apply the MDA model to non-game fans.

Let me give you a quick refresher of what Robin just said. So the MDA model—and I’m sorry if I totally bungle this up but…. all right, so there’s the first level which is mechanics and I’m just going to fly through this. Like in a game of chess, it’s the rules, it’s the pieces, it’s all the things that have a one-to-one relationship with the player. And in product design—and this is where I’m doing in my doctoral research, applying MDA to….. well game design structures to everyday things—like in a car, it’s the feeling of the mechanics like in buttons and switches, the steering wheel and so forth. And then there’s the second layer of the dynamics when we introduce… it’s the run time of the system where the mechanics interact with each other and also the human element comes in. So again, with chess, the rules of chess have been the same for millennia, but no two chess games have ever been the same, and this is an example of dynamics in action and the systems of the car also produce a different dynamic for each car. And finally, there’s the aesthetic level and again, this is where the emotion comes in, this is where we have our affect from the experience, whether it’s driving a car or playing a game of chess. And what’s interesting is, these form a directional stack and from a player or a user’s point of view, they’re most aware of the aesthetic first and slowly become aware up until the mechanic level, while from a designer or developer’s point of view, it’s the reverse where we are most aware of the mechanics and slowly become aware of the implicit aesthetic qualities of the system. All right, so enough of that re-run.

So the interesting thing about this is that the MDA model actually has alignment with the James Lange Theory of Emotion so if we think about the sphere of emotion, here’s your mechanics, dynamics and aesthetics, and this is where it starts to get really interesting. So an example…. well it turns out that one of my good friends is the Head of Games Research at Microsoft, and before Halo came out there was no successful first person shooter game for a console. First person shooter games were all about mouse and keyboard, and what Bunjee and the Games Research Team at Microsoft said was, ‘well, let’s look at the mechanics of the system’, and one mechanic that they looked at was the targeting system of controlling the cross-hairs to target bad guys, and they found a mechanic flaw of why it doesn’t work with joystick and button, but it does work with mouse and keyboard. And if you look at the mechanic of how to target a bad guy, there’s this kind of spectrum where on one end you can snap to a target—you can, like, alt-tab to the target—and it snaps to the target and you can fire, but then you could see MDA in effect where the aesthetic outcome is, this is too easy, I’m not challenged, it’s a boring game, it’s kind of lame. At the other end of the spectrum, if there was a direct relationship or one-to-one relationship of the mechanical interaction of the controller with the cross-hairs, you’d have to be really good at manipulating that controller to actually target something, because most people tend to overshoot the target. And so the aesthetic outcome is, this game’s too hard, again lame. So what the Games Research Team did is they found through a lot of well-documented processes, they found that sweet spot in the spectrum where they finessed the mechanic interaction with targeting the bad guy, so from a user’s perception it seems like there is a direct relationship, a direct one-on-one relationship with the cross-hairs and the target, but there’s actually a little bit of programmatic gain, a little bit of gravity that locks the cross-hairs to the target and kind of eases it in. So while the perception is that this is a really tough thing I’m doing, it actually supports that whole primal fantasy that Halo has, that you’re out to save the world and you’re doing really tough stuff, targeting and aiming with a controller.

And this is where it applies to non-game software and also to industrial design, that if you look at the mechanics of luxury items, like really expensive cars, when you shut a door of a really high end car, it has a certain feel—and I start using Robin’s vocabulary—it’s juicy, it has this juicy feeling, it kind of settles, it does. And also if you buy a really high end Danish stereos, you’ll notice the mechanic interactions of those also have this kind of juiciness, or this richness of experience. And the picture in the middle is a $10 potato peeler, and there are these fins here that serve no practical purpose, this is direct from the manufacturers, Oxo—they say themselves that the only purpose that these fins serve is that it feels good to touch, and this is the most successful vegetable peeler in North America.

So, the sum of this is that the perception of the system output should be greater than the user input, and it should fit within the context of what is trying to be accomplished. So in other words, it kind of feels like magic and this is the first factor in the two-factor design theory that I’m talking about, and this should be enough to start applying it to non-game like things, and there is a whole second factor which I could present, like, for a whole day, but these are the four aspects of that second factor which I’m not going to get into, because I wanted to close with addressing the theme of our conference which is about Designing For Tomorrow And if you interact with Microsoft Surface, you can see how we’re starting to apply some of this MDA thinking and again, that’s a small part of how to apply this pleasurable, emotional design methodology to everyday things. And I wanted to show you an example of how this is enacted, and I love the last question in Robin’s session was about Excel and things that we could do to make Excel emotionally engaging and pleasurable.

And so what I’m going to share to close this talk is how some of us at Microsoft are thinking about this and what it might look like. And so what I’m going to show is… it’s not an advert, but it’s a vision piece that we use—the design teams use—to inspire ourselves, and it’s not totally science fiction, like in Chris’s and Nathan’s talk, but without giving away any corporate secrets, I’m just going to ask you to imagine this, I’m going to ask you to imagine a pixel and I’m calling it the Fifth Element because there are five components to this. In the display pixel of today we have the three elements—red, green and blue. Now imagine we add a fourth part to this pixel, which may be, I don’t know, an IR sensor or some sort of sensor, and imagine this part, this fifth element that existing at the pixel level there’s also a camera—camera capabilities. So now you have this really exciting pixel which I believe will be realised within the next ten years, and you can put this pixel anywhere, and so what I want to show, to close this talk, is what this fifth element might be. This is a piece created by my friends at Office Labs, and it’s kind of a visual answer to the last question of the last session.