Evaluating User Experiences in Games: Workshop at CHI 2008

On April 5th, members of OTOinsights t=zero partnership with the Indiana University School of Informatics participated in a workshop on “Evaluating User Experience in Games” hosted by the International Conference on Human Factors in Computing Systems (CHI 2008) conference in Florence, Italy. CHI is the largest and most prestigious conference in human-computer interaction (HCI), with over 2,000 participants from dozens of countries.

At the workshop, we presented and discussed our plans for executing a multi-modal evaluation of game play experience. Following presentations from the other workshop participants, we participated in a substantive discussion about the current state of game play experience research and practice. The remainder of this post is dedicated to some of the insights from that discussion and a reflection on how the t=zero team’s work is reflective of the latest changes to player experience evaluation methods.

A major theme from the workshop centered on the difficulty or need to distinguish immersion from engagement and determining the effects of each on player experience. Some workshop participants interchanged the terms immersion and engagement in reference to a singular concept; however, other participants saw immersion and engagement as distinct forces. To some, immersion is simply the extent to which a person prefaces the current media over other information (playing a game and not hearing someone yell for you) while engagement is something beyond immersion that fully brings a player into the game world. Participants concluded that we should establish a shared understanding of where immersion and engagement begin and end and to what extent each of those concepts (or a unified concept) affects the game play experience. Without such an understanding, valid and useful measures for evaluating player experience with games will be difficult to develop.

Numerous methods used for evaluating experience with digital media are in use. For example, with the assistance of Quantemo, we simultaneously measure 5 or more biophysical modalities and combine that information with additional behavioral research methods. One identified weakness in all of those methods is a reliance on an expert’s interpretation of the results. Researchers are crucial to the process of evaluating player experience, but in some situations players themselves are best equipped to interpret results generated from our work. Who, then, is the “expert” that interprets the data? A great example from the workshop came from the use of eye tracking equipment in the usability test of a video game. Eye tracking equipment readily shows a researcher “what” a player is looking at, but cannot as easily address “why” a player is looking at an object. Showing eye tracking footage to players and asking why their visual patterns change at any given moment may reveal meaningful information that even expert researchers cannot see.

The final issue brought up during the workshop concerns the difficulty of interpreting biophysical signals as a measure of player experience. Researchers at the workshop commented on the difficulties of relying on any individual biophysical measure to measure changes in experience. Individual measures, especially galvanic skin response, can be susceptible to even subtle changes in a research environment and may not accurately or entirely reflect a players experience with a game due to the noise added to the data by other stimuli (air temperature, background noise, etc). The multi-modal method that our Quantemo lab uses for player experience evaluation was judged by participants to be a promising example of how to compensate for the shortcomings of individual biophysical measures. Monitoring multiple biophysical signals and supplementing those measurements with traditional, vetted behavioral research methods yields results that can be viewed with increased confidence by stakeholders and researchers alike.

The t=zero team was extremely happy to have the opportunity to participate in the workshop and share our experiences with peers from around the world. Methods for evaluating experience with digital media are growing and changing at a rapid pace and events like this workshop help all of us to understand the mutual challenges that we face as a field. While we are currently focused on creating a shared understanding of our terminology, leveraging research participants as knowledge co-creators and developing robust and reliable methods for evaluating experience, you never know what the next challenge will be. Participating in this workshop gives us confidence that t=zero is well positioned to tackle the current and upcoming challenges created by the ever-changing digital landscape.

Tyler Pace, Shaowen Bardzell, Ph.D., Jeffrey Bardzell, Ph.D.

NeuroMarketing - New Tools For Engaging Customers

Below is a post that was originally published on Steve Furman's Expedient Means Blog. I am re-publishing with Steve's permission here.   He attended a Forrester Marketing Forum session I spoke at on April 8th titled "Marketing's New Imperative for Success: Engagement".  Steve did a fantastic job covering a number of  presentation that were given over three days in Los Angeles.  Enjoy!



Fast forward to some time in the future. The marketing game has completely changed, having evolved beyond test and control, research, etc. Imagine you can understand how your customers react to your products. By react I mean physical responses such as eye movements, heart rate, breathing pace, galvanic skin response and body language. You can map these responses to human emotions and cognitive thinking styles. Next you capture how your customers form relationships with your products (abstract, concrete) and how their social preferences interplay with and drive consideration. But wait there’s more. Throw ideolgical values (taste, morals) into the calculus and you will be able to mold a product that satisfies all basic human pleasures and by definition is the most desireable item on the market. You are are flying, and instantly promoted.

Science fiction? Is it even possible? It is possible, and the technology is available now. Welcome to Part III in my weblog series from the Forrester Marketing Forum 2008 (Los Angeles, April 7-9). The Forum’s theme was customer engagement. In this installment I make an attempt to summarize and connect four separate presentations (two breakouts and two keynotes), that starts to show marketers how to create more engaging online experiences by making them more pleasurable and deisrable.

At the heart of this task is a new type of practice called NeuroMarketing. It’s in very early days, having been largely confined to labs using expensive equipment that was uncomfortable for the subjects. As with any technology, it’s getting smaller and cheaper. There is only so far marketers can go with our current practices. In my view it’s critical to employ new tools that can measure human response and desire. Let’s get started.

First - The Four Pleasures Framework by Patrick Jordan. Mr. Jordan is a design, marketing and brand strategist and holds a PhD in psychology. He has worked with major brands to create campaigns and products using his pleasures framework.

The objective is to help people feel good about your product, your brand/company and about themselves. The four pleasures are:

  1. Physio - Physiological, the body and its senses
  2. Psycho - Psychological, the mind, emotions, cognition and interests
  3. Socio - Relationships, social connections in the abstract and concrete
  4. Ideo - Ideological, the values, taste and morals

During his talk Mr. Jordan cited real-life examples for each of the pleasures. To illustrate physio, he spoke about how the car maker Fiat has an entire lab and team devoted to only three parts of a car. The steering wheel, gear shift and inside door handles. Through research and observation, Fiat discovered that these were the first three things a customer actually touched when in a car showroom. The salesman would usually open the door, the customer would step in, put her hands on the wheel, then on the gear shift. When she wanted to exit she would have to touch the door handle. If the designers could elevate the sensory experience of these physical parts to one of pleasure, product consideratin is off to a flying start.

He provided examples for each pleasure, but I won’t go into them here. For those explorers that want to give it a try, he offered this brief summary.

  • Create robust personas
  • Conduct indepth ethnographic research
  • Immerse yourself in your customers
  • Look at what’s going on in the media

Second - Amplifying Engagement: Measuring Customer’s Emotional Reaction to an Experience, was given by Jeremi Karnell, President, One-to-One Interactive. His company(s) are working in the NeuroMarketing space, and he defines it this way.

NeuroMarketing is a new field of marketing that studies consumers sensorimotor, cognitive, and affective response to marketing stimuli.

He discussed what he calls the mind-body nexus of engagement, consisting of perception, attention, brain function and behavior. His firm developed the Quantemo Engagement Index, a scientific approach to measuring a target audience’s emotional reactions to digital media. In short, they put sensors on subjects (simple things like bands, nothing sticks to the skin) show them web sites, ads, emails, then report on heart rate, galvanic skin response and breathing. The sensors can also detect eye tracking and body movement. Are the subjects leaning in (interested), or sitting back (bored). These measurements are graphed and presented alongside the usability testing video and reports to give designers more data points to validate or refine designs or marketing messages. Can be employed against your competitors sites as well.

Third - Creating Personas that Support Engagement was given jointly by Neil Clemmons of Critical Mass and Mike Madaio from QVC. I won’t go into defining personas or how to use them in this post. You can easily find that through a simple search. The value in this talk was how Critical Mass extended the Forrester useful, usable, desireable usability model by adding sustainable and social to the persona matrix.

I have been doing a lot of thinking along these lines lately, and this really made it clear. The more offline experiences migrate to the online world, the more tools designers and marketers will need to be effective. The rapid growth of social computing is being accelerated by technology advances. This will require new ways to think about how to create online experiences that will keep up. Expanding the persona/user-centered design paradigm is a natural next step. Mastering these techniques will be critical to engaging users in your online properties.

Fourth - Designing for Engagement by Forrester Principal Analyst, Kerry Bodine. Her talk orbited around desirability. She didn’t offer a textbook definition, but instead quoted Supreme Court Justice Potter Stewart’s opinion in Jacobellis v. Ohio (1964) as he attempted to define obscenity.

I shall not attempt further to define the kinds of material I understand to be embraced within that shorthand description; and perhaps I could never succeed in intelligibly doing so. But I know it when I see it…

I know it when I see it. Clearly desirability is a subjective call and as unique as humans. But like so many other things the mind processes, it’s real. That’s why NeuroMarketing is going to be important. It pulls the subjective, which is very difficult for marketers to deal with, into focus using something more concrete than a gut feeling.

Kerry Bodine - Photo: Steve A. Furman

Ms. Bodine showed the standard usability strata Forrester has been promoting for years, and suggested it should look more like a point to point map, increasing the role desirability should play when designing. This is a subtle change, but one that challenges designers and web architects to think about desirability along side the other dimensions at the outset, vs. something to aspire to after launch. Makes more sense.

I would love to see Forrester refine, actually update, their persona framework to address the rise in social computing and match what they have done with this change. Since 2002, I have worked with Cooper to create the personas we use today. Their persona philosophy and methodology was a natural fit with how we think about segmentation.

Ms. Bodine used a number of personal and observed examples of desirable experiences. One as mundane as ordering room service in a hotel. Her summary and advice to marketers was as follows.

  • Learn to recognize desirability when you see it
  • Give desirability the recognition it deserves
  • Find a way to create desirable experiences

My take on what it means

Online marketers (DM guys and product managers) need to get much closer to interactive design than they are today. The pure plays are way ahead of the analog legacy firms (less baggage). Traditional direct marketers have the luxury of creating dozens (sometimes hundreds) of test cells and corresponding creatives. But they do this, for the most part, not so much through observing human responses, but by mechanical test and control (trial and error). I’m not suggesting that this is not a valid science, but it leaves out the human emotional reactions that are hallmark to the web’s interactivity.

Online testing tools available to raise interactive marketing practice to DM levels are getting better, but most firms don’t have the understanding, budget, expertise or technology infrastructure to acquire, implement and use them. They cannot support a network of sites or instances of sites or even regions on pages necessary to conduct robust DM-like testing. Don’t get me wrong, some firms are doing this well, but they are the exception. In my company we had at one time over 14,000 direct marketing test cells for one product! Nothing even close to that online.

I know it’s counterintuitive, but the online channel in most companies is fairly static because of tracking challenges, staff support, lack of a content management system and the reality of having to integrate with back end databases and systems real time. Content management suites like Interwoven, are helping, but they are big enterprise solutions. Could there be an Interwoven Lite market out there?

NeuroMarketing, is real today and could be baked into the normal project plan without extending the time line or breaking the budget. It can give the online marketer a new and powerful tool that doesn’t result in an extra large IT project.

What do you tell your CMO when asked to explain desirability? “I know it when I see it” is probably not going to do it. Use the mind. Neurons tell the truth.

In Summary

  • Create personas now. If you already have them be sure they are up to date.
  • Get buy in on personas from your DM marketers and Product Managers.
  • Bring them into the design and development process early and keep them there through the validation cycles.
  • Integrate NeuroMarketing techniques in your usability testing plan.

Read my other Forrester Marketing Forum 2008 posts here for Part I and here for Part II.

MITX | One to One Interactive Engagement Series: "Player Engagement & In-Game Advertising"

Otoengagementseries

(Left Click the image above and choose Download. 19.8mb)

On March 2th, 2008 One to One Interactive and MITX bowed the first presentation in its new Engagement Series.  Jeremi Karnell, President of One to One Interactive and Tyler Pace, Research Assistant Indiana University School of Informatics presented initial findings from their research titled "Player Engagement & In-Game Advertising".  Hosted by UK Trade & Investment at the British Consulate in Cambridge, MA. the presentation is an early insight to what will be a formal report that will be published in Q2 2008. 

Key  Findings:

  • High impact ads may be located at both frequent and terminal game play points such as finish lines, save screens, menus, etc.
  • If you know your brand is not familiar to the player base of the game/genre, highly visible placement will improve recognition and association.
  • If your brand is familiar to the player base for the game/genre, placement is less important to success.
  • Primary research to quantify player physiological engagement with specific game titles may help in media buying selection process and ensure higher brand recognition/recall.

OTOlabs Takes Widgets WAY Beyond the Browser

Polycake2In honor of this past Monday's official launch of PolyThink, the new cross-media messaging platform from OTOlabs, Technology Director Chris Pointon commissioned a set of very special widgets, developed for a decidedly non-traditional delivery system. Possibly the world's first Widget Cookies were unveiled at the company meeting today, with a delicious variety of functionalities to choose from.

Mypicture_bWhile bandwidth and download times are not an issue with this new platform, several members of the office "Biggest Loser" competition were overheard grumbling about beltwidth and off-load time, which could be significant for the product. However, on the whole the reception was warm, and the widgets were downloaded hungrily by the assembled crowd.

As OTOi marketing services tastemaker Mike Lodge observed, "It's a refreshing change from Facebook: a widget that doesn't leave a bad taste in my mouth!"

For a more serious look at the advanced PolyThink platform from OTOlabs, head on over to their official site at http://www.polythink.com/

Facebook To Punish Stupid Apps

Very similar to Google's Quality Index, Facebook today announced that they will begin rolling out a system which rewards applications which appear to be more relevant to users, "based on a range of factors including the rates that users ignore, hide, and report notifications as spam".

Says Facebook:

"The new system aims to provide users with more compelling notifications and fewer notifications that they are likely to ignore or consider spam. We hope this change incentivizes developers to improve the quality of their notifications and encourage their users to send notifications to interested friends."

As someone to recently begin to use Facebook only a few months ago, I applaud this decision, as already the number of Feed updates and friends' applications requesting full control of my Profile seems more than a little out of control.

With a revamped system that promotes those apps that users actually use and punish the more spamlike apps which require -- as a very common example -- that you send the app to 10+ friends before the app will function, hopefully this will result in a social networking site that's far less cluttered with pure crap.

Now if only similar self-controls could begin to be implemented across all other channels, perhaps consumers could begin trusting marketers a little more than the "not at all" they're at right now.

Inspired Viral Video: Black Eyed Peas on Obama

Via Salon.com, I found a blurb about a viral video for the Obama campaign. I checked out the video, and I was quite impressed. The production qualities are high, which is what seemed to impress Salon. But that's not what got me excited enough to blog about it; what impressed me was how authentically YouTube this video was. This isn't some political hackery saying "me too!" to the Facebook generation. This video gets it.

One of the great, if not the greatest, creative strategies of online animations and videos is to take a work of mainstream culture and rework it, commenting on it, distorting it, and appropriating it. This is a dominant cultural logic of "hip" entertainment today.

The best examples of this strategy reveal and celebrate the voice of the remixer. In other words, the source voice is dominated by the viral video's producer/editor. It's not surprising that political campaigns, terrified of relinquishing control over their message, would be leery about fan-created videos. Certainly many inspiring online videos have been made about politicians, but they are commonly parodies (such as this one about Bush, or this one about John Edwards), or they are propaganda pieces.

But the Obama campaign has got to love how the Black Eyed Peas have reworked Obama's oratory into a video that can stand alongside their best work. This video neither parrots nor subverts Obama's message; it personalizes and extends it into the Black Eyed Peas own powerful idiom.

Jeffrey Bardzell, Ph.D.
Indiana University / Quantemo / t=zero

Business Week mentions One to One Interactive in Neuromarketing Article

Bw_255x54

Business Week published an article today regarding Neuromarketing in their Technology Section titled "What you Really Want to Buy". Although the most of the article is rather light (as is most things coming from traditional reporters covering subject matter they don't have a clue about), it does reference Roger Dooley (blog's neuormarketingscience.com), Emsense (Quantemo competitior), and Jeff Bardzell Ph.D. (IU School of Informatics/Quantemo/T=Zero). All of those above individuals and groups are who we are working with to launch the nations first Neuromarketing Association.

Updated Definition of Neuromarketing on Wikipedia

Neuromarketing Late last week I announced that I intended to partially update the definition of Neuromarketing in Wikipedia. Just a couple of minutes ago, I finally made my edits to Wikipedia and I have decided to post about it in order to detail the change and allow for comments, feedback, suggestions, etc.

The old definition
, which I found dated and too narrowly focused on cognitive based research methods, was:

"Neuromarketing is a new field of marketing which uses medical technologies such as functional Magnetic Resonance Imaging (fMRI), and Electroencephalography (EEG) to study the brain's responses to marketing stimuli. Researchers use the fMRI to measure changes in activity in parts of the brain, or EEG to measure activity in specific regional spectra of the brain response, to learn why consumers make the decisions they do, and what part of the brain is telling them to do it."

The new definition, that I published today I believe represents a broader array of research methods deployed in Neuromarketing today. It reads:

"Neuromarketing is a new field of marketing that studies consumers sensorimotor, cognitive, and affective response to marketing stimuli. Researchers use technologies such as functional Magnetic Resonance Imaging (fMRI) to measure changes in activity in parts of the brain, Electroencephalography (EEG) to measure activity in specific regional spectra of the brain response, and/or sensors to measure changes in ones physiological state (Heart Rate, Respiratory Rate, Galvanic Skin Response) to learn why consumers make the decisions they do, and what part of the brain is telling them to do it."

There is no doubt that this will continue to evolve as the industry does. In fact One to One Interactive's OTOinsights division is currently working with other leading Neuromarketing research firms in the United States to establish the country's first Neuromarketing Association. I imagine that such effort will yield an even broader and more robust definition of this emerging industry.

Neuromarketing Defined

Neuromarketing_7 Below is a partial definition of Neuromarketing that I plan to use to update the current Wikipedia page dedicated to the subject  (which I feel is a bit dated and too narrowly focused):



Neuromarketing is a new field of research which uses medical technologies and scientific method to study consumers sensorimotor, cognitive, and affective response to marketing stimuli.


Thoughts, feedback, edits?


Jeremi Karnell-President, One to One Interactive

Head Tracking for Desktop VR Displays using the WiiRemote

The above video was posted to YouTube by Johnny Chung Lee, Ph.D. Graduate Student at Carnegie Mellon Universitie's Human-Computer Interaction Institue.  Using the infrared camera in the Wii remote and a head mounted sensor bar (two IR LEDs), Lee demonstrates an approach to accurately track the location of your head and render view dependent images on the screen. This effectively transforms your display into a portal to a virtual environment. The display properly reacts to head and body movement as if it were a real window creating a realistic illusion of depth and space.  For more information and software visit http://johnnylee.net

One to One Interactive

  • Logo_6_1

Contact Us

E-Mail Digest



  • Powered by FeedBlitz

OTO del.icio.us Links