the rePertory GrId:

Eliciting User Experience Comparisons in the Customers Voice

Michael Hawley, Senior Vice President, Director of Experience at Mad*Pow
IntroductIon
If you do user research, chances are good that you conduct a fair number of user interviews. Our training tells us to minimize bias by asking open-ended questions and choosing our words carefully. But consistently asking unbiased questions is always a challenge, especially when youre following a participant down a line of questioning that is important and you havent prepared your questions ahead of time. Also, if you do a lot of interviews, you might fall into a pattern of asking the same types of questions for different studies. This might not bias participants, but you can bias yourself if you always investigate the same types of issues. Finally, are you sure you are asking the right questions? Your interview questions might be relevant to you and your project team, but are they the questions that will explore important issues from a users perspective? In an effort to address some of these considerations, Ive experimented with the Repertory Grid methodan interview technique that originated in clinical psychology and is useful in a variety of domains, including user experience design. manifest themselves as polar opposites on a scale, so we can easily organize the elements of our world. For example, based on our experiences with people, we know that some are shy and others are outgoing. When we meet new people, we may consciously or subconsciously categorize them according to that construct. An important element of the Personal Construct Theory is that each individual has his or her own unique set of constructs. Taking my example further, whether a person is shy or outgoing might not be important to you in your categorization scheme, but it might be very important to someone else. George Kelly hypothesized that people are constantly challenging and growing their construct systems, but those systems remain unique to the individual, and the sum of each persons experiences shapes them. In addition, the differences in peoples construct systems contribute to our different perceptions of the world and our behavior in it. For example, when choosing a place to live, one person might organize her choices using a construct that rates locations according to how easy it is to get to work, because she has experienced tough commutes in the past. Another person might organize his choices by climate or some other factor. It is this inherent difference in construct systems between people that introduces bias in research: The researcher has one set of constructs, and each participant has another. Especially in survey or structured-interview research, a researcher might ask questions he feels are important. Participants can answer those questions, but are they really the most

Personal construct theory

The Repertory Grid is a data extraction and analysis technique rooted in Personal Construct Theory, which was developed by George Kelly in the 1950s. The central theme of the Personal Construct Theory is that people organize their experiences with the world into conceptual classifications that we can differentiate and describe using attributes of those classifications called constructs. Often, these constructs

www.madpow.net

603.436.7177

40 Pleasant Street Suite 201 Portsmouth, NH 03801

The Repertory Grid: relevant questions? For example, in evaluating the user experience of a website, we can ask participants whether they think the site is trustworthy and why. Each participant might be able to answer those questions, but is a trustworthy site important to that participant for that domain? Even the most well-intentioned researcher, drafting questions that are as open-ended and unbiased as possible, still might lead some participants down an irrelevant path. Kelly developed the Repertory Grid as an interview technique that attempts to minimize the construct bias of the interviewer and systematically extract constructs for a particular domain that are important to participants. Why is it called the Repertory Grid? First, repertory comes from the word repertoire, which refers to a participants repertoire of constructs. The term grid refers to the data extraction and analysis procedure researchers use to gather and compare information from a number of participants in a study.

Michael Hawley how two of the three examples are different from the third. The researcher does not provide a starting point, but just asks the participant about the constructs that are important from his or her perspective. Often, the constructs that are most important to the participant are surprising, and sometimes not related to the topic that the researcher intended. However, this is key aspect of the exerciseto uncover what is important to the participant. Once the participant identifies a construct, or how two examples are different from the third, the participant names the two polar opposites of the construct, identifies which is good and which is bad, then writes its two contrasting poles at the opposite ends of a row in the grid. The participant continues the process of triading examples to identify additional constructs for the domain. Participants can change which two examples are alike and which are different for different constructs. The key is to elicit as many constructs as possible, without any suggestions from the researcher. The researcher can ask probing questions and ask the participant to think aloud, but suggesting dimensions for constructs introduces the bias that the method seeks to avoid. 3. Rating After identifying and naming the contrasting poles for constructs during the triading step of this process, the participant rates all of the original examples in the study (i.e. the 612 examples, including the three the participant used in triading), basing his or her ratings on the constructs the participant developed during triading. For each individual construct, the participant rates an example on a scale of 1 to 5, where 1 represents one end of the pole and 5 represents the other. For example, if the participant identified a construct whose two poles are organized and cluttered, the researcher asks the participant to rate each example on a scale from 1 to 5, where 1 is organized and 5 is cluttered.

the rePertory GrId Process

Traditionally, researchers conduct a Repertory Grid study by choosing several examples in a particular domain with which participants interact. Ideally, there will be 612 different examples that represent a wide variety of approaches and potential constructs. A Repertory Grid study then proceeds according to the following four general steps: 1. Selection During each session, either the participant or the researcher chooses to work with three random examples from the initial set. Ideally, there are multiple participants in the study, and each participant works independently, with a different set of examples. 2. Triading This is the core aspect of eliciting constructs without introducing bias from the researcher. The researcher asks the participant to identify

Depending on the number of examples and constructs the participant identified during the triading step, this rating process can take some time, so be sure to allow for it in your scheduling. 4. Analysis You can analyze the results of a Repertory Grid study both qualitatively and quantitatively. Often, a qualitative analysis is enough to develop a good understanding of the constructs that are important to the target audience. By reviewing notes from the triading sessions and conducting affinity diagramming sessions to assess the various participants constructs and language, researchers can identify themes that can inform their decision making for the domain. In addition, to statistically identify which constructs are most relevant and most clearly distinguish the selected examples, a researcher can apply factor analysis to the participants ratings of the examples. The result is a dendrogram or tree diagramsimilar to what you would get during a card sort exercisethat shows which examples are most closely associated with one another and also the selected examples most differentiating characteristics.

In my work, Ive applied the Repertory Grid method to user experience design in two different ways: 1) getting feedback on user interface paradigms during conceptual modeling 2) understanding a products competitors and positioning in its marketplace Getting Feedback on User Interface Paradigms During the conceptual modeling phase of a project, when I am developing paradigms for how a user might interact with a particular systemwhether I am redesigning a particular page or an entire processIve used the Repertory Grid to get feedback on which paradigm would work best. In this case, the examples for construct elicitation are three different user interface paradigms Ive either sketched, drawn as low-fidelity wireframes, or developed as visual comps. Generally, I ask participants to show me how they would complete particular tasks with each user interfacesimilar to the task scenarios participants follow during in a usability study. Because participants interactions with the three user interfaces help them develop some familiarity with each paradigm, they can then complete a Repertory Grid exercise comparing the three. The triading phase of this process is key. Instead of simply asking the participants which user interface they like best and hoping they have good answers when I ask them to explain why, the triading process brings out the specific attributes that differentiate the user interfaces in the minds of the participants. Additionally, the triading phase is important when comparing wireframes or other prototypes, because two things generally limit the rating and factor analysis: Fig.1 Sample Repertory Grid Layout With Cluster Analysis

aPPlyInG the rePertory GrId to user exPerIence desIGn

George Kelly was a clinical psychologist, so his application of the Repertory Grid was to help identify the constructs that his patients used in interacting with those around them. In his work, the 612 examples for the domain were significant people in the lives of the patients. Kelly used the Repertory Grid method to help patients understand their issues with interacting with those people. In user experience design, the subject matter is obviously different, but we can use the same process to identify the key constructs or considerations people have when interacting with systems.

www.madpow.net

603.436.7177

40 Pleasant Street Portsmouth, NH 03801

1) the number of user interface paradigms we can create 2) the number of user interfaces participants can interact with and retain during an interview session Analyzing a Competitive Marketplace During early phases of projects, Ive used a more traditional application of the Repertory Grid methodwith ratings and statistical analysis of constructsto develop a strong understanding of a products competitors and its current positioning in a market. For example, during the business intelligence phase of a project, when a product team is defining the business goals and objectives for a product, business stakeholders usually have their own perspectives on how a website or application fits in the marketplace and what customers perceptions are. However, their experiences and constructs are likely different from those of their customers. Therefore, what stakeholders think is important might not be important to customers at all. By conducting a Repertory Grid study, using competitive sites or products as examples and choosing participants who are familiar with those competitive products, a researcher can develop a strong understanding of customers perspectives on what is important. In this application of the Repertory Grid, Ive either asked participants to interact with the example systems to bring them back to top of mind or simply shown them images of a website, brand, or application to trigger their memories. Participants complete the triading process using three examples, then rate all of the examples according to the constructs theyve developed. The resulting factor analysis helps identify the differentiating characteristics of the product domain and positive characteristics on which we should focus. Additionally, the statistical aspect of the factor analysis is another tool that can aid in the presentation of the results to stakeholdersespecially if they respond positively to quantitative analysis.

conclusIon
In applying these variations of the Repertory Grid to user experience design, Ive found the method to be fun and engaging for participants and easy for researchers. The Repertory Grid method has a number of benefits for user experience research and design evaluation. Repertory Grid studies: quickly generate a large number of attributes, or constructs, that are useful in comparing different examples elicit differentiating attributes in the participants vocabulary rather than the researchers vocabulary identify constructs that are important to the participants rather than the researcher provide a structured process for eliciting feedback that is easy for participants to understand The most significant limitations of this method concern when you can use it effectively. Triading is an effective technique that you can use when you have just a few examples for comparison. However, to utilize the Repertory Grid to its full potential, it is best to have more numerous examples for comparison, and participants must develop some familiarity with all of them. As with any other qualitative interviewing method, there is potential for bias from a researcher who proposes constructs or leads participants during follow-up questions. However, when applicable, the Repertory Grid method helps researchers minimize bias while developing an understanding of a particular domain from the customers perspective. I recommend using this method as a component of your user-centered design toolkit.

resources
Kelly, George. The Psychology of Personal Constructs. New York: Norton, 1955. Jankowicz, Devi. The Easy Guide to Repertory Grids. New York: Wiley, 2003. Personal Construct Theory and Repertory Grid Interview User Group. Yahoo! Groups. Retrieved November 25, 2007. http://tech.groups.yahoo.com/ group/RepGrid

www.madpow.net

603.436.7177

40 Pleasant Street Portsmouth, NH 03801