Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/1978942.1979336acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience

Published: 07 May 2011 Publication History

Abstract

This paper reviews how empirical research on User Experience (UX) is conducted. It integrates products, dimensions of experience, and methodologies across a systematically selected sample of 51 publications from 2005-2009, reporting a total of 66 empirical studies. Results show a shift in the products and use contexts that are studied, from work towards leisure, from controlled tasks towards open use situations, and from desktop computing towards consumer products and art. Context of use and anticipated use, often named key factors of UX, are rarely researched. Emotions, enjoyment and aesthetics are the most frequently assessed dimensions. The methodologies used are mostly qualitative, and known from traditional usability studies, though constructive methods with unclear validity are being developed and used. Many studies use self-developed questionnaires without providing items or statistical validations. We discuss underexplored research questions and potential improvements of UX research.

References

[1]
R. Agarwal and E. Karahanna. Time flies when you're having fun: Cognitive absorption and beliefs about information technology usage. MIS Quarterly, 24(4):665--694, 2000.
[2]
L. Barkhuus and J. Rode. From Mice to Men--24 years of Evaluation in CHI. Proc. CHI '07, 2007.
[3]
K. Battarbee and I. Koskinen. Co-experience: user experience as interaction. CoDesign, 1(1):5--18, 2005.
[4]
M. Blythe, J. Reid, P. Wright, and E. Geelhoed. Interdisciplinary criticism: analysing the experience of riot! Behaviour & Information Technology, 25(2):127--139, 2006 *.
[5]
M. Blythe, J. Robinson, and D. Frohlich. Interaction design and the critics: what to make of the weegie. In Proc. NordiCHI '08, 53--62. ACM, 2008 *.
[6]
J. Coan and J. Allen. Handbook of emotion elicitation and assessment. Oxford University Press, 2007.
[7]
B. Costello and E. Edmonds. A study in play, pleasure and interaction design. In Proc. DPPI '07, 76--91. ACM, 2007 *.
[8]
B. Costello and E. Edmonds. Directed and emergent play. In Proc. C&C 09, 107--116. ACM, 2009 *.
[9]
M. Csikszentmihalyi. Flow and the Psychology of Discovery and Invention. HarperPerennial, New York, 1997.
[10]
P. Desmet. Designing Emotions. PhD thesis, Delft University of Technology, 2002.
[11]
S. Diefenbach and M. Hassenzahl. The Beauty Dilemma: beauty is valued but discounted in product choice. In Proc. CHI 2009, 1419--1426. ACM, 2009 *.
[12]
J. Forlizzi and K. Battarbee. Understanding experience in interactive systems. In Proc. DIS '04, 261--268. ACM, 2004.
[13]
J. Forlizzi and S. Ford. The building blocks of experience: an early framework for interaction designers. In Proc. DIS '00, 419--423. ACM, 2000.
[14]
W. Gaver, T. Dunne, and E. Pacenti. Cultural probes. Interactions, 6(1):21--29, 1999.
[15]
S. Gilroy, M. Cavazza, and M. Benayoun. Using affective trajectories to describe states of flow in interactive art. In Proc. ACE '09, 165--172. ACM, 2009 *.
[16]
L. Gong. Is happy better than sad even if they are both non-adaptive? Effects of emotional expressions of talking-head interface agents. Int. Journal of Human-Computer Studies, 65(3):183--191, 2007.
[17]
W. Gray and M. Salzman. Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction, 13(3):203--261, 1998.
[18]
J. Hartmann, A. De Angeli, and A. Sutcliffe. Framing the user experience: information biases on website quality judgement. In Proc. CHI '08, 855--864. ACM, 2008 *.
[19]
J. Hartmann, A. Sutcliffe, and A. Angeli. Towards a theory of user judgment of aesthetics and user interface quality. ACM TOCHI, 15(4):1--30, 2008 *.
[20]
M. Hassenzahl. The thing and I: understanding the relationship between user and product. In M. Blythe, C. Overbeeke, A. Monk, and P. Wright, editors, Funology: From Usability to Enjoyment, 31--42. Dordrecht: Kluwer, 2003.
[21]
M. Hassenzahl. The interplay of beauty, goodness, and usability in interactive products. Human-Computer Interaction, 19(4):319--349, 2004.
[22]
M. Hassenzahl. User experience (UX): towards an experiential perspective on product quality. In Proc. IHM '08, 11--15. ACM, 2008.
[23]
M. Hassenzahl, A. Beu, and M. Burmester. Engineering joy. IEEE Software, 18(1):70--76, 2001.
[24]
M. Hassenzahl, S. Diefenbach, and A. Göritz. Needs, affect, and interactive products -- facets of user experience. Interacting with Computers, 22(5):353--362, 2010.
[25]
M. Hassenzahl and A. Monk. The inference of perceived usability from beauty. Human-Computer Interaction, 25(3):235--260, 2010.
[26]
M. Hassenzahl, A. Platz, M. Burmester, and K. Lehner. Hedonic and ergonomic quality aspects determine a software's appeal. In Proc. CHI '00, 201--208. ACM, 2000.
[27]
M. Hassenzahl and N. Sandweg. From mental effort to perceived usability: transforming experiences into summary assessments. In Proc. CHI '04, 1283--1286. ACM, 2004.
[28]
M. Hassenzahl, M. Schöbel, and T. Trautmann. How motivational orientation influences the evaluation and choice of hedonic and pragmatic interactive products: The role of regulatory focus. Interacting with Computers, 20(4--5):473--479, 2008 *.
[29]
M. Hassenzahl and N. Tractinsky. User experience -- a research agenda. Behaviour & Information Technology, 25(2):91--97, 2006.
[30]
M. Hassenzahl and D. Ullrich. To do or not to do: Differences in user experience and retrospective judgments depending on the presence or absence of instrumental goals. Interacting with Computers, 19(4):429--437, 2007 *.
[31]
J. Heikkinen, T. Olsson, and K. V\"a\"an\"anen-Vainio-Mattila. Expectations for user experience in haptic communication with mobile devices. In Proc. MobileHCI '09. ACM, 2009 *.
[32]
M. Hertzum. Images of Usability. International Journal of Human-Computer Interaction, 26(6):567--600, 2010.
[33]
K. Hornbæk. Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies, 64(2):79--102, 2006.
[34]
K. Hornbæk. Dogmas in the assessment of usability evaluation methods. Behaviour & Information Technology, 29(1):97--111, 2010.
[35]
M. Huang. Web performance scale. Information & Management, 42(6):841--852, 2005.
[36]
ISO. 9241--210: Ergonomics of human system interaction-Part 210: Human-centred design for interactive systems. International Organization for Standardization, 2008.
[37]
M. Isomursu, M. Tähti, S. Väinämö, and K. Kuutti. Experimental evaluation of five methods for collecting emotions in field settings with mobile applications. International Journal of Human-Computer Studies, 65(4):404--418, 2007 *.
[38]
S. Jackson and H. Marsh. Development and validation of a scale to measure optimal experience: The Flow State Scale. Journal of Sport and Exercise Psychology, 18:17--35, 1996.
[39]
S. Jacobson and A. Pirinen. Disabled persons as lead users in the domestic environment. In Proc. DPPI '07, 158--167. ACM, 2007 *.
[40]
P. Jordan. Designing pleasurable products: An introduction to the new human factors. CRC, 2002.
[41]
S. Jumisko-Pyykkö, M. Weitzel, and D. Strohmeier. Designing for user experience: what to expect from mobile 3d tv and video? In Proc. UXTV '08, 183--192. ACM, 2008 *.
[42]
E. Karapanos, J. Zimmerman, J. Forlizzi, and J. Martens. User experience over time: an initial framework. In Proc. CHI '09, 729--738. ACM, 2009 *.
[43]
E. Karapanos, J. Zimmerman, J. Forlizzi, and J. Martens. Measuring the Dynamics of Remembered Experience Over Time. Interacting with Computers, 22(5):328--335, 2010.
[44]
A. Koca, M. Funk, E. Karapanos, A. Rozinat, and N. van der Gaarden. Grasping product pragmatics: a case with internet on tv. In Proc. UXTV '08, 193--202. ACM, 2008 *.
[45]
P. Kourouthanassis, G. Giaglis, and A. Vrechopoulos. Enhancing user experience through pervasive information systems: The case of pervasive retailing. International Journal of Information Management, 27(5):319--335, 2007 *.
[46]
J. Landis and G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33(1):159--174, 1977.
[47]
P. J. Lang. Behavioral treatment and bio-behavioral assessment: Computer applications. In J. B. Sidowski, J. H. Johnson, and T. A. Williams, editors, Technology in mental health care delivery systems, 119--137. Ablex Publishing, Norwood, NJ, 1980.
[48]
M. Lankes, S. Riegler, A. Weiss, T. Mirlacher, M. Pirker, and M. Tscheligi. Facial expressions as game input with different emotional feedback conditions. In Proc. ACE '08, 253--256. ACM, 2008 *.
[49]
T. Lavie and N. Tractinsky. Assessing dimensions of perceived visual aesthetics of web sites. International Journal of Human-Computer Studies, 60(3):269--298, 2004.
[50]
E. Law, V. Roto, M. Hassenzahl, A. Vermeeren, and J. Kort. Understanding, scoping and defining user experience: a survey approach. In Proc. CHI '09, 719--728. ACM, 2009.
[51]
E. Law and P. van Schaik. Modelling user experience -- an agenda for research and practice. Interacting with Computers, 22(5):313--322, 2010.
[52]
T. Leong, S. Howard, and F. Vetere. Choice: abidcating or exercising? In Proc. CHI '08, 715--724. ACM, 2008 *.
[53]
A. Lucero, T. Lashina, E. Diederiks, and T. Mattelm\"aki. How probes inform and influence the design process. In Proc. DPPI '07, 377--391. ACM, 2007 *.
[54]
S. Mahlke. Aesthetic and Symbolic Qualities as Antecedents of Overall Judgements of Interactive Products. People and Computers XX -- Engage, 57--64, 2007.
[55]
S. Mahlke and M. Thüring. Studying antecedents of emotional experiences in interactive contexts. In Proc. CHI '07, 915--918. ACM, 2007 *.
[56]
R. Mandryk, K. Inkpen, and T. Calvert. Using psychophysiological techniques to measure user experience with entertainment techno- logies. Behaviour & Information Technology, 25(2):141--158, 2006 *.
[57]
N. Mathiasen and S. Bødker. Threats or threads: from usable security to secure experience? In Proc. NordiCHI '08, 283--289. ACM, 2008 *.
[58]
E. McAuley. Psychometric Properties of the Intrinsic Motivation Inventory in a Competitive Sport Setting. Research Quarterly for Exercise and Sport, 60(1):48--58, 1989.
[59]
J. McCarthy and P. Wright. Technology as experience. Interactions, 11(5):43, 2004.
[60]
J. McCarthy, P. Wright, J. Wallace, and A. Dearden. The experience of enchantment in human-computer interaction. Personal and Ubiquitous Computing, 10(6):369--378, 2006.
[61]
D. McKay, S. Cunningham, and K. Thomson. Exploring the user experience through collage. In Proc. CHINZ '06, 109--115. ACM, 2006 *.
[62]
D. Moher, D. Cook, S. Eastwood, I. Olkin, D. Rennie, and D. Stroup. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. The Lancet, 354:1896--1900, 1999.
[63]
M. Montola, T. Nummenmaa, A. Lucero, M. Boberg, and H. Korhonen. Applying game achievement systems to enhance user experience in a photo sharing service. In Proc. MindTrek '09, 94--97. ACM, 2009 *.
[64]
M. Moshagen and M. Thielsch. Facets of visual aesthetics. Int. Journal of Human-Computer Studies, 68(10):689--709, 2010.
[65]
M. Ní Chonchúir and J. McCarthy. The enchanting potential of technology: a dialogical case study of enchantment and the internet. Personal and Ubiquitous Computing, 12(5):401--409, 2008 *.
[66]
D. Norman. Emotional design. Citeseer, 2004.
[67]
H. O'Brien and E. Toms. What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American Society for Information Science and Technology, 59(6):938--955, 2008 *.
[68]
H. O'Brien and E. Toms. The development and evaluation of a survey to measure user engagement. Journal of the American Society for Information Science and Technology, 61(1):50--69, 2010.
[69]
M. Obrist, A. Meschtscherjakov, and M. Tscheligi. User Experience Evaluation in the Mobile Context. Mobile TV: Customizing Content and Experience, 195--204, 2010.
[70]
Y. Rogers, K. Connelly, L. Tedesco, W. Hazlewood, A. Kurtz, R. Hall, J. Hursey, and T. Toscos. Why it's worth the hassle: the value of in-situ studies when designing Ubicomp. In Proc. UbiComp '07, 336--353. Springer-Verlag, 2007 *.
[71]
P. Ross, C. Overbeeke, S. Wensveen, and C. Hummels. A designerly critique on enchantment. Personal and Ubiquitous Computing, 12(5):359--371, 2008 *.
[72]
K. Scherer. What are emotions? And how can they be measured? Social Science Information, 44(4):695, 2005.
[73]
D. Smith, C. Irby, R. Kimball, and B. Verplank. Designing the Star user interface. Byte, 7(4):242--282, 1982.
[74]
X. Sun and A. May. The role of spatial contextual factors in mobile personalization at large sports events. Personal and Ubiquitous Computing, 13(4):293--302, 2009 *.
[75]
A. Sutcliffe. Assessing the reliability of heuristic evaluation for website attractiveness and usability. In Proc. HICSS '02, 1838--1847. IEEE Computer Society, 2002.
[76]
D. Swallow, M. Blythe, and P. Wright. Grounding experience: relating theory and method to evaluate the user experience of smartphones. In Proc. EACE '05, 91--98. ACM, 2005 *.
[77]
N. Tractinsky, A. Katz, and D. Ikar. What is beautiful is usable. Interacting with Computers, 13(2):127--145, 2000.
[78]
A. Tuch, J. Bargas-Avila, and K. Opwis. Symmetry and aesthetics in website design: It's a man's business. Computers in Human Behavior, 26(6):1831--1837, 2010.
[79]
A. Tuch, J. Bargas-Avila, K. Opwis, and F. Wilhelm. Visual complexity of websites: Effects on users' experience, physiology, performance, and memory. International Journal of Human-Computer Studies, 67(9):703--715, 2009.
[80]
T. Tullis and W. Albert. Measuring The User Experience: collecting, analyzing, and presenting usability metrics. Morgan Kaufmann, 2008.
[81]
M. Turunen, A. Kallinen, I. Sànchez, et al. Multimodal interaction with speech and physical touch interface in a media center application. In Proc. ACE '09, 19--26. ACM, 2009 *.
[82]
K. V\"a\"an\"anen-Vainio-Mattila and M. W\"aljas. Developing an expert evaluation method for user eXperience of cross-platform web services. In MindTrek '09, 162--169. ACM, 2009 *.
[83]
H. V\"a\"at\"aj\"a, T. Koponen, and V. Roto. Developing practical tools for user experience evaluation: a case from mobile news journalism. In Proc. ECCE '09, 1--8. ACM, 2009 *.
[84]
K. van Boerdonk, R. Tieben, S. Klooster, and E. van den Hoven. Contact through canvas: an entertaining encounter. Personal and Ubiquitous Computing, 13(8):551--567, 2009 *.
[85]
H. Van der Heijden. User acceptance of hedonic information systems. MIS Quarterly, 28(4):695--704, 2004.
[86]
P. van Schaik and J. Ling. Modelling user experience with web sites: Usability, hedonic value, beauty and goodness. Interacting with Computers, 20(3):419--432, 2008 *.
[87]
P. van Schaik and J. Ling. The role of context in perceptions of the aesthetics of web pages over time. International Journal of Human-Computer Studies, 67(1):79--89, 2009 *.
[88]
P. van Schaik and J. Ling. An integrated model of interaction experience for information retrieval in a Web-based encyclopaedia. Interacting with Computers, {in press}, 2010.
[89]
R. Wakkary and M. Hatala. ec (h) o: Situated Play in a Tangible and Audio Museum Guide. In Proc. DIS '06, page 290. ACM, 2006 *.
[90]
D. Wilfinger, M. Pirker, R. Bernhaupt, and M. Tscheligi. Evaluating and investigating an iTV interaction concept in the field. In Proc. EuroITV '09, 175--178. ACM, 2009 *.
[91]
P. Wright and M. Blythe. User experience research as an inter- discipline: Towards a UX Manifesto. In Proc. BHCI '07, 65--70, 2007.
[92]
D. Xu, J. Read, G. Sim, B. McManus, and P. Qualter. Children and 'smart' technologies: can children's experiences be interpreted and coded? In Proc. BCS HCI '09, 224--231. British Computer Society, 2009.

Cited By

View all
  • (2024)Beyond usability: innovation in the digital age as an antidote to social isolationInternational Journal of Innovation10.5585/2024.2608812:3(e26088)Online publication date: 19-Nov-2024
  • (2024)Lived experience in human-building interaction (HBI): an initial frameworkFrontiers in Computer Science10.3389/fcomp.2023.12339045Online publication date: 4-Jan-2024
  • (2024)Re-visiting the E-SERVQUAL and Expectancy Disconfirmation Theory in the Context of the Service Industry in MalaysiaInformation Management and Business Review10.22610/imbr.v16i3(I)S.416816:3(I)S(981-993)Online publication date: 9-Oct-2024
  • Show More Cited By

Index Terms

  1. Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience
    Index terms have been assigned to the content through auto-classification.

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
    May 2011
    3530 pages
    ISBN:9781450302289
    DOI:10.1145/1978942
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 07 May 2011

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. meta-analysis
    2. review
    3. usability
    4. user experience
    5. ux

    Qualifiers

    • Research-article

    Conference

    CHI '11
    Sponsor:

    Acceptance Rates

    CHI '11 Paper Acceptance Rate 410 of 1,532 submissions, 27%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)483
    • Downloads (Last 6 weeks)43
    Reflects downloads up to 02 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Beyond usability: innovation in the digital age as an antidote to social isolationInternational Journal of Innovation10.5585/2024.2608812:3(e26088)Online publication date: 19-Nov-2024
    • (2024)Lived experience in human-building interaction (HBI): an initial frameworkFrontiers in Computer Science10.3389/fcomp.2023.12339045Online publication date: 4-Jan-2024
    • (2024)Re-visiting the E-SERVQUAL and Expectancy Disconfirmation Theory in the Context of the Service Industry in MalaysiaInformation Management and Business Review10.22610/imbr.v16i3(I)S.416816:3(I)S(981-993)Online publication date: 9-Oct-2024
    • (2024)Enhancing Existing Reality: A Theoretical Analysis of User-focused Mixed Reality ResearchArchives of Design Research10.15187/adr.2024.11.37.5.24537:5(245-268)Online publication date: 30-Nov-2024
    • (2024)Guidelines for Disc Golf Applications and Design Principles for SportsHCI: A Human-Centered ApproachProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3701584(396-411)Online publication date: 1-Dec-2024
    • (2024)Challenges and Opportunities for Tool Adoption in Industrial UX Research CollaborationsProceedings of the ACM on Human-Computer Interaction10.1145/36869828:CSCW2(1-27)Online publication date: 8-Nov-2024
    • (2024)An Umbrella Review of Reporting Quality in CHI Systematic Reviews: Guiding Questions and Best Practices for HCIACM Transactions on Computer-Human Interaction10.1145/368526631:5(1-55)Online publication date: 31-Jul-2024
    • (2024)Challenges of UX research and long-term UX: A multiple case study with software startupsProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685381(1-11)Online publication date: 13-Oct-2024
    • (2024)Ah-Aloud Method to Comprehend Time-Series Emotion Observation During Gameplay: An Initial Investigation with Japanese SpeakersProceedings of the ACM on Human-Computer Interaction10.1145/36770568:CHI PLAY(1-23)Online publication date: 15-Oct-2024
    • (2024)Charting User Experience in Physical Human–Robot InteractionACM Transactions on Human-Robot Interaction10.1145/365905813:2(1-29)Online publication date: 28-Jun-2024
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media