research-article

The state of user experience evaluation practice

Authors:

Nuno Jardim NunesAuthors Info & Claims

NordiCHI '14: Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational

Pages 93 - 102

https://doi.org/10.1145/2639189.2641208

Published: 26 October 2014 Publication History

Abstract

Despite the growing importance for practice, user experience is often a blurry and doubtful concept both for newcomers and for the industry. Such ambiguity stems from a emergent community of practitioners with diverse backgrounds, to whom user experience encompasses countless interpretations.

This paper reports on an online survey deployed to grasp the state of user experience evaluation practice. We learned that evaluations are mainly conducted by HCIs, software engineers or designers and are perceived to strongly impact the user interface, as well as the business logic level. Additionally, informal, low cost methods are widely used and, although most methods rely on paper prototypes, a single artifact is used per evaluation and working prototypes are favored. Moreover, evaluations happen at multiple project phases and various methods are used. Finally, results shows that evaluations are constrained by evaluators' background or occupation. This compels the community to pursue an end-to-end methodology to prevent it.

References

[1]

Albert, W., Tullis, T., Tedesco, D., and more, & 0. Beyond the Usability Lab: Conducting Large-scale Online User Experience Studies. Morgan Kaufmann, Amsterdam, Burlington, MA, 2010.

Digital Library

[2]

Allen, J. and Chudley, J. Smashing UX Design: Foundations for Designing Online User Experiences. Wiley, Chichester, West Sussex, UK, 2012.

[3]

Alves, R. and Jardim Nunes, N. Towards a Taxonomy of Service Design Methods and Tools. In J. Falcão e Cunha, M. Snene and H. Nóvoa, eds., Exploring Services Science. Springer Berlin Heidelberg, 2013, 215--229.

[4]

Bargas-Avila, J. A. and Hornbæk, K. Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. Proceedings of the 2011 annual conference on Human factors in computing systems, (2011), 2689--2698.

Digital Library

[5]

Coolican, H. Research Methods and Statistics in Psychology. Hodder Education, 2009.

[6]

Csikszentmihalyi, M., PhD. Flow. Harper Perennial, New York.

[7]

Ebling, M. R. and John, B. E. On the contributions of different empirical data in usability testing. Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques, (2000), 289--296.

Digital Library

[8]

Garrett, J. J. The Elements of User Experience: User-Centered Design for the Web and Beyond. New Riders, Berkeley, CA, 2010.

Digital Library

[9]

Gothelf, J. and Seiden, J. Lean UX: Applying Lean Principles to Improve User Experience. O'Reilly Media, Beijing; Sebastopol, CA, 2013.

[10]

Hanington, B. and Martin, B. Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions. Rockport Publishers, 2012.

[11]

Hartson, R. and Pyla, P. The UX Book: Process and Guidelines for Ensuring a Quality User Experience. Morgan Kaufmann, Amsterdam; Boston, 2012.

Digital Library

[12]

Hassenzahl, M. and Carroll, J. Experience Design: Technology for All the Right Reasons. Morgan and Claypool Publishers, San Rafael, Calif., 2010.

[13]

Hollingsed, T. and Novick, D. G. Usability inspection methods after 15 years of research and practice. Proceedings of the 25th annual ACM international conference on Design of communication, (2007), 249--255.

Digital Library

[14]

II, B. J. P. and Gilmore, J. H. The Experience Economy, Updated Edition. Harvard Business Review Press, Boston, Mass, 2011.

[15]

Iivari, J. and Iivari, N. Varieties of user-centeredness. System Sciences, 2006. HICSS'06. Proceedings of the 39th Annual Hawaii International Conference on, (2006), 176a--176a.

Digital Library

[16]

Law, E., Roto, V., Vermeeren, A. P., Kort, J., and Hassenzahl, M. Towards a shared definition of user experience. CHI'08 extended abstracts on Human factors in computing systems, (2008), 2395--2398.

Digital Library

[17]

Law, E. L. C. The measurability and predictability of user experience. Proceedings of the 3rd ACM SIGCHI symposium on Engineering interactive computing systems, (2011), 1--10.

Digital Library

[18]

Mao, J.-Y., Vredenburg, K., Smith, P. W., and Carey, T. The state of user-centered design practice. Commun. ACM 48, 3 (2005), 105--109.

Digital Library

[19]

Nielsen, J. Usability engineering. Kaufmann, San Diego {u.a.}, 1993.

Digital Library

[20]

Nielsen, J. Designing web usability. New Riders, 2000.

[21]

Norman, D. A. The Psychology of Everyday Things. Perseus Books Group, 1990.

[22]

Redish, G. and Barnum, C. Overlap, Influence, Intertwining: The Interplay of UX and Technical Communication. Journal of Usability Studies 6, 3 (2011), 90--101.

Digital Library

[23]

Rubinstein, R., Hersh, H. M., and Ledgard, H. F. The human factor: designing computer systems for people. Digital Press, 1984.

Digital Library

[24]

Sauro, J. and Lewis, J. R. Quantifying the User Experience: Practical Statistics for User Research. Morgan Kaufmann, Amsterdam; Waltham, MA, 2008.

Digital Library

[25]

Shedroff, N. Experience Design. Waite Group Press, Indianapolis, Ind, 2001.

Digital Library

[26]

Shneiderman, B. Designing the user interface: strategies for effective human-computer-interaction. Addison Wesley Longman, 1998.

Digital Library

[27]

Tufte, E. R. The cognitive style of PowerPoint. Graphics Press, 2003.

Digital Library

[28]

Tullis, T. and Albert, W. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Elsevier Science, 2010.

[29]

Unger, R. and Chandler, C. A Project Guide to UX Design: For user experience designers in the field or in the making. New Riders, 2012.

Digital Library

[30]

Väänänen-Vainio-Mattila, K., Roto, V., and Hassenzahl, M. Towards practical user experience evaluation methods. EL-C. Law, N. Bevan, G. Christou, M. Springett & M. Lárusdóttir (eds.) Meaningful Measures: Valid Useful User Experience Measurement (VUUM), (2008), 19--22.

[31]

Statistical Test Selector | Laerd Statistics Premium. https://statistics.laerd.com/premium/sts/index.php.

[32]

Statistical Test Selector - Association - Part II | Laerd Statistics Premium. https://statistics.laerd.com/premium/sts/sts-association-2.php.

Cited By

Deters HDroste JHess AKlös VSchneider KSpeith TVogelsang A(2024)The X Factor: On the Relationship between User eXperience and eXplainabilityProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685352(1-12)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3679318.3685352
Ansari SPrasetya IDastani MDignum FKeller G(2024)EmoSTL: Formal Spatial-Temporal Verification of Emotion Specifications in Computer Games2024 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST60714.2024.00011(13-24)Online publication date: 27-May-2024
https://doi.org/10.1109/ICST60714.2024.00011
Faruk LPal DFunilkul SPerumal TMongkolnam P(2024)Introducing CASUX: A Standardized Scale for Measuring the User Experience of Artificial Intelligence Based Conversational AgentsInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2359206(1-25)Online publication date: 3-Jun-2024
https://doi.org/10.1080/10447318.2024.2359206
Show More Cited By

Index Terms

The state of user experience evaluation practice
1. Human-centered computing

Recommendations

Affordances for practice

This paper argues that Gibson's concept of affordance inserts a powerful conceptual lens for the study of sociomateriality as enacted in contemporary organizational practices. Our objective in this paper is to develop a comprehensive view of affordances ...
The turn to practice in HCI: towards a research agenda
CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

This paper argues that a new paradigm for HCI research, which we label the 'practice' perspective, has been emerging in recent years. This stands in contrast to the prevailing mainstream HCI paradigm, which we term the 'interaction' perspective. The '...
Towards Informed Practice in HCI for Development

Our paper provides an enriched understanding of the relationship between research and practice through the study of practitioners variously engaged in field research on technology interventions in the context of global development. By conducting a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

NordiCHI '14: Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational

October 2014

361 pages

ISBN:9781450325424

DOI:10.1145/2639189

General Chairs:
Virpi Roto,
Jonna Häkkilä,
Program Chairs:
Kaisa Väänänen-Vainio-Mattila,
Oskar Juhlin,
Publications Chairs:
Thomas Olsson,
Ebba Hvannberg

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 October 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Agencia Regional para o Desenvolvimento da Investigacao, Tecnologia e Inovacao (ARDITI)

Conference

NordiCHI '14

NordiCHI '14: The 8th Nordic Conference on Human-Computer Interaction

October 26 - 30, 2014

Helsinki, Finland

Acceptance Rates

NordiCHI '14 Paper Acceptance Rate 89 of 361 submissions, 25%;

Overall Acceptance Rate 379 of 1,572 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

37
Total Citations
View Citations
1,185
Total Downloads

Downloads (Last 12 months)89
Downloads (Last 6 weeks)4

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Deters HDroste JHess AKlös VSchneider KSpeith TVogelsang A(2024)The X Factor: On the Relationship between User eXperience and eXplainabilityProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685352(1-12)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3679318.3685352
Ansari SPrasetya IDastani MDignum FKeller G(2024)EmoSTL: Formal Spatial-Temporal Verification of Emotion Specifications in Computer Games2024 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST60714.2024.00011(13-24)Online publication date: 27-May-2024
https://doi.org/10.1109/ICST60714.2024.00011
Faruk LPal DFunilkul SPerumal TMongkolnam P(2024)Introducing CASUX: A Standardized Scale for Measuring the User Experience of Artificial Intelligence Based Conversational AgentsInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2359206(1-25)Online publication date: 3-Jun-2024
https://doi.org/10.1080/10447318.2024.2359206
Gholizadeh Ansari SPrasetya IDastani MKeller GPrandi DKifetew FDignum F(2024)PX-MBT: a Framework for Model-based Player Experience TestingScience of Computer Programming10.1016/j.scico.2024.103108(103108)Online publication date: Mar-2024
https://doi.org/10.1016/j.scico.2024.103108
Nakamura WC. de Oliveira EH. T. de Oliveira EConte T(2023)UX-MAPPER: A User eXperience Method to Analyze App Store ReviewsProceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems10.1145/3638067.3638109(1-11)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3638067.3638109
Wang RChen SLabbé CFredette MAbdessemed ACourtemanche FCoursaris CSénécal SLéger P(2023)UX Calculator: An Online Tool to Support User TestingDesign, User Experience, and Usability10.1007/978-3-031-35702-2_7(101-111)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-35702-2_7
Ansari SPrasetya IPrandi DKifetew FDastani MDignum FKeller G(2023)Model-based Player Experience Testing with Emotion Pattern VerificationFundamental Approaches to Software Engineering10.1007/978-3-031-30826-0_9(151-172)Online publication date: 22-Apr-2023
https://dl.acm.org/doi/10.1007/978-3-031-30826-0_9
Swoboda DBoasen JLéger PPourchon RSénécal S(2022)Comparing the Effectiveness of Speech and Physiological Features in Explaining Emotional Responses during Voice User Interface InteractionsApplied Sciences10.3390/app1203126912:3(1269)Online publication date: 25-Jan-2022
https://doi.org/10.3390/app12031269
Nakamura WMarques LRedmiles Dde Oliveira EConte T(2022)Investigating the Influence of Different Factors on the UX Evaluation of a Mobile ApplicationInternational Journal of Human–Computer Interaction10.1080/10447318.2022.210865839:20(3948-3968)Online publication date: 16-Aug-2022
https://doi.org/10.1080/10447318.2022.2108658
Namayala PKondo TManyilizu MHassan K(2022)Practical applicability of user experience capability/maturity models in the development processes of free and open-source software: a systematic literature reviewBehaviour & Information Technology10.1080/0144929X.2022.203234842:5(596-623)Online publication date: 8-Feb-2022
https://doi.org/10.1080/0144929X.2022.2032348
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten