Article

Cooperative usability testing: complementing usability tests with user-supported interpretation sessions

Authors:

Erik Frøkjær,

Kasper HornbækAuthors Info & Claims

CHI EA '05: CHI '05 Extended Abstracts on Human Factors in Computing Systems

Pages 1383 - 1386

https://doi.org/10.1145/1056808.1056922

Published: 02 April 2005 Publication History

Get Access

Abstract

Recent criticism of think-aloud testing (TA) discusses discrepancies between theory and practice, the artificiality of the test situation, and inconsistencies in the evaluators' interpretation of the process. Rather than enforcing a more strict TA procedure, we describe Cooperative Usability Testing (CUT), where test users and evaluators join expertise to understand the usability problems of the application evaluated. CUT consists of two sessions. In the interaction session, the test user tries out the application to uncover potential usability problems while the evaluators mainly observe, e.g. as in TA or contextual inquiry. In the interpretation session, evaluators and test users discuss what they consider the most important usability problems, supported by a video of the interaction session. In an exploratory study comparing CUT to TA, seven evaluators find that interpretation sessions contribute important usability information compared to TA. Also test users found participation in the interpretation session interesting.

References

[1]

Beyer, H. & Holtzblatt, K. Contextual Design, Morgan Kaufman Publishers, 1998.

Google Scholar

[2]

Boren, M. T. & Ramey, J. Thinking Aloud: Reconciling Theory and Practice, IEEE Transactions on Professional Communication, 43, 3 (2000), 261--277.

Crossref

Google Scholar

[3]

Buur, J. & Bagger, K. Replacing Usability Testing With User Dialogue, Communications of the ACM, 42, 5 (1999), 63--66.

Digital Library

Google Scholar

[4]

Dumas J. S., User-Based Evaluations, in Jacko, J. A. & Sears, A. The Human-Computer Interaction Handbook, Lawrence Erlbaum Associates, 2003, 1093--1117.

Digital Library

Google Scholar

[5]

Ericsson, K. A. & Simon, H. Protocol Analysis: Verbal Reports As Data, Revised Edition, MIT Press, Cambridge, MA, 1993.

Google Scholar

[6]

Hertzum, M. & Jacobsen, N. E. The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods, International Journal of Human-Computer Interaction, 13 (2001), 421--443.

Crossref

Google Scholar

[7]

Hornbæk, K. & Frষষkjær, E. Usability Inspection by Metaphors of Human Thinking Compared to Heuristic Evaluation, International Journal of Human-Computer Interaction, 17, 3 (2004), 357--374.

Crossref

Google Scholar

[8]

Hornbæk, K. & Frষষkjær, E. Two Psychology-Based Usability Inspection Techniques Studied in a Diary Experiment, Proc. Nordichi 2004, ACM Press (2004)

Digital Library

Google Scholar

[9]

Molich, Rolf, User testing, Discount user testing, 2003, www.dialogdesign.dk.

Google Scholar

[10]

Nielsen, J. Usability Engineering, Academic Press, San Diego CA, 1993.

Digital Library

Google Scholar

[11]

Nielsen, J., Christiansen, N., Clemmensen, T., & Yssing, C. Mindtape-a Technique in Verbal Protocol Analysis, HCI International, Lawrence Earlbaum Associates (2003), 188--192.

Google Scholar

[12]

Vredenburg, K., Mao, J.-Y., Smith, P. W., & Carey, T. A Survey of User-Centered Design Practice, Proc. CHI 2002, ACM Press (2002), 472--478.

Digital Library

Google Scholar

Cited By

View all

Jones SPedraza Pineros IHajas DZong JSatyanarayan A(2024)“Customization is Key”: Reconfigurable Textual Tokens for Accessible Data VisualizationsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641970(1-14)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641970
Sinabell IAmmenwerth E(2024)Challenges and recommendations for eHealth usability evaluation with elderly users: systematic review and case studyUniversal Access in the Information Society10.1007/s10209-022-00949-w23:1(455-474)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s10209-022-00949-w
O'Brien LWilson S(2023)Talking about Thinking Aloud: Perspectives from Interactive Think-Aloud PractitionersJournal of User Experience 10.5555/3605526.360552818:3(113-132)Online publication date: 20-Jun-2023
https://dl.acm.org/doi/10.5555/3605526.3605528
Show More Cited By

Index Terms

Cooperative usability testing: complementing usability tests with user-supported interpretation sessions
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

Usability testing: what have we overlooked?
CHI '07: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

For more than a decade, the number of usability test participants has been a major theme of debate among usability practitioners and researchers keen to improve usability test performance. This paper provides evidence suggesting that the focus be ...
Low-cost usability testing
Software quality and software testing in internet times

Usability is an important aspect of software products. However, in practice not much attention is given to this issue during testing. Testers often do not have the knowledge, instruments and/or time available to test for usability. This article ...
Semiotic analysis combined with usability and ergonomic testing for evaluation of icons in medical user interface
IndiaHCI '11: Proceedings of the 3rd Indian Conference on Human-Computer Interaction

In this research, we have evaluated the medical icons and iconic interfaces of touch screen ventilator systems used in Intensive Care Unit (ICU). Precise communication through iconic interface between ventilator system and medical users like physicians ...

Comments

Information & Contributors

Information

Published In

CHI EA '05: CHI '05 Extended Abstracts on Human Factors in Computing Systems

April 2005

1358 pages

ISBN:1595930027

DOI:10.1145/1056808

Conference Chair:
Gerrit van der Veer
Vrije Universiteit, The Netherlands
,
Program Chair:
Carolyn Gale
Stanford University, Stanford, CA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 April 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

CHI05

Sponsor:

CHI05: CHI 2005 Conference on Human Factors in Computing Systems

April 2 - 7, 2005

OR, Portland, USA

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
1,486
Total Downloads

Downloads (Last 12 months)37
Downloads (Last 6 weeks)6

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Jones SPedraza Pineros IHajas DZong JSatyanarayan A(2024)“Customization is Key”: Reconfigurable Textual Tokens for Accessible Data VisualizationsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641970(1-14)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641970
Sinabell IAmmenwerth E(2024)Challenges and recommendations for eHealth usability evaluation with elderly users: systematic review and case studyUniversal Access in the Information Society10.1007/s10209-022-00949-w23:1(455-474)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s10209-022-00949-w
O'Brien LWilson S(2023)Talking about Thinking Aloud: Perspectives from Interactive Think-Aloud PractitionersJournal of User Experience 10.5555/3605526.360552818:3(113-132)Online publication date: 20-Jun-2023
https://dl.acm.org/doi/10.5555/3605526.3605528
Zong JLee CLundgard AJang JHajas DSatyanarayan A(2022)Rich Screen Reader Experiences for Accessible Data VisualizationComputer Graphics Forum10.1111/cgf.1451941:3(15-27)Online publication date: 12-Aug-2022
https://doi.org/10.1111/cgf.14519
Singh KDabas SSinghal JGautam G(2020)HungerDeal: India's First Food Delivery Price Comparison AppProceedings of the 11th Indian Conference on Human-Computer Interaction10.1145/3429290.3429305(76-81)Online publication date: 5-Nov-2020
https://dl.acm.org/doi/10.1145/3429290.3429305
Paay JKjeldskov JSorensen FJensen TTirosh O(2019)Weight-MateProceedings of the 31st Australian Conference on Human-Computer-Interaction10.1145/3369457.3369466(95-105)Online publication date: 2-Dec-2019
https://dl.acm.org/doi/10.1145/3369457.3369466
Reeves S(2019)How UX Practitioners Produce Findings in Usability TestingACM Transactions on Computer-Human Interaction10.1145/329909626:1(1-38)Online publication date: 30-Jan-2019
https://dl.acm.org/doi/10.1145/3299096
FØlstad A(2017)Users' design feedback in usability evaluationHuman-centric Computing and Information Sciences10.1186/s13673-017-0100-y7:1(1-19)Online publication date: 1-Dec-2017
https://dl.acm.org/doi/10.1186/s13673-017-0100-y
Sagar KSaha A(2017)A systematic review of software usability studiesInternational Journal of Information Technology10.1007/s41870-017-0048-1Online publication date: 11-Dec-2017
https://doi.org/10.1007/s41870-017-0048-1
Andersen SRasmussen CFrøkjær E(2017)Bringing Content Understanding into Usability Testing in Complex Application Domains—a Case Study in eHealthDesign, User Experience, and Usability: Theory, Methodology, and Management10.1007/978-3-319-58634-2_25(327-341)Online publication date: 14-May-2017
https://doi.org/10.1007/978-3-319-58634-2_25
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Usability testing: what have we overlooked?

Low-cost usability testing

Semiotic analysis combined with usability and ergonomic testing for evaluation of icons in medical user interface

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations