research-article

Reviewing and Extending the Five-User Assumption: A Grounded Procedure for Interaction Evaluation

Authors:

Robert D. Macredie,

Jennifer Martin,

Terry YoungAuthors Info & Claims

ACM Transactions on Computer-Human Interaction (TOCHI), Volume 20, Issue 5

Article No.: 29, Pages 1 - 23

https://doi.org/10.1145/2506210

Published: 01 November 2013 Publication History

Abstract

The debate concerning how many participants represents a sufficient number for interaction testing is well-established and long-running, with prominent contributions arguing that five users provide a good benchmark when seeking to discover interaction problems. We argue that adoption of five users in this context is often done with little understanding of the basis for, or implications of, the decision. We present an analysis of relevant research to clarify the meaning of the five-user assumption and to examine the way in which the original research that suggested it has been applied. This includes its blind adoption and application in some studies, and complaints about its inadequacies in others. We argue that the five-user assumption is often misunderstood, not only in the field of Human-Computer Interaction, but also in fields such as medical device design, or in business and information applications. The analysis that we present allows us to define a systematic approach for monitoring the sample discovery likelihood, in formative and summative evaluations, and for gathering information in order to make critical decisions during the interaction testing, while respecting the aim of the evaluation and allotted budget. This approach -- which we call the Grounded Procedure -- is introduced and its value argued.

References

[1]

Alshamari, M. and Mayhew, P. 2009. Tech. Review: Current Issues of Usability Testing. IETE Tech. Rev. 26, 402--406.

[2]

Bias, R. G. and Mayhew, D. J. 2005. Cost-Justifying Usability: An Update for the Internet Age. Morgan Kaufmann Publishers, San Francisco, CA.

Digital Library

[3]

Borsci, S., Federici, S., Mele, M. L., Polimeno, D., and Londei, A. 2012. The bootstrap discovery behaviour model: Why five users are not enough to test user experience. In Cognitively Informed Intelligent Interfaces: Systems Design and Development, E. M. Alkhalifa and K. Gaid Eds., IGI Global Press, Hershey, PA.

[4]

Borsci, S., Londei, A., and Federici, S. 2011. The Bootstrap Discovery Behaviour (BDB): A new outlook on usability evaluation. Cognit. Process. 12, 23--31.

[5]

Caulton, D. A. 2001. Relaxing the homogeneity assumption in usability testing. Behav. Infor. Technol. 20, 1--7.

[6]

Crystal, A. and Greenberg, J. 2005. Usability of a metadata creation application for resource authors. Library Inf. Sci. Res. 27, 177--189.

[7]

Efron, B. 1979. Bootstrap methods: Another look at the jackknife. Ann. Stat. 7, 1--26.

[8]

Faulkner, L. 2003. Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behav. Res. Meth. 35, 379--383.

[9]

Federici, S., Borsci, S., and Stamerra, G. 2010. Web usability evaluation with screen reader users: Implementation of the Partial Concurrent Thinking Aloud technique. Cognit. Process. 11, 263--272.

[10]

Fishman, G. S. 1995. Monte Carlo: Concepts, Algorithms, and Applications. Springer, New York.

[11]

Food and Drug Administration (FDA) 2011. Draft guidance for industry and Food and Drug Administration staff - Applying human factors and usability engineering to optimize medical device design. U.S. Food and Drug Administration, Silver Spring, MD.

[12]

Fox, J. 2002. An R and S-Plus Companion to Applied Regression. SAGE.

Digital Library

[13]

Glaser, B. G. and Strauss, A. L. 1967. The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine Publishing Company, Chicago, IL.

[14]

Good, I. J. 1953. The population frequencies of species and the estimation of population parameters. Biometrika 40, 237--264.

[15]

Hertzum, M. and Jacobsen, N. E. 2003. The evaluator effect: A chilling fact about usability evaluation methods. Int. Jo. Hum.-Comput. Interact. 15, 183--204.

[16]

Hong, J. I., Heer, J., Waterson, S., and Landay, J. A. 2001. WebQuilt: A proxy-based approach to remote web usability testing. ACM Trans. on Inf. Syst. 19, 263--285.

Digital Library

[17]

ISO 1998. ISO 9241-11:1998, Ergonomic requirements for office work with visual display terminals, CEN, Brussels, BE.

[18]

Jelinek, F. 1997. Statistical Methods for Speech Recognition. MIT Press, Cambridge, MA.

Digital Library

[19]

Kirakowski, J. 2005. Chapter 18 - Summative usability testing: Measurement and sample size. In Cost-Justifying Usability 2nd Ed., R. G. Bias and D. J. Mayhew Eds., Morgan Kaufmann, San Francisco, CA, 519--553.

[20]

Kurosu, M. 2007. Concept of usability revisited. In Human-Computer Interaction: Interaction Design and Usability, J. Jacko Ed., Springer, Berlin, 579--586.

Digital Library

[21]

Lewis, J. R. 1994. Sample sizes for usability studies: Additional considerations. Human Factors: J. Human Factors Ergono. Soc. 36, 368--378.

[22]

Lewis, J. R. 2000. Validation of Monte Carlo estimation of problem discovery likelihood. Tech. rep. 29.3357, IBM, Raleigh, NC.

[23]

Lewis, J. R. 2001. Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. Int. J. Hum.-Comput. Interact. 13, 445--479.

[24]

Lewis, J. R. 2006. Sample sizes for usability tests: Mostly math, not magic. Interactions 13, 29--33.

Digital Library

[25]

Lindgaard, G. and Chattratichart, J. 2007. Usability testing: What have we overlooked? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, 1415--1424.

Digital Library

[26]

Manning, C. D. and Schutze, H. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA.

Digital Library

[27]

Molich, R. and Nielsen, J. 1990. Improving a human-computer dialogue. Comm. ACM 33, 338--348.

Digital Library

[28]

Nielsen, J. 1995. Severity ratings for usability problems. http://useit.com/papers/heuristic/severityrating.html.

[29]

Nielsen, J. 2000. Why you only need to test with 5 users. http://www.useit.com/alertbox/20000319.html.

[30]

Nielsen, J. 2012. How many test users in a usability study? http://www.useit.com/alertbox/number-of-test-users.html.

[31]

Nielsen, J. and Landauer, T. K. 1993. A mathematical model of the finding of usability problems. In Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems, ACM, New York, 206--213.

Digital Library

[32]

Nielsen, J. and Mack, R. L. 1994. Usability Inspection Methods. Wiley, New York.

Digital Library

[33]

Norman, D. A. 1983. Some observations on mental models. In Mental Models, D. Gentner and A. Steven Eds., Lawrence Earlbaum Associates, Hillsdale, NJ, 7--14.

[34]

Norman, D. A. 1988. The Psychology of Everyday Things. Basic Books, New York.

[35]

Norman, D. A. and Draper, S. W. 1986. User Centered System Design: New Perspectives on Human-Computer Interaction. Lawrence Erlbaum Associates Inc, Hillsdale, NJ.

Digital Library

[36]

Petrie, H. and Bevan, N. 2009. The evaluation of accessibility, usability, and user experience. In The Universal Access Handbook, C. Stephanidis Ed., CRC Press, London, UK.

[37]

Sauro, J. and Lewis, J. R. 2012. Quantifying the User Experience. Morgan Kaufmann, Waltham, MA.

Digital Library

[38]

Schmettow, M. 2008. Heterogeneity in the usability evaluation process. In Proceedings of the 22nd British HCI Group Annual Conference on People and Computers: Culture, Creativity, Interaction. Vol. 1, UK British Computer Society, Swinton, UK, 89--98.

Digital Library

[39]

Schmettow, M. 2009. Controlling the usability evaluation process under varying defect visibility. In Proceedings of the 23rd British HCI Group Annual Conference on People and Computers: Celebrating People and Technology. British Computer Society, Swinton, UK, 188--197.

Digital Library

[40]

Schmettow, M. 2012. Sample size in usability studies. Comm. ACM 55, 64--70.

Digital Library

[41]

Shneiderman, B. 1986. Designing the User Interface: Strategies for Effective Human-Computer Interaction. Addison-Wesley Longman Publishing Co., Inc, Boston, MA.

Digital Library

[42]

Spool, J. and Schroeder, W. 2001. Testing web sites: Five users is nowhere near enough. In Proceedings of the CHI’01 Extended Abstracts on Human Factors in Computing Systems. New York, 285--286.

Digital Library

[43]

Tullis, T. and Albert, W. 2008. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Morgan, Kaufmann.

Digital Library

[44]

Turner, C. W., Lewis, J. R., and Nielsen, J. 2006. Determining usability test sample size. In International Encyclopedia of Ergonomics and Human Factors, W. Karwowski Ed., CRC Press, Boca Raton, FL, 3084--3088.

[45]

Virzi, R. A. 1990. Streamlining the design process: Running fewer subjects. In Proceedings of the Human Factors Society 34th Annual Meeting. ACM, New York, 291--294.

[46]

Virzi, R. A. 1992. Refining the test phase of usability evaluation: How many subjects is enough? Hum. Fact. 34, 457--468.

Digital Library

[47]

Woolrych, A. and Cockton, G. 2001. Why and when five test users aren’t enough. In Proceedings of the Proceedings of IHM-HCI 2001 Conference, J. Vanderdonckt, A. Blandford, and A. Derycke Eds., Cépaduès Editions, London, UK, 105--108.

Cited By

Sharma SMudgal SGaur RChaturvedi JRulaniya SSharma P(2024)Navigating Sample Size Estimation for Qualitative ResearchJournal of Medical Evidence10.4103/JME.JME_59_245:2(133-139)Online publication date: 24-Jun-2024
https://doi.org/10.4103/JME.JME_59_24
Komianos VTsipis AKontopanagou K(2024)Introducing Digitized Cultural Heritage to Wider Audiences by Employing Virtual and Augmented Reality Experiences: The Case of the v-Corfu ProjectTechnologies10.3390/technologies1210019612:10(196)Online publication date: 13-Oct-2024
https://doi.org/10.3390/technologies12100196
Hertzum M(2024)Concurrent or Retrospective Thinking Aloud in Usability Tests: A Meta-Analytic ReviewACM Transactions on Computer-Human Interaction10.1145/366532731:3(1-29)Online publication date: 17-May-2024
https://dl.acm.org/doi/10.1145/3665327
Show More Cited By

Index Terms

Reviewing and Extending the Five-User Assumption: A Grounded Procedure for Interaction Evaluation
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
    2. HCI theory, concepts and models
  2. Interaction design
    1. Interaction design theory, concepts and paradigms

Recommendations

Factors to actors: implications of posthumanism for social justice work
SIGDOC '15: Proceedings of the 33rd Annual International Conference on the Design of Communication

Contexts, tools, and other nonhuman factors are central to the practice and scholarship of technical communication, particularly communication design. But viewed through the lens of posthumanism, these considerations shift from factors to actors: the ...
Leading participant-centered research: an argument for taking a more strategic role as user experience architects
SIGDOC '15: Proceedings of the 33rd Annual International Conference on the Design of Communication

In this experience report, we discuss our experiences in negotiating participant-based research in industry projects. For the past twenty years, academics and practitioners of participant-based research have worked to integrate practices and methods to ...
Five Provocations for Ethical HCI Research
CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

We present five provocations for ethics, and ethical research, in HCI. We discuss, in turn, informed consent, the researcher-participant power differential, presentation of data in publications, the role of ethical review boards, and, lastly, corporate-...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computer-Human Interaction

ACM Transactions on Computer-Human Interaction Volume 20, Issue 5

November 2013

129 pages

ISSN:1073-0516

EISSN:1557-7325

DOI:10.1145/2533682

Issue’s Table of Contents

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2013

Accepted: 01 July 2013

Revised: 01 June 2013

Received: 01 September 2012

Published in TOCHI Volume 20, Issue 5

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Engineering and Physical Sciences Research Council
MATCH programme

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

32
Total Citations
View Citations
1,004
Total Downloads

Downloads (Last 12 months)30
Downloads (Last 6 weeks)2

Reflects downloads up to 10 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sharma SMudgal SGaur RChaturvedi JRulaniya SSharma P(2024)Navigating Sample Size Estimation for Qualitative ResearchJournal of Medical Evidence10.4103/JME.JME_59_245:2(133-139)Online publication date: 24-Jun-2024
https://doi.org/10.4103/JME.JME_59_24
Komianos VTsipis AKontopanagou K(2024)Introducing Digitized Cultural Heritage to Wider Audiences by Employing Virtual and Augmented Reality Experiences: The Case of the v-Corfu ProjectTechnologies10.3390/technologies1210019612:10(196)Online publication date: 13-Oct-2024
https://doi.org/10.3390/technologies12100196
Hertzum M(2024)Concurrent or Retrospective Thinking Aloud in Usability Tests: A Meta-Analytic ReviewACM Transactions on Computer-Human Interaction10.1145/366532731:3(1-29)Online publication date: 17-May-2024
https://dl.acm.org/doi/10.1145/3665327
Biehl JPatel RLee A(2023)Toward the Design of Sensing-Based Medication Adherence Aids That Support Individualized Activities of Daily Living: Survey and Interviews With Patients and ProvidersJMIR Human Factors10.2196/4017310(e40173)Online publication date: 4-Jul-2023
https://doi.org/10.2196/40173
Bagheri FAbbasi FSadeghi MKhajouei R(2023)Evaluating the usability of a cancer registry system using Cognitive Walkthrough, and assessing user agreement with its problemsBMC Medical Informatics and Decision Making10.1186/s12911-023-02120-823:1Online publication date: 30-Jan-2023
https://doi.org/10.1186/s12911-023-02120-8
van Erven RCanedo E(2023)Measurement of User's Satisfaction of Digital Products through Emotion RecognitionProceedings of the XXII Brazilian Symposium on Software Quality10.1145/3629479.3629488(62-71)Online publication date: 7-Nov-2023
https://dl.acm.org/doi/10.1145/3629479.3629488
Paulino DCorreia AYagui MBarroso JLiberato MVivacqua AGrover ABigham JParedes H(2023)Exploring Stigmergic Collaboration and Task Modularity Through an Expert Crowdsourcing Annotation System: The Case of Storm Phenomena in the Euro-Atlantic RegionIEEE Access10.1109/ACCESS.2023.331959711(106485-106502)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3319597
Kirkscey R(2021)Development and Patient User Experience Evaluation of an mHealth Informational App for OsteoporosisInternational Journal of Human–Computer Interaction10.1080/10447318.2021.196577338:8(707-718)Online publication date: 11-Aug-2021
https://doi.org/10.1080/10447318.2021.1965773
Morris LBellon-Harn M(2021)Development and Functionality of an Internet-Based, Self-Managed Parent Training ProgramAmerican Journal of Speech-Language Pathology10.1044/2020_AJSLP-20-00080(1-12)Online publication date: 8-Feb-2021
https://doi.org/10.1044/2020_AJSLP-20-00080
Marino EBarbieri LColacino BFleri ABruno F(2021)An Augmented Reality inspection tool to support workers in Industry 4.0 environmentsComputers in Industry10.1016/j.compind.2021.103412127(103412)Online publication date: May-2021
https://doi.org/10.1016/j.compind.2021.103412
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents