research-article

Public Access

Local Standards for Sample Size at CHI

Author:

Kelly CaineAuthors Info & Claims

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

Pages 981 - 992

https://doi.org/10.1145/2858036.2858498

Published: 07 May 2016 Publication History

Abstract

We describe the primary ways researchers can determine the size of a sample of research participants, present the benefits and drawbacks of each of those methods, and focus on improving one method that could be useful to the CHI community: local standards. To determine local standards for sample size within the CHI community, we conducted an analysis of all manuscripts published at CHI2014. We find that sample size for manuscripts published at CHI ranges from 1 -- 916,000 and the most common sample size is 12. We also find that sample size differs based on factors such as study setting and type of methodology employed. The outcome of this paper is an overview of the various ways sample size may be determined and an analysis of local standards for sample size within the CHI community. These contributions may be useful to researchers planning studies and reviewers evaluating the validity of results.

References

[1]

ACM SIGCHI 2015. Guide to a Successful Paper or Note Submission.

[2]

ACM SIGCHI 2015. Papers Versus Notes Whats the Difference.

[3]

Herman Aguinis and Erika Harden. 2009. Sample size rules of thumb: evaluating three common practices. In Statistical and methodological myths and urban legends: doctrine, verity and fable in the organizational and social sciences., Charles Lance and Robert Vandenberg Eds., NY, NY, 267--286.

[4]

William Albert and Thomas Tullis. 2013. Measuring the user experience: collecting, analyzing, and presenting usability metrics. Newnes.

Digital Library

[5]

Peter Bacchetti. 2002. Peer review of statistics in medical research: the other problem. BMJ: British Medical Journal 324, 7348, 1271.

[6]

Peter Bacchetti. 2010. Current sample size conventions: Flaws, harms, and alternatives. BMC Medicine 8, 1, 1--7. http://dx.doi.org/10.1186/1741--7015--8--17.

[7]

Peter Bacchetti, Steven G. Deeks, and Joseph M. McCune. 2011. Breaking Free of Sample Size Dogma to Perform Innovative Translational Research. Science Translational Medicine 3, 87 (2011-06--15 00:00:00), 87ps24--87ps24. http://dx.doi.org/10.1126/scitranslmed.3001628.

[8]

Peter Bacchetti, Charles E. McCulloch, and Mark R. Segal. 2008. Simple, Defensible Sample Sizes Based on Cost Efficiency. Biometrics 64, 2, 577--585. http://dx.doi.org/10.1111/j.1541-0420.2008.01004_1.x.

[9]

Louise Barkhuus and Jennifer A. Rode. 2007. From Mice to Men - 24 Years of Evaluation in CHI. In Proceedings of the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA 2007), ACM, 2180963. http://dx.doi.org/10.1145/1240624.2180963.

Digital Library

[10]

Kathy Baxter, Catherine Courage, and Kelly Caine. 2015. Understanding Your Users: A Practical Guide to User Research Methods. Morgan Kaufmann.

Digital Library

[11]

Simone Borsci, Robert D. Macredie, Julie Barnett, Jennifer Martin, Jasna Kuljis, and Terry Young. 2013. Reviewing and Extending the Five-User Assumption: A Grounded Procedure for Interaction Evaluation. ACM Trans. Comput.-Hum. Interact. 20, 5, 1--23. http://dx.doi.org/10.1145/2506210.

Digital Library

[12]

Paul Cairns. 2007. HCI... not as it should be: inferential statistics in HCI research. In Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI... but not as we know it-Volume 1 British Computer Society, 195--201.

Digital Library

[13]

Ed H. Chi. 2011. On the importance of Replication in HCI and Social Computing Research. In BLOG@CACM.

[14]

Jacob Cohen. 1962. The statistical power of abnormalsocial psychological research: A review. The Journal of Abnormal and Social Psychology 65, 3, 145--153. http://dx.doi.org/http://dx.doi.org.libproxy.clemson.edu/ 10.1037/h0045186.

[15]

Paul D Ellis. 2010. The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge University Press.

[16]

Barney G Glaser and Anselm L Strauss. 2009. The discovery of grounded theory: Strategies for qualitative research. Transaction Publishers.

[17]

Henry A Glick. 2011. Sample Size and Power for Cost Effectiveness Analysis (Part 2). Pharmacoeconomics 29, 4, 287--296.

[18]

Saul Greenberg and Bill Buxton. 2008. Usability evaluation considered harmful (some of the time). In Proceedings of the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy2008), ACM, 1357074, 111--120. http://dx.doi.org/10.1145/1357054.1357074.

Digital Library

[19]

Scott A. Hale. 2014. Global connectivity and multilinguals in the Twitter network. In Proceedings of the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada2014), ACM, 2557203, 833--842. http://dx.doi.org/10.1145/2556288.2557203.

Digital Library

[20]

Joseph Henrich, Steven J. Heine, and Ara Norenzayan. 2010. The weirdest people in the world? Behavioral and Brain Sciences 33, 2--3, 61--83. http://dx.doi.org/

[21]

Wonil Hwang and Gavriel Salvendy. 2010. Number of people required for usability evaluation: the 10±2 rule. Commun. ACM 53, 5, 130--133. http://dx.doi.org/10.1145/1735223.1735255.

Digital Library

[22]

Maurits Kaptein and Judy Robertson. 2012. Rethinking statistical analysis methods for CHI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems ACM, 1105--1114.

Digital Library

[23]

Helena Chmura Kraemer and Christine Blasey. 2015. How many subjects?: Statistical power analysis in research. Sage Publications.

[24]

Yong Liu, Jorge Goncalves, Denzil Ferreira, Bei Xiao, Simo Hosio, and Vassilis Kostakos. 2014. CHI 19942013: mapping two decades of intellectual progress through co-word analysis. In Proceedings of the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada2014), ACM, 2556969, 3553--3562. http://dx.doi.org/10.1145/2556288.2556969.

Digital Library

[25]

Jakob Nielsen. 1994. Estimating the number of subjects needed for a thinking aloud test. International journal of human-computer studies 41, 3, 385--397.

Digital Library

[26]

R Nuzzo. 2014. Statistical errors: P values, the 'gold standard' of statistical validity, are not as reliable as many scientists assume. Nature 506, 150, 52.

[27]

Robert J. Ploutz-Snyder, James Fiedler, and Alan H. Feiveson. 2014. Justifying small-n research in scientifically amazing settings: Challenging the notion that only "big-n" studies are worthwhile. Journal of Applied Physiology(2014-01-09 22:33:40). http://dx.doi.org/10.1152/japplphysiol.01335.2013.

[28]

Jenny Preece, Helen Sharp, and Yvonne Rogers. 2015. Interaction Design-beyond human-computer interaction. John Wiley & Sons.

[29]

Daniel Reed and Ed H. Chi. 2012. Online privacy; replicating research results. Commun. ACM 55, 10, 8--9. http://dx.doi.org/10.1145/2347736.2347739.

Digital Library

[30]

Robert Rosenthal. 1965. The volunteer subject. Human relations 18, 4, 389.

[31]

Albrecht Schmidt and Tovi Grossman. 2014. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems ACM, Toronto, Ontario, Canada, 4206.

[32]

Wendie Wulff and Dick E Mahling. 1990. An assessment of HCI: issues and implications. ACM SIGCHI Bulletin 22, 1, 80--87.

Digital Library

Cited By

Weese SMadison RGodley VCassibry TMeaseles MRush M(2024)Collaboration, Socialization, and Support: Accountability Partners in an Online Doctoral Learning EnvironmentThe Pinnacle: A Journal by Scholar-Practitioners10.61643/c302752:1Online publication date: Mar-2024
https://doi.org/10.61643/c30275
Savolainen KKujala S(2024)Testing Two Online Symptom Checkers With Vulnerable Groups: Usability Study to Improve Cognitive Accessibility of eHealth ServicesJMIR Human Factors10.2196/4527511(e45275)Online publication date: 8-Mar-2024
https://doi.org/10.2196/45275
Madaio MChen JWallach HWortman Vaughan J(2024)Tinker, Tailor, Configure, Customize: The Articulation Work of Contextualizing an AI Fairness ChecklistProceedings of the ACM on Human-Computer Interaction10.1145/36537058:CSCW1(1-20)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3653705
Show More Cited By

Index Terms

Local Standards for Sample Size at CHI
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Stratification and sample size of data sources for agricultural mathematical programming models

A comparison is made between the variance of the estimator of the total of a variable obtained from both a simple and a stratified random sampling, in which the sample sizes of some strata are equal to the stratum population size. It is shown that in ...
Sample Size Determination Algorithm for fingerprint-based indoor localization systems

Provision of accurate location information is an important task in the Internet of Things (IoT) applications and scenarios. This need has boosted the research and development of fingerprint based, indoor localization systems, since GPS information is ...
Density estimation-based method to determine sample size for random sample partition of big data
Abstract
Random sample partition (RSP) is a newly developed big data representation and management model to deal with big data approximate computation problems. Academic research and practical applications have confirmed that RSP is an efficient solution ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

May 2016

6108 pages

ISBN:9781450333627

DOI:10.1145/2858036

General Chairs:
Jofish Kaye
Yahoo
,
Allison Druin
University of Maryland / National Park Service
,
Program Chairs:
Cliff Lampe
University of Michigan
,
Dan Morris
Microsoft
,
Juan Pablo Hourcade
University of Iowa

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF

Conference

CHI'16

Sponsor:

SIGCHI

CHI'16: CHI Conference on Human Factors in Computing Systems

May 7 - 12, 2016

California, San Jose, USA

Acceptance Rates

CHI '16 Paper Acceptance Rate 565 of 2,435 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

310
Total Citations
View Citations
5,493
Total Downloads

Downloads (Last 12 months)1,455
Downloads (Last 6 weeks)174

Reflects downloads up to 26 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Weese SMadison RGodley VCassibry TMeaseles MRush M(2024)Collaboration, Socialization, and Support: Accountability Partners in an Online Doctoral Learning EnvironmentThe Pinnacle: A Journal by Scholar-Practitioners10.61643/c302752:1Online publication date: Mar-2024
https://doi.org/10.61643/c30275
Savolainen KKujala S(2024)Testing Two Online Symptom Checkers With Vulnerable Groups: Usability Study to Improve Cognitive Accessibility of eHealth ServicesJMIR Human Factors10.2196/4527511(e45275)Online publication date: 8-Mar-2024
https://doi.org/10.2196/45275
Madaio MChen JWallach HWortman Vaughan J(2024)Tinker, Tailor, Configure, Customize: The Articulation Work of Contextualizing an AI Fairness ChecklistProceedings of the ACM on Human-Computer Interaction10.1145/36537058:CSCW1(1-20)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3653705
Fan ZSanson ARames TCoutrix C(2024)Design and Perception of a Soft Shape Change Beneath a SmartwatchProceedings of the 35th Conference on l'Interaction Humain-Machine10.1145/3649792.3649800(1-14)Online publication date: 25-Mar-2024
https://dl.acm.org/doi/10.1145/3649792.3649800
Widdicks KKnowles BFriday ABlair G(2024)ICT under Constraint: Exposing Tensions in Collaboratively Prioritising ICT Innovation for Climate TargetsACM Journal on Responsible Computing10.1145/36482341:2(1-21)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3648234
Chordia IKim JLiu ZPark HGarrett LErete SLe Dantec CYip JHiniker A(2024)Tuning into the World: Designing Community Safety Technologies to Reduce Dysfunctional Fear of CrimeProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661578(3097-3116)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661578
Alzughbi HCaine KIqbal MKnijnenburg BLee HMcGregor SSidnam-Mauch E(2024)EPIC: Enhanced Privacy and Integrity Considerations for Research (Tutorial)Companion Proceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640544.3645249(166-168)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640544.3645249
Sistrunk ASelf NBiswas SLuther KVerdezoto NRamakrishnan N(2024)Redistrict: Online Public Deliberation Support that Connects and Rebuilds Inclusive CommunitiesProceedings of the ACM on Human-Computer Interaction10.1145/36373938:CSCW1(1-23)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637393
Kwon NSun TGao YZhao LWang XKim JHong S(2024)3DPFIX: Improving Remote Novices' 3D Printing Troubleshooting through Human-AI Collaboration DesignProceedings of the ACM on Human-Computer Interaction10.1145/36372888:CSCW1(1-33)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637288
Colley MSpeidel OStrohbeck JRixen JBelz JRukzio E(2024)Effects of Uncertain Trajectory Prediction Visualization in Highly Automated Vehicles on Trust, Situation Awareness, and Cognitive LoadProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314087:4(1-23)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631408
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents