Abstract
Null hypothesis significance testing (NHST) is a common statistical analysis method in HCI. But its usage and interpretation are often misunderstood. In particular, NHST does not offer the magnitude of differences observed, which is more desirable to determine the effect of comparative studies than the p value. Effect sizes and power analysis can mitigate over-reliance on the p value, and offer researchers better informed preparation and interpretation on experiments. Many research fields now require authors to include effect sizes in NHST results, and this trend is expected to be more and more common. In this chapter, I first discuss common misunderstandings of NHST and p value, and how effect sizes can complement them. I then present methods for calculating effect sizes with examples. I also describe another closely related topic, power analysis. Power analysis can be useful for appropriately designing experiments though it is not frequently used in HCI. I present power analysis methods and discuss how they should and should not be used.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
[R] coin mamual. http://cran.r-project.org/web/packages/coin/coin.pdf
[R] ezANOVA manual. http://cran.r-project.org/web/packages/ez/ez.pdf
[R] MBESS manual. http://cran.r-project.org/web/packages/MBESS/MBESS.pdf
[R] pwr manual. http://cran.r-project.org/web/packages/pwr/pwr.pdf
Cohen J (1998) Statistical power analysis for the behavioral sciences, 2nd edn. Academic Press, New York
Field A (2005) Statistical hell: effect sizes. http://www.statisticshell.com/docs/effectsizes.pdf
Field A (2009) Discovering statistics using SPSS, 3rd edn. Sage Publications
Fisher RA (1925) Statistical methods for research workers. Oliver and Boyd
Hoenig JN, Heisey DM (2001) The abuse of power: the pervasive fallacy of power calculations for data analysis. Am Stat 55:19–24
Kirk RE (2003) The importance of effect magnitude. In: Davis SF (ed) Handbook of research methods in experimental psychology. Oxford, pp 83–105
Mizumoto A, Takeuchi O (2008) Basics and considerations for reporting effect sizes in research papers. English Language Education Society 31:57–66. (written in Japanese)
Nakagawa S, Cuthill IC (2007) Effect size, confidence interval and statistical significance: a practical guide for biologists. Biol Rev 82(4):591–605
Newcombe RG (2012) Confidence intervals for proportions and related measures of effect size. CRC Press, Boca Raton
Thompson B (2007) Effect sizes, confidence intervals, and confidence intervals for effect sizes. Psychol Schools 44(5):423–432
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Yatani, K. (2016). Effect Sizes and Power Analysis in HCI. In: Robertson, J., Kaptein, M. (eds) Modern Statistical Methods for HCI. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-319-26633-6_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-26633-6_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-26631-2
Online ISBN: 978-3-319-26633-6
eBook Packages: Computer ScienceComputer Science (R0)