research-article

Touchstone2: An Interactive Environment for Exploring Trade-offs in HCI Experiment Design

Authors:

Alexander Eiselmayer,

Chat Wacharamanotham,

Michel Beaudouin-Lafon,

Wendy E. MackayAuthors Info & Claims

CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

Paper No.: 217, Pages 1 - 11

https://doi.org/10.1145/3290605.3300447

Published: 02 May 2019 Publication History

Abstract

Touchstone2 offers a direct-manipulation interface for generating and examining trade-offs in experiment designs. Based on interviews with experienced researchers, we developed an interactive environment for manipulating experiment design parameters, revealing patterns in trial tables, and estimating and comparing statistical power. We also developed TSL, a declarative language that precisely represents experiment designs. In two studies, experienced HCI researchers successfully used Touchstone2 to evaluate design trade-offs and calculate how many participants are required for particular effect sizes. We discuss Touchstone2's benefits and limitations, as well as directions for future research.

Supplementary Material

ZIP File (paper217pvc.zip)

Preview video captions

Download
1.34 KB

MP4 File (paper217.mp4)

Supplemental video

Download
164.55 MB

MP4 File (paper217p.mp4)

Preview video

Download
1.36 MB

References

[1]

Monya Baker. 2016. 1500 scientists lift the lid on reproducibility. Nature 533, 11 (2016), 452--454.

[2]

G. E. P. Box and K. B. Wilson. 1992. On the Experimental Attainment of Optimum Conditions. Springer New York, New York, NY, 270--310.

[3]

Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77--101.

[4]

Stephane Champely, Claus Ekstrom, Peter Dalgaard, Jeffrey Gill, Stephan Weibelzahl, Aditya Anandkumar, Clay Ford, Robert Volcic, Helios De Rosario, and Maintainer Helios De Rosario. 2018. Package 'pwr'. https://CRAN.R-project.org/package=pwr R package version 1.2.2.

[5]

Andy Cockburn, Carl Gutwin, and Alan Dix. 2018. HARK No More: On the Preregistration of CHI Experiments. In Proc. Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 141, 12 pages.

Digital Library

[6]

Jacob Cohen. 1988. Statistical power analysis for the behavioral sciences. 2nd. Hillsdale, NJ: erlbaum.

[7]

David Roxbee Cox and Nancy Reid. 2000. The theory of the design of experiments. CRC Press.

[8]

Pierre Dragicevic. 2016. Fair statistical communication in HCI. In Modern Statistical Methods for HCI. Springer, 291--330.

[9]

Franz Faul, Edgar Erdfelder, Albert-Georg Lang, and Axel Buchner. 2007. G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods 39, 2 (01 May 2007), 175--191.

[10]

Ronald Aylmer Fisher. 1937. The design of experiments. Oliver And Boyd; Edinburgh; London.

[11]

Tovi Grossman and Ravin Balakrishnan. 2005. The Bubble Cursor: Enhancing Target Acquisition by Dynamic Resizing of the Cursor's Activation Area. In Proc. Human Factors in Computing Systems (CHI '05). ACM, New York, NY, USA, 281--290.

Digital Library

[12]

Transparent Statistics in Human--Computer Interaction working group. 2018. Transparent Statistics Guidelines. Technical Report. Available at https://transparentstats.github.io/guidelines.

[13]

Matthew A Jaro. 1989. Advances in record-linkage methodology as applied to matching the 1985 census of Tampa, Florida. J. Amer. Statist. Assoc. 84, 406 (1989), 414--420.

[14]

Daniel Kahneman, Jack L. Knetsch, and Richard H. Thaler. 1991. Anomalies: The Endowment Effect, Loss Aversion, and Status Quo Bias. Journal of Economic Perspectives 5, 1 (March 1991), 193--206.

[15]

Matthew Kay, Gregory L. Nelson, and Eric B. Hekler. 2016. Researcher-Centered Design of Statistics: Why Bayesian Statistics Better Fit the Culture and Incentives of HCI. In Proc. Human Factors in Computing Systems (CHI '16). ACM, New York, USA, 4521--4532.

Digital Library

[16]

Ross D. King and others. 2009. The Automation of Science. Science 324, 5923 (2009), 85--89.

[17]

Mark W Lipsey. 1990. Design sensitivity: Statistical power for experimental research. Vol. 19. Sage.

[18]

Wendy E Mackay. 2002. Using video to support interaction design. DVD Tutorial, CHI 2, 5 (2002).

[19]

Wendy E. Mackay, Caroline Appert, Michel Beaudouin-Lafon, Olivier Chapuis, Yangzhou Du, Jean-Daniel Fekete, and Yves Guiard. 2007. Touchstone: Exploratory Design of Experiments. In Proc. Human Factors in Computing Systems (CHI '07). ACM, New York, NY, USA, 1425--1434.

Digital Library

[20]

Xiaojun Meng, Pin Sym Foong, Simon Perrault, and Shengdong Zhao. 2017. NexP: A Beginner Friendly Toolkit for Designing and Conducting Controlled Experiments. Springer International Publishing, Cham, 132--141.

[21]

Tyler Morgan-Wall and George Khoury. 2018. skpr: Design of Experiments Suite: Generate and Evaluate Optimal Designs. https://CRAN.R-project.org/package=skpr R package version 0.54.3.

[22]

Kevin R Murphy, Brett Myors, and Allen Wolach. 2014. Statistical power analysis: A simple and general model for traditional and modern hypothesis tests. Routledge.

[23]

Vijayan N. Nair, Bovas Abraham, Jock MacKay, John A. Nelder, George Box, Madhav S. Phadke, Raghu N. Kacker, Jerome Sacks, William J. Welch, Thomas J. Lorenzen, Anne C. Shoemaker, Kwok L. Tsui, James M. Lucas, Shin Taguchi, Raymond H. Myers, G. Geoffrey Vining, and C. F. Jeff Wu. 1992. Taguchi's Parameter Design: A Panel Discussion. Technometrics 34, 2 (1992), 127--161. http://www.jstor.org/stable/1269231

Digital Library

[24]

C. Papadopoulos, I. Gutenko, and A. E. Kaufman. 2016. VEEVVIE: Visual Explorer for Empirical Visualization, VR and Interaction Experiments. IEEE Transactions on Visualization and Computer Graphics 22, 1 (2016), 111--120.

Digital Library

[25]

Ramana Rao and Stuart K. Card. 1994. The Table Lens: Merging Graphical and Symbolic Representations in an Interactive Focus

[26]

Context Visualization for Tabular Information. In Proc. Human Factors in Computing Systems (CHI '94). ACM, New York, NY, USA, 318--322.

[27]

Martin Oliver Sailer. 2013. crossdes: Construction of Crossover Designs. https://CRAN.R-project.org/package=crossdes R package version 1.1.

[28]

SAS Institute Inc. 2016. JMP ® 13 Design of experiments guide. SAS Institute Inc., SAS Institute Inc., Cary, NC, USA.

[29]

Joseph P Simmons, Leif D Nelson, and Uri Simonsohn. 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science 22, 11 (2011), 1359--1366.

[30]

Larisa N Soldatova and Ross D King. 2006. An ontology of scientific experiments. Journal of The Royal Society Interface 3, 11 (2006), 795--803.

[31]

Lisa Tweedie, Robert Spence, Huw Dawkes, and Hus Su. 1996. Externalising Abstract Mathematical Models. In Proc. Human Factors in Computing Systems (CHI '96). ACM, New York, NY, USA, 406--ff.

Digital Library

[32]

Jelte Wicherts, Coosje Veldkamp, Hilde Augusteijn, Marjan Bakker, Robbie Van Aert, and Marcel Van Assen. 2016. Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology 7 (2016), 1832.

[33]

Daniel Wollschl"ager. 2017. Grundlagen der Datenanalyse mit R. Springer Berlin Heidelberg.

[34]

Koji Yatani. 2016. Effect Sizes and Power Analysis in HCI. Springer International Publishing, Cham, 87--110.

Cited By

Odunayo Olabinjo O(2024)RESPONSE SURFACE TECHNIQUES AS AN INEVITABLE TOOL IN OPTIMIZATION PROCESSResponse Surface Methods - Theory, Applications and Optimization Techniques10.5772/intechopen.1004575Online publication date: 10-Jul-2024
https://doi.org/10.5772/intechopen.1004575
Richer GPister AAbdelaal MFekete JSedlmair MWeiskopf D(2024)Scalability in VisualizationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.323123030:7(3314-3330)Online publication date: Jul-2024
https://doi.org/10.1109/TVCG.2022.3231230
Yamanaka S(2024)Relative Merits of Nominal and Effective Indexes of Difficulty of Fitts’ Law: Effects of Sample Size and the Number of Repetitions on Model FitInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2303201(1-18)Online publication date: 14-Jan-2024
https://doi.org/10.1080/10447318.2024.2303201
Show More Cited By

Index Terms

Touchstone2: An Interactive Environment for Exploring Trade-offs in HCI Experiment Design
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. Laboratory experiments

Recommendations

Supporting the Design and Analysis of HCI Experiments
CHI EA '20: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems

Researchers in HCI commonly use controlled experiments to evaluate artifacts and interaction techniques. However, experiment design and statistical analysis are complex tasks that are prone to errors, especially for novice researchers. In part, this is ...
Design and Development of an Experimental Design and Evaluation System
IMCCC '14: Proceedings of the 2014 Fourth International Conference on Instrumentation and Measurement, Computer, Communication and Control

This paper develops an experimental and Evaluation system which accelerates the construction of experimental scheme. At the same time, it helps with the creation of experiment samples and evaluation of the impact of experiment variable attributes. Based ...
Towards Reproducible Language-Based HRI Experiments: Open-Sourcing a Generalized Experiment Project
HRI '24: Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction

We are witnessing increasing calls for reproducibility and replicability in HRI studies to improve reliability and confidence in empirical findings. One solution to facilitate this is using a robot platform that researchers frequently use, making it ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

May 2019

9077 pages

ISBN:9781450359702

DOI:10.1145/3290605

General Chairs:
Stephen Brewster
University of Glasgow, Scotland, UK
,
Geraldine Fitzpatrick
TU Wien, Austria
,
Program Chairs:
Anna Cox
University College London, UK
,
Vassilis Kostakos
University of Melbourne, Australia

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 May 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Best Paper

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

CHI '19

Sponsor:

SIGCHI

CHI '19: CHI Conference on Human Factors in Computing Systems

May 4 - 9, 2019

Glasgow, Scotland Uk

Acceptance Rates

CHI '19 Paper Acceptance Rate 703 of 2,958 submissions, 24%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
1,113
Total Downloads

Downloads (Last 12 months)164
Downloads (Last 6 weeks)21

Reflects downloads up to 01 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Odunayo Olabinjo O(2024)RESPONSE SURFACE TECHNIQUES AS AN INEVITABLE TOOL IN OPTIMIZATION PROCESSResponse Surface Methods - Theory, Applications and Optimization Techniques10.5772/intechopen.1004575Online publication date: 10-Jul-2024
https://doi.org/10.5772/intechopen.1004575
Richer GPister AAbdelaal MFekete JSedlmair MWeiskopf D(2024)Scalability in VisualizationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.323123030:7(3314-3330)Online publication date: Jul-2024
https://doi.org/10.1109/TVCG.2022.3231230
Yamanaka S(2024)Relative Merits of Nominal and Effective Indexes of Difficulty of Fitts’ Law: Effects of Sample Size and the Number of Repetitions on Model FitInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2303201(1-18)Online publication date: 14-Jan-2024
https://doi.org/10.1080/10447318.2024.2303201
Reza MMusabirov ILaundry NLiut MWilliams J(2023)Using A/B Testing as a Pedagogical Tool for Iterative Design in HCI ClassroomsProceedings of the 5th Annual Symposium on HCI Education10.1145/3587399.3587412(43-48)Online publication date: 28-Apr-2023
https://dl.acm.org/doi/10.1145/3587399.3587412
Wacharamanotham CYang FPu XSarma A(2023)Transparent Practices for Quantitative Empirical ResearchExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3574168(1-5)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3574168
Yamanaka S(2022)Test-retest Reliability on Movement Times and Error Rates in Target PointingProceedings of the 2022 ACM Designing Interactive Systems Conference10.1145/3532106.3533450(178-188)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3532106.3533450
Pang YReinecke KJust R(2022)Apéritif: Scaffolding Preregistrations to Automatically Generate Analysis Code and Methods DescriptionsProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517707(1-15)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517707
Schmidt SZimmermann SMason CSteinicke F(2022)Simulating Human Imprecision in Temporal Statements of Intelligent Virtual AgentsProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517625(1-15)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517625
Kim YHofman JGoldstein D(2022)Putting scientific results in perspective: Improving the communication of standardized effect sizesProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502053(1-14)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502053
Tsandilas TDragicevic P(2022)Gesture Elicitation as a Computational Optimization ProblemProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501942(1-13)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3501942
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents