research-article

reVISit: Looking Under the Hood of Interactive Visualization Studies

Authors:

Carolina Nobre,

Hanspeter Pfister,

Alexander LexAuthors Info & Claims

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Article No.: 25, Pages 1 - 13

https://doi.org/10.1145/3411764.3445382

Published: 07 May 2021 Publication History

Abstract

Quantifying user performance with metrics such as time and accuracy does not show the whole picture when researchers evaluate complex, interactive visualization tools. In such systems, performance is often influenced by different analysis strategies that statistical analysis methods cannot account for. To remedy this lack of nuance, we propose a novel analysis methodology for evaluating complex interactive visualizations at scale. We implement our analysis methods in reVISit, which enables analysts to explore participant interaction performance metrics and responses in the context of users’ analysis strategies. Replays of participant sessions can aid in identifying usability problems during pilot studies and make individual analysis processes salient. To demonstrate the applicability of reVISit to visualization studies, we analyze participant data from two published crowdsourced studies. Our findings show that reVISit can be used to reveal and describe novel interaction patterns, to analyze performance differences between different analysis strategies, and to validate or challenge design decisions.

Supplementary Material

VTT File (3411764.3445382_videofigurecaptions.vtt)

Download
6.66 KB

MP4 File (3411764.3445382_videofigure.mp4)

Supplemental video

Download
89.93 MB

References

[1]

Anne Adams, Peter Lunt, and Paul Cairns. 2008. A Qualititative Approach to HCI Research. In Research Methods for Human-Computer Interaction, Paul Cairns and Anna Cox (Eds.). Cambridge University Press, Cambridge, UK, 138–157.

[2]

Christopher Andrews, Alex Endert, and Chris North. 2010. Space to Think: Large High-Resolution Displays for Sensemaking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’10). ACM, 55–64. https://doi.org/10.1145/1753326.1753336

Digital Library

[3]

Tanja Blascheck, Markus John, Kuno Kurzhals, Steffen Koch, and Thomas Ertl. 2016. VA2: A Visual Analytics Approach for Evaluating Visual Analytics Applications. IEEE Transactions on Visualization and Computer Graphics 22, 1(2016), 61–70. https://doi.org/10.1109/TVCG.2015.2467871

Digital Library

[4]

Rita Borgo, Bongshin Lee, Benjamin Bach, Sara Fabrikant, Radu Jianu, Andreas Kerren, Stephen Kobourov, Fintan McGee, Luana Micallef, Tatiana von Landesberger, Katrin Ballweg, Stephan Diehl, Paolo Simonetto, Michelle Zhou, Stephan Diehl, Paolo Simonetto, and Michelle Zhou. 2017. Crowdsourcing for Information Visualization: Promises and Pitfalls. In Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments. Vol. 10264. Springer, Cham, 96–138. https://doi.org/10.1007/978-3-319-66435-4_5

[5]

Bram C.M. Cappers and Jarke J. van Wijk. 2018. Exploring Multivariate Event Sequences Using Rules, Aggregations, and Selections. IEEE Transactions on Visualization and Computer Graphics 24, 1 (Jan. 2018), 532–541. https://doi.org/10.1109/TVCG.2017.2745278

[6]

Sheelagh Carpendale. 2008. Evaluating Information Visualizations. In Information Visualization: Human-Centered Issues and Perspectives, John T. Stasko, Jean-Daniel Fekete, Chris North, and Chris North (Eds.). Springer, 19–45. https://doi.org/10.1007/978-3-540-70956-5_2

Digital Library

[7]

Juliet Corbin and Anselm Strauss. 2014. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. SAGE Publications.

[8]

Zachary T Cutler, Kiran Gadhave, and Alexander Lex. 2020. Trrack: A Library for Provenance Tracking in Web-Based Visualizations. In Proceedings of IEEE VIS Short Papers. IEEE. https://doi.org/10.31219/osf.io/wnctb

[9]

Pierre Dragicevic. 2016. Fair Statistical Communication in HCI. In Modern Statistical Methods for HCI, Judy Robertson and Maurits Kaptein (Eds.). Springer International Publishing, Cham, 291–330. https://doi.org/10.1007/978-3-319-26633-6_13

[10]

Philipp Eichmann, Emanuel Zgraggen, Zheguang Zhao, Carsten Binnig, and Tim Kraska. 2016. Towards a Benchmark for Interactive Data Exploration. IEEE Data Eng. Bull. 39, 4 (2016), 50–61.

[11]

Philippe Fournier-Viger, Jerry Chun-Wei Lin, Antonio Gomariz, Ted Gueniche, Azadeh Soltani, Zhihong Deng, and Hoang Thanh Lam. 2016. The SPMF Open-Source Data Mining Library Version 2. In Machine Learning and Knowledge Discovery in Databases, Bettina Berendt, Björn Bringmann, Élisa Fromont, Gemma Garriga, Pauli Miettinen, Nikolaj Tatti, and Volker Tresp(Eds.). Vol. 9853. Springer International Publishing, Cham, 36–40. https://doi.org/10.1007/978-3-319-46131-1_8

[12]

Kiran Gadhave, Jochen Görtler, Zach Cutler, Carolina Nobre, Oliver Deussen, Miriah Meyer, Jeff Phillips, and Alexander Lex. 2020. Capturing User Intent When Brushing in Scatterplots. Preprint (2020). https://doi.org/10.31219/osf.io/mq2rk

[13]

David Gotz and Michelle X. Zhou. 2009. Characterizing Users’ Visual Analytic Activity for Insight Provenance. Information Visualization 8, 1 (2009), 42–55. https://doi.org/10.1057/ivs.2008.31

Digital Library

[14]

Samuel Gratzl, Alexander Lex, Nils Gehlenborg, Nicola Cosgrove, and Marc Streit. 2016. From Visual Exploration to Storytelling and Back Again. Computer Graphics Forum 35, 3 (2016), 491–500. https://doi.org/10.1111/cgf.12925

[15]

Hua Guo, Stephen R. Gomez, Caroline Ziemkiewicz, and David H. Laidlaw. 2016. A Case Study Using Visualization Interaction Logs and Insight Metrics to Understand How Analysts Arrive at Insights. IEEE Transactions on Visualization and Computer Graphics 22, 1(2016), 51–60. https://doi.org/10.1109/TVCG.2015.2467613

Digital Library

[16]

Yi Han, Gregory D Abowd, and John Stasko. 2016. Flexible Organization, Exploration, and Analysis of Visualization Application Interaction Events Using Visual Analytics. In IEEE VIS 2016 Workshop on Temporal & Sequential Event Analysis. IEEE, 4.

[17]

Weidong Huang, Peter Eades, and Seok-Hee Hong. 2009. Measuring Effectiveness of Graph Visualizations: A Cognitive Load Perspective. Information Visualization 8, 3 (2009), 139–152. https://doi.org/10.1057/ivs.2009.10

Digital Library

[18]

J. Hullman, E. Adar, and P. Shah. 2011. Benefitting InfoVis with Visual Difficulties. IEEE Transactions on Visualization and Computer Graphics 17, 12 (Dec. 2011), 2213–2222. https://doi.org/10.1109/TVCG.2011.175

Digital Library

[19]

Petra Isenberg, Torre Zuk, Christopher Collins, and Sheelagh Carpendale. 2008. Grounded Evaluation of Information Visualizations. In Proceedings of the 2008 Conference on BEyond Time and Errors: Novel evaLuation Methods for Information Visualization. ACM, Florence, Italy, 1–8. https://doi.org/10.1145/1377966.1377974

Digital Library

[20]

Jian Pei, Jiawei Han, B. Mortazavi-Asl, Jianyong Wang, H. Pinto, Qiming Chen, U. Dayal, and Mei-Chun Hsu. 2004. Mining Sequential Patterns by Pattern-Growth: The PrefixSpan Approach. IEEE Transactions on Knowledge and Data Engineering 16, 11 (Nov. 2004), 1424–1440. https://doi.org/10.1109/TKDE.2004.77

Digital Library

[21]

Alex Kale, Matthew Kay, and Jessica Hullman. 2019. Decision-Making Under Uncertainty in Research Synthesis: Designing for the Garden of Forking Paths. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, Glasgow, Scotland Uk, 1–14. https://doi.org/10.1145/3290605.3300432

Digital Library

[22]

Maurits Kaptein and Judy Robertson. 2012. Rethinking Statistical Analysis Methods for CHI. In Proceedings of the 2012 ACM Annual Conference on Human Factors in Computing Systems - CHI ’12. ACM Press, Austin, Texas, USA, 1105. https://doi.org/10.1145/2207676.2208557

Digital Library

[23]

Yea-Seul Kim, Katharina Reinecke, and Jessica Hullman. 2017. Explaining the Gap: Visualizing One’s Predictions Improves Recall and Comprehension of Data. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems(CHI ’17). Association for Computing Machinery, New York, NY, USA, 1375–1386. https://doi.org/10.1145/3025453.3025592

Digital Library

[24]

Brittany Kondo and Christopher Collins. 2014. DimpVis: Exploring Time-Varying Information Visualizations by Direct Manipulation. IEEE Transactions on Visualization and Computer Graphics (InfoVis ’14) 20, 12(2014), 2003–2012. https://doi.org/10.1109/TVCG.2014.2346250

[25]

Heidi Lam, Enrico Bertini, Petra Isenberg, Catherine Plaisant, and Sheelagh Carpendale. 2012. Empirical Studies in Information Visualization: Seven Scenarios. IEEE Transactions on Visualization and Computer Graphics 18, 9(2012), 1520–1536. https://doi.org/10.1109/TVCG.2011.279

Digital Library

[26]

Miro Mannino and Azza Abouzied. 2018. Qetch: Time Series Querying with Expressive Sketches. In Proceedings of the 2018 International Conference on Management of Data(SIGMOD ’18). Association for Computing Machinery, New York, NY, USA, 1741–1744. https://doi.org/10.1145/3183713.3193547

Digital Library

[27]

A McAlister, D Lee, KM Ehlert, RL Kajfez, CJ Faber, and MS Kennedy. 2017. Qualitative Coding: An Approach to Assess Inter-Rater Reliability. In ASEE Annual Conference & Exposition. ASEE.

[28]

Carolina Nobre, Miriah Meyer, Marc Streit, and Alexander Lex. 2019. The State of the Art in Visualizing Multivariate Networks. Computer Graphics Forum (EuroVis) 38, 3 (2019), 807–832. https://doi.org/10.1111/cgf.13728

[29]

Carolina Nobre, Dylan Wootton, Lane Harrison, and Alexander Lex. 2020. Evaluating Multivariate Network Visualization Techniques Using a Validated Design and Crowdsourcing Approach. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, 1–12. https://doi.org/10.1145/3313831.3376381

Digital Library

[30]

L. M. K. Padilla, S. C. Castro, P. S. Quinan, I. T. Ruginski, and S. H. Creem-Regehr. 2020. Toward Objective Evaluation of Working Memory in Visualizations: A Case Study Using Pupillometry and a Dual-Task Paradigm. IEEE Transactions on Visualization and Computer Graphics 26, 1 (Jan. 2020), 332–342. https://doi.org/10.1109/TVCG.2019.2934286

[31]

D. Park, S. M. Drucker, R. Fernandez, and N. Elmqvist. 2018. Atom: A Grammar for Unit Visualizations. IEEE Transactions on Visualization and Computer Graphics 24, 12(2018), 3032–3043. https://doi.org/10.1109/TVCG.2017.2785807

[32]

E.D. Ragan, A. Endert, J. Sanyal, and J. Chen. 2016. Characterizing Provenance in Visualization and Data Analysis: An Organizational Framework of Provenance Types and Purposes. IEEE Transactions on Visualization and Computer Graphics (VAST ’15) 22, 1(2016), 31–40. https://doi.org/10.1109/TVCG.2015.2467551

Digital Library

[33]

Anselm Strauss and Juliet Corbin. 1994. Grounded Theory Methodology. Handbook of qualitative research 17 (1994), 273–85.

[34]

K. Wongsuphasawat and D. Gotz. 2012. Exploring Flow, Factors, and Outcomes of Temporal Event Sequences with the Outflow Visualization. IEEE Transactions on Visualization and Computer Graphics 18, 12 (Dec. 2012), 2659–2668. https://doi.org/10.1109/TVCG.2012.225

Digital Library

[35]

Kai Xu, Alvitta Ottley, Conny Walchshofer, Marc Streit, Remco Chang, and John Wenskovitch. 2020. Survey on the Analysis of User Interactions and Visualization Provenance. Computer Graphics Forum 39, 3 (2020), 757–783. https://doi.org/10.1111/cgf.14035

[36]

Ji Soo Yi, Youn ah Kang, John Stasko, and J.A. Jacko. 2007. Toward a Deeper Understanding of the Role of Interaction in Information Visualization. IEEE Transactions on Visualization and Computer Graphics (InfoVis ’07) 13, 6(2007), 1224–1231. https://doi.org/10.1109/TVCG.2007.70515

Digital Library

Cited By

Han YAbowd GStasko J(2025)IntiVisor: A Visual Analytics System for Interaction Log AnalysisIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337063731:3(1772-1784)Online publication date: Mar-2025
https://doi.org/10.1109/TVCG.2024.3370637
Rabsahl SBeck F(2024)A Multi-layer Event Visualization for Exploring User Search Patterns in Literature Discovery with PUREsuggestProceedings of Mensch und Computer 202410.1145/3670653.3677502(408-412)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1145/3670653.3677502
Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Show More Cited By

Index Terms

reVISit: Looking Under the Hood of Interactive Visualization Studies
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Index terms have been assigned to the content through auto-classification.

Recommendations

A Model and Framework for Visualization Exploration

Visualization exploration is the process of extracting insight from data via interaction with visual depictions of that data. Visualization exploration is more than presentation; the interaction with both the data and its depiction is as important as ...
Polyarchy visualization: visualizing multiple intersecting hierarchies
CHI '02: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

We describe a new information structure composed of multiple intersecting hierarchies, which we call Polyarchies. Visualizing polyarchies enables use of novel views for discovery of relationships which are very difficult using existing hierarchy ...
Animated visualization of multiple intersecting hierarchies

The authors describe a new information structure composed of multiple intersecting hierarchies, which we call a Polyarchy. Visualizing polyarchies enables use of novel views for discovery of relationships which are very difficult using existing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

May 2021

10862 pages

ISBN:9781450380966

DOI:10.1145/3411764

General Chairs:
Yoshifumi Kitamura
Tohoku University, Japan
,
Aaron Quigley
University of New South Wales, Australia
,
Program Chairs:
Katherine Isbister
University of California Santa Cruz, USA
,
Takeo Igarashi
The University of Tokyo, Japan
,
Publications Chairs:
Pernille Bjørn
University of Copenhagen, Denmark
,
Steven Drucker
Microsoft Research, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '21

Sponsor:

SIGCHI

CHI '21: CHI Conference on Human Factors in Computing Systems

May 8 - 13, 2021

Yokohama, Japan

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
605
Total Downloads

Downloads (Last 12 months)76
Downloads (Last 6 weeks)4

Reflects downloads up to 02 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Han YAbowd GStasko J(2025)IntiVisor: A Visual Analytics System for Interaction Log AnalysisIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337063731:3(1772-1784)Online publication date: Mar-2025
https://doi.org/10.1109/TVCG.2024.3370637
Rabsahl SBeck F(2024)A Multi-layer Event Visualization for Exploring User Search Patterns in Literature Discovery with PUREsuggestProceedings of Mensch und Computer 202410.1145/3670653.3677502(408-412)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1145/3670653.3677502
Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Musleh MCeneda DEhlers HRaidou R(2024)ConAn: Measuring and Evaluating User Confidence in Visual Data Analysis Under UncertaintyComputer Graphics Forum10.1111/cgf.15272Online publication date: 29-Nov-2024
https://doi.org/10.1111/cgf.15272
Wootton DFox APeck ESatyanarayan A(2024)Charting EDA: Characterizing Interactive Visualization Use in Computational Notebooks with a Mixed-Methods FormalismIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345621731:1(1191-1201)Online publication date: 10-Oct-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456217
Narechania AOdak KEl-Assady MEndert A(2024)ProvenanceWidgets: A Library of UI Control Elements to Track and Dynamically Overlay Analytic ProvenanceIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345614431:1(1235-1245)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456144
Liebers CAgarwal SKrug MPitsch KBeck F(2023)VisCoMET: Visually Analyzing Team Collaboration in Medical Emergency TrainingsComputer Graphics Forum10.1111/cgf.1481942:3(149-160)Online publication date: 27-Jun-2023
https://doi.org/10.1111/cgf.14819
Ding YWilburn JShrestha HNdlovu AGadhave KNobre CLex AHarrison L(2023)reVISit: Supporting Scalable Evaluation of Interactive Visualizations2023 IEEE Visualization and Visual Analytics (VIS)10.1109/VIS54172.2023.00015(31-35)Online publication date: 21-Oct-2023
https://doi.org/10.1109/VIS54172.2023.00015
Weckbecker MNeziri SHattab G(2023)Supporting Mathematical Education with Interactive Visual Proofs2023 IEEE Workshop on Visualization for Social Good (VIS4Good)10.1109/VIS4Good60218.2023.00011(21-25)Online publication date: 22-Oct-2023
https://doi.org/10.1109/VIS4Good60218.2023.00011
Walchshofer CHinterreiter AXu KStitz HStreit M(2023)Provectories: Embedding-Based Analysis of Interaction Provenance DataIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.313569729:12(4816-4831)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1109/TVCG.2021.3135697
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten