research-article

Embodied Provenance for Immersive Sensemaking

Authors:

Kadek Ananta Satriadi,

Sarah GoodwinAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 7, Issue ISS

Article No.: 435, Pages 198 - 216

https://doi.org/10.1145/3626471

Published: 01 November 2023 Publication History

Abstract

Immersive analytics research has explored how embodied data representations and interactions can be used to engage users in sensemaking. Prior research has broadly overlooked the potential of immersive space for supporting analytic provenance, the understanding of sensemaking processes through users’ interaction histories. We propose the concept of embodied provenance, the use of three-dimensional space and embodied interactions in supporting recalling, reproducing, annotating and sharing analysis history in immersive environments. We design a conceptual framework for embodied provenance by highlighting a set of design criteria for analytic provenance drawn from prior work and identifying essential properties for embodied provenance. We develop a prototype system in virtual reality to demonstrate the concept and support the conceptual framework by providing multiple data views and embodied interaction metaphors in a large virtual space. We present a use case scenario of energy consumption analysis and evaluated the system through a qualitative evaluation with 17 participants, which show the system’s potential for assisting analytic provenance using embodiment. Our exploration of embodied provenance through this prototype provides lessons learnt to guide the design of immersive analytic tools for embodied provenance.

Supplementary Material

Video (iss23main-p7698-p-video.mp4)

This is a presentation video of my talk at ISS 2023 on our accepted paper: Embodied Provenance for Immersive Sensemaking.

Download
109.85 MB

References

[1]

Christopher Andrews, Alex Endert, and Chris North. 2010. Space to Think: Large High-Resolution Displays for Sensemaking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’10). Association for Computing Machinery, New York, NY, USA. 55–64. isbn:9781605589299 https://doi.org/10.1145/1753326.1753336

Digital Library

[2]

Christopher Andrews, Alex Endert, and Chris North. 2010. Space to Think: Large High-Resolution Displays for Sensemaking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 55–64.

Digital Library

[3]

Lawrence W Barsalou. 1983. Ad hoc categories. Memory & Cognition, 11, 3 (1983), 211–227.

[4]

Andrea Batch, Andrew Cunningham, Maxime Cordeil, Niklas Elmqvist, Tim Dwyer, Bruce H Thomas, and Kim Marriott. 2019. There Is No Spoon: Evaluating Performance, Space Use, and Presence with Expert Domain Users in Immersive Analytics. IEEE Transactions on Visualization and Computer Graphics, 26, 1 (2019), 536–546.

[5]

Michel Beaudouin-Lafon. 2000. Instrumental Interaction: An Interaction Model for Designing Post-WIMP User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’00). Association for Computing Machinery, New York, NY, USA. 446–453. isbn:1581132166 https://doi.org/10.1145/332040.332473

Digital Library

[6]

K. Brodlie, A. Poon, H. Wright, L. Brankin, G. Banecki, and A. Gay. 1993. GRASPARC-A problem solving environment integrating computation and visualization. In Proceedings Visualization ’93. 102–109. https://doi.org/10.1109/VISUAL.1993.398857

[7]

T. Chandler, M. Cordeil, T. Czauderna, T. Dwyer, J. Glowacki, C. Goncu, M. Klapperstueck, K. Klein, K. Marriott, F. Schreiber, and E. Wilson. 2015. Immersive Analytics. In 2015 Big Data Visual Analytics (BDVA). IEEE Computer Society, Los Alamitos, CA, USA. 1–8. https://doi.org/10.1109/BDVA.2015.7314296

[8]

Andy Clark and David Chalmers. 1998. The Extended Mind. Analysis, 58, 1 (1998), 7–19.

[9]

Andy Cockburn and Bruce McKenzie. 2002. Evaluating the Effectiveness of Spatial Memory in 2D and 3D Physical and Virtual Environments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 203–210.

Digital Library

[10]

Maxime Cordeil, Andrew Cunningham, Tim Dwyer, Bruce H. Thomas, and Kim Marriott. 2017. ImAxes: Immersive Axes as Embodied Affordances for Interactive Multivariate Data Visualisation. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST ’17). Association for Computing Machinery, New York, NY, USA. 71–83. isbn:9781450349819 https://doi.org/10.1145/3126594.3126613

Digital Library

[11]

Zach Cutler, Kiran Gadhave, and Alexander Lex. 2020. Trrack: A Library for Provenance-Tracking in Web-Based Visualizations. In 2020 IEEE Visualization Conference (VIS). 116–120. https://doi.org/10.1109/VIS47514.2020.00030

[12]

Paul Dourish. 2001. Where the action is: the foundations of embodied interaction. MIT press.

[13]

Cody Dunne, Nathalie Henry Riche, Bongshin Lee, Ronald Metoyer, and George Robertson. 2012. GraphTrail: Analyzing Large Multivariate, Heterogeneous Networks While Supporting Exploration History. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12). Association for Computing Machinery, New York, NY, USA. 1663–1672. isbn:9781450310154 https://doi.org/10.1145/2207676.2208293

Digital Library

[14]

Niklas Elmqvist, Andrew Vande Moere, Hans-Christian Jetter, Daniel Cernea, Harald Reiterer, and TJ Jankun-Kelly. 2011. Fluid interaction for information visualization. Information Visualization, 10, 4 (2011), 327–340.

Digital Library

[15]

Caroline J Falconer, Mel Slater, Aitor Rovira, John A King, Paul Gilbert, Angus Antley, and Chris R Brewin. 2014. Embodying Compassion: A Virtual Reality Paradigm for Overcoming Excessive Self-Criticism. PloS one, 9, 11 (2014), e111933.

[16]

Jean-Daniel Fekete, T. J. Jankun-Kelly, Melanie Tory, and Kai Xu. 2019. Provenance and Logging for Sense Making (Dagstuhl Seminar 18462). Dagstuhl Reports, 8, 11 (2019), 35–62. issn:2192-5283 https://doi.org/10.4230/DagRep.8.11.35

[17]

David Gotz and Michelle X. Zhou. 2008. Characterizing users’ visual analytic activity for insight provenance. In 2008 IEEE Symposium on Visual Analytics Science and Technology. 123–130. https://doi.org/10.1109/VAST.2008.4677365

[18]

Samuel Gratzl, Alexander Lex, Nils Gehlenborg, Nicola Cosgrove, and Marc Streit. 2016. From visual exploration to storytelling and back again. In Computer Graphics Forum. 35, 491–500.

Digital Library

[19]

Dennis P Groth and Kristy Streefkerk. 2006. Provenance and annotation for visual exploration systems. IEEE transactions on visualization and computer graphics, 12, 6 (2006), 1500–1510.

Digital Library

[20]

Devamardeep Hayatpur, Haijun Xia, and Daniel Wigdor. 2020. DataHop: Spatial Data Exploration in Virtual Reality. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST ’20). Association for Computing Machinery, New York, NY, USA. 818–828. isbn:9781450375146 https://doi.org/10.1145/3379337.3415878

Digital Library

[21]

Jeffrey Heer, Jock Mackinlay, Chris Stolte, and Maneesh Agrawala. 2008. Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation. IEEE Transactions on Visualization and Computer Graphics, 14, 6 (2008), 1189–1196. https://doi.org/10.1109/TVCG.2008.137

Digital Library

[22]

Yi-Jheng Huang, Takanori Fujiwara, Yun-Xuan Lin, Wen-Chieh Lin, and Kwan-Liu Ma. 2017. A Gesture System for Graph Visualization in Virtual Reality Environments. In 2017 IEEE Pacific Visualization Symposium (PacificVis). 41–45.

[23]

Robert JK Jacob, Audrey Girouard, Leanne M Hirshfield, Michael S Horn, Orit Shaer, Erin Treacy Solovey, and Jamie Zigelbaum. 2008. Reality-based interaction: a framework for post-WIMP interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems. 201–210.

Digital Library

[24]

Waqas Javed and Niklas Elmqvist. 2013. ExPlates: spatializing interactive analysis to scaffold visual exploration. In Computer Graphics Forum. 32, 441–450.

Digital Library

[25]

David Kirsh. 1995. The intelligent use of space. Artificial Intelligence, 73, 1 (1995), 31–68. issn:0004-3702 https://doi.org/10.1016/0004-3702(94)00017-U Computational Research on Interaction and Agency, Part 2

Digital Library

[26]

Serena Lee-Cultura and Michail Giannakos. 2020. Embodied Interaction and Spatial Skills: A Systematic Review of Empirical Studies. Interacting with Computers, 32, 4 (2020), 331–366.

[27]

Ang Li, Jiazhou Liu, Maxime Cordeil, Jack Topliss, Thammathip Piumsomboon, and Barrett Ens. 2023. GestureExplorer: Immersive Visualisation and Exploration of Gesture Data. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA. Article 380, 16 pages. isbn:9781450394215 https://doi.org/10.1145/3544548.3580678

Digital Library

[28]

Lee Lisle, Kylie Davidson, Edward J.K. Gitre, Chris North, and Doug A. Bowman. 2021. Sensemaking Strategies with Immersive Space to Think. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). 529–537. https://doi.org/10.1109/VR50410.2021.00077

[29]

Jiazhou Liu, Arnaud Prouzeau, Barrett Ens, and Tim Dwyer. 2020. Design and evaluation of interactive small multiples data visualisation in immersive spaces. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 588–597.

[30]

Weizhou Luo, Anke Lehmann, Yushan Yang, and Raimund Dachselt. 2021. Investigating Document Layout and Placement Strategies for Collaborative Sensemaking in Augmented Reality. CHI EA ’21. Association for Computing Machinery, New York, NY, USA. Article 456, 7 pages. isbn:9781450380959 https://doi.org/10.1145/3411763.3451588

Digital Library

[31]

Kevin Lynch. 1964. The image of the city. MIT press.

[32]

Kim Marriott, Falk Schreiber, Tim Dwyer, Karsten Klein, Nathalie Henry Riche, Takayuki Itoh, Wolfgang Stuerzlinger, and Bruce H Thomas. 2018. Immersive analytics. 11190, Springer.

[33]

Andreas Mathisen, Tom Horak, Clemens Nylandsted Klokmose, Kaj Grønbæ k, and Niklas Elmqvist. 2019. InsideInsights: Integrating Data-Driven Reporting in Collaborative Visual Analytics. In Computer Graphics Forum. 38, 649–661.

[34]

Rhys Newbury, Kadek Ananta Satriadi, Jesse Bolton, Jiazhou Liu, Maxime Cordeil, Arnaud Prouzeau, and Bernhard Jenny. 2021. Embodied gesture interaction for immersive maps. Cartography and Geographic Information Science, 1–15.

[35]

Phong H. Nguyen, Kai Xu, Andy Bardill, Betul Salman, Kate Herd, and B.L. William Wong. 2016. SenseMap: Supporting browser-based online sensemaking through analytic provenance. In 2016 IEEE Conference on Visual Analytics Science and Technology (VAST). 91–100. https://doi.org/10.1109/VAST.2016.7883515

[36]

Annie Murphy Paul. 2021. The Extended Mind: The Power of Thinking Outside the Brain. Eamon Dolan Books.

[37]

Eric D. Ragan, Alex Endert, Jibonananda Sanyal, and Jian Chen. 2016. Characterizing Provenance in Visualization and Data Analysis: An Organizational Framework of Provenance Types and Purposes. IEEE Transactions on Visualization and Computer Graphics, 22, 1 (2016), 31–40. https://doi.org/10.1109/TVCG.2015.2467551

Digital Library

[38]

George Robertson, Mary Czerwinski, Kevin Larson, Daniel C Robbins, David Thiel, and Maarten Van Dantzich. 1998. Data mountain: Using Spatial Memory for Document Management. In Proceedings of the 11th Annual ACM Symposium on User Interface Software and Technology. 153–162.

Digital Library

[39]

David Rose. 2021. Supersight: What Augmented Reality Means for Our Lives, Our Work, and the Way We Imagine the Future. BenBella Books, Incorporated.

[40]

Ali Sarvghad and Melanie Tory. 2015. Exploiting analysis history to support collaborative data analysis. In Graphics Interface. 123–130.

[41]

D.L. Schacter, D.T. Gilbert, and D.M. Wegner. 2009. Semantic and episodic memory. In Psychology. Worth Publishers. isbn:9780716752158

[42]

Yedendra Babu Shrinivasan and Jarke J. van Wijk. 2008. Supporting the Analytical Reasoning Process in Information Visualization. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’08). Association for Computing Machinery, New York, NY, USA. 1237–1246. isbn:9781605580111 https://doi.org/10.1145/1357054.1357247

Digital Library

[43]

Claudio T. Silva, Juliana Freire, and Steven P. Callahan. 2007. Provenance for Visualizations: Reproducibility and Beyond. Computing in Science & Engineering, 9, 5 (2007), 82–89. https://doi.org/10.1109/MCSE.2007.106

Digital Library

[44]

Alice Toniolo, Timothy J. Norman, Anthony Etuk, Federico Cerutti, Robin Wentao Ouyang, Mani Srivastava, Nir Oren, Timothy Dropps, John A. Allen, and Paul Sullivan. 2015. Supporting Reasoning with Different Types of Evidence in Intelligence Analysis. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems (AAMAS ’15). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC. 781–789. isbn:9781450334136

Digital Library

[45]

Jorge A Wagner Filho, Wolfgang Stuerzlinger, and Luciana Nedel. 2019. Evaluating an Immersive Space-Time Cube Geovisualization for Intuitive Trajectory Data Exploration. IEEE transactions on visualization and computer graphics, 26, 1 (2019), 514–524.

[46]

Rick Walker, Aiden Slingsby, Jason Dykes, Kai Xu, Jo Wood, Phong H. Nguyen, Derek Stephens, B.L. William Wong, and Yongjun Zheng. 2013. An Extensible Framework for Provenance in Human Terrain Visual Analytics. IEEE Transactions on Visualization and Computer Graphics, 19, 12 (2013), 2139–2148. https://doi.org/10.1109/TVCG.2013.132

Digital Library

[47]

Alan Wexelblat and Pattie Maes. 1999. Footprints: History-Rich Tools for Information Foraging. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’99). Association for Computing Machinery, New York, NY, USA. 270–277. isbn:0201485591 https://doi.org/10.1145/302979.303060

Digital Library

[48]

Kai Xu, Simon Attfield, T.J. Jankun-Kelly, Ashley Wheat, Phong H. Nguyen, and Nallini Selvaraj. 2015. Analytic Provenance for Sensemaking: A Research Agenda. IEEE Computer Graphics and Applications, 35, 3 (2015), 56–64. https://doi.org/10.1109/MCG.2015.50

Digital Library

[49]

Yalong Yang, Tim Dwyer, Kim Marriott, Bernhard Jenny, and Sarah Goodwin. 2020. Tilt Map: Interactive Transitions Between Choropleth Map, Prism Map and Bar Chart in Immersive Environments. IEEE Transactions on Visualization and Computer Graphics, 27, 12 (2020), 4507–4519.

Digital Library

[50]

Yidan Zhang, Barrett Ens, Kadek Ananta Satriadi, Arnaud Prouzeau, and Sarah Goodwin. 2022. TimeTables: Embodied Exploration of Immersive Spatio-Temporal Data. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 599–605. https://doi.org/10.1109/VR51125.2022.00080

[51]

Yidan Zhang, Barrett Ens, Kadek Ananta Satriadi, Ying Yang, and Sarah Goodwin. 2023. Defining Embodied Provenance for Immersive Sensemaking. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. 1–7.

Cited By

Zhou XLee BOrtega FBatmaz AYang Y(2024)Lights, Headset, Tablet, Action: Exploring the Use of Hybrid User Interfaces for Immersive Situated AnalyticsProceedings of the ACM on Human-Computer Interaction10.1145/36981478:ISS(517-539)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698147
Friedl‐Knirsch JPointecker FPfistermüller SStach CAnthes CRoth D(2024)A Systematic Literature Review of User Evaluation in Immersive AnalyticsComputer Graphics Forum10.1111/cgf.1511143:3Online publication date: 10-Jun-2024
https://doi.org/10.1111/cgf.15111

Index Terms

Embodied Provenance for Immersive Sensemaking
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Virtual reality
    2. Interaction techniques
  2. Visualization

Recommendations

Defining Embodied Provenance for Immersive Sensemaking
CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems

Immersive analytics research has explored how embodied data representations and interactions can be used to engage users in sensemaking. Prior research has broadly overlooked the potential of immersive space for supporting analytic provenance, the ...
Immersive Analytics 2.0: Spatial and Embodied Sensemaking
CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems

Immersive Analytics is now a fully emerged research topic that spans several research communities, including Human-Computer Interaction, Data Visualisation, Virtual Reality and Augmented Reality. Immersive Analytics research has identified and ...
I Did Not Notice: A Comparison of Immersive Analytics with Augmented and Virtual Reality
CHI EA '24: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems

Immersive environments enable users to engage in embodied interaction, enhancing the sensemaking processes involved in completing tasks such as immersive analytics. Previous comparative studies on immersive analytics using augmented and virtual ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 7, Issue ISS

December 2023

482 pages

EISSN:2573-0142

DOI:10.1145/3554314

Editor:
Jeffrey Nichols
Apple, USA

Issue’s Table of Contents

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2023

Published in PACMHCI Volume 7, Issue ISS

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
184
Total Downloads

Downloads (Last 12 months)149
Downloads (Last 6 weeks)6

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhou XLee BOrtega FBatmaz AYang Y(2024)Lights, Headset, Tablet, Action: Exploring the Use of Hybrid User Interfaces for Immersive Situated AnalyticsProceedings of the ACM on Human-Computer Interaction10.1145/36981478:ISS(517-539)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698147
Friedl‐Knirsch JPointecker FPfistermüller SStach CAnthes CRoth D(2024)A Systematic Literature Review of User Evaluation in Immersive AnalyticsComputer Graphics Forum10.1111/cgf.1511143:3Online publication date: 10-Jun-2024
https://doi.org/10.1111/cgf.15111

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents