Cited By
View all- Friedl‐Knirsch JPointecker FPfistermüller SStach CAnthes CRoth D(2024)A Systematic Literature Review of User Evaluation in Immersive AnalyticsComputer Graphics Forum10.1111/cgf.1511143:3Online publication date: 10-Jun-2024
"Is It My Turn?": Assessing Teamwork and Taskwork in Collaborative Immersive Analytics
Article No.: 479, Pages 1 - 23
Abstract
Immersive analytics has the potential to promote collaboration in machine learning (ML). This is desired due to the specific characteristics of ML modeling in practice, namely the complexity of ML, the interdisciplinary approach in industry, and the need for ML interpretability. In this work, we introduce an augmented reality-based system for collaborative immersive analytics that is designed to support ML modeling in interdisciplinary teams. We conduct a user study to examine how collaboration unfolds when users with different professional backgrounds and levels of ML knowledge interact in solving different ML tasks. Specifically, we use the pair analytics methodology and performance assessments to assess collaboration and explore their interactions with each other and the system. Based on this, we provide qualitative and quantitative results on both teamwork and taskwork during collaboration. Our results show how our system elicits sustained collaboration as measured along six distinct dimensions. We finally make recommendations how immersive systems should be designed to elicit sustained collaboration in ML modeling.
Supplementary Material
References
[1]
Richard Arias-Hernández, Linda T. Kaastra, Tera M. Green, and Brian D. Fisher. 2011. Pair analytics: Capturing reasoning processes in collaborative visual analytics. 44th Hawaii International Conference on System Sciences (2011), 1--10.
[2]
Simon Attfield, Gabriella Kazai, Mounia Lalmas, and Benjamin Piwowarski. 2011. Towards a science of user engagement (Position Paper). In WSDM Workshop on User Modelling for Web Applications (Hong Kong, China). 9--12.
[3]
Benjamin Bach, Maxime Cordeil, Ulrich Engelke, Barrett Ens, Marcos Serrano, and Wesley Willett. [n.,d.]. Interaction design & prototyping for immersive analytics. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk). 1--8.
[4]
Adrian Bangerter and Herbert H. Clark. 2003. Navigating joint projects with dialogue. Cognitive Science, Vol. 27, 2 (2003), 195--225.
[5]
Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. 2020. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, Vol. 58 (2020), 82--115.
[6]
Andrea Batch, Andrew Cunningham, Maxime Cordeil, Niklas Elmqvist, Tim Dwyer, Bruce H. Thomas, and Kim Marriott. 2020. There is no spoon: Evaluating performance, space use, and presence with expert domain users in immersive analytics. IEEE Transactions on Visualization and Computer Graphics, Vol. 26, 1 (2020), 536--546.
[7]
Michaela Benk, Raphael Weibel, and Andrea Ferrario. 2022. Creative uses of AI systems and their explanations: A case study from insurance. ACM CHI 2022 Workshop on Human-Centered Explainable AI (HCXAI) (2022).
[8]
Lonni Besançon, Paul Issartel, Mehdi Ammi, and Tobias Isenberg. 2017. Hybrid tactile/tangible interaction for 3D data exploration. IEEE Transactions on Visualization and Computer Graphics, Vol. 23, 1 (2017), 881--890.
[9]
Przemysław Biecek. 2018. DALEX: Explainers for complex predictive models in R. Journal of Machine Learning Research, Vol. 19 (2018), 1--5.
[10]
Mark Billinghurst, Maxime Cordeil, Anastasia Bezerianos, and Todd Margolis. 2018. Collaborative immersive analytics. In Immersive Analytics. Springer International Publishing, Cham, 221--257.
[11]
Reuben Binns, Max Van Kleek, Michael Veale, Ulrik Lyngs, Jun Zhao, and Nigel Shadbolt. [n.,d.]. It's reducing a human being to a percentage' perceptions of justice in algorithmic decisions. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada). 1--14.
[12]
John Brooke. 1996. SUS: A quick and dirty usability scale. Usability Evaluation in Industry (1996).
[13]
Wolfgang Büschel, Jian Chen, Raimund Dachselt, Steven Drucker, Tim Dwyer, Carsten Görg, Tobias Isenberg, Andreas Kerren, Chris North, and Wolfgang Stuerzlinger. 2018. Interaction for immersive analytics. In Immersive Analytics. Springer, Cham, Switzerland, 95--138.
[14]
Wolfgang Büschel, Anke Lehmann, and Raimund Dachselt. 2021. MIRIA: A mixed reality toolkit for the in-situ visualization and analysis of spatio-temporal interaction data. In CHI Conference on Human Factors in Computing Systems (Yokohama, Japan). 1--15.
[15]
Simon Butscher, Sebastian Hubenschmid, Jens Müller, Johannes Fuchs, and Harald Reiterer. 2018. Clusters, trends, and outliers: How immersive technologies can facilitate the collaborative analysis of multidimensional data. In CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada). 1--12.
[16]
Diogo V. Carvalho, E. M. Pereira, and Jaime S. Cardoso. 2019. Machine learning interpretability: A survey on methods and metrics. Electronics, Vol. 8, 8 (2019), 832.
[17]
Tom Chandler, Maxime Cordeil, Tobias Czauderna, Tim Dwyer, Jaroslaw Glowacki, Cagatay Goncu, Matthias Klapperstueck, Karsten Klein, Kim Marriott, Falk Schreiber, and Elliot Wilson. 2015. Immersive analytics. In Big Data Visual Analytics (BDVA) (Hobart, TAS, Australia). 1--8.
[18]
Elizabeth Charters. 2003. The use of think-aloud methods in qualitative research: An introduction to think-aloud methods. Brock Education Journal, Vol. 12, 2 (2003), 68--82.
[19]
Jan Chong and Tom Hurlbutt. 2007. The social dynamics of pair programming. In Proceedings of the 29th International Conference on Software Engineering. 354--363.
[20]
Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, Vol. 5, 2 (2017), 153--163.
[21]
Alexandra Chouldechova and Aaron Roth. 2020. A snapshot of the frontiers of fairness in machine learning. Commun. ACM, Vol. 63, 5 (2020), 82--89.
[22]
Herbert H. Clark and Susan E. Brennan. 1991. Grounding in Communication. In Perspectives on Socially Shared Cognition. American Psychological Association, Washington, DC, 127--149.
[23]
Kristin A Cook and James J Thomas. 2005. Illuminating the path: The research and development agenda for visual analytics. Technical Report. Pacific Northwest National Lab (PNNL), Richland, WA.
[24]
Maxime Cordeil, T. Dwyer, K. Klein, B. Laha, K. Marriott, and B. Thomas. 2017. Immersive collaborative analysis of network connectivity: CAVE-style or head-mounted display? IEEE Transactions on Visualization and Computer Graphics, Vol. 23 (2017), 441--450.
[25]
Andre Doucette, Carl Gutwin, Regan Lee Mandryk, Miguel A. Nacenta, and Sunny Sharma. 2013. Sometimes when we touch: How arm embodiments change reaching and collaboration on digital tables. In Conference on Computer Supported Cooperative Work (San Antonio, Texas, USA). 193--202.
[26]
Paul Dourish and Victoria Bellotti. 1992. Awareness and coordination in shared workspaces. In Conference on Computer-Supported Cooperative Work (Toronto, Ontario, Canada). 107--114.
[27]
Barrett Ens, Benjamin Bach, Maxime Cordeil, Ulrich Engelke, Marcos Serrano, Wesley Willett, Arnaud Prouzeau, Christoph Anthes, Wolfgang Büschel, Cody Dunne, Tim Dwyer, Jens Grubert, Jason H. Haga, Nurit Kirshenbaum, Dylan Kobayashi, Tica Lin, Monsurat Olaosebikan, Fabian Pointecker, David Saffo, Nazmus Saquib, Dieter Schmalstieg, Danielle Albers Szafir, Matt Whitlock, and Yalong Yang. 2021a. Grand challenges in immersive analytics. In CHI Conference on Human Factors in Computing Systems (Yokohama, Japan). 1--17.
[28]
Barrett Ens, Sarah Goodwin, Arnaud Prouzeau, Fraser Anderson, Florence Y. Wang, Samuel Gratzl, Zac Lucarelli, Brendan Moyle, Jim Smiley, and Tim Dwyer. 2021b. Uplift: A tangible and immersive tabletop system for casual collaborative visual analytics. IEEE Transactions on Visualization and Computer Graphics, Vol. 27 (2021), 1193--1203.
[29]
Barrett Ens, Joel Lanir, Anthony Tang, Scott Bateman, Gun Lee, Thammathip Piumsomboon, and Mark Billinghurst. 2019. Revisiting collaboration through mixed reality: The evolution of groupware. International Journal of Human-Computer Studies, Vol. 131 (2019), 81--98.
[30]
Andrea Ferrario, Raphael Weibel, and Stefan Feuerriegel. 2020. ALEEDSA: Augmented reality for interactive machine learning. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA). 1--8.
[31]
Stefan Feuerriegel, Yash Raj Shrestha, Georg von Krogh, and Ce Zhang. 2022. Bringing artificial intelligence to business management. Nature Machine Intelligence (2022).
[32]
Adrien Fonnet and Yannick Prié. 2021. Survey of Immersive Analytics. IEEE Transactions on Visualization and Computer Graphics, Vol. 27, 3 (2021), 2101--2122.
[33]
Sebastian Gehrmann, Hendrik Strobelt, Robert Krüger, Hanspeter Pfister, and Alexander M. Rush. 2019. Visual interaction with deep learning models through collaborative semantic inference. IEEE Transactions on Visualization and Computer Graphics, Vol. 26, 1 (2019), 884--894.
[34]
Jens Grubert, Eyal Ofek, Michel Pahud, and Per Ola Kristensson. 2020. Back to the future: Revisiting mouse and keyboard interaction for HMD-based immersive analytics. In ACM CHI Workshop on Immersive Analytics: Envisioning Future Productivity for Immersive Analytics (Honolulu, Hawaii, USA).
[35]
Carl Gutwin and Ssaul Greenberg. 2000. The mechanics of collaboration: Developing low cost usability evaluation methods for shared workspaces. In IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (Gaithersburg, MD, USA). 98--103.
[36]
Richard Hackathorn and Todd Margolis. 2016. Immersive analytics: Building virtual data worlds for collaborative decision support. In Workshop on Immersive Analytics (Greenville, SC, USA). 44--47.
[37]
Sandra G. Hart. 2006. Nasa-Task Load Index (NASA-TLX); 20 Years Later. Human Factors and Ergonomics Society Annual Meeting, Vol. 50, 9 (2006), 904--908.
[38]
Jeffrey Heer and Maneesh Agrawala. 2008. Design considerations for collaborative visual analytics. Information Visualization, Vol. 7, 1 (2008), 49--62.
[39]
Nicolaus Henke, Jordan Levine, and Paul McInerney. 2018. You don't have to be a data scientist to fill this must-have analytics role. Harvard Business Review (2018).
[40]
Juan David Hincapié-Ramos, Xiang Guo, Paymahn Moghadasian, and Pourang Irani. 2014. Consumed endurance: A metric to quantify arm fatigue of mid-air interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada). 1063--1072.
[41]
Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé, Miro Dudik, and Hanna Wallach. 2019. Improving fairness in machine learning systems: What do industry practitioners need?. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk). 1--16.
[42]
Ming-Tung Hong, Jesse Josua Benjamin, and Claudia Müller-Birn. 2018. Coordinating agents: Promoting shared situational awareness in collaborative sensemaking. In Companion of the ACM Conference on Computer Supported Cooperative Work and Social Computing (Jersey City, NJ, USA). 217--220.
[43]
Sungsoo Ray Hong, Jessica Hullman, and Enrico Bertini. 2020. Human factors in model interpretability: Industry practices, challenges, and needs. In Proceedings of the ACM on Human-Computer Interaction (Virtual Conference), Vol. 4. 1--26.
[44]
Aspen Hopkins and Serena Booth. 2021. Machine learning practices outside big tech: How resource constraints challenge responsible development. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (Virtual conference). 134--145.
[45]
Petra Isenberg, Niklas Elmqvist, Jean Scholtz, Daniel Cernea, Kwan-Liu Ma, and Hans Hagen. 2011. Collaborative visualization: Definition, challenges, and research agenda. Information Visualization, Vol. 10, 4 (2011), 310--326.
[46]
Petra Isenberg, Danyel Fisher, Sharoda A. Paul, Meredith Ringel Morris, Kori Inkpen, and Mary Czerwinski. 2012. Co-located collaborative visual analytics around a tabletop display. IEEE Transactions on Visualization and Computer Graphics, Vol. 18, 5 (2012), 689--702.
[47]
Michael I. Jordan and Tom M. Mitchell. 2015. Machine learning: Trends, perspectives, and prospects. Science, Vol. 349, 6245 (2015), 255--260.
[48]
Malte Jung, Jan Chong, and Larry Leifer. 2012. Group hedonic balance and pair programming performance: Affective interaction dynamics as indicators of performance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA). 829--838.
[49]
Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wortman Vaughan. 2020. Interpreting interpretability: Understanding data scientists' use of interpretability tools for machine learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--14.
[50]
Daniel Keim, Gennady Andrienko, Jean-Daniel Fekete, Carsten Görg, Jörn Kohlhammer, and Guy Melancc on. 2008. Visual analytics: Definition, process, and Challenges. In Information Visualization: Human-Centered Issues and Perspectives. Springer, Berlin, Heidelberg, Germany, 154--175.
[51]
Matthias Kraus, Niklas Weiler, Daniela Oelke, Johannes Kehrer, Daniel A. Keim, and Johannes Fuchs. 2020. The impact of immersion on cluster identification tasks. IEEE Transactions on Visualization and Computer Graphics, Vol. 26, 1 (2020), 525--535.
[52]
Ulla Kruhse-Lehtonen and Dirk Hofmann. 2020. How to define and execute your data and AI strategy. Harvard Data Science Review, Vol. 2 (2020).
[53]
Hannu Kuusela and Pallab Paul. 2000. A comparison of concurrent and retrospective verbal protocol analysis. American Journal of Psychology, Vol. 113, 3 (2000), 387--404.
[54]
Zachary C. Lipton. 2018. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, Vol. 16, 3 (2018), 31--57.
[55]
Narges Mahyar and Melanie Tory. 2014. Supporting communication and coordination in collaborative sensemaking. IEEE Transactions on Visualization and Computer Graphics, Vol. 20, 12 (2014), 1633--1642.
[56]
Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence, Vol. 267 (2019), 1--38.
[57]
Roshanak Zilouchian Moghaddam, Zane. Nicholson, and Brian P. Bailey. 2015. Procid: Bridging consensus building theory with the practice of distributed design discussions. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (Vancouver, BC, Canada). 686--699.
[58]
Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. 2020. Explaining machine learning classifiers through diverse counterfactual explanations. In Conference on Fairness, Accountability, and Transparency (Barcelona, Spain). 607--617.
[59]
Tamara Munzner. 2014. Visualization Analysis and Design. CRC Press, Boca Raton, FL.
[60]
John Ashworth Nelder and Robert WM Wedderburn. 1972. Generalized linear models. Journal of the Royal Statistical Society: Series A (General), Vol. 135, 3 (1972), 370--384.
[61]
Donald A. Norman. 2010. Natural user interfaces are not natural. Interactions, Vol. 17, 3 (2010), 6--10.
[62]
Samir Passi and Steven J. Jackson. 2018. Trust in data science: Collaboration, translation, and accountability in corporate data science projects. Proceedings of the ACM on Human-Computer Interaction, Vol. 2, CSCW (2018), 136.
[63]
Sharoda A. Paul and Madhu C. Reddy. 2010. Understanding together: Sensemaking in collaborative information seeking. In Conference on Computer Supported Cooperative Work (Savannah, Georgia, USA). 321--330.
[64]
David Pinelle, Carl Gutwin, and Saul Greenberg. 2003. Task analysis for groupware usability evaluation: Modeling shared-workspace tasks with the mechanics of collaboration. ACM Transactions on Computer-Human Interaction, Vol. 10, 4 (2003), 281--311.
[65]
Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake M. Hofman, Jennifer Wortman Wortman Vaughan, and Hanna Wallach. 2021. Manipulating and measuring model interpretability. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan). 1--52.
[66]
Nico Reski, Aris Alissandrakis, Jukka Tyrkkö, and Andreas Kerren. 2020. ?Oh, that's where you are!" -- Towards a hybrid asymmetric collaborative immersive analytics system. In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society (Tallinn, Estonia). 1--12.
[67]
Jeremy Roschelle and Stephanie D. Teasley. 1995. The construction of shared knowledge in collaborative problem solving. In Computer Supported Collaborative Learning. Springer, Berlin, Heidelberg, Germany, 69--97.
[68]
Stacey D. Scott, T. C. Nicholas Graham, James R. Wallace, Mark S. Hancock, and Miguel A. Nacenta. 2015. "Local remote" collaboration: Applying remote group awareness techniques to co-located settings. In Conference Companion on Computer Supported Cooperative Work & Social Computing. 319--324.
[69]
Mickael Sereno, Xiyao Wang, Lonni Besançon, Michael J. McGuffin, and Tobias Isenberg. 2022. Collaborative work in augmented reality: A survey. IEEE Transactions on Visualization and Computer Graphics, Vol. 28 (2022), 2530--2549.
[70]
Richard Skarbez, Nicholas F. Polys, J. Todd Ogle, Chris North, and Doug A. Bowman. 2019. Immersive analytics: Theory and research agenda. Frontiers in Robotics and AI, Vol. 6 (2019), 82.
[71]
Thilo Spinner, Udo Schlegel, Hanna Sch"afer, and Mennatallah El-Assady. 2020. explAIner: A visual analytics framework for interactive and explainable machine learning. IEEE Transactions on Visualization and Computer Graphics, Vol. 26, 1 (2020), 1064--1074.
[72]
Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harvard Journal of Law & Technology, Vol. 31 (2017), 841--887.
[73]
April Yi Wang, Anant Mittal, Christopher Brooks, and Steve Oney. 2019a. How data scientists use computational notebooks for real-time collaboration. Proceedings of the ACM on Human-Computer Interaction, Vol. 3, CSCW (2019), 1--30.
[74]
Dakuo Wang, Justin D Weisz, Michael Muller, Parikshit Ram, Werner Geyer, Casey Dugan, Yla Tausczik, Horst Samulowitz, and Alexander Gray. 2019b. Human-AI collaboration in data science: Exploring data scientists' perceptions of automated AI. Proceedings of the ACM on Human-Computer Interaction, Vol. 3, CSCW (2019), 1--24.
[75]
Xiyao Wang, Lonni Besanccon, David Rousseau, Mickael Sereno, Mehdi Ammi, and Tobias Isenberg. 2020. Towards an understanding of augmented reality extensions for existing 3D data analysis tools. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, Hawaii, USA). 1--13.
[76]
Amy X. Zhang, Michael Muller, and Dakuo Wang. 2020. How do data science workers collaborate? Roles, workflows, and tools. Proc. ACM Hum.-Comput. Interact., Vol. 4, CSCW, Article 22 (2020), 23 pages.io
Index Terms
- "Is It My Turn?": Assessing Teamwork and Taskwork in Collaborative Immersive Analytics
Recommendations
Clusters, Trends, and Outliers: How Immersive Technologies Can Facilitate the Collaborative Analysis of Multidimensional Data
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing SystemsImmersive technologies such as augmented reality devices are opening up a new design space for the visual analysis of data. This paper studies the potential of an augmented reality environment for the purpose of collaborative analysis of ...
MagicBook: transitioning between reality and virtuality
CHI EA '01: CHI '01 Extended Abstracts on Human Factors in Computing SystemsThe MagicBook explores how interfaces can be developed that allow for seamless transition between Physical Reality, Augmented Reality (AR), and immersive Virtual Reality (VR) in a collaborative setting. The MagicBook is a normal book and can be read ...
Collaboration in Augmented Reality
Augmented Reality (AR) is a technology that allows users to view and interact in real time with virtual images seamlessly superimposed over the real world. AR systems can be used to create unique collaborative experiences. For example, co-located users ...
Comments
Information & Contributors
Information
Published In
November 2022
8205 pages
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 November 2022
Published in PACMHCI Volume 6, Issue CSCW2
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 306Total Downloads
- Downloads (Last 12 months)123
- Downloads (Last 6 weeks)20
Reflects downloads up to 15 Oct 2024
Other Metrics
Citations
Cited By
View all- Friedl‐Knirsch JPointecker FPfistermüller SStach CAnthes CRoth D(2024)A Systematic Literature Review of User Evaluation in Immersive AnalyticsComputer Graphics Forum10.1111/cgf.1511143:3Online publication date: 10-Jun-2024
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in