research-article

Open access

TeamSlides: a Multimodal Teamwork Analytics Dashboard for Teacher-guided Reflection in a Physical Learning Space

Authors:

Vanessa Echeverria,

Riordan Alfredo,

Hollie Jaggard,

Rosie Wotherspoon,

Simon Buckingham Shum,

Dragan Gasevic,

Roberto Martinez-MaldonadoAuthors Info & Claims

LAK '24: Proceedings of the 14th Learning Analytics and Knowledge Conference

Pages 112 - 122

https://doi.org/10.1145/3636555.3636857

Published: 18 March 2024 Publication History

All formats PDF

Abstract

Advancements in Multimodal Learning Analytics (MMLA) have the potential to enhance the development of effective teamwork skills and foster reflection on collaboration dynamics in physical learning environments. Yet, only a few MMLA studies have closed the learning analytics loop by making MMLA solutions immediately accessible to educators to support reflective practices, especially in authentic settings. Moreover, deploying MMLA solutions in authentic settings can bring new challenges beyond logistic and privacy issues. This paper reports the design and use of TeamSlides, a multimodal teamwork analytics dashboard to support teacher-guided reflection. We conducted an in-the-wild classroom study involving 11 teachers and 138 students. Multimodal data were collected from students working in team healthcare simulations. We examined how teachers used the dashboard in 22 debrief sessions to aid their reflective practices. We also interviewed teachers to discuss their perceptions of the dashboard’s value and the challenges faced during its use. Our results suggest that the dashboard effectively reinforced discussions and augmented teacher-guided reflection practices. However, teachers encountered interpretation conflicts, sometimes leading to mistrust or misrepresenting the information. We discuss the considerations needed to overcome these challenges in MMLA research.

References

[1]

Karan Ahuja, Dohyun Kim, Franceska Xhakaj, Virag Varga, Anne Xie, Stanley Zhang, Jay Eric Townsend, Chris Harrison, Amy Ogan, and Yuvraj Agarwal. 2019. EduSense: Practical classroom sensing at Scale. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 3, Article 71 (2019), 26 pages. https://doi.org/10.1145/3351229

Digital Library

[2]

Haifa Alwahaby, Mutlu Cukurova, Zacharoula Papamitsiou, and Michail Giannakos. 2022. The Evidence of Impact and Ethical Considerations of Multimodal Learning Analytics: A Systematic Literature Review. Springer, 289–325. https://doi.org/10.1007/978-3-031-08076-0_12

[3]

Pengcheng An, Saskia Bakker, Sara Ordanovski, Ruurd Taconis, Chris LE Paffen, and Berry Eggen. 2019. Unobtrusively enhancing reflection-in-action of teachers through spatially distributed ambient information. In Proc. of the 2019 CHI conference on human factors in computing systems. 1–14.

Digital Library

[4]

Natalia Andrienko, Gennady Andrienko, and Peter Gatalsky. 2003. Exploratory spatio-temporal visualization: an analytical review. Journal of Visual Languages & Computing 14, 6 (2003), 503–541. https://doi.org/10.1016/S1045-926X(03)00046-6

[5]

Till Ballendat, Nicolai Marquardt, and Saul Greenberg. 2010. Proxemic Interaction: Designing for a Proximity and Orientation-Aware Environment. In ACM Intl. Conference on Interactive Tabletops and Surfaces (Saarbrücken, Germany). ACM, 121–130. https://doi.org/10.1145/1936652.1936676

Digital Library

[6]

Paulo Blikstein. 2013. Multimodal Learning Analytics. In Proc. of the 3rd Intl. Conference on Learning Analytics and Knowledge (Leuven, Belgium). ACM, 102–106. https://doi.org/10.1145/2460296.2460316

Digital Library

[7]

David Boud, Rosemary Keogh, and David Walker. 2013. Reflection: Turning experience into learning. Routledge.

[8]

Virginia Braun and Victoria Clarke. 2012. Thematic analysis. In Research designs: Quantitative, qualitative, neuropsychological, and biological.APA handbooks in psychology., Vol. 2. American Psychological Association, Washington, DC, US, 57–71. https://doi.org/10.1037/13620-004

[9]

Simon Buckingham Shum, Rebecca Ferguson, and Roberto Martinez-Maldonado. 2019. Human-Centred Learning Analytics. J. of Learning Analytics 6, 2 (2019), 1–9. https://doi.org/10.18608/jla.2019.62.1

[10]

Olga Chernikova, Nicole Heitzmann, Matthias Stadler, Doris Holzberger, Tina Seidel, and Frank Fischer. 2020. Simulation-based learning in higher education: A meta-analysis. Review of Educational Research 90, 4 (2020), 499–541.

[11]

Yi Han Victoria Chua, Justin Dauwels, and Seng Chee Tan. 2019. Technologies for automated analysis of co-located, real-life, physical learning spaces: Where are we now?. In Proc. of the 9th Intl. Conference on Learning Analytics & Knowledge (Tempe, AZ, USA). ACM, 11–20. https://doi.org/10.1145/3303772.3303811

Digital Library

[12]

Doug Clow. 2012. The Learning Analytics Cycle: Closing the Loop Effectively. In Proc. of the 2nd Intl. Conference on Learning Analytics and Knowledge (Vancouver, British Columbia, Canada). ACM, 134–138. https://doi.org/10.1145/2330601.2330636

Digital Library

[13]

Mutlu Cukurova, Michail Giannakos, and Roberto Martinez‐Maldonado. 2020. The promise and challenges of multimodal learning analytics. British J. of Educational Technology 51, 5 (2020), 1441–1449. https://doi.org/10.1111/bjet.13015

[14]

Sharon S. Dawes, Anthony M. Cresswell, and Bruce B. Cahan. 2004. Learning From Crisis: Lessons in Human and Information Infrastructure From the World Trade Center Response. Social Science Computer Review 22, 1 (2004), 52–66. https://doi.org/10.1177/0894439303259887

Digital Library

[15]

Daniele Di Mitri, Jan Schneider, Kevin Trebing, Sasa Sopka, Marcus Specht, and Hendrik Drachsler. 2020. Real-Time Multimodal Feedback with the CPR Tutor. In Artificial Intelligence in Education. Springer, 141–152.

[16]

Vanessa Echeverria, Roberto Martinez-Maldonado, and Simon Buckingham Shum. 2019. Towards Collaboration Translucence: Giving Meaning to Multimodal Group Data. In Proc. of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland UK). ACM, 1–16. https://doi.org/10.1145/3290605.3300269

Digital Library

[17]

Vanessa Echeverria, Roberto Martinez-Maldonado, Lixiang Yan, Linxuan Zhao, Gloria Fernandez-Nieto, Dragan Gašević, and Simon Buckingham Shum. 2023. HuCETA: A Framework for Human-Centered Embodied Teamwork Analytics. IEEE Pervasive Computing 22, 1 (2023), 39–49. https://doi.org/10.1109/MPRV.2022.3217454

Digital Library

[18]

Gloria Fernandez-Nieto, Pengcheng An, Jian Zhao, Simon Buckingham Shum, and Roberto Martinez-Maldonado. 2022. Classroom Dandelions: Visualising Participant Position, Trajectory and Body Orientation Augments Teachers’ Sensemaking. In 2022 CHI Conference on Human Factors in Computing Systems. 1–17.

[19]

Gloria Milena Fernandez Nieto, Kirsty Kitto, Simon Buckingham Shum, and Roberto Martinez-Maldonado. 2022. Beyond the Learning Analytics Dashboard: Alternative Ways to Communicate Student Data Insights Combining Visualisation, Narrative and Storytelling. In Proc. of the 12th Intl. Learning Analytics and Knowledge Conference (Online, USA). ACM, 219–229. https://doi.org/10.1145/3506860.3506895

Digital Library

[20]

Kristin L Fraser, Michael J Meguerdichian, Jolene T Haws, Vincent J Grant, Komal Bajaj, and Adam Cheng. 2018. Cognitive Load Theory for debriefing simulations: implications for faculty development. Advances in Simulation 3, 1 (2018), 1–8.

[21]

Dragan Gašević, Srećko Joksimović, Brendan R. Eagan, and David Williamson Shaffer. 2019. SENS: Network analytics to combine social and cognitive perspectives of collaborative learning. Computers in Human Behavior 92 (2019), 562–577. https://doi.org/10.1016/j.chb.2018.07.003

Digital Library

[22]

Michail Giannakos, Mutlu Cukurova, and Sofia Papavlasopoulou. 2022. Sensor-Based Analytics in Education: Lessons Learned from Research in Multimodal Learning Analytics. Springer Intl. Publishing, Cham, 329–358. https://doi.org/10.1007/978-3-031-08076-0_13

[23]

Karen Hall and Kathleen Tori. 2017. Best practice recommendations for debriefing in simulation-based education for Australian undergraduate nursing students: An integrative review. Clinical Simulation in Nursing 13, 1 (2017), 39–50.

[24]

Adam Kendon. 2010. Spacing and Orientation in Co-present Interaction. Springer, 1–15. https://doi.org/10.1007/978-3-642-12397-9_1

Digital Library

[25]

Charlotte Larmuseau, Jan Cornelis, Luigi Lancieri, Piet Desmet, and Fien Depaepe. 2020. Multimodal learning analytics to investigate cognitive load during online problem solving. British Journal of Educational Technology 51, 5 (2020), 1548–1562. https://doi.org/10.1111/bjet.12958

[26]

Sukwon Lee, Sung-Hee Kim, and Bum Chul Kwon. 2016. Vlat: Development of a visualization literacy assessment test. IEEE transactions on visualization and computer graphics 23, 1 (2016), 551–560.

[27]

Serena Lee-Cultura, Kshitij Sharma, and Michail Giannakos. 2023. MultiModal Teacher Dashboards: Challenges and Opportunities of Enhancing Teacher Insights through a case study. IEEE Transactions on Learning Technologies (2023), 1–19. https://doi.org/10.1109/TLT.2023.3276848

Digital Library

[28]

Lisa Lim, Shane Dawson, Srecko Joksimovic, and Dragan Gašević. 2019. Exploring Students’ Sensemaking of Learning Analytics Dashboards: Does Frame of Reference Make a Difference?. In Proc. of the 9th Intl. Conference on Learning Analytics & Knowledge (Tempe, AZ, USA). ACM, 250–259.

Digital Library

[29]

Roberto Martinez-Maldonado, Vanessa Echeverria, Gloria Fernandez Nieto, and Simon Buckingham Shum. 2020. From Data to Insights: A Layered Storytelling Approach for Multimodal Learning Analytics. In Proc, of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA). ACM, 1–15. https://doi.org/10.1145/3313831.3376148

Digital Library

[30]

Roberto Martinez-Maldonado, Vanessa Echeverria, Olga C. Santos, Augusto Dias Pereira Dos Santos, and Kalina Yacef. 2018. Physical Learning Analytics: A Multimodal Perspective. In Proc. of the 8th Intl. Conference on Learning Analytics and Knowledge (Sydney, NSW, Australia). ACM, 375–379.

Digital Library

[31]

Roberto Martinez-Maldonado, Abelardo Pardo, Negin Mirriahi, Kalina Yacef, Judy Kay, and Andrew Clayphan. 2015. LATUX: An Iterative Workflow for Designing, Validating, and Deploying Learning Analytics Visualizations.J. of Learning Analytics 2, 3 (2015), 9–39.

[32]

Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and Inter-rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice. Proc. of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–23. https://doi.org/10.1145/3359174

Digital Library

[33]

Xavier Ochoa. 2022. Multimodal learning analytics - Rationale, process, examples, and direction. Handbook of Learning Analytics (2022), 54–65.

[34]

Xavier Ochoa and Federico Dominguez. 2020. Controlled evaluation of a multimodal system to improve oral presentation skills in a real learning setting. British Journal of Educational Technology 51, 5 (2020), 1615–1630. https://doi.org/10.1111/bjet.12987

[35]

Sambit Praharaj, Maren Scheffel, Hendrik Drachsler, and Marcus Specht. 2021. Literature review on co-located collaboration modeling using multimodal learning analytics—can we go the whole nine yards?IEEE Transactions on Learning Technologies 14, 3 (2021), 367–385.

[36]

Sambit Praharaj, Maren Scheffel, Marcel Schmitz, Marcus Specht, and Hendrik Drachsler. 2022. Towards Collaborative Convergence: Quantifying Collaboration Quality with Automated Co-Located Collaboration Analytics. In Proc. of the 12th Intl. Learning Analytics and Knowledge Conference (Online, USA). ACM, 358–369.

Digital Library

[37]

Peter Reimann. 2016. Connecting learning analytics with learning research: the role of design-based research. Learning: Research and Practice 2, 2 (2016), 130–142. https://doi.org/10.1080/23735082.2016.1210198

[38]

Daniel T Risser, Matthew M Rice, Mary L Salisbury, Robert Simon, Gregory D Jay, Scott D Berns, MedTeams Research Consortium, 1999. The potential for improved teamwork to reduce medical errors in the emergency department. Annals of emergency medicine 34, 3 (1999), 373–383.

[39]

Yvonne Rogers, Paul Marshall, and John M Carroll. 2017. Research in the Wild. Springer. https://doi.org/10.1007/978-3-031-02220-3

[40]

Marija Sablić, Ana Mirosavljević, and Alma Škugor. 2021. Video-based learning (VBL)—past, present and future: An overview of the research published from 2008 to 2019. Technology, Knowledge and Learning 26, 4 (2021), 1061–1077.

[41]

Eduardo Salas, Denise L Reyes, and Susan H McDaniel. 2018. The science of teamwork: Progress, reflections, and the road ahead.American Psychologist 73, 4 (2018), 593.

[42]

Nazmus Saquib, Ayesha Bose, Dwyane George, and Sepandar Kamvar. 2018. Sensei: Sensing Educational Interaction. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 4 (2018), 27 pages. https://doi.org/10.1145/3161172

Digital Library

[43]

Donald A Schön. 2017. The reflective practitioner: How professionals think in action. Routledge.

[44]

Michael Seropian and Robert Lavey. 2010. Design considerations for healthcare simulation facilities. Simulation in Healthcare 5, 6 (2010), 338–345.

[45]

David Williamson Shaffer, Wesley Collier, and Andrew R Ruis. 2016. A tutorial on epistemic network analysis: Analyzing the structure of connections in cognitive, social, and interaction data. J. of Learning Analytics 3, 3 (2016), 9–45.

[46]

Kshitij Sharma and Michail Giannakos. 2020. Multimodal data capabilities for learning: What can multimodal data tell us about learning?British Journal of Educational Technology 51, 5 (2020), 1450–1484.

[47]

Antonette Shibani, Simon Knight, and Simon Buckingham Shum. 2020. Educator perspectives on learning analytics in classroom practice. The Internet and Higher Education 46 (2020). https://doi.org/10.1016/j.iheduc.2020.100730

[48]

Yi-Shan Tsai, Alexander Whitelock-Wainwright, and Dragan Gašević. 2021. More Than Figures on Your Laptop: (Dis)trustful Implementation of Learning Analytics. J. of Learning Analytics 8, 3 (2021), 81–100. https://doi.org/10.18608/jla.2021.7379

[49]

Katrien Verbert, Erik Duval, Joris Klerkx, Sten Govaerts, and José Luis Santos. 2013. Learning Analytics Dashboard Applications. American Behavioral Scientist 57, 10 (2013), 1500–1509. https://doi.org/10.1177/0002764213479363

[50]

Alyssa Friend Wise and Yeonji Jung. 2019. Teaching with analytics: Towards a situated model of instructional decision-making. J. of Learning Analytics 6, 2 (2019), 53–69.

[51]

Marcelo Worsley, Roberto Martinez-Maldonado, and Cynthia D’Angelo. 2021. A New Era in Multimodal Learning Analytics: Twelve Core Commitments to Ground and Grow MMLA.J. of Learning Analytics 8, 3 (2021), 10–27.

[52]

Lixiang Yan, Roberto Martinez-Maldonado, Beatriz Gallo Cordoba, Joanne Deppeler, Deborah Corrigan, Gloria Fernandez Nieto, and Dragan Gasevic. 2021. Footprints at School: Modelling In-Class Social Dynamics from Students’ Physical Positioning Traces. In Proc. of the 11th Intl. Learning Analytics and Knowledge Conference (Irvine, CA, USA). ACM, 43–54. https://doi.org/10.1145/3448139.3448144

Digital Library

[53]

Lixiang Yan, Roberto Martinez-Maldonado, Linxuan Zhao, Samantha Dix, Hollie Jaggard, Rosie Wotherspoon, Xinyu Li, and Dragan Gašević. 2023. The role of indoor positioning analytics in assessment of simulation-based learning. British Journal of Educational Technology 54, 1 (2023), 267–292.

[54]

Lixiang Yan, Linxuan Zhao, Dragan Gašević, and Roberto Martinez-Maldonado. 2022. Scalability, Sustainability, and Ethicality of Multimodal Learning Analytics. In Proc. of the 12th Intl. Learning Analytics and Knowledge Conference. ACM, New York, NY, USA, 13–23. https://doi.org/10.1145/3506860.3506862

Digital Library

[55]

Linxuan Zhao, Lixiang Yan, Dragan Gasevic, Samantha Dix, Hollie Jaggard, Rosie Wotherspoon, Riordan Alfredo, Xinyu Li, and Roberto Martinez-Maldonado. 2022. Modelling Co-Located Team Communication from Voice Detection and Positioning Data in Healthcare Simulation. In LAK22: 12th International Learning Analytics and Knowledge Conference (Online, USA). ACM, 370–380. https://doi.org/10.1145/3506860.3506935

Digital Library

[56]

Qi Zhou, Wannapon Suraworachet, Stanislav Pozdniakov, Roberto Martinez-Maldonado, Tom Bartindale, Peter Chen, Dan Richardson, and Mutlu Cukurova. 2021. Investigating students’ experiences with collaboration analytics for remote group meetings. In Proc. of the 22nd Intl. Conf. of Artificial Intelligence in Education (Utrecht, NL). Springer, 472–485.

Cited By

Milesi MAlfredo REcheverria VYan LZhao LTsai YMartinez-Maldonado R(2024)"It's Really Enjoyable to See Me Solve the Problem like a Hero": GenAI-enhanced Data Comics as a Learning Analytics ToolExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651111(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3651111
Echeverria VNieto GZhao LPalominos ESrivastava NGašević DPammer‐Schindler VMartinez‐Maldonado R(2024)A learning analytics dashboard to support students' reflection on collaborationJournal of Computer Assisted Learning10.1111/jcal.1308841:1Online publication date: 7-Nov-2024
https://doi.org/10.1111/jcal.13088
Yan LEcheverria VJin YFernandez‐Nieto GZhao LLi XAlfredo RSwiecki ZGašević DMartinez‐Maldonado R(2024)Evidence‐based multimodal learning analytics for feedback and reflection in collaborative learningBritish Journal of Educational Technology10.1111/bjet.1349855:5(1900-1925)Online publication date: 22-Jun-2024
https://doi.org/10.1111/bjet.13498
Show More Cited By

Index Terms

TeamSlides: a Multimodal Teamwork Analytics Dashboard for Teacher-guided Reflection in a Physical Learning Space
1. Applied computing
  1. Education
    1. Collaborative learning
2. Human-centered computing
  1. Collaborative and social computing
  2. Visualization
    1. Visualization systems and tools

Recommendations

Designing a learning analytics dashboard for twitter-facilitated teaching
L@S '18: Proceedings of the Fifth Annual ACM Conference on Learning at Scale

Social media sites are increasingly being adopted to support teaching practice in higher education. Learning Analytics (LA) dashboards can be used to reveal how students engage with course material and others in the class. However, research on the best ...
Multimodal Analytics for Collaborative Teacher Reflection of Human-AI Hybrid Teaching: Design Opportunities and Constraints
Responsive and Sustainable Educational Futures
Abstract
Past research shows that teachers benefit immensely from reflecting on their classroom practices. At the same time, adaptive and artificially intelligent (AI) tutors are shown to be highly effective for students, especially when teachers are ...
Learning Analytics as a Metacognitive Tool
CSEDU 2014: Proceedings of the 6th International Conference on Computer Supported Education - Volume 1

The use of learning analytics is entering in the field of research in education as a promising way to support learning. However, in many cases data are not transparent for the learner. In this regard, Educational institutions shouldnâ t escape the need ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

LAK '24: Proceedings of the 14th Learning Analytics and Knowledge Conference

March 2024

962 pages

ISBN:9798400716188

DOI:10.1145/3636555

Copyright © 2024 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 March 2024

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Australian Research Council

Conference

LAK '24

LAK '24: The 14th Learning Analytics and Knowledge Conference

March 18 - 22, 2024

Kyoto, Japan

Acceptance Rates

Overall Acceptance Rate 236 of 782 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
861
Total Downloads

Downloads (Last 12 months)861
Downloads (Last 6 weeks)133

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Milesi MAlfredo REcheverria VYan LZhao LTsai YMartinez-Maldonado R(2024)"It's Really Enjoyable to See Me Solve the Problem like a Hero": GenAI-enhanced Data Comics as a Learning Analytics ToolExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651111(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3651111
Echeverria VNieto GZhao LPalominos ESrivastava NGašević DPammer‐Schindler VMartinez‐Maldonado R(2024)A learning analytics dashboard to support students' reflection on collaborationJournal of Computer Assisted Learning10.1111/jcal.1308841:1Online publication date: 7-Nov-2024
https://doi.org/10.1111/jcal.13088
Yan LEcheverria VJin YFernandez‐Nieto GZhao LLi XAlfredo RSwiecki ZGašević DMartinez‐Maldonado R(2024)Evidence‐based multimodal learning analytics for feedback and reflection in collaborative learningBritish Journal of Educational Technology10.1111/bjet.1349855:5(1900-1925)Online publication date: 22-Jun-2024
https://doi.org/10.1111/bjet.13498
Yan LZhao LEcheverria VJin YAlfredo RLi XGaševi’c DMartinez-Maldonado R(2024)VizChat: Enhancing Learning Analytics Dashboards with Contextualised Explanations Using Multimodal Generative AI ChatbotsArtificial Intelligence in Education10.1007/978-3-031-64299-9_13(180-193)Online publication date: 2-Jul-2024
https://doi.org/10.1007/978-3-031-64299-9_13

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten