research-article

Public Access

Focused or stuck together: multimodal patterns reveal triads' performance in collaborative problem solving

Authors:

Mary Jean Amon,

Angela Stewart,

Nicholas D. Duran,

Sidney K. D'MelloAuthors Info & Claims

LAK '20: Proceedings of the Tenth International Conference on Learning Analytics & Knowledge

Pages 295 - 304

https://doi.org/10.1145/3375462.3375467

Published: 23 March 2020 Publication History

Abstract

Collaborative problem solving (CPS) in virtual environments is an increasingly important context of 21st century learning. However, our understanding of this complex and dynamic phenomenon is still limited. Here, we examine unimodal primitives (activity on the screen, speech, and body movements), and their multimodal combinations during remote CPS. We analyze two datasets where 116 triads collaboratively engaged in a challenging visual programming task using video conferencing software. We investigate how UI-interactions, behavioral primitives, and multimodal patterns were associated with teams' subjective and objective performance outcomes. We found that idling with limited speech (i.e., silence or backchannel feedback only) and without movement was negatively correlated with task performance and with participants' subjective perceptions of the collaboration. However, being silent and focused during solution execution was positively correlated with task performance. Results illustrate that in some cases, multimodal patterns improved the predictions and improved explanatory power over the unimodal primitives. We discuss how the findings can inform the design of real-time interventions for remote CPS.

References

[1]

Mary Jean Amon, Hana Vrzakova, and Sidney K D'Mello. 2019. Beyond Dyadic Coordination: Multimodal Behavioral Irregularity in Triads Predicts Facets of Collaborative Problem Solving. Cogn. Sci. 43, 10 (2019).

[2]

Paulo Blikstein. 2013. Multimodal learning analytics. In ACM International Conference Proceeding Series.

Digital Library

[3]

Dan Calacci, Oren Lederman, David Shrier, and Alex "Sandy" Pentland. 2016. Breakout: An Open Measurement and Intervention Tool for Distributed Peer Learning Groups. CoRR abs/1607.0, (2016).

[4]

Prerna Chikersal, Maria Tomprou, Young Ji Kim, Anita Williams Woolley, and Laura Dabbish. 2017. Deep Structures of Collaboration: Physiological Correlates of Collective Intelligence and Group Satisfaction. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17), 873--888.

Digital Library

[5]

Michael Diehl and Wolfgang Stroebe. 1987. Productivity loss in brainstorming groups: Toward the solution of a riddle. J. Pers. Soc. Psychol. 53, 3 (1987), 497.

[6]

Pierre Dillenbourg. 1999. What do you mean by collaborative learning?

[7]

Muhterem Dindar, Iman Alikhani, Jonna Malmberg, Sanna Järvelä, and Tapio Seppänen. 2019. Examining shared monitoring in collaborative learning: A case of a recurrence quantification analysis approach. Comput. Human Behav. April 2018 (2019).

[8]

Lucca Eloy, Angela E.B. Stewart, Mary Jean Amon, Caroline Reinhardt, Amanda Michaels, Chen Sun, Valerie Shute, Nicholas D Duran, and Sidney D'Mello. 2019. Modeling Team-level Multimodal Dynamics During Multiparty Collaboration. In 2019 International Conference on Multimodal Interaction (ICMI '19), 244--258. Retrieved from

Digital Library

[9]

Abigail C Evans, Jacob O Wobbrock, and Katie Davis. 2016. Modeling Collaboration Patterns on an Interactive Tabletop in a Classroom Setting. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16), 860--871.

Digital Library

[10]

Michael Flor, Su-Youn Yoon, Jiangang Hao, Lei Liu, and Alina von Davier. 2016. Automated classification of collaborative problem solving interactions in simulated science tasks. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, 31--41.

[11]

Neil Fraser. 2015. Ten things we've learned from Blockly. In Proceedings of the 2015 IEEE Blocks and Beyond Workshop, 49--50.

Digital Library

[12]

Donald G Gardner and Jon L Pierce. 2016. Organization-based self-esteem in work teams. Gr. Process. Intergr. Relations 19, 3 (2016), 394--408.

[13]

Daniel Gigone and Reid Hastie. 1993. The common knowledge effect: Information sharing and group judgment. J. Pers. Soc. Psychol. 65, 5 (1993), 959.

[14]

Carl Gutwin, Scott Bateman, Gaurav Arora, and Ashley Coveney. 2017. Looking Away and Catching Up: Dealing with Brief Attentional Disconnection in Synchronous Groupware. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17), 2221--2235.

Digital Library

[15]

Jiangang Hao, Lei Chen, Michael Flor, Lei Liu, and Alina A von Davier. 2017. CPS-Rater: Automated Sequential Annotation for Conversations in Collaborative Problem-Solving Activities. ETS Res. Rep. Ser. 2017, 1 (2017), 1--9.

[16]

Tobias Hecking, Dorian Doberstein, and H Ulrich Hoppe. 2019. Predicting the Well-functioning of Learning Groups Under Privacy Restrictions. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK19), 245--249. Retrieved from

Digital Library

[17]

Tiffany Herder, Zachari Swiecki, Simon Skov Fougt, Andreas Lindenskov Tamborg, Benjamin Brink Allsopp, David Williamson Shaffer, and Morten Misfeldt. 2018. Supporting Teachers' Intervention in Students' Virtual Collaboration Using a Network Based Model. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK '18), 21--25. Retrieved from

Digital Library

[18]

Cindy E. Hmelo-Silver and Howard S. Barrows. 2008. Facilitating collaborative knowledge building. Cogn. Instr. (2008).

[19]

Fred Hohman, Andrew Head, Rich Caruana, Robert DeLine, and Steven M. Drucker. 2019. Gamut: A design probe to understand how data scientists understand machine learning models. Conf. Hum. Factors Comput. Syst. - Proc. (2019), 1--13.

[20]

Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Ryuichiro Higashinaka, and Junji Tomita. 2018. Analyzing Gaze Behavior and Dialogue Act During Turn-taking for Estimating Empathy Skill Level. In Proceedings of the 2018 on International Conference on Multimodal Interaction (ICMI '18), 31--39.

Digital Library

[21]

Irving Lester Janis. 1982. Groupthink: Psychological studies of policy decisions and fiascoes. Houghton Mifflin Boston.

[22]

Patrick Jermann and Kshitij Sharma. 2018. Gaze as a Proxy for Cognition and Communication. In 2018 IEEE 18th International Conference on Advanced Learning Technologies (ICALT), 152--154.

[23]

Steven J Karau and Kipling D Williams. 1993. Social loafing: A meta-analytic review and theoretical integration. J. Pers. Soc. Psychol. 65, 4 (1993), 681.

[24]

Norbert L Kerr and Steven E Bruun. 1983. Dispensability of member effort and group motivation losses: Free-rider effects. J. Pers. Soc. Psychol. 44, 1 (1983), 78.

[25]

Norbert L Kerr and R Scott Tindale. 2004. Group performance and decision making. Annu. Rev. Psychol. 55, (2004), 623--655.

[26]

Charlotte Larmuseau, Pieter Vanneste, Piet Desmet, and Fien Depaepe. 2019. Multichannel Data for Understanding Cognitive Affordances During Complex Problem Solving. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK19), 61--70.

Digital Library

[27]

Yiwen Lin, Nia Dowell, Andrew Godfrey, Heeryung Choi, and Christopher Brooks. 2019. Modeling Gender Dynamics in Intra and Interpersonal Interactions During Online Collaborative Learning. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK19), 431--435.

Digital Library

[28]

Emma M. Mercier, Steven E. Higgins, and Laura da Costa. 2014. Different leaders: Emergent organizational and intellectual leadership in children's collaborative learning groups. Int. J. Comput. Collab. Learn. (2014).

[29]

Daniele Di Mitri, Jan Schneider, Roland Klemke, Marcus Specht, and Hendrik Drachsler. 2019. Read between the lines: An annotation tool for multimodal data for learning. In ACM International Conference Proceeding Series.

Digital Library

[30]

Gabriel Murray and Catharine Oertel. 2018. Predicting Group Performance in Task-Based Interaction. In Proceedings of the 20th ACM International Conference on Multimodal Interaction (ICMI '18), 14--20.

Digital Library

[31]

Bernard A Nijstad, Wolfgang Stroebe, and Hein F M Lodewijkx. 2003. Production blocking and idea generation: Does blocking interfere with cognitive processes? J. Exp. Soc. Psychol. 39, 6 (2003), 531--548.

[32]

Xavier Ochoa, Nadir Weibel, Marcelo Worsley, and Sharon Oviatt. 2016. Multimodal learning analytics data challenges. In ACM International Conference Proceeding Series.

Digital Library

[33]

Kazuhiro Otsuka, Keisuke Kasuga, and Martina Köhler. 2018. Estimating Visual Focus of Attention in Multiparty Meetings Using Deep Convolutional Neural Networks. In Proceedings of the 20th ACM International Conference on Multimodal Interaction (ICMI '18), 191--199.

Digital Library

[34]

Richard V Palumbo, Marisa E Marraccini, Lisa L Weyandt, Oliver Wilder-Smith, Heather A McGee, Siwei Liu, and Matthew S Goodwin. 2017. Interpersonal autonomic physiology: A systematic review of the literature. Personal. Soc. Psychol. Rev. 21, 2 (2017), 99--141.

[35]

Joseph M Reilly and Chris Dede. 2019. Differences in Student Trajectories via Filtered Time Series Analysis in an Immersive Virtual World. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK19), 130--134.

Digital Library

[36]

Joseph M Reilly, Milan Ravenell, and Bertrand Schneider. 2018. Exploring Collaboration Using Motion Sensors and Multi-Modal Learning Analytics. Int. Educ. Data Min. Soc. (2018).

[37]

Jeremy Roschelle and Stephanie D Teasley. 1995. The construction of shared knowledge in collaborative problem solving. In Computer supported collaborative learning, 69--97.

[38]

Bertrand Schneider and Paulo Blikstein. 2018. Tangible User Interfaces and Contrasting Cases as a Preparation for Future Learning. J. Sci. Educ. Technol. (2018).

[39]

Jan Schneider, Dirk Börner, Peter Van Rosmalen, and Marcus Specht. 2015. Augmenting the senses: A review on sensor-based learning support. Sensors (Switzerland) (2015).

[40]

Julian Schulze and Stefan Krumm. 2017. The "virtual team player": A review and initial model of knowledge, skills, abilities, and other characteristics for virtual collaboration. Organ. Psychol. Rev. 7, 1 (2017), 66--95.

[41]

Daniel Spikol, Marcelo Worsley, Luis P. Prieto, Xavier Ochoa, M. J. Rodríguez-Triana, and Mutlu Cukurova. 2017. Current and future multimodal learning analytics data challenges. In ACM International Conference Proceeding Series.

Digital Library

[42]

Emma L Starr, Joseph M Reilly, and Bertrand Schneider. 2018. Toward Using Multi-Modal Learning Analytics to Support and Measure Collaboration in Co-Located Dyads. . International Society of the Learning Sciences, Inc.[ISLS].

[43]

Angela Stewart and Sidney K D'Mello. 2018. Connecting the Dots Towards Collaborative AIED: Linking Group Makeup to Process to Learning. In International Conference on Artificial Intelligence in Education, 545--556.

[44]

Angela E. B. Stewart, Zachary A. Keirn, and Sidney K D'Mello. 2018. Multimodal Modeling of Coordination and Coregulation Patterns in Speech Rate During Triadic Collaborative Problem Solving. In Proceedings of the 20th ACM International Conference on Multimodal Interaction (ICMI '18), 21--30.

Digital Library

[45]

Angela E.B. Stewart, Hana Vrzakova, Chen Sun, Jade; Yonehiro, Cathlyn Adele Stone, D Duran, Nicholas, Valerie Shute, and Sidney K. D'Mello. 2019. I Say, You Say, We Say: Using Spoken Language to Model Socio-Cognitive Processes during Computer-Supported Collaborative Problem Solving. Proc. ACM Hum. Comput. Interact. 3, November (2019), 19.

[46]

Angela Stewart and Sidney K D Mello. Connecting the Dots towards Collaborative AIED : Linking Group Makeup to Process to Learning.

[47]

Code Studio. Code Studio.

[48]

Chen Sun, Valerie J Shute, Angela Stewart, Jade Yonehiro, Nicholas Duran, and Sidney D'Mello. 2020. Towards a generalized competency model of collaborative problem solving. Comput. Educ. 143, (2020), 103672.

[49]

Hana Vrzakova, Mary Jean Amon, Angela E B Stewart, and Sidney K D'Mello. 2019. Dynamics of Visual Attention in Multiparty Collaborative Problem Solving Using Multidimensional Recurrence Quantification Analysis. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19), 342:1--342:14.

Digital Library

[50]

Sebastian Wallot, Andreas Roepstorff, and Dan Mønster. 2016. Multidimensional Recurrence Quantification Analysis (MdRQA) for the analysis of multidimensional time-series: A software implementation in MATLAB and its application to group-level data in joint action. Front. Psychol. 7, (2016), 1835.

[51]

Jacqueline Kory Westlund, Sidney K D'Mello, and Andrew M Olney. 2015. Motion Tracker: camera-based monitoring of bodily movements using motion silhouettes. PLoS One 10, 6 (2015), e0130293.

[52]

M Worsley. 2018. (Dis)Engagement matters: Identifying efficacious learning practices with multimodal learning analytics. (2018), 365--369.

[53]

Marcelo Worsley. 2014. Multimodal learning analytics as a tool for bridging learning theory and complex learning behaviors. In MLA 2014 - Proceedings of the 2014 ACM Multimodal Learning Analytics Workshop and Grand Challenge, Co-located with ICMI 2014.

Digital Library

[54]

Bin Xie, Joseph M Reilly, Yong Li Dich, and Bertrand Schneider. 2018. Augmenting Qualitative Analyses of Collaborative Learning Groups Through Multi-Modal Sensing. . International Society of the Learning Sciences, Inc.[ISLS].

[55]

Qian Yang, Alex Scuito, John Zimmerman, Jodi Forlizzi, and Aaron Steinfeld. 2018. Investigating how experienced UX designers effectively work with machine learning. DIS 2018 - Proc. 2018 Des. Interact. Syst. Conf. (2018), 585--596.

Digital Library

[56]

Jaebong Yoo and Jihie Kim. 2014. Can online discussion participation predict group project performance? Investigating the roles of linguistic features and participation patterns. Int. J. Artif. Intell. Educ. 24, 1 (2014), 8--32.

[57]

Guang Yong Zou. 2007. Toward using confidence intervals to compare correlations. Psychol. Methods 12, 4 (2007), 399.

[58]

IBM. Retrieved May 2, 2018 from https://www.ibm.com/watson/services/speech-to-text/

[59]

2015. PISA 2015 Collaborative Problem Solving Framework.

Cited By

Chandler CBreideband TReitman JChitwood MBush JHoward ALeonhart SFoltz PPenuel WD'Mello S(2024)Computational Modeling of Collaborative Discourse to Enable Feedback and Reflection in Middle School ClassroomsProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636917(576-586)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636917
Anwar ATick ALim YKee DSurienty L(2024)Business student problem-solving skills and e-learning management in Malaysia: mediating role of student psychological motivation and peer collaborationAsian Education and Development Studies10.1108/AEDS-11-2023-0155Online publication date: 4-Jun-2024
https://doi.org/10.1108/AEDS-11-2023-0155
Mugula LTufon RShi YValero MRandolph ANino V(2024)Investigating the Impact of Emotions on the Quality of Novice Programmers’ CodeInformation Systems and Neuroscience10.1007/978-3-031-58396-4_7(67-78)Online publication date: 26-Jul-2024
https://doi.org/10.1007/978-3-031-58396-4_7
Show More Cited By

Index Terms

Focused or stuck together: multimodal patterns reveal triads' performance in collaborative problem solving
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing theory, concepts and paradigms
      1. Collaborative content creation
      2. Computer supported cooperative work
    2. Empirical studies in collaborative and social computing

Recommendations

Dynamics of Visual Attention in Multiparty Collaborative Problem Solving using Multidimensional Recurrence Quantification Analysis
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

Multiparty collaborative problem solving - an increasingly important context in the 21st century workforce - suffers from a degradation of social and behavioral signals when attempted remotely, resulting in suboptimal outcomes. We investigate teams' ...
Editable Chat Logs: A Concept for Seamless Integration of Chat Conversations and Documents in Shared Workspaces
Proceedings of the 2006 conference on Cooperative Systems Design: Seamless Integration of Artifacts and Conversations -- Enhanced Concepts of Infrastructure for Communication

This paper introduces the concept of editable chat logs for shared workspace systems. In shared workspaces offering a chat for synchronous communication, editable chat logs allow to keep and archive transcripts of chat conversations as documents in the ...
Multimodal Analytics for Real-Time Feedback in Co-located Collaboration
Lifelong Technology-Enhanced Learning
Abstract
Collaboration is an important 21st century skill; it can take place in a remote or co-located setting. Co-located collaboration (CC) is a very complex process which involves subtle human interactions that can be described with multimodal ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

LAK '20: Proceedings of the Tenth International Conference on Learning Analytics & Knowledge

March 2020

679 pages

ISBN:9781450377126

DOI:10.1145/3375462

General Chairs:
Christoph Rensing
Technische Universität Darmstadt, Germany
,
Hendrik Drachsler
DIPF Leibniz Institute for Research and Information in Education & University of Frankfurt, Germany
,
Program Chairs:
Vitomir Kovanović
University of South Australia, Australia
,
Niels Pinkwart
Humboldt-Universität zu Berlin, Germany
,
Maren Scheffel
Open Universiteit, Netherlands
,
Katrien Verbert
KU Leuven, Belgium

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 March 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

LAK '20

LAK '20: 10th International Conference on Learning Analytics and Knowledge

March 23 - 27, 2020

Frankfurt, Germany

Acceptance Rates

LAK '20 Paper Acceptance Rate 80 of 261 submissions, 31%;

Overall Acceptance Rate 236 of 782 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

30
Total Citations
View Citations
1,330
Total Downloads

Downloads (Last 12 months)290
Downloads (Last 6 weeks)33

Reflects downloads up to 12 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chandler CBreideband TReitman JChitwood MBush JHoward ALeonhart SFoltz PPenuel WD'Mello S(2024)Computational Modeling of Collaborative Discourse to Enable Feedback and Reflection in Middle School ClassroomsProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636917(576-586)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636917
Anwar ATick ALim YKee DSurienty L(2024)Business student problem-solving skills and e-learning management in Malaysia: mediating role of student psychological motivation and peer collaborationAsian Education and Development Studies10.1108/AEDS-11-2023-0155Online publication date: 4-Jun-2024
https://doi.org/10.1108/AEDS-11-2023-0155
Mugula LTufon RShi YValero MRandolph ANino V(2024)Investigating the Impact of Emotions on the Quality of Novice Programmers’ CodeInformation Systems and Neuroscience10.1007/978-3-031-58396-4_7(67-78)Online publication date: 26-Jul-2024
https://doi.org/10.1007/978-3-031-58396-4_7
Doherty ESpencer CBurnison JČeko MChin JEloy LHaring KKim PPittman DPowers SPugh SRoumis DStephens JYeh THirshfield L(2023)Interdisciplinary views of fNIRS: Current advancements, equity challenges, and an agenda for future needs of a diverse fNIRS research communityFrontiers in Integrative Neuroscience10.3389/fnint.2023.105967917Online publication date: 27-Feb-2023
https://doi.org/10.3389/fnint.2023.1059679
Martinez-Maldonado REcheverria VFernandez-Nieto GYan LZhao LAlfredo RLi XDix SJaggard HWotherspoon ROsborne AShum SGašević D(2023)Lessons Learnt from a Multimodal Learning Analytics Deployment In-the-WildACM Transactions on Computer-Human Interaction10.1145/362278431:1(1-41)Online publication date: 29-Nov-2023
https://dl.acm.org/doi/10.1145/3622784
Eloy LSpencer CDoherty EHirshfield L(2023)Capturing the Dynamics of Trust and Team Processes in Human-Human-Agent Teams via Multidimensional Neural Recurrence AnalysesProceedings of the ACM on Human-Computer Interaction10.1145/35795987:CSCW1(1-23)Online publication date: 16-Apr-2023
https://dl.acm.org/doi/10.1145/3579598
Ma YCelepkolu MBoyer KLynch CWiebe EIsrael M(2023)How Noisy is Too Noisy? The Impact of Data Noise on Multimodal Recognition of Confusion and Conflict During Collaborative LearningProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614127(326-335)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614127
Moulder RBooth BAbitino AD'Mello SHilliger IKhosravi HRienties BDawson S(2023)Recurrence Quantification Analysis of Eye Gaze Dynamics During Team CollaborationLAK23: 13th International Learning Analytics and Knowledge Conference10.1145/3576050.3576113(430-440)Online publication date: 13-Mar-2023
https://dl.acm.org/doi/10.1145/3576050.3576113
Rajarathinam RD'Angelo CHilliger IKhosravi HRienties BDawson S(2023)Turn-taking analysis of small group collaboration in an engineering discussion classroomLAK23: 13th International Learning Analytics and Knowledge Conference10.1145/3576050.3576099(650-656)Online publication date: 13-Mar-2023
https://dl.acm.org/doi/10.1145/3576050.3576099
Buseyne SRajagopal KDanquigny TDepaepe FHeutte JRaes A(2023)Assessing verbal interaction of adult learners in computer‐supported collaborative problem solvingBritish Journal of Educational Technology10.1111/bjet.13391Online publication date: 29-Sep-2023
https://doi.org/10.1111/bjet.13391
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents