research-article

Public Access

Multimodal Modeling of Coordination and Coregulation Patterns in Speech Rate during Triadic Collaborative Problem Solving

Authors:

Angela E.B. Stewart,

Zachary A. Keirn, and

Sidney K. D'MelloAuthors Info & Claims

ICMI '18: Proceedings of the 20th ACM International Conference on Multimodal Interaction

October 2018

Pages 21 - 30

https://doi.org/10.1145/3242969.3242989

Published: 02 October 2018 Publication History

Abstract

We model coordination and coregulation patterns in 33 triads engaged in collaboratively solving a challenging computer programming task for approximately 20 minutes. Our goal is to prospectively model speech rate (words/sec) - an important signal of turn taking and active participation - of one teammate (A or B or C) from time lagged nonverbal signals (speech rate and acoustic-prosodic features) of the other two (i.e., A + B → C; A + C → B; B + C → A) and task-related context features. We trained feed-forward neural networks (FFNNs) and long short-term memory recurrent neural networks (LSTMs) using group-level nested cross-validation. LSTMs outperformed FFNNs and a chance baseline and could predict speech rate up to 6s into the future. A multimodal combination of speech rate, acoustic-prosodic, and task context features outperformed unimodal and bimodal signals. The extent to which the models could predict an individual's speech rate was positively related to that individual's scores on a subsequent posttest, suggesting a link between coordination/coregulation and collaborative learning outcomes. We discuss applications of the models for real-time systems that monitor the collaborative process and intervene to promote positive collaborative outcomes.

References

[1]

Drew H. Abney, Alexandra Paxton, Rick Dale, and Christopher T. Kello. 2014. Complexity matching in dyadic conversation. J. Exp. Psychol. Gen. 143, 6 (2014), 2304--2315.

[2]

Richard Alterman and Kendall Harsch. 2017. A more reflective form of joint problem solving. Int. J. Comput. Collab. Learn. 12, 1 (March 2017), 9--33.

[3]

Brigid Barron. 2000. Achieving coordination in collaborative problem-solving groups. J. Learn. Sci. 9, 4 (2000), 403--436.

[4]

Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting Linear Mixed-Effects Models Using lme4. J. Stat. Softw. 67, 1 (2015), 1--48.

[5]

Emily A. Butler and Ashley K. Randall. 2013. Emotional Coregulation in Close Relationships. Emot. Rev. 5, 2 (April 2013), 202--210.

[6]

Herbert H. Clark and Susan E. Brennan. 1991. Grounding in communication. Perspect. Soc. Shar. Cogn. 13, 1991 (1991), 127--149.

[7]

Jennifer Coates. 1994. No gap, lots of overlap; turn-taking patterns in the talk of women friends. In Researching Language and Literacy in Social Context: A Reader, David Graddol, Janet Maybin and Barry Stierer (eds.). Multilingual Matters, 177--191.

[8]

Traci Y. Craig and Janice R. Kelly. 1999. Group cohesiveness and creative performance. Gr. Dyn. Theory, Res. Pract. 3, 4 (1999), 243--256.

[9]

Emilie Delaherche, Mohamed Chetouani, Ammar Mahdhaoui, Catherine Saint-Georges, Sylvie Viaux, and David Cohen. 2012. Interpersonal synchrony: a survey of evaluation methods across disciplines. IEEE Trans. Affect. Comput. 3, 3 (October 2012), 349--365.

Digital Library

[10]

Michael Diehl and Wolfgang Stroebe. 1987. Productivity loss in brainstorming groups: Toward the solution of a riddle. J. Pers. Soc. Psychol. 53, 3 (1987), 497--509.

[11]

Alfred Dielmann, Giulia Garau, and Hervé Bourlard. 2010. Floor holder detection and end of speaker turn prediction in meetings. In Proceedings of the International Conference on Speech and Language Processing, Interspeech.

[12]

Pierre Dillenbourg. 1999. What do you mean by collaborative learning? Collab. Cogn. Comput. Approaches 1, 6 (1999), 1--19.

[13]

Timothy Dozat. 2016. Incorporating nesterov momentum into adam. In Proceedings of the International Conference on Learning Representations.

[14]

Nicholas D. Duran and Riccardo Fusaroli. 2017. Conversing with a devil's advocate: Interpersonal coordination in deception and disagreement. PLoS One 12, 6 (June 2017), e0178140.

[15]

Charles R. Evans and Kenneth L. Dion. 1991. Group cohesion and performance: A meta-analysis. Small Gr. Res. 22, 2 (1991), 175--186.

[16]

Florian Eyben, Felix Weninger, Florian Gross, and Björn Schuller. 2013. Recent developments in openSMILE, the Munich open-source multimedia feature extractor. In Proceedings of the 21st ACM international conference on Multimedia (MM '13), 835--838.

Digital Library

[17]

Bo Fan, Lijuan Wang, Frank K. Soong, and Lei Xie. 2015. Photo-real talking head with deep bidirectional LSTM. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 4884--4888.

[18]

Will Feng, Anitha Kannan, Georgia Gkioxari, and C. Lawrence Zitnick. 2017. Learn2Smile: Learning non-verbal interaction through observation. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 4131--4138.

[19]

John Fox, Michael Friendly, and Sanford Weisberg. 2013. Hypothesis tests for multivariate linear models using the car package. R J. 5, 1 (2013), 39--52.

[20]

Neil Fraser. 2015. Ten things we've learned from Blockly. In Proceedings of the 2015 IEEE Blocks and Beyond Workshop, 49--50.

Digital Library

[21]

Daniel Gigone and Reid Hastie. 1993. The common knowledge effect: Information sharing and group judgment. J. Pers. Soc. Psychol. 65, 5 (1993), 959--974.

[22]

Jamie C. Gorman, Nancy J. Cooke, Polemnia G. Amazeen, and Shannon Fouse. 2012. Measuring Patterns in Team Interaction Sequences Using a Discrete Recurrence Approach. Hum. Factors J. Hum. Factors Ergon. Soc. 54, 4 (August 2012), 503--517.

[23]

Joseph Grafsgaard, Nicholas Duran, Ashley Randall, Chun Tao, and Sidney D'Mello. 2018. Generative multimodal models of nonverbal synchrony in close relationships. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), 195--202.

Digital Library

[24]

Stanley M. Gully, Dennis J. Devine, and David J. Whitney. 1995. A meta-analysis of cohesion and performance. Effects of level of analysis and task interdependence. Small Gr. Res. 26, 4 (November 1995), 497--520.

[25]

Cindy E. Hmelo-Silver and Howard S. Barrows. 2008. Facilitating collaborative knowledge building. Cogn. Instr. 26, 1 (January 2008), 48--94.

[26]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Comput. 9, 8 (November 1997), 1735--1780.

Digital Library

[27]

Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, 448--456.

Digital Library

[28]

Irving Lester Janis. 1982. Groupthink: Psychological studies of policy decisions and fiascoes. Houghton Mifflin, Boston.

[29]

Steven J. Karau and Kipling D. Williams. 1993. Social loafing: A meta-analytic review and theoretical integration. J. Pers. Soc. Psychol. 65, 4 (1993), 681--706.

[30]

Norbert L. Kerr. 1983. Motivation losses in small groups: A social dilemma analysis. J. Pers. Soc. Psychol. 45, 4 (1983), 819--828.

[31]

Norbert L. Kerr and R. Scott Tindale. 2004. Group performance and decision making. Annu. Rev. Psychol. 55, 1 (February 2004), 623--655.

[32]

Norbert L Kerr and Steven E Bruun. 1983. Dispensability of member effort and group motivation losses: Free-rider effects. J. Pers. Soc. Psychol. 44, 1 (1983), 78--94.

[33]

Iwan de Kok and Dirk Heylen. 2009. Multimodal end-of-turn prediction in multi-party meetings. In Proceedings of the 2009 International Conference on Multimodal Interfaces (ICMI-MLMI '09), 91--98.

Digital Library

[34]

Claus W. Langfred. 1998. Is group cohesiveness a double-edged sword? Small Gr. Res. 29, 1 (February 1998), 124--143.

[35]

Edward W. Large and Mari Riess Jones. 1999. The dynamics of attending: How people track time-varying events. Psychol. Rev. 106, 1 (1999), 119--159.

[36]

Patrick R. Laughlin and Alan L. Ellis. 1986. Demonstrability and social combination processes on mathematical intellective tasks. J. Exp. Soc. Psychol. 22, 3 (May 1986), 177--189.

[37]

Patrick R. Laughlin, Erin C. Hatch, Jonathan S. Silver, and Lee Boh. 2006. Groups perform better than the best individuals on letters-to-numbers problems: Effects of group size. J. Pers. Soc. Psychol. 90, 4 (2006), 644--651.

[38]

Patrick R. Laughlin, Norbert L. Kerr, James. H. Davis, Henry. M. Halff, and Kenneth. A. Marciniak. 1975. Group size, member ability, and social decision schemes on an intellective task. J. Pers. Soc. Psychol. 31, 3 (1975), 522--535.

[39]

R. Lenth. 2018. Emmeans: Estimated marginal means, aka least-squares means. R Package.

[40]

Russell V. Lenth. 2016. Least-squares means: the R package lsmeans. J. Stat. Softw. 69, 1 (2016), 1--33.

[41]

Rivka Levitan, Agustin Gravano, Laura Willson, Stefan Benus, Julia Hirschberg, and Ani Nenkova. 2012. Acoustic-prosodic entrainment and social behavior. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT '12), 11--19.

Digital Library

[42]

Zachary C. Lipton, David C. Kale, and Randall Wetzel. 2016. Directly modeling missing data in sequences with RNNs: Improved classification of clinical time series. In Proceedigns of Machine Learning Research, 253--270.

[43]

Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the International Conference on Machine Learning.

[44]

Joseph H. Manson, Gregory A. Bryant, Matthew M. Gervais, and Michelle A. Kline. 2013. Convergence of speech rate in conversation predicts cooperation. Evol. Hum. Behav. 34, 6 (November 2013), 419--426.

[45]

Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan Yuille. 2015. Deep captioning with multimodal recurrent neural networks (M-RNN). In Proceedings of the 2015 International Conference on Learning Representations.

[46]

Bernard A. Nijstad, Wolfgang Stroebe, and Hein F.M. Lodewijkx. 2003. Production blocking and idea generation: Does blocking interfere with cognitive processes? J. Exp. Soc. Psychol. 39, 6 (November 2003), 531--548.

[47]

Hai X. Pham, Samuel Cheung, and Vladamir Pavlovic. 2017. Speech-driven 3D facial animation with implicit emotional awareness: A deep learning approach. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2328--2336.

[48]

Daniel C. Richardson and Rick Dale. 2005. Looking to understand: The coupling between speakers' and listeners' eye movements and its relationship to discourse comprehension. Cogn. Sci. 29, 6 (November 2005), 1045--1060.

[49]

Daniel C. Richardson, Rick Dale, and Natasha Z. Kirkham. 2007. The art of conversation is coordination. Psychol. Sci. 18, 5 (May 2007), 407--413.

[50]

Michael A. Riley, Michael J. Richardson, Kevin Shockley, and Verónica C. Ramenzoni. 2011. Interpersonal synergies. Front. Psychol. 2, (2011), 38.

[51]

Emanuel A. Schegloff. 2000. Overlapping talk and the organization of turn-taking for conversation. Lang. Soc. 29, 1 (2000), 1--63.

[52]

Julian Schulze and Stefan Krumm. 2017. The "virtual team player": A review and initial model of knowledge, skills, abilities, and other characteristics for virtual collaboration. Organ. Psychol. Rev. 7, 1 (February 2017), 66--95.

[53]

David A. Sears and James Michael Reagin. 2013. Individual versus collaborative problem solving: Divergent outcomes depending on task complexity. Instr. Sci. 41, 6 (November 2013), 1153--1172.

[54]

Kevin Shockley, Daniel C. Richardson, and Rick Dale. 2009. Conversation and Coordinative Structures. Top. Cogn. Sci. 1, 2 (April 2009), 305--319.

[55]

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1 (2014), 1929--1958.

Digital Library

[56]

I. D. Steiner. 1972. Group processes and group productivity. New York Acad. (1972).

[57]

Chen Sun, Valerie Shute, Angela E.B. Stewart, Jade Yonehiro, Nicholas Duran, and Sidney K. D'Mello. Toward a generalized competency model of collaborative problem solving. Rev.

[58]

Jacqueline Kory Westlund, Sidney K. D'Mello, and Andrew M. Olney. 2015. Motion Tracker: Camera-based monitoring of bodily movements using motion silhouettes. PLoS One 10, 6 (June 2015), e0130293.

[59]

Stephen J. Zaccaro. 1991. Nonequivalent associations between forms of cohesiveness and group-related outcomes: evidence for multidimensionality. J. Soc. Psychol. 131, 3 (June 1991), 387--399.

[60]

Code Studio. Retrieved April 1, 2018 from https://studio.code.org/s/mc/stage/1/puzzle/1

[61]

IBM. Retrieved May 2, 2018 from https://www.ibm.com/watson/services/speech-to-text/

[62]

Keras. Retrieved May 2, 2018 from https://github.com/keras-team/keras

[63]

Convolutional neural networks for visual recognition. Retrieved from http://cs231n.github.io/neural-networks-1/

[64]

Scikit Learn. Retrieved May 3, 2018 from https://github.com/scikit-learn/scikit-learn

[65]

Convolutional neural networks for visual recognition. Retrieved May 3, 2018 from http://cs231n.github.io/neural-networks-2/

[66]

2015. PISA 2015 Collaborative Problem Solving Framework.

Cited By

Chandler CBreideband TReitman JChitwood MBush JHoward ALeonhart SFoltz PPenuel WD'Mello S(2024)Computational Modeling of Collaborative Discourse to Enable Feedback and Reflection in Middle School ClassroomsProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636917(576-586)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636917
D’Mello SDuran NMichaels AStewart A(2024)Improving collaborative problem-solving skills via automated feedback and scaffolding: a quasi-experimental study with CPSCoach 2.0User Modeling and User-Adapted Interaction10.1007/s11257-023-09387-6Online publication date: 14-Feb-2024
https://doi.org/10.1007/s11257-023-09387-6
Moulder RBooth BAbitino AD'Mello SHilliger IKhosravi HRienties BDawson S(2023)Recurrence Quantification Analysis of Eye Gaze Dynamics During Team CollaborationLAK23: 13th International Learning Analytics and Knowledge Conference10.1145/3576050.3576113(430-440)Online publication date: 13-Mar-2023
https://dl.acm.org/doi/10.1145/3576050.3576113
Show More Cited By

Index Terms

Multimodal Modeling of Coordination and Coregulation Patterns in Speech Rate during Triadic Collaborative Problem Solving
1. Human-centered computing
  1. Collaborative and social computing
    1. Empirical studies in collaborative and social computing

Recommendations

Multimodal, Multiparty Modeling of Collaborative Problem Solving Performance
ICMI '20: Proceedings of the 2020 International Conference on Multimodal Interaction

Modeling team phenomena from multiparty interactions inherently requires combining signals from multiple teammates, often by weighting strategies. Here, we explored the hypothesis that strategic weighting signals from individual teammates would ...
Read More
Acoustic-Prosodic Entrainment and Rapport in Collaborative Learning Dialogues
MLA '14: Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge

In spoken dialogue analysis, the speech signal is a rich source of information. We explore in this paper how low level features of the speech signal, such as pitch, loudness, and speaking rate, can inform a model of student interaction in collaborative ...
Read More
Detecting Impasse During Collaborative Problem Solving with Multimodal Learning Analytics
LAK22: LAK22: 12th International Learning Analytics and Knowledge Conference

Collaborative problem solving has numerous benefits for learners, such as improving higher-level reasoning and developing critical thinking. While learners engage in collaborative activities, they often experience impasse, a potentially brief encounter ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICMI '18: Proceedings of the 20th ACM International Conference on Multimodal Interaction

October 2018

687 pages

ISBN:9781450356923

DOI:10.1145/3242969

General Chairs:
Sidney K. D'Mello
University of Illinois, USA
,
Panayiotis (Panos) Georgiou
University of Southern California, USA
,
Stefan Scherer
University of Southern California, USA
,
Program Chairs:
Emily Mower Provost
University of Michigan, USA
,
Mohammad Soleymani
University of Southern California, USA
,
Marcelo Worsley
Northwestern University, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: Specialist Interest Group in Computer-Human Interaction of the ACM

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

ICMI '18

Sponsor:

SIGCHI

ICMI '18: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 16 - 20, 2018

CO, Boulder, USA

Acceptance Rates

ICMI '18 Paper Acceptance Rate 63 of 149 submissions, 42%;

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
699
Total Downloads

Downloads (Last 12 months)87
Downloads (Last 6 weeks)11

Other Metrics

View Author Metrics

Citations

Cited By

Chandler CBreideband TReitman JChitwood MBush JHoward ALeonhart SFoltz PPenuel WD'Mello S(2024)Computational Modeling of Collaborative Discourse to Enable Feedback and Reflection in Middle School ClassroomsProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636917(576-586)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636917
D’Mello SDuran NMichaels AStewart A(2024)Improving collaborative problem-solving skills via automated feedback and scaffolding: a quasi-experimental study with CPSCoach 2.0User Modeling and User-Adapted Interaction10.1007/s11257-023-09387-6Online publication date: 14-Feb-2024
https://doi.org/10.1007/s11257-023-09387-6
Moulder RBooth BAbitino AD'Mello SHilliger IKhosravi HRienties BDawson S(2023)Recurrence Quantification Analysis of Eye Gaze Dynamics During Team CollaborationLAK23: 13th International Learning Analytics and Knowledge Conference10.1145/3576050.3576113(430-440)Online publication date: 13-Mar-2023
https://dl.acm.org/doi/10.1145/3576050.3576113
Lewis AOchoa XQamra RHilliger IKhosravi HRienties BDawson S(2023)Instructor-in-the-Loop Exploratory Analytics to Support Group WorkLAK23: 13th International Learning Analytics and Knowledge Conference10.1145/3576050.3576093(284-292)Online publication date: 13-Mar-2023
https://dl.acm.org/doi/10.1145/3576050.3576093
Patzak AVytasek J(2023)Operationalizing Learning Processes Through Learning AnalyticsApplied Data Science10.1007/978-3-031-29937-7_6(69-81)Online publication date: 10-May-2023
https://doi.org/10.1007/978-3-031-29937-7_6
Pugh SRao AStewart AD'Mello SWise AMartinez-Maldonado RHilliger I(2022)Do Speech-Based Collaboration Analytics Generalize Across Task Contexts?LAK22: 12th International Learning Analytics and Knowledge Conference10.1145/3506860.3506894(208-218)Online publication date: 21-Mar-2022
https://dl.acm.org/doi/10.1145/3506860.3506894
Ma YCelepkolu MBoyer KWise AMartinez-Maldonado RHilliger I(2022)Detecting Impasse During Collaborative Problem Solving with Multimodal Learning AnalyticsLAK22: 12th International Learning Analytics and Knowledge Conference10.1145/3506860.3506865(45-55)Online publication date: 21-Mar-2022
https://dl.acm.org/doi/10.1145/3506860.3506865
Schneider BBryant T(2022)Using Mobile Dual Eye-Tracking to Capture Cycles of Collaboration and Cooperation in Co-located DyadsCognition and Instruction10.1080/07370008.2022.215741842:1(26-55)Online publication date: 23-Dec-2022
https://doi.org/10.1080/07370008.2022.2157418
Praharaj SScheffel MDrachsler HSpecht M(2021)Literature Review on Co-Located Collaboration Modeling Using Multimodal Learning Analytics—Can We Go the Whole Nine Yards?IEEE Transactions on Learning Technologies10.1109/TLT.2021.309776614:3(367-385)Online publication date: 1-Jun-2021
https://doi.org/10.1109/TLT.2021.3097766
Stewart AKeirn ZD’Mello S(2021)Multimodal modeling of collaborative problem-solving facets in triadsUser Modeling and User-Adapted Interaction10.1007/s11257-021-09290-yOnline publication date: 2-Feb-2021
https://doi.org/10.1007/s11257-021-09290-y
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents