-
Who Said What? An Automated Approach to Analyzing Speech in Preschool Classrooms
Authors:
Anchen Sun,
Juan J Londono,
Batya Elbaum,
Luis Estrada,
Roberto Jose Lazo,
Laura Vitale,
Hugo Gonzalez Villasanti,
Riccardo Fusaroli,
Lynn K Perry,
Daniel S Messinger
Abstract:
Young children spend substantial portions of their waking hours in noisy preschool classrooms. In these environments, children's vocal interactions with teachers are critical contributors to their language outcomes, but manually transcribing these interactions is prohibitive. Using audio from child- and teacher-worn recorders, we propose an automated framework that uses open source software both t…
▽ More
Young children spend substantial portions of their waking hours in noisy preschool classrooms. In these environments, children's vocal interactions with teachers are critical contributors to their language outcomes, but manually transcribing these interactions is prohibitive. Using audio from child- and teacher-worn recorders, we propose an automated framework that uses open source software both to classify speakers (ALICE) and to transcribe their utterances (Whisper). We compare results from our framework to those from a human expert for 110 minutes of classroom recordings, including 85 minutes from child-word microphones (n=4 children) and 25 minutes from teacher-worn microphones (n=2 teachers). The overall proportion of agreement, that is, the proportion of correctly classified teacher and child utterances, was .76, with an error-corrected kappa of .50 and a weighted F1 of .76. The word error rate for both teacher and child transcriptions was .15, meaning that 15% of words would need to be deleted, added, or changed to equate the Whisper and expert transcriptions. Moreover, speech features such as the mean length of utterances in words, the proportion of teacher and child utterances that were questions, and the proportion of utterances that were responded to within 2.5 seconds were similar when calculated separately from expert and automated transcriptions. The results suggest substantial progress in analyzing classroom speech that may support children's language development. Future research using natural language processing is under way to improve speaker classification and to analyze results from the application of the automated framework to a larger dataset containing classroom recordings from 13 children and 3 teachers observed on 17 occasions over one year.
△ Less
Submitted 10 April, 2024; v1 submitted 14 January, 2024;
originally announced January 2024.
-
Automated speech- and text-based classification of neuropsychiatric conditions in a multidiagnostic setting
Authors:
Lasse Hansen,
Roberta Rocca,
Arndis Simonsen,
Alberto Parola,
Vibeke Bliksted,
Nicolai Ladegaard,
Dan Bang,
Kristian Tylén,
Ethan Weed,
Søren Dinesen Østergaard,
Riccardo Fusaroli
Abstract:
Speech patterns have been identified as potential diagnostic markers for neuropsychiatric conditions. However, most studies only compare a single clinical group to healthy controls, whereas clinical practice often requires differentiating between multiple potential diagnoses (multiclass settings). To address this, we assembled a dataset of repeated recordings from 420 participants (67 with major d…
▽ More
Speech patterns have been identified as potential diagnostic markers for neuropsychiatric conditions. However, most studies only compare a single clinical group to healthy controls, whereas clinical practice often requires differentiating between multiple potential diagnoses (multiclass settings). To address this, we assembled a dataset of repeated recordings from 420 participants (67 with major depressive disorder, 106 with schizophrenia and 46 with autism, as well as matched controls), and tested the performance of a range of conventional machine learning models and advanced Transformer models on both binary and multiclass classification, based on voice and text features.
While binary models performed comparably to previous research (F1 scores between 0.54-0.75 for autism spectrum disorder, ASD; 0.67-0.92 for major depressive disorder, MDD; and 0.71-0.83 for schizophrenia); when differentiating between multiple diagnostic groups performance decreased markedly (F1 scores between 0.35-0.44 for ASD, 0.57-0.75 for MDD, 0.15-0.66 for schizophrenia, and 0.38-0.52 macro F1). Combining voice and text-based models yielded increased performance, suggesting that they capture complementary diagnostic information.
Our results indicate that models trained on binary classification may learn to rely on markers of generic differences between clinical and non-clinical populations, or markers of clinical features that overlap across conditions, rather than identifying markers specific to individual conditions. We provide recommendations for future research in the field, suggesting increased focus on developing larger transdiagnostic datasets that include more fine-grained clinical features, and that can support the development of models that better capture the complexity of neuropsychiatric conditions and naturalistic diagnostic assessment.
△ Less
Submitted 31 January, 2023; v1 submitted 13 January, 2023;
originally announced January 2023.
-
The Danish Gigaword Project
Authors:
Leon Strømberg-Derczynski,
Manuel R. Ciosici,
Rebekah Baglini,
Morten H. Christiansen,
Jacob Aarup Dalsgaard,
Riccardo Fusaroli,
Peter Juel Henrichsen,
Rasmus Hvingelby,
Andreas Kirkedal,
Alex Speed Kjeldsen,
Claus Ladefoged,
Finn Årup Nielsen,
Malte Lau Petersen,
Jonathan Hvithamar Rystrøm,
Daniel Varab
Abstract:
Danish language technology has been hindered by a lack of broad-coverage corpora at the scale modern NLP prefers. This paper describes the Danish Gigaword Corpus, the result of a focused effort to provide a diverse and freely-available one billion word corpus of Danish text. The Danish Gigaword corpus covers a wide array of time periods, domains, speakers' socio-economic status, and Danish dialect…
▽ More
Danish language technology has been hindered by a lack of broad-coverage corpora at the scale modern NLP prefers. This paper describes the Danish Gigaword Corpus, the result of a focused effort to provide a diverse and freely-available one billion word corpus of Danish text. The Danish Gigaword corpus covers a wide array of time periods, domains, speakers' socio-economic status, and Danish dialects.
△ Less
Submitted 12 May, 2021; v1 submitted 7 May, 2020;
originally announced May 2020.
-
Inferring causality from noisy time series data
Authors:
Dan Mønster,
Riccardo Fusaroli,
Kristian Tylén,
Andreas Roepstorff,
Jacob F. Sherson
Abstract:
Convergent Cross-Mapping (CCM) has shown high potential to perform causal inference in the absence of models. We assess the strengths and weaknesses of the method by varying coupling strength and noise levels in coupled logistic maps. We find that CCM fails to infer accurate coupling strength and even causality direction in synchronized time-series and in the presence of intermediate coupling. We…
▽ More
Convergent Cross-Mapping (CCM) has shown high potential to perform causal inference in the absence of models. We assess the strengths and weaknesses of the method by varying coupling strength and noise levels in coupled logistic maps. We find that CCM fails to infer accurate coupling strength and even causality direction in synchronized time-series and in the presence of intermediate coupling. We find that the presence of noise deterministically reduces the level of cross-mapping fidelity, while the convergence rate exhibits higher levels of robustness. Finally, we propose that controlled noise injections in intermediate-to-strongly coupled systems could enable more accurate causal inferences. Given the inherent noisy nature of real-world systems, our findings enable a more accurate evaluation of CCM applicability and advance suggestions on how to overcome its weaknesses.
△ Less
Submitted 3 March, 2016;
originally announced March 2016.
-
A Heart for Interaction: Shared Physiological Dynamics and Behavioral Coordination in a Collective, Creative Construction Task
Authors:
Riccardo Fusaroli,
Johanne S. Bjørndahl,
Andreas Roepstorff,
Kristian Tylén
Abstract:
Interpersonally shared physiological dynamics are increasingly argued to underlie rapport, empathy and even team performance. Inspired by the model of interpersonal synergy, we critically investigate the presence, temporal development, possible mechanisms and impact of shared interpersonal heart rate dynamics during individual and collective creative LEGO construction tasks. In Study 1 we show how…
▽ More
Interpersonally shared physiological dynamics are increasingly argued to underlie rapport, empathy and even team performance. Inspired by the model of interpersonal synergy, we critically investigate the presence, temporal development, possible mechanisms and impact of shared interpersonal heart rate dynamics during individual and collective creative LEGO construction tasks. In Study 1 we show how shared HR dynamics are driven by a plurality of sources including task constraints and behavioral coordination. Generally, shared HR dynamics are more prevalent in individual trials (involving participants doing the same things) than in collective ones (involving participants taking turns and performing complementary actions). However, when contrasted against virtual pairs, collective trials display more stable shared HR dynamics suggesting that online social interaction plays an important role. Furthermore, in contrast to individual trials, shared HR dynamics are found to increase across collective trials. Study 2 investigates which aspects of social interaction might drive these effects. We show that shared HR dynamics are statistically predicted by interpersonal speech and building coordination. In Study 3, we explore the relation between HR dynamics, behavioral coordination, and self-reported measures of rapport and group competence. While behavioral coordination predicts rapport and group competence, shared HR dynamics do not. Although shared physiological dynamics were reliably observed in our study, our results warrant not to consider HR dynamics a general driving mechanism of social coordination. Behavioral coordination - on the other hand - seems to be more informative of both shared physiological dynamics and collective experience.
△ Less
Submitted 6 December, 2015; v1 submitted 22 April, 2015;
originally announced April 2015.
-
Timescales of Massive Human Entrainment
Authors:
Riccardo Fusaroli,
Marcus Perlman,
Alan Mislove,
Alexandra Paxton,
Teenie Matlock,
Rick Dale
Abstract:
The past two decades have seen an upsurge of interest in the collective behaviors of complex systems composed of many agents entrained to each other and to external events. In this paper, we extend concepts of entrainment to the dynamics of human collective attention. We conducted a detailed investigation of the unfolding of human entrainment - as expressed by the content and patterns of hundreds…
▽ More
The past two decades have seen an upsurge of interest in the collective behaviors of complex systems composed of many agents entrained to each other and to external events. In this paper, we extend concepts of entrainment to the dynamics of human collective attention. We conducted a detailed investigation of the unfolding of human entrainment - as expressed by the content and patterns of hundreds of thousands of messages on Twitter - during the 2012 US presidential debates. By time locking these data sources, we quantify the impact of the unfolding debate on human attention. We show that collective social behavior covaries second-by-second to the interactional dynamics of the debates: A candidate speaking induces rapid increases in mentions of his name on social media and decreases in mentions of the other candidate. Moreover, interruptions by an interlocutor increase the attention received. We also highlight a distinct time scale for the impact of salient moments in the debate: Mentions in social media start within 5-10 seconds after the moment; peak at approximately one minute; and slowly decay in a consistent fashion across well-known events during the debates. Finally, we show that public attention after an initial burst slowly decays through the course of the debates. Thus we demonstrate that large-scale human entrainment may hold across a number of distinct scales, in an exquisitely time-locked fashion. The methods and results pave the way for careful study of the dynamics and mechanisms of large-scale human entrainment.
△ Less
Submitted 11 January, 2015; v1 submitted 29 October, 2014;
originally announced October 2014.
