Body Rhythmic Entrainment and Pragmatics in Musical and Linguistic Improvisation Tasks Satinder Gill*1, Marc R. Thompson#2, Tommi Himberg #3 * # Centre for Music and Science, Faculty of Music, University of Cambridge, UK Finnish Centre of Excellence in Interdisciplinary Music Research, Department of Music, University of Jyväskylä, Finland 1 spg12@cam.ac.uk, 2marc.thompson@jyu.fi, 3tommi.himberg@gmail.com ABSTRACT Background This interdisciplinary study combines researchers and methods from linguistic communication (Gill, 2007), as well as music and movement (Himberg & Thompson, 2011). We consider conversation as performance, and improvisation in music as akin to this performance. Improvisation, musical or linguistic, involves rules/conventions, but the interactive performance will often unfold in unpredictable ways, involving heightened moments of rhythmic and empathic connection (salient rhythmic moments, SRM), and require synchrony. We investigate the interpersonal entrainment and the properties of rhythm in the performance of music and language to discern what properties are shared and what differs. Aims Our aims are twofold: 1) to combine qualitative observational analysis and quantitative movement analysis to identify SRM in the data, and to describe kinematics related to the SRM’s 2) to compare periodicity and entrainment of body movements across different conditions (participants facing each other vs. not facing; music making vs. story telling). Method 8 pairs of participants performed musical and linguistic improvisations (2 min) while audio & video and movement recordings were made. In the trials, participants were instructed to co-create a musical improvisation using shakers, or co-create a story. As an inspiration, they were presented soundscapes and slideshows of either urban and social environments (restaurants, city streets) or natural environments (forest, beach). It was emphasised that the stimuli were for inspiration only, the participants were free to create whatever kind of music or stories they liked. (footnote: this instruction seems to have worked, as most stories had only tangential relationships with the stimuli.) Observation analysis of the videos to identify SRM’s and kinematic and statistical analyses of the motion capture data (Toiviainen & Burger, 2010) were performed. The analyses focused on the body sway (inspired by Shockley et al., 2003) (anterior-posterior movement of the centre of mass and chest), movements of the head and hands. The observation analysis method has involved the use of ELAN, a gesture and interaction analysis tool, in the analysis 347 of motion data, which in this study includes video and motion capture animation. The aim is to identify salient moments of rhythmic synchrony between the bodies of the participants. This analysis builds on earlier work by Gill on Body Moves (BM) (Gill, 2007), which identified salient rhythmic moments in interaction that both maintain the information flow and help transform it, facilitating understanding. These BMs were originally conceived as collective acts across the bodies and speech of the participants, although primarily identified by the collective action of their bodies. In this study, the analysis has been extended from the linguistic context to enable the identification of salient rhythmic moments in music and in speech. The analysis involves shifting between the video and the motion capture animations, with insights from one data form providing clarity about movement patterns in the other. Motion capture animation reduces the richness in the environment and draws attention to patterns that can be checked to discern their significance in the video. Video on the other hand, allows you to see rich movement, and when noticing an SRM, one can move to the motion capture to check its structure for clarity. In order to investigate movement patterns across the various conditions (music vs. speech, facing vs. non-facing), feature extraction and Principal Components Analysis (PCA) was applied to the raw movement data. The aim of using PCA was to reduce the dimensionality of the feature data and expose how disparate movements features could be grouped together according to their amount of variance. In addition, it presented an objective way of selecting which parts of the body would be used for subsequent analysis as it was assumed that parts of the body with high variance would presumably have a higher chance of producing Salient Rhythmic Movements. Cross-recurrence analysis (CRA) (Marwan et al. 2007) is a way of detecting entrainment and structure in noisy data. It involves the reconstruction of the phase space of the processes, and then an analysis to detect when the processes come near each other in the space. The points of recurrence can be plotted visually, but there are also methods to quantify essential features of the plot, such as the overall rate of recurrence and determinism, the proportion of recurrence that occurs in longer, diagonal segments (which are indicative of strong entrainment of the processes or influence from one process to the other). Results Analysis is currently ongoing. Preliminary findings include: • SRM’s in the linguistic trial correlate with moments of less distance between the bodies, indicating increased contact. • SRM’s in the music trial correlate with less distance, but only when the interaction was mutually cooperative. In general, there were more disruptions and moments of non-cooperative behaviour in the music trials. • in contrast, there are no major differences in the periodicity of movements between the linguistic and musical trials, suggesting the two systems share rhythmic properties at the relational level of communication. • in music trials, facing each other or not influences the balance between autonomous and cooperative behaviour of the participants; the necessity to cooperate is greater when facing each other whereas in the non-face-to-face condition the participants were more likely to spontaneously coordinate their gestures. • based on the factor loadings of the PCA, the hand movements and the anterior-posterior body sway seem to be the main contributors to the variance of the movement data, and these body parts are flagged for more detailed analysis. • preliminary cross-recurrence analysis suggests that in the music tasks, body sway entrainment does not seem to be influenced by whether participants can see each other or not. Conclusions The observational analysis and the kinematic and entrainment analyses formed a complementary set of methods that helped identify similarities and differences in the interaction in musical and linguistic improvisations. Music and language as communicative performance appear very likely to share properties of body rhythmic interpersonal synchrony. Keywords movement, entrainment, communication, coordination, common ground, joint action REFERENCES Gill, S. (2007). Entrainment and Musicality in the Human System Interface. In AI&Society, 21 (4), 567-605. Himberg, T. & Thompson, M.R. (2011). Learning and synchronising dance movements in South African songs: Cross-cultural motion-capture study. Dance Research, 29 (2), 303-326. Marwan, N., Romano, M. C., Thiel, M., and Kurths, J. (2007). Recurrence plots for the analysis of complex systems. Physics Reports, 438(5-6), 237 – 329. Shockley, K., Santana, M.-V., and Fowler, C. A. (2003). Mutual interpersonal postural constraints are involved in cooperative conversation. Journal of Experimental Psychology-Human Perception and Performance, 29(2), 326–332. Toiviainen, P., & Burger, B. (2010). MoCap Toolbox Manual. University of Jyväskylä: Jyväskylä, Finland. Available at http://www.jyu.fi/music/coe/materials/mocaptoolbox/MCTmanu al. 348