Almost Human: Moving Expressive Gesture from Cello to Spine Ian Hattwick Seth Woods Marcelo M. Wanderley Center for Interdisciplinary Research in Music Media and Technology Input Devices and Music Interaction Lab McGill University ian.hattwick@mail.mcgill.ca Center for Research in New Music University of Huddersfield u1269816@mail.hud.ac.uk Center for Interdisciplinary Research in Music Media and Technology Input Devices and Music Interaction Lab McGill University marcelo.wanderley@mcgill.ca ABSTRACT Developing approaches to the creation of expert performance practice on new DMIs is a longstanding research challenge. We present an approach which leverages an expert performer’s embodied knowledge of expressive performance gesture on an existing instrument. This knowledge forms the basis for the creation of a performance practice of expressive gestures an a new DMI. The research process took place in five stages: observation of musical performance on an existing instrument, analysis of the observed expressive gestures, transformation of these gestures for the new instrument, creation of an artistic work utilizing these gestures, and ultimately the reproduction of these gestures in performance. The results of this research may prove useful to performers seeking to adopt a new DMI into their performance practice, instrument designers seeking to characterize expressive gestural affordances for new DMIs, and researchers seeking to understand the role of personal performance practice in the manifestation of expressivity. transform it for use on a new instrument. In particular, we were interested in expressive gestures which were distinctive not just to the original instrument but also to the performer’s individual performance practice. Can this embodied knowledge of expressivity be transferred even though the gestural affordances of the instruments may be significantly different? This research brought together an expert performer on cello, who also has experience in dance performance and research on gestural performance, with an instrument designer and composer interested in gestural affordances of DMIs. The desired outcome was the creation of an artistic work composed by the first author and performed by the second author. This work consists of two parts. The first part consists of Woods performing on cello while wearing the Spine, a new DMI described below. The second part consists of Woods performing choreographed movements while wearing the Spine. In the first section, the Spine mostly serves to control transformations of the acoustic cello sound, while in the second part the Spine directly controls sound synthesis. Keywords Expressive Gesture, Practice-Based Research, Gestural Analysis, Prosthetic Instrument 1. INTRODUCTION For the performer, establishing a performance practice with a new digital musical instrument (DMI) can be a challenging task. The goal of this task can described variously as attaining virtuosity [11] or performing with expressivity [2]. One approach to this challenge has been recruiting performers with a previous experience of expert performance on an acoustic instrument [3, 5]. Even given such performance experience, it is often recognized that a certain longevity of performance practice on an instrument is necessary in order for the development of a mature performance practice [2, 3]. In this paper we present an example of practice-led research which presents a variation to this approach. In our approach, we seek to leverage an expert performer’s performance practice on an existing instrument in order to utilize their embodied knowledge of expressive performance and Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NIME’14, June 30 – July 03, 2014, Goldsmiths, University of London, UK. Copyright remains with the author(s). Figure 1: Woods rehearses while wearing the Spine. 1.1 Background The primary researchers for this project consisted of instrument designer and composer Ian Hattwick and cellist and researcher Seth Woods. Hattwick co-created the family of Prosthetic Instruments [9, 6] and is interested in ways in which an instruments’ gestural affordances can facilitate ensemble interaction in performance. Woods as a performer specializes in contemporary classical and avant-garde music. His research lies within the area of physical and chore- ographic analysis of instrumentalists and wearable technology performance. The DMI used for this research is the Spine, one of the family of prosthetic instruments developed by Joe Malloch and Ian Hattwick. The development of these instruments took place as part of a project titled Les Gestes: une nouvelle génération des instruments de musique numérique pour le contrôle de la synthèse et le traitement de la musique en performance par les musiciens et les danseurs 1 , a collaborative research project directed by Sean Ferguson and Marcelo Wanderley at McGill University and choreographer Isabelle van Grimde from the Montreal-based dance company Van Grimde Corps Secrets 2 . Briefly, the Spine (seen in figure 1) is a digital musical instrument designed to be worn by dancers in an interactive dance performance. It consists of a flexible truss-like structure attached at two locations on the dancer’s body, behind the head and at the base of their back. It is equipped with two inertial measurement units, each of which provides its absolute orientation. Together they provide information regarding the Spine’s deformation in three axes. For a more detailed description see Malloch (2014)[9]. 2. RESEARCH PROCESS In this section we will first present an overall description of the research process and describe a sample of our analysis approach before going into detail regarding certain stages in order to facilitate further discussion. 2.1 as force, contraction, flux and flow, which originate from Rudolf von Laban’s theories on Choreutics and Eukinetics [8, 7]. The third was an even higher level of analysis which attempts to characterize the movement by the expressivity inherent in the gesture, as experienced by the performer or communicated visually. Following this analysis we began the process of using these gestures as well as their transformations for the creation of the artistic work. This process took place in three stages. The first was the creation of a section for cello and live electronics controlled by the Spine. Our goal for this section was not to provide a demonstration of the gestures we analyzed, but rather to present the expressive vocabulary of Wood’s personal performance practice. Nonetheless, we hoped that the music written for this section would facilitate the manifestation of our chosen gestures, and made reasonable efforts towards that end. The second stage of the artistic creation was the creation of the choreography. At this point we thought carefully about ways in which our analyses could facilitate the transformation of the expressive gestures into whole-body movement. The final stage was the creation of mappings using the data produced by the Spine both while Woods performs with the cello as well as when he solely performs the choreographed section. Overview The first step of our research process was a discussion of our overall goals in order to identify the perspectives we each brought to the collaboration. For Woods this consisted of his previous research using Laban notation to characterize instrumental gesture on the cello, as well as his experience as a performer. For Hattwick this consisted of his previous work with dancers both as a composer and also during the creation of the Prosthetic Instruments. Following this discussion we closely viewed three videorecordings, as well as one recording made using a Qualisys motion capture system, of Woods performing compositions for solo cello. These pieces varied from traditional repertoire, one of J.S. Bach’s cello suites, to recent compositions commissioned by Woods. Our goal was to identify specific moments of gestural expressivity that reflected Woods’ personal performance practice. Once we had selected nine specific moments we began to analyze them for the gestures’ physical components as well as their expressive characteristics. Since our goal was to transform and transfer these gestures to a new instrument, we were interested not only in their physical characteristics, but also by the performer’s subjective experience of the gesture as well as the visual manifestation as perceived by external observers. The analysis took place through closely watching the selected videos, sketching the important motions (see figure 2), discussing our individual perspectives on the gesture’s expressive characteristics, and collective brainstorming. The results of this analysis were on three levels. The first was a kinaesthetic analysis which looked at where the gesture originates as well as the movements which occur during the course of the gesture. The second is a higher-level analysis of types of movement, drawn largely from concepts such 1 “Gestures: a new generation of digital musical instruments for controlling synthesis and processing of live music by musicians and dancers.” 2 www.vangrimdecorpssecrets.com/) Figure 2: Sketch identifying key aspects of the gesture we labelled ‘holding extension’. 2.2 Gestural Analyses The most crucial stage of this research was the analyses of our chosen gestures for their expressive content. We were interested in exploring as many modalities of expression inherent within these gestures as possible. To this end we considered not only the physical manifestation of the gesture and its visual encodings of expressivity, but also Woods’ embodied experience of performance and the observer’s sympathetic perception of the performer’s experience. In this latter respect we were much influenced by Godøy’s theory of motor-mimesis. Briefly, Godøy argues that we “mentally imitate sound-producing actions when we actively listen to music” [4]. We also observe that when observing someone performing a gesture we mentally conceptualize the kinaesthetic feeling of performing that gesture ourselves. In reference to our example analysis described below, that can involve a contradiction wherein the actual effort involved in performing the gesture is much greater than the perceived effort. After analyzing all of the gestures we were able to further characterize them into three categories: breath and embrace, trajectory, and extension. 2.3 An Example Analysis 3.1.1 Priorities in the Creation Process As an example we will describe in more detail the sixth gesture we analyzed, taken from a performance by Woods of the composition PNOM, composed in 1995 by Claudio Gabriele.3 We refer to this gesture as ‘Holding Extension’. Our first analysis of the gesture consisted of its physical manifestation. Part of this included making a rough sketch of the gesture highlighting important elements, shown in figure 2. At this point we identified as important and interesting the way the right elbow raises and the forearm rotates when increasing bowing intensity at 5:06 and the extension of the left elbow and rotation of the torso at 0:07. In our discussions we identified several important concepts drawn from Wood’s embodied experience of performing this gesture. The first is the difficulty of holding the arm in extension, both during moments of low intensity as well as high intensity. In addition Woods noted the difficulty of applying enough bow pressure to get the string to speak while still keeping maintaining a light timbre. We identified this as both a balance and a contradiction – balancing between not enough and too much pressure, as well as the contradiction of exerting great force to control small motions. We also identified the role of the subtle use of rotation to draw a little more intensity out of the gesture. In considering transformations of this gesture it was easy to imagine whole body extensions rather than arm extensions. In addition, the gesture strongly suggested mappings which would take advantage of the small movements made while holding this kind of extension. 2.4 Gestural Transformations We considered that there were three rough approaches to transferring these gestures both to a new composition for cello as well as to a new instrument. The first was the literal recreation of the gesture as closely as possible. The second was a creation of a new gesture in which the physical characteristics of the original gesture were moved to a new body part or performance technique. The third was the creation of a gesture which recreates the essence of the original gesture while taking a wholly different physical form. This last is the most difficult to conceptualize but may be expressed as an energy flow or as an embodied experience of performance. For all of these approaches what is important are the ways in which the gesture is recognizable both to the performer as well as to spectators. 3. DISCUSSION In this section we discuss observations and discoveries we found interesting and meaningful. 3.1 Research Context We cannot overemphasize the fact that this research took place during the creation of an artistic work and with an instrument that is still under development. These factors greatly influenced the ways in which our research took shape. Perhaps the most difficult aspect was the time pressure they created. During our early discussions the idea of the methodical analysis of Woods’ gestures and their subsequent transformation greatly appealed to us. However, it took great effort to resist rushing through the observation and analysis stages, especially once it became apparent that they would take more time than we had allotted for them. 3 Video of this recording is available https://www.youtube.com/watch?v=8gpI51fFvZk, cessed May 10, 2014. at ac- A constant tension during our research was balancing our desired research outcomes with the creation of a successful work. One way in which this manifested itself was in the priorities inherent in the work itself. As an artistic piece, was it more important that it be visually striking, aurally appealing, conceptually strong, well-composed, or an effective demonstration of the technology and research process? We strongly felt that over-emphasizing the latter would be greatly detrimental. However we remained strongly in favor of sticking with our initial intentions of creating a work which grows out of the performer’s gestures and our chosen research process. We were keenly aware of the temptation felt by creators of live electronic works to create fixed electronic elements on top of which the live elements are ornamental. In our initial discussions we considered including a third collaborator to write the actual composition. However, our concern was that a composer, brought into the project for that purpose, would favor the sonic outcome over the conceptual basis of the work. In the end, we decided that it would be better for all of the collaborators to share the same commitment and focus on performer gesture. We also considered the problem that, if expressive gestures are spontaneous and embodied in performance, it might not be possible to create a piece in which our chosen gestures come out spontaneously and without “forcing” them. We decided that indicating the intended expressive gesture in the written score would help direct the practice of rehearsal and suggest the interpretation of the piece. The clear labeling of the gestures as described below was a great help in this process. 3.2 Observations Many of our most interesting observations came through our experience and discussions during the research process. 3.2.1 Labeling Gestures Maintaining clarity of the gestures proved to be difficult, both in the creation of the written scores and in the act of performance. Clearly labeling them proved to be a helpful way of crystallizing their essence, and referring to these labels during the compositional and rehearsal process helped to preserve their expressive intent. Once a gesture was placed into a sequence with other gestures, it frequently occurred that we had a difficult time maintaining the essence of the gesture and not just its form. This was particularly true for gestures whose physical manifestation differed greatly from the initial gesture. We also found it helpful to use clear labels during our discussions. During our analyses we described each gesture in many different ways, and we found that we would often gravitate towards different phrasings describing the same gesture. This was particularly problematic when discussing two similar gestures. 3.2.2 Transformations through Process During the creation of the work it sometimes happened that the transformation would take place through embodied knowledge, in the body rather than conceptually. One example of this occurred when, while creating the choreography, Woods found a particular movement within the gesture we labelled ‘impulse’, and worked at trying to find a way of transferring the embodied experience of the gesture from the cello into a form of dance movement. In this case the indication of success was as much through his experience of the transformed gesture as it was through any outside observation. In this case the conceptual transfor- mation might not be visible to the outside observer but will still inform the artistic result. 3.2.3 Performative Attention This work deals heavily with the ongoing challenge of dancers using DMIs in performance, an ongoing research question [10], with the additional complication of how a musician is supposed to incorporate into their performance practice gestures which blur the line between instrumental gesture and choreography. In particular, since our work is so heavily focused on the performance of specific gestures, there are many times during the choreography where the performer’s entire attention is on the gesture’s physical manifestation. What this means is that there is often little ability for him to focus on the sonic outcome of his movements. We consider that the continuum of performative attention for a dancer performing with a DMI lies between focusing on the one hand on their kinaesthetic experience with no attention given to the sonic results, and on the other hand on performing gestures with an intention of creating a specific sonic result.4 Broadly speaking, there will be times when the performer is entirely focused on their movements, and times when they will be able to specifically focus on the sonic outcome of their movements. In the piece we created, we found that large, whole-body gestures tend to fall into the first category, while gestures such as ‘holding extension’ allow for the performer’s attention to move towards their control of the sound. This presents a huge challenge in mapping, as it is necessary both for the sonic results of the performer’s actions to be appropriate when they are making large movements and not focused on sound production, while still giving them the affordance for nuanced control of sound using small gestures when appropriate. 4. CONCLUSION The primary research question which drove the research described here is whether it is possible for an expert performer on one instrument to transfer some element of their embodied knowledge of expressive gesture onto performance using a new instrument. Can this embodied knowledge of expressivity be transferred even though the instrumental gestures between the instruments are differ significantly? And further, is it possible for this practice to facilitate performance on a new DMI, one which does not already have an established performance practice? In our experience, our research has demonstrated that the analysis of a performer’s expressive gestures as described above can be used successfully as the basis for the creation of an artistic work. However, that is quite different than a performance practice per se. Our goal is for the creation of a series of works for Woods performing on the Spine in order to help clarify the actual performance practice which he creates over time, and to help identify idiomatic performance practice on the Spine in general. 5. REFERENCES [1] C. Cadoz, M. M. Wanderley, and I. C. Pompidou. Gesture - Music. In M. M. Wanderley and M. Battier, editors, Trends in Gestural Control of Music, pages 28–65. Ircam, Paris, 2000. [2] C. Dobrian and D. Koppelman. The ”E” in NIME : Musical Expression with New Computer Interfaces. In Proceedings of the 2006 conference on New interfaces 4 This later approach can be termed instrumental gesture, as defined in Cadoz and Wanderley (2000) [1]. [3] [4] [5] [6] [7] [8] [9] [10] [11] for musical expression, pages 277–282, Paris, France, 2006. IRCAM - Centre Pompidou in collaboration with Sorbonne University. S. Ferguson and M. Wanderley. The McGill Digital Orchestra: An Interdisciplinary Project on Digital Musical Instruments. Journal of Interdisciplinary Music Studies, 4(2):17–35, 2010. R. I. Godø y. Motor-Mimetic Music Cognition. Leonardo Music Journal, 36(4):317–319, 2003. I. Hattwick, P. Beebe, Z. Hale, M. M. Wanderley, P. Leroux, and F. Marandola. Unsounding Objects: Audio Feature Extraction for Control of Sound Synthesis in a Digital Percussion Instrument. Technical report, 2013. I. Hattwick, J. Malloch, and M. M. Wanderley. Forming Shapes to Bodies: Design for Manufacturing in the Prosthetic Instruments. In Proceedings of Conference on New Interfaces for Musical Expression, 2014. R. Laban. Modern Educational Dance. MacDonald & Evans Ltd., London, 1963. R. Laban and F. C. Lawrence. Effort. MacDonald & Evans Ltd., London, 1947. J. Malloch. A Framework and Tools for Mapping of Digital Musical Instruments. Phd thesis, McGill University, 2014. W. Siegel. Dancing the Music: Interactive Dance and Music. In R. T. Dean, editor, The Oxford Handbook of Computer Music, pages 191–213. Oxford University Press, Oxford, 2009. D. Wessel and M. Wright. Problems and Prospects for Intimate Musical Control of Computers. Computer Music Journal, 26(3):11–22, 2002.