Gesture and Response in Field-Based Performance Sha Xin Wei Topological Media Lab [12] Graphics, Visualization, and Usability Center Georgia Institute of Technology & Program in Science, Technology, and Society Massachusetts Institute of Technology xinwei@mit.edu Satinder Gill Interaction Design Centre, School of Computing Science, Middlesex University, London, UK & Centre for Music and Science, Cambridge, UK s.gill@mdx.ac.uk spg12@cam.ac.uk responsive media space like the TGarden environment? [9] How can we build environments in which people can become more virtuosic in their performance with continued play? How can people coordinate powerful experiences without appealing to verbal language or to a linguistic representation? In order to sustain such improvisatory but non-random play, TGarden is built explicitly from metaphorical, dense tangible material substrates and fieldbased rather than object-based or agent-based responses to gesture and movement. These material substrates include live, gesturally parameterized projection video, gesturally modified sound, and image-bearing or sensate fabrics. ABSTRACT Ambience and immersive technological environments allow us to explore some basics of human pragmatics that lie beyond linguistics, intentionality and the subject-agency perspectives of human interaction. We focus on gesture and the body in sense-making and propose a discussion drawing on a non-dualist and agent-free account of embodied, material experience. By agent-free we mean an approach that does not presume a monolithic subject. Moreover, we deal with the problem of intersubjectivity by studying the human coordination of activity without appealing to a transmission theory of communication. [6] Body Moves is an analytical method by the second author [4] that deals with the pragmatics of meaning where salient body rhythms span more than one body, and are in relation to each other. In extending this work to the responsive media environment of the TGarden, the relation is not limited to the rhythm of one body with another, but of one body with the salient responsive elements in the environment. Learning to master this responsive space is to be skilled in extending one's own body field. We achieve this by considering how gesture spans multiple bodies and how aesthetic design works with this and facilitates it. The paper is in two parts, the first part covers movement studies, focusing on gesture and body movement, drawing on the acting and pragmatics, and the second part develops this with the example of the TGarden, a responsive play space for experimental performance augmented by gesturally nuanced computational media. Gesture, responsive play. Embodied material experience, body field, material substrate, rhythmic coordination. As people are creating the TGarden, they are very receptive to fields and conscious of their gestures emerging out of the field through rhythm, with precision, and reflexivity. ACM Classification Keywords BODY MOVES - GESTURE (GILL) Author Keywords Body Moves are a form of what we will term metacommunication, which means they serve ‘to instruct about or alter the ongoing communicational process’ [8]. Body Moves are rhythmic configurations between persons, a form of rhythmic synchrony [1]. These rhythmic coordinations shape the engagement space they inhabit, and maintain, form and re-form it. Each Body Move is a composite of rhythm of more than one person. H1. Models and Principles, H5. Information interfaces and Representation (HCI), J. Computer Applications. INTRODUCTION We ask the following questions: how do people collectively and individually improvise meaningful gestures in a highly Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. C&C’05, April 12-15, 2005, London, United Kingdom. Copyright 2005 ACM 1-59593-025-6/05/0004…$5.00. Two kinds of Body Moves, having sequential and parallel structures, have been identified and analysed within the engagement space: sequential Body Moves have the structure of action-reaction motion, whilst Parallel Coordinated Moves have the structure of parallel motion [1, 3, 5]. They have different priorities in their functionality. 205 whether they are acoustic or visual, or perhaps sociopsychological objects. In time you discover people who have invented virtuosic ways of playing and engaging this responsive space, and without a word you are able to learn from their deft action and inaction. As you walk past another body, you leave behind material traces of yourself: shadow, hair, echoes, and air currents. Even if you do not explicitly and actively acknowledge the passerby, your residues intertwine with the other’s and conduct material conversations in your wake. The dynamic physical and symbolic matter of these residues and traces constitute a continuous substrate or field of activity. Sequential moves serve to maintain the steady-state communication, whilst parallel moves serve to transform the communication. There is a pulsation in the movement from sequential to parallel action that facilitates the process of the building of a common ground or sense-making in the interaction environment. Each person has a body field of engagement, and together, the aggregate of their fields, forms the engagement space. This space is therefore also called, the Body Field of Engagement. It is a variable field and alters with the degrees of comfort and discomfort, expressed in our work as ‘contact.’ Within the engagement space, persons cooperate to sustain the space that enables them to remain committed to be together. It necessitates that the membranes of the person’s body fields are in contact, the degree of which alters with levels of commitment and nature of attitude. Overlap or mergence of the fields occurs when bodies move in parallel coordinated action, where the overlap is complete for the period of that action. However, this overlap is only metacommunicatively shared, and does not denote a common focus of attention. In fact, in parallel coordinated action, persons are acting autonomously but simultaneously in rhythmic pulsation on different foci of attention, and in doing so they are aware and attending to each other at the same time [5, 3, 4]. Space is considered as a resonating space. A particular gesture does not always elicit exactly the same sound; it seems as if you are dragging your fingers and limbs across materials like wool or metal sheet or rubber. As your movements couple to the responsive dynamics of the dragged sounds or visual textures, you learn to intentionally "bow" or brush calligraphically through the medium. TGARDEN The TGarden [9, 11] is a responsive media space in which small groups of ordinary people costumed in expressive sensors create and modulate fields of sound and visual texture as they move. The first author proposed TGarden as a space filled thickly with visual and sonic media in which people could improvise gestures that would stir together meaningful and, with practice, even symbolically charged patterns. Over two years, TGarden was built by a consortium of 26 artists and engineers associated with the Sponge and FoAM art research collectives [9], and exhibited in 10 cities. It is one of a series of public experiments in phenomenology of performance, whose context and construction is described in an accompanying paper [7]. Figure 1. Solo epiphany. Ars Electronica, Linz Austria, 2001. Courtesy Sponge. Figure 2. Professional dancers in TG2001, V2 Las Palmas, Rotterdam, 2001. When you walk into a TGarden, you choose from a set of sumptuous garments, each with a different unfamiliarity. Some billow around you in clouds of fabric so that you grow three times larger but no heavier. Some add an odd elasticity to your body so you tend to flop as you walk. Some may rip as you walk, or glue to each other or the walls so you must tear yourself free as you disambiguate your body from ambient matter. Figure 3. Swapping projected wings upon close encounter. In a TGarden, salient rhythms occur within the substrate of the combined activity, indicating particular resonances as body fields move in response to each other We will illustrate how the TGarden creative space works through some examples of activity that span how an individual and a group are coupled with the environment and with objects in the environment. You notice that there are no well-defined objects in the room, but as you play in it over time (minutes or days of repeated visits) you learn certain ways of playing that characteristically elicit more or less well defined entities, 206 ILLUSTRATIVE ANALYSES OF TGARDEN (SHA AND GILL) dynamics, the intertwining between simulated physics and material physics or body physics. Hop-Skip Example Another critical aspect of this responsive system is how it deals with the characteristic time of decay or response. If the characteristic time is too long, the environment begins to feel decoupled from the person and if it is too short the environment responds as a simple discrete series of stimulus-response events. With just the right characteristic time of response, the player imputes a strong sense of elasticity to projected, structured light shining on a hard floor. In the documentary video titled ‘Hop-skip,’ a person is hopping up and down periodically every 8 or 9 beats to sound patterns. The strong beats in the musical textures in this Hop-skip environment elevate the overall excitement in the room, but our question is why and how does the human first begin to hop and skip about the room. At 3 beats after the third hop, the person leaves their position and begins to hop and skip around over the floor space. The analysis of this sudden change in movement helps in understanding the TGarden. During the third hop (21 seconds into the action) there is a white flash on the floor. Just following the hop, the flash re-emerges and moves across under the feet space and shadow of the person. Since the TGarden is engineered with such low latencies as to produce computed media that the human perceives as concurrent with his or her activity, the human interprets the computed response not as a macroscopic interactive reply but as a tangible quality. In the hop-skip example, this tangible quality derives from the micro-physics of the body intertwined with the synthesized dynamics of the visual texture and the rhythmic sound. V2 Professional Dancers Example In the second example, ‘TGarden V2 dancers’, four professional dancers walk into the space, and as they do so and find positions for themselves, the textures and colours on the floor move with them and connect together. The form their positions in relation to each other, coming towards each other in the centre and then working from that (24 seconds). Figure 4. Hop-skip. This is the salient responsive change in the environment that cues in turn the response of the person. Why does the flash re-emerge? The 3D graphics is filled in with a ‘texture map’ and this texture of pale light colour is filled by using 2 rules: a) it is triggered by the person's hop, and b) it is interpolated such that its echo, the echo of the person’s hop, goes on. In other words, the texture map is continuous function of both the internal clock of the machine, as well as the rich real-time data from the human body's ongoing physical movement. We look more carefully at what is happening with the dynamical response. There are two kinds of responses in the TGarden environment to a person's movement. The first is the response of the real-time video synthesis software to the person’s physical movement, as measured by accelerometer on his chest. In the video, for example, a bright texture map fills in as an immediate response to the person's hop. The other is the system's preprogrammed, retraction of the texture map back to a plain, open mesh, interpolated over a fixed time interval. However, there is a recoil, an echo of the white flash. This is not due to a programmed computed response, but is in fact due to the actual physical recoil of the accelerometer on the chest of the human body. Figure 5. Finding their positions in relation to each other. Once positioned, the dancers being to warm up in an improvised rehearsal, sensuously moving with sounds and colours. The shifting shapes on the floor occasionally and very momentarily detach from a dancer who then reaches out and regains contact. As they move in the space the dancers quickly (within 51 seconds) find resonant connections with each other as an engagement field. The filling in of the mesh by texture comes from prior logic, but the echo of the flash – that recoil - comes from the recoil of the physical body. Hence the responsivity in the TGarden arises from both software dynamics and body 207 Figure 6. Notice the triangular space holding the attention of their body fields (55 seconds). Figure 8. Slo-mo. Converging on balls. As they do so, there is a change in their rhythmic coordination. The dynamics and tempo of their field space shift from a smoothly coordinated rhythm to a seemingly staccato random tempo, affected by their individual movement with the balls and the physical contact between bodies that comes with rolling the balls to each other. The rhythm alters again as they disperse and their body fields engage in smoother coordinated autonomous choreographies. In analyzing this, one could produce an agent-based description of a sequence of actions, for example, a) the person with the tiger striped costume initiated the movement of the balls in the center of the play space, and this entrained the others to do likewise. Alternatively, and as we do, one could produce a fieldbased analysis where the overall movement in the space and the relations of engagement undergoes change. In this example, the "tiger stripes person …" description is an egobased and object-oriented account, whilst the latter description of rhythm and tempo is an account of the entire scene in terms of rhythmic fields. By the end of their performance, they are fully rhythmically coordinated. This is enabled through grounding their coordinations with the environment and each other during discovery and improvisation. Figure 7. Two of the dancers sway together in aligned synchrony (2 min 11 sec). In analyzing how they coordinate or attending to each other, it is not essential to know their gaze activity, which is subsumed as background knowledge to their body movement. In the TGarden, gaze is not a core part of ‘attention’. In much cognitive work, visual attention or gaze is often used as an explicit indicator of attention. However, to correlate gaze with cognition and attention is to reduce the connectivity in the field space of the TGarden. Our analysis focuses body movements as fields, i.e. of the whole body and thereby avoids reduction to solely geometric data and visual perception. TGARDEN: SUBSTRATE AND CONCURRENCY (SHA) Within the environment we are always permeated, coincident with fields of physical and symbolic material. A core concept for the TGarden is the ‘substrate’. In the Slomo example above, the agent-based description of activity is that of individual beings and it takes the form of a graph. However, ‘substrate’ is a way of looking at the entire room as a continuous distribution of, for example, sound, light, fabrics, costumes and bodies, and more abstractly, gestures, and fields of speech or attention. Considering the changes in the distribution over time of fields is a dynamic approach that lends itself naturally to notions such as waves and rhythms. By ‘substrate’ we mean the union of all these continuous, time-varying distributions. Slo-mo Example The ‘Slo-mo’ video illustrates body movement fields. About 41 seconds into the action, there is a scene where four dancers converge around the centre of the play space and move with four large balls. Concurrency Concurrency is a crucial aspect of the TGarden's field-based computationally mediated experience. It substantially 208 players in a TGarden form tacit awareness in overlapping and autonomous space and gauge elements and patterns of connectivity, and through this tacit learning, shape the media space and are concurrently shaped by it. differs from the standard graph-theoretic model of agent-toagent information-passing and causality, which is an analytic framework whereby humans and devices are seen to operate in sequential chains. In the TGarden, however, all the environmental processes evolve in semi-autonomy coupled by relatively long-time (O(100) sec) state information and relatively short-time (O(10) msec) time series derived from sensor data. In any case, the visual and sound processes are engineered under the requirement that they compute their responses to new sensor data within the threshold of human perception of concurrency: sensor data and sounds and visual imagery are computed fast enough to appear phenomenologically concurrent with human gestures. This concurrency enables people to become coupled with the room and with one another. In a sufficiently small space the concurrency sustains an embodied sense of co-present experience rather than a combinatorially complex game of atomic agents. REFERENCES 1. Birdwhistle, R.L. Kinesics and Context, University of Pennsylvania, 1970. 2. Gill, S.P., Kawamori, M, Katagiri, Y, Shimojima, A. The Role of Body Moves in Dialogue. International Journal for Language and Communication (RASK), 12 (2000), 89-114. 3. Gill, S.P. The Engagement Space and Parallel Coordinated Movement: Case of a Conceptual Drawing Task. CKIR (Centre for Knowledge and Innovation Research) Working Paper series, CKIR-1, (2002) (ISBN 951-791-660-4). 4. Gill, S.P. Body Moves and Tacit Knowing. In Gorayska, B. and Mey, J. (Eds.) Cognition and Technology. Amsterdam: John Benjamin, 2004, 241-265. CONCLUSIONS AND REFLECTIONS The TGarden has been reflexively constructed by creators who are expert students of the interactive and responsive strategies operating in the human performers and the computational media systems. In the design, the creators explicitly designed the environment to allow gesturing bodies to emerge from and dissolve into various kinds of fields: musical sonic textures, choreographical bodies, kinetic visual textures, and fabrics. The construction carries on a phenomenological research agenda that explicitly informs the design of the responsive space and includes the following questions: How is agency diffused? What is meaning-making movement? What is individual and group agency? And what is nature of continuous materiality. These questions are addressed by ‘concurrent activity’ that is made possible through conceiving of the responsive space as ‘material substrate’, whereby the entire room is bathed in sound. This research agenda is described in detail in [7]. 5. Gill, S.P. and Borchers. Knowledge in Co-Action: Social Intelligence in Collaborative, Design Activity, AI & Society, 17.3 (2003), 322-339. 6. Harris, R., Signs of Writing, Routledge, London, 1997. 7. Salter, C., and Sha, X.W. Sponge: A Case Study in Practice-based Collaborative Art Research, Creativity and Cognition, submitted (2005). 8. Scheflen, A.E. How Behaviour Means. Exploring the Contexts of Speech and Meaning: Kinesics, Posture, Interaction, Setting, and Culture. New York: Anchor, Press/Doubleday, 1974. 9. Sponge. TGarden Project: TG2001, http://sponge.org/projects/m3_tg_intro.html 10. Sha, X.W. Resistance Is Fertile: Gesture and Agency in the Field of Responsive Media, in Makeover: Writing the Body into the Posthuman Technoscape, Configurations, 10.3, Summer (2002), 439-472. The TGarden as constructed does not interpret movement presuming intentionality or a model of the ego subject. The salient rhythms are essentially resonances of spontaneous actions and non-symbolic, providing an example of an ‘alinguistic semiology of human performance’ [10]. We extend the concept of the Body Move based on its essential fields of resonant performance to movement based on fields instead of particular human bodies. We consider how the 11. Sha, X.W. The TGarden As a Phenomenological Experiment, (in preparation). 12. Sha, X.W. Topological Media Lab, http://topologicalmedia.net . 209