Gesture and Response in Field-Based Performance
Sha Xin Wei
Topological Media Lab [12]
Graphics, Visualization, and Usability Center
Georgia Institute of Technology
&
Program in Science, Technology, and Society
Massachusetts Institute of Technology
xinwei@mit.edu
Satinder Gill
Interaction Design Centre,
School of Computing Science,
Middlesex University, London, UK
&
Centre for Music and Science, Cambridge, UK
s.gill@mdx.ac.uk
spg12@cam.ac.uk
responsive media space like the TGarden environment? [9]
How can we build environments in which people can
become more virtuosic in their performance with continued
play? How can people coordinate powerful experiences
without appealing to verbal language or to a linguistic
representation? In order to sustain such improvisatory but
non-random play, TGarden is built explicitly from
metaphorical, dense tangible material substrates and fieldbased rather than object-based or agent-based responses to
gesture and movement. These material substrates include
live, gesturally parameterized projection video, gesturally
modified sound, and image-bearing or sensate fabrics.
ABSTRACT
Ambience and immersive technological environments allow
us to explore some basics of human pragmatics that lie
beyond linguistics, intentionality and the subject-agency
perspectives of human interaction. We focus on gesture and
the body in sense-making and propose a discussion drawing
on a non-dualist and agent-free account of embodied,
material experience. By agent-free we mean an approach
that does not presume a monolithic subject. Moreover, we
deal with the problem of intersubjectivity by studying the
human coordination of activity without appealing to a
transmission theory of communication. [6]
Body Moves is an analytical method by the second author
[4] that deals with the pragmatics of meaning where salient
body rhythms span more than one body, and are in relation
to each other. In extending this work to the responsive
media environment of the TGarden, the relation is not
limited to the rhythm of one body with another, but of one
body with the salient responsive elements in the
environment. Learning to master this responsive space is to
be skilled in extending one's own body field.
We achieve this by considering how gesture spans multiple
bodies and how aesthetic design works with this and
facilitates it. The paper is in two parts, the first part covers
movement studies, focusing on gesture and body
movement, drawing on the acting and pragmatics, and the
second part develops this with the example of the TGarden,
a responsive play space for experimental performance
augmented by gesturally nuanced computational media.
Gesture, responsive play. Embodied material experience,
body field, material substrate, rhythmic coordination.
As people are creating the TGarden, they are very receptive
to fields and conscious of their gestures emerging out of the
field through rhythm, with precision, and reflexivity.
ACM Classification Keywords
BODY MOVES - GESTURE (GILL)
Author Keywords
Body Moves are a form of what we will term metacommunication, which means they serve ‘to instruct about
or alter the ongoing communicational process’ [8]. Body
Moves are rhythmic configurations between persons, a form
of rhythmic synchrony [1]. These rhythmic coordinations
shape the engagement space they inhabit, and maintain,
form and re-form it. Each Body Move is a composite of
rhythm of more than one person.
H1. Models and Principles, H5. Information interfaces and
Representation (HCI), J. Computer Applications.
INTRODUCTION
We ask the following questions: how do people collectively
and individually improvise meaningful gestures in a highly
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
C&C’05, April 12-15, 2005, London, United Kingdom.
Copyright 2005 ACM 1-59593-025-6/05/0004…$5.00.
Two kinds of Body Moves, having sequential and parallel
structures, have been identified and analysed within the
engagement space: sequential Body Moves have the
structure of action-reaction motion, whilst Parallel
Coordinated Moves have the structure of parallel motion [1,
3, 5]. They have different priorities in their functionality.
205
whether they are acoustic or visual, or perhaps sociopsychological objects. In time you discover people who
have invented virtuosic ways of playing and engaging this
responsive space, and without a word you are able to learn
from their deft action and inaction. As you walk past
another body, you leave behind material traces of yourself:
shadow, hair, echoes, and air currents. Even if you do not
explicitly and actively acknowledge the passerby, your
residues intertwine with the other’s and conduct material
conversations in your wake. The dynamic physical and
symbolic matter of these residues and traces constitute a
continuous substrate or field of activity.
Sequential moves serve to maintain the steady-state
communication, whilst parallel moves serve to transform
the communication. There is a pulsation in the movement
from sequential to parallel action that facilitates the process
of the building of a common ground or sense-making in the
interaction environment. Each person has a body field of
engagement, and together, the aggregate of their fields,
forms the engagement space. This space is therefore also
called, the Body Field of Engagement. It is a variable field
and alters with the degrees of comfort and discomfort,
expressed in our work as ‘contact.’
Within the engagement space, persons cooperate to sustain
the space that enables them to remain committed to be
together. It necessitates that the membranes of the person’s
body fields are in contact, the degree of which alters with
levels of commitment and nature of attitude. Overlap or
mergence of the fields occurs when bodies move in parallel
coordinated action, where the overlap is complete for the
period of that action. However, this overlap is only metacommunicatively shared, and does not denote a common
focus of attention. In fact, in parallel coordinated action,
persons are acting autonomously but simultaneously in
rhythmic pulsation on different foci of attention, and in
doing so they are aware and attending to each other at the
same time [5, 3, 4]. Space is considered as a resonating
space.
A particular gesture does not always elicit exactly the same
sound; it seems as if you are dragging your fingers and
limbs across materials like wool or metal sheet or rubber.
As your movements couple to the responsive dynamics of
the dragged sounds or visual textures, you learn to
intentionally "bow" or brush calligraphically through the
medium.
TGARDEN
The TGarden [9, 11] is a responsive media space in which
small groups of ordinary people costumed in expressive
sensors create and modulate fields of sound and visual
texture as they move. The first author proposed TGarden as
a space filled thickly with visual and sonic media in which
people could improvise gestures that would stir together
meaningful and, with practice, even symbolically charged
patterns. Over two years, TGarden was built by a
consortium of 26 artists and engineers associated with the
Sponge and FoAM art research collectives [9], and
exhibited in 10 cities. It is one of a series of public
experiments in phenomenology of performance, whose
context and construction is described in an accompanying
paper [7].
Figure 1. Solo epiphany. Ars Electronica, Linz Austria, 2001.
Courtesy Sponge.
Figure 2. Professional dancers in TG2001, V2 Las Palmas,
Rotterdam, 2001.
When you walk into a TGarden, you choose from a set of
sumptuous garments, each with a different unfamiliarity.
Some billow around you in clouds of fabric so that you
grow three times larger but no heavier. Some add an odd
elasticity to your body so you tend to flop as you walk.
Some may rip as you walk, or glue to each other or the
walls so you must tear yourself free as you disambiguate
your body from ambient matter.
Figure 3. Swapping projected wings upon close encounter.
In a TGarden, salient rhythms occur within the substrate of
the combined activity, indicating particular resonances as
body fields move in response to each other We will
illustrate how the TGarden creative space works through
some examples of activity that span how an individual and
a group are coupled with the environment and with objects
in the environment.
You notice that there are no well-defined objects in the
room, but as you play in it over time (minutes or days of
repeated visits) you learn certain ways of playing that
characteristically elicit more or less well defined entities,
206
ILLUSTRATIVE ANALYSES OF TGARDEN (SHA AND
GILL)
dynamics, the intertwining between simulated physics and
material physics or body physics.
Hop-Skip Example
Another critical aspect of this responsive system is how it
deals with the characteristic time of decay or response. If
the characteristic time is too long, the environment begins
to feel decoupled from the person and if it is too short the
environment responds as a simple discrete series of
stimulus-response events. With just the right characteristic
time of response, the player imputes a strong sense of
elasticity to projected, structured light shining on a hard
floor.
In the documentary video titled ‘Hop-skip,’ a person is
hopping up and down periodically every 8 or 9 beats to
sound patterns. The strong beats in the musical textures in
this Hop-skip environment elevate the overall excitement in
the room, but our question is why and how does the human
first begin to hop and skip about the room. At 3 beats after
the third hop, the person leaves their position and begins to
hop and skip around over the floor space. The analysis of
this sudden change in movement helps in understanding the
TGarden. During the third hop (21 seconds into the action)
there is a white flash on the floor. Just following the hop,
the flash re-emerges and moves across under the feet space
and shadow of the person.
Since the TGarden is engineered with such low latencies as
to produce computed media that the human perceives as
concurrent with his or her activity, the human interprets the
computed response not as a macroscopic interactive reply
but as a tangible quality. In the hop-skip example, this
tangible quality derives from the micro-physics of the body
intertwined with the synthesized dynamics of the visual
texture and the rhythmic sound.
V2 Professional Dancers Example
In the second example, ‘TGarden V2 dancers’, four
professional dancers walk into the space, and as they do so
and find positions for themselves, the textures and colours
on the floor move with them and connect together. The
form their positions in relation to each other, coming
towards each other in the centre and then working from that
(24 seconds).
Figure 4. Hop-skip.
This is the salient responsive change in the environment
that cues in turn the response of the person. Why does the
flash re-emerge? The 3D graphics is filled in with a ‘texture
map’ and this texture of pale light colour is filled by using 2
rules: a) it is triggered by the person's hop, and b) it is
interpolated such that its echo, the echo of the person’s hop,
goes on. In other words, the texture map is continuous
function of both the internal clock of the machine, as well
as the rich real-time data from the human body's ongoing
physical movement.
We look more carefully at what is happening with the
dynamical response. There are two kinds of responses in the
TGarden environment to a person's movement. The first is
the response of the real-time video synthesis software to the
person’s physical movement, as measured by accelerometer
on his chest. In the video, for example, a bright texture map
fills in as an immediate response to the person's hop. The
other is the system's preprogrammed, retraction of the
texture map back to a plain, open mesh, interpolated over a
fixed time interval. However, there is a recoil, an echo of
the white flash. This is not due to a programmed computed
response, but is in fact due to the actual physical recoil of
the accelerometer on the chest of the human body.
Figure 5. Finding their positions in relation to each other.
Once positioned, the dancers being to warm up in an
improvised rehearsal, sensuously moving with sounds and
colours. The shifting shapes on the floor occasionally and
very momentarily detach from a dancer who then reaches
out and regains contact. As they move in the space the
dancers quickly (within 51 seconds) find resonant
connections with each other as an engagement field.
The filling in of the mesh by texture comes from prior
logic, but the echo of the flash – that recoil - comes from
the recoil of the physical body. Hence the responsivity in
the TGarden arises from both software dynamics and body
207
Figure 6. Notice the triangular space holding the attention of
their body fields (55 seconds).
Figure 8. Slo-mo. Converging on balls.
As they do so, there is a change in their rhythmic
coordination. The dynamics and tempo of their field space
shift from a smoothly coordinated rhythm to a seemingly
staccato random tempo, affected by their individual
movement with the balls and the physical contact between
bodies that comes with rolling the balls to each other. The
rhythm alters again as they disperse and their body fields
engage
in
smoother
coordinated
autonomous
choreographies. In analyzing this, one could produce an
agent-based description of a sequence of actions, for
example, a) the person with the tiger striped costume
initiated the movement of the balls in the center of the play
space, and this entrained the others to do likewise.
Alternatively, and as we do, one could produce a fieldbased analysis where the overall movement in the space and
the relations of engagement undergoes change. In this
example, the "tiger stripes person …" description is an egobased and object-oriented account, whilst the latter
description of rhythm and tempo is an account of the entire
scene in terms of rhythmic fields.
By the end of their performance, they are fully rhythmically
coordinated.
This is enabled through grounding their coordinations with
the environment and each other during discovery and
improvisation.
Figure 7. Two of the dancers sway together in aligned
synchrony (2 min 11 sec).
In analyzing how they coordinate or attending to each other,
it is not essential to know their gaze activity, which is
subsumed as background knowledge to their body
movement. In the TGarden, gaze is not a core part of
‘attention’. In much cognitive work, visual attention or gaze
is often used as an explicit indicator of attention. However,
to correlate gaze with cognition and attention is to reduce
the connectivity in the field space of the TGarden. Our
analysis focuses body movements as fields, i.e. of the whole
body and thereby avoids reduction to solely geometric data
and visual perception.
TGARDEN: SUBSTRATE AND CONCURRENCY (SHA)
Within the environment we are always permeated,
coincident with fields of physical and symbolic material. A
core concept for the TGarden is the ‘substrate’. In the Slomo example above, the agent-based description of activity
is that of individual beings and it takes the form of a graph.
However, ‘substrate’ is a way of looking at the entire room
as a continuous distribution of, for example, sound, light,
fabrics, costumes and bodies, and more abstractly, gestures,
and fields of speech or attention. Considering the changes
in the distribution over time of fields is a dynamic approach
that lends itself naturally to notions such as waves and
rhythms. By ‘substrate’ we mean the union of all these
continuous, time-varying distributions.
Slo-mo Example
The ‘Slo-mo’ video illustrates body movement fields.
About 41 seconds into the action, there is a scene where
four dancers converge around the centre of the play space
and move with four large balls.
Concurrency
Concurrency is a crucial aspect of the TGarden's field-based
computationally mediated experience. It substantially
208
players in a TGarden form tacit awareness in overlapping
and autonomous space and gauge elements and patterns of
connectivity, and through this tacit learning, shape the
media space and are concurrently shaped by it.
differs from the standard graph-theoretic model of agent-toagent information-passing and causality, which is an
analytic framework whereby humans and devices are seen
to operate in sequential chains. In the TGarden, however,
all the environmental processes evolve in semi-autonomy
coupled by relatively long-time (O(100) sec) state
information and relatively short-time (O(10) msec) time
series derived from sensor data. In any case, the visual and
sound processes are engineered under the requirement that
they compute their responses to new sensor data within the
threshold of human perception of concurrency: sensor data
and sounds and visual imagery are computed fast enough to
appear phenomenologically concurrent with human
gestures. This concurrency enables people to become
coupled with the room and with one another. In a
sufficiently small space the concurrency sustains an
embodied sense of co-present experience rather than a
combinatorially complex game of atomic agents.
REFERENCES
1. Birdwhistle, R.L. Kinesics and Context, University of
Pennsylvania, 1970.
2. Gill, S.P., Kawamori, M, Katagiri, Y, Shimojima, A.
The Role of Body Moves in Dialogue. International
Journal for Language and Communication (RASK), 12
(2000), 89-114.
3. Gill, S.P. The Engagement Space and Parallel
Coordinated Movement: Case of a Conceptual Drawing
Task. CKIR (Centre for Knowledge and Innovation
Research) Working Paper series, CKIR-1, (2002)
(ISBN 951-791-660-4).
4. Gill, S.P. Body Moves and Tacit Knowing. In Gorayska,
B. and Mey, J. (Eds.) Cognition and Technology.
Amsterdam: John Benjamin, 2004, 241-265.
CONCLUSIONS AND REFLECTIONS
The TGarden has been reflexively constructed by creators
who are expert students of the interactive and responsive
strategies operating in the human performers and the
computational media systems. In the design, the creators
explicitly designed the environment to allow gesturing
bodies to emerge from and dissolve into various kinds of
fields: musical sonic textures, choreographical bodies,
kinetic visual textures, and fabrics. The construction carries
on a phenomenological research agenda that explicitly
informs the design of the responsive space and includes the
following questions: How is agency diffused? What is
meaning-making movement? What is individual and group
agency? And what is nature of continuous materiality.
These questions are addressed by ‘concurrent activity’ that
is made possible through conceiving of the responsive
space as ‘material substrate’, whereby the entire room is
bathed in sound. This research agenda is described in detail
in [7].
5. Gill, S.P. and Borchers. Knowledge in Co-Action:
Social Intelligence in Collaborative, Design Activity, AI
& Society, 17.3 (2003), 322-339.
6. Harris, R., Signs of Writing, Routledge, London, 1997.
7. Salter, C., and Sha, X.W. Sponge: A Case Study in
Practice-based Collaborative Art Research, Creativity
and Cognition, submitted (2005).
8. Scheflen, A.E. How Behaviour Means. Exploring the
Contexts of Speech and Meaning: Kinesics, Posture,
Interaction, Setting, and Culture. New York: Anchor,
Press/Doubleday, 1974.
9. Sponge. TGarden Project: TG2001,
http://sponge.org/projects/m3_tg_intro.html
10. Sha, X.W. Resistance Is Fertile: Gesture and Agency in
the Field of Responsive Media, in Makeover: Writing
the Body into the Posthuman Technoscape,
Configurations, 10.3, Summer (2002), 439-472.
The TGarden as constructed does not interpret movement
presuming intentionality or a model of the ego subject. The
salient rhythms are essentially resonances of spontaneous
actions and non-symbolic, providing an example of an ‘alinguistic semiology of human performance’ [10]. We
extend the concept of the Body Move based on its essential
fields of resonant performance to movement based on fields
instead of particular human bodies. We consider how the
11. Sha, X.W. The TGarden As a Phenomenological
Experiment, (in preparation).
12. Sha, X.W. Topological Media Lab,
http://topologicalmedia.net .
209