Touching Sound: Vulnerability and
Synchronicity
Sam Aaron
Richard Hoadley, Helen Odell-
Abstract
Computer Lab, University of
Miller,
Cambridge, UK.
Department of Music and
Samaaron@gmail.com
Performing Arts,
In this paper, vulnerability is considered in terms of our
ability to synchronise in time with others, and how the
consequences of being unable to do so leads to
isolation and a range of social, emotional, and
psychological issues. We describe a project to develop a
shared musical instrument to facilitate the process of
joint interaction and the emergence of moments of
synchronising in time with others. The design involves a
multidisciplinary and multi-practice team that includes
music therapists, artists, performance artists,
musicians, music psychologists, music technologists,
software and hardware sound engineers.
Anglia Ruskin University,
Phil Barnard
Cambridge UK.
MRC Cognition and Brain
richard.hoadley@anglia.ac.uk
Sciences Unit, Cambridge, UK.
Helen.Odell-Miller@anglia.ac.uk
Philip.Barnard@mrc-cbu.cam.ac.uk
Rob Toulson
Ian Cross, Satinder Gill
Faculty of Music, University of
Cambridge, UK.
ic108@cam.ac.uk
spg12@cam.ac.uk
Cultures of the Digital Economy
Research Institute (CoDE),
Anglia Ruskin University,
Cambridge UK.
Rob.Toulson@anglia.ac.uk
Tommi Himberg
Brain Research Unit, O.V.
Lounasmaa Laboratory, Aalto
University, Finland.
tommi.himberg@aalto.fi
Author Keywords
Synchrony; Cooperative Action; Music Therapy;
Entrainment; Aspergers; Autism; Design.
ACM Classification Keywords
H.5.m. Information interfaces and presentation (e.g.,
HCI), H.5.2.User interfaces, H.5.3 Computer supported
cooperative work, evaluation/methodology,
synchronous interaction, theories and models.
General Terms
Copyright is held by the author/owner(s).
CHI’13, April 27 – May 2, 2013, Paris, France.
ACM 978-1-XXXX-XXXX-X/XX/XX.
Human Factors; Design.
Introduction
Being unable to connect in time with another human
being can lead to isolation, loneliness, and frustration.
For most of us, we take our ability to move in
synchrony with each other for granted and only become
aware of this capacity when the timing is not quite
right, leaving an awkward feeling. For some, however,
this capacity is inhibited, and notably for people with
autism. At the mild end of the autism spectrum, those
with Aspergers suffer depression and degrees of
intolerance from others that comes with this
discrepancy in timed interaction. They are fully aware
that they have difficulties in establishing bonds with
others in the same way as those around them, and
although at the surface they appear to be like anyone
else, their behavioural reactions are slightly different
which results in a lack of understanding towards them.
Those with Aspergers are vulnerable in their emotions
and their isolation.
In his work on autism, amongst other disorders,
Condon [1] identified that the temporal coordination
between someone with autism and someone without,
created problems in being able to understand the other
due to the temporal delay of perceiving and responding
to sound.
Musicians have developed therapeutic methods [2] to
facilitate moments of connection through making music
together, seeking to help people with a range of
emotional and psychological issues to synchronise in
time with another person.
Over the past two years, a team of us including
musicians, music psychologists, music therapists,
performance artists, software and hardware designers,
have been developing a design process and prototype
of a shared musical instrument to address how to
support synchronisation in time and facilitate the
capacity to cooperate with another person. Such an
instrument would be tangible and pleasurable, requiring
at least two people to play by touching it with their
hands to create musical sounds. It would be a
complement to the current instruments and methods
used by music therapists, and we see it as something
that, in the future, could be used by families and
friends to engage with those they love and seek to
support. At the heart of music therapy is experiencing
each other through sound and movement.
In the past, music therapists have sometimes been
disappointed and wary in working with technologists as
the design process and end result has tended to be
technology driven rather than therapy and human
engagement driven, and thereby fallen short of
therapeutic needs [3]. From the beginning, we realised
that we needed to work together in order for us to
develop any meaningful and useful interface for
vulnerable children and adults. This paper presents our
work in progress, and in the next section we discuss
our design method and framework, followed by
examples of some findings from the pilot work, and
concluding with some reflections on what needs to be
done.
Design and Research
Our shared musical instrument investigates how a
gestural and tangible interface can facilitate synchrony
and cooperative action in musical interaction. The
design involves an iterative process of developing and
testing what we have termed Generic Interfaces for
Socio-Musical Orientation (GISMOs). We are integrating
gesture-based interfaces with interactive digital music
systems to produce an instrument that responds to the
degrees of cooperation exhibited in the joint interaction
with it. The patterns of communicative interaction, both
verbal and gestural are analysed to gain an
understanding of the emergence of cooperative
behaviour, and the findings are fed back into the design
to optimise the interface’s facilitation of joint
interaction.
The research draws on work in Interactive Musical
Digital systems, Computer-Supported Cooperative Work
(CSCW), and tangible interfaces. In the field of
interactive digital music, many systems have been
developed in which performers use gesture to shape or
generate sound. Murray-Rust & Smaill [4] provide a
preliminary system for describing and evaluating the
dynamics of digital musical interactivity in generic
terms. Cross [5] has integrated aspects of this system
in a framework to guide and to test the development of
digital music technologies for cooperative interaction.
This research builds on recent musicological research
[6] which views music as a medium for cooperative
interaction that engenders social solidarity. Music
achieves this by aligning the affective states of
interacting participants, by aligning their actions in time
through processes of entrainment [7]. Musical
interaction supports behaviours such as empathy,
mimicry or synchrony of gesture [8, 9], providing a
means for exploring the conditions for the emergence
and maintenance of cooperation.
While collaborative or cooperative interactions lie at the
heart of much research in CSCW, their study is
primarily considered with regard to the achievement of
the goals of the collaboration [10]. CSCW has
traditionally focused on transactional features involved
in the coordination of joint, goal-directed action.
However, research (e.g. [11]) has shown that
interactive features other than those directly concerned
with the extrinsic goals of the interaction, are highly
influential for the success of interaction. These
features, characterised as phatic or relational, serve to
set up and sustain cooperative social relationships. The
CSCW and Tangible Embedded and Embodied
Interfaces (TEI) literature is beginning to consider this
in the last decade (see, e.g. [12] and [13]).
The relational dimension is central to the maintenance
and success of interaction, and mediating interfaces
need to be sensitive to cues of both transactional and
relational dimensions. The design of the interface of the
shared musical instrument is being developed with
sensitivity to these cues.
The “physicality” of touch and movement are also part
of cooperative interaction. The area of TEI addresses
this aspect in the design of interactive and cooperative
touch and movement-based interfaces that explore
human emotion, sharing of experience, and sensory
interaction (e.g. [14] on the experience of intimacy
through the cooperative manipulation of sound). The
sensory immediacy of tangibility, combined with realtime connectivity, allows for the sharing of experience
that is essential for ‘being with others’ [15]. In HCI
there is a growing interest in temporality and bodily
interaction (e.g. Mobile Life lab at Stockholm
University).
Designing The Prototype
The team decided that due to sensitivity, early
prototyping of the GISMO should be tested out on
people who are not vulnerable. This is for ethical
reasons so as not to cause any disturbance and harm.
The music therapists will be able to gauge when the
prototype has reached a level where vulnerable people
can use it.
GISMO Prototype
The design of the first prototype was undertaken
following discussions in the team. One of the music
therapists working with children asked us to think of
the interface as being simple to engage with and
‘dribble proof’. Another therapist suggested that one of
the simplest interfaces might just involve the laying
down of hands. In design terms, this was easily
accomplished using electrically conductive paint which
is sensitive to the skin’s natural capacitance, and a
perspex surface was used that emphasised and
protected the interface’s functionality. The two hand
prints were configured so as to emphasise the
requirement that the device should be operated by two
different people, although it is accepted that this could
be made more clear in future versions. Subsequent to
testing and the pilot study it was also decided that any
future version would be improved by the inclusion of
raised, moulded hand shapes on the surface of the
Perspex. These would help make the type of contact
required, clearer.
One of the interactive goals of the prototype is
simultaneous synchrony. Music therapists term this the
‘pivotal moment’. It is brief but significant, and is a felt
experience by both the client and the therapist. The
music technologist, software engineer, and sound
engineer selected a high pitch musical sound to cue
simultaneous synchrony. This sound would only occur
when two people touched the GISMO at the same time.
The prototype has been tested in the pilot study briefly
presented below (see www.touchingsound.rt60.co.uk).
The Pilot Study
The Pilot study was conducted in the Centre for Music
and Science, with five pairs of subjects, all with
experience in music. For the Pilot study, as we were
testing what could be understood about the interface,
we deliberately did not inform the participants about
the relationship between their actions and the sound
outputs. We asked them discover what is possible, and
once they felt that had achieved this they were asked
to play with the GISMO together. Each session lasted
20 minutes and was videotaped.
PRELIMINARY OBSERVATIONS
The participants seemed to find the organ sound to be
the most pleasurable. Opinions differed about the high
pitched sound of tinkling bells, yet when it occurred the
participants were curious as to what actions caused it.
One pair liked the organ sound so much that they
focused on discovering how to achieve it, yet their
coordination with each other at the surface of the
GISMO appeared to be correlated to the higher pitched
musical sound. In another pair, where one of the
participants found it somewhat irritating, the pair still
found themselves oriented to achieving it. The higher
pitched sound seemed to facilitate getting people to
come together in moments of simultaneous synchrony.
They all came to realise that the high pitched sound
occurred when they touched the surface at the same
time, but they were not so clear as to the correlation of
other sounds to other forms of synchronised
coordination.
The participants used a wide range of gestures from
finger movements, to sliding the hand across the
surface, to waving the hand in choreographed
movements around the surface, and in the case of one
pair (both pianists), used both their hands and all ten
fingers. Most of them realised that the palm or whole
hand needs to be placed on the surface or just above
the surface.
The observations raise a complex range of design
issues, including: what is a pleasurable sound? What
kinds of gestures does the interface need to afford?
How do we correlate sounds with different timings in
coordinated synchronisation? How do we ensure the
system does not bias leadership possibilities? How
would patterns of self and self-with other entrainment
to sounds evolve with the design?
Participants entrained each other in movement at the
levels of individual and joint entrainment to sound,
both on and above the surface of the GISMO. Consider
this interaction starting at 2.43 mins into the session:
2.43 mins - both have hands touching and flat
on the Gismo;
For the GISMO to be applied in the context of music
therapy, and be used by families and with friends, it
will need to be able to handle the particular aesthetics
of sounds for the vulnerable person, and adapt to their
interaction capabilities. It would also need to be
sensitive to the touch sensitivity of participants to
particular textures, materials, and physical feedback.
2.44 mins – A lifts her hand and B follows;
Conclusion
2.45 mins - B places her hand down and A
follows;
2.46 mins - both simultaneously lift their
hands;
2.47 mins - A places hand down and B follows,
then A then lifts her hand before B touches
down as she hears the organ sound, causing
B's hand to hover above Gismo's surface; both
then lift their hands higher at the organ sound.
There was a common held view that one person was
the 'leader', in 'charge', because one of the hands
seemed to activate the sound more than the other.
This project is informed from its outset by music
therapy, understanding its practices and experiential
knowledge of the therapist and the experiential
knowledge of the vulnerable person. We consider
vulnerability here as lying in the difficulties of moving in
time with another human being that ordinarily allows us
to be connected, understand, and be understood by
others at the phatic and relational level. The result can
be isolation, loneliness, and in cases, depression.
In the development of this shared musical instrument,
we aim to help elucidate principles that underlie the
emergence of cooperation in real-time interaction,
using music as a medium through which interactions
can be conducted, and apply these to the development
of GISMO as an interface that affords moments of
sharing an experience with another person.
Acknowledgements
We thank Amelia Oldfield, Bonnie Kempske, Jane
Turner, and Cecily Morrison for their advice on the
Touching Sound project and the design of GISMO.
References
[1] Condon, WS, Ogston WD. Sound film analysis of
normal and pathological behaviour patterns. Journal of
Nervous and Mental Diseases, 143, 4 (1996), 338-347.
[2] Wigram, T. Improvisation. Methods and techniques
for Music Therapy Clinicians, Educators and Students.
Jessica Kingsley Publishers, 2004.
[3] Magee, W. L., Bertolami, M., Kubicek, L., LaJoie,
M., Martino, L., Sankowski, A., et al. Using Music
Technology in Music Therapy With Populations Across
the Life Span in Medical and Educational Programs.
Music and Medicine, 3, 3 (2011): 146-153.
[4] Murray-Rust, D., & Smaill, A. Towards a model of
musical interaction and communication. Artificial
Intelligence, 175, 9-10 (2011), 1697-1721.
[5] Cross, I. (in press). "Does not compute"? Music as
real-time communicative interaction. AI & Society.
[6] Turino, T. Music as social life. London: University of
Chicago Press, 2008.
[7] Clayton, M., Sager, R., & Will, U. In time with the
music: The concept of entrainment and its significance
for ethnomusicology. ESEM counterpoint, 1 (2005), 145.
[8] Miles, L. K., Nind, L. K., & Macrae, C. N. The rhythm of rapport:
Interpersonal synchrony and social perception. Journal
of Experimental Social Psychology, 45, 3 (2009): 585589.
[9] Rabinowitch, T.-C., Cross, I., & Burnard, P. Longterm musical group interaction has a positive influence
on empathy in children. Psychology of Music (2012) 115.
[10] Schmidt, K. Cooperative Work and Coordinative
Practices: Contributions to the Conceptual Foundations
of Computer-Supported Cooperative Work (CSCW).
London: Springer, 2011.
[11] Coupland, J., Coupland, N., & Robinson, J. D. "How
Are You?": Negotiating Phatic Communion. Language in
Society, 21, 2 (1992), 207-230.
[12] Oleksik, G., Jones, R., & Milic-Frayling, N. Use of
gestures in a multi-device environment (Technical
Report MSR-TR-2010). Redmond, WA 98052: Microsoft
Research, 2010.
[13] Olson, I. C., Leong, Z. A., Wilensky, U. & Horn, M.
S. (2011). “It’s just a toolbar!” Using tangibles to help
children manage conflict around a multi-touch tabletop.
In Proceedings of TEI’11, (2011), 29-36.
[14] Balaam, M., Fitzpatrick, G., Good, J., & Harris, E.
Enhancing interactional synchrony with an ambient
display, Proceedings of the 2011 annual conference on
Human factors in computing systems, ACM Press
(2011), 867-876.
[15] Nagargoje, A., Maybach, K., Sokoler, T. Social Yoga
Mats: Designing for Exercising/Socialising Synergy. In
Proceedings of TEI’12, (2012), 87-90.