XTH SENSE: A STUDY OF MUSCLE SOUNDS
FOR AN EXPERIMENTAL PARADIGM
OF MUSICAL PERFORMANCE
Marco Donnarumma
Sound Design, ACE, The University of Edinburgh,
Alison House, Nicolson Square
Edinburgh, UK, EH8 9DF
m.donnarumma@sms.ed.ac.uk
ABSTRACT
This paper seeks to outline methods underlying the
development of the Xth Sense project, an ongoing
research which investigates exploratory applications of
biophysical sound design for musical performance and
responsive milieux.
Firstly, I describe the development and design of the
Xth Sense, a wearable hardware sensor device for
capturing biological body sounds; this was implemented
in the realization of Music for Flesh I, a first attempt at
musical performance. Next, the array of principles
underpinning the application of muscle sounds to a
musical performance is illustrated. Drawing from such
principles, I eventually describe the methods by which
useful features were extracted from the muscle sounds,
and the mapping techniques used to deploy these
features as control data for real time sound processing.
1.
INTRODUCTION
Biosensing musical technologies use biological signals
of a human subject to control music. One of the earliest
applications can be identified in Alvin Lucier's Music
for Solo Performer (1965). Alpha waves generated when
the performer enters a peculiar mind state are transduced
in electrical signals used to vibrate percussion
instruments. Over the past thirty years biosensing
technologies have been comprehensively studied [3, 8,
13, 14, 15, 18, 22] and presently notable biophysicalonly music performances1 are being implemented at
SARC2 by a research group lead by the main contributor
to the Bio Muse project3 Ben Knapp (10).
Whereas biological motion and movement and music
are arising topics of interest in neuroscience research [5,
12, 21], the biologic body is being studied by music
researchers as a means to control virtual instruments.
Although such approach has informed gestural control of
music, I argue that it overlooks the expressive
capabilities of muscle sounds. They are inaudible to the
human ear, but can be amplified and data extracted from
these signal inside a computer may retain a meaningful
vocabulary of intimate interactions with the musicians'
actions.
1
Bio Muse Trio, GroundMe!.
Queen's University, Sonic Art Research Center, Belfast, UK.
3
A commercialized product exploiting electromyography and
brainwaves analysis systems for musical applications.
2
To what extent could muscle sounds be employed
musically? In which ways could the performer's
perceptual experience be affected? How could such
experimental paradigm motivate an original perspective
on musical performance?
2.
AESTHETIC PRINCIPLES
The long-term outcome of the research is the
implementation of low cost, open source tools (software
and hardware) capable of providing musicians,
performers and dancers with a framework for
biosensors-aided auditive design (BAAD)4 in a real
time5 environment. The framework will be redistributable, customizable and easy to set up. However,
given the substantial interdisciplinary quality of such
project, its realization process needed to be fragmented
into more specific and measurable steps.
At a first stage, primary aim of the inquiry was to
explore the musical deployment and design of biological
sounds of the body in a functional context – the
production of Music for Flesh I a solo sound
performance for wearable biosensing device, which
could demonstrate an experimental coupling between
unheard sounds of muscle gestures and corresponding
sound synthesis played back through loudspeakers. In an
attempt to inform the present state of augmented musical
performance and embodied interaction in performing
environments, the characteristics of this pairing were
identified in the authenticity of the performer's somatic
interaction, the natural responsiveness of the system and
the expressive immediacy and transparency of the
mapping of muscle sounds to the performer's kinetic
behaviour. Such work required an interdisciplinary
approach embracing biomedical computing studies,
music technology and most importantly sound design. In
fact, as I will demonstrate later in this text, the major
research issue was not a technical implementation, but
rather the definition of design paradigms by which the
captured biological sounds could achieve a meaningful
and detailed expressiveness.
4
BAAD is a novel term used by the author to indicate a specific
sound design practice which relies on the use of biological signals.
Although in this context is not possible to further elaborate on this
practice, its essential principles are defined in paragraph 4.1.
5
Real time refers here to a computing system in which there exists no
perceivable delay between performer's actions and sonic response.
3.
METHODS: UNDERSTANDING AND
CAPTURING MUSCLES SOUNDS
The project consisted of two interrelated strands of
research. The first concerned the design of muscle
sounds and their meaningful mapping to the somatic
behaviour of the performer; the second included the
design and implementation of a wearable biosensing
hardware device for musical performance. Chosen
research methods are discussed in the following
paragraphs; however, being the focus of this paper on
the research methodology, specific signal processing
techniques and other technical information are not
illustrated in detail, but they are fully referenced.
3.1. Xth Sense: first prototype sensor implementation
Before undertaking the development of the Xth Sense
sensor hardware, few crucial criteria were defined:
•
•
•
•
to develop a wearable, unobtrusive device,
allowing a performer to freely move on
stage;
to implement an extremely sensitive
hardware device which could efficiently
capture in real time and with very low
latency diverse muscle sounds;
to make use of the most inexpensive
hardware solutions, assuring a low
implementation cost;
to implement the most accessible and
straightforward production methodology
in order to foster the future re-distribution
and openness of the hardware.
Study of the hardware sensor design began with a
contextual review of biomedical engineering papers and
publications focused on mechanical myography (MMG).
The mechanical signal which can be observed from the
surface of a muscle when it is contracted is called a
MMG signal. At the onset of muscle contraction,
significant changes in the muscle shape produce a large
peak in the MMG. The oscillations of the muscle's fibers
at the resonant frequency of the muscle generate
subsequent vibrations. The mechanomyogram is
commonly known also as the phonomyogram, acoustic
myogram, sound myogram or vibromyogram.
Interestingly, MMG seems not to be a topic of interest
in the study of gestural control of music and music
technology; apparently many researchers in this field
focus their attention on electromyography (EMG),
electroencephalography (EEG), or multidimensional
control data which can be obtained through the use of
wearable accelerometers, gyroscopes and other similar
sensors. Notwithstanding the apparent lack of pertinent
documentation in the studies of gestural control of music
and music technologies, useful technical information
regarding different MMG sensor designs were collected
by reviewing recent biomedical engineering literature.
In fact, MMG is currently the subject of several
investigations in this field as alternative control data for
low cost, open source prosthetics research and for
general biomedical applications [1, 6, 9, 20]. Most
notably the work of Jorge Silva at Prism Lab; his MASc
thesis extensively documents the design of a coupled
microphone-accelerometer sensor pair (CMASP) and
represents a comprehensive resource of information and
technical insights on the use and analysis of MMG
signals [19]. The device designed at Prism Lab is
capable of capturing the audio signal of muscles sounds
in real time. Muscle sonic resonance is transmitted to the
skin, which in turn vibrates, exciting an air chamber.
These vibrations are captured by an omnidirectional
condenser microphone adequately shielded from noise
and interferences by mean of a silicon case. A printed
circuit board (PCB) is used to couple the microphone
with an accelerometer in order to filter out vibrations
caused by global motion of the arm, and precisely
identify muscle signals (figure 1). Microphone
sensitivity ranges from 20Hz up to 16kHz, thus it is
capable of capturing a relevant part of the spectrum of
muscle resonance6.
Figure 1. CMASP schematic
Although this design has been proved effectively
functional through several academic reports, criteria of
my investigation could have been satisfied with a less
complex device. Supported by the research group at
Dorkbot ALBA7, I could develop a first, simpler MMG
sensor: the circuit did not make use of a PCB and
accelerometer, but deployed the same omnidirectional
electret condenser microphone indicated by Silva
(Panasonic WM-63PRT). This first prototype was
successfully used to capture actual heart and forearm
muscles sounds; earliest recordings and analysis of
MMG signals were produced with the open source
digital audio workstation Ardour2 and a benchmark was
set in order to evaluate the signal-to-noise ratio (SNR).
In spite of the positive results obtained with the first
prototype, the microphone shielding required further
trials. The importance of the shield was manifold; an
optimal shield had to fit specific requirements: to bypass
the 60Hz electrical interference which can be heard
when alternating electric current distribute itself within
6
On a side note, it is interesting to observe that the biggest part of
muscles sounds spectra seems to sit below 20Hz, thus pertaining to the
realm of infra-sounds. Such characteristic is not being explored at the
moment only due to technical constraints, although it suggests
appealing prospects for a further research.
7
Electronics open research group based in Edinburgh.
See: http://dorkbot.noodlefactory.co.uk/wiki
the skin after a direct contact with the microphone metal
case; to narrow the sensitive area of the microphone,
filtering out external noises; to keep the microphone
static, avoiding external air pressure to affect the signal;
to provide a suitable air chamber for the microphone, in
order to amplify sonic vibrations of the muscles,
allowing to capture also deeper muscle contractions.
First, microphone was insulated by means of a
polyurethane shield, but due to the strong malleability of
this material, its initial shape tended to undergo
substantial alterations. Eventually, sensor was insulated
in a common silicon case that positively satisfied the
requirements and further enhanced the SNR. Once the
early prototype had reached a good degree of efficiency
and reliability, the circuit was embedded in a portable
plastic box (3.15 x 1.57 x 0.67) along with an audio
output (¼ mono chassis jack socket) and a cell holder for
a 3V coin lithium battery. The shielded microphone was
embedded in a Velcro bracelet and needed wiring cables
were connected to the circuit box (figure 2).
according to their capability of enhancing the
metaphorical interpretation of the performer's
physiological and spatial behaviour.
From this perspective, the essential principles of
BAAD in a performing environment were defined as
follows:
•
•
•
•
•
•
to make use of biological sounds as major sonic
source and control data;
to exclude the direct interaction of the
performer with a computer and to conceal the
latter from the view of the public;
to demonstrate a distinct, natural and non-linear
interaction between kinetic energy and sonic
outcome which could be instinctively controlled
by the performer;
to provide a rich, specific and unconventional
vocabulary of gesture/sound definitions which
can be unambiguously interpreted by the
audience;
to allow the performer to flexibly execute the
composition, or even improvise a new one with
the same sonic vocabulary;
to make both performer and public perceive the
former's body as a musical instrument and its
kinetic energy as an exclusive sound generating
force.
4.2. MMG features extraction
Figure 2. Xth Sense wearable MMG sensor
prototype
4.
PERFORMANCE TESTING: MAPPING AND
DESIGN DEFINITIONS
At this stage of the project the creation of design
paradigms for mapping muscle sounds was the major
goal. The main principles and some technical
implementations are illustrated below.
4.1. Sound performance and design principles
Major aim of the design of the MMG audio signals was
to avoid a perception of the sound being dissociated
from the performer's gesture. The dissociation I point at
not only refers to the visual feedback of the performer's
actions being disjointed from the sonic experience, but it
also, and most importantly, concerns a metaphorical
level affecting the listener's interpretation of the sounds
generated by the performer's somatic behavior [2]. In
this project the use of muscle sounds had to be clearly
motivated in order to inform classical approaches to
gestural control of music. Therefore, chosen sound
processing and data mapping techniques were evaluated
Since the project dealt with sound data, a pitch tracking
system may have been a straightforward solution for an
automated evaluation and recognition of gestures,
however muscle sound's resonance frequency is not
affected by any external agent and its pitch seems not to
change significantly with different movements [17].
Whereas muscle sounds are mostly short, discrete events
with no meaningful pitch change information, the most
interesting and unique aspect of their acoustic
composition is their extremely rich and fast dynamic;
therefore, extraction of useful data can be achieved by
RMS amplitude analysis and tracking, contractions
onset and gesture pattern recognition. In fact, each
human muscle exerts a different amount of kinetic
energy when contracting and a computing system can be
trained in order to measure and recognize different
levels of force, i.e. different gestures. Feature extraction
enabled the performer to calibrate software parameters
according to the different intensity of the contractions of
each finger or the wrist and provided 8 variables: 6
discrete events, 1 continuous moving event and 1
continuous exponential event.
First, the sensor was subjected to a series of
movements and contractions with different intensity to
identify a sensitivity range; this was measured between
57.79 dB (weakest contraction) and 89.04 dB (strongest
contraction). The force threshold of each finger discrete
contraction was set by normalizing and measuring the
individual maximum force exertion level; although
some minor issues arisen from the resemblance between
the force amplitude exerted by the minimus (little
finger) and the thumb still need to be solved, this
method allowed the determination of 6 independent
binary trigger control messages (fingers and wrist
contractions).
Secondly, by measuring the continuous amplitude
average of the overall contractions, it was possible to
extract the running maximum amplitude of performer's
gestures; in order to correct the jitter of this data, which
otherwise could not have been usefully deployed, value
was extracted every 2 seconds, then interpolated with
the prior one to generate a continuous event and
eventually normalized to MIDI range. Lastly, a basic
equation of single exponential smoothing (SES) was
applied to the moving global RMS amplitude in order to
forecast a less sensitive continuous control value [16].
4.3. Mapping kinetic energy to control data
A first mapping model deployed the 6 triggers
previously described as control messages. These were
used to enable the performer to control the real time
SSB modulation algorithm by choosing a specific
frequency among six different preset frequencies; the
performer could select which target frequency to apply
according to the contracted finger; therefore, the
voluntary contraction of a specific finger would enable
the performer to “play” a certain note.
A one-to-many mapping model, instead, used the
continuous values obtained through the RMS analysis to
control several processing parameters within five digital
signal processing (DSP) chains simultaneously. Being
that this paper does not offer enough room to fully
describe the whole DSP system which was eventually
implemented, I will concentrate on one example chain
which can provide a relevant insight on the chosen
mapping methodology; namely, this DSP chain included
a SSB modulation algorithm, a lofi distortion module, a
stereo reverb, and a band-pass filter.
The SSB algorithm was employed to increase the
original pitch of the raw muscle sounds by 20Hz, thus
making it more easily audible. Following an aesthetic
choice, the amount of distortion over the source audio
signal was subtle and static, thus adding a light
granulation to the body of the sound; therefore, the
moving global RMS amplitude was mapped to the
reverb decay time and to the moving frequency and
Quality factor8 (Q) of the band-pass filter.
The most interesting performance feature of such
mapping model consisted of the possibility to control a
multi-layered processing of the MMG audio signal by
exerting different amounts of kinetic energy. Stronger
and wider gestures would generate sharp, higher
resonating frequencies coupled with a very short reverb
time, whereas weaker and more confined gestures
would produce gentle, lower resonances with longer
reverb time.
Such direct interaction among the perceived force and
spatiality of the gesture and the moving form and color
of the sonic outcome happened with very low latency,
and seemed to suggest promising further applications in
a more complex DSP system.
8
Narrowness of the filter.
Figure 3. Music for Flesh I first public performance,
2010
The Xth Sense framework was tested live during a
first public performance of Music for Flesh I (figure 3)
at the University of Edinburgh (December 2010).
Although the system was still in development, it proved
reliable and efficient. Audience feedback was positive,
and apparently what most appealed some listeners was
an authentic, neat and natural responsiveness of the
system along with a suggestive and unconventional
coupling of sound and gestures.
5.
CONCLUSIONS
Results reported in this paper appear to disclose
promising prospects of an experimental paradigm for
musical performance based on MMG. The development
of the Xth Sense and the composition and public
performance of Music for Flesh I can possibly
demonstrate an uncharted potential of biological sounds
of the human body, specifically muscle sounds, in a
musical performance.
Notwithstanding the apparently scarce interest of the
relevant academic community towards the study and the
use of muscle sounds, the experiment described here
shows that these sounds could retain a relevant potential
for an exploration of meaningful and unconventional
sound-gesture metaphors. Besides, if compared to EMG
and EEG sensing devices, the use of MMG sensors
could depict a new prospect for a simpler
implementation of unobtrusive and low-cost biosensing
technologies for biophysical generation and control of
music.
Whereas the development of the sensor hardware
device did not present complex issues, several
improvements to the tracking and mapping techniques
can lead to a further enhancement of the expressive
vocabulary of sound-gestures. In an attempt to enrich
the performer's musical control over a longer period of
time, hereafter priority will be given to the extraction of
other useful features, to the development of a gesture
pattern recognition system and to the implementation of
a system for multiple sensors.
6.
REFERENCES
[1] Alves, N., Sejdić, E., Sahota, B. and Chau, T.
“The effect of accelerometer location on the
classification
of
single-site
forearm
mechanomyograms,” Biomedical Engineering
Online, vol. 9, 2010, p. 23.
[2] D. Arfib, J.M. Couturier, L. Kessous, and V.
Verfaille, “Strategies of mapping between
gesture data and synthesis model parameters
using perceptual spaces,” Organised Sound, vol.
7, 2003.
[3] Arslan, B., Brouse, A., Castet, J., Léhembre, R.,
Simon, C., Filatriau J.J. And Noirhomme, Q.
“A real time music synthesis environment
driven with biological signals,” Proceedings of
the IEEE International Conference on
Acoustics, Speech and Signal Processing, 2006,
pp. 14–19.
Computer Music Journal, vol. 14, Apr. 1990,
pp. 42-47.
[11] Mäki-patola, T. and Hämäläinen, P. “Latency
Tolerance for Gesture Controlled Continuous
Sound Instrument Without Tactile Feedback,”
Proceedings of the 2004 International
Computer Music Conference (ICMC’04), 2004.
[12] Manera, V., Schouten, B., Becchio, C., Bara,
B.G. and Verfaillie, K. “Inferring intentions
from biological motion: A stimulus set of pointlight communicative interactions,” Behavior
Research Methods, vol. 42, Feb. 2010, pp. 168178.
[13] Miranda, E.R. and Brouse, A. “Toward direct
Brain-Computer
Musical
Interfaces.”
Proceedings of the Conference on New
Interfaces for Musical Expression (NIME05),
2005.
[4] Clark, J.J. “Frequency and Pitch Shifting.”
[14] Miranda, E.R. “Brain-Computer Music Interface
Advanced Programming Techniques for
Modular Synthesizers. Available at:
http://www.cim.mcgill.ca/~clark/nordmodularb
ook/nm_spectrum_shift.html [Accessed January
12, 2011].
for
Composition
and
Performance.”
International Journal on Disability and Human
Development, 2006, Vol. 5, Issue 2, pp. 119125.
[5] Deen, B. and McCarthy, G. “Reading about the
actions of others: Biological motion imagery
and action congruency influence brain activity,”
Neuropsychologia, vol. 48, May. 2010, pp.
1607-1615.
[15] Nagashima, Y. “Bio-Sensing Systems and Bio-
Feedback Systems for Interactive Media Arts,”
Proceedings of the 2003 Conference on New
Interfaces for Musical Expression (NIME-03,
2003, pp. 48—53).
[16] “NIST/SEMATECH e-Handbook of Statistical
E., Rampichini, S. and
Veicsteinas, A. “Acute passive stretching in a
previously fatigued muscle: Electrical and
mechanical
response
during
tetanic
stimulation,” Journal of Sports Sciences, vol.
27, 2009, p. 1347.
Methods.” Single Exponential Smoothing, eHandbook of Statistical Methods, 2003.
Available at:
http://www.itl.nist.gov/div898/handbook/pmc/s
ection4/pmc431.htm [Accessed January 12,
2011].
[7] Moorer, J. “The Synthesis of Complex Audio
Spectra by Means of Discrete Summation
Formulas”, Journal of the Audio Engineering
Society, vol. 24, 1976, pp. 717—727.
[17] Oster, G. and Jaffe, J. “Low frequency sounds
[8] Filatriau, J.J. and Kessous,
[18] Rosenboom, D. Biofeedback and the arts;:
[6] Esposito, F., Cè,
L. “Visual and
sound generation driven by brain, heart and
respiration signals,” Proceedings of the 2008
International Computer Music Conference
(ICMC’08), 2008.
[9] Garcia, M.A.C., Vargas, C.D., de Souza, M.N.,
Imbiriba, L.A. and de Oliveira, L.F.
“Evaluation of arm dominance by using the
mechanomyographic signal,” Journal of Motor
Behavior, vol. 40, Mar. 2008, pp. 83-89.
from sustained contraction of human skeletal
muscle,” Biophysical Journal, vol. 30, Apr.
1980, pp. 119-127.
Results of early experiments,
Research Centre of Canada, 1974.
Aesthetic
[19] Silva, J. Mechanomyography Sensor Design
and Multisensor Fusion for Upper-Limb
Prosthesis Control, PhD Thesis, Master of
Applied Sciences, Graduate Department of
Mechanical & Industrial Engineering University
of Toronto, 2004.
[20] Silva, J. and Chau, T. “Coupled microphone-
[10] Knapp, R.B. and Lusted, H.S. “A Bioelectric
Controller for Computer Music Applications,”
accelerometer sensor pair for dynamic noise
reduction in MMG signal recording,”
Electronics Letters, vol. 39, 2003, pp. 14961498.
[21] Stevens, C.. Ginsborg, J. and Lester,
G.
“Backwards and forwards in space and time:
Recalling dance movement from long-term
memory,” Memory Studies. Available at:
http://mss.sagepub.com/content/early/2010/11/0
5/1750698010387018.full.pdf+html [Accessed
January 12, 2011].
[22] Tanaka, A. “Introduction Musical Performance
Practice on Sensor-based Instruments.” Trends
in Gestural Control of Music, M. Battier and M.
Wanderley eds., 2000, pp. 389-406.