Abstract
Skalldans is an audiovisual improvisation piece for
solo laptop performer., developed in Max [2]. Sound
and video syntheses are piloted with a MIDI interface,
a camera, and a Wiimote; also, audiovisual streams
influence each other. The proceedings paper
discusses some of the hardware and software points
of interest, for example, how audio and video
syntheses are piloted, how the streams interact, and
the camera tracking method with a linear regression
stabiliser. It also touches upon the sources of
inspiration for the piece.
Background
‣
aiming for more involvement of the performer in
laptop performance [1];
‣
accident --> cranial MRI scan --> 3D “skull” object;
‣
patcher ‘framework’ developed in stages since 2008;
Performance outline (Total duration ≈ 10 min.)
‣
first section: noise, close-up details of the 3D object;
‣
2: skull revealed, musical elements added: Drone,
slow Rhythm chops up Noise;
‣
3: tempo picks up, other video streams mixed with 3D
skull (live input, tracking);
‣
4: Rhythm input from Drone --> Drum’n’Bass sample,
down-sampled visuals with more colour;
‣
5: Rhythm gradually faster & simpler (synchronising 2
step sequencers); performer ‘live head’ fuses visually
with 3D skull;
‣
last bit: ‘retro’ analog video delay, performer stands
up, head-bang dance-> abrupt end & image freeze.
Motion tracking
‣
iSight camera (computer) towards performer’s face;
‣
blob tracking [3] of performer face in relation to laptop,
mapped to 3D skull position on screen {x, y};
‣
tracking continuity and stability improved by linear
regression prediction from the proceeding 5 frames to
most likely ‘next frame position’ of the target blob;
‣
Wiimote (hidden in a cap), performer head angle
directly mapped onto 3D skull {roll, yaw, pitch}.
Physical interface
‣
Evolution uc33e MIDI interface, 9 sliders & 24 knobs;
Audio synthesis
‣
Drone: additive synthesis, directly controlled by the
performer with uc33e interface;
‣
‘Coloured Noise’: subtractive synthesis, with FFT filter
shape based on 3D skull appearance;
‣
Rhythm: two coupled step sequencers, based on [4],
variations depend on 3D skull appearance; input
comes from Drone, Noise, or a Drum’n’Bass sample.
Visual synthesis
‣
video sources:
‣
VFX with Jitter [2]: compositing, downsampling, and
Sobel outlining: directly controlled by the performer
with uc33e interface;
‣
‘retro’ analog feedback, loop with screen projection;
Skalldans, audiovisual performance.
PerMagnus Lindborg
Nanyang Technological University, Singapore / KTH Royal Institute of Technology, Stockholm
permagnus@ntu.edu.sg
www.permagnus.net
E=expensive
C=cheap
References
[1] Lindborg, PerMagnus (2008). "Reflections on aspects of
music interactivity in performance situations". eContact , 4.10.
[2] Cycling74 (). Max . www.cycling74.com (acc. Feb. 2013).
[3] Pelletier, J.-M. (2010). Cv.jit, Computer Vision library for
Jitter. http://jmpelletier.com/cvjit/ (acc. Feb. 2013).
[4] Tanaka, A. (2003) “Modsquad Relooper”. [Max example]
‣
3D skull;
‣
linear regression line, prediction ‘hair-pin’;
‣
live camera input (performer’s face);
Skalldans, audiovisual performance.
PerMagnus Lindborg
Nanyang Technological University, Singapore / KTH Royal Institute of Technology, Stockholm
permagnus@ntu.edu.sg
www.permagnus.net
Abstract
Skalldans is an audiovisual improvisation piece for
solo laptop performer., developed in Max [2]. Sound
and video syntheses are piloted with a MIDI interface,
a camera, and a Wiimote; also, audiovisual streams
influence each other. The proceedings paper
discusses some of the hardware and software points
of interest, for example, how audio and video
syntheses are piloted, how the streams interact, and
the camera tracking method with a linear regression
stabiliser. It also touches upon the sources of
inspiration for the piece.
Motion tracking
‣ iSight camera (computer) towards performer’s face;
‣ blob tracking [3] of performer face in relation to laptop,
mapped to 3D skull position on screen {x, y};
‣ tracking continuity and stability improved by linear
regression prediction from the proceeding 5 frames to
most likely ‘next frame position’ of the target blob;
‣ Wiimote (hidden in a cap), performer head angle
directly mapped onto 3D skull {roll, yaw, pitch}.
Physical interface
‣ Evolution uc33e MIDI interface, 9 sliders & 24 knobs;
Background
‣ aiming for more involvement of the performer in Audio synthesis
‣ Drone: additive synthesis, directly controlled by the
laptop performance [1];
performer
with
uc33e
interface;
‣ accident --> cranial MRI scan --> 3D “skull” object;
‣ patcher ‘framework’ developed in stages since 2008; ‣ ‘Coloured Noise’: subtractive synthesis, with FFT filter
shape based on 3D skull appearance;
‣ Rhythm: two coupled step sequencers, based on [4],
variations depend on 3D skull appearance; input
comes from Drone, Noise, or a Drum’n’Bass sample.
Visual synthesis
‣ video sources:
‣ 3D skull;
‣ linear regression line, prediction ‘hair-pin’;
‣ live camera input (performer’s face);
‣ VFX with Jitter [2]: compositing, downsampling, and
Sobel outlining: directly controlled by the performer
with uc33e interface;
‣ ‘retro’ analog feedback, loop with screen projection;
Performance outline (Total duration ≈ 10 min.)
‣ first section: noise, close-up details of the 3D object;
‣ 2: skull revealed, musical elements added: Drone,
slow Rhythm chops up Noise;
‣ 3: tempo picks up, other video streams mixed with 3D
skull (live input, tracking);
‣ 4: Rhythm input from Drone --> Drum’n’Bass sample,
down-sampled visuals with more colour;
‣ 5: Rhythm gradually faster & simpler (synchronising 2
step sequencers); performer ‘live head’ fuses visually
with 3D skull;
‣ last bit: ‘retro’ analog video delay, performer stands
up, head-bang dance-> abrupt end & image freeze.
E=expensive
C=cheap
References
[1] Lindborg, PerMagnus (2008). "Reflections on aspects of
music interactivity in performance situations". eContact, 4.10.
[2] Cycling74 (). Max. www.cycling74.com (acc. Feb. 2013).
[3] Pelletier, J.-M. (2010). Cv.jit, Computer Vision library for
Jitter. http://jmpelletier.com/cvjit/ (acc. Feb. 2013).
[4] Tanaka, A. (2003) “Modsquad Relooper”. [Max example]
Keep reading this paper — and 50 million others — with a free Academia account
Used by leading Academics
Johannes Preiser-Kapeller
Austrian Academy of Sciences
Petra Sijpesteijn
Leiden University
Justine Firnhaber-Baker
University of St Andrews
Emilia Jamroziak
University of Leeds
Related Papers
Download
A common feature found in most musical instruments is the usage of a physical interface to retrieve musician gestures. This is found from traditional musical instruments (e.g. a piano that retrieves gestures throughout keyboards and pedals) to electronic musical instruments, with their digital interfaces, (e.g. MIDI controllers). This article introduces a new type of computer music instrument that does not require a physical (touchable) interface to mediate gestural retrieval. Rather, it is controlled directly by musician's movements without any physical part to be touched, grasped or attached. They are here named as Bodiless Musical Instruments (BMIs). This article presents two BMIs and discusses their computational implementation, artistic performance and aesthetic achievement.
Download
These de doctorat, Department of Musicology, …, 2007
Download
Suzy Styles, PerMagnus Lindborg, Vibeke Sorensen, Charles Spence, Tham Chin Pang, Natalie A . Tse, Nan Shang, Shao-Min (Sean) Hung, Rachel C, paul fletcher
Welcome to this Special Issue of Array: Proceedings of Si15, the 2nd International Symposium on Sound and Interactivity.
The articles in the present issue originated in the Si15 Soundislands Festival, which was held in Singapore 18–23 August 2015. The festival events included five invited artist performances, two scientific keynotes and two days of proceedings, a commissioned sound installation, an afternoon of public talks, an internet panel, two pedagogic workshops, a concert with young performers, and more than fifty artworks and scientific papers in numerous forms and formats selected from an open call (http://soundislands.com/si15).
We are thrilled to present 20 articles, by 31 authors, emanating from Si15. The articles have been extended and thoroughly revised for this special issue of Array. They cover a range of topics related to aesthetics, percep-tion, technology, and sound art. We hope that you will enjoy the fruits of the authors' labour and therein discover many a stimulating thought.
Download
Using a research through design approach, this thesis presents a first prototype of Plot Twister, a novel interactive, audiovisual interface specifically designed for solo musicians improvising alo ...
Download
Music interactivity is a sub-field of human-computer interaction studies. Interactive situations have different degree of structural openness and musical “ludicity” or playfulness. Discussing music seems inherently impossible since it is essentially a non-verbal activity. Music can produce an understanding (or at least prepare for an understanding) of creativity that is of an order neither verbal nor written. A human listener might perceive beauty to be of this kind in a particular music. But can machine-generated music be considered creative and if so, wherein lies the creativity? What are the conceptual limits of notions such as instrument, computer and machine? A work of interactive music might be more pertinently described by the processes involved than by one or several instanciations. While humans spontaneously deal with multiple process descriptions (verbal, visual, kinetic…) and are very good at synthesising, the computer is limited to handling processes describable in a formal language such as computer code. But if the code can be considered a score, does it not make a musician out of the computer? As tools for creative stimulus, composers have created musical systems employing artificial intelligence in different forms since the dawn of computer music. A large part of music interactivity research concerns interface design, which involves ergonomics and traditional instrument maker concepts. I will show examples of how I work with interactivity in my compositions, from straight-forward applications as composition tools to more complex artistic work.
Download
Download
Download
The Integrated Multimodal Score-following Environment (IMuSE) [5] is a software project aimed at developing a system for the creation, rehearsal and performance of score-based interactive computer music compositions. An enhanced version of the NoteAbilityPro [7] music notation software is the central controller in IMuSE. The score contains conventional notes and performance indications as well as discrete and continuous control messages that can be sent to other applications such as MaxMSP [15] or Pure data (Pd) [11] during performance. As well, multiple modes of score-following [2] can be used to synchronize the live performance to the score. Score-following strategies include pitch-tracking of monophonic instruments, pitchtracking and amplitude-tracking of polyphonic instruments and gesture-tracking of performers’ hand movements. In this paper, we present an overview of the IMuSE system, with a focus on its abilities to monitor and coordinate multiple pitch and gesture trackers. A...
Download
Performing artists have frequently used technology to sense and extend the body's natural expressivity via live control of multimedia. However, the sophistication, emotional content, and variety of expression possible through the original physical channels of voice and movement are generally not captured or represented by these technologies and thus cannot be intuitively transferred from body to digital media. Additionally, relevant components of expression vary between different artists, performance pieces, and output modalities, such that any single model for describing movement and the voice cannot be meaningful in all contexts. This dissertation presents a new framework for flexible parametric abstraction of expression in vocal and physical performance, the Expressive Performance Extension Framework. This framework includes a set of questions and principles to guide the development of new extended performance works and systems for performance extension, particularly those i...
Download
As human movement is an incredibly rich mode of communication and expression, performance artists working with digital media often use performers' movement and gestures to control and shape that digital media as part of a theatrical, choreographic, or musical performance. In my own work, I have found that strong, semantically-meaningful mappings between gesture and sound or visuals are necessary to create compelling performance interactions. However, the existing systems for developing mappings between incoming data streams and output media have extremely low-level concepts of “gesture.” The actual programming process focuses on low-level sensor data, such as the voltage values of a particular sensor, which limits the user in his or her thinking process, requires users to have significant programming experience, and loses the expressive, meaningful, and metaphor-rich content of the movement. To remedy these difficulties, I have created a new framework and development environment...
Download
QPSR of the numediart …
After the first numediart project, we present in this paper our study of sensors in details. We focus on testing different sensors divided into two main groups: motion and positioning sensors for the first group and biological sensors for the second group. For each study, ...
Download
This paper presents a numediart project in collaboration with two artistic projects: Musichoregraphie d’appartement by Andre Serre-Milan [63] and BioDiva by Laurence Moletta (Nios Karma) [34]. The scope of this project was to offer technological forecasting and development consultancy to these two artists that both share a common goal of using gestures to control on audiovisual rendering on stage (see section 2). We developed a first prototype for synchronized recording, visualization and editing of multimodal signals (audio, video, sensors) (see section 3.1). We updated our technologies for gesture recognition using Dynamic Time Warping (DTW) (see section 3.2) and mapping interpolation (see section 3.3). We ported our longterm attention computer vision algorithm from EyesWeb to Max / MSP (see section 3.4). We achieved initial promising results for the prototyping of the BioDiva gestural interface (see section 4.1) and for the analysis using the long-term attention model of the Musi...
Download
Within this portfolio are three pieces which explore the use of video, improvisation and noise in the performance and production of electronic music. The pieces are presented as both fixed media and videos of performances. Also included is preliminary research in the form of two smaller research projects. Video has been used in various forms either as a stimulus to improvisation and composition, as an input for sound control, or more traditionally, as an accompaniment to a composition. The use of improvisation both in the composition and performance of the pieces was also investigated. Noise was used in the composition of the pieces, recorded from field recordings, performed by live instrumentalists or generated by synthesisers. Noise is an important theme in the work and is used to bind sounds together, to create tension and release and to provide a contrast to the more traditional melodic and rhythmic structures.
This research endeavors to expand the idea of electronic music performance and explore different approaches to presenting electronic music in a live context. The aim being to break out of the paradigm of the laptop musician staring at a screen and doing little else whilst performing. For each piece I have explored a different mode of performance. For example in the piece Aberfan (2013) the traditional three piece band line up of guitar, percussion and bass, was mutated, creating instruments from springs, heavily distorting conventional instruments such as double bass and using improvising musicians to accompany a film.
Download
Computer Music Journal, 2010
Download
Despite the wealth of alternative peripherals available, electroacoustic composers often limit themselves to just a mouse and keyboard. This paper outlines various applications of gesture recognition available to electroacoustic composers. This is achieved through analysis of both historical and contemporary applications, as well as the presentation of a new suite of tools developed in Pure Data for the Leap Motion device.
(This dissertation was submitted in partial fullfilment of the requirements for the degree of BMus)
Download
Download
In this article we report the development of a suite of Android musical instruments “Música Móvel” based on combination of LibPD for the DSP and Processing language for the graphical interface. The article shows
the interaction design decision details for the interface of the first eight applications developed to the present stage of work: Looper, D-Sonus, Arvoritmo, Horizons, PhotoSintese, B/I/T/S/L/C, Multigranular and Anartistas.
Download
Organised Sound, 2013
This article describes a manual actions expressive system (MAES) which aims to enable music creation and performance using natural hand actions (e.g. hitting virtual objects, or shaking them). Gestures are fully programmable and result from tracking and analysing hand motion and finger bend, potentially allowing performers to concentrate on natural actions from our daily use of the hands (e.g. the physical movement associated with hitting and shaking). Work carried out focused on the development of an approach for the creation of gestures based on intuitive metaphors, their implementation as software for composition and performance, and their realisation within a musical composition through the choice of suitable mappings, sonic materials and processes.
Download
Download
Task 6.1 is concerned with developing new interfaces and new metaphors for more physical interaction with virtual environments, involving the entire body and its physical properties. The deliverable is divided into three parts: • “A Characterization of Input Devices used in Interactive Installations” develops a taxonomy of how input devices and space have been used in interactive installations. • “Navigation for the Senses”describes several devices for whole-body interaction developed or under development at GMD. • “Some Elementary Gestural Techniques for Real-Time Interaction in Artistic Performances” describes gesture-based interfaces for multimedia performances. Document eRENA-D6.1 Type Deliverable report with video material on eRENA tape Status Final Version 1.0 Date 30 May 1998 Author(s) John Bowers (KTH), Monika Fleischmann (GMD), Sten-Olof Hellström (KTH), Michael Hoch (ZKM), Kai-Mikael Jää-Aro (KTH), Thomas Kulessa (GMD), Jasminko Novak (GMD), Jeffrey Shaw (ZKM), Wolfgang St...
Download
Download
Mongolian overtone singing (höömii) and Mongolian wrestling songs (tsols) are vocal styles that evoke physical and mental strength in the vocalist through the accessing of nature. The phrase “höömii-tsol-thinking computer” conveys my end-goal while composing, performing, and researching for my original composition strong.mng. I wanted to create a work in which the computer would be informed by the performance methods and philosophies employed during Mongolian höömii and tsols.
Strong.mng is a 25-minute production for dancer, live digital illustrator, and overtone singer with a laptop computer serving as both a fixed and interactive responsive musical instrument. The music draws upon themes from höömii and tsols through the lens of virtual fieldwork, which was the research method I used to inform strong.mng. Through the composing and performing of strong.mng, I arrived at the following three-part hypothesis: firstly, the development of a robust symbiotic relationship between höömii, tsols, and today’s electronic music technology may transform the technological devices used into agents of deep ecology and bodily interconnectedness. Secondly, this transformation may metamorphose the performer into a more courageous being who is strengthened both physically and mentally by the Mongolian belief that, when performing höömii and tsols, the musician is drawn into kinship with nature. Lastly, I believe some computer music is restrained in its potential by techno-somatic discreteness as well as anthropocentrism, and that applying philosophies from höömii and tsols can help move computer music more towards a physically embodying means of sonification; one that is also akin with the natural world.
Download
In the absence of a well suited measure for quantifying binaural data variations, this study presents the use of a global perceptual distance metric which can describe both HRTF as well as listener similarities. The metric is derived based on subjective evaluations of binaural renderings of a sound moving along predefined trajectories in the horizontal and median planes. Its characteristics and advantages in describing data distributions based on perceptually relevant attributes are discussed. In addition, the use of 24 HRTFs from two different databases of origin allows for an evaluation of the perceptual impact of some database-dependent characteristics on spatialization. The effectiveness of the experimental design as well as the correlation between the HRTF evaluations of the two plane trajectories are also discussed.
Download
