The "Piano-of-Lights": a visual-music instrument for FullDome display

The “Piano-of-Lights”: a visual-music instrument for FullDome display Yan Breuleux, Rémi Lapierre Université du Québec à Chicoutimi (UQAC-NAD) NAD - École des arts numériques, de l’animation et du design Montréal, Canada ybreuleux@nad.ca, remi.lapierre1@uqac.ca Abstract Les Planètes is a “FullDome” immersive concert generated entirely in real time. The concert is made possible by the “Pianoof-Lights,” an instrument that enables a visual dialogue with pianist Louise Bessette during a novel performance of the work Les Planètes from composer Walter Boudreau. The Piano-ofLights’ dynamic constellations of small spheres are projected across the space of the satosphere of the Society of Art and Technology (SAT) in Montreal. The visual shapes are based on an analysis of the different sections of the score. During this process, the Piano-of-Lights emits light particles, becoming a catalyst for the dynamic constellations distributed across the dome. The project blends the tradition of colour organs and the live A/V approach of digital technology, bringing together an analogue piano, digital recording and immersive projection. The originality of this research primarily lies in the use of a colour-piano, which was designed to create a synthesis of the colour-sound association in an immersive format. Keywords Sound and Music Computing, Live performance, Colour Organ, Virtual Colour Organ (VCO), Visual Music, FullDome, Environmental Storytelling, Immersive Media, Virtual Reality. Introduction The first clavecin oculaire or “Ocular Harpsichord” was invented by Father Castel to address criticism of the overly static dimension of painting. Starting in 1725, and throughout his life, Castel worked on designing a colour harpsichord that would make for “hearing with the eyes” and thus creating visual music [1]. As Kenneth Peacock has noted, the 18th and 19th centuries gave rise to much experimentation shaped by the colour theory of Isaac Newton [2]. It is worth pointing out that, in spite of his observations, Newton strived to align his theory with the notes in the musical octave [3]. The modern period, building on Father Castel, would be marked by the work of the master in this field, Alexander Wallace Rimington, who designed multiple versions of his light organ. Influenced by this research, Alexander Scriabin composed a synesthetic symphony, Prometheus or the Poem of Fire, which made its own light-music associations. The use of colour organs supported the composer’s artistic and spiritual vision, and tended toward a Total Art project. According to Françoise Roy-Gerboud: “In this composition, he [Scriabin] uses the possibilities of all the arts, such as music, visual arts (colours and the art of miming), choreography and poetry” [4]. Figure 1: Three centuries of colour scales. © 2004 Fred Collopy - RhythmicLight.com (author permission) Scriabin’s aesthetic project was consistent with a synesthetic vision of performance involving a dynamic union of the senses of sight, smell, touch and hearing. The futurists would inaugurate the art of noise by building their own tools, such as “chromatic music” or Russolo’s “rumorarmonio.” In 1922, Raoul Hausmann devised his “Optophone” to be able to electronically transform light into sound waves and vice versa. In his own words, “Light is vibrating electricity, and sound is also vibrating electricity” [5]. Another striking instrument to emerge from continued experimentation was the famed organ-like “Clavilux” by Thomas Wilfred. Each section of the mechanical invention contained a series of sliding keys divided into three groups: shapes, colours and movements. A complex arrangement of lenses helped create movement animated by different rhythms. The device could be adapted to a musical score, and each light composition could be played on the instrument with a degree of artistic latitude [6]. The tradition of colour organs was continued in the field of analogue technology-driven video art with the creation of multiple devices that responded to sound amplitude and frequencies. Gene Youngblood, in his book Expanded Cinema, discusses this topic at length in a chapter entitled “Television as a creative medium.” Of Nam June Paik, he writes: “In recent years Paik has abandoned his mixed-media environmental Happenings to concentrate exclusively on television as an aesthetic and communicative instrument” [7]. A musician by training, Paik always treasured music and the notion of instrument in his conception of art. This initiator of the Fluxus movement was naturally attracted to different forms of music visualization [8]. Using the video collage to this end, Paik adopted an intermedia approach in his projections, leading him to continually work at the frontier of all the arts [9]. The current of colour organs is vast, and is reflected in many trends of visual music. It is important here to emphasize that artists, developers, designers and researchers have never ceased working on this project of associating sound and light by means of the invention Purform duo, in collaboration with composer Alain Thibault, has been working on visual synthesizers via several projects such as Black Box (2003) and After Dark (2007). The project Les Planètes, based on a piece by Walter Boudreau, is a first attempt at systematizing the colour organ using a FullDome installation [21]. Les Planètes: score by Walter Boudreau Figure 2: Black Box. Usine C, Ekektra Festival, Montreal. © 2003. Purform: Yan Breuleux, Alain Thibault of synesthetic instruments. In an article recounting the history of the colour organ phenomenon, Kenneth Peacock concludes with the following reflection: “Although experimenters during the past two centuries hardly could have anticipated today’s widespread use of laser light in combination with electronic computers, these marvellous inventions are, in some ways, refinements of earlier technological proposals for a viable colour-music instrument. Every generation, it seems, must re-discover and re-define the art of colour-music for itself” [10]. Indeed, the generation of digital art and A/V performance or Live A/V [11] artists has reinvented certain aspects of the light organ approach. The research of Golan Levine, in the context of his M.S. thesis at the Massachusetts Institute of Technology (MIT), has precisely examined such efforts. Levine abundantly refers to the colour organ current in connection with the conception of a real-time visual creation interface [12]. Along these lines, artist Edwin van der Heider [13] in the context of the Ondulations project from the artistic duo The Users [14] uses light to give visibility to musical structures. The Rutt/Etra raster-based analogue synthesizer (1973) used by artists Steina and Woody Vasulka has been reproduced in the shape of a plugin by V002 (Anton Marini alis vade & Tom Butterworth) [15]. Another example is the Simple Harmonic Motion project from the artist Memo Atken. Even if the performances do not specifically involve the use of a keyboard, this project clearly entails visualizing certain structures of musical rhythms [16]. The simulation is in keeping with the whole current of emulators or “soft synth” in the audio field. As part of the Canadian and international community of artists, and more specifically those gravitating around the Technical Arts Society (TAS) [17] and the Mutek [18] and Elektra [19] festivals, many artists such as Mathieu Lesourd, LouisPhilippe St-Arnaud, Éric Raynaud, Ouchhh Studio, Francis Théberge, Joanie Lemercier, Abstract Birds, Matthew Biederman, Kurt Hentschläger, Ryoji Ikeda, Herman Kolgen, Ryoichi Kurokawa, Ulf Langheinrich, Carsten Nicolai and more are taking an interest in sound visualization [20]. The Piano-of-Lights strives to contribute to this flourishing community of practice, which is carrying out research-creation projects as a means of studying different forms of sound visualization. However, even if this community is exploring the medium of immersion in relation to issues of sound visualization, it has not directly addressed the specific question of designing colour organs. Since 2003, the Composed and commissioned by Louis-Philippe Pelletier in 1983, Les Planètes was created on May 21, 1998 in Montreal’s Bon-Pasteur Historic Chapel after a long and winding journey. The realization of the piece was jeopardized several times over the years for a host of reasons beyond the composer’s control. The piece holds a unique place in the artist’s works as it was composed over a very long period, with a 13-year “gap” from conception to final completion. As with the entire series of works that come under Le Cercle Gnostique, the piece is essentially described as an exploration in which characteristics of the nine main planets of the solar system (Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and *Pluto) are transposed into music. Partly “calculated” long ago using a CYBER 7400 computer from the Université de Montréal computing centre, the music is exclusively informed and developed based on structural premises that embrace a vast sonic and psychological range in which the delicate balance between form and substance, expression and impression(s), is both the theme and the object [22]. The Piano-of-Lights instrument In his article on the history and origins of colour organs, Teun Lucassen gives the following definition: “A colour organ can be defined as an instrument which is able to simultaneously produce colour and music. The produced colours are based on the music by a certain mapping” [23]. The Piano-of-Lights transforms the score into an immersive format based on an analysis of the composer’s vision for FullDome. It translates the abstract structures at the heart of Walter Boudreau’s composition into virtual environments. Louise Bessette’s playing is amplified so as to transform the energy of her performance into an immersive experience. The Piano-of-Lights is an instrument that creates worlds, generating multiple virtual spaces that enable the audience to see, feel and live the sounds in their material dimension. In this context, the sounds become objects materializing in sound visualization environments. The visuals are generated by the score, with the shapes forming abstract environments. The audiovisual objects offer the audience an open work in which they can imagine their own narrative space. The Piano-of-Lights uses the 88 keys of the Yamaha Disklavier concert piano to generate environments of animated particles, to bring various stage elements to life, and to modify the lighting. The instrument explores multiple forms of sound visualization, with a programming structure that allows it to bring visibility to multiple scores. The principle is always the same: to design procedurally generated environments that respond to the actions of the pianist. The Piano-of-Lights uses the entire colour scale from the Fred Collopy chart entitled “Three centuries of colour scales” [24] (Figure 1) in order to contribute to the re- search on what Jack Ox is dubbing the “Virtual Colour Organ” (VCO). The goal is to imagine a “21st century virtual reality colour organ” [19]. According to Ox, the VCO can be defined as a visualization instrument: “The Virtual Colour Organ™ (VCO) is a 3D immersive environment in which music is visually realized in coloured and image-textured shapes as it is heard” [25]. Our project is based on the Touch Designer software programming environment. The goal is to contribute to the VCO field by means of a study on the spatialization of visual representations of piano notes within a Full Dome projection environment. Environmental storytelling for FullDome Environmental storytelling is based on a complex structure of interconnected virtual environments. The story emerges from the sensory aesthetics and the colour visualization of each planet. The objective is to translate Henry Jenkins’ conceptual framework, Marie-Laure Ryan’s narrative architecture [26], and the idea of spatialized narrative [27] in the service of the expressive potential of the FullDome medium. Our goal is to add the FullDome 3D space to the visual music colour organ tradition. The notes of the scores are three-dimensional objects that move around the dome surface or form objects. For example, depending on the note and the octave, some pulses produce the rings of Saturn. Each environment corresponds to the score’s narrative logic while creating note-colour associations. The visual music structure of the experience In the context of the Piano-of-Lights, I designed a system that allows for rapidly putting in place different forms of mapping. The system is able to incorporate any association. This will make it possible to work with researcher-artists and composers in order to create various forms of association between the colours and the notes of the octave. The project is thus located between the artistic approaches of sound visualization and the tradition of colour organs. The correlations are sometimes immediate and directly correspond to the instrument’s pulses, and sometimes less direct, responding to various logics. This said, the environments are assembled as follows. The first environment [Mercury] demonstrates the piano’s system. Playing with the keys generates light shapes consistent with the Newton colour palette [28]. The second environment [Sun] portrays an eclipse, with the pianist’s actions producing rays of light. The third environment [Re-Sun] projects a space of light turbulence. The fourth space [Venus] is References [1] Anne-Marie Chouillet, “Le Clavecin oculaire du P. Castel,” Les Jésuites: Dix-huitième Siècle 8, (1976): 141-166. [2] Kenneth Peacock, “Instruments to perform color-music: Two centuries of technological experimentation,” Leonardo 21, no.4, (1988): 397-406. Figure 3: Les Planètes. Society for Arts and Technology (SAT). © 2018 Yan Breuleux made up of spheres concentrically embedded in one another. The particles in the middle respond to the higher octaves whereas the vast spheres forming the circumference are activated by the lower ones. The [Earth] environment has the performer make a drawing in space, in sync with the inflections of the music. Then, [Jupiter] associates colours with the seven octaves of the musical scale, which are spatialized in the SATosphere. In the next environment [Saturn] (Figure 3), the generated notes are transformed into the rings of Saturn. The piano generates a column of light using a Remington colour palette. After an intense experience, [Uranus] scatters notes against stellar space, turning stars into sounds. Almost at the end [Pluto] displays the score by controlling light emissions. Each scene is the result of generating principles that bring visibility to the space of the score. The piece closes with infinite space as the audience is drawn into a chaotic interstellar void. The ISEA demo will display the piece inside a virtual environment using the HTC Vive Headset. We are showing our previsualization system. To control the environments with a midi score, the Disklavier piano is replaced with a computer. Our latest production, Enigma, takes up the tools of the Piano-ofLights to create an A/V performance. The principle remains unchanged; midi input is associated with objects, which, taken together, form environments. This approach, introduced in 2003 with the Purform group and the Black Box project, leverages an iterative process based on the production of multiple objects in order to produce a visual synthesizer for immersive formats. Thus, consistent with the perspective of environmental storytelling composition, the system allows for adapting the project to a wide array of projection setups such as architectural video projection, multi-screen, poly-screen, virtual reality and FullDome. The overall idea is to design synthetic environments that serve as a springboard for experiences in a wide variety of contexts and installations. [3] Alan E. Shapiro, “Artists' colors and Newton's colors,” Isis 85, no. 4, (1994): 600-630. [4] Françoise Roy-Gerboud, La musique comme Art total au XXe siècle. Son-couleurs-formes. Systémique et symbolique (Paris, L’harmattan: Univers Musical, 2009), 130. [5] Jean-Noël Von der Weid, Le Flux Et Le Fixe : Peinture Et Musique (Paris: Fayard, 2012), 29. [6] Fred Colopy, “Color, form, and motion: Dimensions of a musical art of light,” Leonardo 33, no. 5, (2000): 355-360. [7] Gene Youngblood, and Richard Buckminster Fuller, Expanded cinema (New York: Dutton, 1970), 302. [8] Holly Rogers, “The Unification of the senses: intermediality in video art-music,” Journal of the Royal Musical Association 136, no. 2, (2011): 399-428. [9] Man-Yong Song and Kim Chee-Yong, “Research on the Visualization of Music and Hypermediacy in Paik Nam-June's Video Art,” Journal of Korea Multimedia Society 10, no. 12, (2007): 1687-1697. [10] Kenneth Peacock, “Instruments to perform color-music: Two centuries of technological experimentation,” Leonardo 21, no.4, (1988): 397-406. [11] Hors-Série, “LIVE A/V Performances Audiovisuelles,” MCD, Musiques et Cultures Digitales, (2010). [12] Golan Levin, “Painterly Interfaces for Audiovisual Performance,” (M.S. Thesis, School of Architecture and and Planning, Massachusetts Institute of Technology, 2000). [13] Edwin van der Heide, “Lsp (2003)”, Artist website, accessed April 13, 2019, http://www.evdh.net/lsp/ [14] Thomas McIntosh and Emmanuel Madan, “Ondulations (2002)”, Artist website, accessed April 13, 2019, http://www.undefine.ca/en/projects/ondulation/ [15] Anton Marini alias vade and Tom Butterworth, “Rutt-Etra plugin (2009)”, Artist website, accessed April 13, 2019, http://v002.info/plugins/v002-rutt-etra/ [16] Memo Akten, “Simple Harmonic Motion #5 (2011)”, Artist website, accessed April 13, 2019, http://www.memo.tv/portfolio/simple-harmonic-motion-5/ [17] Society of Arts and Technology, “Les Planètes (2018)”, accessed April 13, 2019, http://sat.qc.ca/fr/planetes [18] Mutek Festival, accessed April 13, 2019, http://www.mutek.ca/ [19] Elektra Festival, accessed April 13, 2019, https://www.elektrafestival.ca/ [20] Il est possible de consulter de nombreux projets sur le catalogue de la Society of Arts and Technology (SAT). Online page, “Immersive Catalog”, Society of Arts and Technology, accessed April 13, 2019, http://sat.qc.ca/catalog/ [21] Online page, Documentation and credits of “Les Planètes (2018)”, Society of Arts and Technology, accessed December 10, 2018, http://sat.qc.ca/fr/planetes [22] Description from Walter Boudreau (with author permission). [23] Teun Lucassen, “Color Organs,” University of Twente [interaktyvus], (2008). [24] Fred Collopy, “Playing (with) color,” Glimpse 2, no.3, (2009): 62-67. [25] Jack Ox and David Britton, “The 21st century virtual reality color organ,” IEEE MultiMedia 3, (2000): 6-9. [26] Henry Jenkins, “Game design as narrative,” Computer 44, (2004): 53. [27] Marie-Laure Ryan, Kenneth Foote, and Maoz Azaryahu, Narrating space / spatializing narrative: Where narrative theory and geography meet (Ohio State University Press, 2016), 266. [28] Society of Arts and Technology, Online documentation of “Les Planètes (2018)”, vimeo album, accessed December 10, 2018, https://vimeo.com/album/5622492 Author(s) Biography(ies) Yan Breuleux is a professor at NAD-UQAC (École des arts numériques, de l'animation et du design à Montréal). He is also a researcher and practitioner in the field of visual music for immersive display. For 20 years, he has collaborated with musicians and composers to create multi-screen, panoramic, architectural projection and FullDome pieces. Since 1998, he has created powerful sensorial A/V performances as part of the PURFORM duo with the composer Alain Thibault. Rémi Lapierre has a solid background in programming, graphic novelling, 3D modelling and video games. His multidisciplinary skill set gives him an in-depth understanding of advanced technical issues in software design as well as the expressive dynamics of digital creation. Rémi is currently devoting his efforts to automated virtual environment generation. He is a master’s degree student at the NAD (École des arts numériques, de l’animation et du design).