On Sound

TA 9 (2+3) pp. 215–222 Intellect Limited 2011 Technoetic Arts: A Journal of Speculative Research Volume 9 Numbers 2 and 3 © 2011 Intellect Ltd Article. English language. doi: 10.1386/tear.9.2-3.215_1 C N opy ot r fo igh rd tI is nte tri ll bu ec tio t L n td dimitri Batsis and Xenophon BitsiKas university of Ioannina 20 11 ON SOuNd anastasia GeorGaKi university of Athens anGelos evaGGelou university of Ioannina panaGiotis tiGas university of Bristol Biomusic: the carrier 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. aBstract Keywords This article investigates the concept of sound, in relation to the new means and sciences from different perspectives, ultimately providing an analysis of the newborn artistic movement of bioart. It is divided into two parts. The first part of the study is based upon reference, investigating the interconnection between art and science. This mechanism is characterized by transformation processes in the interdisciplinary practices that are applied mainly by various artists and movements of the postSecond World War period. The expressive element seeks an unworldly explanation through audio and visual conjunctions. This nature is obvious in Paul Klee’s reflections of musical elements in his paintings, and Rimmington’s attempts to marry audio-visual influences in his ‘colour organs’. The experimentations of composers such as Xenakis and Stockhausen at various locations with light and colour illustrate the continuous quest to render sound by the use of new means. Technology is a vital component of transformation as it enhances syncretic creativity for various contemporary art new media bioart biomusic bioart ECG sonification 215 TA_9.2-3_Batsis_215-222.indd 215 4/6/12 5:04:27 PM dimitri Batsis | Xenophon Bitsikas ... C N opy ot r fo igh rd tI is nte tri ll bu ec tio t L n td 20 11 art domains such as those that Fluxus deployed. Nam Jun Paik and Dick Higgins introduce radical techniques in their performances as they detach their selves from the parameters that define composition, and use the mind and power of sentiment in order to identify reality aurally and optically. Towards the end of the twentieth century, we witnessed the appearance of new art forms such as bioart. The human body, host of material and immaterial functions, comes to the forefront of art practice. Its relation to elements such as oscillations and vibrations that express the energy flow is analysed through the model of spiritualism that came from eastern thought. The notion of digital embodiment is presented as a reminder, highlighting the importance of technology in biotechnology and genetics. The second part of the article involves an experiment. This describes how the concept of biomusic is applied with the use of electrocardiography (ECG) data from the MIT PhysioNet database. As sound penetrates the entire human body, it can be analysed in all of its phasma. Using this information, we attempt to translate/transform these biological sound phenomena into music. The sound produced by the elaboration of data that result from biological functions can be described as biomusic. It can be transformed into frequencies related to time and can be expressed in musical themes. Sonification plays an important role in this research as it constitutes a rapid and precise rendering of polymorphic information (in this case the ECG) in musical notes. This modelling and musical attribution leads to two distinguishable results, each concerning different clinical cases (all data belong to a normal heart function and a pathological one). The invention of this novel system is suggested for scientific as well as musical disciplines. It has the ability to be implemented in an experimental form and obtain an educational character. The transformation process avoids compensation throughout the matching process between ECG functions and music, while focusing on the aesthetic factor at the same time. Sound meets art in the world of biomusic as it takes shape through technology, constituting a new medium to further evolve the model of ‘biology into art’ transformation. introduction The marriage of audio and visual is a classic approach in the speech acts of artistic practice. Paul Klee tried to reflect upon his paintings with musical elements (Duchting 2004: 11–12). He used colour inspired by the movement of musical notes on the stave. He inevitably conveyed the spirit of a musician on the canvas. The idea of motion is not a compositional component for Klee, as he aims for the elimination of time. The polyphony in musical creation has responded to this demand to some extent. Klee wants to overcome music’s periodical element through a conscious choice. He wants to escape from the programmatic style that is dominant in classical music composition. He creates polyphonic structures, of higher provision, that constitute free explanations of (virtual musical) scores. Such projects include ‘Fugue in rot’/‘Fugue in Red’ (1921) and ‘Landschaft in A dur’/‘Landscape in A major’ (Duchting 2004: 28–29). Klee’s musical paintings are a combination of scientific observation and artistic creation, with strong abstract elements similar to those defined in the twentieth century. The audio-visual connection between the arts is dominant in various devices that host experimental application by composers and inventors alike. Alexander Rimmington first used the term ‘colour organs’ for his combinatory patent of musical/sonic elements with the art world (Peacock 1988: 397–406). About 50 years later, Rimmington’s innovation led to the invention 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51. 52. 216 TA_9.2-3_Batsis_215-222.indd 216 4/6/12 5:04:27 PM Biomusic 20 11 of the ‘Lumigrah’ by abstract cinema animator Oskar Fischinger. Its function is based upon a collection of images that are projected with the touch of a rubberized screen. Depressing the screen would cause it to intersect with the light. In 1932 Fischinger researched the graphic composition of specific sounds through filmic practice. Thereby he introduced a new component to the conception of image, a contrapuntal mode (in this case, one melody represents image and the other sound). In our attempt to deliver a syncretic approach to creativity, we focus on the Fluxus (Rush 1999: 24) movement. In Fluxus sound acquires energy substance. Vibration and sonification for and from the body are some of its particular characteristics. Nam Jun Paik and Dick Higgins, two artists from utterly different backgrounds, were innovators in the field of sound art. The former transformed the video image from a literal representation of subjects and events into an expression of artistic views of issues/events of the time. Television and the screen generally was one of the main tools/elements for Paik from the beginning of the 1960s. That information (no matter what its content) is transferred to the world through the television or generally the screen. Paik includes it in his work, pointing out its domination over our senses and the control of attention (Mc Luhan 1966: 6). On the other hand, Dick Higgins belongs to the league of artists who approached sound as a real natural substance that sometimes has low intensity and durability. The natural substance of intense sounds that Higgins was keen on could be perceived in different ways: (1) as a sonic energy that is mass-perceived, (2) as a carrier of the material substance of sound that overturns the natural substance of sound or (3) as a means to pervade and define space and others. The research on sound is part of the ‘sonic art’ legacy that composers such as Xenakis and Stockhausen have left us. Karlheinz Stockhausen created entirely electronic music for his project ‘Contact’ (Kontakte, 1958–1960), signifying a new era in audio control. He takes a multidimensional approach to space issue and absorption of sound. The project is characterized by the composer’s need to include all the sound elements (pitch, colour, intensity, duration, etc.) with a single manipulation (Clarke 1998: 222). The composition involves groups of classical and electronic instruments that contribute to a transformative process, a kind of contact between groups separated by sound categories (metal, leather and wood) and electronic sound. The spatial distribution of four different speakers is the composer’s attempt to bring the listener into contact with more than one dimension, a kind of contact with several aspects of reality. Iannis Xenakis’ quest to explore the idea of general morphology (combinatorial forms of various disciplines that meet in the field of arts) establishes the substantial application of mathematics. These disciplinary fields form two mechanisms: The first mechanism deals with theories that encompass all the scientific fields and the second mechanism belongs to a complete theoretical approach mechanism that relates to the conclusion. Questions and verification of these theories are presented and investigated in an experimental apparatus. Nonetheless, art suggests a third mechanism revealing the link between integrated and experimental methods, where the ‘art object’ obtains dimensions, following mysterious paths, combining the cores or elements from both mechanisms (Xenakis 1992: 4). The third mechanism that Xenakis refers to is evident in his projects, known as ‘polytopes’ (the term has a mathematical significance too, existing in Euclidian geometry). Polytopes are large-scale multimedia works that take place in specific C N opy ot r fo igh rd tI is nte tri ll bu ec tio t L n td 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51. 52. 217 TA_9.2-3_Batsis_215-222.indd 217 4/6/12 5:04:28 PM dimitri Batsis | Xenophon Bitsikas ... C N opy ot r fo igh rd tI is nte tri ll bu ec tio t L n td 20 11 locations (usually of historical archaeological significance) and involve music performance and light. The human body, host and creator of energy, plays a special role in the multimedia combination of art and science. Digital embodiment is part of this process in the arts. Stelarc’s main idea is that the human body is obsolete (1994). The idea of interaction varies between the two poles: the anthropomorphic machine on one end and the technological transfer of the organic element through digital technologies on the other. It could be argued that all the approaches that relate directly or indirectly to life and have some kind of application in the biotechnology laboratories are part of what constitutes bioart. Science has been moving in complementary ways to art for centuries, including recently, with the introduction of biotechnology into the arts. The mix of eastern/western and holistic/analytical-technocratic thinking contributed to a multi-angular approach to human nature. The informatics that supports biotechnology became a craftsperson’s tool. According to Whitelaw, especially biotechnology involves technologies such as genetic engineering, tissue culture and cloning, while it produces results that are the source of inspiration for those occupied with the subject (2004: 12). Bioart rather suggests that any future outcome for embodiment in the field of informatics should leave some space for the aesthetic processes of composition. The term bioart, an invention of artist Eduardo Kac for his work ‘Time Capsule’ in 1997 (Kac 2000: 243–49), and its derivatives, such as biomusic, belong to what could be described as the next level of syncretic creativity. It is about a technoetic evolution, where the self comes to the forefront through generative arrangements and processes. The self is shaped through new dimensions of consciousness. This transformative mechanism is the framework in which our case study is taking place. case study As part of our research we designed and conducted an experiment during which we applied the concept of biomusic on electrocardiography (ECG) data. In our experiment we transformed data retrieved from PhysioNet into music using simple computational procedures. We mainly focused on sonification of cardiac arrhythmia. Arrhythmia is a medical term describing heart rhythms that differ from normal sinus rhythms. The main difference is that in contrast to normal sinus rhythm, RR intervals show irregularities and variations of the heartbeat. The dataset created for the purposes of 2001 Challenge of the conference Computers in Cardiology, and consists of recordings of 48 different subjects. This dataset was divided into a learning set and a testing test for the task of atrial fibrillation (AF) onset prediction using machine-learning methods. Our main goal was to create a sonification process of AF, which is one of the most common cardiac arrhythmias. Our hypothesis is that the aesthetic result of the final musical piece correlates with the health of the heart; thus, a normal heart will have a more pleasant result than an arrhythmic heart. In our approach we use RR intervals and ∆RR intervals (difference of two successive RR intervals). In the literature methods exist for the prediction of AF using these features (Tateno and Glass 2000: 391–94). Thus, we strongly believe that our approach captures and sonifies statistical properties of normal and arrhythmic hearts. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51. 52. 218 TA_9.2-3_Batsis_215-222.indd 218 4/6/12 5:04:28 PM Biomusic The algorithm/method we used to produce biomusic is summarized in the following section. alGorithm Step 1. We retrieve data from PhysioNet. Step 2. For a given recording we retrieve the corresponding RR intervals. Step 3. Note pitch computation Step 4. Note duration computation We compute ∆RR with the following formula: 20 11 For the note pitch computation we use RR intervals that we map to integers of range [0,n](n is a parameter of our choice with which we control the note pitch range). Those integers are then mapped to MIDI notes and we audiofy them by using audio sequencers. C N opy ot r fo igh rd tI is nte tri ll bu ec tio t L n td 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51. 52. ∆RR(t)=R(t)-R(t-1),t>1 A i we normalize ∆RR so that 0 ≤฀∆RR(t) ≤1, ∆RR(t)=฀ RR(t) max ∆฀RR We create nine bins as follows: No. bin Note value 1 2 3 4 5 6 7 8 9 64 32 16 8 4 2 1 0.5 0.25 Range [0,0.111) [0.111,0.222) [0.222,0.333) [0.333,0.444) [0.444,0.555) [0.555,0.666) [0.666,0.777) [0.777,0.888) [0.888,1] Table 1: Note value bins. Each bin corresponds to a note value. For example, the first corresponds to the 64th note and the ninth to a longa. Then we map the scaled ∆RR to these bins in such a way that values from 0 to 0.111 are mapped to bin 1 , from 0.111 to 0.222 to bin 2 , etc. More formally, the duration of note i is: duri=duri–1+durations(floor(∆RR(i . 8)).notelength where notelength is a parameter that we choose arbitrarily and floor(x) is the floor function (largest integer not greater than x). 219 TA_9.2-3_Batsis_215-222.indd 219 4/6/12 5:04:29 PM dimitri Batsis | Xenophon Bitsikas ... Step 5. Note dynamics For the note dynamics we use the heart beat dynamics. The reason for this is that we want the transformation procedure to be as simple and bias-free as possible. Step 6. Track tempo The result of this process is a series of notes that have duration, are dynamic and are pitch-assigned. The only part missing is the tempo of the track. For this we compute the mean value of RR intervals. conclusion C N opy ot r fo igh rd tI is nte tri ll bu ec tio t L n td 20 11 Although the sonic result is a series of notes that seem to obey rules, it takes the essence of contemporary composition a step further. The main feature of twelve-tone composition is that all twelve notes of the chromatic scale are of equal importance, and thus none of them is being given more or less importance in a music piece. The use of tone rows is significant as the target of twelvetone composition is to avoid writing a piece in a specific key. Affiliations to our composition (having the composition made with note combinations that fall outside of classic harmony rules) can be made to a certain extent. The listener is not prepared for a compositional thought that relates to even and equal use of notes as well as tonal pitches. As a result we do not bias the listener towards an aesthetic judgment. For example, by mapping the ECG to western scales that are familiar to the listener, we would restrict the musical outcome in the sense that we already maximize the possibilities of a familiar listening in its traditional sense. ECG is a biological signal, an imprint that carries data that can be sonified. Sound carries information that can be mapped/translated through the use of variable techniques. An ECG is a primal example of music. It refers to nature primarily. The pulse is energy; its presence is obvious from archaic shaman rituals to music/art and science practice encompassing holistic/ eastern and western thought at once. references Batsis, D. (2012), ‘Investigation on sound as a new way of expression in contemporary art. Biomusic’, Ph.D. thesis, Ioannina: Ioannina University. Clarke, M., (1998). ‘Extending contacts: the concept of unity in computer music’, Perspectives of New Music, 36 (1), pp. 221–246. Duchting, Hajo (2004), Paul Klee: Painting Music (trans. Penelope Crowe), London: Prestel Publishing, pp. 11–12, 28–29. Kac, E., (2000), ‘Time capsule’, AI & Society, 14 (2), pp. 243–249. Mc Luhan, Marshall (1966), Understanding Media, New York: Signet, p. 6. Peacock, K., 1988. ‘Instruments to perform color-music: Two centuries of technological experimentation’, Leonardo, 21 (4), pp. 397–406. Rush, Michael (1999), New Media in Art, London: Thames & Hudson, p. 24. Stelarc (1994), ‘Obsolete body’, http://www.stelarc.va.com.au/obsolete/ obsolete.html. Accessed 15 January 2010. Tateno, K. & Glass, L., (2000), ‘A method for detection of atrial fibrillation using RR intervals’ in Computers in Cardiology, Cambridge, MA. Whitelaw, Mitchell (2004), Metacreation: Art and Artificial Life, Cambridge, MA: MIT Press, p. 12. Xenakis, I. (1992), Formalized Music: Thought and Mathematics in Composition, New York: Pendragon Press, p. 4. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51. 52. 220 TA_9.2-3_Batsis_215-222.indd 220 4/6/12 5:04:30 PM Biomusic suGGested citation Batsis, D., Bitsikas, X., Georgaki, A., Evaggelou, A. and Tigas, P. (2011), ‘Biomusic: The carrier’, Technoetic Arts: A Journal of Speculative Research 9: 2+3, pp. 215–222, doi: 10.1386/tear.9.2-3.215_1 contriButor details Dimitri Batsis completed his BA in Music at Anglia Ruskin University in 2002 and his MA in Contemporary Arts and Music at Oxford Brookes University in 2003. He received his Ph.D. in Biomusic from the Fine Arts and Sciences of Art School at the University of Ioannina. His research covers a range of disciplines including contemporary art, music composition, sound art, and recently bio-informatics and sonification. His interest focuses on new technologies and media applications focusing primarily on sound. 20 11 Contact: 2, Kapetan Lepenioti, Ioannina, 45332 Greece. E-mail: dbatsis@gmail.com Xenofon Bitsikas is an artist and Associate Professor at the Department of Fine Arts and Art Sciences , University of Ioannina. He studied at the University of Athens School of Fine Arts and completed his Ph.D. at the School of Fine Arts UCMadrid. His work has been awarded and exhibited in Greece and Europe. He has been working on the idea of limit and its relation to the spectator, considering the artwork as a two-way transitorial space, redefined by both artist and spectator. The field of his research and academic curriculum is centered around questions of systems-mechanisms-tools used for structuring space with a specific side interest on matters of new media. He has published papers in congress proceedings and reviews while collaborating with Greek (DUTH) and European (UCMadrid) universities. C N opy ot r fo igh rd tI is nte tri ll bu ec tio t L n td 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51. 52. Contact: School of Plastic Arts and Sciences of Art, University of Ioannina, Ioannina, 45510, Greece. E-mail: xbitsikas@cc.uoi.gr Anastasia Georgaki studied Physics (University of Athens, 1986) and Music (accordion, piano, harmony, counterpoint/Hellenic Conservatory of Athens, 1981–1990). She continued her studies at IRCAM (Paris, 1990–1995) in computer music and music technology (DEA and Ph.D. in Music and Musicology, IRCAM/EHESS). During the period 1995–2002 she has been teaching as a lecturer in Music Acoustics and music technology at the Music Department of the Ionian University at Corfu. Since 2002, she is has been lecturing and currently she is an Assistant Professor in Music Technology at the Music Department of the University of Athens. Since 2008 she has been teaching at three different Master programs at the University of Athens and the School of the Fine Arts (applications of new technologies in music creation). She is also supervisor of Ph.D. candidates on the area of vocal analysis and new media. Contact: Laboratory of Music Acoustics ad Technology, Music Department, University of Athens Athens, 15784, Greece. E-mail: georgaki@music.uoa.gr 221 TA_9.2-3_Batsis_215-222.indd 221 4/6/12 5:04:30 PM dimitri Batsis | Xenophon Bitsikas ... Angelos Evaggelou is an Associate Professor and Head of the Laboratory of Physiology at the University of Ioannina Medical School, Greece. He received his Medical degree in 1971 and a Ph.D. in 1978 from the Faculty of Medicine of the University of Athens. He also specialized in Internal Medicine at the University Hospital ‘Alexandra’, Therapeutic Clinic of the University of Athens in 1975 and in Physiology at the Karolinska Institute of Clinical Physiology, the Rayne Institute, London and the Organ Physiology Unit, INSERM U-200 in 1984 and 1989. He worked as a Registrar in Internal Medicine Clinic of the General Hospital of Ioannina, before joining the Laboratory of Physiology as a Lecturer and then as an Assistant Professor of Physiology. His research concentrates on the use of vanadium compounds in the prevention and treatment of experimental cancer. 11 Contact: Faculty of Medicine, Laboratory of Physiology, University of Ioannina, 45110 Ioannina, Greece. E-mail: aevaggel@cc.uoi.gr C N opy ot r fo igh rd tI is nte tri ll bu ec tio t L n td 20 Panagiotis Tigkas studied Computer Science in University of Ioannina. After that he pursued his studies in Machine Learning and Data Mining in University of Bristol where he graduated with distinctions in 2011. He is a research collaborator at the same university, at department of Engineering Mathematics and his research is focused on Music Information Retrieval, Machine Learning and Theoretical Computer Science. Contact: 89B Shirland Road, London, W92EL, UK. E-mail: ptigas@gmail.com Dimitri Batsis, Xenophon Bitsikas, Anastasia Georgaki, Angelos Evaggelou and Panagiotis Tigas have asserted their right under the Copyright, Designs and Patents Act, 1988, to be identified as the author of this work in the format that was submitted to Intellect Ltd. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51. 52. 222 TA_9.2-3_Batsis_215-222.indd 222 4/6/12 5:04:31 PM