Proceedings of the International Conference on the Multimodal Experience of Music 2015 Inside-Outside: 3-D music through Tissue conduction Ian McKenzie1, Peter Lennox1, Bruce Wiggins1 1 University of Derby Eliciting an auditory perception by means of mechanical transduction bypassing the peripheral hearing apparatus has been recorded as early as the 16th century. Excluding its audiometric use to assess ear pathology, bone and soft tissue conduction has received very little interest until the last two decades. Previous work during this time (Stanley and Walker 2006, MacDonald and Letowski 2006) has indicated robust lateralisation is feasible via mechanical transduction. This paper reports on an extension to this work, adding the front-back and up-down axes. Keywords: 3-D Music, bone conduction, soft tissue conduction, haptic cues, multimodal. Background Eliciting an auditory perception by means of mechanical transduction has been in use for hundreds of years. Several transmission pathways may be employed to deliver an audible experience, these pathways are often collectively referred to as bone conduction; as soft tissues and cerebrospinal fluid may feature significantly in these we refer to them as tissue conduction (TC). Four pathways out of several have been identified in numerous research studies as primary conduction pathways. 1) Inertial movement of the ossicular bones relative to the skull at low frequencies 2) Distortion of the temporal bone and cochlear shell at high frequencies 3) Osseo-tympanic transmission of sound radiated from the walls of an occluded ear canal 4) Sound conduction via fluid pathways connecting the cochlea to the brain cerebrospinal fluid “The resultant sound level at the cochlea is a frequency-dependent vector sum of the contributions from each of these transmission mechanisms.” Dietz et al. (2005). Scepticism prevailed about tissue conduction’s ability to convey the same localisation cues as air conduction. Stanley and Walker (2006) discuss experiments that indicate that, using inter-signal variations at two tactile transducers in contact with the head, lateralisation performance equivalent to that in binaural lateralisation experiments is feasible. MacDonald et al. (2006) using similar apparatus but alternative locations showed lateralisation performance that was almost identical to those utilizing stereo headphones. Stenfelt & Zeitooni (2013) found a benefit in twenty normal hearing participants of bilaterally applied transducers indicating coherent use of binaural cues. 002 Aims Having established the feasibility of coherent lateralisation control via tissue conduction, a greater range of auditory spatial experiences that might be available was explored. These could include localisation of sources and features within a sphere surrounding the perceiver, front/back discrimination, externalization, elevation, range perception and movement perception (including auditory looming, Seifritz et al 2002). Other attributes might include environment shape, spaciousness, enclosed-ness, surface textures, and ‘clutter’. Method A multiple transducer array was designed, constructed and used to present tissue conducted audio signals via the head; initial experimentation would investigate the feasibility of using multiple transducers. Five fairly equidistant transducers were used in the array; the placement of these may afford a degree of control of left/right, front/back and height localisation. Data from a study conducted by McBride et al. (2005) comparing the sensitivity of eleven locations on the skull assisted in the choice of transducer placement. The five locations chosen here had been found to be reasonably equivalent in terms of attenuation for the frequencies in our investigation. Figure 1. Headset Array and Transducer locations. 1 – Left Mastoid, 2 – 1” above Left Temple, 3 – Between Forehead and Vertex, 4 – 1” above Right Temple, 5 – Right Mastoid. Dayton Audio BCT-1 tactile transducers were used, 8 Ohms impedance, 1 Watt RMS and reported frequency response 300 – 19 kHz. Each transducer had its own discrete signal feed and contact with the head made through a 16mm hemi-spherical hard plastic medium. Proceedings of ICMEM, 23-25 March 2015 003 Subjective Testing Lateralisation performance having previously been established, a short series of tests were conducted to consider any degree of externalization, elevation or phantom image control using a combination of signal attributes and transducer location. The tests were of a qualitative exploratory nature and are discussed in McKenzie et al. (2014). Informal Testing Informal testing into how musical stimuli and recordings of real environment sounds might be manipulated and presented yielded positive results. The signals were processed using Reaper digital audio workstation and a range of plugins. The same signals were also presented Ambisonically using WigWare (Wiggins 2014); additionally some 1st order B-format recordings were used. Ambisonics was included as a convenient method to manipulate the presented audio as it possesses many useful attributes. In 2014 the headset was on demonstration for three days at the National Exhibition Centre during the Institute of Acoustics conference; 35 participants took place in the demo providing positive feedback. Results Lateralisation was experienced when presented bilaterally and easily manipulated using amplitude panning, similar to that of headphones; the ears remain un-occluded providing ‘openness’ and a very different listening experience, often commented by test subjects. When presented via multiple locations, amplitude manipulations alone produced limited image movement. Using amplitude and delay modifications a greater level of image control, elevation and a degree of externalization was realized. Good separation with a widened image was achieved with modified stereo presentation, some degree of externalization was experienced and in some cases height and range perception comments were made. Ambisonic presentation, despite the transducer patch having been empirically derived, provided positive feedback. Acceptable image control was experienced in the main although frontal presentation of externalized sound was poor. Elevation panning was possible with fairly smooth control and reasonable height achieved. To most participants sound presented via tissue conduction was a novel (and for some, initially confusing) experience. After a short adjustment period most were able to make sense of spatial separation and appreciate a degree of phantom image control and externalization. 004 Discussion We note the anomaly that elevation, perceptible in normal hearing conditions due to pinnae filtering is achievable despite the absence of the outer ear contributions, possibly due to phase interference arising out of multiple signal paths exhibiting fine timing differences. An intriguing possibility is that of multimodal cueing; conventional binaural auditory cues merging with additional information provided through the somatosensory system via haptic cues. Somatic information can alter perceived auditory localisation; while investigating whether perceptual auditory localisation on the median plane could be altered by body vibration, Tajadura-Jiménez et al. (2007) suggest a multisensory integration of the sensory information. Their findings indicate that vibro-tactile information concurrent with a sound source affects the perception of localisation on the median plane. Meredith & Stein (1986) examined the convergence of inputs from different sensory modalities in the superior colliculus concluding that multisensory integration might represent a basic mechanism by which the brain integrates complex environmental stimuli, profoundly influencing perception. “The superior colliculus is a structure intimately involved in attending to, localizing, and orienting to sensory stimuli.” (p.641). Several studies consider key elements of somatoperception, one element is that of remapping information from the body surface into an egocentric reference frame (Longo et al. 2009). With regard to body schema and spatial orientation Lackner (1988) showed, in a dark room, the vibration of certain muscles can produce apparent displacement and motion. If the subject were to hold the tip of their nose while a vibrational stimulus were applied to the biceps they would perceive their nose to grow in length as the forearm extends away from the face. Although the lengthening of the nose is an illusion it shows that the anatomy and dimensions of the body are taken into account by the perceptual interpretation of the afferent signal. It is known that sensory modalities interact, that they functionally reorganize, the characteristics of multimodal contributions may tend towards a perceptual viewpoint otherwise not achieved through a single modality. The ‘settling in’ period that was observed in participants when first wearing the headset may lend possible insight into the stimulation of alternative perceptual systems (Pantev et al. 1999); compensatory plasticity and sensory supplementation are areas of further interest and to be included in later studies. (Rauschecker 1995; Good et al. 2014; Bach-y-Rita 2003). Proceedings of ICMEM, 23-25 March 2015 005 Although informal testing provided positive feedback, the presentation of music lacked low frequency content (<150Hz). The novelty of avoiding the “in head” experience when listening through earphones and perceptual differences gained via alternative information pathways may account for some initial interest. Improvement in transducer technology and dynamic range is required and will feature in further works; with regard to the low frequency content alternative locations on the body may provide the answer based on the premise that good bass is “felt” in the lower abdomen. Several participants at the NEC however were hearing impaired and found the experience quite unique; one in particular with unilateral conductive hearing loss was able to appreciate stereo separation again, which wasn’t possible using standard headphones. Conclusions Multimodal presentation of music through tissue conduction seems feasible and may offer renewed enjoyment for those with certain hearing impairment. Improvements in transducer technology and low frequency consideration would enhance the quality. Considering alternative information pathways could lead to changes in music composition and reproduction providing a different spatial perceptual experience. Address for correspondence Ian McKenzie - i.mckenzie1@derby.ac.uk ; Dr. Peter Lennox p.lennox@derby.ac.uk; Dr Bruce Wigins – b.j.wiggins@derby.ac.uk University of Derby, Markeaton Street, Derby, DE22 3AW, England. References Bach-y-Rita, P., Kercel, W. S. (2003) Sensory substitution and the human–machine interface. TRENDS in Cognitive Sciences Vol.7 No.12 December 2003. Dietz, A.J.; May, B.S.; Knaus, D.A.; Greeley, H.P. (2005) Hearing Protection for Bone-Conducted Sound. In New Directions for Improving Audio Effectiveness (pp. 14-1 – 14-18). Meeting Proceedings RTO-MP-HFM-123, Paper 14. Neuilly-sur-Seine, France: RTO. Good, A., Reed, M. J., & Russo, F.A. (2014) Compensatory Plasticity in the Deaf Brain: Effects on Perception of Music. Brain Sci. 2014, 4, 560-574; doi:10.3390/brainsci4040560. Lackner, J. R. (1988) Some proprioceptive influences on the perceptual representations of body shape and orientation. Brain, 111, 281-297. 006 Longo, M. R., Aza˜nóna, E., & Haggarda, P. (2009) More than skin deep: Body representation beyond primary somatosensory cortex. Neuropsychologia 48 (2010) 655–668. MacDonald, J.A., Henry, P.P. & Letowski, T.R. (2006) Spatial audio through a bone conduction interface. International Journal of Audiology, 2006, 45, pp. 595-599. McBride, M., Letowski, T.R., & Tran, P.K. (2005) Bone Conduction Head Sensitivity Mapping: Bone Vibrator. ARL-TR-3556. McKenzie, I., Lennox, P., & Wiggins, B.J. (2014) Tissue conducted spatial sound fields. Proceedings of the Institute of Acoustics. Vol. 36. Pt. 2. 2014. Meredith, M.A., & Stein, B. E. (1986) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. Journal of Neurophysiology Published 1 September 1986 Vol. 56 no. 3, 640-662 DOI: Niemoeller, A F. (1940) Handbook of Hearing Aids. New York: Harvest House. Pantev, C., Wollbrink, A., Roberts, L.E., Engelien, A., & Lu¨tkenho¨ner, B. (1999) Short-term plasticity of the human auditory cortex. Brain Research 842 (1999) 192–199. Rauschecker, J. P. (1995) Compensatory Plasticity and Sensory substitution in the Cerebral Cortex. TINS Vol. 18, No. 1, 1995. Seifritz E1, Neuhoff JG, Bilecen D, Scheffler K, Mustovic H, Schächinger H, Elefante R, Di Salle F.(2002) Neural processing of auditory looming in the human brain. Curr. Biol. 2002 Dec 23;12(24):2147-51 Stanley, R., & Walker, B. N. (2006). Lateralisation of sounds using bone-conduction headsets. Proceedings of the Annual Meeting of the Human Factors and Ergonomics Society (HFES 2006) (pp. 15711575), San Francisco, CA. Stenfelt, S., & Zeitooni, M. (2013) Binaural hearing ability with mastoid applied bilateral bone conduction stimulation in normal hearing subjects, 2013, Journal of Acoustical Society of America, (134), 1, 481-493. Tajadura-Jiménez, A., Väljamäe, A., Kitagawa, N., and Ho, H.-N. (2007) Whole-Body Vibration Influences Sound Localisation in the Median Plane, Proceedings of the 10th Annual International Workshop on Presence, Barcelona, Spain, Oct. 2007. Tsuchitani, C. (2014) Chapter 2: Somatosensory Systems. [available online] http://neuroscience.uth.tmc.edu/s2/chapter02.html Wiggins, B. (2014) 'WigWare' The Blog of Bruce. [online] Available at: http://www.brucewiggins.co.uk/?page_id=78