Tonal Cognition in Sonification: Exploring the Needs of Practitioners in Sonic Interaction Design
Pages 218 - 228
Abstract
Research into tonal music examines the structural relationships among sounds and how they align with our auditory perception. The exploration of integrating tonal cognition into sonic interaction design, particularly for practitioners lacking extensive musical knowledge, and developing an accessible software tool, remains limited. We report on a study of designers to understand the sound creation practices of industry experts and explore how infusing tonal music principles into a sound design tool can better support their craft and enhance the sonic experiences they create. Our study collected qualitative data through semi-structured individual and focus group interviews with six participants. We developed a low-fidelity prototype sound design tool that involves practical methods of functional harmony and interaction design discussed in focus groups. We identified four themes through reflexive thematic analysis: decision-making, domain knowledge and terminology, collaboration, and contexts in sound creation. Finally, we discussed design considerations for an accessible sonic interaction design tool that aligns auditory experience more closely with tonal cognition.
References
[1]
David Bernstein. 2002. Nineteenth-century harmonic theory: the Austro-German legacy. Cambridge University Press, 778–811.
[2]
Emmanuel Bigand and Bénédicte Poulin-Charronnat. 2016. Tonal cognition. The Oxford handbook of music psychology (2016), 95–111.
[3]
Meera M. Blattner, Denise A. Sumikawa, and Robert M. Greenberg. 1989. Earcons and Icons: Their Structure and Common Design Principles. Human–Computer Interaction 4, 1 (1989), 11–44. https://doi.org/10.1207/s15327051hci0401_1
[4]
Jorge Boehringer, Marcin Pietruszewski, John Bowers, Bennett Hogg, Joseph Newbold, Gerriet Sharma, Tim Shaw, and Paul Vickers. 2022. Listener-centred Sonification Practice as Transdisciplinary Experimental Artistic Engagement. (2022).
[5]
Stephen A Brewster. 1998. Using nonspeech sounds to provide navigation cues. ACM Transactions on Computer-Human Interaction (TOCHI) 5, 3 (1998), 224–259.
[6]
Stephen A Brewster, Peter C Wright, and Alastair DN Edwards. 1994. A detailed investigation into the effectiveness of earcons. In SANTA FE INSTITUTE STUDIES IN THE SCIENCES OF COMPLEXITY-PROCEEDINGS VOLUME-, Vol. 18. Addison-Wesley Publishing CO, 471–471.
[7]
Stephen A Brewster, Peter C Wright, and Alistair DN Edwards. 1995. Experimentally derived guidelines for the creation of earcons. In Adjunct Proceedings of HCI, Vol. 95. 155–159.
[8]
J Peter Burkholder, Donald Jay Grout, and Claude V Palisca. 2019. A history of western music. WW Norton & Company.
[9]
Edward Burnell, Michael Stern, Ana Flooks, and Maria C Yang. 2017. Integrating design and optimization tools: a designer centered study. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol. 58219. American Society of Mechanical Engineers, V007T06A029.
[10]
Minsik Choi. 2023. Music theoretical approach to auditory interface design: progressive, explainable, and accessible. In Proceedings of the 15th Conference on Creativity and Cognition. 39–42.
[11]
Thomas Christensen. 2006. The Cambridge history of Western music theory. Cambridge University Press.
[12]
Brent Clement. 2023. Rock Tonality Amplified: A Theory of Modality, Harmonic Function, and Tonal Hierarchy. Routledge. https://doi.org/10.4324/9781003300236
[13]
Jaap Daalhuizen, Petra Badke-Schaub, and Stephen Batill. 2009. Dealing with uncertainty in design practice: issues for designer-centered methodology. Vol. 9. Design Society, 147–158. http://www.designsociety.org/iced09_17th_international_conference_on_engineering_design.event.62.2009-08-24.htm 17th International Conference on Engineering Design, ICED09 ; Conference date: 24-08-2009 Through 27-08-2009.
[14]
Dan Diaper and Neville Stanton. 2003. The handbook of task analysis for human-computer interaction. CRC Press.
[15]
Karmen Franinovic and Stefania Serafin. 2013. Sonic interaction design. Mit Press.
[16]
Robert Gauldin. 2004. Harmonic Practice in Tonal Music (2nd ed.). W.W. Norton.
[17]
William W. Gaver. 1986. Auditory Icons: Using Sound in Computer Interfaces. Hum.-Comput. Interact. 2, 2 (jun 1986), 167–177. https://doi.org/10.1207/s15327051hci0202_3
[18]
William W Gaver. 1989. The SonicFinder: An interface that uses auditory icons. Human–Computer Interaction 4, 1 (1989), 67–94.
[19]
Thomas Hermann, Andy Hunt, John G Neuhoff, 2011. The sonification handbook. Vol. 1. Logos Verlag Berlin.
[20]
Myounghoon Jeon, Areti Andreopoulou, and Brian FG Katz. 2020. Auditory displays and auditory user interfaces: art, design, science, and research. Journal on Multimodal User Interfaces 14 (2020), 139–141.
[21]
Purnima Kamath, Fabio Morreale, Priambudi Lintang Bagaskara, Yize Wei, and Suranga Nanayakkara. 2024. Sound Designer-Generative AI Interactions: Towards Designing Creative Support Tools for Professional Sound Designers. In Proceedings of the CHI Conference on Human Factors in Computing Systems (, Honolulu, HI, USA,) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 730, 17 pages. https://doi.org/10.1145/3613904.3642040
[22]
Stefan Koelsch. 2013. Brain and Music. John Wiley & Sons.
[23]
Marina Korsakova-Kreyn. 2018. Two-level model of embodied cognition in music.Psychomusicology: Music, Mind, and Brain 28, 4 (2018), 240.
[24]
Marc Leman. 2008. Embodied music cognition and mediation technology. MIT press.
[25]
Ronnie Lipton. 2011. The practical guide to information design. John Wiley & Sons.
[26]
Indrani Medhi. 2007. User-centered design for development. interactions 14, 4 (2007), 12–14.
[27]
Frederik Moesgaard, Lasse Hulgaard, and Mads Bødker. 2020. Involving Users in Sound Design. In Design, User Experience, and Usability. Interaction Design: 9th International Conference, DUXU 2020, Held as Part of the 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19–24, 2020, Proceedings, Part I 22. Springer, 405–425.
[28]
J Newbold, N Gold, and NL Bianchi-Berthouze. 2017. Musical expectancy in squat sonification for people who struggle with physical activity. In ICAD’17, Vol. 23. ICAD’17, 65–72.
[29]
Joseph Newbold, Nicolas E Gold, and Nadia Bianchi-Berthouze. 2020. Movement sonification expectancy model: Leveraging musical expectancy theory to create movement-altering sonifications. Journal on Multimodal User Interfaces 14 (2020), 153–166.
[30]
Joseph W. Newbold, Nadia Bianchi-Berthouze, Nicolas E. Gold, Ana Tajadura-Jiménez, and Amanda CdC Williams. 2016. Musically Informed Sonification for Chronic Pain Rehabilitation: Facilitating Progress & Avoiding Over-Doing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 5698–5703. https://doi.org/10.1145/2858036.2858302
[31]
Don Norman. 2013. The design of everyday things: Revised and expanded edition. Basic books.
[32]
Stephen Roddy and Brian Bridges. 2020. Mapping for meaning: the embodied sonification listening model and its implications for the mapping problem in sonic information design. Journal on Multimodal User Interfaces 14, 2 (2020), 143–151.
[33]
Yvonne Rogers, Helen Sharp, and Jenny Preece. 2011. Interaction Design: Beyond Human-Computer Interaction (3rd ed.). John Wiley & Sons.
[34]
Kalpna Sagar and Anju Saha. 2017. A systematic review of software usability studies. International Journal of Information Technology (2017), 1–24.
[35]
Carla Scaletti. 2018. Sonification ≠ Music. In The Oxford Handbook of Algorithmic Music, Roger T. Dean and Alex McLean (Eds.). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780190226992.013.9 Online edition, Oxford Academic, 5 Feb. 2018.
[36]
Rod Selfridge and Sandra Pauletto. 2022. Sound design ideation: comparing four sound designers’ approaches. In 19th Sound and Music Computing Conference, June 5-12th, 2022, Saint-Étienne (France).
[37]
Stefania Serafin, Bill Buxton, Bill Gaver, and Sara Bly. 2022. Auditory Interfaces. Focal Press.
[38]
Neville A Stanton. 2006. Hierarchical task analysis: Developments, applications, and extensions. Applied ergonomics 37, 1 (2006), 55–79.
[39]
Neville A Stanton, Paul M Salmon, Laura A Rafferty, Guy H Walker, Chris Baber, and Daniel P Jenkins. 2017. Human factors methods: a practical guide for engineering and design. CRC Press.
[40]
Gareth Terry and Nikki Hayfield. 2021. Essentials of thematic analysis.American Psychological Association.
[41]
Paul Vickers. 2016. Sonification and music, music and sonification. In The Routledge Companion to Sounding Art. Routledge, 135–144.
[42]
Daniel Wallach and Sebastian C. Scholz. 2012. User-Centered Design: Why and How to Put Users First in Software Development. In Software for People, Alexander Maedche, Andreas Botzenhardt, and Lars Neer (Eds.). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-31371-4_2
Index Terms
- Tonal Cognition in Sonification: Exploring the Needs of Practitioners in Sonic Interaction Design
Recommendations
A review of methods and frameworks for sonic interaction design: exploring existing approaches
CMMR/ICAD'09: Proceedings of the 6th international conference on Auditory DisplayThis article presents a review of methods and frameworks focused on the early conceptual design of sonic interactions. The aim of the article is to provide novice and expert designers in human computer interaction an introduction to sonic interaction ...
Interaction by ear
Highlights- An overview of sonic information design and sonic interaction design in relation to the three HCI waves.
AbstractSpeech-based interaction is now part of our everyday experiences, in the home and on the move. More subtle is the presence of designed non-speech sounds in human-machine interactions, and far less evident is their importance to create ...
Sonic interaction design: sound, information and experience
CHI EA '08: CHI '08 Extended Abstracts on Human Factors in Computing SystemsSonic Interaction Design (SID) is an emerging field that is positioned at the intersection of auditory display, ubiquitous computing, interaction design, and interactive arts. SID can be used to describe practice and inquiry into any of various roles ...
Comments
Information & Contributors
Information
Published In
September 2024
565 pages
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 18 September 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
AM '24
Acceptance Rates
Overall Acceptance Rate 177 of 275 submissions, 64%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 73Total Downloads
- Downloads (Last 12 months)73
- Downloads (Last 6 weeks)53
Reflects downloads up to 10 Nov 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in