research-article

SoundSignaling: Realtime, Stylistic Modification of a Personal Music Corpus for Information Delivery

Authors:

Ishwarya Ananthabhotla,

Joseph A. ParadisoAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 2, Issue 4

Article No.: 154, Pages 1 - 23

https://doi.org/10.1145/3287032

Published: 27 December 2018 Publication History

Abstract

Drawing inspiration from the notion of cognitive incongruence associated with Stroop's famous experiment, from musical principles, and from the observation that music consumption on an individual basis is becoming increasingly ubiquitous, we present the SoundSignaling system -- a software platform designed to make real-time, stylistically relevant modifications to a personal corpus of music as a means of conveying information or notifications. In this work, we discuss in detail the system's technical implementation and its motivation from a musical perspective, and validate these design choices through a crowd-sourced signal identification experiment consisting of 200 independent tasks performed by 50 online participants. We then qualitatively discuss the potential implications of such a system from the standpoint of switch cost, cognitive load, and listening behavior by considering the anecdotal outcomes of a small-scale, in-the-wild experiment consisting of over 180 hours of usage from 6 participants. Through this work, we suggest a re-evaluation of the age-old paradigm of binary audio notifications in favor of a system designed to operate upon the relatively unexplored medium of a user's musical preferences.

Supplementary Material

ananthabhotla (ananthabhotla.zip)

Supplemental movie, appendix, image and software files for, SoundSignaling: Real-time, Stylistic Modification of a Personal Music Corpus for Information Delivery

Download
26.30 KB

References

[1]

{n. d.}. Google GMail API. https://developers.google.com/gmail/api/v1/reference/. ({n. d.}). Accessed: 2018-01-05.

[2]

2013. Brain, Interrupted. http://www.nytimes.com/2013/05/05/opinion/sunday/a-focus-on-distraction.html?mtrref=www.google.com&gwh=F707B2B27C6A05114FC9FBB2A0370200&gwt=pay&assetType=opinion. (2013). Accessed: 2018-01-05.

[3]

2015. Addicted to Distraction. https://www.nytimes.com/2015/11/29/opinion/sunday/addicted-to-distraction.html?mtrref=www.google.com&gwh=18A28DCD1302DA9AA94A0EFEE1B55D61&gwt=pay&assetType=opinion. (2015). Accessed: 2018-01-05.

[4]

2017. Adobe Email Survey 2017. https://blogs.adobe.com/digitaleurope/files/2017/08/20170815_Email2017_EMEA_Report.pdf. (2017). Accessed: 2018-01-05.

[5]

2017. AudioTechnica User Consumption Study 2017. https://www.realwire.com/releases/Official-study-reveals-that-the-average-person-will-spend-13-years-of-their-lives-listening-to-music. (2017). Accessed: 2018-01-05.

[6]

2017. Global Music Report 2017. http://www.ifpi.org/downloads/GMR2017.pdf. (2017). Accessed: 2018-01-05.

[7]

2017. Spotify Listening Diversity. https://insights.spotify.com/us/2017/11/02/listening-diversity-spotify/. (2017). Accessed: 2018-01-05.

[8]

Piotr D Adamczyk and Brian P Bailey. 2004. If not now, when?: the effects of interruption at different moments within task execution. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 271--278.

Digital Library

[9]

Miguel A Alonso, Gaël Richard, and Bertrand David. 2004. Tempo And Beat Estimation Of Musical Signals. In ISMIR.

[10]

Ishwarya Ananthabhotla and Joseph Paradiso. 2017. VisualSoundtrack: A Novel Approach to Style Transfer in the Context of Soundtrack Prototyping. In Proceedings of the International Computer Music Conference (ICMC). ICMA.

[11]

Luke Barrington, Michael J Lyons, Dominique Diegmann, and Shinji Abe. 2006. Ambient display using musical effects. In Proceedings of the 11th international conference on Intelligent user interfaces. ACM, 372--374.

Digital Library

[12]

Matthias Böhmer, Brent Hecht, Johannes Schöning, Antonio Krüger, and Gernot Bauer. 2011. Falling asleep with Angry Birds, Facebook and Kindle: a large scale study on mobile application usage. In Proceedings of the 13th international conference on Human computer interaction with mobile devices and services. ACM, 47--56.

Digital Library

[13]

Andreas Butz and Ralf Jung. 2005. Seamless user notification in ambient soundscapes. In Proceedings of the 10th international conference on Intelligent user interfaces. ACM, 320--322.

Digital Library

[14]

Daniel Chen and Roel Vertegaal. 2004. Using mental load for managing interruptions in physiologically attentive user interfaces. In CHI'04 extended abstracts on Human factors in computing systems. ACM, 1513--1516.

Digital Library

[15]

Zhe Chen. 2003. Attentional focus, processing load, and Stroop interference. Perception & Psychophysics 65, 6 (2003), 888--900.

[16]

Mary Czerwinski, Edward Cutrell, and Eric Horvitz. 2000. Instant messaging and interruption: Influence of task type on performance. In OZCHI 2000 conference proceedings, Vol. 356. 361--367.

[17]

Mary Czerwinski, Eric Horvitz, and Susan Wilhite. 2004. A diary study of task switching and interruptions. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 175--182.

Digital Library

[18]

Deloitte. 2017. 2017 Global Mobile Consumer Survey: The Dawn of the Next Era in Mobile.

[19]

Gaël Dubus and Roberto Bresin. 2013. A systematic review of mapping strategies for the sonification of physical quantities. PloS one 8, 12 (2013), e82491.

[20]

Daniel PW Ellis. 2002. A phase vocoder in Matlab. Columbia University (http://www.ee.columbia.edu/dpwe/resources/matlab/pvoc/) (2002).

[21]

Daniel PW Ellis. 2007. Beat tracking by dynamic programming. Journal of New Music Research 36, 1 (2007), 51--60.

[22]

Joel E Fischer, Chris Greenhalgh, and Steve Benford. 2011. Investigating episodes of mobile phone activity as indicators of opportune moments to deliver notifications. In Proceedings of the 13th international conference on human computer interaction with mobile devices and services. ACM, 181--190.

Digital Library

[23]

Derry Fitzgerald. 2010. Harmonic/percussive separation using median filtering. (2010).

[24]

Hans-W Gellersen, Albrecht Schmidt, and Michael Beigl. 1999. Ambient media for peripheral information display. Personal Technologies 3, 4 (1999), 199--208.

[25]

Joyce Ho and Stephen S Intille. 2005. Using context-aware computing to reduce the perceived burden of interruptions from mobile devices. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 909--918.

Digital Library

[26]

Edward Cutrell Mary Czerwinski Eric Horvitz. 2001. Notification, disruption, and memory: Effects of messaging interruptions on memory and performance. In Human-computer Interaction: INTERACT'01: IFIP TC. 13 International Conference on Human-Comupter Interaction, 9th-13th July 2001, Tokyo, Japan. IOS Press, 263.

[27]

Shamsi T Iqbal and Eric Horvitz. 2007. Disruption and recovery of computing tasks: field study, analysis, and directions. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 677--686.

Digital Library

[28]

Shamsi T Iqbal and Eric Horvitz. 2010. Notifications and awareness: a field study of alert usage and preferences. In Proceedings of the 2010 ACM conference on Computer supported cooperative work. ACM, 27--30.

Digital Library

[29]

Hiroshi Ishii, Craig Wisneski, Scott Brave, Andrew Dahley, Matt Gorbet, Brygg Ullmer, and Paul Yarin. 1998. ambientROOM: integrating ambient media with architectural space. In CHI 98 conference summary on Human factors in computing systems. ACM, 173--174.

Digital Library

[30]

Addie Johnson and Robert W Proctor. 2004. Attention: Theory and practice. Sage.

[31]

Ralf Jung. 2008. Ambience for auditory displays: Embedded musical instruments as peripheral audio cues. International Community for Auditory Display.

[32]

Ralf Jung. 2010. Non-intrusive audio notification in emotion classified background music. In Proceedings of Meetings on Acoustics 159ASA, Vol. 9. ASA, 050001.

[33]

Anssi Klapuri, Jouni Paulus, and Meinard Müller. 2010. Audio-based music structure analysis. In ISMIR, in Proc. of the Int. Society for Music Information Retrieval Conference.

[34]

Gregory Kramer, Bruce Walker, Terri Bonebright, Perry Cook, John H Flowers, Nadine Miner, and John Neuhoff. 2010. Sonification report: Status of the field and research agenda. (2010).

[35]

P Lamere. 2012. The infinite jukebox. url: www.infinitejuke.com/(accessed 24 May, 2016) (2012).

[36]

Elizabeth Hellmuth Margulis. 2014. On repeat: How music plays the mind. Oxford University Press.

[37]

Gloria Mark, Daniela Gudith, and Ulrich Klocke. 2008. The cost of interrupted work: more speed and stress. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. ACM, 107--110.

Digital Library

[38]

Andrii Matviienko, Maria Rauschenberger, Vanessa Cobus, Janko Timmermann, Jutta Fortmann, Andreas Löcken, Heiko Müller, Christoph Trappe, Wilko Heuten, and Susanne Boll. 2015. Towards new ambient light systems: a close look at existing encodings of ambient light systems. Interaction Design and Architecture (s). 2015;(26): 10--24. (2015).

[39]

Abhinav Mehrotra, Mirco Musolesi, Robert Hendley, and Veljko Pejovic. 2015. Designing content-driven intelligent notification mechanisms for mobile applications. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 813--824.

Digital Library

[40]

Abhinav Mehrotra, Veljko Pejovic, Jo Vermeulen, Robert Hendley, and Mirco Musolesi. 2016. My phone and me: understanding people's receptivity to mobile notifications. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1021--1032.

Digital Library

[41]

Christopher A Monk, Deborah A Boehm-Davis, and J Gregory Trafton. 2002. The attentional costs of interrupting task performance at various stages. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 46. SAGE Publications Sage CA: Los Angeles, CA, 1824--1828.

[42]

Stephen Monsell. 2003. Task switching. Trends in cognitive sciences 7, 3 (2003), 134--140.

[43]

Tadashi Okoshi, Julian Ramos, Hiroki Nozaki, Jin Nakazawa, Anind K Dey, and Hideyuki Tokuda. 2015. Reducing users' perceived mental effort due to interruptive notifications in multi-device mobile environments. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 475--486.

Digital Library

[44]

Antti Oulasvirta, Tye Rattenbury, Lingyi Ma, and Eeva Raita. 2012. Habits make smartphone use more pervasive. Personal and Ubiquitous Computing 16, 1 (2012), 105--114.

Digital Library

[45]

Veljko Pejovic and Mirco Musolesi. 2014. InterruptMe: designing intelligent prompting mechanisms for pervasive applications. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 897--908.

Digital Library

[46]

Martin Pielot, Karen Church, and Rodrigo De Oliveira. 2014. An in-situ study of mobile phone notifications. In Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services. ACM, 233--242.

Digital Library

[47]

J Ridley Stroop. 1935. Studies of interference in serial verbal reactions. Journal of experimental psychology 18, 6 (1935), 643.

[48]

Douglas Turnbull, Gert RG Lanckriet, Elias Pampalk, and Masataka Goto. 2007. A Supervised Approach for Detecting Boundaries in Music Using Difference Features and Boosting. In ISMIR. 51--54.

[49]

Liam D Turner, Stuart M Allen, and Roger M Whitaker. 2015. Interruptibility prediction for ubiquitous systems: conventions and new directions from a growing field. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 801--812.

Digital Library

[50]

Frederik Wiehr, Alexandra Voit, Dominik Weber, Sven Gehring, Christoph Witte, Daniel Kärcher, Niels Henze, and Antonio Krüger. 2016. Challenges in designing and implementing adaptive ambient notification environments. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. ACM, 1578--1583.

Digital Library

Cited By

Wang ACheng YLindlbauer D(2024)MARingBA: Music-Adaptive Ringtones for Blended Audio Notification DeliveryProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642376(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642376
Clark MDoryab A(2023)Sounds of HealthProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35703466:4(1-31)Online publication date: 11-Jan-2023
https://dl.acm.org/doi/10.1145/3570346
Ramsay DParadiso J(2022)Peripheral Light Cues as a Naturalistic Measure of FocusProceedings of the 2022 ACM International Conference on Interactive Media Experiences10.1145/3505284.3532984(375-380)Online publication date: 21-Jun-2022
https://dl.acm.org/doi/10.1145/3505284.3532984
Show More Cited By

Index Terms

SoundSignaling: Realtime, Stylistic Modification of a Personal Music Corpus for Information Delivery
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Sound-based input / output
    2. Interaction techniques
      1. Auditory feedback

Recommendations

Resynchronize Japanese "Geisha" Dance Video Using Music of Different Styles
CULTURECOMPUTING '13: Proceedings of the 2013 International Conference on Culture and Computing

Music and dancing are two different arts yet inseparable and which can be both powerful expression channels for a society or an artist. In most cases, the rhythm, tempo and performance of a dance depend on those of the music. This paper presents a new ...
Music/lyrics composition system considering user's image and music genre
SMC'09: Proceedings of the 2009 IEEE international conference on Systems, Man and Cybernetics

This paper proposes a music/lyrics composition system consisting of two sections, a lyric composing section and a music composing section, which considers user's image of a song and music genre. First of all, a user has an image of music/lyrics to ...
The Greek Music Dataset
EANN '15: Proceedings of the 16th International Conference on Engineering Applications of Neural Networks (INNS)

Music Information Research (MIR) requires musical data in order to test methods and to compare results. Greek music presents a number of unique characteristics that make its musical pieces distinct from popular tracks existing in currently available ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 2, Issue 4

December 2018

1169 pages

EISSN:2474-9567

DOI:10.1145/3301777

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 December 2018

Accepted: 01 October 2018

Revised: 01 July 2018

Received: 01 February 2018

Published in IMWUT Volume 2, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Science Foundation GRFP

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
308
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)2

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang ACheng YLindlbauer D(2024)MARingBA: Music-Adaptive Ringtones for Blended Audio Notification DeliveryProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642376(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642376
Clark MDoryab A(2023)Sounds of HealthProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35703466:4(1-31)Online publication date: 11-Jan-2023
https://dl.acm.org/doi/10.1145/3570346
Ramsay DParadiso J(2022)Peripheral Light Cues as a Naturalistic Measure of FocusProceedings of the 2022 ACM International Conference on Interactive Media Experiences10.1145/3505284.3532984(375-380)Online publication date: 21-Jun-2022
https://dl.acm.org/doi/10.1145/3505284.3532984
Ananthabhotla IRamsay DDuhart CParadiso J(2021)Cognitive Audio Interfaces: Mediating Sonic Information With an Understanding of How We HearIEEE Pervasive Computing10.1109/MPRV.2021.305265920:2(36-45)Online publication date: 1-Apr-2021
https://doi.org/10.1109/MPRV.2021.3052659
Yang JCinquin TSoros G(2021)Unsupervised Musical Timbre Transfer for Notification SoundsICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP39728.2021.9414760(3735-3739)Online publication date: 6-Jun-2021
https://doi.org/10.1109/ICASSP39728.2021.9414760

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents