M.I.N.U.E.T.: Procedural Musical Accompaniment for Textual Narratives
Authors: 
Mehak Maniktala, Chris Miller, Aaron Margolese-Malin, Arnav Jhala, Chris MartensAuthors Info & Claims
Mehak Maniktala, Chris Miller, Aaron Margolese-Malin, Arnav Jhala, Chris MartensAuthors Info & ClaimsArticle No.: 100, Pages 1 - 7
Abstract
Extensive research has been conducted on using procedural music generation in real-time applications such as accompaniment to musicians, visual narratives, and games. However, less attention has been paid to the enhancement of textual narratives through music. In this paper, we present Mood Into Note Using Extracted Text (MINUET), a novel system that can procedurally generate music for textual narrative segments using sentiment analysis. Textual analysis of the flow and sentiment derived from the text is used as input to condition accompanying music. Music generation systems have addressed variations through changes in sentiment. By using an ensemble predictor model to classify sentences as belonging to particular emotions, MINUET generates text-accompanying music with the goal of enhancing a reader’s experience beyond the limits of the author’s words. Music is played via the JMusic library and a set of Markov chains specific to each emotion with mood classifications evaluated via stratified 10-fold cross validation. The development of MINUET affords the reflection and analysis of features that affect the quality of generated musical accompaniment for text. It also serves as a sandbox for further evaluating sentiment-based systems on both text and music generation sides in a coherent experience of an implemented and extendable experiential artifact.
References
[1]
Timothey Adam, Michael Haungs, and Foaad Khosmood. 2014. Procedurally generated, adaptive music for rapid game development. In FDG 2014 Workshop Proceedings, Foundation of Digital Games.
[2]
Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat. 2005. Emotions from text: machine learning for text-based emotion prediction. In Proceedings of the conference on human language technology and empirical methods in natural language processing. Association for Computational Linguistics, 579–586.
[3]
Heike Argstatter. 2016. Perception of basic emotions in music: Culture-specific or multicultural?Psychology of Music 44, 4 (2016), 674–690.
[4]
Laura-Lee Balkwill and William Forde Thompson. 1999. A cross-cultural investigation of the perception of emotion in music: Psychophysical and cultural cues. Music perception: an interdisciplinary journal 17, 1 (1999), 43–64.
[5]
Karen Collins. 2009. An introduction to procedural music in video games. Contemporary Music Review 28, 1 (2009), 5–15.
[6]
Kate Compton, Ben Kybartas, and Michael Mateas. 2015. Tracery: an author-focused generative text tool. In International Conference on Interactive Digital Storytelling. Springer, 154–161.
[7]
Kate Compton and Michael Mateas. 2015. Casual Creators. In ICCC. 228–235.
[8]
Kim G Dolgin and Edward H Adelson. 1990. Age changes in the ability to interpret affect in sung and instrumentally-presented melodies. Psychology of Music 18, 1 (1990), 87–98.
[9]
Chris Donahue, Julian McAuley, and Miller Puckette. 2018. Synthesizing Audio with Generative Adversarial Networks. arXiv preprint arXiv:1802.04208(2018).
[10]
Hao-Wen Dong, Wen-Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang. 2018. MuseGAN: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment. In Proc. AAAI Conf. Artificial Intelligence.
[11]
Cicero dos Santos and Maira Gatti. 2014. Deep convolutional neural networks for sentiment analysis of short texts. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. 69–78.
[12]
Richard Evans and Emily Short. 2013. Versu—a simulationist storytelling system. IEEE Transactions on Computational Intelligence and AI in Games 6, 2(2013), 113–130.
[13]
John Fuegi and Jo Francis. 2003. Lovelace & Babbage and the creation of the 1843’notes’. IEEE Annals of the History of Computing 25, 4 (2003), 16–26.
[14]
Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford 1, 12 (2009).
[15]
Antonio Gulli and Sujit Pal. 2017. Deep learning with Keras. Packt Publishing Ltd.
[16]
Julia C Hailstone, Rohani Omar, Susie MD Henley, Chris Frost, Michael G Kenward, and Jason D Warren. 2009. It’s not what you play, it’s how you play it: Timbre affects perception of emotion in music. The quarterly Journal of Experimental psychology 62, 11(2009), 2141–2155.
[17]
Dorien Herremans, Ching-Hua Chuan, and Elaine Chew. 2017. A functional taxonomy of music generation systems. ACM Computing Surveys (CSUR) 50, 5 (2017), 1–30.
[18]
David A Jurgens, Peter D Turney, Saif M Mohammad, and Keith J Holyoak. 2012. Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation. Association for Computational Linguistics, 356–364.
[19]
Patrik N Juslin. 2001. Communicating emotion in music performance: A review and a theoretical framework.(2001).
[20]
Vojislav Kecman. 2005. Support vector machines–an introduction. In Support vector machines: theory and applications. Springer, 1–47.
[21]
Balázs Kégl. 2009. Introduction to AdaBoost.
[22]
Hyun-Chul Lee and In-Kwon Lee. 2005. Automatic synchronization of background music and motion in computer animation. In Computer Graphics Forum, Vol. 24. Wiley Online Library, 353–361.
[23]
Edward Loper and Steven Bird. 2002. NLTK: the natural language toolkit. arXiv preprint cs/0205028(2002).
[24]
Lie Lu, Dan Liu, and Hong-Jiang Zhang. 2006. Automatic mood detection and tracking of music audio signals. IEEE Transactions on audio, speech, and language processing 14, 1(2006), 5–18.
[25]
Todd Lubart. 2005. How can computers be partners in the creative process: classification and commentary on the special issue. International Journal of Human-Computer Studies 63, 4-5 (2005), 365–369.
[26]
Rishi Madhok, Shivali Goel, and Shweta Garg. 2018. SentiMozart: Music Generation based on Emotions. (2018).
[27]
Saif M. Mohammad. [n.d.]. Emotion, Sentiment, and Stance Labeled Data. https://web.archive.org/web/20170623152301http://saifmohammad.com:80/WebPages/SentimentEmotionLabeledData.html.
[28]
Saif M. Mohammad. [n.d.]. Fine tuning a classifier in scikit-learn. http://saifmohammad.com/WebPages/EmotionIntensity-SharedTask.html.
[29]
Saif M Mohammad and Felipe Bravo-Marquez. 2017. Emotion intensities in tweets. arXiv preprint arXiv:1708.03696(2017).
[30]
Dan Morris, Sumit Basu, and Ian Simon. 2010. Automatic accompaniment for vocal melodies. US Patent 7,705,231.
[31]
Preslav Nakov, Sara Rosenthal, Svetlana Kiritchenko, Saif M Mohammad, Zornitsa Kozareva, Alan Ritter, Veselin Stoyanov, and Xiaodan Zhu. 2016. Developing a successful SemEval task in sentiment analysis of Twitter and other social media texts. Language Resources and Evaluation 50, 1 (2016), 35–65.
[32]
BBC News. [n.d.]. Could a computer ever create better art than a human?https://www.bbc.com/news/business-47700701.
[33]
Alexander Pak and Patrick Paroubek. 2010. Twitter as a corpus for sentiment analysis and opinion mining. In LREc, Vol. 10. 1320–1326.
[34]
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, 2011. Scikit-learn: Machine learning in Python. the Journal of machine Learning research 12 (2011), 2825–2830.
[35]
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. In Empirical Methods in Natural Language Processing (EMNLP). 1532–1543. http://www.aclweb.org/anthology/D14-1162
[36]
Whitney Quesenbery and Kevin Brooks. 2010. Storytelling for user experience: Crafting stories for better design. Rosenfeld Media.
[37]
Adhika Sigit Ramanto and Nur Ulfa Maulidevi. 2017. Markov Chain Based Procedural Music Generator with User Chosen Mood Compatibility. International Journal of Asia Digital Art and Design Association 21, 1(2017), 19–24.
[38]
D Ramos, JLO Bueno, and E Bigand. 2011. Manipulating Greek musical modes and tempo affects perceived musical emotion in musicians and nonmusicians. Brazilian Journal of Medical and Biological Research 44, 2 (2011), 165–172.
[39]
Syeda Rida-E-Fatima, Ali Javed, Ameen Banjar, Aun Irtaza, Hassan Dawood, Hussain Dawood, and Abdullah Alamri. 2019. A multi-layer dual attention deep learning model with refined word embeddings for aspect-based sentiment analysis. IEEE Access 7(2019), 114795–114807.
[40]
Irina Rish 2001. An empirical study of the naive Bayes classifier. In IJCAI 2001 workshop on empirical methods in artificial intelligence, Vol. 3. 41–46.
[41]
S Rasoul Safavian and David Landgrebe. 1991. A survey of decision tree classifier methodology. IEEE transactions on systems, man, and cybernetics 21, 3(1991), 660–674.
[42]
Ian Simon, Dan Morris, and Sumit Basu. 2008. MySong: automatic accompaniment generation for vocal melodies. In Proceedings of the SIGCHI conference on human factors in computing systems. 725–734.
[43]
Swathi Swaminathan and E Glenn Schellenberg. 2015. Current emotion research in music psychology. Emotion review 7, 2 (2015), 189–197.
[44]
Manuelde Vega. 1996. The representation of changing emotions in reading comprehension. Cognition & Emotion 10, 3 (1996), 303–322.
[45]
Sandrine Vieillard, Isabelle Peretz, Nathalie Gosselin, Stéphanie Khalfa, Lise Gagnon, and Bernard Bouchard. 2008. Happy, sad, scary and peaceful musical excerpts for research on emotions. Cognition & Emotion 22, 4 (2008), 720–752.
[46]
Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the conference on human language technology and empirical methods in natural language processing. Association for Computational Linguistics, 347–354.
[47]
Rene Wooller, Andrew R Brown, Eduardo Miranda, Joachim Diederich, and Rodney Berry. 2005. A framework for comparison of process in algorithmic music systems. (2005).
[48]
Georgios N Yannakakis, Antonios Liapis, and Constantine Alexopoulos. 2014. Mixed-initiative co-creativity. (2014).
Recommendations
Dynamic Procedural Music Generation from NPC Attributes
FDG '20: Proceedings of the 15th International Conference on the Foundations of Digital GamesProcedural Music Generation in Games (PMGG) can enrich the playing experience by providing both entertainment and communication to the player. We present a system that generates unique procedural thematic music for non-player characters (NPC) based on ...
Pop Music Generation: From Melody to Multi-style Arrangement
Special Issue on KDD 2018, Regular Papers and Survey PaperMusic plays an important role in our daily life. With the development of deep learning and modern generation techniques, researchers have done plenty of works on automatic music generation. However, due to the special requirements of both melody and ...
PopMAG: Pop Music Accompaniment Generation
MM '20: Proceedings of the 28th ACM International Conference on MultimediaIn pop music, accompaniments are usually played by multiple instruments (tracks) such as drum, bass, string and guitar, and can make a song more expressive and contagious by arranging together with its melody. Previous works usually generate multiple ...
Comments
Information & Contributors
Information
Published In
September 2020
804 pages
ISBN:9781450388078
DOI:10.1145/3402942
Copyright © 2020 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 17 September 2020
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
FDG '20
FDG '20: International Conference on the Foundations of Digital Games
September 15 - 18, 2020
Bugibba, Malta
Acceptance Rates
Overall Acceptance Rate 152 of 415 submissions, 37%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 102Total Downloads
- Downloads (Last 12 months)18
- Downloads (Last 6 weeks)1
Reflects downloads up to 09 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format