article

Tune in to your emotions: a robust personalized affective music player

Authors:

Joris H. Janssen,

Joyce H. WesterinkAuthors Info & Claims

User Modeling and User-Adapted Interaction, Volume 22, Issue 3

Pages 255 - 279

https://doi.org/10.1007/s11257-011-9107-7

Published: 01 July 2012 Publication History

Abstract

The emotional power of music is exploited in a personalized affective music player (AMP) that selects music for mood enhancement. A biosignal approach is used to measure listeners' personal emotional reactions to their own music as input for affective user models. Regression and kernel density estimation are applied to model the physiological changes the music elicits. Using these models, personalized music selections based on an affective goal state can be made. The AMP was validated in real-world trials over the course of several weeks. Results show that our models can cope with noisy situations and handle large inter-individual differences in the music domain. The AMP augments music listening where its techniques enable automated affect guidance. Our approach provides valuable insights for affective computing and user modeling, for which the AMP is a suitable carrier application.

References

[1]

Baas, M., De Dreu, C.K.W., Nijstad, B.A.: A meta-analysis of 25 years of moodcreativity research: hedonic tone, activation, or regulatory focus?. Psychol. Bull. 134, 779-806 (2008).

[2]

Bak, A.A., Grobbee, D.E.: A randomized study on coffee and blood pressure. J. Hum. Hypertens. 4, 259-264 (1990).

[3]

Baumgartner, T., Esslen, M., Jäncke, L.: From emotion perception to emotion experience: emotions evoked by pictures and classical music. Int. J. Psychophysiol. 60(1), 34-43 (2006).

[4]

Beedie, C.J., Terry, P.C., Lane, A.M.: Distinctions between emotion and mood. Cogn. Emot. 19, 847-878 (2005).

[5]

Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006).

[6]

Boehner, K., DePaula, R., Dourish, P., Sengers, P.: How emotion ismade and measured. Int. J. Hum.-Comput. Stud. 65, 275-291 (2007).

Digital Library

[7]

Boucsein, W.: Electrodermal Activity. Plenum Press, New York (1992).

[8]

Caberletti, L., Elfmann, K., Kümmel, M., Schierz, C.: Influence of ambient lighting in vehicle interior on the driver's perception. In: de Kort, Y., IJsselsteijn, W., Vogels, I., Aarts, M., Tenner, A., Smolders, K. (eds.) Proceedings of Experiencing Light 2009 International Conference on the Effects of Light on Wellbeing, pp. 5-13, Eindhoven, The Netherlands (2009).

[9]

Cacioppo, J., Tassinary, L.: Inferring psychological significance from physiological signals. Am. Psychol. 45, 16-28 (1990).

[10]

Calvo, R.A., D'Mello, S.: Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans. Affect. Comput. 1, 18-37 (2010).

Digital Library

[11]

Carberry, S., de Rosis, F.: Introduction to special issue on affective modeling and adaptation. User Model. User-Adapt. Interact. 18, 1-9 (2008).

Digital Library

[12]

Chin, N.D.: Emperical evaluation of user models and user adaptive systems. User Model. User-Adapt. Interact. 11, 181-194 (2001).

[13]

Clore, G.L., Palmer, J.: Affective guidance of intelligent agents: how emotion controls cognition. Cogn. Syst. Res. 10(1), 21-30 (2009).

Digital Library

[14]

Csíkszentmihályi, M.: Flow: The Psychology of Optimal Experience. Harper Collins, Sussex, UK (1990).

[15]

de Rosis, F.: Preface: towards adaptation of interaction to affective factors.UserModel. User-Adapt. Interact. 11, 267-278 (2001).

[16]

D'Mello, S., Craig, S.D., Witherspoon, A., McDaniel, B., Graesser, A.: Automatic detection of learners affect from conversational cues. User Model. User-Adapt. Interact. 18, 45-80 (2008).

Digital Library

[17]

Fairclough, S.H.: Fundamentals of physiological computing. Interact. Comput. 21, 133-145 (2009).

Digital Library

[18]

Fogg, B.J.: Persuas. Technology. Morgan Kaufmann Publishers, New York (2003).

[19]

Frijda, N.H.: The Emotions. Cambridge University Press, New York (1986).

[20]

Geenen, R., van de Vijver, F.J.R.: A simple test of the law of initial values. Psychophysiology 30(5), 525-530 (1993).

[21]

Gendolla, G.H.E.: On the impact of mood on behavior: an integrative theory and a review. Rev. Gen. Psychol. 4, 378-408 (2000).

[22]

Gendolla, G.H.E., Brinkman, K.: The role of mood states in self-regulation: effects on action preferences and resource mobilization. Eur. Psychol. 10, 187-198 (2005).

[23]

Gendolla, G.H.E., Krüsken, J.: Mood state and cardiovascular response in active coping with an affect-regulative challenge. Int. J. Psychophysiol. 41, 169-180 (2001).

[24]

Hanson, M.A., Powell, H.C. Jr., Barth, A.T., Ringgenberg, K., Calhoun, B.H., Aylor, J.H. et al.: Body area sensor networks: challenges and opportunities. IEEE Comput. 42, 58-65 (2009).

Digital Library

[25]

Härdle, W.: Smoothing Techniques, with Implementations in S. Springer, New York (1991).

[26]

Healey, J.A.: Affect detection in the realworld: recording and processing physiological signals. In: Proceedings of the IEEE 3rd International Conference on Affective Computing and Intelligent Interaction, ACII, Vol. 1, pp. 729-734. IEEE Press, Amsterdam (2009).

[27]

Healey, J.A., Picard, R.W.: Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intell. Transp. Syst. 6, 156-166 (2005).

Digital Library

[28]

Healey, J.A., Picard, R.W., Dabek, F.: A new affect-perceiving interface and its application to personalized music selection. In: Turk, M. (ed.) Proceedings of the 1998 Workshop on Perceptual User Interfaces (PUI), San Francisco, CA, USA (1998).

[29]

Heinz, C., Seeger, B.: Cluster kernels: resource-aware kernel density estimators over streaming data. IEEE Trans. Knowl. Data Eng. 20, 880-893 (2008).

Digital Library

[30]

Höök, K.: Affective loop experiences: designing for interactional embodiment. Philos. Trans. R. Soc. B 364, 3585-3595 (2009).

[31]

Husain, G., Thompson, W.F., Schellenberg, E.G.: Effects of musical tempo and mode on arousal, mood, and spatial abilities. Music Percept. 20, 151-171 (2002).

[32]

Janssen, J.H., Bailenson, J.N., IJsselstein, W.A., Westerink, J.H.D.M.: Intimate heartbeats: opportunities for affective communication technology. IEEE Trans. Affect. Comput. 1(2), 72-80 (2010).

Digital Library

[33]

Kaptein, M., Eckles, D. : Selecting effective means to any end: futures and ethics of per30 suasion profiling. In: Ploug, T., Hasle, P., Oinas-Kukkonen, H. (eds.) Persuasive Technology, pp. 82-93. Springer, Berlin (2010).

[34]

Kim, J., André, E.: Emotion recognition based on physiological changes in music listening. IEEE Trans. Pattern Anal. Mach. Intell. 30, 2067-2083 (2008).

Digital Library

[35]

Kowal, J., Fortier, M.: Motivational determinants of flow: contributions from selfdetermination theory. J. Soc. Psychol. 139, 355-368 (1999).

[36]

Lemmens, P., Crompvoets, F., Brokken, D., van den Eerenbeemd, J., De Vries, G.-J.: A body-conforming tactile jacket to enrich movie viewing. In World Haptics Conference, pp. 7-12. IEEE, Los Alamitos (2009).

Digital Library

[37]

Lesiuk, T.: The effect of music listening on work performance. Psychol. Music 33, 173-191 (2005).

[38]

Liljedahl, M., Sjömark, C., Lefford, N.: Usingmusic to promote physical well-being via computer-mediated interaction. In MusicNetwork Open Workshop, 5 (2005).

[39]

Mandel, M., Poliner, G., Ellis, D.: Support vector machine active learning for music retrieval. ACM Multimed. Syst. J. 12, 3-13 (2006).

Digital Library

[40]

Matthews, G., Jones, D.M., Chamberlain, A.G.: Refining the measurement of mood: the UWIST mood adjective checklist. Br. J. Psychol. 81, 17-42 (1990).

[41]

McFarland, R.A., Kennison, R.: Asymmetry in the relationship between finger temperature changes and emotional state in males. Appl. Psychophysiol. Biofeedback 14(4), 281-290 (1989).

[42]

North, A.C.H., David, J.: Musical preferences during and after relaxation and exercise. Am. J. Psychol. 113, 43-67 (2000).

[43]

North, A.C.H., Hargreaves, D.J., Hargreaves, J.J.: Uses of music in everyday life. Music Percept. 22, 41-77 (2004).

[44]

Oliver, N., Flores-Mangas, F.: MPTrain: a mobile, music and physiology-based personal trainer. In: Proceedings of the 8th Conference on Human-Computer Interaction with Mobile Devices and Services, pp. 21-28. ACM, New York (2006).

Digital Library

[45]

Oliver, N., Kregor-Stickles, L.: PAPA: physiology and purpose-aware automatic playlist generation. In: Lemström, K., Tindale, A., Dannenberg, R. (eds.) Proceedings of the 7th International Conference on Music Information Retrieval (ISMIR), pp. 250-253, Victoria, Canada, 8-12 October 2006.

[46]

Ophira, E., Nass, C., Wagner, A.D.: Cognitive control in media multitaskers. Proc. Natl Acad. Sci. 106, 15583-15587 (2009).

[47]

Pantic, M., Patras, I.: Dynamics of facial expressions: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans. Man Syst. Cybernet. B 36, 433-449 (2006).

Digital Library

[48]

Pelletier, C.L.: The effect of music on decreasing arousal due to stress: a meta-analysis. J. Music Ther. 41, 192-214 (2004).

[49]

Peter, C., Herbon, A.: Emotion representation and physiology assignments in digital systems. Interact. Comput. 18, 139-170 (2006).

[50]

Picard, R.W.: Affective Computing. MIT Press, Cambridge (1997).

[51]

Picard, R.W.: Affective computing: challenges. Int. J. Hum.-Comput. Stud. 59, 55-64 (2003).

[52]

Prinz, J.J.: Gut Reactions: A Perceptual Theory of Emotion. Oxford University Press, New York (2004).

[53]

Rentfrow, P.J., Gosling, S.D.: The do re mi's of everyday life: the structure and personality correlates of music preference. J. Pers. Soc. Psychol. 84, 1236-1256 (2003).

[54]

Rickard, N.S.: Intense emotional responses to music: A test of the physiological arousal hypothesis. Psychol. Music 32, 371-388 (2004).

[55]

Rimm-Kaufman, S.E., Kagan, J.: The psychological significance of changes in skin temperature. Motiv. Emot. 20(1), 64-78 (1996).

[56]

Ritossa, D.A., Rickard, N.S.: The relative utility of 'pleasantness' and 'liking' dimensions in predicting the emotions expressed by music. Psychol. Music 32, 5-22 (2004).

[57]

Russell, J.A.: Core affect and the psychological construction of emotion. Psychol. Rev. 110, 145-172 (2003).

[58]

Rusting, C.L.: Personality, mood, and cognitive processing of emotional information: three conceptual frameworks. Psychol. Bull. 124, 165-196 (1998).

[59]

Saarikallio, S., Erkkilä, J.: Role ofmusic in adolescents'mood regulation. Psychol.Music 35, 88-109 (2007).

[60]

Scott, D., Sain, S.: Multidimensional density estimation. In: Rao, C.R., Wegman, E.J., Solka, J.L. (eds.) Handbook of Statistics, Vol. 24, pp. 229-261. Elsevier, North Holland (2005).

[61]

Silverman, B.W.: Density Estimation for Statistics and Data Analysis. Chapman and Hall, London (1986).

[62]

Sloboda, J.A.: Exploring the Musical Mind: Cognition, Emotion, Ability, Function. Oxford University Press, New York (2005).

[63]

Sotiropoulos, D.N., Lampropoulos, A.S., Tsihrintzis, G.A.: MUSIPER: a system for modeling music similarity perception based on objective feature subset selection. User Model. User-Adapt. Interact. 18, 315-348 (2008).

Digital Library

[64]

Thayer, R.E.: The Biopsychology of Mood and Activation. Oxford University Press, New York (1989).

[65]

Tractinsky, N.: Tools over solutions? comments on interacting with computers special issue on affective computing. Interact. Comput. 16, 751-757 (2004).

[66]

Turlach, B.A.: Bandwidth selection in kernel density estimation: a review. Discussion Paper 9317, Institut de Statistique, Voie du Roman Pays 34, B-1348 Louvain-la-Neuve (1993).

[67]

Vaillant, G.: Aging Well: Surprising Guideposts to a Happier Life from the Landmark Harvard Study of Adult Development. Little, Brown and Company, Boston (2003).

[68]

Van den Broek, E.L., Janssen, J.H., Westerink, J.H.D.M.: Guidelines for Affective Signal Processing (ASP): From lab to life. In Proceedings of the IEEE 3rd international conference on affective computing and intelligent interaction, ACII, Vol. 1, pp. 704-709. IEEE Press, Amsterdam, The Netherlands (2009a).

[69]

Van den Broek, E.L., Janssen, J.H., Westerink, J.H.D.M., Healey, J.A.: Prerequisites for Affective Signal Processing (ASP). In: Encarnaçã, P., Veloso, A. (eds.) Biosignals 2009: Proceedings of the International Conference on Bio-Inspired Systems and Signal Processing, pp. 426-433, Porto, Portugal (2009b).

[70]

Van den Broek, E.L., Lisy, V., Janssen, J.H., Westerink, J.H.D.M., Schut, M.H., Tuinenbreijer, K.: Affective man-machine interface: unveiling human emotions through biosignals. In: Fred, A., Filipe, J., Gamboa, H. (eds.) Biomedical Engineering Systems and Technologies: BIOSTEC2009 Selected Revised Papers, Vol. 52., pp. 21-47. Springer, Berlin (2010).

[71]

Wagenaar, W.A.: Note on the construction of digram-balanced latin squares. Psychol. Bull. 72, 384-386 (1969).

[72]

Webster, G.D., Weir, C.G.: Emotional responses to music: interactive effects of mode, texture, and tempo. Motiv. Emot. 29, 19-39 (2005).

[73]

Westerink, J.H.D.M., De Vries, G., Waele, S., Eerenbeemd, J., Boven, M., Ouwerkerk, M.: Emotion measurement platform for daily life situations. In: Nijholt, A., Cohn, J., Pantic, M. (eds.) Proceedings of ACII'09: Affective Computing and Intelligent Interaction, pp. 217-223. IEEE, Los Alamitos (2009).

[74]

Wilder, J.: Stimulus and Response: The Law of Initial Values. Wright, Bristol (1967).

[75]

Wilhelm, P., Schoebi, D.: Assessing mood in daily life: structural validity, sensitivity to change, and reliability of a short-scale to measure three basic dimensions of mood. Eur. J. Psychol. Assess. 23, 258-267 (2007).

[76]

Wilson, G.F., Russell, C.A.: Real-time assessment ofmental workload using psychophysiological measures and artificial neural networks. Hum. Factors 45, 635-644 (2003).

[77]

Yannakakis, G.N., Hallam, J., Lund, H.H.: Entertainment capture through heart rate activity in physical interactive playgrounds. User Model. User-Adapt. Interact. 18, 207-243 (2008).

Digital Library

[78]

Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31, 39-58 (2009).

Digital Library

Cited By

de Assunção WZaina LSantos CBento Villela MRios da Hora Rodrigues Kde Gois Ribeiro Darin T(2022)Evaluating user experience in music discovery on deezer and spotifyProceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems10.1145/3554364.3560901(1-11)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3554364.3560901
Spelt HWesterink JFrank LHam JIJsselsteijn W(2022)Physiology-based personalization of persuasive technology: a user modeling perspectiveUser Modeling and User-Adapted Interaction10.1007/s11257-021-09313-832:1-2(133-163)Online publication date: 1-Apr-2022
https://dl.acm.org/doi/10.1007/s11257-021-09313-8
Assuncao WPiccolo LZaina L(2022)Considering emotions and contextual factors in music recommendation: a systematic literature reviewMultimedia Tools and Applications10.1007/s11042-022-12110-z81:6(8367-8407)Online publication date: 2-Feb-2022
https://dl.acm.org/doi/10.1007/s11042-022-12110-z
Show More Cited By

Tune in to your emotions: a robust personalized affective music player
1. Human-centered computing

Recommendations

Directing Physiology and Mood through Music: Validation of an Affective Music Player

Music is important in everyday life, as it provides entertainment and influences our moods. As music is widely available, it is becoming increasingly difficult to select songs to suit our mood. An affective music player can remove this obstacle by ...
Tangible Affect: A Literature Review of Tangible Interactive Systems Addressing Human Core Affect, Emotions and Moods
DIS '24: Proceedings of the 2024 ACM Designing Interactive Systems Conference

Tangible user interfaces (TUIs) have been applied to assess, communicate, or regulate human core affect, emotions, or moods. Previous studies identified TUIs as an innovative way to serve people’s affective needs. This review examines the design and ...
Emotions and heart rate while sitting on a chair
CHI '05: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

New methods for unobtrusive monitoring of computer users' emotion psychophysiology are very much needed in human-computer interaction research. The present aim was to study heart rate changes during emotionally provocative stimulation. Six-second long ...

Comments

Information & Contributors

Information

Published In

cover image User Modeling and User-Adapted Interaction

User Modeling and User-Adapted Interaction Volume 22, Issue 3

July 2012

88 pages

ISSN:0924-1868

Issue’s Table of Contents

Copyright © Copyright © 2012 Springer Science+Business Media B.V.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 July 2012

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

de Assunção WZaina LSantos CBento Villela MRios da Hora Rodrigues Kde Gois Ribeiro Darin T(2022)Evaluating user experience in music discovery on deezer and spotifyProceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems10.1145/3554364.3560901(1-11)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3554364.3560901
Spelt HWesterink JFrank LHam JIJsselsteijn W(2022)Physiology-based personalization of persuasive technology: a user modeling perspectiveUser Modeling and User-Adapted Interaction10.1007/s11257-021-09313-832:1-2(133-163)Online publication date: 1-Apr-2022
https://dl.acm.org/doi/10.1007/s11257-021-09313-8
Assuncao WPiccolo LZaina L(2022)Considering emotions and contextual factors in music recommendation: a systematic literature reviewMultimedia Tools and Applications10.1007/s11042-022-12110-z81:6(8367-8407)Online publication date: 2-Feb-2022
https://dl.acm.org/doi/10.1007/s11042-022-12110-z
Brandstetter CSuda FPapariello LDopler FGöschlberger B(2021)Exploring Learning Resource Recommendation Approaches for Secondary EducationThe 23rd International Conference on Information Integration and Web Intelligence10.1145/3487664.3487774(28-33)Online publication date: 29-Nov-2021
https://dl.acm.org/doi/10.1145/3487664.3487774
Liang YWillemsen MHammond TVerbert KParra DKnijnenburg BO'Donovan JTeale P(2021)Interactive Music Genre Exploration with Visualization and Mood ControlProceedings of the 26th International Conference on Intelligent User Interfaces10.1145/3397481.3450700(175-185)Online publication date: 14-Apr-2021
https://dl.acm.org/doi/10.1145/3397481.3450700
Gasparini FGiltri MBandini S(2020)Discriminating affective state intensity using physiological responsesMultimedia Tools and Applications10.1007/s11042-020-09114-y79:47-48(35845-35865)Online publication date: 1-Dec-2020
https://dl.acm.org/doi/10.1007/s11042-020-09114-y
de Assunção Wde Almeida Neris VGuimaraes RSaibel Santos Cde Santana VKronbauer A(2019)m-MotionProceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems10.1145/3357155.3358459(1-11)Online publication date: 22-Oct-2019
https://dl.acm.org/doi/10.1145/3357155.3358459
de Assuncao Wde Almeida Neris V(2018)An algorithm for music recommendation based on the user's musical preferences and desired emotionsProceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia10.1145/3282894.3282915(205-213)Online publication date: 25-Nov-2018
https://dl.acm.org/doi/10.1145/3282894.3282915
López-Ortega OCastro-Espinoza FPérez-Cortés O(2018)An intelligent multi-agent system to create and classify fractal musicComputing10.1007/s00607-017-0584-3100:7(671-688)Online publication date: 1-Jul-2018
https://dl.acm.org/doi/10.1007/s00607-017-0584-3
Yang XDong YLi J(2018)Review of data features-based music emotion recognition methodsMultimedia Systems10.1007/s00530-017-0559-424:4(365-389)Online publication date: 1-Jul-2018
https://dl.acm.org/doi/10.1007/s00530-017-0559-4
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents