Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Emotion-Driven Interactive Storytelling: Let Me Tell You How to Feel

  • Conference paper
  • First Online:
Artificial Intelligence in Music, Sound, Art and Design (EvoMUSART 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13221))

  • 2331 Accesses

Abstract

Interactive storytelling is a form of digital entertainment that has gained attention with the development of creative computational methodologies. However, one of the main problems this field is facing is the poor control that the content creator (e.g. film director or game designer) has over the experience of the user (e.g. viewer or player) once the story starts. Hence, we leverage artificial intelligence to increase the creative control of the content creator by designing a system that guides the user’s emotions towards a particular state as the story unfolds. Specifically, we have developed an EEG-based emotion recognition system trained on EEG recordings acquired from 5 participants watching a selection of 384 videos. The system is able to operate a binary classification on both valence and arousal with an accuracy of 62% and 57%, respectively. A short film was then created, where each scene automatically adapts to the user’s emotion, based on a set of predefined interactions established by the content creator (i.e. the actual film director). The analysis shows that the system not only improves the engagement of the user, but also induces an emotion closer to the one intended and specified ahead of time by the content creator for the story. Our results indicate that there is a practical application of emotion-based studies for future content creators to better control an intended emotional response delivered and received by the audience.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://www.pwc.com/us/en/industries/tmt/library/global-entertainment-media-outlook.html.

  2. 2.

    https://www.nasdaq.com/articles/investing-video-games-industry-pulls-more-revenue-movies-music-2016-06-13.

  3. 3.

    https://nebulas.sfwa.org/.

  4. 4.

    https://shop.openbci.com/products/ultracortex-mark-iv?variant=23280742211.

  5. 5.

    A teaser for this short film is available at https://sendvid.com/nlxw45pi.

References

  1. Al Machot, F., Elmachot, A., Ali, M., Al Machot, E., Kyamakya, K.: A deep-learning model for subject-independent human emotion recognition using electrodermal activity sensors. Sensors 19(7) (2019). https://doi.org/10.3390/s19071659, https://www.mdpi.com/1424-8220/19/7/1659

  2. Alserri, S.A., Zin, N.A.M., Wook, T.S.M.T.: Instrument validation for evaluating serious game engagement model. In: 2019 International Conference on Electrical Engineering and Informatics (ICEEI), pp. 170–175. IEEE (2019)

    Google Scholar 

  3. Barrett, L.F., Adolphs, R., Marsella, S., Martinez, A.M., Pollak, S.D.: Emotional expressions reconsidered: challenges to inferring emotion from human facial movements. Psychol. Sci. Public Interest 20(1), 1–68 (2019)

    Article  Google Scholar 

  4. Baveye, Y., Dellandréa, E., Chamaret, C., Chen, L.: LIRIS-ACCEDE: a video database for affective content analysis. IEEE Trans. Affect. Comput. 6, 43–55 (2015). https://doi.org/10.1109/TAFFC.2015.2396531

  5. Bellantoni, P.: If It’s Purple, Someone’s Gonna Die: The Power of Color in Visual Storytelling. Taylor & Francis (2012). https://books.google.co.jp/books?id=E57cAwAAQBAJ

  6. Böck, R., et al.: Intraindividual and interindividual multimodal emotion analyses in human-machine-interaction. In: 2012 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support, pp. 59–64. IEEE (2012)

    Google Scholar 

  7. Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment manikin and the semantic differential. J. Behav. Therapy Exp. Psychiatry 25(1), 49–59 (1994). https://doi.org/10.1016/0005-7916(94)90063-9, https://www.sciencedirect.com/science/article/pii/0005791694900639

  8. Bradley, M.M., Miccoli, L., Escrig, M.A., Lang, P.J.: The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology 45(4), 602–607 (2008)

    Article  Google Scholar 

  9. Carlton, J., Brown, A., Jay, C., Keane, J.: Inferring user engagement from interaction data. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–6 (2019)

    Google Scholar 

  10. Caruso, E.M., Burns, Z.C., Converse, B.A.: Slow motion increases perceived intent. Proc. Natl. Acad. Sci. 113(33), 9250–9255 (2016). https://doi.org/10.1073/pnas.1603865113, https://www.pnas.org/content/113/33/9250

  11. Ciancone Chama, A.G., Monaro, M., Piccoli, E., Gamberini, L., Spagnolli, A.: Engaging the audience with biased news: an exploratory study on prejudice and engagement. In: Oinas-Kukkonen, H., Win, K.T., Karapanos, E., Karppinen, P., Kyza, E. (eds.) PERSUASIVE 2019. LNCS, vol. 11433, pp. 350–361. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-17287-9_28

    Chapter  Google Scholar 

  12. Damiano, R., Lombardo, V., Monticone, G., Pizzo, A.: All about face. An experiment in face emotion recognition in interactive dramatic performance. In: 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), pp. 1–7 (2019). https://doi.org/10.1109/ACIIW.2019.8925032

  13. De Andrés, I., Garzón, M., Reinoso-Suárez, F.: Functional anatomy of non-REM sleep. Front. Neurol. 1–14 (2011). https://doi.org/10.3389/fneur.2011.00070

  14. Deldin, P.J., Keller, J., Gergen, J.A., Miller, G.A.: Cognitive bias and emotion in neuropsychological models of depression. Cogn. Emot. 15(6), 787–802 (2001)

    Article  Google Scholar 

  15. Frey, J., Ostrin, G., Grabli, M., Cauchard, J.R.: Physiologically driven storytelling: concept and software tool. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI 2020, pp. 1–13. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3313831.3376643, https://doi.org/10.1145/3313831.3376643

  16. Hauge, M.: Writing Screenplays That Sell. Bloomsbury Publishing (2011). https://books.google.co.jp/books?id=6I9qDwAAQBAJ

  17. Iwamoto, S.: Epic and interactive music in ‘final fantasy xv’ (2017). https://www.gdcvault.com/play/1023971/Epic-AND-Interactive-Music-in

  18. Koelstra, S., et al.: DEAP: a database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2012). https://doi.org/10.1109/T-AFFC.2011.15

  19. Lang, P.J.: The emotion probe. Am. Psychol. Assoc. 50, 372–385 (1995). https://doi.org/10.1037/0003-066X.50.5.372

  20. Laurans, G., Desmet, P.M., Hekkert, P.P.: Assessing emotion in interaction: some problems and a new approach. In: Proceedings of the 4th International Conference on Designing Pleasurable Products and Interfaces, DPPI 2009, Compiegne, October 2009. Universite de Technologie de Compiegne (2009)

    Google Scholar 

  21. Levenson, R.W., Ekman, P., Friesen, W.V.: Voluntary facial action generates emotion-specific autonomic nervous system activity. Psychophysiology 27(4), 363–384 (1990)

    Article  Google Scholar 

  22. Liu, J., et al.: EEG-based emotion classification using a deep neural network and sparse autoencoder. Front. Syst. Neurosci. 14, 43 (2020). https://doi.org/10.3389/fnsys.2020.00043, https://www.frontiersin.org/article/10.3389/fnsys.2020.00043

  23. Liu, Y., Sourina, O., Nguyen, M.K.: Real-time EEG-based human emotion recognition and visualization. In: 2010 International Conference on Cyberworlds, pp. 262–269. IEEE (2010)

    Google Scholar 

  24. Marlowe, D., Crowne, D.P.: Social desirability and response to perceived situational demands. J. Consult. Psychol. 25, 109–15 (1961)

    Article  Google Scholar 

  25. Nardelli, M., Valenza, G., Greco, A., Lanata, A., Scilingo, E.P.: Recognizing emotions induced by affective sounds through heart rate variability. IEEE Trans. Affect. Comput. 6(4), 385–394 (2015). https://doi.org/10.1109/TAFFC.2015.2432810

    Article  Google Scholar 

  26. O’Brien, H.: Translating theory into methodological practice. In: O’Brien, H., Cairns, P. (eds.) Why Engagement Matters, pp. 27–52. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-27446-1_2

    Chapter  Google Scholar 

  27. O’Brien, H.L., Cairns, P., Hall, M.: A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form. Int. J. Hum.-Comput. Stud. 112, 28–39 (2018). https://doi.org/10.1016/j.ijhcs.2018.01.004, https://www.sciencedirect.com/science/article/pii/S1071581918300041

  28. Pasquali, A., Cleeremans, A., Gaillard, V.: Reversible second-order conditional sequences in incidental sequence learning tasks. Q. J. Exp. Psychol. 72(5), 1164–1175 (2019). https://doi.org/10.1177/1747021818780690, https://doi.org/10.1177/1747021818780690, pMID: 29779443

  29. Perea Restrepo, C.M.: Limpieza social. Una violencia mal nombrada. Bogotá: Centro Nacional de Memoria Histórica (2019). http://www.cervantesvirtual.com/obra/limpieza-social-una-violencia-mal-nombrada-879231

  30. Perron, B., Arsenault, D., Picard, M., Therrien, C.: Methodological questions in ‘interactive film studies’. New Rev. Film Telev. Stud. 6(3), 233–252 (2008)

    Article  Google Scholar 

  31. Potel, M.J., Sayre, R.E., Robertson, A.: A system for interactive film analysis. Comput. Biol. Med. 9(3), 237–256 (1979)

    Article  Google Scholar 

  32. Prokasy, W.: Electrodermal activity in psychological research. Elsevier Sci. (2012). https://books.google.co.jp/books?id=m9l5ApC3avoC

  33. Radúz, C., Ján, R., Vladimír, S.: Kinoautomat: One Man and His House. Czechoslovakia (1967)

    Google Scholar 

  34. Rico, O., Tag, B., Ohta, N., Sugiura, K.: Seamless multithread films in virtual reality. In: Proceedings of the Eleventh International Conference on Tangible, Embedded, and Embodied Interaction, TEI 2017. ACM, New York, pp. 641–646 (2017). https://doi.org/10.1145/3024969.3025096, http://doi.acm.org/10.1145/3024969.3025096

  35. Saeghe, P., et al.: Augmenting television with augmented reality. In: Proceedings of the 2019 ACM International Conference on Interactive Experiences for TV and Online Video, pp. 255–261 (2019)

    Google Scholar 

  36. Shu, L., et al.: A review of emotion recognition using physiological signals. Sensors (Switzerland) 18 (2018). https://doi.org/10.3390/s18072074

  37. Slade, D.: Black Mirror: Bandersnatch (2018)

    Google Scholar 

  38. Welch, P.D.: The use of Fast Fourier transform for the estimation of power spectra. Digit. Sig. Process. 15(2), 532–574 (1975)

    Google Scholar 

  39. Wu, S., Du, Z., Li, W., Huang, D., Wang, Y.: Continuous emotion recognition in videos by fusing facial expression, head pose and eye gaze. In: 2019 International Conference on Multimodal Interaction, pp. 40–48 (2019)

    Google Scholar 

  40. Yadati, K., Katti, H., Kankanhalli, M.: Interactive video advertising: a multimodal affective approach. In: Li, S., et al. (eds.) MMM 2013. LNCS, vol. 7732, pp. 106–117. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-35725-1_10

    Chapter  Google Scholar 

  41. Zheng, W.L., Dong, B.N., Lu, B.L.: Multimodal emotion recognition using EEG and eye tracking data. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 5040–5043. IEEE (2014)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Cross Labs, Cross Compass, Kyoto, Japan

    Oneris Daniel Rico Garcia, Javier Fernandez Fernandez & Olaf Witkowski

  2. Bogotá, Colombia

    Rafael Andres Becerra Saldana

Authors
  1. Oneris Daniel Rico Garcia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Javier Fernandez Fernandez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rafael Andres Becerra Saldana
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Olaf Witkowski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Oneris Daniel Rico Garcia .

Editor information

Editors and Affiliations

  1. University of Coimbra, Coimbra, Portugal

    Tiago Martins

  2. University of A Coruña, A Coruña, Spain

    Nereida Rodríguez-Fernández

  3. University of Coimbra, Coimbra, Portugal

    Sérgio M. Rebelo

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rico Garcia, O.D., Fernandez Fernandez, J., Becerra Saldana, R.A., Witkowski, O. (2022). Emotion-Driven Interactive Storytelling: Let Me Tell You How to Feel. In: Martins, T., Rodríguez-Fernández, N., Rebelo, S.M. (eds) Artificial Intelligence in Music, Sound, Art and Design. EvoMUSART 2022. Lecture Notes in Computer Science, vol 13221. Springer, Cham. https://doi.org/10.1007/978-3-031-03789-4_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-03789-4_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-03788-7

  • Online ISBN: 978-3-031-03789-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation