research-article

Open access

Investigating the effect of visual realism on empathic responses to emotionally expressive virtual humans

Authors:

Darragh Higgins,

Benjamin R. Cowan, and

Rachel McDonnellAuthors Info & Claims

SAP '23: ACM Symposium on Applied Perception 2023

August 2023

Article No.: 5, Pages 1 - 7

https://doi.org/10.1145/3605495.3605799

Published: 05 August 2023 Publication History

All formats PDF

Abstract

With the remarkable improvement in technical systems for generating realistic virtual humans, there comes a requirement to quantify the effects that different aspects of realism can have on users. The study outlined here sought to advance research on emotion perception and virtual humans by assessing basic empathic responses to high fidelity emotionally expressive characters. We report findings on participants experiences of cognitive, affective and compassionate empathy, as well as measurements for the uncanny valley at two levels of visual realism. We find that the levels of emotion expressed by virtual humans within our study influenced ratings of empathy and the uncanny valley, with positive and negative valence having significant effects on empathic responses and perceived appeal. We discuss these findings in relation to studies which have measured empathy responses to animated characters, as well as notable differences in uncanny valley measurements.

Supplementary Material

MP4 File (SAP_Empathy_2023_CamReadyAuthors.mp4)

Supplemental video for SAP 23-10

Download
44.26 MB

References

[1]

Victor Araujo, Bruna Dalmoro, and Soraia Raupp Musse. 2021. Analysis of charisma, comfort and realism in CG characters from a gender perspective. Visual Computer 37 (9 2021), 2685–2698. Issue 9-11. https://doi.org/10.1007/s00371-021-02214-2

Digital Library

[2]

Lisa Feldman Barrett, Ralph Adolphs, Stacy Marsella, Aleix M. Martinez, and Seth D. Pollak. 2019. Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements. Psychological science in the public interest : a journal of the American Psychological Society 20 (7 2019), 1–68. Issue 1. https://doi.org/10.1177/1529100619832930

[3]

Stéphane Bouchard, François Bernier, Éric Boivin, Stéphanie Dumoulin, Mylène Laforest, Tanya Guitard, Geneviève Robillard, Johana Monthuy-Blanc, and Patrice Renaud. 2013. Empathy toward virtual humans depicting a known or unknown person expressing pain. Cyberpsychology, Behavior, and Social Networking 16 (1 2013), 61–71. Issue 1. https://doi.org/10.1089/cyber.2012.1571

[4]

Tyler J. Burleigh, Jordan R. Schoenherr, and Guy L. Lacroix. 2013. Does the Uncanny Valley exist? An empirical test of the relationship between eeriness and the human likeness of digitally created faces. Computers in Human Behavior 29, 3 (2013), 759–771.

Digital Library

[5]

Marcus Cheetham, Lingdan Wu, Paul Pauli, and Lutz Jancke. 2015. Arousal, valence, and the uncanny valley: psychophysiological and self-report findings. Frontiers in Psychology 6 (7 2015). https://doi.org/10.3389/fpsyg.2015.00981

[6]

M Davis. 1980. A multidimensional approach to individual differences in empathy. JSAS Catalog of Selected Documents in Psychology 10 (1980), 85–103.

[7]

Mark H. Davis. 1983. Measuring individual differences in empathy: Evidence for a multidimensional approach. Journal of Personality and Social Psychology 44 (1983), 113–126. Issue 1. https://doi.org/10.1037/0022-3514.44.1.113

[8]

Ronan de Kervenoael, Rajibul Hasan, Alexandre Schwob, and Edwin Goh. 2020. Leveraging human-robot interaction in hospitality services: Incorporating the role of perceived value, empathy, and information sharing into visitors’ intentions to use social robots. Tourism Management 78 (2020). https://doi.org/10.1016/j.tourman.2019.104042

[9]

Celso M. de Melo, Jonathan Gratch, Stacy Marsella, and Catherine Pelachaud. 2023. Social Functions of Machine Emotional Expressions. Proc. IEEE (2023), 1–16. https://doi.org/10.1109/JPROC.2023.3261137

[10]

Annelies Devesse, Alexander Dudek, Astrid van Wieringen, and Jan Wouters. 2018. Speech intelligibility of virtual humans. International Journal of Audiology 57 (12 2018), 908–916. Issue 12. https://doi.org/10.1080/14992027.2018.1511922

[11]

Changming Duan. 1996. The Current State of Empathy Research. Journal of Counseling Psychology 43 (1996), 261–274. Issue 3.

[12]

Changming Duan. 2000. Being Empathic:The Role of Motivation to Empathize and the Nature of Target Emotions. Motivation and Emotion 24 (2000). Issue 1.

[13]

Funda Durupinar and Jiehyun Kim. 2022. Facial Emotion Recognition of Virtual Humans with Different Genders, Races, and Ages. Proceedings - SAP 2022: ACM Symposium on Applied Perception. https://doi.org/10.1145/3548814.3551464

Digital Library

[14]

Paul Ekman. 1999. Handbook of Cognition and Emotion. John Wiley & Sons, Ltd.

[15]

Rasmus Gahrn-Andersen. 2020. Seeming autonomy, technology and the uncanny valley. AI and Society (2020). https://doi.org/10.1007/s00146-020-01040-9

Digital Library

[16]

Xuemei Gao, Lei Weng, Yuhong Zhou, and Hongling Yu. 2017. The influence of empathy and morality of violent video game characters on gamers’ aggression. Frontiers in Psychology 8 (11 2017). Issue NOV. https://doi.org/10.3389/fpsyg.2017.01863

[17]

Kurt Gray and Daniel M. Wegner. 2012. Feeling robots and human zombies: Mind perception and the uncanny valley. Cognition 125 (10 2012), 125–130. Issue 1. https://doi.org/10.1016/j.cognition.2012.06.007

[18]

Shivashankar Halan, Isaac Sia, Michael Crary, and Benjamin Lok. 2015. Exploring the effects of healthcare students creating virtual patients for empathy training. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 9238, 239–249. https://doi.org/10.1007/978-3-319-21996-7_24

[19]

Darragh Higgins, Donal Egan, Rebecca Fribourg, Benjamin Cowan, and Rachel McDonnell. 2021a. Ascending from the valley: Can state-of-the-art photorealism avoid the uncanny?ACM Symposium on Applied Perception 2021, 1–5. https://doi.org/10.1145/3474451.3476242

Digital Library

[20]

Darragh Higgins, Rebecca Fribourg, and Rachel Mcdonnell. 2021b. Remotely Perceived: Investigating the Influence of Valence on Self-Perception and Social Experience for Dyadic Video-Conferencing With Personalized Avatars. (2021). https://doi.org/10.3389/frvir.2021.668499

[21]

Darragh Higgins, Katja Zibrek, Joao Cabral, Donal Egan, and Rachel McDonnell. 2022. Sympathy for the digital: Influence of synthetic voice on affinity, social presence and empathy for photorealistic virtual humans. Computers & Graphics (3 2022). https://doi.org/10.1016/J.CAG.2022.03.009

[22]

Patrik Jonell, Taras Kucherenko, Ilaria Torre, and Jonas Beskow. 2020. Can we trust online crowdworkers?: Comparing online and offline participants in a preference test of virtual agents. Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents, IVA 2020. https://doi.org/10.1145/3383652.3423860

Digital Library

[23]

Julian Kiverstein. 2015. Empathy and the responsiveness to social affordances. (2015). https://doi.org/10.1016/j.concog.2015.05.002

[24]

Andrea Kleinsmith, Diego Rivera-Gutierrez, Glen Finney, Juan Cendan, and Benjamin Lok. 2015. Understanding empathy training with virtual patients. Computers in Human Behavior 52 (6 2015), 151–158. https://doi.org/10.1016/j.chb.2015.05.033

Digital Library

[25]

Demi Krystallidou, Aline Remael, Esther de Boe, Kristin Hendrickx, Giannoula Tsakitzidis, Sofie van de Geuchte, and Peter Pype. 2018. Investigating empathy in interpreter-mediated simulated consultations: An explorative study. Patient Education and Counseling 101 (1 2018), 33–42. Issue 1. https://doi.org/10.1016/j.pec.2017.07.022

[26]

Jari Kätsyri, Meeri Mäkäräinen, and Tapio Takala. 2017. Testing the ‘uncanny valley’ hypothesis in semirealistic computer-animated film characters: An empirical evaluation of natural film stimuli. International Journal of Human Computer Studies 97 (1 2017), 149–161. https://doi.org/10.1016/j.ijhcs.2016.09.010

[27]

Charles Lassiter. 2021. Could a robot flirt? 4E cognition, reactive attitudes, and robot autonomy. AI and Society (2021). https://doi.org/10.1007/s00146-020-01116-6

Digital Library

[28]

Yong Il Lee, Yeojeong Choi, and Jaeseung Jeong. 2017. Character drawing style in cartoons on empathy induction: An eye-tracking and EEG study. PeerJ 2017 (2017). Issue 11. https://doi.org/10.7717/peerj.3988

[29]

P. Lush, V. Botan, R. B. Scott, A. K. Seth, J. Ward, and Z. Dienes. 2020. Trait phenomenological control predicts experience of mirror synaesthesia and the rubber hand illusion. Nature Communications 2020 11:1 11 (9 2020), 1–10. Issue 1. https://doi.org/10.1038/s41467-020-18591-6

[30]

Karl F. MacDorman, Robert D. Green, Chin-Chang Ho, and Clinton T. Koch. 2009. Too Real for Comfort? Uncanny Responses to Computer Generated Faces. Comput. Hum. Behav. 25, 3 (May 2009), 695–710. https://doi.org/10.1016/j.chb.2008.12.026

Digital Library

[31]

Wade J. Mitchell, Kevin A. Szerszen, Amy Shirong Lu, Paul W. Schermerhorn, Matthias Scheutz, and Karl F. MacDorman. 2011. A mismatch in the human realism of face and voice produces an uncanny valley. i-Perception 2 (2011), 10–12. Issue 1. https://doi.org/10.1068/i0415

[32]

M. Mori. 1970. The Uncanny Valley. Energy 7, 4 (1970), 33–35.

[33]

Masahiro Mori, Karl F. MacDorman, and Norri Kageki. 2012. The uncanny valley. IEEE Robotics and Automation Magazine 19 (2012), 98–100. Issue 2. https://doi.org/10.1109/MRA.2012.2192811

[34]

Ewald Moser, Birgit Derntl, Simon Robinson, Bernhard Fink, Ruben C. Gur, and Karl Grammer. 2007. Amygdala activation at 3T in response to human and avatar facial expressions of emotions. Journal of Neuroscience Methods 161 (3 2007), 126–133. Issue 1. https://doi.org/10.1016/j.jneumeth.2006.10.016

[35]

Christos Mousas, Dimitris Anastasiou, and Ourania Spantidi. 2018. The effects of appearance and motion of virtual characters on emotional reactivity. Computers in Human Behavior 86 (9 2018), 99–108. https://doi.org/10.1016/j.chb.2018.04.036

[36]

Meeri Mäkäräinen, Jari Kätsyri, Klaus Förger, and Tapio Takala. 2015. The funcanny valley: A study of positive emotional reactions to strangeness. ACADEMICMINDTREK 2015 - Proceedings of the 19th International Academic Mindtrek Conference, 175–181. https://doi.org/10.1145/2818187.2818292

Digital Library

[37]

Soo Youn Oh, Jeremy Bailenson, Nicole Krämer, and Benjamin Li. 2016. Let the avatar brighten your smile: Effects of enhancing facial expressions in virtual environments. PLoS ONE 11 (9 2016). Issue 9. https://doi.org/10.1371/journal.pone.0161794

[38]

Philip A. Powell and Jennifer Roberts. 2017. Situational determinants of cognitive, affective, and compassionate empathy in naturalistic digital interactions. Computers in Human Behavior 68 (3 2017), 137–148. https://doi.org/10.1016/j.chb.2016.11.024

Digital Library

[39]

Astrid M. Rosenthal-Von Der Pütten, Frank P. Schulte, Sabrina C. Eimler, Sabrina Sobieraj, Laura Hoffmann, Stefan Maderwald, Matthias Brand, and Nicole C. Krämer. 2014. Investigations on empathy towards humans and robots using fMRI. Computers in Human Behavior 33 (4 2014), 201–212. https://doi.org/10.1016/j.chb.2014.01.004

Digital Library

[40]

Aravind Sesagiri Raamkumar and Yinping Yang. 2020. IEEE TRANSACTIONS ON AFFECTIVE COMPUTING 1 Empathetic Conversational Systems: A Review of Current Advances, Gaps, and Opportunities. (2020).

[41]

Elizabeth B. D. Ripley. 2006. A Review of Paying Research Participants: It’s Time to Move beyond the Ethical Debate. Journal of Empirical Research on Human Research Ethics 1, 4 (2006), 9–19. https://doi.org/10.1525/jer.2006.1.4.9 arXiv:https://doi.org/10.1525/jer.2006.1.4.9PMID: 19385834.

[42]

Mike Seymour. 2019. FACS at 40; Facial action coding System panel. ACM SIGGRAPH 2019 Panels, SIGGRAPH 2019. https://doi.org/10.1145/3306212.3328132

Digital Library

[43]

Mike Seymour, Lingyao Yuan, Alan R Dennis, and Kai Riemer. 2019. Crossing the Uncanny Valley? Understanding Affinity, Trustworthiness, and Preference for More Realistic Virtual Humans in Immersive Environments. https://hdl.handle.net/10125/59615

[44]

Mel Slater, Angus Antley, Adam Davison, David Swapp, Christoph Guger, Chris Barker, Nancy Pistrang, and Maria V. Sanchez-Vives. 2006. A Virtual Reprise of the Stanley Milgram Obedience Experiments. PLOS ONE 1, 1 (12 2006), 1–10. https://doi.org/10.1371/journal.pone.0000039

[45]

Mel Slater, Bruno Herbelin, Ryan Patrick McMahan, Philippe Bertrand, Franck Zenasni, Jérôme Guegan, Léonore Robieux, and Cade Andrew McCall. 2018. Learning empathy Through virtual Reality: Multiple Strategies for Training empathy-Related Abilities Using Body Ownership illusions in embodied virtual Reality. Article 5 (2018), 1. https://doi.org/10.3389/frobt.2018.00026

[46]

Raphaël Thézé, Mehdi Ali Gadiri, Louis Albert, Antoine Provost, Anne Lise Giraud, and Pierre Mégevand. 2020. Animated virtual characters to explore audio-visual speech in controlled and naturalistic environments. Scientific Reports 10 (12 2020). Issue 1. https://doi.org/10.1038/s41598-020-72375-y

[47]

Angela Tinwell, Mark Grimshaw, Debbie Abdel Nabi, and Andrew Williams. 2011. Facial expression of emotion and perception of the Uncanny Valley in virtual characters. Computers in Human Behavior 27 (3 2011), 741–749. Issue 2. https://doi.org/10.1016/j.chb.2010.10.018

Digital Library

[48]

Rahma M Tolba, Taha Al-Arif, and El-Sayed M El Horbaty. 2018. Realistic Facial Animation Review: Based on Facial Action Coding System. Egyptian Computer Science Journal 42 (2018). Issue 1.

[49]

Ilaria Torre, Emma Carrigan, Killian McCabe, Rachel McDonnell, and Naomi Harte. 2018. Survival at the museum: A cooperation experiment with emotionally expressive virtual characters. ICMI 2018 - Proceedings of the 2018 International Conference on Multimodal Interaction, 423–427. https://doi.org/10.1145/3242969.3242984

Digital Library

[50]

Austin van Loon, Jeremy Bailenson, Jamil Zaki, Joshua Bostick, and Robb Willer. 2018. Virtual reality perspective-taking increases cognitive empathy for specific others. PLoS ONE 13 (8 2018). Issue 8. https://doi.org/10.1371/journal.pone.0202442

[51]

Matias Volonte, Reza Ghaiumy Anaraky, Bart P. Knijnenburg, Andrew T. Duchowski, and Sabarish V. Babu. 2019. Empirical evaluation of the interplay of emotion and visual attention in human-virtual human interaction. Proceedings - SAP 2019: ACM Conference on Applied Perception. https://doi.org/10.1145/3343036.3343118

Digital Library

[52]

Katja Zibrek, Sean Martin, and Rachel McDonnell. 2019a. Is Photorealism Important for Perception of Expressive Virtual Humans in Virtual Reality?ACM Trans. Appl. Percept. 16, 3, Article 14 (Sept. 2019), 19 pages. https://doi.org/10.1145/3349609

Digital Library

[53]

Katja Zibrek, Sean Martin, and Rachel McDonnell. 2019b. Is Photorealism Important for Perception of Expressive Virtual Humans in Virtual Reality?ACM Transactions on Applied Perception 16, 3, Article 14 (sep 2019), 19 pages. https://doi.org/10.1145/3349609

Digital Library

Cited By

Rekik RWuhrer SHoyet LZibrek KOlivier A(2024)A Survey on Realistic Virtual Human Animations: Definitions, Features and EvaluationsComputer Graphics Forum10.1111/cgf.1506443:2Online publication date: 30-Apr-2024
https://doi.org/10.1111/cgf.15064
Lee YJi Y(2024)Effects of Visual Realism on Avatar Perception in Immersive and Non-Immersive Virtual EnvironmentsInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2351713(1-14)Online publication date: 14-May-2024
https://doi.org/10.1080/10447318.2024.2351713
Yurtoğlu ASonlu SDoğan YGüdükbay U(2024)Personality perception in human videos altered by motion transfer networksComputers & Graphics10.1016/j.cag.2024.01.013119(103886)Online publication date: Apr-2024
https://doi.org/10.1016/j.cag.2024.01.013

Index Terms

Investigating the effect of visual realism on empathic responses to emotionally expressive virtual humans
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Perception

Recommendations

Social media users’ affective, attitudinal, and behavioral responses to virtual human emotions
Highlights
- Social media users demonstrate affective responses toward virtual human emotions.
- Affective responses lead to increased attitudes and behavioral intentions related to the virtual human.
- Lust has the strongest impact on social media ...
Abstract
Virtual humans’ communication of emotions in social media posts are considered heuristics of human behavior. The current study extends the Cognitive-Functional Model of Emotions to virtual influencers and investigates how social media users ...
Read More
Effect of Appearance and Animation Realism on the Perception of Emotionally Expressive Virtual Humans
IVA '23: Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents

3D Virtual Human technology is growing with several potential applications in health, education, business and telecommunications. Investigating the perception of these virtual humans can help guide to develop better and more effective applications. ...
Read More
Psychological responses to simulated displays of mismatched emotional expressions

Embodied agents are often designed with the ability to simulate human emotion. This paper investigates the psychological impact of simulated emotional expressions on computer users with a particular emphasis on how mismatched facial and audio ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SAP '23: ACM Symposium on Applied Perception 2023

August 2023

111 pages

ISBN:9798400702525

DOI:10.1145/3605495

Editors:
Alex Chapiro
Meta, USA
,
Andrew Robb
Clemson University, USA
,
Funda Durupinar
UMass Boston, USA
,
Qi Sun
New York University, USA
,
Lauren Buck
Trinity College, Ireland

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 August 2023

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

RADICal
ADAPT Centre for Digital Content Technology
Science Foundation Ireland Centre for Research Training in DigitallyEnhanced Reality (d-real)

Conference

SAP '23

Sponsor:

SIGGRAPH

SAP '23: ACM Symposium on Applied Perception 2023

August 5 - 6, 2023

CA, Los Angeles, USA

Acceptance Rates

Overall Acceptance Rate 43 of 94 submissions, 46%

Upcoming Conference

SAP '24

Sponsor:
siggraph

ACM Symposium on Applied Perception 2024

August 30 - 31, 2024

Dublin , Ireland

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
666
Total Downloads

Downloads (Last 12 months)666
Downloads (Last 6 weeks)67

Other Metrics

View Author Metrics

Citations

Cited By

Rekik RWuhrer SHoyet LZibrek KOlivier A(2024)A Survey on Realistic Virtual Human Animations: Definitions, Features and EvaluationsComputer Graphics Forum10.1111/cgf.1506443:2Online publication date: 30-Apr-2024
https://doi.org/10.1111/cgf.15064
Lee YJi Y(2024)Effects of Visual Realism on Avatar Perception in Immersive and Non-Immersive Virtual EnvironmentsInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2351713(1-14)Online publication date: 14-May-2024
https://doi.org/10.1080/10447318.2024.2351713
Yurtoğlu ASonlu SDoğan YGüdükbay U(2024)Personality perception in human videos altered by motion transfer networksComputers & Graphics10.1016/j.cag.2024.01.013119(103886)Online publication date: Apr-2024
https://doi.org/10.1016/j.cag.2024.01.013

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents