research-article

Empirical Evaluation of the Interplay of Emotion and Visual Attention in Human-Virtual Human Interaction

Authors:

Matias Volonte,

Reza Ghaiumy Anaraky,

Bart Knijnenburg,

Andrew T. Duchowski,

Sabarish V. BabuAuthors Info & Claims

SAP '19: ACM Symposium on Applied Perception 2019

Article No.: 1, Pages 1 - 9

https://doi.org/10.1145/3343036.3343118

Published: 19 September 2019 Publication History

Abstract

We examined the effect of rendering style and the interplay between attention and emotion in users during interaction with a virtual patient in a medical training simulator. The virtual simulation was rendered representing a sample from the photo-realistic to the non-photorealistic continuum, namely Near-Realistic, Cartoon or Pencil-Shader. In a mixed design study, we collected 45 participants’ emotional responses and gaze behavior using surveys and an eye tracker while interacting with a virtual patient who was medically deteriorating over time. We used a cross-lagged panel analysis of attention and emotion to understand their reciprocal relationship over time. We also performed a mediation analysis to compare the extent to which the virtual agent’s appearance and his affective behavior impacted users’ emotional and attentional responses. Results showed the interplay between participants’ visual attention and emotion over time and also showed that attention was a stronger variable than emotion during the interaction with the virtual human.

References

[1]

Russell Beale and Chris Creed. 2009. Affective interaction: How emotional agents affect users. International Journal of Human-Computer Studies 67, 9 (2009), 755–776.

Digital Library

[2]

Edward Bernat, Christopher J Patrick, Stephen D Benning, and Auke Tellegen. 2006. Effects of picture content and intensity on affective physiological response. Psychophysiology 43, 1 (2006), 93–103.

[3]

Tobias Brosch, Klaus R Scherer, Didier Maurice Grandjean, and David Sander. 2013. The impact of emotion on perception, attention, memory, and decision-making. Swiss medical weekly 143(2013), w13786.

[4]

Manuel G Calvo and Lauri Nummenmaa. 2007. Processing of unattended emotional visual scenes.Journal of Experimental Psychology: General 136, 3 (2007), 347.

[5]

Elizabeth J Carter, Moshe Mahler, and Jessica K Hodgins. 2013. Unpleasantness of animated characters corresponds to increased viewer attention to faces. In Proceedings of the ACM Symposium on Applied Perception. ACM, 35–40.

Digital Library

[6]

Marcus Cheetham, Ivana Pavlovic, Nicola Jordan, Pascal Suter, and Lutz Jancke. 2013. Category processing and the human likeness dimension of the uncanny valley hypothesis: eye-tracking data. Frontiers in psychology 4 (2013), 108.

[7]

Maurizio Corbetta and Gordon L Shulman. 2002. Control of goal-directed and stimulus-driven attention in the brain. Nature reviews neuroscience 3, 3 (2002), 201.

[8]

Andrew Duchowski. 2017. Eye tracking methodology: Theory and practice (3rd ed.). Springer-Verlag, Inc., London, UK.

Digital Library

[9]

Qi Dunsworth and Robert K Atkinson. 2007. Fostering multimedia learning of science: Exploring the role of an animated agent’s image. Computers & Education 49, 3 (2007), 677–690.

Digital Library

[10]

John D Eastwood, Daniel Smilek, and Philip M Merikle. 2001. Differential attentional guidance by unattended faces expressing positive and negative emotion. Attention, Perception, & Psychophysics 63, 6 (2001), 1004–1013.

[11]

Nancy Eisenberg, Tracy L Spinrad, and Natalie D Eggum. 2010. Emotion-related self-regulation and its relation to children’s maladjustment. Annual review of clinical psychology 6 (2010), 495–525.

[12]

Paula M Ellis and Joanna J Bryson. 2005. The significance of textures for affective interfaces. In International Workshop on Intelligent Virtual Agents. Springer, 394–404.

Digital Library

[13]

James J Gross. 2001. Emotion regulation in adulthood: Timing is everything. Current directions in psychological science 10, 6 (2001), 214–219.

[14]

Nick Halper, Mara Mellin, Christoph S Herrmann, Volker Linneweber, and Thomas Strothotte. 2003. Psychology and non-photorealistic rendering: The beginning of a beautiful relationship. In Mensch & Computer 2003. Springer, 277–286.

[15]

Bernhard Jung, Abdul Ahad, and Matthias Weber. 2005. The affective virtual patient: An E-learning tool for social interaction training within the medical field. In Proceedings of International Conference TESI-Training Education & Education.

[16]

Michael W Kearney. 2017. Cross lagged panel analysis. The SAGE encyclopedia of communication research methods (2017), 1–6.

[17]

David A Kenny. 1975. Cross-lagged panel correlation: A test for spuriousness.Psychological bulletin 82, 6 (1975), 887.

[18]

Bart P Knijnenburg and Martijn C Willemsen. 2016. Inferring capabilities of intelligent agents from their external traits. ACM Transactions on Interactive Intelligent Systems (TiiS) 6, 4(2016), 28.

Digital Library

[19]

Peter J Lang, Robert F Simons, Marie Balaban, and Robert Simons. 2013. Attention and orienting: Sensory and motivational processes. Psychology Press.

[20]

Richard S Lazarus. 1982. Thoughts on the relations between emotion and cognition.American psychologist 37, 9 (1982), 1019.

[21]

Karl F MacDorman, Robert D Green, Chin-Chang Ho, and Clinton T Koch. 2009. Too real for comfort? Uncanny responses to computer generated faces. Computers in human behavior 25, 3 (2009), 695–710.

Digital Library

[22]

Rachel McDonnell, Martin Breidt, and Heinrich H Bülthoff. 2012. Render me real?: investigating the effect of render style on the perception of animated virtual humans. ACM Transactions on Graphics (TOG) 31, 4 (2012), 91.

Digital Library

[23]

Lauri Nummenmaa, Jukka Hyönä, and Manuel G Calvo. 2006. Eye movement assessment of selective attentional capture by emotional pictures.Emotion 6, 2 (2006), 257.

[24]

Marcus Nyström and Kenneth Holmqvist. 2010. An adaptive algorithm for fixation, saccade, and glissade detection in eyetracking data. Behavior research methods 42, 1 (2010), 188–204.

[25]

Arne Öhman. 2005. The role of the amygdala in human fear: automatic detection of threat. Psychoneuroendocrinology 30, 10 (2005), 953–958.

[26]

Derrick Parkhurst, Klinton Law, and Ernst Niebur. 2002. Modeling the role of salience in the allocation of overt visual attention. Vision research 42, 1 (2002), 107–123.

[27]

Harold Pashler, James C Johnston, and Eric Ruthruff. 2001. Attention and performance. Annual review of psychology 52, 1 (2001), 629–651.

[28]

Toni B Pence, Lauren C Dukes, Larry F Hodges, Nancy K Meehan, and Arlene Johnson. 2014. An eye tracking evaluation of a virtual pediatric patient training system for nurses. In International Conference on Intelligent Virtual Agents. Springer, 329–338.

[29]

Michael I Posner. 1982. Cumulative development of attentional theory.American Psychologist 37, 2 (1982), 168.

[30]

Helmut Prendinger, Chunling Ma, and Mitsuru Ishizuka. 2006. Eye movements as indices for the utility of life-like interface agents: A pilot study. Interacting with Computers 19, 2 (2006), 281–292.

Digital Library

[31]

Matthias Rehm and Elisabeth André. 2005. Where do they look? Gaze behaviors of multiple users interacting with an embodied conversational agent. In International Workshop on Intelligent Virtual Agents. Springer, 241–252.

Digital Library

[32]

Andrew Robb, Andrea Kleinsmith, Andrew Cordar, Casey White, Samsun Lampotang, Adam Wendling, and Lok Benjamin. 2016. Do Variations in Agency Indirectly Affect Behavior with Others? An Analysis of Gaze Behavior.IEEE Transactions on Visualizations and Computer Graphics 22, 4 (April 2016), 1336–1345. http://www.ncbi.nlm.nih.gov/ /23428451

Digital Library

[33]

David Rogosa. 1980. A critique of cross-lagged correlation.Psychological Bulletin 88, 2 (1980), 245.

[34]

Abraham Savitzky and Marcel JE Golay. 1964. Smoothing and differentiation of data by simplified least squares procedures.Analytical chemistry 36, 8 (1964), 1627–1639.

[35]

Harald T Schupp, Jessica Stockburger, Maurizio Codispoti, Markus Junghöfer, Almut I Weike, and Alfons O Hamm. 2007. Selective visual attention to emotion. Journal of neuroscience 27, 5 (2007), 1082–1089.

[36]

Rebecca M Todd, William A Cunningham, Adam K Anderson, and Evan Thompson. 2012. Affect-biased attention as emotion regulation. Trends in cognitive sciences 16, 7 (2012), 365–372.

[37]

Job Van Der Schalk, Agneta Fischer, Bertjan Doosje, Daniël Wigboldus, Skyler Hawk, Mark Rotteveel, and Ursula Hess. 2011. Convergent and divergent responses to emotional displays of ingroup and outgroup.Emotion 11, 2 (2011), 286.

[38]

Toni Vanhala, Veikko Surakka, Harri Siirtola, Kari-Jouko Räihä, Benoît Morel, and Laurent Ach. 2010. Virtual proximity and facial expressions of computer agents regulate human emotions and attention. Computer Animation and Virtual Worlds 21, 3-4 (2010), 215–224.

Digital Library

[39]

Matias Volante, Sabarish V Babu, Himanshu Chaturvedi, Nathan Newsome, Elham Ebrahimi, Tania Roy, Shaundra B Daily, and Tracy Fasolino. 2016. Effects of virtual human appearance fidelity on emotion contagion in affective inter-personal simulations. IEEE transactions on visualization and computer graphics 22, 4(2016), 1326–1335.

Digital Library

[40]

Heather A Wadlinger and Derek M Isaacowitz. 2011. Fixing our focus: Training attention to regulate emotion. Personality and Social Psychology Review 15, 1 (2011), 75–102.

[41]

Yanxiang Wu, Sabarish V Babu, Rowan Armstrong, Jeffrey W Bertrand, Jun Luo, Tania Roy, Shaundra B Daily, Lauren Cairco Dukes, Larry F Hodges, and Tracy Fasolino. 2014. Effects of virtual human animation on emotion contagion in simulated inter-personal experiences. IEEE transactions on visualization and computer graphics 20, 4(2014), 626–635.

Digital Library

[42]

Katja Zibrek and Rachel McDonnell. 2014. Does render style affect perception of personality in virtual humans?. In Proceedings of the ACM Symposium on Applied Perception. ACM, 111–115.

Digital Library

Cited By

Huang WWong SVolonte MBabu S(2024)Impact of Socio-Demographic Attributes and Mutual Gaze of Virtual Humans on Users’ Visual Attention and Collision Avoidance in VRIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332951530:9(6146-6163)Online publication date: Sep-2024
https://doi.org/10.1109/TVCG.2023.3329515
Higgins DZhan YCowan BMcDonnell R(2023)Investigating the effect of visual realism on empathic responses to emotionally expressive virtual humansACM Symposium on Applied Perception 202310.1145/3605495.3605799(1-7)Online publication date: 5-Aug-2023
https://dl.acm.org/doi/10.1145/3605495.3605799
Jhan XWong SEbrahimi ELai YHuang WBabu S(2022)Effects of Small Talk With a Crowd of Virtual Humans on Users' Emotional and Behavioral ResponsesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.320310728:11(3767-3777)Online publication date: Nov-2022
https://doi.org/10.1109/TVCG.2022.3203107
Show More Cited By

Empirical Evaluation of the Interplay of Emotion and Visual Attention in Human-Virtual Human Interaction
1. Human-centered computing

Recommendations

Human perception of a conversational virtual human: an empirical study on the effect of emotion and culture

Virtual reality applications with virtual humans, such as virtual reality exposure therapy, health coaches and negotiation simulators, are developed for different contexts and usually for users from different countries. The emphasis on a virtual human's ...
How is your user feeling? inferring emotion through human-computer interaction devices

Emotion can influence important user behaviors, including purchasing decisions, technology use, and customer loyalty. The ability to easily assess users' emotion during live system use therefore has practical significance for the design and improvement ...
Affective computing with primary and secondary emotions in a virtual human

We introduce the WASABI ([W]ASABI [A]ffect [S]imulation for [A]gents with [B]elievable [I]nteractivity) Affect Simulation Architecture, in which a virtual human's cognitive reasoning capabilities are combined with simulated embodiment to achieve the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SAP '19: ACM Symposium on Applied Perception 2019

September 2019

188 pages

ISBN:9781450368902

DOI:10.1145/3343036

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 September 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SAP '19

SAP '19: ACM Symposium on Applied Perception 2019

September 19 - 20, 2019

Barcelona, Spain

Acceptance Rates

Overall Acceptance Rate 43 of 94 submissions, 46%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
428
Total Downloads

Downloads (Last 12 months)35
Downloads (Last 6 weeks)4

Reflects downloads up to 02 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Huang WWong SVolonte MBabu S(2024)Impact of Socio-Demographic Attributes and Mutual Gaze of Virtual Humans on Users’ Visual Attention and Collision Avoidance in VRIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332951530:9(6146-6163)Online publication date: Sep-2024
https://doi.org/10.1109/TVCG.2023.3329515
Higgins DZhan YCowan BMcDonnell R(2023)Investigating the effect of visual realism on empathic responses to emotionally expressive virtual humansACM Symposium on Applied Perception 202310.1145/3605495.3605799(1-7)Online publication date: 5-Aug-2023
https://dl.acm.org/doi/10.1145/3605495.3605799
Jhan XWong SEbrahimi ELai YHuang WBabu S(2022)Effects of Small Talk With a Crowd of Virtual Humans on Users' Emotional and Behavioral ResponsesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.320310728:11(3767-3777)Online publication date: Nov-2022
https://doi.org/10.1109/TVCG.2022.3203107
Nam HKim CKim KYeo JPark J(2022)An Emotionally Responsive Virtual Parent for Pediatric Nursing Education: A Framework for Multimodal Momentary and Accumulated Interventions2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR55827.2022.00052(365-374)Online publication date: Oct-2022
https://doi.org/10.1109/ISMAR55827.2022.00052
Volonte MWang CEbrahimi EHsu YLiu KWong SBabu S(2021)Effects of Language Familiarity in Simulated Natural Dialogue with a Virtual Crowd of Digital Humans on Emotion Contagion in Virtual Reality2021 IEEE Virtual Reality and 3D User Interfaces (VR)10.1109/VR50410.2021.00040(188-197)Online publication date: Mar-2021
https://doi.org/10.1109/VR50410.2021.00040
Agha ZGhaiumy Anaraky RBadillo-Urquiola KMcHugh BWisniewski P(2021)‘Just-in-Time’ Parenting: A Two-Month Examination of the Bi-directional Influences Between Parental Mediation and Adolescent Online Risk ExposureHCI for Cybersecurity, Privacy and Trust10.1007/978-3-030-77392-2_17(261-280)Online publication date: 3-Jul-2021
https://doi.org/10.1007/978-3-030-77392-2_17
Volonte MAnaraky RVenkatakrishnan RVenkatakrishnan RKnijnenburg BDuchowski ABabu S(2020)Empirical evaluation and pathway modeling of visual attention to virtual humans in an appearance fidelity continuumJournal on Multimodal User Interfaces10.1007/s12193-020-00341-z15:2(109-119)Online publication date: 6-Aug-2020
https://doi.org/10.1007/s12193-020-00341-z

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten