research-article

Investigating the Effects of Virtual Patients’ Nonsensical Responses on Users’ Facial Expressions in Mental Health Training Scenarios

Authors:

Alexandre Gomes de Siqueira,

Sarah Bloch-Elkouby,

Jenelle Richards,

Lauren B. Lloveras,

Kathleen Feeney,

Stephanie Morris,

Erica D. Musser,

Benjamin Lok, and

Igor GalynkerAuthors Info & Claims

VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology

December 2021

Article No.: 16, Pages 1 - 10

https://doi.org/10.1145/3489849.3489864

Published: 08 December 2021 Publication History

Abstract

This report investigates how clinician-participants react to virtual patients’ sensical vs. nonsensical responses in a training simulation that aims to help clinicians acquire empathetic skills toward high-risk patients with symptoms of the Suicide Crisis Syndrome (SCS). Two suicidal virtual patients were developed, and clinician-participants interactions with them were recorded. Their facial emotions were analysed in three key moments: after a baseline sensical response, after a nonsensical response, and after the following sensical response. We compared their basic facial emotions aggregated into Negative and Positive facial affective behaviors (FABs). We describe our study involving ten clinician-participants and the results of the facial expression analysis with Noldus FaceReader. Our results suggest that nonsensical responses from virtual humans have an overall impact in both Positive and Negative facial affective emotions, and may lead to an increased percent of time participants demonstrate Negative facial affective behaviors when interacting with virtual humans. We discuss several aspects regarding the impacts and importance of considering nonsensical responses in the context of virtual human-based interactions.

References

[1]

2021. Facial expression recognition software | FaceReader. https://www.noldus.com. (Accessed on 06/12/2021).

[2]

Shira Barzilay, Krystel Assounga, Jacqueline Veras, Courtnie Beaubian, Sarah Bloch-Elkouby, and Igor Galynker. 2020. Assessment of near-term risk for suicide attempts using the suicide crisis inventory. Journal of affective disorders 276 (2020), 183–190.

[3]

Shira Barzilay, Zimri S Yaseen, Mariah Hawes, Bernard Gorman, Rachel Altman, Adriana Foster, Alan Apter, Paul Rosenfield, and Igor Galynker. 2018. Emotional responses to suicidal patients: Factor structure, construct, and predictive validity of the Therapist Response Questionnaire-Suicide Form. Frontiers in psychiatry 9 (2018), 104.

[4]

Timothy Bickmore, Amanda Gruber, and Rosalind Picard. 2005. Establishing the computer–patient working alliance in automated health behavior change interventions. Patient education and counseling 59, 1 (2005), 21–30.

[5]

Jim Blascovich. 2002. Social influence within immersive virtual environments. In The social life of avatars. Springer, 127–145.

Digital Library

[6]

Jim Blascovich. 2002. A theoretical model of social influence for increasing the utility of collaborative virtual environments. In Proceedings of the 4th international conference on Collaborative virtual environments. 25–30.

Digital Library

[7]

Jim Blascovich and Cade McCall. 2013. Social influence in virtual environments.(2013).

[8]

George A Bonanno and Dacher Keltner. 1997. Facial expressions of emotion and the course of conjugal bereavement.Journal of abnormal psychology 106, 1 (1997), 126.

[9]

Michael Borish, Andrew Cordar, Adriana Foster, Thomas Kim, James Murphy, and Benjamin Lok. 2014. Utilizing real-time human-assisted virtual humans to increase real-world interaction empathy. Kansei Engineering & Emotion Research (KEER’14) 15 (2014).

[10]

Carma L Bylund and Gregory Makoul. 2005. Examining empathy in medical encounters: an observational study using the empathic communication coding system. Health communication 18, 2 (2005), 123–140.

[11]

Alex S Cohen, Sean C Morrison, and Dallas A Callaway. 2013. Computerized facial analysis for understanding constricted/blunted affect: initial feasibility, reliability, and validity data. Schizophrenia research 148, 1-3 (2013), 111–116.

[12]

TF Cootes and CJ Taylor. 2004. Statistical models of appearance for computer vision, imaging science and biomedical engineering. University of Manchester 17 (2004), 2011.

[13]

David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, 2014. SimSensei Kiosk: A virtual human interviewer for healthcare decision support. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems. 1061–1068.

[14]

Wilfrid J Dixon and Alexander M Mood. 1946. The statistical sign test. J. Amer. Statist. Assoc. 41, 236 (1946), 557–566.

[15]

Paul Ekman and Daniel Cordaro. 2011. What is meant by calling emotions basic. Emotion review 3, 4 (2011), 364–370.

[16]

Paul Ekman, E Richard Sorenson, and Wallace V Friesen. 1969. Pan-cultural elements in facial displays of emotion. Science 164, 3875 (1969), 86–88.

[17]

Adriana Foster, Neelam Chaudhary, Thomas Kim, Jennifer L Waller, Joyce Wong, Michael Borish, Andrew Cordar, Benjamin Lok, and Peter F Buckley. 2016. Using virtual patients to teach empathy: a randomized controlled study to enhance medical students’ empathic communication. Simulation in Healthcare 11, 3 (2016), 181–189.

[18]

Arlene F Frank and John G Gunderson. 1990. The role of the therapeutic alliance in the treatment of schizophrenia: Relationship to course and outcome. Archives of general psychiatry 47, 3 (1990), 228–236.

[19]

Igor Galynker. 2017. The suicidal crisis: Clinical guide to the assessment of imminent suicide risk. Oxford University Press.

[20]

Randy Goebel, Ajay Chander, Katharina Holzinger, Freddy Lecue, Zeynep Akata, Simone Stumpf, Peter Kieseberg, and Andreas Holzinger. 2018. Explainable ai: the new 42?. In International cross-domain conference for machine learning and knowledge extraction. Springer, 295–303.

[21]

Jonathan Gratch, Ning Wang, Jillian Gerten, Edward Fast, and Robin Duffy. 2007. Creating rapport with virtual agents. In International workshop on intelligent virtual agents. Springer, 125–138.

Digital Library

[22]

Adam N Joinson. 2001. Self-disclosure in computer-mediated communication: The role of self-awareness and visual anonymity. European journal of social psychology 31, 2 (2001), 177–192.

[23]

Peter Khooshabeh, Morteza Dehghani, Angela Nazarian, and Jonathan Gratch. 2017. The cultural influence model: When accented natural language spoken by virtual characters matters. AI & society 32, 1 (2017), 9–16.

[24]

Sung Soo Kim, Stan Kaplowitz, and Mark V Johnston. 2004. The effects of physician empathy on patient satisfaction and compliance. Evaluation & the health professions 27, 3 (2004), 237–251.

[25]

Janice L Krieger, Jordan M Neil, Kyle A Duke, Mohan S Zalake, Fatemeh Tavassoli, Melissa J Vilaro, Danyell S Wilson-Howard, Sarah Y Chavez, Eric B Laber, Marie Davidian, 2021. A pilot study examining the efficacy of delivering colorectal cancer screening messages via virtual health assistants. American journal of preventive medicine(2021).

[26]

Peter Lewinski, Tim M den Uyl, and Crystal Butler. 2014. Automated facial coding: validation of basic emotions and FACS AUs in FaceReader.Journal of Neuroscience, Psychology, and Economics 7, 4 (2014), 227.

[27]

Gale M Lucas, Jill Boberg, David Traum, Ron Artstein, Jonathan Gratch, Alesia Gainer, Emmanuel Johnson, Anton Leuski, and Mikio Nakano. 2018. Culture, errors, and rapport-building dialogue in social agents. In Proceedings of the 18th International Conference on intelligent virtual agents. 51–58.

Digital Library

[28]

Gale M Lucas, Jonathan Gratch, Aisha King, and Louis-Philippe Morency. 2014. It’s only a computer: Virtual humans increase willingness to disclose. Computers in Human Behavior 37 (2014), 94–100.

Digital Library

[29]

Clifford Nass, Jonathan Steuer, and Ellen R Tauber. 1994. Computers are social actors. In Proceedings of the SIGCHI conference on Human factors in computing systems. 72–78.

Digital Library

[30]

Neelang Parghi, Lakshmi Chennapragada, Shira Barzilay, Saskia Newkirk, Brian Ahmedani, Benjamin Lok, and Igor Galynker. 2021. Assessing the predictive ability of the Suicide Crisis Inventory for near-term suicidal behavior using machine learning approaches. International journal of methods in psychiatric research 30, 1(2021), e1863.

[31]

John E Perez and Ronald E Riggio. 2003. Nonverbal Social Skills and Psychopathology.(2003).

[32]

Melissa D Pinto, Amy M Greenblatt, Ronald L Hickman, Heather M Rice, Tami L Thomas, and John M Clochesy. 2015. Assessing the Critical Parameters of eSMART-MH: A Promising Avatar-Based Digital Therapeutic Intervention to Reduce Depressive Symptoms.Perspectives in psychiatric care 52, 3 (2015), 157–168.

[33]

Mores Prachyabrued, Disathon Wattanadhirach, Richard B Dudrow, Nat Krairojananan, and Pusit Fuengfoo. 2019. Toward virtual stress inoculation training of prehospital healthcare personnel: A stress-inducing environment design and investigation of an emotional connection factor. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 671–679.

[34]

Albert Rizzo, Greg Reger, Karen Perlman, Barbara Rothbaum, JoAnn Difede, Rob McLay, Ken Graap, Greg Gahm, Scott Johnston, Rob Deal, 2011. Virtual reality posttraumatic stress disorder (PTSD) exposure therapy results with active duty OIF/OEF service members. (2011).

[35]

James Ross and Chris Watling. 2017. Use of empathy in psychiatric practice: constructivist grounded theory study. BJPsych open 3, 1 (2017), 26–33.

[36]

Brent Rossen and Benjamin Lok. 2012. A crowdsourcing method to develop virtual human conversational agents. International Journal of Human-Computer Studies 70, 4 (2012), 301–319.

Digital Library

[37]

Allison Schuck, Raffaella Calati, Shira Barzilay, Sarah Bloch-Elkouby, and Igor Galynker. 2019. Suicide Crisis Syndrome: A review of supporting evidence for a new suicide-specific diagnosis. Behavioral sciences & the law 37, 3 (2019), 223–239.

[38]

Richard Skarbez, Aaron Kotranza, Frederick P Brooks, Benjamin Lok, and Mary C Whitton. 2011. An initial exploration of conversational errors as a novel method for evaluating virtual human experiences. In 2011 IEEE Virtual Reality Conference. IEEE, 243–244.

Digital Library

[39]

Mel Slater and Anthony Steed. 2000. A virtual presence counter. Presence 9, 5 (2000), 413–434.

Digital Library

[40]

Matthew J Smith, Emily J Ginger, Michael Wright, Katherine Wright, Laura Boteler Humm, Dale Olsen, Morris D Bell, and Michael F Fleming. 2014. Virtual reality job interview training for individuals with psychiatric disabilities. The Journal of nervous and mental disease 202, 9 (2014), 659.

[41]

Jacob Stuart, Ileri Akinnola, Frank Guido-Sanz, Mindi Anderson, Desiree Diaz, Greg Welch, and Benjamin Lok. 2020. Applying stress management techniques in augmented reality: Stress induction and reduction in healthcare providers during virtual triage simulation. In 2020 IEEE conference on virtual reality and 3D user interfaces abstracts and workshops (VRW). IEEE, 171–172.

[42]

Yuqiong Wang, Peter Khooshabeh, and Jonathan Gratch. 2013. Looking real and making mistakes. In International Workshop on Intelligent Virtual Agents. Springer, 339–348.

[43]

Lidewij A Wind, Jan Van Dalen, Arno MM Muijtjens, and Jan-Joost Rethans. 2004. Assessing simulated patients in an educational setting: the MaSP (Maastricht Assessment of Simulated Patients). Medical Education 38, 1 (2004), 39–44.

[44]

Heng Yao, Alexandre Gomes de Siqueira, Adriana Foster, Igor Galynker, and Benjamin Lok. 2020. Toward Automated Evaluation of Empathetic Responses in Virtual Human Interaction Systems for Mental Health Scenarios. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents. 1–8.

Digital Library

[45]

Zimri S Yaseen, Mariah Hawes, Shira Barzilay, and Igor Galynker. 2019. Predictive validity of proposed diagnostic criteria for the suicide crisis syndrome: an acute presuicidal state. Suicide and Life-Threatening Behavior 49, 4 (2019), 1124–1135.

Cited By

Yao HGomes de Siqueira ARogers MBloch-Elkouby SLawrence OSarli GFoster AMitelman SGalynker ILok B(2024)The impact of scaffolded and non-scaffolded suicidal virtual human interaction training on clinician emotional self-awareness, empathic communication, and clinical efficacyBMC Medical Education10.1186/s12909-024-05371-924:1Online publication date: 15-Apr-2024
https://doi.org/10.1186/s12909-024-05371-9
Kessassi CChollet MDumas CCao C(2023)Social Presence Mediates Audience Behavior Effects on Social Stress in Virtual Public Speaking2023 11th International Conference on Affective Computing and Intelligent Interaction (ACII)10.1109/ACII59096.2023.10388126(1-8)Online publication date: 10-Sep-2023
https://doi.org/10.1109/ACII59096.2023.10388126

Recommendations

Toward Automated Evaluation of Empathetic Responses in Virtual Human Interaction Systems for Mental Health Scenarios
IVA '20: Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents

This paper investigates the process of automating the evaluation of empathetic response levels in virtual human interaction systems implementing mental health scenarios. Two suicidal virtual patients were developed to collect clinician participants' ...
Read More
Understanding empathy training with virtual patients

Early stage medical students are able to practice empathy with virtual patients.Responses rated by coders were more empathetic with virtual patients than standardized patients.The level of empathetic response positively correlates to response ...
Read More
Is virtual reality emotionally arousing? Investigating five emotion inducing virtual park scenarios

Following the idea of using virtual environments (VEs) as mood induction procedures (MIPs), this study set out to examine whether five different virtual park scenarios would each elicit a specific affective state (i.e., joy, sadness, boredom, anger and ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology

December 2021

563 pages

ISBN:9781450390927

DOI:10.1145/3489849

Editors:
Yuichi Itoh
Aoyama Gakuin University, Japan
,
Kazuki Takashima
Tohoku University, Japan
,
Parinya Punpongsanon
Osaka University, Japan
,
Misha Sra
University of California, Santa Barbara, USA
,
Kazuyuki Fujita
Tohoku University, Japan
,
Shigeo Yoshida
The University of Tokyo, Japan
,
Tham Piumsomboon
University of Canterbury, New Zealand

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 December 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

American Foundation for Suicide Prevention
National Institutes of Health

Conference

VRST '21

Sponsor:

VRST '21: 27th ACM Symposium on Virtual Reality Software and Technology

December 8 - 10, 2021

Osaka, Japan

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Upcoming Conference

VRST '24

Sponsor:
sigchi
sigchi

30th ACM Symposium on Virtual Reality Software and Technology

October 9 - 11, 2024

Trier , Germany

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
185
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)2

Other Metrics

View Author Metrics

Citations

Cited By

Yao HGomes de Siqueira ARogers MBloch-Elkouby SLawrence OSarli GFoster AMitelman SGalynker ILok B(2024)The impact of scaffolded and non-scaffolded suicidal virtual human interaction training on clinician emotional self-awareness, empathic communication, and clinical efficacyBMC Medical Education10.1186/s12909-024-05371-924:1Online publication date: 15-Apr-2024
https://doi.org/10.1186/s12909-024-05371-9
Kessassi CChollet MDumas CCao C(2023)Social Presence Mediates Audience Behavior Effects on Social Stress in Virtual Public Speaking2023 11th International Conference on Affective Computing and Intelligent Interaction (ACII)10.1109/ACII59096.2023.10388126(1-8)Online publication date: 10-Sep-2023
https://doi.org/10.1109/ACII59096.2023.10388126

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents