short-paper

Injecting Nonverbal Mimicry with Hybrid Avatar-Agent Technologies: A Naïve Approach

Authors:

Christian Felix Purps,

Peter Kullmann,

Marc Erich LatoschikAuthors Info & Claims

SUI '18: Proceedings of the 2018 ACM Symposium on Spatial User Interaction

Pages 69 - 73

https://doi.org/10.1145/3267782.3267791

Published: 13 October 2018 Publication History

Abstract

Humans communicate to a large degree through nonverbal behavior. Nonverbal mimicry, i.e., the imitation of another's behavior can positively affect the social interactions. In virtual environments, user behavior can be replicated to avatars, and agent behaviors can be artificially constructed. By combining both, hybrid avatar-agent technologies aim at actively mediating virtual communication to foster interpersonal understanding and rapport. We present a naïve prototype, the "Mimicry Injector", that injects artificial mimicry in real-time virtual interactions. In an evaluation study, two participants were embodied in a Virtual Reality (VR) simulation, and had to perform a negotiation task. Their virtual characters either a) replicated only the original behavior or b) displayed the original behavior plus induced mimicry. We found that most participants did not detect the modification. However, the modification did not have a significant impact on the perception of the communication.

Supplementary Material

MP4 File (p69-roth.mp4)

Download
186.02 MB

References

[1]

Arthur Aron, Elaine N. Aron, and Danny Smollan. 1992. Inclusion of Other in the Self Scale and the structure of interpersonal closeness. Journal of personality and social psychology 63, 4 (1992), 596.

[2]

Jeremy N. Bailenson, Andrew C. Beall, Jack Loomis, Jim Blascovich, and Matthew Turk. 2004. Transformed Social Interaction: Decoupling Representation from Behavior and Form in Collaborative Virtual Environments. Presence: Teleoperators and Virtual Environments 13, 4 (Aug. 2004), 428--441.

Digital Library

[3]

Jeremy N Bailenson and Nick Yee. 2005. Digital chameleons: Automatic assimilation of nonverbal gestures in immersive virtual environments. Psychological science 16, 10 (2005), 814--819.

[4]

John A Bargh and Tanya L Chartrand. 1999. The unbearable automaticity of being. American psychologist 54, 7 (1999), 462.

[5]

Janet B Bavelas, Alex Black, Charles R Lemery, and Jennifer Mullett. 1986. "I show how you feel": Motor mimicry as a communicative act. Journal of personality and social psychology 50, 2 (1986), 322.

[6]

Gary Bente, Maria Senokozlieva, Sibylle Pennig, Ahmad Al-Issa, and Oliver Fischer. 2008. Deciphering the secret code: A new methodology for the cross-cultural analysis of nonverbal behavior. Behavior Research Methods 40, 1 (Feb. 2008), 269--277.

[7]

Elisabetta Bevacqua, Romain Richard, Julien Soler, and Pierre De Loor. 2016. INGREDIBLE: A platform for full body interaction between human and virtual agent that improves co-presence. In Proceedings of the 3rd International Symposium on Movement and Computing. ACM, 22.

Digital Library

[8]

Steven M Boker, Jeffrey F Cohn, Barry-John Theobald, Iain Matthews, Timothy R Brick, and Jeffrey R Spies. 2009. Effects of damping head movement and facial expression in dyadic conversation using real-time facial expression tracking and synthesized avatars. Philosophical Transactions of the Royal Society of London B: Biological Sciences 364, 1535 (2009), 3485--3495.

[9]

Tanya L Chartrand and Rick Van Baaren. 2009. Human mimicry. Advances in experimental social psychology 41 (2009), 219--274.

[10]

Antonia F. de C. Hamilton. 2008. Emulation and mimicry for social interaction: A theoretical approach to imitation in autism. The Quarterly Journal of Experimental Psychology 61, 1 (2008), 101--115.

[11]

Pierre De Loor, Romain Richard, Julien Soler, and Elisabetta Bevacqua. 2016. Aliveness metaphor for an evolutive gesture interaction based on coupling between a human and a virtual agent. In Proceedings of the 29th International Conference on Computer Animation and Social Agents. ACM, 147--155.

Digital Library

[12]

Alexandra Livia Georgescu, Bojana Kuzmanovic, Daniel Roth, Gary Bente, and Kai Vogeley. 2014. The use of virtual characters to assess and train nonverbal communication in high-functioning autism. Frontiers in human neuroscience 8 (2014), 807.

[13]

Jonathan Gratch, David DeVault, Gale M. Lucas, and Stacy Marsella. 2015. Negotiation as a challenge problem for virtual humans. In International Conference on Intelligent Virtual Agents. Springer, 201--215.

[14]

Jonathan Gratch, Anna Okhmatovskaia, Francois Lamothe, Stacy Marsella, Mathieu Morales, Rick J van der Werf, and Louis-Philippe Morency. 2006. Virtual rapport. In International Workshop on Intelligent Virtual Agents. Springer, 14--27.

Digital Library

[15]

Joanna Hale and F De C Antonia. 2016. Testing the relationship between mimicry, trust and rapport in virtual reality conversations. Scientific reports 6 (2016), 35295.

[16]

Lixing Huang, Louis-Philippe Morency, and Jonathan Gratch. 2011. Virtual Rapport 2.0. In International Workshop on Intelligent Virtual Agents. Springer, 68--79.

Digital Library

[17]

Robert S. Kennedy, Norman E. Lane, Kevin S. Berbaum, and Michael G. Lilienthal. 1993. Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness. The International Journal of Aviation Psychology 3, 3 (July 1993), 203--220.

[18]

Felix Kistler, Birgit Endrass, Ionut Damian, Chi Tai Dang, and Elisabeth André. 2012. Natural interaction with culturally adaptive virtual characters. Journal on Multimodal User Interfaces 6, 1--2 (2012), 39--47.

[19]

Jessica L Lakin, Valerie E Jefferis, Clara Michelle Cheng, and Tanya L Chartrand. 2003. The chameleon effect as social glue: Evidence for the evolutionary significance of nonconscious mimicry. Journal of nonverbal behavior 27, 3 (2003), 145--162.

[20]

David Matsumoto, Mark G. Frank, and Hyi Sung Hwang. 2012. Reading People. Nonverbal Communication: Science and Applications: Science and Applications (2012), 1.

[21]

Richard E Maurer and Jeffrey H Tindall. 1983. Effect of postural congruence on client's perception of counselor empathy. Journal of Counseling Psychology 30, 2 (1983), 158.

[22]

Daniel N McIntosh. 2006. Spontaneous facial mimicry, liking and emotional contagion. Polish Psychological Bulletin 37, 1 (2006), 31.

[23]

Kristine L. Nowak and Frank Biocca. 2003. The Effect of the Agency and Anthropomorphism on Users' Sense of Telepresence, Copresence, and Social Presence in Virtual Environments. Presence: Teleoperators and Virtual Environments 12, 5 (Oct. 2003), 481--494.

Digital Library

[24]

Soo Youn Oh, Jeremy Bailenson, Nicole Krämer, and Benjamin Li. 2016. Let the avatar brighten your smile: Effects of enhancing facial expressions in virtual environments. PloS one 11, 9 (2016), e0161794.

[25]

Soo Youn Oh, Jeremy Bailenson, Nicole Krämer, and Benjamin Li. 2016. Let the Avatar Brighten Your Smile: Effects of Enhancing Facial Expressions in Virtual Environments. PLOS ONE 11, 9 (Sept. 2016), e0161794.

[26]

Beatrice Rammstedt and Oliver P. John. 2007. Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German. Journal of research in Personality 41, 1 (2007), 203--212.

[27]

Daniel Roth, Constantin Kleinbeck, Tobias Feigl, Christopher Mutschler, and Marc Erich Latoschik. 2018. Beyond Replication: Augmenting Social Behaviors in Multi-User Virtual Realities. In Proceedings of the 25th IEEE Virtual Reality (VR) conference.

[28]

Daniel Roth, Peter Kullmann, Gary Bente, Dominik Gall, and Marc Erich Latoschik. 2018, in press. Effects of Hybrid and Synthetic Social Gaze in Avatar-Mediated Interactions. In 17th IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, Munich, Germany.

[29]

Daniel Roth, Marc Erich Latoschik, Kai Vogeley, and Gary Bente. 2015. Hybrid Avatar-Agent Technology -- A Conceptual Step Towards Mediated "Social" Virtual Reality and its Respective Challenges. icom 14, 2 (Jan. 2015).

[30]

Daniel Roth, Jean-Luc Lugrin, Dmitri Galakhov, Arvid Hofmann, Gary Bente, Marc Erich Latoschik, and Arnulph Fuhrmann. 2016. Avatar realism and social interaction quality in virtual reality. In Virtual Reality (VR), 2016 IEEE. IEEE, 277--278.

[31]

Daniel Roth, Kristoffer Waldow, Felix Stetter, Gary Bente, Marc Erich Latoschik, and Arnulph Fuhrmann. 2016. SIAM-C - A Socially Immersive Avatar Mediated Communication Platform. In Proceedings of the International Conference on Artificial Reality and Telexistence / Eurographics Symposium on Virtual Environments.

Digital Library

[32]

Edward Sapir. 1927. The unconscious patterning of behavior in society. Selected Writings of Edward Sapir in Language, Culture and Personality, ed. David E. Mandelbaum (Berkeley: U of California P, 1949) 556, 11 (1927).

[33]

Mariëlle Stel and Roos Vonk. 2010. Mimicry in social interaction: Benefits for mimickers, mimickees, and their interaction. British Journal of Psychology 101, 2 (2010), 311--323.

[34]

Johannes Wagner, Florian Lingenfelser, Tobias Baur, Ionut Damian, Felix Kistler, and Elisabeth André. 2013. The social signal interpretation (SSI) framework: multimodal signal processing and recognition in real-time. In Proceedings of the 21st ACM international conference on Multimedia. ACM, 831--834.

Digital Library

[35]

David Watson, Lee Anna Clark, and Auke Tellegen. 1988. Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology 54, 6 (1988), 1063--1070.

Cited By

Zhang RDuan WFlathmann CMcNeese NKnijnenburg BFreeman G(2024)Verbal vs. Visual: How Humans Perceive and Collaborate with AI Teammates Using Different Communication Modalities in Various Human-AI Team CompositionsProceedings of the ACM on Human-Computer Interaction10.1145/36869768:CSCW2(1-34)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686976
Le Tarnec HAugereau OBevacqua EDe Loor P(2024)Impact of Augmented Engagement Model for Collaborative Avatars on a Collaborative Task in Virtual RealityProceedings of the 2024 International Conference on Advanced Visual Interfaces10.1145/3656650.3656684(1-9)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3656650.3656684
Bhattacharya S(2024)Towards Asynchronous VR Collaboration with Virtual Agents as Placeholder UsersProceedings of the 24th ACM International Conference on Intelligent Virtual Agents10.1145/3652988.3673945(1-4)Online publication date: 16-Sep-2024
https://dl.acm.org/doi/10.1145/3652988.3673945
Show More Cited By

Index Terms

Injecting Nonverbal Mimicry with Hybrid Avatar-Agent Technologies: A Naïve Approach
1. Human-centered computing
  1. Collaborative and social computing
    1. Empirical studies in collaborative and social computing
  2. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Virtual reality

Recommendations

The effect of avatar realism in immersive social virtual realities
VRST '17: Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology

This paper investigates the effect of avatar realism on embodiment and social interactions in Virtual Reality (VR). We compared abstract avatar representations based on a wooden mannequin with high fidelity avatars generated from photogrammetry 3D scan ...
The Effects of an Embodied Conversational Agent's Nonverbal Behavior on User's Evaluation and Behavioral Mimicry
IVA '07: Proceedings of the 7th international conference on Intelligent Virtual Agents

Against the background that recent studies on embodied conversational agents demonstrate the importance of their behavior, an experimental study is presented that assessed the effects of different nonverbal behaviors of an embodied conversational agent ...
Multimodal embodied mimicry in interaction
COST'10: Proceedings of the 2010 international conference on Analysis of Verbal and Nonverbal Communication and Enactment

Nonverbal behavior plays an important role in human-human interaction. One particular kind of nonverbal behavior is mimicry. Behavioral mimicry supports harmonious relationships in social interaction through creating affiliation, rapport, and liking ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SUI '18: Proceedings of the 2018 ACM Symposium on Spatial User Interaction

October 2018

203 pages

ISBN:9781450357081

DOI:10.1145/3267782

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Conference

SUI '18

Sponsor:

SUI '18: Symposium on Spatial User Interaction

October 13 - 14, 2018

Berlin, Germany

Acceptance Rates

SUI '18 Paper Acceptance Rate 19 of 61 submissions, 31%;

Overall Acceptance Rate 86 of 279 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
446
Total Downloads

Downloads (Last 12 months)81
Downloads (Last 6 weeks)4

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang RDuan WFlathmann CMcNeese NKnijnenburg BFreeman G(2024)Verbal vs. Visual: How Humans Perceive and Collaborate with AI Teammates Using Different Communication Modalities in Various Human-AI Team CompositionsProceedings of the ACM on Human-Computer Interaction10.1145/36869768:CSCW2(1-34)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686976
Le Tarnec HAugereau OBevacqua EDe Loor P(2024)Impact of Augmented Engagement Model for Collaborative Avatars on a Collaborative Task in Virtual RealityProceedings of the 2024 International Conference on Advanced Visual Interfaces10.1145/3656650.3656684(1-9)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3656650.3656684
Bhattacharya S(2024)Towards Asynchronous VR Collaboration with Virtual Agents as Placeholder UsersProceedings of the 24th ACM International Conference on Intelligent Virtual Agents10.1145/3652988.3673945(1-4)Online publication date: 16-Sep-2024
https://dl.acm.org/doi/10.1145/3652988.3673945
Wang ICalvo RWang HRuiz JLugrin BLatoschik Mvon Mammen SKopp SPécune FPelachaud C(2023)Stop Copying MeProceedings of the 23rd ACM International Conference on Intelligent Virtual Agents10.1145/3570945.3607322(1-4)Online publication date: 19-Sep-2023
https://dl.acm.org/doi/10.1145/3570945.3607322
Schneeberger TReinwarth AWensky RAnglet MGebhard PWessler JLugrin BLatoschik Mvon Mammen SKopp SPécune FPelachaud C(2023)Fast FriendsProceedings of the 23rd ACM International Conference on Intelligent Virtual Agents10.1145/3570945.3607302(1-8)Online publication date: 19-Sep-2023
https://dl.acm.org/doi/10.1145/3570945.3607302
Kyrlitsias CMichael-Grigoriou D(2022)Social Interaction With Agents and Avatars in Immersive Virtual Environments: A SurveyFrontiers in Virtual Reality10.3389/frvir.2021.7866652Online publication date: 11-Jan-2022
https://doi.org/10.3389/frvir.2021.786665
Wei XJin XFan M(2022)Communication in Immersive Social Virtual Reality: A Systematic Review of 10 Years’ StudiesProceedings of the Tenth International Symposium of Chinese CHI10.1145/3565698.3565767(27-37)Online publication date: 22-Oct-2022
https://dl.acm.org/doi/10.1145/3565698.3565767
Nguyen HBruhn TSandor CBourdot P(2022)Absence Agents: Mitigating Interruptions in Extended Reality Remote Collaboration2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW55335.2022.00128(554-555)Online publication date: Mar-2022
https://doi.org/10.1109/VRW55335.2022.00128
Ratan RKlein MUcha CCherchiglia L(2022)Avatar customization orientation and undergraduate-course outcomesComputers & Education10.1016/j.compedu.2022.104643191:COnline publication date: 1-Dec-2022
https://dl.acm.org/doi/10.1016/j.compedu.2022.104643
Wu YWang YJung SHoermann SLindeman R(2021)Using a Fully Expressive Avatar to Collaborate in Virtual Reality: Evaluation of Task Performance, Presence, and AttractionFrontiers in Virtual Reality10.3389/frvir.2021.6412962Online publication date: 7-Apr-2021
https://doi.org/10.3389/frvir.2021.641296
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten