Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3313831.3376606acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article
Open access

"How do I make this thing smile?": An Inventory of Expressive Nonverbal Communication in Commercial Social Virtual Reality Platforms

Published: 23 April 2020 Publication History

Abstract

Despite the proliferation of platforms for social Virtual Reality (VR) communicating emotional expression via an avatar remains a significant design challenge. In order to better understand the design space for expressive Nonverbal Communication (NVC) in social VR we undertook an inventory of the ten most prominent social VR platforms. Our inventory identifies the dominant design strategies for movement, facial control, and gesture in commercial VR applications, and identifies opportunities and challenges for future design and research into social expression in VR. Specifically, we highlight the paucity of interaction paradigms for facial expression and the near nonexistence of meaningful control over ambient aspects of nonverbal communication such as posture, pose, and social status.

References

[1]
Sang Chul Ahn, Ig-Jae Kim, Hyoung-Gon Kim, Yong-Moo Kwon, and Heedong Ko. 2001. Audience Interaction for Virtual Reality Theater and Its Implementation. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST '01), 41--45. https://doi.org/10.1145/505008.505016
[2]
Jeremy N. Bailenson, Jim Blascovich, Andrew C. Beall, and Jack M. Loomis. 2001. Equilibrium Theory Revisited: Mutual Gaze and Personal Space in Virtual Environments. Presence 10, 6: 583--598.
[3]
Jeremy N. Bailenson, Jim Blascovich, Andrew C. Beall, and Jack M. Loomis. 2003. Interpersonal Distance in Immersive Virtual Environments. Personality and Social Psychology Bulletin 29: 819--833.
[4]
Irmgard Bartenieff. 1980. Body Movement: Coping with the Enviornment. Routledge, New York.
[5]
Steve Benford, John Bowers, Lennart E. Fahl´en, Chris Greenhalgh, and Dave Snowdon. 1997. Embodiments, avatars, clones and agents for multi-user, multi-sensory virtual worlds. Multimedia Systems 5: 93--104.
[6]
Ray L. Birdwhistell. 1952. Introduction to Kinesics: an annotation system for analysis of body motion and gesture. Louisville University Press, Louisville, KY.
[7]
Jim Bizzocchi and Theresa Jean Tanenbaum. 2011. Well read: applying close reading techniques to gameplay experiences. In Drew Davidson (ed.). ETC Press, Pittsburgh, PA, USA, 262--290. Retrieved October 10, 2018 from http://dl.acm.org/citation.cfm?id=2031858.2031881
[8]
Staffan Bjork and Jussi Holopainen. 2005. Patterns in game design. Charles River Media Hingham.
[9]
Sarah Blackburn. 2017. Top Five VR Social Experiences. VRFocus. Retrieved April 3, 2019 from https://www.vrfocus.com/2017/08/top-five-vr-social-experiences/
[10]
Costas Boletsis. 2017. The New Era of Virtual Reality Locomotion: A Systematic Literature Review of Techniques and a Proposed Typology. Multimodal Technologies and Interaction 1, 4: 24. https://doi.org/10.3390/mti1040024
[11]
Dario Bombari, Marianne Schmid Mast, Elena Canadas, and Manuel Bachmann. 2015. Studying social interactions through immersive virtual environment technology: virtues, pitfalls, and future challenges. Frontiers in Psychology 6. https://doi.org/10.3389/fpsyg.2015.00869
[12]
Jerome Bruner. 1987. Life as Narrative. Social Research: An International Quarterly 54, 1: 11--32.
[13]
Jerome Bruner. 1991. The Narrative Construction of Reality. Critical Inquiry 18, 1: 1--21.
[14]
Tuncay Cakmak and Holger Hager. 2014. Cyberith virtualizer: a locomotion device for virtual reality. In ACM SIGGRAPH 2014 Emerging Technologies, 6.
[15]
J. Cassell and H. Vilhjálmsson. 1999. Fully Embodied Conversational Avatars: Making Communicative Behaviors Autonomous. Autonomous Agents and Multi-Agent Systems 2, 1: 45--64.
[16]
Justine Cassell. 1989. Embodied Conversation: Integrating Face and Gesture into Automatic Spoken Dialogue Systems. Retrieved October 24, 2009 from http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.34.7936
[17]
Bruce Damer. 1997. Avatars!; Exploring and Building Virtual Worlds on the Internet. Peachpit Press, Berkeley, CA, USA.
[18]
Rudolph P. Darken, William R. Cockayne, and David Carmein. 1997. The omni-directional treadmill: a locomotion device for virtual worlds.
[19]
M. Eddy. 2009. A brief history of somatic practices and dance: historical development of the field of somatic education and its relationship to dance. Journal of Dance and Somatic Practices 1, 1: 5--27.
[20]
Paul Ekman and Wallace V. Friesen. 1981. The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding. In Nonverbal Communication, Interaction, and Gesture, Thomas A. Sebeok, Jean Umiker-Sebeok and Adam Kendon (eds.). Mouton Publishers, The Hague.
[21]
Robert Giusti, Kenneth Hullett, and Jim Whitehead. 2012. Weapon Design Patterns in Shooter Games. In Proceedings of the First Workshop on Design Patterns in Games (DPG '12), 3:1--3:7. https://doi.org/10.1145/2427116.2427119
[22]
David Gochfeld, Corinne Brenner, Kris Layng, Sebastian Herscher, Connor DeFanti, Marta Olko, David Shinn, Stephanie Riggs, Clara Fernández-Vara, and Ken Perlin. 2018. Holojam in Wonderland: Immersive Mixed Reality Theater. Leonardo 51, 4: 362--367. https://doi.org/10.1162/leon_a_01644
[23]
Lindsay D. Grace. 2013. Affection Games in Digital Play: A Content Analysis of Web Playable Games. In DiGRA Conference.
[24]
Arline Groenewald. 2017. We Visit The Best Free PC Social VR Apps/Games Available Now. VR The Gamers. Retrieved April 3, 2019 from http://vrthegamers.com/best-free-social-vr-apps-games/
[25]
Anthony Guye-Vuillieme, Tolga K. Capin, Igor Sunday Pandzic, Nadia Magnenat, and Daniel Thalmann. 1999. Nonverbal communication interface for collaborative virtual environments. Virtual Reality Journal 4: 49--59.
[26]
Edward T. Hall. 1966. The Hidden Dimension. Anchor Books.
[27]
Linjia He, Hongsong Li, Tong Xue, Deyuan Sun, Shoulun Zhu, and Gangyi Ding. 2018. Am I in the Theater?: Usability Study of Live Performance Based Virtual Reality. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology (VRST '18), 28:1--28:11. https://doi.org/10.1145/3281505.3281508
[28]
H. Iwata. 1999. The Torus Treadmill: realizing locomotion in VEs. IEEE Computer Graphics and Applications 19, 6: 30--35. https://doi.org/10.1109/38.799737
[29]
Hiroo Iwata and Takashi Fujii. 1996. Virtual perambulator: a novel interface device for locomotion in virtual environment. In Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium, 60--65.
[30]
Hiroo Iwata, Hiroaki Yano, Hiroyuki Fukushima, and Haruo Noma. 2005. CirculaFloor [locomotion interface]. IEEE Computer Graphics and Applications 25, 1: 64--67.
[31]
Keith Johnstone. 1999. Impro for Storytellers. Routledge/Theatre Arts Books.
[32]
Stephen Jolly. 2000. Understanding Body Language: Birdwhistell's Theory of Kinesics. Corporate Communications: An International Journal 5, 3: 133--139.
[33]
Jen Karner. 2017. The Best VR social apps to share with friends. VRHeads. Retrieved April 3, 2019 from https://www.vrheads.com/best-vr-social-apps-share-friends
[34]
J. S. Kestenberg. 1979. The role of movement patterns in development. Dance Notation Bureau, Inc., New York.
[35]
SJ Kim. 2018. List Of The Most Popular Social VR Platforms. VR News, Games, And Reviews. Retrieved April 3, 2019 from https://www.vrandfun.com/popular-social-vr-platform-list/
[36]
Anya Kolesnichenko, Joshua McVeigh-Schultz, and Katherine Isbister. 2019. Understanding Emerging Design Practices for Avatar Systems in the Commercial Social VR Ecology. In Proceedings of the 2019 on Designing Interactive Systems Conference (DIS '19), 241--252. https://doi.org/10.1145/3322276.3322352
[37]
George Lakoff and Mark Johnson. 1980. Metaphors We Live By. University of Chicago Press, Chicago.
[38]
W. Lamb and E. Watson. 1987. Body code: The meaning in movement. Princeton Book Company.
[39]
Lara Maister, Mel Slater, Maria V. Sanchez-Vives, and Manos Tsakiris. 2015. Changing bodies changes minds: owning another body affects social cognition. Trends in Cognitive Sciences 19, 1: 6--12. https://doi.org/10.1016/j.tics.2014.11.001
[40]
Vera Maletic. 1987. Body - Space - Expression: The Development of Rudolf Laban's Movement and Dance Concepts. Walter de Gruyter & Co., Berlin.
[41]
David McNeill. 1992. Hand and mind: what gestures reveal about thought. University of Chicago Press.
[42]
David McNeill and Susan Duncan. 2000. Growth points in thinking-for-speaking. In Language and Gesture, David McNeill (ed.). Cambrige University Press, Cambridge, 162--185.
[43]
Joshua McVeigh-Schultz, Elena Márquez Segura, Nick Merrill, and Katherine Isbister. 2018. What's It Mean to "Be Social" in VR?: Mapping the Social VR Design Ecology. In Proceedings of the 2018 ACM Conference Companion Publication on Designing Interactive Systems (DIS '18 Companion), 289--294. https://doi.org/10.1145/3197391.3205451
[44]
Mark Metry. 2017. Top 7 Social VR Networks You Can Hang out in Now. VU Dream. Retrieved April 3, 2019 from http://www.vudream.com/top-7-social-vr-networks-you-can-hang-out-in-now/
[45]
Jacquelyn Ford Morie. 2014. Avatar appearance as prima facie non-verbal communication. In Nonverbal Communication in Virtual Worlds, Theresa Jean Tanenbaum, Magy Seif El-Nasr and Michael Nixon (eds.). ETC Press, Pittsburgh, PA, USA, 77--102. Retrieved April 8, 2019 from http://dl.acm.org/citation.cfm?id=2812748.2812755
[46]
Fares Moustafa and Anthony Steed. 2018. A Longitudinal Study of Small Group Interaction in Social Virtual Reality. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology (VRST '18), 22:1--22:10. https://doi.org/10.1145/3281505.3281527
[47]
Brian Myers. 2017. Friends With Benefits: Plausible Optimism and the Practice of Teabagging in Video Games. Games and Culture: 1555412017732855. https://doi.org/10.1177/1555412017732855
[48]
S. H. Oh and T. K. Whangbo. 2017. A study on the effective interaction method to improve the presence in social virtual reality game. In 2017 4th International Conference on Computer Applications and Information Processing Technology (CAIPT), 1--2. https://doi.org/10.1109/CAIPT.2017.8320715
[49]
Ken Perlin and Athomas Goldberg. 1996. Improv: A System for Scripting Interactive Actors in Virtual Worlds. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '96), 205--216. https://doi.org/10.1145/237170.237258
[50]
T. S. Perry. 2016. Virtual reality goes social. IEEE Spectrum 53, 1: 56--57. https://doi.org/10.1109/MSPEC.2016.7367470
[51]
Pierre Pita. 2017. List of Social VR Apps and Projects. Virtual Reality Times. Retrieved April 3, 2019 from https://virtualrealitytimes.com/2017/04/16/social-vr/
[52]
Fabio Pittarello and Eugenio Franchin. 2016. PlayVR: A VR Experience for the World of Theater. In Proceedings of the International Working Conference on Advanced Visual Interfaces (AVI '16), 338--339. https://doi.org/10.1145/2909132.2926063
[53]
Ralph Schroeder. 2002. Social Interaction in Virtual Environments: Key Issues, Common Themes, and a Framework for Research. In The Social Life of Avatars: Presence and Interaction in Shared Virtual Environments, Ralph Scroeder (ed.). Springer-Verlag, London, 1--18.
[54]
Author Ryan Schultz. 2018. A Complete List of Every Social VR Space and Virtual World Platform I Have Written About on The RyanSchultz.com Blog. Ryan Schultz. Retrieved April 3, 2019 from https://ryanschultz.com/2018/05/10/a-complete-list-of-every-social-vr-space-and-virtual-world-platform-i-have-written-about-on-the-ryanschultz-com-blog/
[55]
Author Ryan Schultz. 2018. UPDATED: A Comparison Chart of Twelve Popular Social VR Platforms. Ryan Schultz. Retrieved April 3, 2019 from https://ryanschultz.com/2018/10/22/a-comparison-chart-of-twelve-popular-social-vr-platforms/
[56]
Nicola S. Schutte and Emma J. Stilinovi?. 2017. Facilitating empathy through virtual reality. Motivation and Emotion 41, 6: 708--712. https://doi.org/10.1007/s11031-017--9641--7
[57]
Magy Seif El-Nasr and Huaxin Wei. 2008. Exploring Non-verbal Behavior Models for Believable Characters. In Interactive Storytelling (Lecture Notes in Computer Science), 71--82.
[58]
Koichi Shimizu, Gaku Sueta, Kentaro Yamaoka, Kazuki Sawamura, Yujin Suzuki, Keisuke Yoshida, Vibol Yem, Yasushi Ikei, Tomohiro Amemiya, Makoto Sato, Koichi Hirota, and Michiteru Kitazaki. 2018. FiveStar VR: Shareable Travel Experience Through Multisensory Stimulation to the Whole Body. In SIGGRAPH Asia 2018 Virtual & Augmented Reality (SA '18), 2:1--2:2. https://doi.org/10.1145/3275495.3275502
[59]
Husain Sumra. 2017. Social Reality: From Facebook Spaces to Altspace, VR social apps compared. Wareable. Retrieved April 3, 2019 from https://www.wareable.com/vr/best-virtual-reality-social-experiences-967
[60]
Theresa Jean Tanenbaum, Magy Seif El-Nasr, and Michael Nixon. 2014. Nonverbal Communication in Virtual Worlds: Understanding and Designing Expressive Characters. ETC Press. Retrieved from http://press.etc.cmu.edu/content/nonverbal-communication-virtual-worlds-understanding-and-designing-expressive-characters
[61]
B. Tarr, M. Slater, and E. Cohen. 2018. Synchrony and social connection in immersive Virtual Reality. Scientific Reports 8, 1: 3693. https://doi.org/10.1038/s41598-018--21765--4
[62]
T. L. Taylor. 2002. Living Digitally: Embodiment in Virtual Worlds. In The Social Life of Avatars: Presence and Interaction in Shared Virtual Environments, Ralph Scroeder (ed.). Springer-Verlag, London, 40--62.
[63]
Mickey Thibault. 2018. VR Social Networks will be all the rage in 2018. Mickey Thibault. Retrieved April 3, 2019 from https://medium.com/@tickarawr/vr-social-networks-will-be-all-the-rage-in-2018-fda5093c65
[64]
Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. 2016. FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality. arXiv:1610.03151 [cs]. Retrieved April 8, 2019 from http://arxiv.org/abs/1610.03151
[65]
Luiz Velho, Leonardo Carvalho, and Djalma Lucio. 2018. VR Kino+ Theater: from the ancient greeks into the future of media. In Proceedings of the Conference on Computer Animation, 9--10.
[66]
Jeffrey Ventrella. 2014. Virtual Gaze: the communicative energy between avatar faces. In Nonverbal Communication in Virtual Worlds, Theresa Jean Tanenbaum, Magy Seif El-Nasr and Michael Nixon (eds.). ETC Press, Pittsburgh, PA, USA, 61--75. Retrieved April 8, 2019 from http://dl.acm.org/citation.cfm?id=2812748.2812754
[67]
Jeffrey Ventrella. 2014. Avatar Puppeteering: direct manipulation of avatar joints for spontaneous body language. In Nonverbal Communication in Virtual Worlds, Theresa Jean Tanenbaum, Magy Seif El-Nasr and Michael Nixon (eds.). ETC Press, Pittsburgh, PA, USA, 237--249. Retrieved April 8, 2019 from http://dl.acm.org/citation.cfm?id=2812748.2812764
[68]
Alessandro Vinciarelli, Maja Pantic, Herve Bourlard, and Alex Pentland. 2008. Social Signals, their Function, and Automatic Analysis: A Survey. ACM.
[69]
Nick Yee, Jeremy N. Bailenson, Mark Urbanek, Francis Chang, and Dan Merget. 2007. The Unbearable Likeness of Being Digital: The Persistence of Nonverbal Social Norms in Online Virtual Environments. CyberPsychology & Behavior 10, 1: 115--121.
[70]
Y. Zhang and Z. Zhu. 2017. On Stage Interactive Spatial AR for Drama Performance. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct)(ISMARW), 280--283. https://doi.org/10.1109/ISMAR-Adjunct.2016.0095

Cited By

View all
  • (2024)Sensing the Intentions to Speak in VR Group DiscussionsSensors10.3390/s2402036224:2(362)Online publication date: 7-Jan-2024
  • (2024)Coordination of Speaking Opportunities in Virtual Reality: Analyzing Interaction Dynamics and Context-Aware StrategiesApplied Sciences10.3390/app14241207114:24(12071)Online publication date: 23-Dec-2024
  • (2024)The impact of first-person avatar customization on embodiment in immersive virtual realityFrontiers in Virtual Reality10.3389/frvir.2024.14367525Online publication date: 21-Aug-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
April 2020
10688 pages
ISBN:9781450367080
DOI:10.1145/3313831
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 April 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. nonverbal communication
  2. social interactions
  3. virtual reality

Qualifiers

  • Research-article

Conference

CHI '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)566
  • Downloads (Last 6 weeks)59
Reflects downloads up to 24 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Sensing the Intentions to Speak in VR Group DiscussionsSensors10.3390/s2402036224:2(362)Online publication date: 7-Jan-2024
  • (2024)Coordination of Speaking Opportunities in Virtual Reality: Analyzing Interaction Dynamics and Context-Aware StrategiesApplied Sciences10.3390/app14241207114:24(12071)Online publication date: 23-Dec-2024
  • (2024)The impact of first-person avatar customization on embodiment in immersive virtual realityFrontiers in Virtual Reality10.3389/frvir.2024.14367525Online publication date: 21-Aug-2024
  • (2024)How Do We Embody Our Narratives?Newhouse Impact Journal10.14305/jn.29960819.2024.2.1.42:1(17-22)Online publication date: 2024
  • (2024)There Is More to Avatars Than Visuals: Investigating Combinations of Visual and Auditory User Representations for Remote Collaboration in Augmented RealityProceedings of the ACM on Human-Computer Interaction10.1145/36981488:ISS(540-568)Online publication date: 24-Oct-2024
  • (2024)Press A or Wave: User Expectations for NPC Interactions and Nonverbal Behaviour in Virtual RealityProceedings of the ACM on Human-Computer Interaction10.1145/36770988:CHI PLAY(1-25)Online publication date: 15-Oct-2024
  • (2024)Accessible Nonverbal Cues to Support Conversations in VR for Blind and Low Vision PeopleProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675663(1-13)Online publication date: 27-Oct-2024
  • (2024)Disembodied, Asocial, and Unreal: How Users Reinterpret Designed Affordances of Social VRProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661548(1914-1925)Online publication date: 1-Jul-2024
  • (2024)Body Language for VUIs: Exploring Gestures to Enhance Interactions with Voice User InterfacesProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660691(133-150)Online publication date: 1-Jul-2024
  • (2024)Hands or Controllers? How Input Devices and Audio Impact Collaborative Virtual RealityProceedings of the 30th ACM Symposium on Virtual Reality Software and Technology10.1145/3641825.3687718(1-12)Online publication date: 9-Oct-2024
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media