research-article

The Impact of Sharing Gaze Behaviours in Collaborative Mixed Reality

Authors:

Brandon Matthews,

Mark BillinghurstAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 6, Issue CSCW2

Article No.: 463, Pages 1 - 27

https://doi.org/10.1145/3555564

Published: 11 November 2022 Publication History

Abstract

In a remote collaboration involving a physical task, visualising gaze behaviours may compensate for other unavailable communication channels. In this paper, we report on a 360° panoramic Mixed Reality (MR) remote collaboration system that shares gaze behaviour visualisations between a local user in Augmented Reality and a remote collaborator in Virtual Reality. We conducted two user studies to evaluate the design of MR gaze interfaces and the effect of gaze behaviour (on/off) and gaze style (bi-/uni-directional). The results indicate that gaze visualisations amplify meaningful joint attention and improve co-presence compared to a no gaze condition. Gaze behaviour visualisations enable communication to be less verbally complex therefore lowering collaborators' cognitive load while improving mutual understanding. Users felt that bi-directional behaviour visualisation, showing both collaborator's gaze state, was the preferred condition since it enabled easy identification of shared interests and task progress.

References

[1]

Huidong Bai, Prasanth Sasikumar, Jing Yang, and Mark Billinghurst. 2020. A User Study on Mixed Reality Remote Collaboration with Eye Gaze and Hand Gesture Sharing. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3313831.3376550

Digital Library

[2]

Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. Journal of Usability Studies 4 (2009), 114--123. https://doi.org/10.5555/2835587.2835589

Digital Library

[3]

Simon Baron-Cohen, Sally Wheelwright, and Therese Jolliffe. 1997. Is There a "Language of the Eyes"? Evidence from Normal Adults, and Adults with Autism or Asperger Syndrome. Psychology Press Ltd. VISUAL COGNITION 4, 3 (1997), 311--331.

[4]

M Bauer, G Kortuem, and Z Segall. 1999. "Where are you pointing at?" A study of remote collaboration in a wearable videoconference system. In Digest of Papers. Third International Symposium on Wearable Computers. 151--158. https: //doi.org/10.1109/ISWC.1999.806696

[5]

John Brooke. [n.d.]. SUS - A quick and dirty usability scale. Technical Report.

[6]

Simona Buetti, Zoe Jing Xu, and Alejandro Lleras. 2021. Predicting how surface texture and shape combine in the human visual system to direct attention. Scientific Reports 11, 1 (2021), 1--11. https://doi.org/10.1038/s41598-021--85605--8

[7]

Roser Cañigueral and Antonia F.de C Hamilton. 2019. The role of eye gaze during natural social interactions in typical and autistic people. Frontiers in Psychology 10, MAR (2019), 1--18. https://doi.org/10.3389/fpsyg.2019.00560

[8]

Sarah D'Angelo and Andrew Begel. 2017. Improving communication between pair programmers using shared gaze awareness. In Conference on Human Factors in Computing Systems - Proceedings, Vol. 2017-Janua. 6245--6255. https://doi.org/10.1145/3025453.3025573

Digital Library

[9]

Sarah D'Angelo and Darren Gergle. 2016. Gazed and Confused: Understanding and Designing Shared Gaze for Remote Collaboration. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). Association for Computing Machinery, New York, NY, USA, 2492--2496. https://doi.org/10.1145/2858036.2858499

Digital Library

[10]

Sarah D'Angelo and Darren Gergle. 2018. An eye for design: Gaze visualizations for remote collaborative work. Conference on Human Factors in Computing Systems - Proceedings 2018-April (2018), 1--12. https://doi.org/10.1145/ 3173574.3173923

Digital Library

[11]

Sarah D'Angelo and Bertrand Schneider. 2021. Shared Gaze Visualizations in Collaborative Interactions: Past, Present and Future. Interacting with Computers 00, 0 (2021). https://doi.org/10.1093/iwcomp/iwab015

[12]

Jay S. Efran. 1968. Looking for approval: Effects on visual behavior of approbation from persons differing in importance. Journal of Personality and Social Psychology 10, 1 (9 1968), 21--25. https://doi.org/10.1037/h0026383

[13]

Austin Erickson, Nahal Norouzi, Kangsoo Kim, Ryan Schubert, Jonathan Jules, Joseph J. LaViola, Gerd Bruder, and Gregory F. Welch. 2020. Sharing gaze rays for visual target identification tasks in collaborative augmented reality. Journal on Multimodal User Interfaces 14, 4 (2020), 353--371. https://doi.org/10.1007/s12193-020-00330--2

[14]

Susan R Fussell, Leslie D Setlock, and Robert E Kraut. 2003. Effects of Head-Mounted and Scene-Oriented Video Systems on Remote Collaboration on Physical Tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '03). Association for Computing Machinery, New York, NY, USA, 513--520. https://doi.org/10. 1145/642611.642701

Digital Library

[15]

Kunal Gupta, Gun A. Lee, and Mark Billinghurst. 2016. Do you see what i see? the effect of gaze tracking on task space remote collaboration. IEEE Transactions on Visualization and Computer Graphics 22, 11 (2016), 2413--2422. https://doi.org/10.1109/TVCG.2016.2593778

Digital Library

[16]

Chad Harms and Frank Biocca. 2004. Internal consistency and reliability of the networked minds measure of social presence. (2004).

[17]

Keita Higuch, Ryo Yonetani, and Yoichi Sato. 2016. Can Eye Help You? Effects of Visualizing Eye Fixations on Remote Collaboration Scenarios for Physical Tasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). Association for Computing Machinery, New York, NY, USA, 5180--5190. https: //doi.org/10.1145/2858036.2858438

Digital Library

[18]

Allison Jing, Gun Lee, and Mark Billinghurst. 2022. Using Speech to Visualise Shared Gaze Cues in MR Remote Collaboration. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 250--259. https://doi.org/ 10.1109/VR51125.2022.00044

[19]

Allison Jing, Kieran May, Gun Lee, and Mark Billinghurst. 2021. Eye See What You See: Exploring How Bi-Directional Augmented Reality Gaze Visualisation Influences Co-Located Symmetric Collaboration. Frontiers in Virtual Reality 2, June (2021), 79. https://doi.org/10.3389/frvir.2021.697367

[20]

Allison Jing, Kieran May, Mahnoor Naeem, Gun Lee, and Mark Billinghurst. 2021. EyemR-Vis: A Mixed Reality System to Visualise Bi-Directional Gaze Behavioural Cues Between Remote Collaborators. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (CHI EA '21). Association for Computing Machinery, New York, NY, USA, 19--22. https://doi.org/10.1145/3411763.3451545

Digital Library

[21]

Allison Jing, Kieran May, Mahnoor Naeem, Gun Lee, and Mark Billinghurst. 2021. EyemR-Vis: Using Bi-Directional Gaze Behavioural Cues to Improve Mixed Reality Remote Collaboration. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (CHI EA '21). Association for Computing Machinery, New York, NY, USA, 1--7. https://doi.org/10.1145/3411763.3451844

Digital Library

[22]

Shunichi Kasahara, Shohei Nagai, and Jun Rekimoto. 2017. JackIn Head: Immersive Visual Telepresence System with Omnidirectional Wearable Camera. IEEE Transactions on Visualization and Computer Graphics 23, 3 (2017), 1222--1234. https://doi.org/10.1109/TVCG.2016.2642947

Digital Library

[23]

Seungwon Kim, Allison Jing, Hanhoon Park, Soo-hyung Kim, Gun Lee, and Mark Billinghurst. 2020. Use of Gaze and Hand Pointers in Mixed Reality Remote Collaboration. In The 9th International Conference on Smart Media and Applications. SMA, Jeju, Republic of Korea, 1--6.

[24]

Seungwon Kim, Gun A. Lee, Nobuchika Sakata, Elina Vartiainen, and Mark Billinghurst. 2013. Comparing pointing and drawing for remote collaboration. 2013 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2013 (2013). https://doi.org/10.1109/ISMAR.2013.6671833

[25]

Gustav Kuhn, Benjamin W. Tatler, and Geoff G. Cole. 2009. You look where I look! Effect of gaze cues on overt and covert attention in misdirection. Visual Cognition 17, 6--7 (8 2009), 925--944. https://doi.org/10.1080/13506280902826775

[26]

Grete Helena Kütt and Ethan Hardacre. 2019. Eye-Write : Gaze Sharing for Collaborative Writing. (2019), 1--12.

[27]

Grete Helena Kütt, Teerapaun Tanprasert, Jay Rodolitz, Bernardo Moyza, Samuel So, Georgia Kenderova, and Alexandra Papoutsaki. 2020. Effects of Shared Gaze on Audio-Versus Text-Based Remote Collaborations. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020). https://doi.org/10.1145/3415207

Digital Library

[28]

Hideaki Kuzuoka, Toshio Kosuge, and Masatomo Tanaka. 1994. GestureCam: A Video Communication System for Sympathetic Remote Collaboration. In Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work (CSCW '94). Association for Computing Machinery, New York, NY, USA, 35--43. https://doi.org/10.1145/192844.192866

Digital Library

[29]

Gun Lee, Seungwon Kim, Youngho Lee, Arindam Dey, Thammatip Piumsomboon, Mitchell Norman, and Mark Billinghurst. 2017. Improving Collaboration in Augmented Video Conference using Mutually Shared Gaze. In ICATEGVE 2017 - International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments. The Eurographics Association, Adelaide, Australia, 197--204. https://doi.org/10.2312/egve.20171359

[30]

Gun A. Lee, Theophilus Teo, Seungwon Kim, and Mark Billinghurst. 2019. A User Study on MR Remote Collaboration Using Live 360 Video. Proceedings of the 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2018 (2019), 153--164. https://doi.org/10.1109/ISMAR.2018.00051

[31]

Gun A. Lee, Theophilus Teo, Seungwon Kim, and Mark Billinghurst. 2019. A user study on mr remote collaboration using live 360 video. In 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 153--164. https://doi.org/10.1109/ISMAR.2018.00051

[32]

Yuan Li, Feiyu Lu, Wallace S Lages, and Doug Bowman. 2019. Gaze Direction Visualization Techniques for Collaborative Wide-Area Model-Free Augmented Reality. In Symposium on Spatial User Interaction (SUI '19). Association for Computing Machinery, New York, NY, USA, 1--11. https://doi.org/10.1145/3357251.3357583

Digital Library

[33]

Fred G W C Paas. 1992. Training Strategies for Attaining Transfer of Problem-Solving Skill in Statistics: A Cognitive- Load Approach. Journal of Educational Psychology 84, 4 (1992), 429--434. https://doi.org/10.1037/0022-0663.84.4.429

[34]

Mohsen Parisay, Charalambos Poullis, and Marta Kersten-Oertel. 2020. FELiX: Fixation-based eye fatigue load index a multi-factor measure for gaze-based interactions. International Conference on Human System Interaction, HSI 2020-June, June (2020), 74--81. https://doi.org/10.1109/HSI49210.2020.9142677

[35]

Thammathip Piumsomboon, Arindam Dey, Barrett Ens, Gun Lee, and Mark Billinghurst. 2017. CoVAR: Mixed-Platform Remote Collaborative Augmented and Virtual Realities System with Shared Collaboration Cues. In Adjunct Proceedings of the 2017 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2017, Vol. NA. IEEE, Nantes, France, 218--219. https://doi.org/10.1109/ISMAR-Adjunct.2017.72

[36]

Thammathip Piumsomboon, Gun A Lee, Andrew Irlitti, Barrett Ens, Bruce H Thomas, and Mark Billinghurst. 2019. On the Shoulder of the Giant: A Multi-Scale Mixed Reality Collaboration with 360 Video Sharing and Tangible Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--17. https://doi.org/10.1145/3290605.3300458

Digital Library

[37]

Dario D. Salvucci and Joseph H Goldberg. 2000. Identifying Fixations and Saccades in Eye-Tracking Protocols. In Proceedings of the 2000 Symposium on Eye Tracking Research & Applications (ETRA '00, Vol. 10). Association for Computing Machinery, New York, NY, USA, 71--78. https://doi.org/10.1145/355017.355028

Digital Library

[38]

Theophilus Teo, Louise Lawrence, Gun A Lee, Mark Billinghurst, and Matt Adcock. 2019. Mixed Reality Remote Collaboration Combining 360 Video and 3D Reconstruction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--14. https: //doi.org/10.1145/3290605.3300431

Digital Library

[39]

Theophilus Teo, Mitchell Norman, Gun A. Lee, Mark Billinghurst, and Matt Adcock. 2020. Exploring interaction techniques for 360 panoramas inside a 3D reconstructed scene for mixed reality remote collaboration. Journal on Multimodal User Interfaces 14, 4 (2020), 373--385. https://doi.org/10.1007/s12193-020-00343-x

[40]

Yanxia Zhang, Jörg Müller, Ming Ki Chong, Andreas Bulling, and Hans Gellersen. 2014. GazeHorizon: Enabling passers-by to interact with public displays by gaze. UbiComp 2014 - Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (2014), 559--563. https://doi.org/10.1145/2632048.2636071

Digital Library

[41]

Yanxia Zhang, Ken Pfeuffer, Ming Ki Chong, Jason Alexander, Andreas Bulling, and Hans Gellersen. 2017. Look together: using gaze for assisting co-located collaborative search. Personal and Ubiquitous Computing 21, 1 (2017), 173--186. https://doi.org/10.1007/s00779-016-0969-x

Digital Library

[42]

Kaixing Zhao, Sandra Bardot, and Marcos Serrano. 2021. Tactile fixations: A behavioral marker on how people with visual impairments explore raised-line graphics. Conference on Human Factors in Computing Systems - Proceedings (2021). https://doi.org/10.1145/3411764.3445578

Digital Library

Cited By

SHINDO AFUKUSHIMA SHAUTASAARI ANAEMURA T(2024)CoVR+: Design of Visual Effects for Promoting Joint Attention During Shared VR Experiences via a Projection of HMD User's ViewIEICE Transactions on Information and Systems10.1587/transinf.2023HCP0011E107.D:3(374-382)Online publication date: 1-Mar-2024
https://doi.org/10.1587/transinf.2023HCP0011
Atweh JEl Iskandarani MRiggs S(2024)Real-Time Gaze Sharing Techniques and Their Influence on Performance and Shared Situational Awareness of Teammates in UAV C2 OperationsProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/10711813241260316Online publication date: 12-Aug-2024
https://doi.org/10.1177/10711813241260316
Atweh JRiggs S(2024)Gaze Sharing, a Double-Edged Sword: Examining the Effect of Real-Time Gaze Sharing Visualizations on Team Performance and Situational AwarenessHuman Factors: The Journal of the Human Factors and Ergonomics Society10.1177/00187208241272060Online publication date: 19-Aug-2024
https://doi.org/10.1177/00187208241272060
Show More Cited By

Index Terms

The Impact of Sharing Gaze Behaviours in Collaborative Mixed Reality
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Mixed / augmented reality
    2. Interactive systems and tools
  2. Visualization
    1. Visualization design and evaluation methods

Recommendations

eyemR-Vis: Using Bi-Directional Gaze Behavioural Cues to Improve Mixed Reality Remote Collaboration
CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems

Gaze is one of the most important communication cues in face-to-face collaboration. However, in remote collaboration, sharing dynamic gaze information is more difficult. In this research, we investigate how sharing gaze behavioural cues can improve ...
A User Study on Mixed Reality Remote Collaboration with Eye Gaze and Hand Gesture Sharing
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

Supporting natural communication cues is critical for people to work together remotely and face-to-face. In this paper we present a Mixed Reality (MR) remote collaboration system that enables a local worker to share a live 3D panorama of his/her ...
eyemR-Talk: Using Speech to Visualise Shared MR Gaze Cues
SA '21 Posters: SIGGRAPH Asia 2021 Posters

In this poster we present eyemR-Talk, a Mixed Reality (MR) collaboration system that uses speech input to trigger shared gaze visualisations between remote users. The system uses 360° panoramic video to support collaboration between a local user in the ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 6, Issue CSCW2

CSCW

November 2022

8205 pages

EISSN:2573-0142

DOI:10.1145/3571154

Editor:
Jeff Nichols
Google

Issue’s Table of Contents

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 November 2022

Published in PACMHCI Volume 6, Issue CSCW2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
744
Total Downloads

Downloads (Last 12 months)331
Downloads (Last 6 weeks)25

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

SHINDO AFUKUSHIMA SHAUTASAARI ANAEMURA T(2024)CoVR+: Design of Visual Effects for Promoting Joint Attention During Shared VR Experiences via a Projection of HMD User's ViewIEICE Transactions on Information and Systems10.1587/transinf.2023HCP0011E107.D:3(374-382)Online publication date: 1-Mar-2024
https://doi.org/10.1587/transinf.2023HCP0011
Atweh JEl Iskandarani MRiggs S(2024)Real-Time Gaze Sharing Techniques and Their Influence on Performance and Shared Situational Awareness of Teammates in UAV C2 OperationsProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/10711813241260316Online publication date: 12-Aug-2024
https://doi.org/10.1177/10711813241260316
Atweh JRiggs S(2024)Gaze Sharing, a Double-Edged Sword: Examining the Effect of Real-Time Gaze Sharing Visualizations on Team Performance and Situational AwarenessHuman Factors: The Journal of the Human Factors and Ergonomics Society10.1177/00187208241272060Online publication date: 19-Aug-2024
https://doi.org/10.1177/00187208241272060
Zhang KCao ZZheng XBillinghurst M(2024)Identifying Hand-based Input Preference Based on Wearable EEGProceedings of the Augmented Humans International Conference 202410.1145/3652920.3653028(102-118)Online publication date: 4-Apr-2024
https://dl.acm.org/doi/10.1145/3652920.3653028
Aoyagi SYamada TCheng KMasuko SZempo K(2024)VR remote tourism system with natural gaze induction without causing user discomfortProceedings of the Augmented Humans International Conference 202410.1145/3652920.3652938(181-192)Online publication date: 4-Apr-2024
https://dl.acm.org/doi/10.1145/3652920.3652938
Wang LLi XWu JZhou DSio Kei IPopescu V(2024)AVICol: Adaptive Visual Instruction for Remote Collaboration Using Mixed RealityInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2313920(1-20)Online publication date: 18-Feb-2024
https://doi.org/10.1080/10447318.2024.2313920
Chang ELee YBillinghurst MYoo B(2024)Efficient VR-AR communication method using virtual replicas in XR remote collaborationInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103304190(103304)Online publication date: Oct-2024
https://doi.org/10.1016/j.ijhcs.2024.103304
Atweh JHazimeh RRiggs S(2023)Can Real-Time Gaze Sharing Help Team Collaboration? A Preliminary Examination of its Effectiveness with PairsProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/2169506723119365967:1(716-721)Online publication date: 21-Oct-2023
https://doi.org/10.1177/21695067231193659
Lee SKim JLee J(2023)Effects of Reward Schedule and Avatar Visibility on Joint Agency During VR CollaborationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332022129:11(4372-4382)Online publication date: 3-Oct-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3320221
Jing AFrederick MSewell MKarlson ASimpson BSmith M(2023)How Visualising Emotions Affects Interpersonal Trust and Task Collaboration in a Shared Virtual Space2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR59233.2023.00100(849-858)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR59233.2023.00100
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents