research-article

In AI We Trust: Investigating the Relationship between Biosignals, Trust and Cognitive Load in VR

Authors:

Andreas Duenser,

Martin Lochner,

Mark BillinghurstAuthors Info & Claims

VRST '19: Proceedings of the 25th ACM Symposium on Virtual Reality Software and Technology

Article No.: 33, Pages 1 - 10

https://doi.org/10.1145/3359996.3364276

Published: 12 November 2019 Publication History

Abstract

Human trust is a psycho-physiological state that is difficult to measure, yet is becoming increasingly important for the design of human-computer interactions. This paper explores if human trust can be measured using physiological measures when interacting with a computer interface, and how it correlates with cognitive load. In this work, we present a pilot study in Virtual Reality (VR) that uses a multi-sensory approach of Electroencephalography (EEG), galvanic skin response (GSR), and Heart Rate Variability (HRV) to measure trust with a virtual agent and explore the correlation between trust and cognitive load. The goal of this study is twofold; 1) to determine the relationship between biosignals, or physiological signals with trust and cognitive load, and 2) to introduce a pilot study in VR based on cognitive load level to evaluate trust. Even though we could not report any significant main effect or interaction of cognitive load and trust from the physiological signal, we found that in low cognitive load tasks, EEG alpha band power reflects trustworthiness on the agent. Moreover, cognitive load of the user decreases when the agent is accurate regardless of task’s cognitive load. This could be possible because of small sample size, tasks not stressful enough to induce high cognitive load due to lab study and comfortable environment or timestamp synchronisation error due to fusing data from various physiological sensors with different sample rate.

Supplementary Material

a33-gupta-supplement (inaiwetrust.m4v)

video

Download
50.78 MB

References

[1]

Mohamed Abouelenien, Mihai Burzo, Rada Mihalcea, Kristen Rusinek, and David Van Alstine. 2017. Detecting Human Thermal Discomfort via Physiological Signals. In Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments(PETRA ’17). ACM, New York, NY, USA, 146–149. https://doi.org/10.1145/3056540.3064957

Digital Library

[2]

Kumar Akash, Wan-Lin Hu, Neera Jain, and Tahira Reid. 2018. A classification model for sensing human trust in machines using eeg and gsr. ACM Transactions on Interactive Intelligent Systems (TiiS) 8, 4(2018), 27.

[3]

Kumar Akash, Wan-Lin Hu, Tahira Reid, and Neera Jain. 2017. Dynamic modeling of trust in human-machine interactions. In 2017 American Control Conference (ACC). IEEE, 1542–1548.

[4]

Pavlo Antonenko, Fred Paas, Roland Grabner, and Tamara Van Gog. 2010. Using electroencephalography to measure cognitive load. Educational Psychology Review 22, 4 (2010), 425–438.

[5]

Olivier Augereau, Benjamin Tag, and Koichi Kise. 2018. Mental State Analysis on Eyewear. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers(UbiComp ’18). ACM, New York, NY, USA, 968–973. https://doi.org/10.1145/3267305.3274119

Digital Library

[6]

Bernard Barber. 1983. The logic and limits of trust. Vol. 96. Rutgers University Press New Brunswick, NJ.

[7]

Mathias Benedek and Christian Kaernbach. 2010. A continuous measure of phasic electrodermal activity. Journal of neuroscience methods 190, 1 (2010), 80–91.

[8]

Alanah Davis, John D Murphy, Dawn Owens, Deepak Khazanchi, and Ilze Zigurs. 2009. Avatars, people, and virtual worlds: Foundations for research in metaverses. Journal of the Association for Information Systems 10, 2 (2009), 90.

[9]

Arindam Dey, Alex Chatburn, and Mark Billinghurst. 2019. Exploration of an EEG-Based Cognitively Adaptive Training System in Virtual Reality. (2019).

[10]

Suh-Yeon Dong, Bo-Kyeong Kim, Kyeongho Lee, and Soo-Young Lee. 2015. A Preliminary Study on Human Trust Measurements by EEG for Human-Machine Interactions. In Proceedings of the 3rd International Conference on Human-Agent Interaction(HAI ’15). ACM, New York, NY, USA, 265–268. https://doi.org/10.1145/2814940.2814993

Digital Library

[11]

Jérémy Frey, Maxime Daniel, Julien Castet, Martin Hachet, and Fabien Lotte. 2016. Framework for electroencephalography-based evaluation of user experience. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2283–2294.

Digital Library

[12]

Lynda Gerry, Barrett Ens, Adam Drogemuller, Bruce Thomas, and Mark Billinghurst. 2018. Levity: A Virtual Reality System That Responds to Cognitive Load. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems(CHI EA ’18). ACM, New York, NY, USA, Article LBW610, 6 pages. https://doi.org/10.1145/3170427.3188479

Digital Library

[13]

Google Glass [n.d.]. Google Glass - Glass Partners. https://www.google.com/glass/partners/

[14]

David Grimes, Desney S Tan, Scott E Hudson, Pradeep Shenoy, and Rajesh PN Rao. 2008. Feasibility and pragmatics of classifying working memory load with an electroencephalograph. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 835–844.

Digital Library

[15]

Jacek Gwizdka. 2010. Using Stroop task to assess cognitive load. In Proceedings of the 28th Annual European Conference on Cognitive Ergonomics. ACM, 219–222.

Digital Library

[16]

Joanna Hale and F De C Antonia. 2016. Testing the relationship between mimicry, trust and rapport in virtual reality conversations. Scientific reports 6(2016), 35295.

[17]

Peter A Hancock, Deborah R Billings, Kristin E Schaefer, Jessie YC Chen, Ewart J De Visser, and Raja Parasuraman. 2011. A meta-analysis of factors affecting trust in human-robot interaction. Human factors 53, 5 (2011), 517–527.

[18]

Sandra G Hart. 1986. NASA Task load Index (TLX). Volume 1.0; Paper and pencil package. (1986).

[19]

Blanca Hernandez-Ortega. 2011. The role of post-use trust in the acceptance of a technology: Drivers and consequences. Technovation 31, 10-11 (2011), 523–538.

[20]

Leanne M Hirshfield, Erin Treacy Solovey, Audrey Girouard, James Kebinger, Robert JK Jacob, Angelo Sassaroli, and Sergio Fantini. 2009. Brain measurement for usability testing and adaptive interfaces: an example of uncovering syntactic workload with functional near infrared spectroscopy. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2185–2194.

Digital Library

[21]

Kevin Anthony Hoff and Masooda Bashir. 2015. Trust in automation: Integrating empirical evidence on factors that influence trust. Human factors 57, 3 (2015), 407–434.

[22]

Wan-Lin Hu, Kumar Akash, Neera Jain, and Tahira Reid. 2016. Real-time sensing of trust in human-machine interactions. IFAC-PapersOnLine 49, 32 (2016), 48–53.

[23]

Sirkka L Jarvenpaa, Kathleen Knoll, and Dorothy E Leidner. 1998. Is anybody out there? Antecedents of trust in global virtual teams. Journal of management information systems 14, 4 (1998), 29–64.

Digital Library

[24]

Ole Jensen and Claudia D Tesche. 2002. Frontal theta activity in humans increases with memory load in a working memory task. European journal of Neuroscience 15, 8 (2002), 1395–1399.

[25]

Jiun-Yin Jian, Ann M Bisantz, and Colin G Drury. 2000. Foundations for an empirically determined scale of trust in automated systems. International Journal of Cognitive Ergonomics 4, 1 (2000), 53–71.

[26]

Ahmad Khawaji, Jianlong Zhou, Fang Chen, and Nadine Marcus. 2015. Using galvanic skin response (GSR) to measure trust and cognitive load in the text-chat environment. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 1989–1994.

Digital Library

[27]

Wayne K Kirchner. 1958. Age differences in short-term retention of rapidly changing information.Journal of experimental psychology 55, 4 (1958), 352.

[28]

Juyoung Lee, Hui-Shyong Yeo, Murtaza Dhuliawala, Jedidiah Akano, Junichi Shimizu, Thad Starner, Aaron Quigley, Woontack Woo, and Kai Kunze. 2017. Itchy Nose: Discreet Gesture Interaction Using EOG Sensors in Smart Eyewear. In Proceedings of the 2017 ACM International Symposium on Wearable Computers(ISWC ’17). ACM, New York, NY, USA, 94–97. https://doi.org/10.1145/3123021.3123060

Digital Library

[29]

Rachel McDonnell and Martin Breidt. 2010. Face reality: investigating the uncanny valley for virtual faces. In ACM SIGGRAPH ASIA 2010 Sketches. ACM, 41.

Digital Library

[30]

Daniel J. McDuff, Javier Hernandez, Sarah Gontarek, and Rosalind W. Picard. 2016. COGCAM: Contact-free Measurement of Cognitive Stress During Computer Tasks with a Digital Camera. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems(CHI ’16). ACM, New York, NY, USA, 4000–4004. https://doi.org/10.1145/2858036.2858247

Digital Library

[31]

Bonnie M Muir. 1987. Trust between humans and machines, and the design of decision aids. International journal of man-machine studies 27, 5-6 (1987), 527–539.

Digital Library

[32]

OpenBCI [n.d.]. OpenBCI - Open Source Brain Computer Interfaces. https://shop.openbci.com/collections/frontpage/products/openbci-eeg-electrocap-kit

[33]

Oxford Medical Simulation [n.d.]. Oxford Medical Simulation. https://oxfordmedicalsimulation.com/

[34]

Yun Suen Pai. 2016. Physiological Signal-Driven Virtual Reality in Social Spaces. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology(UIST ’16 Adjunct). ACM, New York, NY, USA, 25–28. https://doi.org/10.1145/2984751.2984787

Digital Library

[35]

J. Posada, C. Toro, I. Barandiaran, D. Oyarzun, D. Stricker, R. de Amicis, E. B. Pinto, P. Eisert, J. Döllner, and I. Vallarino. 2015. Visual Computing as a Key Enabling Technology for Industrie 4.0 and Industrial Internet. IEEE Computer Graphics and Applications 35, 2 (2015), 26–40.

Digital Library

[36]

Lingyun Qiu and Izak Benbasat. 2005. Online consumer trust and live help interfaces: The effects of text-to-speech voice and three-dimensional avatars. International journal of human-computer interaction 19, 1(2005), 75–94.

[37]

John K Rempel, John G Holmes, and Mark P Zanna. 1985. Trust in close relationships.Journal of personality and social psychology 49, 1(1985), 95.

[38]

David Rozado and Andreas Dunser. 2015. Combining EEG with pupillometry to improve cognitive workload detection. Computer 48, 10 (2015), 18–25.

Digital Library

[39]

Davide Salanitri, Glyn Lawson, and Brian Waterfield. 2016. The Relationship Between Presence and Trust in Virtual Reality. In Proceedings of the European Conference on Cognitive Ergonomics(ECCE ’16). ACM, New York, NY, USA, Article 16, 4 pages. https://doi.org/10.1145/2970930.2970947

Digital Library

[40]

Katarzyna Samson and Patrycjusz Kostyszyn. 2015. Effects of cognitive load on trusting behavior–an experiment using the trust game. PloS one 10, 5 (2015), e0127680.

[41]

Shimmer [n.d.]. Shimmer GSR+ Sensor. https://www.shimmersensing.com/products/shimmer3-wireless-gsr-sensor

[42]

Sharon Stansfield, Daniel Shawver, Annette Sobel, Monica Prasad, and Lydia Tapia. 2000. Design and Implementation of a Virtual Reality System and Its Application to Training Medical First Responders. Presence: Teleoperators and Virtual Environments 9, 6(2000), 524–556.

Digital Library

[43]

D Susan and John G Holmes. 1991. The dynamics of interpersonal trust: Resolving uncertainty in the face of risk. Cooperation and Prosocial Behavior; Cambridge University Press: New York, NY, USA (1991), 190.

[44]

Wioleta Szwoch. 2015. Emotion Recognition Using Physiological Signals. In Proceedings of the Mulitimedia, Interaction, Design and Innnovation(MIDI ’15). ACM, New York, NY, USA, Article 15, 8 pages. https://doi.org/10.1145/2814464.2814479

Digital Library

[45]

Mika P Tarvainen, Juha-Pekka Niskanen, Jukka A Lipponen, Perttu O Ranta-Aho, and Pasi A Karjalainen. 2014. Kubios HRV–heart rate variability analysis software. Computer methods and programs in biomedicine 113, 1(2014), 210–220.

[46]

Unity [n.d.]. Unity. https://unity3d.com/unity/whats-new/2019.2.4

[47]

Vive [n.d.]. Vive. https://www.vive.com/us/product/vive-virtual-reality-system/

[48]

Willem I. M. Willaert, Rajesh Aggarwal, Isabelle Van Herzeele, Nicholas J. Cheshire, and Frank E. Vermassen. 2012. Recent Advancements in Medical Simulation: Patient-Specific Virtual Reality Simulation. World Journal of Surgery 36, 7 (2012), 1703–1712.

[49]

Lian Zhang, Joshua Wade, Dayi Bian, Jing Fan, Amy Swanson, Amy Weitlauf, Zachary Warren, and Nilanjan Sarkar. 2017. Cognitive load measurement in a virtual reality-based driving system for autism intervention. IEEE transactions on affective computing 8, 2 (2017), 176–189.

Digital Library

Cited By

Ahmadi MMichalka SNajafabadi MWünsche BBillinghurst M(2024)EEG, Pupil Dilations, and Other Physiological Measures of Working Memory Load in the Sternberg TaskMultimodal Technologies and Interaction10.3390/mti80400348:4(34)Online publication date: 19-Apr-2024
https://doi.org/10.3390/mti8040034
Zhang KCao ZZheng XBillinghurst M(2024)Identifying Hand-based Input Preference Based on Wearable EEGProceedings of the Augmented Humans International Conference 202410.1145/3652920.3653028(102-118)Online publication date: 4-Apr-2024
https://dl.acm.org/doi/10.1145/3652920.3653028
Colley MSpeidel OStrohbeck JRixen JBelz JRukzio E(2024)Effects of Uncertain Trajectory Prediction Visualization in Highly Automated Vehicles on Trust, Situation Awareness, and Cognitive LoadProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314087:4(1-23)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631408
Show More Cited By

Recommendations

Using Galvanic Skin Response (GSR) to Measure Trust and Cognitive Load in the Text-Chat Environment
CHI EA '15: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems

Exchanging text messages via software on smart phones and computers has recently become one of the most popular ways for people to communicate and accomplish their tasks. However, there are negative aspects to using this kind of software, for example, ...
Trust and Cognitive Load in semi-automated UAV operation
OzCHI '19: Proceedings of the 31st Australian Conference on Human-Computer-Interaction

Trust in automation is an essential precursor to system adoption and use. Given the emerging wave of autonomous systems available for public consumption and the resources devoted to this trend, it's important to understand trust, and how to measure it. ...
Trust and cognitive load in the text-chat environment: the role of mouse movement
OzCHI '14: Proceedings of the 26th Australian Computer-Human Interaction Conference on Designing Futures: the Future of Design

This paper examines how different levels of cognitive load can affect trust in the text-chat environment. It also examines how the mouse movements of participants can indicate the level of cognitive load when they chat with each other. We designed two ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VRST '19: Proceedings of the 25th ACM Symposium on Virtual Reality Software and Technology

November 2019

498 pages

ISBN:9781450370011

DOI:10.1145/3359996

Editors:
Tomas Trescak
Western Sydney University
,
Simeon Simoff
Western Sydney University
,
Deborah Richards
Macquarie University
,
Anton Bogdanovych
Western Sydney University
,
Thierry Duval
IMT Atlantique
,
Torsten Kuhlen
RWTH Aachen University
,
Huyen Nguyen
University of New South Wales
,
Shigeo Morishima
University of Waseda
,
Yuichi Itoh
University of Osaka
,
Richard Skarbez
La Trobe
,
Anton Bogdanovych
Western Sydney University
,
Martin Masek
Edith Cowan University

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

VRST '19

Sponsor:

VRST '19: 25th ACM Symposium on Virtual Reality Software and Technology

November 12 - 15, 2019

NSW, Parramatta, Australia

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

37
Total Citations
View Citations
1,604
Total Downloads

Downloads (Last 12 months)288
Downloads (Last 6 weeks)30

Reflects downloads up to 04 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ahmadi MMichalka SNajafabadi MWünsche BBillinghurst M(2024)EEG, Pupil Dilations, and Other Physiological Measures of Working Memory Load in the Sternberg TaskMultimodal Technologies and Interaction10.3390/mti80400348:4(34)Online publication date: 19-Apr-2024
https://doi.org/10.3390/mti8040034
Zhang KCao ZZheng XBillinghurst M(2024)Identifying Hand-based Input Preference Based on Wearable EEGProceedings of the Augmented Humans International Conference 202410.1145/3652920.3653028(102-118)Online publication date: 4-Apr-2024
https://dl.acm.org/doi/10.1145/3652920.3653028
Colley MSpeidel OStrohbeck JRixen JBelz JRukzio E(2024)Effects of Uncertain Trajectory Prediction Visualization in Highly Automated Vehicles on Trust, Situation Awareness, and Cognitive LoadProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314087:4(1-23)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631408
Wang RLu A(2024)Exploring Trustful AI Augmentation with Virtual Evacuation Study2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00191(789-790)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00191
Kyriaki KKoukopoulos DFidas C(2024)A Comprehensive Survey of EEG Preprocessing Methods for Cognitive Load AssessmentIEEE Access10.1109/ACCESS.2024.336032812(23466-23489)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3360328
Li HLiang MNiu KZhang Y(2024)A Human-Machine Trust Evaluation Method for High-Speed Train Drivers Based on Multi-Modal Physiological InformationInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2327188(1-18)Online publication date: 3-Apr-2024
https://doi.org/10.1080/10447318.2024.2327188
Afroogh SAkbari AMalone EKargar MAlambeigi H(2024)Trust in AI: progress, challenges, and future directionsHumanities and Social Sciences Communications10.1057/s41599-024-04044-811:1Online publication date: 18-Nov-2024
https://doi.org/10.1057/s41599-024-04044-8
Leschanowsky ARech SPopp BBäckström T(2024)Evaluating privacy, security, and trust perceptions in conversational AI: A systematic reviewComputers in Human Behavior10.1016/j.chb.2024.108344159(108344)Online publication date: Oct-2024
https://doi.org/10.1016/j.chb.2024.108344
Chatterjee JSpruyt LPirson NVega M(2024)Effects of 6DoF Motion on Cybersickness in Interactive Virtual RealityExtended Reality10.1007/978-3-031-71713-0_2(21-37)Online publication date: 3-Sep-2024
https://doi.org/10.1007/978-3-031-71713-0_2
Yi BCao HSong XWang JGuo WHuang Z(2023)Measurement and Real-Time Recognition of Driver Trust in Conditionally Automated Vehicles: Using Multimodal Feature Fusions NetworkTransportation Research Record: Journal of the Transportation Research Board10.1177/036119812311565762677:8(311-330)Online publication date: 13-Mar-2023
https://doi.org/10.1177/03611981231156576
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents