research-article

‘Can you Set It Up On Your Own?’ – Investigating Users’ Ability To Participate in Remote-Based Virtual Reality Studies

Authors:

Pascal Knierim,

Florian AltAuthors Info & Claims

MUM '22: Proceedings of the 21st International Conference on Mobile and Ubiquitous Multimedia

Pages 121 - 127

https://doi.org/10.1145/3568444.3568462

Published: 29 December 2022 Publication History

Abstract

The availability of consumer-grade virtual reality (VR) devices allows user studies to be conducted remotely, that is in users’ homes. In this way, diverse populations can be reached and studies using virtual reality can be conducted in settings characterized by high ecologic validity. In this study (N=21) we investigate challenges participants face as they are required to set up and calibrate a virtual reality system in their home without assistance from experimenters. This allowed us to identify key reasons why participants struggle with this task. Our findings suggest that providing illustrative instructions and additional assistance on request can notably increase the success rate of setting up a VR environment for participating in a remote study. Interestingly, we also find that it is harder to recruit participants who do not have prior VR experience to participate in remote VR studies. We derive suggestions on how to support this task based on our findings.

References

[1]

Yasmeen Abdrabou, Radiah Rivu, Tarek Ammar, Jonathan Liebers, Alia Saad, Carina Liebers, Uwe Gruenefeld, Pascal Knierim, Mohamed Khamis, Ville Maekelae, Stefan Schneegass, and Florian Alt. 2022. Understanding Shoulder Surfer Behavior Using Virtual Reality. In Adjunct Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces. http://www.florian-alt.org/unibw/wp-content/publications/abdrabou2022ieeevr.pdf abdrabou2022ieeevr.

[2]

Neena L Chappell and Zachary Zimmer. 1999. Receptivity to new technology among older adults. Disability and rehabilitation 21, 5-6 (1999), 222–230.

[3]

Claiton Marques Correa, Gabriel Viegas Maciel de Freitas, André Luis dos Santos Eberhardt, and Milene Selbach Silveira. 2021. From now on: experiences from user-based research in remote settings. In Proceedings of the XX Brazilian Symposium on Human Factors in Computing Systems. 1–7.

Digital Library

[4]

Melinda Heinz, Peter Martin, Jennifer A Margrett, Mary Yearns, Warren Franke, Hen-I Yang, Johnny Wong, and Carl K Chang. 2013. Perceptions of technology among older adults. Journal of gerontological nursing 39, 1 (2013), 42–51.

[5]

Kai Holländer, Ashley Colley, Christian Mai, Jonna Häkkilä, Florian Alt, and Bastian Pfleging. 2019. Investigating the Influence of External Car Displays on Pedestrians’ Crossing Behavior in Virtual Reality. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (Taipei, Taiwan) (MobileHCI ’19). Association for Computing Machinery, New York, NY, USA, Article 27, 11 pages. https://doi.org/10.1145/3338286.3340138 hollaender2019mobilehci.

Digital Library

[6]

Felix Kosmalla, André Zenner, Corinna Tasch, Florian Daiber, and Antonio Krüger. 2020. The Importance of Virtual Hands and Feet for Virtual Reality Climbing. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3334480.3383067

Digital Library

[7]

Chiara Leonardi, Claudio Mennecozzi, Elena Not, Fabio Pianesi, and Massimo Zancanaro. 2008. Designing a familiar technology for elderly people. Gerontechnology 7, 2 (2008), 151.

[8]

Christian Mai, Tim Wiltzius, Florian Alt, and Heinrich Hußmann. 2018. Feeling Alone in Public: Investigating the Influence of Spatial Layout on Users’ VR Experience. In Proceedings of the 10th Nordic Conference on Human-Computer Interaction (Oslo, Norway) (NordiCHI ’18). Association for Computing Machinery, New York, NY, USA, 286–298. https://doi.org/10.1145/3240167.3240200 mai2018nordichi.

Digital Library

[9]

Aske Mottelson and Kasper Hornbæk. 2017. Virtual Reality Studies Outside the Laboratory. In Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology (Gothenburg, Sweden) (VRST ’17). Association for Computing Machinery, New York, NY, USA, Article 9, 10 pages. https://doi.org/10.1145/3139131.3139141

Digital Library

[10]

Aske Mottelson and Kasper Hornbæk. 2017. Virtual reality studies outside the laboratory. In Proceedings of the 23rd acm symposium on virtual reality software and technology. 1–10.

Digital Library

[11]

Aske Mottelson, Gustav Bøg Petersen, Klemen Lilija, and Guido Makransky. 2021. Conducting Unsupervised Virtual Reality User Studies Online. Frontiers in Virtual Reality 2 (2021), 66.

[12]

Tabitha C Peck, Laura E Sockol, and Sarah M Hancock. 2020. Mind the gap: The underrepresentation of female participants and authors in virtual reality research. IEEE transactions on visualization and computer graphics 26, 5(2020), 1945–1954.

[13]

Gustav Bøg Petersen, Aske Mottelson, and Guido Makransky. 2021. Pedagogical Agents in Educational VR: An in the Wild Study. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.3445760

Digital Library

[14]

Lara Reis, Emília Duarte, and Francisco Rebelo. 2013. Main Usability Issues in Using Virtual Environments for Older Population Warning Studies. In Design, User Experience, and Usability. User Experience in Novel Technological Environments, Aaron Marcus (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 189–198.

[15]

Radiah Rivu, Ville Mäkelä, Sarah Prange, Sarah Delgado Rodriguez, Robin Piening, Yumeng Zhou, Kay Köhle, Ken Pfeuffer, Yomna Abdelrahman, Matthias Hoppe, Albrecht Schmidt, and Florian Alt. 2021. Remote VR Studies - A Framework for Running Virtual Reality Studies Remotely Via Participant-Owned HMDs. CoRR abs/2102.11207(2021). arxiv:2102.11207https://arxiv.org/abs/2102.11207

[16]

Radiah Rivu, Yumeng Zhou, Robin Welsch, Ville Mäkelä, and Florian Alt. 2021. When Friends become Strangers: Understanding the Influence of Avatar Gender On Interpersonal Distance in Virtual Reality. In IFIP Conference on Human-Computer Interaction. Springer, 234–250.

Digital Library

[17]

David Saffo, Caglar Yildirim, Sara Di Bartolomeo, and Cody Dunne. 2020. Crowdsourcing Virtual Reality Experiments Using VRChat. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3334480.3382829

Digital Library

[18]

Albrecht Schmidt, Florian Alt, and Ville Mäkelä. 2021. Evaluation in Human-Computer Interaction – Beyond Lab Studies. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411763.3445022

Digital Library

[19]

Sanni Siltanen, Hanna Heinonen, Alisa Burova, Paulina Becerril Palma, Phong Truong, Viveka Opas, and Markku Turunen. 2021. There is Always a Way: Organizing VR User Tests with Remote and Hybrid Setups during a Pandemic—Learnings from Five Case Studies. Multimodal Technologies and Interaction 5, 10 (2021), 62.

[20]

Rômulo Silva, Artur Mol, and Lucila Ishitani. 2019. Virtual reality for older users: a systematic literature review. International Journal of Virtual Reality 19 (03 2019). https://doi.org/10.20870/IJVR.2019.19.1.2908

[21]

Marc B Sokol. 1994. Adaptation to difficult designs: Facilitating use of new technology. Journal of Business and Psychology 8, 3 (1994), 277–296.

[22]

Becky Spittle, Wenge Xu, Maite Frutos-Pascual, Chris Creed, and Ian Williams. 2021. Socially Distanced: Have user evaluation methods for Immersive Technologies changed during the COVID-19 pandemic?. In 2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). IEEE, 415–420.

[23]

Anthony Steed, Daniel Archer, Ben Congdon, Sebastian Friston, David Swapp, and Felix J Thiel. 2021. Some Lessons Learned Running Virtual Reality Experiments Out of the Laboratory. arXiv preprint arXiv:2104.05359(2021).

[24]

Anthony Steed, Sebastian Frlston, Maria Murcia Lopez, Jason Drummond, Ye Pan, and David Swapp. 2016. An ‘in the wild’experiment on presence and embodiment using consumer virtual reality equipment. IEEE transactions on visualization and computer graphics 22, 4(2016), 1406–1414.

Digital Library

[25]

Anthony Steed, Francisco R Ortega, Adam S Williams, Ernst Kruijff, Wolfgang Stuerzlinger, Anil Ufuk Batmaz, Andrea Stevenson Won, Evan Suma Rosenberg, Adalberto L Simeone, and Aleshia Hayes. 2020. Evaluating immersive experiences during Covid-19 and beyond. interactions 27, 4 (2020), 62–67.

[26]

Shabbir Syed-Abdul, Shwetambara Malwade, Aldilas Achmad Nursetyo, Mishika Sood, Madhu Bhatia, Diana Barsasella, Megan F Liu, Chia-Chi Chang, Kathiravan Srinivasan, M Raja, 2019. Virtual reality among the elderly: a usefulness and acceptance study from Taiwan. BMC geriatrics 19, 1 (2019), 1–10.

[27]

Jennifer Tichon and Robin Burgess-Limerick. 2011. A review of virtual reality as a medium for safety related training in mining. Journal of Health & Safety Research & Practice 3, 1 (2011), 33–40.

[28]

Clara Vandeweerdt, Tiffany Luong, Michael Atchapero, Aske Mottelson, Christian Holz, Guido Makransky, and Robert Böhm. 2021. Virtual reality reduces COVID-19 vaccine hesitancy in the wild. (2021).

Cited By

Warsinke MKojić TVergari MSpang RVoigt-Antons JMöller S(2024)Comparing Continuous and Retrospective Emotion Ratings in Remote VR Study2024 16th International Conference on Quality of Multimedia Experience (QoMEX)10.1109/QoMEX61742.2024.10598301(139-145)Online publication date: 18-Jun-2024
https://doi.org/10.1109/QoMEX61742.2024.10598301
Marques BSilva SMaio RAlves JFerreira CDias PSantos B(2023)Evaluating Outside the Box: Lessons Learned on eXtended Reality Multi-modal Experiments Beyond the LaboratoryProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614134(234-242)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614134

Index Terms

‘Can you Set It Up On Your Own?’ – Investigating Users’ Ability To Participate in Remote-Based Virtual Reality Studies
1. Human-centered computing
  1. Human computer interaction (HCI)
2. Security and privacy
  1. Security services
    1. Authentication

Recommendations

Remote VR Studies: A Framework for Running Virtual Reality Studies Remotely Via Participant-Owned HMDs
We investigate opportunities and challenges of running virtual reality (VR) studies remotely. Today, many consumers own head-mounted displays (HMDs), allowing them to participate in scientific studies from their homes using their own equipment. ...
The presence of an avatar can reduce cybersickness in Virtual Reality
Abstract
Virtual Reality (VR) applications are increasingly being utilized for research, healthcare, and education. Despite their benefits, many VR users report motion sickness-like sensations (cybersickness), such as headache, disorientation, or nausea. ...
Don’t make me sick: investigating the incidence of cybersickness in commercial virtual reality headsets
Abstract
The resurgence of interest in the use of virtual reality (VR) technology for research and entertainment purposes has led to an increase in concerns about human factor issues inherent in VR technology. One issue that has received a great deal of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

MUM '22: Proceedings of the 21st International Conference on Mobile and Ubiquitous Multimedia

November 2022

315 pages

ISBN:9781450398206

DOI:10.1145/3568444

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 December 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

MUM 2022

MUM 2022: 21th International Conference on Mobile and Ubiquitous Multimedia

November 27 - 30, 2022

Lisbon, Portugal

Acceptance Rates

Overall Acceptance Rate 190 of 465 submissions, 41%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
99
Total Downloads

Downloads (Last 12 months)30
Downloads (Last 6 weeks)7

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Warsinke MKojić TVergari MSpang RVoigt-Antons JMöller S(2024)Comparing Continuous and Retrospective Emotion Ratings in Remote VR Study2024 16th International Conference on Quality of Multimedia Experience (QoMEX)10.1109/QoMEX61742.2024.10598301(139-145)Online publication date: 18-Jun-2024
https://doi.org/10.1109/QoMEX61742.2024.10598301
Marques BSilva SMaio RAlves JFerreira CDias PSantos B(2023)Evaluating Outside the Box: Lessons Learned on eXtended Reality Multi-modal Experiments Beyond the LaboratoryProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614134(234-242)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614134

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten