"Never The Same": Systematic Analysis of the Methodological Issues in the Presence Studies That Employ Questionnaires
Authors: 
Eugene Kukshinov, 
Joseph Tu, 
Kata Szita, Kaushall Senthil Nathan, 
Lennart E. NackeAuthors Info & Claims
Eugene Kukshinov, Joseph Tu, Kata Szita, Kaushall Senthil Nathan, Lennart E. NackeAuthors Info & ClaimsArticle No.: 8, Pages 1 - 7
Abstract
Presence is a psychological state that is usually measured via questionnaires. Many presence researchers assume self-report questionnaires are standardized. However, we do not know how reliable they are. This knowledge gap impacts the accuracy and validity of data collected through these questionnaires. Reliable and accurate data collection is crucial to trust findings in presence research. Inaccurate or unreliable data could lead to incorrect conclusions. This impacts theoretical understanding and practical applications. To address this, we conducted a comprehensive systematic review of 100 empirical quantitative presence studies. Our goal was to uncover the underlying issues with these self-report questionnaires. So, we explored the employment of these questionnaires and analyzed the specific reasons for measuring presence. We show patterns and inconsistencies in the current methodologies, as presence questionnaires are frequently utilized in a non-standardised manner. We will propose well-grounded improvements and constructive approaches based on our findings. These will improve the validity and effectiveness of these questionnaires and lead to more consistent and replicable results in presence research.
Supplemental Material
MP4 File
Talk Video
- Download
- 13.52 MB
- Transcript
References
[1]
Lena Fanya Aeschbach, Sebastian A.C. Perrig, Lorena Weder, Klaus Opwis, and Florian Brühlmann. 2021. Transparency in Measurement Reporting: A Systematic Literature Review of CHI PLAY. Proceedings of the ACM on Human-Computer Interaction 5, CHI PLAY (Oct. 2021), 1–21. https://doi.org/10.1145/3474660
[2]
Frank Biocca, Chad Harms, and Judee K. Burgoon. 2003. Toward a More Robust Theory and Measure of Social Presence: Review and Suggested Criteria. Presence: Teleoperators and Virtual Environments 12, 5 (Oct. 2003), 456–480. https://doi.org/10.1162/105474603322761270
[3]
Carmen Bisogni, Lucia Cascone, Aniello Castiglione, and Ignazio Passero. 2021. Deep learning for emotion driven user experiences. Pattern Recognition Letters 152 (Dec. 2021), 115–121. https://doi.org/10.1016/j.patrec.2021.09.004
[4]
Yasra Chandio, Noman Bashir, Victoria Interrante, and Fatima M. Anwar. 2023. Investigating the Correlation Between Presence and Reaction Time in Mixed Reality. IEEE Transactions on Visualization and Computer Graphics Early access (2023), 1–13. https://doi.org/10.1109/TVCG.2023.3319563
[5]
Lee Anna Clark and David Watson. 2019. Constructing validity: New developments in creating objective measuring instruments.Psychological Assessment 31, 12 (Dec. 2019), 1412–1427. https://doi.org/10.1037/pas0000626
[6]
James J Cummings and Erin E Wertz. 2022. Capturing social presence: concept explication through an empirical analysis of social presence measures. Journal of Computer-Mediated Communication 28, 1 (Nov. 2022), zmac027. https://doi.org/10.1093/jcmc/zmac027
[7]
Zoltan Dienes. 2004. Assumptions of Subjective Measures of Unconscious Mental States: Higher Order Thoughts and Bias. Journal of Consciousness Studies 11, 9 (Jan. 2004), 25–45.
[8]
William M. Felton and Russell E. Jackson. 2022. Presence: A Review. International Journal of Human–Computer Interaction 38, 1 (Jan. 2022), 1–18. https://doi.org/10.1080/10447318.2021.1921368
[9]
Jonathan Freeman, S. E. Avons, Ray Meddis, Don E. Pearson, and Wijnand IJsselsteijn. 2000. Using Behavioral Realism to Estimate Presence: A Study of the Utility of Postural Responses to Motion Stimuli. Presence: Teleoperators and Virtual Environments 9, 2 (April 2000), 149–164. https://doi.org/10.1162/105474600566691
[10]
Jonathan Freeman, S. E. Avons, Don E. Pearson, and Wijnand A. IJsselsteijn. 1999. Effects of Sensory Information and Prior Experience on Direct Subjective Ratings of Presence. Presence: Teleoperators and Virtual Environments 8, 1 (Feb. 1999), 1–13. https://doi.org/10.1162/105474699566017
[11]
Maia Garau, Doron Friedman, Hila Ritter Widenfeld, Angus Antley, Andrea Brogni, and Mel Slater. 2008. Temporal and Spatial Variations in Presence: Qualitative Analysis of Interviews from an Experiment on Breaks in Presence. Presence: Teleoperators and Virtual Environments 17, 3 (June 2008), 293–309. https://doi.org/10.1162/pres.17.3.293
[12]
Sarah Graf and Valentin Schwind. 2020. Inconsistencies of Presence Questionnaires in Virtual Reality. In 26th ACM Symposium on Virtual Reality Software and Technology. ACM, Virtual Event Canada, 1–3. https://doi.org/10.1145/3385956.3422105
[13]
Simone Grassini and Karin Laumann. 2020. Questionnaire Measures and Physiological Correlates of Presence: A Systematic Review. Frontiers in Psychology 11 (2020), 1–21. https://www.frontiersin.org/articles/10.3389/fpsyg.2020.00349
[14]
Wijnand IJsselsteijn, Huib De Ridder, Jonathan Freeman, S. E. Avons, and Don Bouwhuis. 2001. Effects of Stereoscopic Presentation, Image Motion, and Screen Size on Subjective and Objective Corroborative Measures of Presence. Presence: Teleoperators and Virtual Environments 10, 3 (June 2001), 298–311. https://doi.org/10.1162/105474601300343621
[15]
Karel Kreijns, Kate Xu, and Joshua Weidlich. 2022. Social Presence: Conceptualization and Measurement. Educational Psychology Review 34, 1 (March 2022), 139–170. https://doi.org/10.1007/s10648-021-09623-8
[16]
Jari Laarni, Niklas Ravaja, Timo Saari, Saskia Böcking, Tilo Hartmann, and Holger Schramm. 2015. Ways to Measure Spatial Presence: Review and Future Directions. In Immersed in Media: Telepresence Theory, Measurement & Technology, Matthew Lombard, Frank Biocca, Jonathan Freeman, Wijnand IJsselsteijn, and Rachel J. Schaevitz (Eds.). Springer International Publishing, Cham, 139–185. https://doi.org/10.1007/978-3-319-10190-3_8
[17]
Cha Lee, Gustavo A. Rincon, Greg Meyer, Tobias Hollerer, and Doug A. Bowman. 2013. The Effects of Visual Realism on Search Tasks in Mixed Reality Simulation. IEEE Transactions on Visualization and Computer Graphics 19, 4 (April 2013), 547–556. https://doi.org/10.1109/TVCG.2013.41
[18]
Jane Lessiter, Jonathan Freeman, Edmund Keogh, and Jules Davidoff. 2001. A Cross-Media Presence Questionnaire: The ITC-Sense of Presence Inventory. Presence: Teleoperators and Virtual Environments 10, 3 (June 2001), 282–297. https://doi.org/10.1162/105474601300343612
[19]
Matthew Lombard, Frank Biocca, Jonathan Freeman, Wijnand IJsselsteijn, and Rachel J. Schaevitz (Eds.). 2015. Immersed in Media. Springer International Publishing, Cham. https://doi.org/10.1007/978-3-319-10190-3
[20]
Marvin Minsky. 1980. Telepresence. OMNI magazine 2, 9 (1980), 45–51. https://cir.nii.ac.jp/crid/1570854175773549312
[21]
Olivier Nannipieri. 2022. Do Presence Questionnaires Actually Measure Presence? A Content Analysis of Presence Measurement Scales. In Extended Reality. Vol. 13445. Springer International Publishing, Cham, 273–295. https://doi.org/10.1007/978-3-031-15546-8_24
[22]
Richard E. Nisbett and Timothy D. Wilson. 1977. The halo effect: Evidence for unconscious alteration of judgments.Journal of Personality and Social Psychology 35, 4 (April 1977), 250–256. https://doi.org/10.1037/0022-3514.35.4.250
[23]
Kristine L. Nowak and Frank Biocca. 2003. The Effect of the Agency and Anthropomorphism on Users’ Sense of Telepresence, Copresence, and Social Presence in Virtual Environments. Presence: Teleoperators and Virtual Environments 12, 5 (Oct. 2003), 481–494. https://doi.org/10.1162/105474603322761289
[24]
Seok Hee Oh, Jung Woon Park, and Seong-Jin Cho. 2022. Effectiveness of the VR Cognitive Training for Symptom Relief in Patients with ADHD. Journal of Web Engineering 21, 3 (March 2022), 767–788. https://doi.org/10.13052/jwe1540-9589.21310
[25]
Matthew J. Page, Joanne E. McKenzie, Patrick M. Bossuyt, Isabelle Boutron, Tammy C. Hoffmann, Cynthia D. Mulrow, Larissa Shamseer, Jennifer M. Tetzlaff, Elie A. Akl, Sue E. Brennan, Roger Chou, Julie Glanville, Jeremy M. Grimshaw, Asbjørn Hróbjartsson, Manoj M. Lalu, Tianjing Li, Elizabeth W. Loder, Evan Mayo-Wilson, Steve McDonald, Luke A. McGuinness, Lesley A. Stewart, James Thomas, Andrea C. Tricco, Vivian A. Welch, Penny Whiting, and David Moher. 2021. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372 (March 2021), n71. https://doi.org/10.1136/bmj.n71
[26]
Sebastian A. C. Perrig, Nicolas Scharowski, and Florian Brühlmann. 2023. Trust Issues with Trust Scales: Examining the Psychometric Quality of Trust Measures in the Context of AI. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, Hamburg Germany, 1–7. https://doi.org/10.1145/3544549.3585808
[27]
Giuseppe Riva, Fabrizia Mantovani, Claret Samantha Capideville, Alessandra Preziosa, Francesca Morganti, Daniela Villani, Andrea Gaggioli, Cristina Botella, and Mariano Alcañiz. 2007. Affective Interactions Using Virtual Reality: The Link between Presence and Emotions. CyberPsychology & Behavior 10, 1 (Feb. 2007), 45–56. https://doi.org/10.1089/cpb.2006.9993
[28]
Giuseppe Riva, John Waterworth, and Dianne Murray. 2014. Interacting with Presence: HCI and the Sense of Presence in Computer-mediated Environments. DE GRUYTER OPEN, Poland. https://doi.org/10.2478/9783110409697
[29]
Richard Skarbez, Frederick P. Brooks, Jr., and Mary C. Whitton. 2018. A Survey of Presence and Related Concepts. Comput. Surveys 50, 6 (Nov. 2018), 1–39. https://doi.org/10.1145/3134301
[30]
Mel Slater. 2004. How Colorful Was Your Day? Why Questionnaires Cannot Assess Presence in Virtual Environments. Presence: Teleoperators and Virtual Environments 13, 4 (Aug. 2004), 484–493. https://doi.org/10.1162/1054746041944849
[31]
Mel Slater, Christoph Guger, Guenter Edlinger, Robert Leeb, Gert Pfurtscheller, Angus Antley, Maia Garau, Andrea Brogni, and Doron Friedman. 2006. Analysis of Physiological Responses to a Social Situation in an Immersive Virtual Environment. Presence: Teleoperators and Virtual Environments 15, 5 (Oct. 2006), 553–569. https://doi.org/10.1162/pres.15.5.553
[32]
Stella Sylaiou, Katerina Mania, Athanasis Karoulis, and Martin White. 2010. Exploring the relationship between presence and enjoyment in a virtual museum. International Journal of Human-Computer Studies 68, 5 (May 2010), 243–253. https://doi.org/10.1016/j.ijhcs.2009.11.002
[33]
Phil Turner and Susan Turner. 2006. Place, Sense of Place, and Presence. Presence: Teleoperators and Virtual Environments 15, 2 (April 2006), 204–217. https://doi.org/10.1162/pres.2006.15.2.204
[34]
Bartholomäus Wissmath, David Weibel, and D. Stricker. 2008. When and how to assess subjective overall judgments of presence?. In Wissmath, Bartholomäus; Weibel, David; Stricker, D. (2008). When and how to assess subjective overall judgments of presence? In: Spagnolli, Anna; Gamberini, Luciano (eds.) Proceedings of the 11th Annual International Workshop on Presence, Padova, 16-18 October 2008 (pp. 238-243). Padova: CLEUP, Anna Spagnolli and Luciano Gamberini (Eds.). CLEUP, Padova, 238–243. https://boris.unibe.ch/33931/
[35]
Bob G. Witmer and Michael J. Singer. 1998. Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence: Teleoperators and Virtual Environments 7, 3 (June 1998), 225–240. https://doi.org/10.1162/105474698565686
Index Terms
- "Never The Same": Systematic Analysis of the Methodological Issues in the Presence Studies That Employ Questionnaires
Recommendations
Reliability, equivalence and respondent preference of computerized versus paper-and-pencil mental health questionnaires
The purpose of this study was to examine the reliability, equivalence and respondent preference of a computerized version of the General Health Questionnaire (GHQ-12), Symptom Checklist (SCL-90-R), Medical Outcomes Study Social Support Survey (MOSSSS), ...
Interchangeability of computer and paper based questionnaires in gathering computer experience data from young children
CHI '12: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsThe study compares self-reported computer experience (CE) obtained from children in computer-assisted and paper-and-pencil self-administered questionnaires. Twenty primary school children aged between 8 and 9 years completed a set of CE questions in ...
The Influence of in-VR Questionnaire Design on the User Experience
VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and TechnologyResearchers study the user experience in Virtual Reality (VR) typically by collecting either sensory data or using questionnaires. While traditional questionnaire formats present it through web-based survey tools (out-VR), recent studies investigate ...
Comments
Information & Contributors
Information
Published In
May 2024
4761 pages
ISBN:9798400703317
DOI:10.1145/3613905
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 May 2024
Check for updates
Author Tags
Qualifiers
- Work in progress
- Research
- Refereed limited
Data Availability
Funding Sources
Conference
Acceptance Rates
Overall Acceptance Rate 6,164 of 23,696 submissions, 26%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 162Total Downloads
- Downloads (Last 12 months)162
- Downloads (Last 6 weeks)38
Reflects downloads up to 13 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format