Experimental Analysis of Freehand Multi-object Selection Techniques in Virtual Reality Head-Mounted Displays
Authors: 
Rongkai Shi, Yushi Wei, Xuning Hu, Yu Liu, Yong Yue, Lingyun Yu, Hai-Ning LiangAuthors Info & Claims
Rongkai Shi, Yushi Wei, Xuning Hu, Yu Liu, Yong Yue, Lingyun Yu, Hai-Ning LiangAuthors Info & ClaimsArticle No.: 529, Pages 93 - 111
Abstract
Object selection is essential in virtual reality (VR) head-mounted displays (HMDs). Prior work mainly focuses on enhancing and evaluating techniques for selecting a single object in VR, leaving a gap in the techniques for multi-object selection, a more complex but common selection scenario. To enable multi-object selection, the interaction technique should support group selection in addition to the default pointing selection mode for acquiring a single target. This composite interaction could be particularly challenging when using freehand gestural input. In this work, we present an empirical comparison of six freehand techniques, which are comprised of three mode-switching gestures (Finger Segment, Multi-Finger, and Wrist Orientation) and two group selection techniques (Cone-casting Selection and Crossing Selection) derived from prior work. Our results demonstrate the performance, user experience, and preference of each technique. The findings derive three design implications that can guide the design of freehand techniques for multi-object selection in VR HMDs.
References
[1]
Ferran Argelaguet and Carlos Andujar. 2013. A survey of 3D object selection techniques for virtual environments. Computers & Graphics, 37, 3 (2013), 121–136. issn:0097-8493 https://doi.org/10.1016/j.cag.2012.12.003
[2]
Joanna Bergström, Tor-Salve Dalsgaard, Jason Alexander, and Kasper Hornbæ k. 2021. How to Evaluate Object Selection and Manipulation in VR? Guidelines from 20 Years of Studies. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21). Association for Computing Machinery, New York, NY, USA. Article 533, 20 pages. isbn:9781450380966 https://doi.org/10.1145/3411764.3445193
[3]
Gunnar A Borg. 1982. Psychophysical bases of perceived exertion. Medicine and science in sports and exercise, 14, 5 (1982), 377–381.
[4]
Doug Bowman, Chadwick Wingrave, Joshua Campbell, and Vinh Ly. 2001. Using pinch gloves (tm) for both natural and abstract interaction techniques in virtual environments. Virginia Tech.
[5]
James V. Bradley. 1958. Complete Counterbalancing of Immediate Sequential Effects in a Latin Square Design. J. Amer. Statist. Assoc., 53, 282 (1958), 525–528. https://doi.org/10.1080/01621459.1958.10501456 arxiv:https://www.tandfonline.com/doi/pdf/10.1080/01621459.1958.10501456.
[6]
Gavin Buckingham. 2021. Hand Tracking for Immersive Virtual Reality: Opportunities and Challenges. Frontiers in Virtual Reality, 2 (2021), issn:2673-4192 https://doi.org/10.3389/frvir.2021.728461
[7]
Di Laura Chen, Ravin Balakrishnan, and Tovi Grossman. 2020. Disambiguation Techniques for Freehand Object Manipulations in Virtual Reality. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 285–292. https://doi.org/10.1109/VR46266.2020.00048
[8]
Tor-Salve Dalsgaard, Jarrod Knibbe, and Joanna Bergström. 2021. Modeling Pointing for 3D Target Selection in VR. In Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology (VRST ’21). Association for Computing Machinery, New York, NY, USA. Article 42, 10 pages. isbn:9781450390927 https://doi.org/10.1145/3489849.3489853
[9]
Lisa A. Elkin, Matthew Kay, James J. Higgins, and Jacob O. Wobbrock. 2021. An Aligned Rank Transform Procedure for Multifactor Contrast Tests. In The 34th Annual ACM Symposium on User Interface Software and Technology (UIST ’21). Association for Computing Machinery, New York, NY, USA. 754–768. isbn:9781450386357 https://doi.org/10.1145/3472749.3474784
[10]
S. Frees and G.D. Kessler. 2005. Precise and rapid interaction through scaled manipulation in immersive virtual environments. In IEEE Proceedings. VR 2005. Virtual Reality, 2005. 99–106. https://doi.org/10.1109/VR.2005.1492759
[11]
Sandra G. Hart. 2006. Nasa-Task Load Index (NASA-TLX); 20 Years Later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 50, 9 (2006), 904–908. https://doi.org/10.1177/154193120605000909 arxiv:https://doi.org/10.1177/154193120605000909.
[12]
Ken Hinckley, Francois Guimbretiere, Patrick Baudisch, Raman Sarin, Maneesh Agrawala, and Ed Cutrell. 2006. The Springboard: Multiple Modes in One Spring-Loaded Control. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’06). Association for Computing Machinery, New York, NY, USA. 181–190. isbn:1595933727 https://doi.org/10.1145/1124772.1124801
[13]
Susu HUANG, Daqing QI, Jiabin YUAN, and Huawei TU. 2019. Review of studies on target acquisition in virtual reality based on the crossing paradigm. Virtual Reality & Intelligent Hardware, 1, 3 (2019), 251–264. issn:2096-5796 https://doi.org/10.3724/SP.J.2096-5796.2019.0006 Human-computer interactions for virtual reality
[14]
Akira Ishii, Takuya Adachi, Keigo Shima, Shuta Nakamae, Buntarou Shizuki, and Shin Takahashi. 2017. FistPointer: Target Selection Technique using Mid-air Interaction for Mobile VR Environment. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ’17). Association for Computing Machinery, New York, NY, USA. 474. isbn:9781450346566 https://doi.org/10.1145/3027063.3049795
[15]
Joseph J LaViola Jr, Ernst Kruijff, Ryan P McMahan, Doug Bowman, and Ivan P Poupyrev. 2017. 3D user interfaces: theory and practice. Addison-Wesley Professional, Boston, MA, USA. isbn:0134034325
[16]
James R. Lewis. 2018. The System Usability Scale: Past, Present, and Future. International Journal of Human–Computer Interaction, 34, 7 (2018), 577–590. https://doi.org/10.1080/10447318.2018.1455307 arxiv:https://doi.org/10.1080/10447318.2018.1455307.
[17]
John Finley Lucas. 2005. Design and evaluation of 3D multiple object selection techniques. Ph. D. Dissertation. Virginia Tech.
[18]
Mykola Maslych, Yahya Hmaiti, Ryan Ghamandi, Paige Leber, Ravi Kiran Kattoju, Jacob Belga, and Joseph J. LaViola. 2023. Toward Intuitive Acquisition of Occluded VR Objects Through an Interactive Disocclusion Mini-map. In 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR). 460–470. https://doi.org/10.1109/VR55154.2023.00061
[19]
Sven Mayer, Valentin Schwind, Robin Schweigert, and Niels Henze. 2018. The Effect of Offset Correction and Cursor on Mid-Air Pointing in Real and Virtual Environments. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18). Association for Computing Machinery, New York, NY, USA. 1–13. isbn:9781450356206 https://doi.org/10.1145/3173574.3174227
[20]
Daniel Mendes, Daniel Medeiros, Maurício Sousa, Eduardo Cordeiro, Alfredo Ferreira, and Joaquim A. Jorge. 2017. Design and evaluation of a novel out-of-reach selection technique for VR using iterative refinement. Computers & Graphics, 67 (2017), 95–102. issn:0097-8493 https://doi.org/10.1016/j.cag.2017.06.003
[21]
Mark R Mine. 1995. Virtual environment interaction techniques. UNC Chapel Hill CS Dept.
[22]
Mark R. Mine, Frederick P. Brooks, and Carlo H. Sequin. 1997. Moving Objects in Space: Exploiting Proprioception in Virtual-Environment Interaction. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’97). ACM Press/Addison-Wesley Publishing Co., USA. 19–26. isbn:0897918967 https://doi.org/10.1145/258734.258747
[23]
Donald A. Norman. 2010. Natural User Interfaces Are Not Natural. Interactions, 17, 3 (2010), may, 6–10. issn:1072-5520 https://doi.org/10.1145/1744161.1744163
[24]
Chanho Park, Hyunwoo Cho, Sangheon Park, Young-Suk Yoon, and Sung-Uk Jung. 2019. HandPoseMenu: Hand Posture-Based Virtual Menus for Changing Interaction Mode in 3D Space. In Proceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces (ISS ’19). Association for Computing Machinery, New York, NY, USA. 361–366. isbn:9781450368919 https://doi.org/10.1145/3343055.3360752
[25]
Ken Pfeuffer, Benedikt Mayer, Diako Mardanbegi, and Hans Gellersen. 2017. Gaze + Pinch Interaction in Virtual Reality. In Proceedings of the 5th Symposium on Spatial User Interaction (SUI ’17). Association for Computing Machinery, New York, NY, USA. 99–108. isbn:9781450354868 https://doi.org/10.1145/3131277.3132180
[26]
Ken Pfeuffer, Lukas Mecke, Sarah Delgado Rodriguez, Mariam Hassib, Hannah Maier, and Florian Alt. 2020. Empirical Evaluation of Gaze-Enhanced Menus in Virtual Reality. In Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology (VRST ’20). Association for Computing Machinery, New York, NY, USA. Article 20, 11 pages. isbn:9781450376198 https://doi.org/10.1145/3385956.3418962
[27]
Henning Pohl, Klemen Lilija, Jess McIntosh, and Kasper Hornbæ k. 2021. Poros: Configurable Proxies for Distant Interactions in VR. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21). Association for Computing Machinery, New York, NY, USA. Article 532, 12 pages. isbn:9781450380966 https://doi.org/10.1145/3411764.3445685
[28]
Ivan Poupyrev, Mark Billinghurst, Suzanne Weghorst, and Tadao Ichikawa. 1996. The Go-Go Interaction Technique: Non-Linear Mapping for Direct Manipulation in VR. In Proceedings of the 9th Annual ACM Symposium on User Interface Software and Technology (UIST ’96). Association for Computing Machinery, New York, NY, USA. 79–80. isbn:0897917987 https://doi.org/10.1145/237091.237102
[29]
Jef Raskin. 2000. The humane interface: new directions for designing interactive systems. Addison-Wesley Professional.
[30]
Gang Ren and Eamonn O’Neill. 2013. 3D selection with freehand gesture. Computers & Graphics, 37, 3 (2013), 101–120. issn:0097-8493 https://doi.org/10.1016/j.cag.2012.12.006
[31]
Abigail J. Sellen, Gordon P. Kurtenbach, and William A.S. Buxton. 1992. The Prevention of Mode Errors Through Sensory Feedback. Human–Computer Interaction, 7, 2 (1992), 141–164. https://doi.org/10.1207/s15327051hci0702_1
[32]
Rongkai Shi, Yushi Wei, Xueying Qin, Pan Hui, and Hai-Ning Liang. 2023. Exploring Gaze-Assisted and Hand-Based Region Selection in Augmented Reality. Proc. ACM Hum.-Comput. Interact., 7, ETRA (2023), Article 160, may, 19 pages. https://doi.org/10.1145/3591129
[33]
Rongkai Shi, Jialin Zhang, Wolfgang Stuerzlinger, and Hai-Ning Liang. 2022. Group-Based Object Alignment in Virtual Reality Environments. In Proceedings of the 2022 ACM Symposium on Spatial User Interaction (SUI ’22). Association for Computing Machinery, New York, NY, USA. Article 2, 11 pages. isbn:9781450399487 https://doi.org/10.1145/3565970.3567682
[34]
Rongkai Shi, Jialin Zhang, Yong Yue, Lingyun Yu, and Hai-Ning Liang. 2023. Exploration of Bare-Hand Mid-Air Pointing Selection Techniques for Dense Virtual Reality Environments. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA ’23). Association for Computing Machinery, New York, NY, USA. Article 109, 7 pages. isbn:9781450394222 https://doi.org/10.1145/3544549.3585615
[35]
Rongkai Shi, Nan Zhu, Hai-Ning Liang, and Shengdong Zhao. 2021. Exploring Head-based Mode-Switching in Virtual Reality. In 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 118–127. https://doi.org/10.1109/ISMAR52148.2021.00026
[36]
Jesse Smith, Isaac Wang, Winston Wei, Julia Woodward, and Jaime Ruiz. 2020. Evaluating the Scalability of Non-Preferred Hand Mode Switching in Augmented Reality. In Proceedings of the International Conference on Advanced Visual Interfaces (AVI ’20). Association for Computing Machinery, New York, NY, USA. Article 19, 9 pages. isbn:9781450375351 https://doi.org/10.1145/3399715.3399850
[37]
Jesse Smith, Isaac Wang, Julia Woodward, and Jaime Ruiz. 2019. Experimental Analysis of Single Mode Switching Techniques in Augmented Reality. In Proceedings of the 45th Graphics Interface Conference on Proceedings of Graphics Interface 2019 (GI’19). Canadian Human-Computer Communications Society, Waterloo, CAN. Article 20, 8 pages. isbn:9780994786845 https://doi.org/10.20380/GI2019.20
[38]
Zhaomou Song, John J. Dudley, and Per Ola Kristensson. 2022. Efficient Special Character Entry on a Virtual Keyboard by Hand Gesture-Based Mode Switching. In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 864–871. https://doi.org/10.1109/ISMAR55827.2022.00105
[39]
Hemant Bhaskar Surale, Fabrice Matulic, and Daniel Vogel. 2019. Experimental Analysis of Barehand Mid-Air Mode-Switching Techniques in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA. 1–14. isbn:9781450359702 https://doi.org/10.1145/3290605.3300426
[40]
Huawei Tu, Susu Huang, Jiabin Yuan, Xiangshi Ren, and Feng Tian. 2019. Crossing-Based Selection with Virtual Reality Head-Mounted Displays. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA. 1–14. isbn:9781450359702 https://doi.org/10.1145/3290605.3300848
[41]
Stephen Uzor and Per Ola Kristensson. 2021. An Exploration of Freehand Crossing Selection in Head-Mounted Augmented Reality. ACM Trans. Comput.-Hum. Interact., 28, 5 (2021), Article 33, aug, 27 pages. issn:1073-0516 https://doi.org/10.1145/3462546
[42]
Tingjie Wan, Rongkai Shi, Wenge Xu, Yue Li, Katie Atkinson, Lingyun Yu, and Hai-Ning Liang. 2024. Hands-free multi-type character text entry in virtual reality. Virtual Reality, 28, 8 (2024), 1–19. https://doi.org/10.1007/s10055-023-00902-z
[43]
Tingjie Wan, Yushi Wei, Rongkai Shi, Junxiao Shen, Per Ola Kristensson, Katie Atkinson, and Hai-Ning Liang. 2024. Design and Evaluation of Controller-Based Raycasting Methods for Efficient Alphanumeric and Special Character Entry in Virtual Reality. IEEE Transactions on Visualization and Computer Graphics, 30, 9 (2024), 6493–6506. https://doi.org/10.1109/TVCG.2024.3349428
[44]
Curtis Wilkes and Doug A. Bowman. 2008. Advantages of Velocity-Based Scaling for Distant 3D Manipulation. In Proceedings of the 2008 ACM Symposium on Virtual Reality Software and Technology (VRST ’08). Association for Computing Machinery, New York, NY, USA. 23–29. isbn:9781595939517 https://doi.org/10.1145/1450579.1450585
[45]
Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins. 2011. The aligned rank transform for nonparametric factorial analyses using only anova procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). Association for Computing Machinery, New York, NY, USA. 143–146. isbn:9781450302289 https://doi.org/10.1145/1978942.1978963
[46]
Zhiqing Wu, Difeng Yu, and Jorge Goncalves. 2023. Point- and Volume-Based Multi-object Acquisition in VR. In Human-Computer Interaction – INTERACT 2023, José Abdelnour Nocera, Marta Kristín Lárusdóttir, Helen Petrie, Antonio Piccinno, and Marco Winckler (Eds.). Springer Nature Switzerland, Cham. 20–42. isbn:978-3-031-42280-5
[47]
Haijun Xia, Sebastian Herscher, Ken Perlin, and Daniel Wigdor. 2018. Spacetime: Enabling Fluid Individual and Collaborative Editing in Virtual Reality. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST ’18). Association for Computing Machinery, New York, NY, USA. 853–866. isbn:9781450359481 https://doi.org/10.1145/3242587.3242597
[48]
Difeng Yu, Qiushi Zhou, Tilman Dingler, Eduardo Velloso, and Jorge Goncalves. 2022. Blending On-Body and Mid-Air Interaction in Virtual Reality. In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 637–646. https://doi.org/10.1109/ISMAR55827.2022.00081
[49]
Difeng Yu, Qiushi Zhou, Joshua Newn, Tilman Dingler, Eduardo Velloso, and Jorge Goncalves. 2020. Fully-Occluded Target Selection in Virtual Reality. IEEE Transactions on Visualization and Computer Graphics, 26, 12 (2020), 3402–3413. https://doi.org/10.1109/TVCG.2020.3023606
[50]
Lixiang Zhao, Tobias Isenberg, Fuqi Xie, Hai-Ning Liang, and Lingyun Yu. 2024. MeTACAST: Target- and Context-Aware Spatial Selection in VR. IEEE Transactions on Visualization and Computer Graphics, 30, 1 (2024), 480–494. https://doi.org/10.1109/TVCG.2023.3326517
Index Terms
- Experimental Analysis of Freehand Multi-object Selection Techniques in Virtual Reality Head-Mounted Displays
Recommendations
Do virtual reality head-mounted displays make a difference? A comparison of presence and self-efficacy between head-mounted displays and desktop computer-facilitated virtual environments
AbstractVirtual reality (VR) has made it possible for users to access novel digital experiences. An interesting question that arises in the context of VR is whether it appears or feels different to users when different virtual environments are used. This ...
Differences in virtual and physical head orientation predict sickness during active head-mounted display-based virtual reality
AbstractDuring head-mounted display (HMD)-based virtual reality (VR), head movements and motion-to-photon-based display lag generate differences in our virtual and physical head pose (referred to as DVP). We propose that large-amplitude, time-varying ...
EZCursorVR: 2D Selection with Virtual Reality Head-Mounted Displays
GI '18: Proceedings of the 44th Graphics Interface ConferenceWe present an evaluation of a new selection technique for virtual reality (VR) systems presented on head-mounted displays. The technique, dubbed EZCursorVR, presents a 2D cursor that moves in a head-fixed plane, simulating 2D desktop-like cursor control ...
Comments
Information & Contributors
Information
Published In
December 2024
693 pages
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 24 October 2024
Published in PACMHCI Volume 8, Issue ISS
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Suzhou Municipal Key Laboratory for Intelligent Virtual Engineering
- National Natural Science Foundation of China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 146Total Downloads
- Downloads (Last 12 months)146
- Downloads (Last 6 weeks)47
Reflects downloads up to 18 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in