poster

Terrain Modelling with a Pen & Touch Tablet and Mid-Air Gestures in Virtual Reality

Authors:

Fabrice Matulic,

Daniel VogelAuthors Info & Claims

CHI EA '22: CHI Conference on Human Factors in Computing Systems Extended Abstracts

Article No.: 301, Pages 1 - 7

https://doi.org/10.1145/3491101.3519867

Published: 28 April 2022 Publication History

Abstract

World building or terrain modelling is an essential task when designing games, natural simulations or artistic creations involving virtual 3D landscapes. To support this task, we propose a virtual reality (VR) system based on a pen and touch tablet used in a sitting position (desktop VR) such that both hands are free to interact in an asymmetric way (pen hand + other bare hand). We present and compare several techniques to perform navigation, sculpting and menu operations using the two hands, which interact on and above the tablet surface, i.e. using the pen, touch and mid-air input spaces. A qualitative evaluation with 16 participants confirms the entertaining nature and practical benefits of our system. The study further underlines the complementarity of the different modalities and identifies the promising—and as of yet underexplored—combination of bimanual touch + pen + mid-air interaction in desktop VR.

Supplementary Material

MP4 File (3491101.3519867-video-figure.mp4)

Video Figure

Download
52.29 MB

MP4 File (3491101.3519867-video-preview.mp4)

Video Preview

Download
20.07 MB

References

[1]

Bruno R De Araùjo, Géry Casiez, and Joaquim A Jorge. 2012. Mockup Builder: Direct 3D Modeling on and above the Surface in a Continuous Interaction Space. In Proceedings of Graphics Interface 2012 (GI ’12), 173–180.

[2]

Rahul Arora, Rubaiat Habib Kazi, Tovi Grossman, George Fitzmaurice, and Karan Singh. 2018. SymbiosisSketch: Combining 2D & 3D Sketching for Designing Detailed 3D Objects in Situ. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–15. Retrieved from https://doi.org/10.1145/3173574.3173759

Digital Library

[3]

Peter Brandl, Clifton Forlines, Daniel Wigdor, Michael Haller, and Chia Shen. 2008. Combining and measuring the benefits of bimanual pen and direct-touch interaction on horizontal interfaces. Proceedings of the working conference on Advanced visual interfaces, 154–161. https://doi.org/10.1145/1385569.1385595

Digital Library

[4]

Theocharis Chatzis, Andreas Stergioulas, Dimitrios Konstantinidis, Kosmas Dimitropoulos, and Petros Daras. 2020. A Comprehensive Study on Deep Learning-Based 3D Hand Pose Estimation Methods. Applied Sciences 10, 19. https://doi.org/10.3390/app10196850

[5]

Noah Coomer, Sadler Bullard, William Clinton, and Betsy Williams-Sanders. 2018. Evaluating the Effects of Four VR Locomotion Methods: Joystick, Arm-Cycling, Point-Tugging, and Teleporting. In Proceedings of the 15th ACM Symposium on Applied Perception (SAP ’18). https://doi.org/10.1145/3225153.3225175

Digital Library

[6]

Tobias Drey, Jan Gugenheimer, Julian Karlbauer, Maximilian Milo, and Enrico Rukzio. 2020. VRSketchIn: Exploring the Design Space of Pen and Tablet Interaction for 3D Sketching in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376628

Digital Library

[7]

Sevinc Eroglu, Frederic Stefan, Alain Chevalier, Daniel Roettger, Daniel Zielasko, Torsten W Kuhlen, and Benjamin Weyers. 2021. Design and Evaluation of a Free-Hand VR-based Authoring Environment for Automated Vehicle Testing. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), 1–10. https://doi.org/10.1109/VR50410.2021.00020

[8]

Thomas Fröhlich, Dmitry Alexandrovsky, Timo Stabbert, Tanja Döring, and Rainer Malaka. 2018. VRBox: A Virtual Reality Augmented Sandbox for Immersive Playfulness, Creativity and Exploration. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play (CHI PLAY ’18), 153–162. https://doi.org/10.1145/3242671.3242697

Digital Library

[9]

Travis Gesslein, Verena Biener, Philipp Gagel, Daniel Schneider, Per Ola Kristensson, Eyal Ofek, Michel Pahud, and Jens Grubert. 2020. Pen-based Interaction with Spreadsheets in Mobile Virtual Reality. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 361–373. https://doi.org/10.1109/ISMAR50242.2020.00063

[10]

Yves Guiard. 1987. Asymmetric division of labor in human skilled bimanual action: the kinematic chain as a model. Journal of Motor Behavior 19, 4: 486–517.

[11]

François Guimbretière and Chau Nguyen. 2012. Bimanual Marking Menu for near Surface Interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 825–828. Retrieved from https://doi.org/10.1145/2207676.2208521

Digital Library

[12]

Ken Hinckley, Koji Yatani, Michel Pahud, Nicole Coddington, Jenny Rodenhouse, Andy Wilson, Hrvoje Benko, and Bill Buxton. 2010. Pen + touch = new tools. Proceedings of the 23nd annual ACM symposium on User interface software and technology, 27–36. https://doi.org/10.1145/1866029.1866036

Digital Library

[13]

Kenrick Kin, Björn Hartmann, and Maneesh Agrawala. 2011. Two-Handed Marking Menus for Multitouch Devices. ACM Trans. Comput.-Hum. Interact. 18, 3. https://doi.org/10.1145/1993060.1993066

Digital Library

[14]

Josen Daniel O De Leon, Romelio P Tavas, Rodolfo A Aranzanso, and Rowel O Atienza. 2016. Genesys: A Virtual Reality scene builder. In 2016 IEEE Region 10 Conference (TENCON), 3708–3711. https://doi.org/10.1109/TENCON.2016.7848751

[15]

Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. 2021. Semi-supervised 3d hand-object poses estimation with interactions in time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14687–14697.

[16]

Fabrice Matulic and Moira Norrie. 2013. Pen and Touch Gestural Environment for Document Editing on Interactive Tabletops. Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces, 41–50.

Digital Library

[17]

Manuel Meier, Paul Streli, Andreas Fender, and Christian Holz. 2021. TaplD: Rapid Touch Interaction in Virtual Reality using Wearable Sensing. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), 519–528. https://doi.org/10.1109/VR50410.2021.00076

[18]

Meta. 2021. Horizon Worlds. Retrieved from https://www.oculus.com/facebook-horizon/

[19]

Daniel L Odell, Richard C Davis, Andrew Smith, and Paul K Wright. 2004. Toolglasses, Marking Menus, and Hotkeys: A Comparison of One and Two-Handed Command Selection Techniques. In Proceedings of Graphics Interface 2004 (GI ’04), 17–24.

[20]

Duc-Minh Pham and Wolfgang Stuerzlinger. 2019. Is the Pen Mightier than the Controller? A Comparison of Input Devices for Selection in Virtual and Augmented Reality. In 25th ACM Symposium on Virtual Reality Software and Technology (VRST ’19). https://doi.org/10.1145/3359996.3364264

Digital Library

[21]

Hugo Romat, Andreas Fender, Manuel Meier, and Christian Holz. 2021. Flashpen: A High-Fidelity and High-Precision Multi-Surface Pen for Virtual Reality. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), 306–315. https://doi.org/10.1109/VR50410.2021.00053

[22]

Hemant Bhaskar Surale, Aakar Gupta, Mark Hancock, and Daniel Vogel. 2019. TabletInVR: Exploring the Design Space for Using a Multi-Touch Tablet in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–13. Retrieved from https://doi.org/10.1145/3290605.3300243

Digital Library

[23]

Stephan P Swinnen. 2002. Intermanual coordination: From behavioural principles to neural-network interactions. Nature Reviews Neuroscience 3, 5: 348–359. https://doi.org/10.1038/nrn807

[24]

Daniel Vogel, Matthew Cudmore, Géry Casiez, Ravin Balakrishnan, and Liam Keliher. 2009. Hand Occlusion with Tablet-Sized Direct Pen Input. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 557–566. Retrieved from https://doi.org/10.1145/1518701.1518787

Digital Library

[25]

Jia Wang, Owen Leach, and Robert W Lindeman. 2013. DIY World Builder: An immersive level-editing system. In 2013 IEEE Symposium on 3D User Interfaces (3DUI), 195–196. https://doi.org/10.1109/3DUI.2013.6550245

[26]

Mike Wu, Shen Chia, Kathy Ryall, Clifton Forlines, and Ravin Balakrishnan. 2006. Gesture registration, relaxation, and reuse for multi-point direct-touch surfaces. First IEEE International Workshop on Horizontal Interactive Human-Computer Systems, 185–192. https://doi.org/10.1109/tabletop.2006.19

Digital Library

[27]

Robert Zeleznik, Andrew Bragdon, Ferdi Adeputra, and Hsu-Sheng Ko. 2010. Hands-on math: a page-based multi-touch and pen desktop for technical work and problem solving. Proceedings of the 23nd annual ACM symposium on User interface software and technology, 17–26. https://doi.org/10.1145/1866029.1866035

Digital Library

[28]

Unity. https://unity3d.com

[29]

Unreal Engine VR Mode. https://docs.unrealengine.com/4.27/en-US/BuildingWorlds/VRMode

[30]

Minecraft in VR. https://www.minecraft.net/en-us/vr

Cited By

Matulic FKashima TBeker DSuzuo DFujiwara HVogel D(2024)Above-Screen Fingertip Tracking and Hand Representation for Precise Touch Input with a Phone in Virtual RealityProceedings of the 50th Graphics Interface Conference10.1145/3670947.3670961(1-15)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3670947.3670961
Fender AKari M(2024)OptiBasePen: Mobile Base+Pen Input on Passive Surfaces by Sensing Relative Base Motion Plus Close-Range Pen PositionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676467(1-9)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676467
Kim TNisar HLindgren RZhang JTang XLira MTalhan A(2024)Beyond the Screen: Gestural Perspective-Taking with a Biochemistry SimulationExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650816(1-6)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650816
Show More Cited By

Recommendations

Pen + Mid-Air: An Exploration of Mid-Air Gestures to Complement Pen Input on Tablets
NordiCHI '16: Proceedings of the 9th Nordic Conference on Human-Computer Interaction

In this paper, we report on a series of studies, exploring the potential of pen and mid-air input on tablets. We describe a field study with an early prototype of a drawing application and follow-up inquiries, such as the development and comparison of ...
Eliciting Mid-Air Gestures for Wall-Display Interaction
NordiCHI '16: Proceedings of the 9th Nordic Conference on Human-Computer Interaction

Freehand mid-air gestures are a promising input method for interacting with wall displays. However, work on mid-air gestures for wall-display interaction has mainly explored what is technically possible, which might not result in gestures that users ...
Towards document engineering on pen and touch-operated interactive tabletops
UIST Adjunct Proceedings '12: Adjunct proceedings of the 25th annual ACM symposium on User interface software and technology

Touch interfaces have now become mainstream thanks to modern smartphones and tablets. However, there are still very few "productivity" applications, i.e. tools that support mundane but essential work, especially for large interactive surfaces such as ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems

April 2022

3066 pages

ISBN:9781450391566

DOI:10.1145/3491101

Editors:
Simone Barbosa
PUC-Rio, Brazil
,
Cliff Lampe
University of Michigan, USA
,
Caroline Appert
Université Paris-Saclay, France
,
David A. Shamma
Toyota Research Institute, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster
Research
Refereed limited

Conference

CHI '22

Sponsor:

SIGCHI

CHI '22: CHI Conference on Human Factors in Computing Systems

April 29 - May 5, 2022

LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
271
Total Downloads

Downloads (Last 12 months)67
Downloads (Last 6 weeks)11

Reflects downloads up to 28 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Matulic FKashima TBeker DSuzuo DFujiwara HVogel D(2024)Above-Screen Fingertip Tracking and Hand Representation for Precise Touch Input with a Phone in Virtual RealityProceedings of the 50th Graphics Interface Conference10.1145/3670947.3670961(1-15)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3670947.3670961
Fender AKari M(2024)OptiBasePen: Mobile Base+Pen Input on Passive Surfaces by Sensing Relative Base Motion Plus Close-Range Pen PositionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676467(1-9)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676467
Kim TNisar HLindgren RZhang JTang XLira MTalhan A(2024)Beyond the Screen: Gestural Perspective-Taking with a Biochemistry SimulationExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650816(1-6)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650816
Matulic FKashima TBeker DSuzuo DFujiwara HVogel D(2023)Above-Screen Fingertip Tracking with a Phone in Virtual RealityExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585728(1-7)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585728
Bhuyan RKenny S(2023)Continuous Authentication of Tablet Users Using Raw Swipe Characteristics: Tree Based ApproachesIntelligent Systems Design and Applications10.1007/978-3-031-35507-3_29(299-310)Online publication date: 3-Jun-2023
https://doi.org/10.1007/978-3-031-35507-3_29

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten