research-article

Phonetroller: Visual Representations of Fingers for Precise Touch Input with Mobile Phones in VR

Authors:

Fabrice Matulic,

Aditya Ganeshan,

Hiroshi Fujiwara,

Daniel VogelAuthors Info & Claims

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Article No.: 129, Pages 1 - 13

https://doi.org/10.1145/3411764.3445583

Published: 07 May 2021 Publication History

Abstract

Smartphone touch screens are potentially attractive for interaction in virtual reality (VR). However, the user cannot see the phone or their hands in a fully immersive VR setting, impeding their ability for precise touch input. We propose mounting a mirror above the phone screen such that the front-facing camera captures the thumbs on or near the screen. This enables the creation of semi-transparent overlays of thumb shadows and inference of fingertip hover points with deep learning, which help the user aim for targets on the phone. A study compares the effect of visual feedback on touch precision in a controlled task and qualitatively evaluates three example applications demonstrating the potential of the technique. The results show that the enabled style of feedback is effective for thumb-size targets, and that the VR experience can be enriched by using smartphones as VR controllers supporting precise touch input.

Supplementary Material

VTT File (3411764.3445583_videofigurecaptions.vtt)

Download
6.50 KB

Supplementary Materials (3411764.3445583_supplementalmaterials.zip)

Download
507.79 KB

MP4 File (3411764.3445583_videofigure.mp4)

Supplementary video

Download
64.59 MB

References

[1]

Amal Benzina, Arindam Dey, Marcus Toennis, and Gudrun Klinker. 2012. Empirical Evaluation of Mapping Functions for Navigation in Virtual Reality Using Phones with Integrated Sensors. In Proceedings of the 10th Asia Pacific Conference on Computer Human Interaction (Matsue-city, Shimane, Japan) (APCHI ’12). Association for Computing Machinery, New York, NY, USA, 149–158. https://doi.org/10.1145/2350046.2350078

Digital Library

[2]

Verena Biener, Daniel Schneider, Travis Gesslein, Alexander Otte, Bastian Kuth, Per Ola Kristensson, Eyal Ofek, Michel Pahud, and Jens Grubert. 2020. Breaking the Screen: Interaction Across Touchscreen Boundaries in Virtual Reality for Mobile Knowledge Workers. IEEE Transactions on Visualization and Computer Graphics 26, 12(2020), 3490–3502.

[3]

Costas Boletsis and Stian Kongsvik. 2019. Controller-based Text-input Techniques for Virtual Reality: An Empirical Comparison. International Journal of Virtual Reality (IJVR) 19, 3 (Oct. 2019), 2–15. https://doi.org/10.20870/IJVR.2019.19.3.2917

[4]

Sabah Boustila, Thomas Guégan, Kazuki Takashima, and Yoshifumi Kitamura. 2019. Text Typing in VR Using Smartphones Touchscreen and HMD. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, New York, NY, USA, 860–861. https://doi.org/10.1109/VR.2019.8798238

[5]

Sibo Chen, Junce Wang, Santiago Guerra, Neha Mittal, and Soravis Prakkamakul. 2019. Exploring Word-Gesture Text Entry Techniques in Virtual Reality. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3290607.3312762

Digital Library

[6]

Xiang ’Anthony’ Chen, Julia Schwarz, Chris Harrison, Jennifer Mankoff, and Scott E. Hudson. 2014. Air+touch: Interweaving Touch & in-Air Gestures. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA) (UIST ’14). Association for Computing Machinery, New York, NY, USA, 519–525. https://doi.org/10.1145/2642918.2647392

Digital Library

[7]

Lawrence D. Cutler, Bernd Fröhlich, and Pat Hanrahan. 1997. Two-Handed Direct Manipulation on the Responsive Workbench. In Proceedings of the 1997 Symposium on Interactive 3D Graphics (Providence, Rhode Island, USA) (I3D ’97). Association for Computing Machinery, New York, NY, USA, 107–114. https://doi.org/10.1145/253284.253315

Digital Library

[8]

Tobias Drey, Jan Gugenheimer, Julian Karlbauer, Maximilian Milo, and Enrico Rukzio. 2020. VRSketchIn: Exploring the Design Space of Pen and Tablet Interaction for 3D Sketching in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376628

Digital Library

[9]

Wei Fang, Lianyu Zheng, and Xiangyong Wu. 2017. Multi-sensor based real-time 6-DoF pose tracking for wearable augmented reality. Computers in Industry 92-93 (2017), 91 – 103. https://doi.org/10.1016/j.compind.2017.06.002

[10]

Travis Gesslein, Verena Biener, Philipp Gagel, Daniel Schneider, Per Ola Kristensson, Eyal Ofek, Michel Pahud, and Jens Grubert. 2020. Pen-based Interaction with Spreadsheets in Mobile Virtual Reality. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, New York, NY, USA, 361–373.

[11]

J. Grubert, L. Witzani, E. Ofek, M. Pahud, M. Kranz, and P. O. Kristensson. 2018. Effects of Hand Representations for Typing in Virtual Reality. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, New York, NY, USA, 151–158.

[12]

Jan Gugenheimer, David Dobbelstein, Christian Winkler, Gabriel Haas, and Enrico Rukzio. 2016. FaceTouch: Enabling Touch Interaction in Display Fixed UIs for Mobile Virtual Reality. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST ’16). Association for Computing Machinery, New York, NY, USA, 49–60. https://doi.org/10.1145/2984511.2984576

Digital Library

[13]

Yves Guiard. 1987. Asymmetric division of labor in human skilled bimanual action: The kinematic chain as a model. Journal of motor behavior 19, 4 (1987), 486–517.

[14]

Shangchen Han, Beibei Liu, Randi Cabezas, Christopher D. Twigg, Peizhao Zhang, Jeff Petkau, Tsz-Ho Yu, Chun-Jung Tai, Muzaffer Akbay, Zheng Wang, Asaf Nitzan, Gang Dong, Yuting Ye, Lingling Tao, Chengde Wan, and Robert Wang. 2020. MEgATrack: Monochrome Egocentric Articulated Hand-Tracking for Virtual Reality. ACM Trans. Graph. 39, 4, Article 87 (July 2020), 13 pages. https://doi.org/10.1145/3386569.3392452

Digital Library

[15]

Mohamed Abudulaziz Ali Haseeb and Ramviyas Parasuraman. 2017. Wisture: RNN-based Learning of Wireless Signals for Gesture Recognition in Unmodified Smartphones. arxiv:1707.08569 [cs.HC]

[16]

Keisuke Hattori and Tatsunori Hirai. 2020. Inside-out Tracking Controller for VR/AR HMD Using Image Recognition with Smartphones. In ACM SIGGRAPH 2020 Posters (Virtual Event, USA) (SIGGRAPH ’20). Association for Computing Machinery, New York, NY, USA, Article 23, 2 pages. https://doi.org/10.1145/3388770.3407430

Digital Library

[17]

Ken Hinckley, Seongkook Heo, Michel Pahud, Christian Holz, Hrvoje Benko, Abigail Sellen, Richard Banks, Kenton O’Hara, Gavin Smyth, and William Buxton. 2016. Pre-Touch Sensing for Mobile Interaction. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 2869–2881. https://doi.org/10.1145/2858036.2858095

Digital Library

[18]

Ken Hinckley, Koji Yatani, Michel Pahud, Nicole Coddington, Jenny Rodenhouse, Andy Wilson, Hrvoje Benko, and Bill Buxton. 2010. Pen+ touch= new tools. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. Association for Computing Machinery, New York, NY, USA, 27–36.

Digital Library

[19]

Christian Holz and Patrick Baudisch. 2011. Understanding Touch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 2501–2510. https://doi.org/10.1145/1978942.1979308

Digital Library

[20]

H. Jiang and D. Weng. 2020. HiPad: Text entry for Head-Mounted Displays Using Circular Touchpad. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, New York, NY, USA, 692–703.

[21]

Ricardo Jota, Albert Ng, Paul Dietz, and Daniel Wigdor. 2013. How Fast is Fast Enough? A Study of the Effects of Latency in Direct-Touch Pointing Tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA, 2291–2300. https://doi.org/10.1145/2470654.2481317

Digital Library

[22]

Jin Woo Jung, Kwang Seog Kim, Jun Ho Shin, Yu Jin Kwon, Jae Ha Hwang, and Sam Yong Lee. 2015. Fingernail configuration. Archives of plastic surgery 42, 6 (2015), 753.

[23]

Y. R. Kim and G. J. Kim. 2017. HoVR-Type: Smartphone as a typing interface in VR using hovering. In 2017 IEEE International Conference on Consumer Electronics (ICCE). IEEE, New York, NY, USA, 200–203.

[24]

Pascal Knierim, Valentin Schwind, Anna Maria Feit, Florian Nieuwenhuizen, and Niels Henze. 2018. Physical Keyboards in Virtual Reality: Analysis of Typing Performance and Effects of Avatar Hands. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–9. https://doi.org/10.1145/3173574.3173919

Digital Library

[25]

Jihyun Lee, Byungmoon Kim, Bongwon Suh, and Eunyee Koh. 2016. Exploring the Front Touch Interface for Virtual Reality Headsets. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (San Jose, California, USA) (CHI EA ’16). Association for Computing Machinery, New York, NY, USA, 2585–2591. https://doi.org/10.1145/2851581.2892344

Digital Library

[26]

Florian Lehmann and Michael Kipp. 2018. How to Hold Your Phone When Tapping: A Comparative Study of Performance, Precision, and Errors. In Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces (Tokyo, Japan) (ISS ’18). Association for Computing Machinery, New York, NY, USA, 115–127. https://doi.org/10.1145/3279778.3279791

Digital Library

[27]

Hai-Ning Liang, Yuwei Shi, Feiyu Lu, Jizhou Yang, and Konstantinos Papangelis. 2016. VRMController: An Input Device for Navigation Activities in Virtual Reality Environments. In Proceedings of the 15th ACM SIGGRAPH Conference on Virtual-Reality Continuum and Its Applications in Industry - Volume 1(Zhuhai, China) (VRCAI ’16). Association for Computing Machinery, New York, NY, USA, 455–460. https://doi.org/10.1145/3013971.3014005

Digital Library

[28]

Jaime Lien, Nicholas Gillian, M. Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous Gesture Sensing with Millimeter Wave Radar. ACM Trans. Graph. 35, 4, Article 142 (July 2016), 19 pages. https://doi.org/10.1145/2897824.2925953

Digital Library

[29]

N. G. Lipari and C. W. Borst. 2015. Handymenu: Integrating menu selection into a multifunction smartphone-based VR controller. In 2015 IEEE Symposium on 3D User Interfaces (3DUI). IEEE, New York, NY, USA, 129–132.

[30]

Logitech. 2020. Logitech VR Ink Pilot Edition. https://www.logitech.com/en-roeu/promo/vr-ink.html. Accessed: 2020-09-15.

[31]

Shahzad Malik and Joe Laszlo. 2004. Visual Touchpad: A Two-Handed Gestural Input Device. In Proceedings of the 6th International Conference on Multimodal Interfaces (State College, PA, USA) (ICMI ’04). Association for Computing Machinery, New York, NY, USA, 289–296. https://doi.org/10.1145/1027933.1027980

Digital Library

[32]

Jess McIntosh, Paul Strohmeier, Jarrod Knibbe, Sebastian Boring, and Kasper Hornbæk. 2019. Magnetips: Combining Fingertip Tracking and Haptic Feedback for Around-Device Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300638

Digital Library

[33]

Tim Menzner, Travis Gesslein, Alexander Otte, and Jens Grubert. 2020. Above Surface Interaction for Multiscale Navigation in Mobile Virtual Reality. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, New York, NY, USA, 372–381.

[34]

Alexandre Millette and Michael J McGuffin. 2016. DualCAD: integrating augmented reality with a desktop GUI and smartphone interaction. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). IEEE, New York, NY, USA, 21–26.

[35]

Peter Mohr, Markus Tatzgern, Tobias Langlotz, Andreas Lang, Dieter Schmalstieg, and Denis Kalkofen. 2019. TrackCap: Enabling Smartphones for 3D Interaction on Mobile Head-Mounted Displays. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300815

Digital Library

[36]

Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. FingerIO: Using Active Sonar for Fine-Grained Finger Tracking. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 1515–1525. https://doi.org/10.1145/2858036.2858580

Digital Library

[37]

Russell Owen, Gordon Kurtenbach, George Fitzmaurice, Thomas Baudel, and Bill Buxton. 2005. When It Gets More Difficult, Use Both Hands: Exploring Bimanual Curve Manipulation. In Proceedings of Graphics Interface 2005 (Victoria, British Columbia) (GI ’05). Canadian Human-Computer Communications Society, Waterloo, CAN, 17–24.

[38]

Keunwoo Park, Sunbum Kim, Youngwoo Yoon, Tae-Kyun Kim, and Geehyuk Lee. 2020. DeepFisheye: Near-Surface Multi-Finger Tracking Technology Using Fisheye Camera. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology(Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 1132–1146. https://doi.org/10.1145/3379337.3415818

Digital Library

[39]

Ken Pfeuffer, Benedikt Mayer, Diako Mardanbegi, and Hans Gellersen. 2017. Gaze + Pinch Interaction in Virtual Reality. In Proceedings of the 5th Symposium on Spatial User Interaction (Brighton, United Kingdom) (SUI ’17). Association for Computing Machinery, New York, NY, USA, 99–108. https://doi.org/10.1145/3131277.3132180

Digital Library

[40]

Alwin Poulose and Dong Seog Han. 2019. Hybrid Indoor Localization Using IMU Sensors and Smartphone Camera. Sensors 19, 23 (2019), 5084.

[41]

Jing Qian, Jiaju Ma, Xiangyu Li, Benjamin Attal, Haoming Lai, James Tompkin, John F. Hughes, and Jeff Huang. 2019. Portal-Ble: Intuitive Free-Hand Manipulation in Unbounded Smartphone-Based Augmented Reality. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology(UIST ’19). Association for Computing Machinery, New York, NY, USA, 133–145. https://doi.org/10.1145/3332165.3347904

Digital Library

[42]

RiftCat. 2020. VRidge Controller. https://play.google.com/store/apps/details?id=com.riftcat.vridgecontroller. Accessed: 2020-08-21.

[43]

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (Eds.). Springer International Publishing, Cham, 234–241.

[44]

Samsung. 2013. How does Air view work?https://www.samsung.com/global/galaxy/what-is/air-view/. Accessed: 2020-08-21.

[45]

Shizof. 2019. Touch Gesture VR. https://www.nexusmods.com/skyrimspecialedition/mods/27815. Accessed: 2020-08-21.

[46]

Shaishav Siddhpuria, Sylvain Malacria, Mathieu Nancel, and Edward Lank. 2018. Pointing at a Distance with Everyday Smart Devices. Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3173574.3173747

Digital Library

[47]

José Somolinos. 2017. How to design common/basic interactions in VR. https://blog.prototypr.io/how-to-design-common-basic-interactions-in-vr-f958cf160cfc?gi=e3daf7483cd1. Accessed: 2020-08-21.

[48]

Jeongmin Son, Sunggeun Ahn, Sunbum Kim, and Geehyuk Lee. 2019. Improving Two-Thumb Touchpad Typing in Virtual Reality. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3290607.3312926

Digital Library

[49]

Jie Song, Gábor Sörös, Fabrizio Pece, Sean Ryan Fanello, Shahram Izadi, Cem Keskin, and Otmar Hilliges. 2014. In-Air Gestures around Unmodified Mobile Devices. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA) (UIST ’14). Association for Computing Machinery, New York, NY, USA, 319–329. https://doi.org/10.1145/2642918.2647373

Digital Library

[50]

Jonathan Dyssel Stets, Yongbin Sun, Wiley Corning, and Scott W. Greenwald. 2017. Visualization and Labeling of Point Clouds in Virtual Reality. In SIGGRAPH Asia 2017 Posters(Bangkok, Thailand) (SA ’17). Association for Computing Machinery, New York, NY, USA, Article 31, 2 pages. https://doi.org/10.1145/3145690.3145729

Digital Library

[51]

Hemant Bhaskar Surale, Aakar Gupta, Mark Hancock, and Daniel Vogel. 2019. TabletInVR: Exploring the Design Space for Using a Multi-Touch Tablet in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300243

Digital Library

[52]

Jonathan Taylor, Lucas Bordeaux, Thomas Cashman, Bob Corish, Cem Keskin, Toby Sharp, Eduardo Soto, David Sweeney, Julien Valentin, Benjamin Luff, Arran Topalian, Erroll Wood, Sameh Khamis, Pushmeet Kohli, Shahram Izadi, Richard Banks, Andrew Fitzgibbon, and Jamie Shotton. 2016. Efficient and Precise Interactive Hand Tracking through Joint, Continuous Optimization of Pose and Correspondences. ACM Trans. Graph. 35, 4, Article 143 (July 2016), 12 pages. https://doi.org/10.1145/2897824.2925965

Digital Library

[53]

J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bregler. 2015. Efficient object localization using Convolutional Networks. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, New York, NY, USA, 648–656.

[54]

Rishi Vanukuru, Amarnath Murugan, and Jayesh Pillai. 2020. Dual Phone AR: Using a Second Phone as a Controller for Mobile Augmented Reality. In 26th ACM Symposium on Virtual Reality Software and Technology (Virtual Event, USA) (UIST ’20 Adjunct). Association for Computing Machinery, New York, NY, USA, 117–119. https://doi.org/10.1145/3379350.3416139

Digital Library

[55]

Spyros Vosinakis and Panayiotis Koutsabasis. 2018. Evaluation of visual feedback techniques for virtual grasping with bare hands using Leap Motion and Oculus Rift. Virtual Reality 22, 1 (2018), 47–62.

Digital Library

[56]

James Walker, Bochao Li, Keith Vertanen, and Scott Kuhl. 2017. Efficient Typing on a Visually Occluded Physical Keyboard. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 5457–5461. https://doi.org/10.1145/3025453.3025783

Digital Library

[57]

Wei Wang, Alex X. Liu, and Ke Sun. 2016. Device-Free Gesture Tracking Using Acoustic Signals. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking (New York City, New York) (MobiCom ’16). Association for Computing Machinery, New York, NY, USA, 82–94. https://doi.org/10.1145/2973750.2973764

Digital Library

[58]

Daniel Wigdor, Clifton Forlines, Patrick Baudisch, John Barnwell, and Chia Shen. 2007. Lucid Touch: A See-through Mobile Device. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (Newport, Rhode Island, USA) (UIST ’07). Association for Computing Machinery, New York, NY, USA, 269–278. https://doi.org/10.1145/1294211.1294259

Digital Library

[59]

Wikipedia. 2012. Sony Xperia sola - Wikipedia. https://en.wikipedia.org/wiki/Sony_Xperia_sola. Accessed: 2020-08-21.

[60]

Florian Wirth, Jannik Quchl, Jeffrey Ota, and Christoph Stiller. 2019. PointAtMe: Efficient 3D Point Cloud Labeling in Virtual Reality. In 2019 IEEE Intelligent Vehicles Symposium (IV). IEEE, New York, NY, USA, 1693–1698.

Digital Library

[61]

Pui Chung Wong, Hongbo Fu, and Kening Zhu. 2016. Back-Mirror: Back-of-Device One-Handed Interaction on Smartphones. In SIGGRAPH ASIA 2016 Mobile Graphics and Interactive Applications (Macau) (SA ’16). Association for Computing Machinery, New York, NY, USA, Article 13, 2 pages. https://doi.org/10.1145/2999508.2999512

Digital Library

[62]

Peter Wozniak, Oliver Vauderwange, Avikarsha Mandal, Nicolas Javahiraly, and Dan Curticapean. 2016. Possible applications of the LEAP motion controller for more interactive simulated experiments in augmented or virtual reality. In Optics Education and Outreach IV, G. Groot Gregory (Ed.), Vol. 9946. International Society for Optics and Photonics, SPIE, Bellingham WA 98227-0010 USA, 234 – 245. https://doi.org/10.1117/12.2237673

[63]

Xing-Dong Yang, Khalad Hasan, Neil Bruce, and Pourang Irani. 2013. Surround-See: Enabling Peripheral Vision on Smartphones during Active Use. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology(St. Andrews, Scotland, United Kingdom) (UIST ’13). Association for Computing Machinery, New York, NY, USA, 291–300. https://doi.org/10.1145/2501988.2502049

Digital Library

[64]

Chungkuk Yoo, Inseok Hwang, Eric Rozner, Yu Gu, and Robert F. Dickerson. 2016. SymmetriSense: Enabling Near-Surface Interactivity on Glossy Surfaces Using a Single Commodity Smartphone. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 5126–5137. https://doi.org/10.1145/2858036.2858286

Digital Library

[65]

Shahrouz Yousefi, Mhretab Kidane, Yeray Delgado, Julio Chana, and Nico Reski. 2016. 3D gesture-based interaction for immersive experience in mobile VR. In 2016 23rd International Conference on Pattern Recognition (ICPR). IEEE, New York, NY, USA, 2121–2126.

[66]

Chun Yu, Xiaoying Wei, Shubh Vachher, Yue Qin, Chen Liang, Yueting Weng, Yizheng Gu, and Yuanchun Shi. 2019. HandSee: Enabling Full Hand Interaction on Smartphone with Front Camera-Based Stereo Vision. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300935

Digital Library

[67]

Sangki Yun, Yi-Chao Chen, Huihuang Zheng, Lili Qiu, and Wenguang Mao. 2017. Strata: Fine-Grained Acoustic-Based Device-Free Tracking. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services(Niagara Falls, New York, USA) (MobiSys ’17). Association for Computing Machinery, New York, NY, USA, 15–28. https://doi.org/10.1145/3081333.3081356

Digital Library

[68]

Fengyuan Zhu and Tovi Grossman. 2020. BISHARE: Exploring Bidirectional Interactions Between Smartphones and Head-Mounted Augmented Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376233

Digital Library

Cited By

Matulic FKashima TBeker DSuzuo DFujiwara HVogel D(2024)Above-Screen Fingertip Tracking and Hand Representation for Precise Touch Input with a Phone in Virtual RealityProceedings of the 50th Graphics Interface Conference10.1145/3670947.3670961(1-15)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3670947.3670961
Zhu FSousa MSidenmark LGrossman T(2024)PhoneInVR: An Evaluation of Spatial Anchoring and Interaction Techniques for Smartphone Usage in Virtual RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642582(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642582
Gonzalez EPatel KAhuja KGonzalez-Franco M(2024)XDTK: A Cross-Device Toolkit for Input & Interaction in XR2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00092(467-470)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00092
Show More Cited By

Index Terms

Phonetroller: Visual Representations of Fingers for Precise Touch Input with Mobile Phones in VR
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
    2. Interaction paradigms
      1. Mixed / augmented reality
      2. Virtual reality

Index terms have been assigned to the content through auto-classification.

Recommendations

Air+touch: interweaving touch & in-air gestures
UIST '14: Proceedings of the 27th annual ACM symposium on User interface software and technology

We present Air+Touch, a new class of interactions that interweave touch events with in-air gestures, offering a unified input modality with expressiveness greater than each input modality alone. We demonstrate how air and touch are highly complementary: ...
A mobile see-through 3D display with front- and back-touch
NordiCHI '14: Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational

Touch screens are currently the dominant technology for facilitating input and output for mobile devices. Several directions to extend the possibilities of current touch screen technologies have been explored. In this demonstration we showcase a ...
A study of direction's impact on single-handed thumb interaction with touch-screen mobile phones
CHI EA '14: CHI '14 Extended Abstracts on Human Factors in Computing Systems

This study is aimed to investigate how the performance of thumb interaction with touch-screen mobile devices via double tap and swipe varies with movement directions of the thumb. A target selection game was used in an empirical study to evaluate users' ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

May 2021

10862 pages

ISBN:9781450380966

DOI:10.1145/3411764

General Chairs:
Yoshifumi Kitamura
Tohoku University, Japan
,
Aaron Quigley
University of New South Wales, Australia
,
Program Chairs:
Katherine Isbister
University of California Santa Cruz, USA
,
Takeo Igarashi
The University of Tokyo, Japan
,
Publications Chairs:
Pernille Bjørn
University of Copenhagen, Denmark
,
Steven Drucker
Microsoft Research, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '21

Sponsor:

SIGCHI

CHI '21: CHI Conference on Human Factors in Computing Systems

May 8 - 13, 2021

Yokohama, Japan

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
1,435
Total Downloads

Downloads (Last 12 months)282
Downloads (Last 6 weeks)20

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Matulic FKashima TBeker DSuzuo DFujiwara HVogel D(2024)Above-Screen Fingertip Tracking and Hand Representation for Precise Touch Input with a Phone in Virtual RealityProceedings of the 50th Graphics Interface Conference10.1145/3670947.3670961(1-15)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3670947.3670961
Zhu FSousa MSidenmark LGrossman T(2024)PhoneInVR: An Evaluation of Spatial Anchoring and Interaction Techniques for Smartphone Usage in Virtual RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642582(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642582
Gonzalez EPatel KAhuja KGonzalez-Franco M(2024)XDTK: A Cross-Device Toolkit for Input & Interaction in XR2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00092(467-470)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00092
Wu JWang ZWang LDuan YLi J(2024)FanPad: A Fan Layout Touchpad Keyboard for Text Entry in VR2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00045(222-232)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VR58804.2024.00045
Mishra ASingh HParnami AShukla J(2024)MobiTangibles: Enabling Physical Manipulation Experiences of Virtual Precision Hand-Held Tools' Miniature Control in VRIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345619130:11(7321-7331)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456191
Tian YZheng YZhao SMa XWang Y(2023)Balancing Accuracy and Speed in Gaze-Touch Grid Menu Selection in AR via Mapping Sub-Menus to a Hand-Held DeviceSensors10.3390/s2323958723:23(9587)Online publication date: 3-Dec-2023
https://doi.org/10.3390/s23239587
Gomi RTakashima KOnishi YFujita KKitamura Y(2023)UbiSurface: A Robotic Touch Surface for Supporting Mid-air Planar Interactions in Room-Scale VRProceedings of the ACM on Human-Computer Interaction10.1145/36264797:ISS(376-397)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1145/3626479
Lee JJin TLee SLee SBae S(2023)Stereoscopic Viewing and Monoscopic Touching: Selecting Distant Objects in VR Through a Mobile DeviceProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606809(1-7)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606809
Liang CWang XLi ZHsia CFan MYu CShi Y(2023)ShadowTouch: Enabling Free-Form Touch-Based Hand-to-Surface Interaction with Wrist-Mounted Illuminant by Shadow ProjectionProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606785(1-14)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606785
Gil HOakley I(2023)ThumbAirProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35694746:4(1-30)Online publication date: 11-Jan-2023
https://dl.acm.org/doi/10.1145/3569474
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents