research-article

Open access

ShadowTouch: Enabling Free-Form Touch-Based Hand-to-Surface Interaction with Wrist-Mounted Illuminant by Shadow Projection

Authors:

Yuanchun ShiAuthors Info & Claims

UIST '23: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

Article No.: 27, Pages 1 - 14

https://doi.org/10.1145/3586183.3606785

Published: 29 October 2023 Publication History

All formats PDF

Abstract

We present ShadowTouch, a novel sensing method to recognize the subtle hand-to-surface touch state for independent fingers based on optical auxiliary. ShadowTouch mounts a forward-facing light source on the user’s wrist to construct shadows on the surface in front of the fingers when the corresponding fingers are close to the surface. With such an optical design, the subtle vertical movements of near-surface fingers are magnified and turned to shadow features cast on the surface, which are recognizable for computer vision algorithms. To efficiently recognize the touch state of each finger, we devised a two-stage CNN-based algorithm that first extracted all the fingertip regions from each frame and then classified the touch state of each region from the cropped consecutive frames. Evaluations showed our touch state detection algorithm achieved a recognition accuracy of 99.1% and an F-1 score of 96.8% in the leave-one-out cross-user evaluation setting. We further outlined the hand-to-surface interaction space enabled by ShadowTouch’s sensing capability from the aspects of touch-based interaction, stroke-based interaction, and out-of-surface information and developed four application prototypes to showcase ShadowTouch’s interaction potential. The usability evaluation study showed the advantages of ShadowTouch over threshold-based techniques in aspects of lower mental demand, lower effort, lower frustration, more willing to use, easier to use, better integrity, and higher confidence.

Supplementary Material

ZIP File (3606785.zip)

Supplemental File

Download
41.06 MB

References

[1]

Ankur Agarwal, Shahram Izadi, Manmohan Chandraker, and Andrew Blake. 2007. High Precision Multi-touch Sensing on Surfaces using Overhead Cameras. In Second Annual IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP’07). 197–200. https://doi.org/10.1109/TABLETOP.2007.29

[2]

Rahul Arora, Rubaiat Habib Kazi, Tovi Grossman, George Fitzmaurice, and Karan Singh. 2018. SymbiosisSketch: Combining 2D & 3D Sketching for Designing Detailed 3D Objects in Situ. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3173574.3173759

Digital Library

[3]

Rahul Arora, Rubaiat Habib Kazi, Fraser Anderson, Tovi Grossman, Karan Singh, and George Fitzmaurice. 2017. Experimental Evaluation of Sketching on Surfaces in VR. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 5643–5654. https://doi.org/10.1145/3025453.3025474

Digital Library

[4]

Aaron Bangor, Philip T Kortum, and James T Miller. 2008. An empirical evaluation of the system usability scale. Intl. Journal of Human–Computer Interaction 24, 6 (2008), 574–594.

[5]

Samarth Brahmbhatt, Cusuh Ham, Charles C. Kemp, and James Hays. 2019. ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 8701–8711. https://doi.org/10.1109/CVPR.2019.00891

[6]

Liwei Chan, Rong-Hao Liang, Ming-Chang Tsai, Kai-Yin Cheng, Chao-Huai Su, Mike Y. Chen, Wen-Huang Cheng, and Bing-Yu Chen. 2013. FingerPad: Private and Subtle Interaction Using Fingertips. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST ’13). Association for Computing Machinery, New York, NY, USA, 255–260. https://doi.org/10.1145/2501988.2502016

Digital Library

[7]

Pak-Kiu Chung, Bing Fang, and Francis Quek. 2008. Mirrortrack-a vision based multi-touch system for glossy display surfaces. (2008).

[8]

Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. 2014. Training deep neural networks with low precision multiplications. arxiv:1412.7024 [cs.LG]

[9]

Tim Dettmers. 2015. 8-Bit Approximations for Parallelism in Deep Learning. arxiv:1511.04561 [cs.NE]

[10]

Tobias Drey, Jan Gugenheimer, Julian Karlbauer, Maximilian Milo, and Enrico Rukzio. 2020. VRSketchIn: Exploring the Design Space of Pen and Tablet Interaction for 3D Sketching in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376628

Digital Library

[11]

Cathy Mengying Fang and Chris Harrison. 2021. Retargeted Self-Haptics for Increased Immersion in VR without Instrumentation. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 1109–1121. https://doi.org/10.1145/3472749.3474810

Digital Library

[12]

Patrick Grady, Chengcheng Tang, Samarth Brahmbhatt, Christopher D Twigg, Chengde Wan, James Hays, and Charles C Kemp. 2022. PressureVision: Estimating Hand Pressure from a Single RGB Image. arXiv preprint arXiv:2203.10385 (2022).

[13]

Patrick Grady, Chengcheng Tang, Samarth Brahmbhatt, Christopher D. Twigg, Chengde Wan, James Hays, and Charles C. Kemp. 2022. PressureVision: Estimating Hand Pressure from a Single RGB Image. In Computer Vision – ECCV 2022, Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (Eds.). Springer Nature Switzerland, Cham, 328–345.

Digital Library

[14]

Yizheng Gu, Chun Yu, Zhipeng Li, Weiqi Li, Shuchang Xu, Xiaoying Wei, and Yuanchun Shi. 2019. Accurate and Low-Latency Sensing of Touch Contact on Any Surface with Finger-Worn IMU Sensor. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 1059–1070. https://doi.org/10.1145/3332165.3347947

Digital Library

[15]

Yizheng Gu, Chun Yu, Zhipeng Li, Zhaoheng Li, Xiaoying Wei, and Yuanchun Shi. 2020. QwertyRing: Text Entry on Physical Surfaces Using a Ring. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 4, Article 128 (dec 2020), 29 pages. https://doi.org/10.1145/3432204

Digital Library

[16]

Hung-Jui Guo and Balakrishnan Prabhakaran. 2022. HoloLens 2 Technical Evaluation as Mixed Reality Guide. https://doi.org/10.48550/ARXIV.2207.09554

[17]

Sean Gustafson, Christian Holz, and Patrick Baudisch. 2011. Imaginary Phone: Learning Imaginary Interfaces by Transferring Spatial Memory from a Familiar Device. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (Santa Barbara, California, USA) (UIST ’11). Association for Computing Machinery, New York, NY, USA, 283–292. https://doi.org/10.1145/2047196.2047233

Digital Library

[18]

Shangchen Han, Beibei Liu, Randi Cabezas, Christopher D. Twigg, Peizhao Zhang, Jeff Petkau, Tsz-Ho Yu, Chun-Jung Tai, Muzaffer Akbay, Zheng Wang, Asaf Nitzan, Gang Dong, Yuting Ye, Lingling Tao, Chengde Wan, and Robert Wang. 2020. MEgATrack: Monochrome Egocentric Articulated Hand-Tracking for Virtual Reality. ACM Trans. Graph. 39, 4, Article 87 (jul 2020), 13 pages. https://doi.org/10.1145/3386569.3392452

Digital Library

[19]

Chris Harrison, Hrvoje Benko, and Andrew D. Wilson. 2011. OmniTouch: Wearable Multitouch Interaction Everywhere. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (Santa Barbara, California, USA) (UIST ’11). Association for Computing Machinery, New York, NY, USA, 441–450. https://doi.org/10.1145/2047196.2047255

Digital Library

[20]

Chris Harrison and Scott E. Hudson. 2008. Scratch Input: Creating Large, Inexpensive, Unpowered and Mobile Finger Input Surfaces. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (Monterey, CA, USA) (UIST ’08). Association for Computing Machinery, New York, NY, USA, 205–208. https://doi.org/10.1145/1449715.1449747

Digital Library

[21]

Chris Harrison, Desney Tan, and Dan Morris. 2010. Skinput: Appropriating the Body as an Input Surface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). Association for Computing Machinery, New York, NY, USA, 453–462. https://doi.org/10.1145/1753326.1753394

Digital Library

[22]

Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. 2019. Searching for MobileNetV3. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).

[23]

Tzu Hsuan Hsia, Shogo Okamoto, Yasuhiro Akiyama, and Yoji Yamada. 2021. HumTouch: Localization of Touch on Semi-Conductive Surfaces by Sensing Human Body Antenna Signal. Sensors 21, 3 (2021). https://doi.org/10.3390/s21030859

[24]

Yuhan Hu, Sara Maria Bejarano, and Guy Hoffman. 2020. ShadowSense: Detecting Human Touch in a Social Robot Using Shadow Image Classification. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 4, Article 132 (dec 2020), 24 pages. https://doi.org/10.1145/3432202

Digital Library

[25]

Ying Jiang, Congyi Zhang, Hongbo Fu, Alberto Cannavò, Fabrizio Lamberti, Henry Y K Lau, and Wenping Wang. 2021. HandPainter - 3D Sketching in VR with Hand-Based Physical Proxy. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 412, 13 pages. https://doi.org/10.1145/3411764.3445302

Digital Library

[26]

Yongkwan Kim, Sang-Gyun An, Joon Hyub Lee, and Seok-Hyung Bae. 2018. Agile 3D Sketching with Air Scaffolding. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3173812

Digital Library

[27]

Yuki Kubo, Yuto Koguchi, Buntarou Shizuki, Shin Takahashi, and Otmar Hilliges. 2019. AudioTouch: Minimally Invasive Sensing of Micro-Gestures via Active Bio-Acoustic Sensing. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (Taipei, Taiwan) (MobileHCI ’19). Association for Computing Machinery, New York, NY, USA, Article 36, 13 pages. https://doi.org/10.1145/3338286.3340147

Digital Library

[28]

Eric Larson, Gabe Cohn, Sidhant Gupta, Xiaofeng Ren, Beverly Harrison, Dieter Fox, and Shwetak Patel. 2011. HeatWave: Thermal Imaging for Surface User Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 2565–2574. https://doi.org/10.1145/1978942.1979317

Digital Library

[29]

Chen Liang, Chi Hsia, Chun Yu, Yukang Yan, Yuntao Wang, and Yuanchun Shi. 2023. DRG-Keyboard: Enabling Subtle Gesture Typing on the Fingertip with Dual IMU Rings. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 4, Article 170 (jan 2023), 30 pages. https://doi.org/10.1145/3569463

Digital Library

[30]

Chen Liang, Chun Yu, Yue Qin, Yuntao Wang, and Yuanchun Shi. 2021. DualRing: Enabling Subtle and Expressive Hand Interaction with Dual IMU Rings. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 3, Article 115 (sep 2021), 27 pages. https://doi.org/10.1145/3478114

Digital Library

[31]

Guan Ming Lim, Prayook Jatesiktat, Christopher Wee Keong Kuah, and Wei Tech Ang. 2020. Camera-based Hand Tracking using a Mirror-based Multi-view Setup. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). 5789–5793. https://doi.org/10.1109/EMBC44109.2020.9176728

[32]

Damien Masson, Alix Goguey, Sylvain Malacria, and Géry Casiez. 2017. WhichFingers: Identifying Fingers on Touch Surfaces and Keyboards Using Vibration Sensors. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 41–48. https://doi.org/10.1145/3126594.3126619

Digital Library

[33]

Fabrice Matulic, Aditya Ganeshan, Hiroshi Fujiwara, and Daniel Vogel. 2021. Phonetroller: Visual Representations of Fingers for Precise Touch Input with Mobile Phones in VR. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 129, 13 pages. https://doi.org/10.1145/3411764.3445583

Digital Library

[34]

Manuel Meier, Paul Streli, Andreas Fender, and Christian Holz. 2021. Demonstrating the Use of Rapid Touch Interaction in Virtual Reality for Prolonged Interaction in Productivity Scenarios. In 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). 761–762. https://doi.org/10.1109/VRW52623.2021.00263

[35]

Manuel Meier, Paul Streli, Andreas Fender, and Christian Holz. 2021. TapID: Rapid Touch Interaction in Virtual Reality using Wearable Sensing. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). 519–528. https://doi.org/10.1109/VR50410.2021.00076

[36]

Pranav Mistry and Pattie Maes. 2011. Mouseless: A Computer Mouse as Small as Invisible. In CHI ’11 Extended Abstracts on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI EA ’11). Association for Computing Machinery, New York, NY, USA, 1099–1104. https://doi.org/10.1145/1979742.1979715

Digital Library

[37]

Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. 2020. Interhand2. 6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In European Conference on Computer Vision. Springer, 548–564.

Digital Library

[38]

Anh Nguyen and Amy Banic. 2015. 3DTouch: A wearable 3D input device for 3D applications. In 2015 IEEE Virtual Reality (VR). 55–61. https://doi.org/10.1109/VR.2015.7223324

[39]

Takehiro Niikura, Takashi Matsubara, and Naoki Mori. 2016. Touch Detection System for Various Surfaces Using Shadow of Finger. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces (Niagara Falls, Ontario, Canada) (ISS ’16). Association for Computing Machinery, New York, NY, USA, 337–342. https://doi.org/10.1145/2992154.2996777

Digital Library

[40]

Takehiro Niikura, Yoshihiro Watanabe, and Masatoshi Ishikawa. 2014. Anywhere Surface Touch: Utilizing Any Surface as an Input Area. In Proceedings of the 5th Augmented Human International Conference (Kobe, Japan) (AH ’14). Association for Computing Machinery, New York, NY, USA, Article 39, 8 pages. https://doi.org/10.1145/2582051.2582090

Digital Library

[41]

Ju Young Oh, Jun Lee, Joong Ho Lee, and Ji Hyung Park. 2017. AnywhereTouch: Finger Tracking Method on Arbitrary Surface Using Nailed-Mounted IMU for Mobile HMD. In HCI International 2017 – Posters’ Extended Abstracts, Constantine Stephanidis (Ed.). Springer International Publishing, Cham, 185–191.

[42]

Ju Young Oh, Ji-Hyung Park, and Jung-Min Park. 2020. FingerTouch: Touch Interaction Using a Fingernail-Mounted Sensor on a Head-Mounted Display for Augmented Reality. IEEE Access 8 (2020), 101192–101208. https://doi.org/10.1109/ACCESS.2020.2997972

[43]

Joseph A Paradiso and Che King Leo. 2005. Tracking and characterizing knocks atop large interactive displays. Sensor Review (2005).

[44]

Luis Paredes, Ananya Ipsita, Juan C. Mesa, Ramses V. Martinez Garrido, and Karthik Ramani. 2022. StretchAR: Exploiting Touch and Stretch as a Method of Interaction for Smart Glasses Using Wearable Straps. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 3, Article 134 (sep 2022), 26 pages. https://doi.org/10.1145/3550305

Digital Library

[45]

Yuto Sekiya, Takeshi Umezawa, and Noritaka Osawa. 2021. Detection of Finger Contact with Skin Based on Shadows and Texture Around Fingertips. In Human-Computer Interaction. Interaction Techniques and Novel Applications: Thematic Area, HCI 2021, Held as Part of the 23rd HCI International Conference, HCII 2021, Virtual Event, July 24–29, 2021, Proceedings, Part II 23. Springer, 109–122.

[46]

Yilei Shi, Haimo Zhang, Kaixing Zhao, Jiashuo Cao, Mengmeng Sun, and Suranga Nanayakkara. 2020. Ready, Steady, Touch! Sensing Physical Contact with a Finger-Mounted IMU. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 2, Article 59 (June 2020), 25 pages. https://doi.org/10.1145/3397309

Digital Library

[47]

Garth Shoemaker, Anthony Tang, and Kellogg S. Booth. 2007. Shadow Reaching: A New Perspective on Interaction for Large Displays. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (Newport, Rhode Island, USA) (UIST ’07). Association for Computing Machinery, New York, NY, USA, 53–56. https://doi.org/10.1145/1294211.1294221

Digital Library

[48]

Chengqun Song and Jun Cheng. 2017. A robust projector–camera interactive display system based on finger touch control by fusing finger and its shadow. Journal of the Society for Information Display 25, 9 (2017), 568–576.

[49]

Naoki Sugita, Daisuke Iwai, and Kosuke Sato. 2008. Touch sensing by image analysis of fingernail. In 2008 SICE Annual Conference. 1520–1525. https://doi.org/10.1109/SICE.2008.4654901

[50]

Joseph Thomas. 2013. A Camera Based Virtual Keyboard with Touch Detection by Shadow Analysis. (2013).

[51]

Cheng-Yao Wang, Min-Chieh Hsiu, Po-Tsung Chiu, Chiao-Hui Chang, Liwei Chan, Bing-Yu Chen, and Mike Y Chen. 2015. Palmgesture: Using palms as gesture interfaces for eyes-free input. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services. 217–226.

Digital Library

[52]

Eric Whitmire, Mohit Jain, Divye Jain, Greg Nelson, Ravi Karkar, Shwetak Patel, and Mayank Goel. 2017. DigiTouch: Reconfigurable Thumb-to-Finger Input and Text Entry on Head-Mounted Displays. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 113 (sep 2017), 21 pages. https://doi.org/10.1145/3130978

Digital Library

[53]

Andrew D Wilson. 2004. TouchLight: an imaging touch screen and display for gesture-based interaction. In Proceedings of the 6th international conference on Multimodal interfaces. 69–76.

Digital Library

[54]

Andrew D Wilson. 2005. PlayAnywhere: a compact interactive tabletop projection-vision system. In Proceedings of the 18th annual ACM symposium on User interface software and technology. 83–92.

Digital Library

[55]

Pui Chung Wong, Hongbo Fu, and Kening Zhu. 2016. Back-Mirror: Back-of-Device One-Handed Interaction on Smartphones. In SIGGRAPH ASIA 2016 Mobile Graphics and Interactive Applications (Macau) (SA ’16). Association for Computing Machinery, New York, NY, USA, Article 10, 5 pages. https://doi.org/10.1145/2999508.2999522

Digital Library

[56]

M. Yang, M. Al-Kutubi, and D.T. Pham. 2013. Continuous acoustic source tracking for tangible acoustic interfaces. Measurement 46, 3 (2013), 1272–1278. https://doi.org/10.1016/j.measurement.2012.11.019

[57]

Xing-Dong Yang, Tovi Grossman, Daniel Wigdor, and George Fitzmaurice. 2012. Magic Finger: Always-Available Input through Finger Instrumentation. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (Cambridge, Massachusetts, USA) (UIST ’12). Association for Computing Machinery, New York, NY, USA, 147–156. https://doi.org/10.1145/2380116.2380137

Digital Library

[58]

Xing-Dong Yang, Khalad Hasan, Neil Bruce, and Pourang Irani. 2013. Surround-See: Enabling Peripheral Vision on Smartphones during Active Use. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST ’13). Association for Computing Machinery, New York, NY, USA, 291–300. https://doi.org/10.1145/2501988.2502049

Digital Library

[59]

Xin Yi, Chen Liang, Haozhan Chen, Jiuxu Song, Chun Yu, Hewu Li, and Yuanchun Shi. 2023. From 2D to 3D: Facilitating Single-Finger Mid-Air Typing on QWERTY Keyboards with Probabilistic Touch Modeling. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 7, 1, Article 38 (mar 2023), 25 pages. https://doi.org/10.1145/3580829

Digital Library

[60]

Chungkuk Yoo, Inseok Hwang, Eric Rozner, Yu Gu, and Robert F. Dickerson. 2016. SymmetriSense: Enabling Near-Surface Interactivity on Glossy Surfaces Using a Single Commodity Smartphone. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 5126–5137. https://doi.org/10.1145/2858036.2858286

Digital Library

[61]

Takatoshi Yoshida, Junichi Ogawa, Kyung Yun Choi, Sanad Bushnaq, Ken Nakagaki, and Hiroshi Ishii. 2021. InDepth: Force-Based Interaction with Objects beyond A Physical Barrier. In Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction (Salzburg, Austria) (TEI ’21). Association for Computing Machinery, New York, NY, USA, Article 42, 6 pages. https://doi.org/10.1145/3430524.3442447

Digital Library

[62]

Chun Yu, Xiaoying Wei, Shubh Vachher, Yue Qin, Chen Liang, Yueting Weng, Yizheng Gu, and Yuanchun Shi. 2019. HandSee: Enabling Full Hand Interaction on Smartphone with Front Camera-Based Stereo Vision. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300935

Digital Library

[63]

Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung, Chuo-Ling Chang, and Matthias Grundmann. 2020. MediaPipe Hands: On-device Real-time Hand Tracking. https://doi.org/10.48550/ARXIV.2006.10214

[64]

Mingrui Ray Zhang, Shumin Zhai, and Jacob O. Wobbrock. 2022. TypeAnywhere: A QWERTY-Based Text Entry Solution for Ubiquitous Computing. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 339, 16 pages. https://doi.org/10.1145/3491102.3517686

Digital Library

[65]

Xiang Zhang, Kaori Ikematsu, Kunihiro Kato, and Yuta Sugiura. 2022. ReflecTouch: Detecting Grasp Posture of Smartphone Using Corneal Reflection Images. In CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 289, 8 pages. https://doi.org/10.1145/3491102.3517440

Digital Library

[66]

Yang Zhang, Wolf Kienzle, Yanjun Ma, Shiu S. Ng, Hrvoje Benko, and Chris Harrison. 2019. ActiTouch: Robust Touch Detection for On-Skin AR/VR Interfaces. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 1151–1159. https://doi.org/10.1145/3332165.3347869

Digital Library

Cited By

Kim DWhitmire EBoldu RKienzle WBenko HKostakos VKay JHoang T(2024)SoundScroll: Robust Finger Slide Detection Using Friction Sound and Wrist-Worn MicrophonesProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676614(63-70)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676614
Mollyn VHarrison C(2024)EgoTouch: On-Body Touch Input Using AR/VR Headset CamerasProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676455(1-11)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676455
Streli PRichardson MBotros FMa SWang RHolz C(2024)TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric VisionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676330(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676330

Index Terms

ShadowTouch: Enabling Free-Form Touch-Based Hand-to-Surface Interaction with Wrist-Mounted Illuminant by Shadow Projection
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction techniques
      1. Gestural input
  2. Ubiquitous and mobile computing
    1. Ubiquitous and mobile devices

Recommendations

A Real Time Vision-Based Hand Gesture Interaction
AMS '10: Proceedings of the 2010 Fourth Asia International Conference on Mathematical/Analytical Modelling and Computer Simulation

Moving beyond mouse and keyboard, the evolution of human-computer interaction (HCI) has been an interest research in recent years which witnessed the development from text-based like using a keyboard to graphic user interface (GUI) based on a mouse, ...
Hand distinction for multi-touch tabletop interaction
ITS '09: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces

Recent multi-touch multi-user tabletop systems offer rich touch contact properties to applications. Not only they provide touch positions, but also finger orientations. Applications can use these properties separated for each finger or derive ...
Enabling tangible interaction on capacitive touch panels
UIST '10: Adjunct proceedings of the 23nd annual ACM symposium on User interface software and technology

We propose two approaches to sense tangible objects on capacitive touch screens, which are used in off-the-shelf multi-touch devices such as Apple iPad, iPhone, and 3M's multi-touch displays. We seek for the approaches that do not require modifications ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '23: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

October 2023

1825 pages

ISBN:9798400701320

DOI:10.1145/3586183

Editors:
Sean Follmer
Stanford University, USA
,
Jeff Han,
Jürgen Steimle
Saarland University, Germany
,
Nathalie Henry Riche
Microsoft Research, USA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

This work is supported by the Natural Science Foundation of China
2025 Key Technological Innovation Program of Ningbo City
Beijing Municipal Science & Technology Commission, Administrative Commission of Zhongguancun Science Park

Conference

UIST '23

Sponsor:

UIST '23: The 36th Annual ACM Symposium on User Interface Software and Technology

October 29 - November 1, 2023

CA, San Francisco, USA

Acceptance Rates

Overall Acceptance Rate 842 of 3,967 submissions, 21%

Upcoming Conference

UIST '24

Sponsor:
sigchi
sigchi

The 37th Annual ACM Symposium on User Interface Software and Technology

October 13 - 16, 2024

Pittsburgh , PA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
1,330
Total Downloads

Downloads (Last 12 months)1,330
Downloads (Last 6 weeks)158

Reflects downloads up to 06 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kim DWhitmire EBoldu RKienzle WBenko HKostakos VKay JHoang T(2024)SoundScroll: Robust Finger Slide Detection Using Friction Sound and Wrist-Worn MicrophonesProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676614(63-70)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676614
Mollyn VHarrison C(2024)EgoTouch: On-Body Touch Input Using AR/VR Headset CamerasProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676455(1-11)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676455
Streli PRichardson MBotros FMa SWang RHolz C(2024)TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric VisionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676330(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676330

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents