research-article

Reducing the Latency of Touch Tracking on Ad-hoc Surfaces

Authors:

Robert XiaoAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 6, Issue ISS

Article No.: 577, Pages 489 - 499

https://doi.org/10.1145/3567730

Published: 14 November 2022 Publication History

Abstract

Touch sensing on ad-hoc surfaces has the potential to transform everyday surfaces in the environment - desks, tables and walls - into tactile, touch-interactive surfaces, creating large, comfortable interactive spaces without the cost of large touch sensors. Depth sensors are a promising way to provide touch sensing on arbitrary surfaces, but past systems have suffered from high latency and poor touch detection accuracy. We apply a novel state machine-based approach to analyzing touch events, combined with a machine-learning approach to predictively classify touch events from depth data with lower latency and higher touch accuracy than previous approaches. Our system can reduce end-to-end touch latency to under 70ms, comparable to conventional capacitive touchscreens. Additionally, we open-source our dataset of over 30,000 touch events recorded in depth, infrared and RGB for the benefit of future researchers.

Supplementary Material

Teaser (iss22main-id4514-p-teaser.mp4)

This is the Teaser Video for our work: Reducing the Latency of Touch Tracking on Ad-hoc Surfaces

Download
186.67 MB

References

[1]

Arturo Cadena, Rubén Carvajal, Bruno Guamán, Roger Granda, Enrique Peláez, and Katherine Chiluiza. 2016. Fingertip detection approach on depth image sequences for interactive projection system. In 2016 IEEE Ecuador Technical Chapters Meeting (ETCM). 1–6. https://doi.org/10.1109/ETCM.2016.7750827

[2]

Elie Cattan, Amélie Rochet-Capellan, and François Bérard. 2015. A Predictive Approach for an End-to-End Touch-Latency Measurement. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces (ITS ’15). Association for Computing Machinery, New York, NY, USA. 215–218. isbn:978-1-4503-3899-8 https://doi.org/10.1145/2817721.2817747

Digital Library

[3]

Elie Cattan, Amélie Rochet-Capellan, Pascal Perrier, and François Bérard. 2015. Reducing Latency with a Continuous Prediction: Effects on Users’ Performance in Direct-Touch Target Acquisitions. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces (ITS ’15). Association for Computing Machinery, New York, NY, USA. 205–214. isbn:978-1-4503-3899-8 https://doi.org/10.1145/2817721.2817736

Digital Library

[4]

Zhi Chai and Roy Shilkrot. 2018. Enhanced Touchable Projector-depth System with Deep Hand Pose Estimation. arXiv:1812.11090 [cs], Dec., arxiv:1812.11090 arXiv: 1812.11090

[5]

Jonathan Deber, Bruno Araujo, Ricardo Jota, Clifton Forlines, Darren Leigh, Steven Sanders, and Daniel Wigdor. 2016. Hammer Time! A Low-Cost, High Precision, High Accuracy Tool to Measure the Latency of Touchscreen Devices. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). Association for Computing Machinery, New York, NY, USA. 2857–2868. isbn:978-1-4503-3362-7 https://doi.org/10.1145/2858036.2858394

Digital Library

[6]

Andreas Dippon and Gudrun Klinker. 2011. KinectTouch: accuracy test for a very low-cost 2.5D multitouch tracking system. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS ’11). Association for Computing Machinery, New York, NY, USA. 49–52. isbn:978-1-4503-0871-7 https://doi.org/10.1145/2076354.2076363

Digital Library

[7]

Florian Echtler, Manuel Huber, and Gudrun Klinker. 2008. Shadow tracking on multi-touch tables. In Proceedings of the working conference on Advanced visual interfaces (AVI ’08). Association for Computing Machinery, New York, NY, USA. 388–391. isbn:978-1-60558-141-5 https://doi.org/10.1145/1385569.1385640

Digital Library

[8]

Thomas B. Fitzpatrick. 1988. The Validity and Practicality of Sun-Reactive Skin Types I Through VI. Archives of Dermatology, 124, 6 (1988), June, 869–871. issn:0003-987X https://doi.org/10.1001/archderm.1988.01670060015008

[9]

Yuxiang Gao and Chien-Ming Huang. 2019. PATI: a projection-based augmented table-top interface for robot programming. In Proceedings of the 24th International Conference on Intelligent User Interfaces (IUI ’19). Association for Computing Machinery, New York, NY, USA. 345–355. isbn:978-1-4503-6272-6 https://doi.org/10.1145/3301275.3302326

Digital Library

[10]

Jun Gong, Aakar Gupta, and Hrvoje Benko. 2020. Acustico: Surface Tap Detection and Localization using Wrist-based Acoustic TDOA Sensing. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, New York, NY, USA. 406–419. isbn:978-1-4503-7514-6 https://doi.org/10.1145/3379337.3415901

Digital Library

[11]

Dan Gregor, Ondrej Prucha, Jakub Rocek, and Josef Kortan. 2017. Digital playgroundz. In ACM SIGGRAPH 2017 VR Village (SIGGRAPH ’17). Association for Computing Machinery, New York, NY, USA. 1–2. isbn:978-1-4503-5013-6 https://doi.org/10.1145/3089269.3089288

Digital Library

[12]

Yizheng Gu, Chun Yu, Zhipeng Li, Weiqi Li, Shuchang Xu, Xiaoying Wei, and Yuanchun Shi. 2019. Accurate and Low-Latency Sensing of Touch Contact on Any Surface with Finger-Worn IMU Sensor. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST ’19). Association for Computing Machinery, New York, NY, USA. 1059–1070. isbn:978-1-4503-6816-2 https://doi.org/10.1145/3332165.3347947

Digital Library

[13]

Taku Hachisu and Hiroyuki Kajimoto. 2013. HACHIStack: dual-layer photo touch sensing for haptic and auditory tapping interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13). Association for Computing Machinery, New York, NY, USA. 1411–1420. isbn:978-1-4503-1899-0 https://doi.org/10.1145/2470654.2466187

Digital Library

[14]

Ken Hinckley, Seongkook Heo, Michel Pahud, Christian Holz, Hrvoje Benko, Abigail Sellen, Richard Banks, Kenton O’Hara, Gavin Smyth, and William Buxton. 2016. Pre-Touch Sensing for Mobile Interaction. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). Association for Computing Machinery, New York, NY, USA. 2869–2881. isbn:978-1-4503-3362-7 https://doi.org/10.1145/2858036.2858095

Digital Library

[15]

Daisuke Iwai and Kosuke Sato. 2005. Heat sensation in image creation with thermal vision. In Proceedings of the 2005 ACM SIGCHI International Conference on Advances in computer entertainment technology (ACE ’05). Association for Computing Machinery, New York, NY, USA. 213–216. isbn:978-1-59593-110-8 https://doi.org/10.1145/1178477.1178510

Digital Library

[16]

Daniel Kurz. 2014. Thermal touch: Thermography-enabled everywhere touch interfaces for mobile augmented reality applications. In 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 9–16. https://doi.org/10.1109/ISMAR.2014.6948403

[17]

Gierad Laput and Chris Harrison. 2019. SurfaceSight: A New Spin on Touch, User, and Object Sensing for IoT Experiences. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA. 1–12. isbn:978-1-4503-5970-2 https://doi.org/10.1145/3290605.3300559

Digital Library

[18]

Jiajun Lu, Hrvoje Benko, and Andrew D. Wilson. 2017. Hybrid HFR Depth: Fusing Commodity Depth and Color Cameras to Achieve High Frame Rate, Low Latency Depth Camera Interactions. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17). Association for Computing Machinery, New York, NY, USA. 5966–5975. isbn:978-1-4503-4655-9 https://doi.org/10.1145/3025453.3025478

Digital Library

[19]

Joe Marshall, Tony Pridmore, Mike Pound, Steve Benford, and Boriana Koleva. 2008. Pressing the Flesh: Sensing Multiple Touch and Finger Pressure on Arbitrary Surfaces. In Pervasive Computing, Jadwiga Indulska, Donald J. Patterson, Tom Rodden, and Max Ott (Eds.). 5013, Springer Berlin Heidelberg, Berlin, Heidelberg. 38–55. isbn:978-3-540-79575-9 978-3-540-79576-6 https://doi.org/10.1007/978-3-540-79576-6_3 Series Title: Lecture Notes in Computer Science

Digital Library

[20]

Takashi Matsubara, Naoki Mori, Takehiro Niikura, and Shun’ichi Tano. 2017. Touch detection method for non-display surface using multiple shadows of finger. In 2017 IEEE 6th Global Conference on Consumer Electronics (GCCE). 1–5. https://doi.org/10.1109/GCCE.2017.8229364

[21]

Manuel Meier, Paul Streli, Andreas Fender, and Christian Holz. 2021. TapID: Rapid Touch Interaction in Virtual Reality using Wearable Sensing. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). 519–528. https://doi.org/10.1109/VR50410.2021.00076 ISSN: 2642-5254

[22]

Sundar Murugappan, Vinayak, Niklas Elmqvist, and Karthik Ramani. 2012. Extended multitouch: recovering touch posture and differentiating users using a depth camera. In Proceedings of the 25th annual ACM symposium on User interface software and technology. Association for Computing Machinery, New York, NY, USA. 487–496. isbn:978-1-4503-1580-7 https://doi.org/10.1145/2380116.2380177

Digital Library

[23]

Takehiro Niikura, Takashi Matsubara, and Naoki Mori. 2016. Touch Detection System for Various Surfaces Using Shadow of Finger. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces (ISS ’16). Association for Computing Machinery, New York, NY, USA. 337–342. isbn:978-1-4503-4248-3 https://doi.org/10.1145/2992154.2996777

Digital Library

[24]

Vivian Shen, James Spann, and Chris Harrison. 2021. FarOut Touch: Extending the Range of ad hoc Touch Sensing with Depth Cameras. In Symposium on Spatial User Interaction (SUI ’21). Association for Computing Machinery, New York, NY, USA. 1–12. isbn:978-1-4503-9091-0 https://doi.org/10.1145/3485279.3485281

Digital Library

[25]

Ziwei Song, Yuichiro Kinoshita, Kentaro Go, and Gangyong Jia. 2021. Touch Point Prediction for Interactive Public Displays Based on Camera Images. In 2021 International Conference on Cyberworlds (CW). 133–136. https://doi.org/10.1109/CW52790.2021.00029 ISSN: 2642-3596

[26]

Naoki Sugita, Daisuke Iwai, and Kosuke Sato. 2008. Touch sensing by image analysis of fingernail. In 2008 SICE Annual Conference. 1520–1525. https://doi.org/10.1109/SICE.2008.4654901

[27]

Ryo Takahashi, Masaaki Fukumoto, Changyo Han, Takuya Sasatani, Yoshiaki Narusue, and Yoshihiro Kawahara. 2020. TelemetRing: A Batteryless and Wireless Ring-shaped Keyboard using Passive Inductive Telemetry. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, New York, NY, USA. 1161–1168. isbn:978-1-4503-7514-6 https://doi.org/10.1145/3379337.3415873

Digital Library

[28]

Mayuka Tsuji, Hiroyuki Kubo, Suren Jayasuriya, Takuya Funatomi, and Yasuhiro Mukaigawa. 2021. Touch Sensing for a Projected Screen Using Slope Disparity Gating. IEEE Access, 9 (2021), 106005–106013. issn:2169-3536 https://doi.org/10.1109/ACCESS.2021.3099901 Conference Name: IEEE Access

[29]

Andrew D. Wilson. 2010. Using a depth camera as a touch sensor. In ACM International Conference on Interactive Tabletops and Surfaces (ITS ’10). Association for Computing Machinery, New York, NY, USA. 69–72. isbn:978-1-4503-0399-6 https://doi.org/10.1145/1936652.1936665

Digital Library

[30]

Haijun Xia, Ricardo Jota, Benjamin McCanny, Zhe Yu, Clifton Forlines, Karan Singh, and Daniel Wigdor. 2014. Zero-latency tapping: using hover information to predict touch locations and eliminate touchdown latency. In Proceedings of the 27th annual ACM symposium on User interface software and technology (UIST ’14). Association for Computing Machinery, New York, NY, USA. 205–214. isbn:978-1-4503-3069-5 https://doi.org/10.1145/2642918.2647348

Digital Library

[31]

Robert Xiao, Scott Hudson, and Chris Harrison. 2016. DIRECT: Making Touch Tracking on Ordinary Surfaces Practical with Hybrid Depth-Infrared Sensing. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces (ISS ’16). Association for Computing Machinery, New York, NY, USA. 85–94. isbn:978-1-4503-4248-3 https://doi.org/10.1145/2992154.2992173

Digital Library

[32]

Robert Xiao, Julia Schwarz, Nick Throm, Andrew D. Wilson, and Hrvoje Benko. 2018. MRTouch: Adding Touch Input to Head-Mounted Mixed Reality. IEEE Transactions on Visualization and Computer Graphics, 24, 4 (2018), April, 1653–1660. issn:1077-2626 https://doi.org/10.1109/TVCG.2018.2794222

Digital Library

[33]

Lixing Zhang and Takafumi Matsumaru. 2016. Near-Field Touch Interface Using Time-of-Flight Camera. Journal of Robotics and Mechatronics, 28, 5 (2016), Oct., 759–775. https://doi.org/10.20965/jrm.2016.p0759 Publisher: Fuji Technology Press Ltd.

[34]

Yang Zhang, Gierad Laput, and Chris Harrison. 2017. Electrick: Low-Cost Touch Sensing Using Electric Field Tomography. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA. 1–14. isbn:978-1-4503-4655-9 https://doi.org/10.1145/3025453.3025842

Digital Library

Cited By

Streli PRichardson MBotros FMa SWang RHolz C(2024)TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric VisionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676330(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676330
Dupré CAppert CRey SSaidi HPietriga E(2024)TriPad: Touch Input in AR on Ordinary Surfaces with Hand Tracking OnlyProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642323(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642323
Streli PJiang JRossie JHolz C(2023)Structured Light Speckle: Joint Ego-Centric Depth Estimation and Low-Latency Contact Detection via Remote VibrometryProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606749(1-12)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606749

Index Terms

Reducing the Latency of Touch Tracking on Ad-hoc Surfaces
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction techniques

Recommendations

Bimanual Interaction with Interscopic Multi-Touch Surfaces
INTERACT '09: Proceedings of the 12th IFIP TC 13 International Conference on Human-Computer Interaction: Part II

Multi-touch interaction has received considerable attention in the last few years, in particular for natural two-dimensional (2D) interaction. However, many application areas deal with three-dimensional (3D) data and require intuitive 3D interaction ...
Single-tap Latency Reduction with Single- or Double- tap Prediction
MHCI

Touch surfaces are widely utilized for smartphones, tablet PCs, and laptops (touchpad), and single and double taps are the most basic and common operations on them. The detection of single or double taps causes the single-tap latency problem, which ...
To touch or not to touch?: comparing 2D touch and 3D mid-air interaction on stereoscopic tabletop surfaces
SUI '13: Proceedings of the 1st symposium on Spatial user interaction

Recent developments in touch and display technologies have laid the groundwork to combine touch-sensitive display systems with stereoscopic three-dimensional (3D) display. Although this combination provides a compelling user experience, interaction with ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 6, Issue ISS

December 2022

746 pages

EISSN:2573-0142

DOI:10.1145/3554337

Editor:
Jeffrey Nichols
Apple, USA

Issue’s Table of Contents

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 November 2022

Published in PACMHCI Volume 6, Issue ISS

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Natural Sciences and Engineering Research Council of Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
167
Total Downloads

Downloads (Last 12 months)47
Downloads (Last 6 weeks)4

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Streli PRichardson MBotros FMa SWang RHolz C(2024)TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric VisionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676330(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676330
Dupré CAppert CRey SSaidi HPietriga E(2024)TriPad: Touch Input in AR on Ordinary Surfaces with Hand Tracking OnlyProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642323(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642323
Streli PJiang JRossie JHolz C(2023)Structured Light Speckle: Joint Ego-Centric Depth Estimation and Low-Latency Contact Detection via Remote VibrometryProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606749(1-12)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606749

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents