research-article

Open access

ActiTouch: Robust Touch Detection for On-Skin AR/VR Interfaces

Authors:

Chris HarrisonAuthors Info & Claims

UIST '19: Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology

Pages 1151 - 1159

https://doi.org/10.1145/3332165.3347869

Published: 17 October 2019 Publication History

Abstract

Contemporary AR/VR systems use in-air gestures or handheld controllers for interactivity. This overlooks the skin as a convenient surface for tactile, touch-driven interactions, which are generally more accurate and comfortable than free space interactions. In response, we developed ActiTouch, a new electrical method that enables precise on-skin touch segmentation by using the body as an RF waveguide. We combine this method with computer vision, enabling a system with both high tracking precision and robust touch detection. Our system requires no cumbersome instrumentation of the fingers or hands, requiring only a single wristband (e.g., smartwatch) and sensors integrated into an AR/VR headset. We quantify the accuracy of our approach through a user study and demonstrate how it can enable touchscreen-like interactions on the skin.

Supplementary Material

MP4 File (ufp1312pv.mp4)

Preview video

Download
10.07 MB

MP4 File (ufp1312vf.mp4)

Supplemental video

Download
125.16 MB

References

[1]

Adafruit. Knit Conductive Fabric. Retrieved July 4, 2019 from https://www.adafruit.com/product/1167

[2]

Analog Devices. AD5930. Retrieved July 4, 2019 from https://www.analog.com/media/en/technical-documentation/data-sheets/ad5930.pdf

[3]

Gabe Cohn, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. Humantenna: using the body as an antenna for real-time whole-body interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 1901--1910.

Digital Library

[4]

Niloofar Dezfuli, Mohammadreza Khalilbeigi, Jochen Huber, Florian Müller, and Max Mühlhäuser. 2012. PalmRC: imaginary palm-based remote control for eyes-free television interaction. In Proceedings of the 10th European Conference on Interactive TV and Vid-eo (EuroITV '12). ACM, New York, NY, USA, 27--34.

Digital Library

[5]

Paul Dietz and Darren Leigh. 2001. DiamondTouch: a multi-user touch technology. In Proceedings of the 14th annual ACM symposium on User interface soft-ware and technology (UIST '01). ACM, New York, NY, USA, 219--226.

Digital Library

[6]

Facebook Inc. Oculus Rift and Touch. Retrieved July 4, 2019 from https://www.oculus.com/rift/accessories

[7]

Freescale. K20P64M72SF1. Retrieved 7/4/19 from http://cache.freescale.com/files/32bit/doc/data_sheet/ K20P64M72SF1.pdf

[8]

Sean Gustafson, Christian Holz, and Patrick Baudisch. 2011. Imaginary phone: learning imaginary interfaces by transferring spatial memory from a familiar device. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11). ACM, New York, NY, USA, 283--292.

Digital Library

[9]

Taku Hachisu, Baptiste Bourreau, and Kenji Suzuki. 2019. EnhancedTouchX: Smart Bracelets for Aug-menting Interpersonal Touch Interactions. In Proceed-ings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, Paper 321, 12 pages.

Digital Library

[10]

Keisuke Hachisuka, Teruhito Takeda, Yusuke Terau-chi, Ken Sasaki, Hiroshi Hosaka, and Kiyoshi Itao. In-tra-body data transmission for the personal area net-work. Microsystem Technologies 11, no. 8--10 (2005): 1020--1027.

[11]

Chris Harrison, Desney Tan, and Dan Morris. 2010. Skinput: appropriating the body as an input surface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 453--462.

Digital Library

[12]

Chris Harrison, Hrvoje Benko, and Andrew D. Wilson. 2011. OmniTouch: wearable multitouch interaction everywhere. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11). ACM, New York, NY, USA, 441--450.

[13]

Christian Holz, Tovi Grossman, George Fitzmaurice, and Anne Agur. 2012. Implanted user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 503--512.

Digital Library

[14]

HTC. Vive. Retrieved July 4, 2019 from https://www.vive.com/us/

[15]

Toshikazu Karitsuka, and Kosuke Sato. "A wearable mixed reality with an on-board projector." In The Sec-ond IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings., pp. 321--322. IEEE, 2003.

[16]

Gierad Laput, Robert Xiao, Xiang 'Anthony' Chen, Scott E. Hudson, and Chris Harrison. 2014. Skin but-tons: cheap, small, low-powered and clickable fixed-icon laser projectors. In Proceedings of the 27th annual ACM symposium on User interface software and technology (UIST '14). ACM, New York, NY, USA, 389--394.

Digital Library

[17]

Gierad Laput, Chouchang Yang, Robert Xiao, Alanson Sample, and Chris Harrison. 2015. EM-Sense: Touch Recognition of Uninstrumented, Electrical and Elec-tromechanical Objects. In Proceedings of the 28th An-nual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, New York, NY, USA, 157--166.

Digital Library

[18]

Gierad Laput, Robert Xiao, and Chris Harrison. 2016. ViBand: High-Fidelity Bio-Acoustic Sensing Using Commodity Smartwatch Accelerometers. In Proceed-ings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 321--333.

Digital Library

[19]

Leap Motion Inc. Leap Motion. Retrieved July 4, 2019 from https://www.leapmotion.com

[20]

Linear Technology. LT1806. Retrieved July 4, 2019 from https://www.analog.com/media/en/technical-documentation/data-sheets/18067fc.pdf

[21]

Rong-Hao Liang, Shu-Yang Lin, Chao-Huai Su, Kai-Yin Cheng, Bing-Yu Chen, and De-Nian Yang. 2011. SonarWatch: appropriating the forearm as a slider bar. In SIGGRAPH Asia 2011 Emerging Technologies (SA '11), Article 5, 1 pages.

Digital Library

[22]

Robert W. Lindeman, John L. Sibert, and James K. Hahn. 1999. Towards usable VR: an empirical study of user interfaces for immersive virtual environments. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (CHI '99). ACM, New York, NY, USA, 64--71.

[23]

Denys J. C. Matthies, Simon T. Perrault, Bodo Urban, and Shengdong Zhao. 2015. Botential: Localizing On-Body Gestures by Measuring Electrical Signatures on the Human Skin. In Proceedings of the 17th Interna-tional Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '15). ACM, New York, NY, USA, 207--216.

Digital Library

[24]

Microsoft. HoloLens 2. Retrieved July 4, 2019 from https://www.microsoft.com/en-US/hololens

[25]

Pranav Mistry, Pattie Maes, and Liyan Chang. WUW-wear Ur world: a wearable gestural interface. In CHI'09 extended abstracts on Human factors in computing systems, pp. 4111--4116. ACM, 2009.

Digital Library

[26]

Adiyan Mujibiya, Xiang Cao, Desney S. Tan, Dan Morris, Shwetak N. Patel, and Jun Rekimoto. 2013. The sound of touch: on-body touch and gesture sensing based on transdermal ultrasound propagation. In Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces (ITS '13). ACM, New York, NY, USA, 189--198.

Digital Library

[27]

PJRC Inc. Teensy 3.2. Retrieved July 4, 2019 from https://www.pjrc.com/teensy/teensy31.html

[28]

Manuel Prätorius, Aaron Scherzinger, and Klaus Hin-richs. SkInteract: An on-body interaction system based on skin-texture recognition. In IFIP Conference on Human-Computer Interaction, pp. 425--432. Springer, Cham, 2015.

[29]

Nobuchika Sakata, Teppei Konishi, and Shogo Nishida. Mobile interfaces using body worn projector and camera. In International Conference on Virtual and Mixed Reality, pp. 106--113. Springer, Berlin, Hei-delberg, 2009.

Digital Library

[30]

Kenji Suzuki, Taku Hachisu, and Kazuki Iida. 2016. EnhancedTouch: A Smart Bracelet for Enhancing Hu-man-Human Physical Touch. In Proceedings of the 2016 CHI Conference on Human Factors in Compu-ting Systems (CHI '16). ACM, New York, NY, USA, 1282--1293.

Digital Library

[31]

Texas Instruments. REF2030. Retrieved July 4, 2019 from http://www.ti.com/lit/ds/sbos600d/sbos600d.pdf

[32]

Bruce Thomas, K. Grimmer, J. Zucco, and S. Milanese. 2002. Where Does the Mouse Go? An Investigation in-to the Placement of a Body-Attached TouchPad Mouse for Wearable Computers. Personal Ubiquitous Comput. 6, 2 (January 2002), 97--112.

[33]

Virag Varga, Marc Wyss, Gergely Vakulya, Alanson Sample, and Thomas R. Gross. 2018. Designing Groundless Body Channel Communication Systems: Performance and Implications. In Proceedings of the 31st Annual ACM Symposium on User Interface Soft-ware and Technology (UIST '18). ACM, New York, NY, USA, 683--695.

[34]

Martin Weigel, Tong Lu, Gilles Bailly, Antti Oulasvirta, Carmel Majidi, and Jürgen Steimle. 2015. iSkin: Flexible, Stretchable and Visually Customizable On-Body Touch Sensors for Mobile Computing. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2991--3000.

Digital Library

[35]

Robert Xiao, Scott Hudson, and Chris Harrison. 2016. DIRECT: Making Touch Tracking on Ordinary Sur-faces Practical with Hybrid Depth-Infrared Sensing. In Proceedings of the 2016 ACM International Confer-ence on Interactive Surfaces and Spaces (ISS '16). ACM, New York, NY, USA, 85--94.

Digital Library

[36]

Robert Xiao, Julia Schwarz, Nick Throm, Andrew D. Wilson, and Hrvoje Benko. "MRTouch: adding touch input to head-mounted mixed reality." IEEE transac-tions on visualization and computer graphics 24, no. 4 (2018): 1653--1660.

[37]

Robert Xiao, Teng Cao, Ning Guo, Jun Zhuo, Yang Zhang, and Chris Harrison. 2018. LumiWatch: On-Arm Projected Graphics and Touch Input. In Proceed-ings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Paper 95, 11 pages.

Digital Library

[38]

Goshiro Yamamoto and Kosuke Sato. PALMbit: A Body Interface Utilizing Light Projection onto Palms. Inst. Image Information and Television Engineers 61, no. 6 (2007): 797--804.

[39]

Yang Zhang, Junhan Zhou, Gierad Laput, and Chris Harrison. 2016. SkinTrack: Using the Body as an Elec-trical Waveguide for Continuous Finger Tracking on the Skin. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1491--1503.

Digital Library

[40]

Junhan Zhou, Yang Zhang, Gierad Laput, and Chris Harrison. 2016. AuraSense: Enabling Expressive Around-Smartwatch Interactions with Electric Field Sensing. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 81--86.

Digital Library

Cited By

Hoffmann PHoffmann P(2025)The Merging of Worlds and … of ImmersionNext Generation Internet10.1007/978-3-658-46424-0_3(25-77)Online publication date: 8-Jan-2025
https://doi.org/10.1007/978-3-658-46424-0_3
Darbar RHu XYan XWei YLiang HXu WSarcar S(2024)OnArmQWERTY: An Empirical Evaluation of On-Arm Tap Typing for AR HMDsProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682084(1-12)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3682084
Kim DWhitmire EBoldu RKienzle WBenko HKostakos VKay JHoang T(2024)SoundScroll: Robust Finger Slide Detection Using Friction Sound and Wrist-Worn MicrophonesProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676614(63-70)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676614
Show More Cited By

Index Terms

ActiTouch: Robust Touch Detection for On-Skin AR/VR Interfaces
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Graphics input devices

Recommendations

Immersive Multiplayer Games With Tangible and Physical Interaction

In this paper, we present a new immersive multiplayer game system developed for two different environments, namely, virtual reality (VR) and augmented reality (AR). To evaluate our system, we developed three game applications-a first-person-shooter game ...
Comparative Study of HMD-based Virtual and Augmented Realities for Immersive Museums: User Acceptance, Medium, and Learning
Technologies like Head Mounted Display (HMD)-based Virtual Reality (VR) and Augmented Reality (AR) have made HMD-based immersive museums possible. To investigate the user acceptance, medium, and interaction experience for HMD-based immersive museums, an ...
Magic Lenses for Augmented Virtual Environments

The authors developed a Magic Lens framework for Scape, an augmented virtual environment (AVE). They generalize the functional characteristics of Magic Lenses in terms of 3D visualization in AVEs and present two tangible Magic Lens-enabled devices with ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '19: Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology

October 2019

1229 pages

ISBN:9781450368162

DOI:10.1145/3332165

General Chair:
François Guimbretière
Cornell University, USA
,
Program Chairs:
Michael Bernstein
Stanford University, USA
,
Katharina Reinecke
University of Washington, USA

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

UIST '19

Sponsor:

UIST '19: The 32nd Annual ACM Symposium on User Interface Software and Technology

October 20 - 23, 2019

LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

50
Total Citations
View Citations
3,080
Total Downloads

Downloads (Last 12 months)424
Downloads (Last 6 weeks)59

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hoffmann PHoffmann P(2025)The Merging of Worlds and … of ImmersionNext Generation Internet10.1007/978-3-658-46424-0_3(25-77)Online publication date: 8-Jan-2025
https://doi.org/10.1007/978-3-658-46424-0_3
Darbar RHu XYan XWei YLiang HXu WSarcar S(2024)OnArmQWERTY: An Empirical Evaluation of On-Arm Tap Typing for AR HMDsProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682084(1-12)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3682084
Kim DWhitmire EBoldu RKienzle WBenko HKostakos VKay JHoang T(2024)SoundScroll: Robust Finger Slide Detection Using Friction Sound and Wrist-Worn MicrophonesProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676614(63-70)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676614
Bartels JSanchez-Tamayo NSedlmair MKuchenbecker K(2024)Active Haptic Feedback for a Virtual Wrist-Anchored User InterfaceAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686765(1-3)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3672539.3686765
Mollyn VHarrison C(2024)EgoTouch: On-Body Touch Input Using AR/VR Headset CamerasProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676455(1-11)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676455
Cheng WChan L(2024)BodyTouchProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314267:4(1-22)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631426
Yang XSayono JXu JZhang Y(2024)Interaction-Power Stations: Turning Environments into Ubiquitous Power Stations for Charging WearablesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650769(1-8)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650769
Zhu FSousa MSidenmark LGrossman T(2024)PhoneInVR: An Evaluation of Spatial Anchoring and Interaction Techniques for Smartphone Usage in Virtual RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642582(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642582
Grady PCollins JTang CTwigg CAneja KHays JKemp C(2024)PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00850(8683-8693)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00850
Hoffmann PHoffmann P(2024)Das Verschmelzen von Welten und … versenNext Generation Internet10.1007/978-3-658-43029-0_3(27-86)Online publication date: 2-Feb-2024
https://doi.org/10.1007/978-3-658-43029-0_3
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten