research-article

TapSkin: Recognizing On-Skin Input for Smartwatches

Authors:

AbdelKareem Bedri,

Thad E. Starner,

Gregory D. AbowdAuthors Info & Claims

ISS '16: Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces

Pages 13 - 22

https://doi.org/10.1145/2992154.2992187

Published: 06 November 2016 Publication History

Abstract

The touchscreen has been the dominant input surface for smartphones and smartwatches. However, its small size compared to a phone limits the richness of the input gestures that can be supported. We present TapSkin, an interaction technique that recognizes up to 11 distinct tap gestures on the skin around the watch using only the inertial sensors and microphone on a commodity smartwatch. An evaluation with 12 participants shows our system can provide classification accuracies from 90.69% to 97.32% in three gesture families -- number pad, d-pad, and corner taps. We discuss the opportunities and remaining challenges for widespread use of this technique to increase input richness on a smartwatch without requiring further on-body instrumentation.

Supplementary Material

suppl.mov (iss234.mp4)

Supplemental video

Download
31.32 MB

References

[1]

Ashbrook, D., Baudisch, P., and White, S. Nenya: Subtle and eyes-free mobile input with a magnetically-tracked finger ring. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '11, ACM (New York, NY, USA, 2011), 2043--2046.

Digital Library

[2]

Baudisch, P., and Chu, G. Back-of-device interaction allows creating very small touch devices. In Proceedings of the 27th international conference on Human factors in computing systems - CHI 09, Association for Computing Machinery (ACM) (2009).

Digital Library

[3]

Bernaerts, Y., Druwé, M., Steensels, S., Vermeulen, J., and Schöning, J. The office smartwatch: development and design of a smartwatch app to digitally augment interactions in an office environment. In Proceedings of the 2014 companion publication on Designing interactive systems, ACM (2014), 41--44.

Digital Library

[4]

Bi, X., Li, Y., and Zhai, S. FFitts law. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI 13, Association for Computing Machinery (ACM) (2013).

Digital Library

[5]

Chen, K.-Y., Lyons, K., White, S., and Patel, S. utrack. In Proceedings of the 26th annual ACM symposium on User interface software and technology - UIST 13, Association for Computing Machinery (ACM) (2013).

Digital Library

[6]

Dementyev, A., and Paradiso, J. A. Wristflex. In Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST 14, Association for Computing Machinery (ACM) (2014).

[7]

Deyle, T., Palinko, S., Poole, E. S., and Starner, T. Hambone: A bio-acoustic gesture interface. In 2007 11th IEEE International Symposium on Wearable Computers, Institute of Electrical & Electronics Engineers (IEEE) (oct 2007).

Digital Library

[8]

Funk, M., Sahami, A., Henze, N., and Schmidt, A. Using a touch-sensitive wristband for text entry on smart watches. In Proceedings of the extended abstracts of the 32nd annual ACM conference on Human factors in computing systems - CHI EA 14, Association for Computing Machinery (ACM) (2014).

Digital Library

[9]

Goel, M., Wobbrock, J., and Patel, S. Gripsense. In Proceedings of the 25th annual ACM symposium on User interface software and technology - UIST 12, Association for Computing Machinery (ACM) (2012).

Digital Library

[10]

Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., and Witten, I. H. The weka data mining software: an update. ACM SIGKDD explorations newsletter 11, 1 (2009), 10--18.

Digital Library

[11]

Harrison, C., Benko, H., and Wilson, A. D. Omnitouch: wearable multitouch interaction everywhere. In Proceedings of the 24th annual ACM symposium on User interface software and technology, ACM (2011), 441--450.

Digital Library

[12]

Harrison, C., Tan, D., and Morris, D. Skinput: appropriating the body as an input surface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2010), 453--462.

Digital Library

[13]

Hinckley, K., and Horvitz, E. Toward more sensitive mobile phones. In Proceedings of the 14th annual ACM symposium on User interface software and technology, ACM (2001), 191--192.

Digital Library

[14]

Kim, J., He, J., Lyons, K., and Starner, T. The gesture watch: A wireless contact-free gesture based wrist interface. In 2007 11th IEEE International Symposium on Wearable Computers, Institute of Electrical & Electronics Engineers (IEEE) (oct 2007).

Digital Library

[15]

Laput, G., Xiao, R., Chen, X. ., Hudson, S. E., and Harrison, C. Skin buttons. In Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST 14, Association for Computing Machinery (ACM) (2014).

Digital Library

[16]

Lyons, K., Nguyen, D., Ashbrook, D., and White, S. Facet. In Proceedings of the 25th annual ACM symposium on User interface software and technology UIST 12, Association for Computing Machinery (ACM) (2012).

[17]

Lyons, K., Starner, T., Plaisted, D., Fusia, J., Lyons, A., Drew, A., and Looney, E. Twiddler typing: one-handed chording text entry for mobile phones. In Proceedings of the SIGCHI conference on Human factors in computing systems, ACM (2004), 671--678.

Digital Library

[18]

Lyons, K., Starner, T., Plaisted, D., Fusia, J., Lyons, A., Drew, A., and Looney, E. W. Twiddler typing. In Proceedings of the 2004 conference on Human factors in computing systems - CHI 04, Association for Computing Machinery (ACM) (2004).

Digital Library

[19]

McGrath, W., and Li, Y. Detecting tapping motion on the side of mobile devices by probabilistically combining hand postures. In Proceedings of the 27th annual ACM symposium on User interface software and technology, ACM (2014), 215--219.

Digital Library

[20]

Merrill, D., and Raffie, H. The sound of touch. In CHI'07 Extended Abstracts on Human Factors in Computing Systems, ACM (2007), 2807--2812.

Digital Library

[21]

Nanayakkara, S., Shilkrot, R., Yeo, K. P., and Maes, P. Eyering: a finger-worn input device for seamless interactions with our surroundings. In Proceedings of the 4th Augmented Human International Conference, ACM (2013), 13--20.

Digital Library

[22]

Nandakumar, R., Iyer, V., Tan, D., and Gollakota, S. Fingerio: Using active sonar for fine-grained finger tracking. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, ACM (2016), 1515--1525.

Digital Library

[23]

Oakley, I., Lee, D., Islam, M. R., and Esteves, A. Beats. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI 15, Association for Computing Machinery (ACM) (2015).

[24]

Perrault, S. T., Lecolinet, E., Eagan, J., and Guiard, Y. Watchit. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI 13, Association for Computing Machinery (ACM) (2013).

[25]

Rekimoto, J. GestureWrist and GesturePad: unobtrusive wearable interaction devices. In Proceedings Fifth International Symposium on Wearable Computers,

Digital Library

[26]

Roudaut, A., Huot, S., and Lecolinet, E. TapTap and MagStick. In Proceedings of the working conference on Advanced visual interfaces - AVI 08, Association for Computing Machinery (ACM) (2008).

Digital Library

[27]

Ruiz, J., Li, Y., and Lank, E. User-defined motion gestures for mobile interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2011), 197--206.

Digital Library

[28]

Saponas, T. S., Tan, D. S., Morris, D., Balakrishnan, R., Turner, J., and Landay, J. A. Enabling always-available input with muscle-computer interfaces. In Proceedings of the 22nd annual ACM symposium on User interface software and technology, ACM (2009), 167--176.

Digital Library

[29]

Weigel, M., Mehta, V., and Steimle, J. More than touch: Understanding how people use skin as an input surface for mobile computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '14, ACM (New York, NY, USA, 2014), 179--188.

Digital Library

[30]

Wen, H., Ramos Rojas, J., and Dey, A. K. Serendipity: Finger gesture recognition using an off-the-shelf smartwatch. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, ACM (2016), 3847--3851.

Digital Library

[31]

Xia, H., Grossman, T., and Fitzmaurice, G. Nanostylus. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology - UIST 15, Association for Computing Machinery (ACM) (2015).

[32]

Xiao, R., Laput, G., and Harrison, C. Expanding the input expressivity of smartwatches with mechanical pan, twist, tilt and click. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI 14, Association for Computing Machinery (ACM) (2014).

Digital Library

[33]

Xu, C., Pathak, P. H., and Mohapatra, P. Finger-writing with smartwatch: A case for finger and hand gesture recognition using smartwatch. In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications, ACM (2015), 9--14.

Digital Library

[34]

Yang, X.-D., Grossman, T., Wigdor, D., and Fitzmaurice, G. Magic finger: always-available input through finger instrumentation. In Proceedings of the 25th annual ACM symposium on User interface software and technology, ACM (2012), 147--156.

Digital Library

[35]

Zhang, C., Guo, A., Zhang, D., Southern, C., Arriaga, R., and Abowd, G. Beyondtouch: Extending the input language with built-in sensors on commodity smartphones. In Proceedings of the 20th International Conference on Intelligent User Interfaces, IUI '15, ACM (New York, NY, USA, 2015), 67--77.

Digital Library

[36]

Zhang, C., Yang, J., Southern, C., Starner, T., and Abowd, G. D. Watchout: Extending interactions on a smartwatch with inertial sensing. In 2016 20th IEEE International Symposium on Wearable Computers (2016).

Digital Library

[37]

Zhang, Y., Zhou, J., Laput, G., and Harrison, C. Skintrack: Using the body as an electrical waveguide for continuous finger tracking on the skin. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, ACM (2016), 1491--1503.

Digital Library

Cited By

Cheng WChan L(2024)BodyTouchProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314267:4(1-22)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631426
Hajika RGunasekaran THaigh CPai YHayashi ELien JLottridge DBillinghurst M(2024)RadarHand: A Wrist-Worn Radar for On-Skin Touch-Based Proprioceptive GesturesACM Transactions on Computer-Human Interaction10.1145/361736531:2(1-36)Online publication date: 29-Jan-2024
https://dl.acm.org/doi/10.1145/3617365
Yang XWang XDong GYan ZSrivastava MHayashi EZhang Y(2023)HeadarProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109007:3(1-28)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610900
Show More Cited By

Index Terms

TapSkin: Recognizing On-Skin Input for Smartwatches
1. Human-centered computing

Recommendations

NanoStylus: Enhancing Input on Ultra-Small Displays with a Finger-Mounted Stylus
UIST '15: Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology

Due to their limited input area, ultra-small devices, such as smartwatches, are even more prone to occlusion or the fat finger problem, than their larger counterparts, such as smart phones, tablets, and tabletop displays. We present NanoStylus -- a ...
WatchOut: extending interactions on a smartwatch with inertial sensing
ISWC '16: Proceedings of the 2016 ACM International Symposium on Wearable Computers

Current interactions on a smartwatch are generally limited to a tiny touchscreen, physical buttons or knobs, and speech. We present WatchOut, a suite of interaction techniques that includes three families of tap and swipe gestures which extend input ...
The effects of mobility, encumbrance, and (non-)dominant hand on interaction with smartwatches
ISWC '17: Proceedings of the 2017 ACM International Symposium on Wearable Computers

Smartwatches are designed for short interactions in varying mobile contexts. However little data is available on how present mobile conditions affect interaction with these devices. In this work, we investigate the effects of mobility (walking), ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISS '16: Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces

November 2016

554 pages

ISBN:9781450342483

DOI:10.1145/2992154

General Chairs:
Mark Hancock
University of Waterloo, Canada
,
Nicolai Marquardt
University College London, UK
,
Program Chairs:
Johannes Schöning
Hasselt University, Belgium
,
Melanie Tory
Tableau Research, USA

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 November 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISS '16

Sponsor:

SIGCHI

ISS '16: 2016 ACM International Conference on Interactive Surfaces and Spaces

November 6 - 9, 2016

Ontario, Niagara Falls, Canada

Acceptance Rates

ISS '16 Paper Acceptance Rate 33 of 119 submissions, 28%;

Overall Acceptance Rate 147 of 533 submissions, 28%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

51
Total Citations
View Citations
1,211
Total Downloads

Downloads (Last 12 months)95
Downloads (Last 6 weeks)11

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cheng WChan L(2024)BodyTouchProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314267:4(1-22)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631426
Hajika RGunasekaran THaigh CPai YHayashi ELien JLottridge DBillinghurst M(2024)RadarHand: A Wrist-Worn Radar for On-Skin Touch-Based Proprioceptive GesturesACM Transactions on Computer-Human Interaction10.1145/361736531:2(1-36)Online publication date: 29-Jan-2024
https://dl.acm.org/doi/10.1145/3617365
Yang XWang XDong GYan ZSrivastava MHayashi EZhang Y(2023)HeadarProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109007:3(1-28)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610900
Zhao GYan YYang Z(2023)Low-cost and Non-visual Labels Using Magnetic PrintingProceedings of the ACM on Human-Computer Interaction10.1145/35932327:EICS(1-18)Online publication date: 19-Jun-2023
https://dl.acm.org/doi/10.1145/3593232
Chen WWang ZQuan PPeng ZLin SSrivastava MMatusik WStankovic J(2023)Robust Finger Interactions with COTS Smartwatches via Unsupervised Siamese AdaptationProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606794(1-14)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606794
Huh JShin HKim HCheon ESong YLee COakley I(2023)WristAcousticProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35694736:4(1-34)Online publication date: 11-Jan-2023
https://dl.acm.org/doi/10.1145/3569473
Xu CZhou BKrishnan GNayar S(2023)AO-Finger: Hands-free Fine-grained Finger Gesture Recognition via Acoustic-Optic Sensor FusingProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581264(1-14)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581264
Jannat MYang XHasan K(2023)Around-device finger input on commodity smartwatches with learning guidance through discoverabilityInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103105179:COnline publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1016/j.ijhcs.2023.103105
Noh HAhn CChae SKu YSim J(2022)Multichannel Acoustic Spectroscopy of the Human Body for Inviolable Biometric AuthenticationBiosensors10.3390/bios1209070012:9(700)Online publication date: 31-Aug-2022
https://doi.org/10.3390/bios12090700
Rateau HLank ELiu Z(2022)Leveraging Smartwatch and Earbuds Gesture Capture to Support Wearable InteractionProceedings of the ACM on Human-Computer Interaction10.1145/35677106:ISS(31-50)Online publication date: 14-Nov-2022
https://dl.acm.org/doi/10.1145/3567710
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents