Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3565970.3567698acmconferencesArticle/Chapter ViewAbstractPublication PagessuiConference Proceedingsconference-collections
research-article

MicroPress: Detecting Pressure and Hover Distance in Thumb-to-Finger Interactions

Published: 01 December 2022 Publication History

Abstract

Thumb-to-finger interactions leverage the thumb for precise, eyes-free input with high sensory bandwidth. While previous research explored gestures based on touch contact and finger movement on the skin, interactions leveraging depth such as pressure and hovering input are still underinvestigated. We present MicroPress, a proof-of-concept device that can detect both, precise thumb pressure applied on the skin and hover distance between the thumb and the index finger. We rely on a wearable IMU sensor array and a bi-directional RNN deep learning approach to enable fine-grained control while preserving the natural tactile feedback and touch of the skin. We demonstrate MicroPress’ efficacy with two interactive scenarios that pose challenges for real-time input and we validate its design with a study involving eight participants. With short per-user calibration steps, MicroPress is capable of predicting hover distance with 0.57mm accuracy, and on-skin pressure with normalized pressure error at 6 locations on the index finger.

Supplemental Material

MP4 File
Video Figure
MP4 File
Video figure
MP4 File
Video Figure
ZIP File
Study data and R analysis script
ZIP File
Supplementary materials
ZIP File
Supplementary materials

References

[1]
Daniel Ashbrook, Patrick Baudisch, and Sean White. 2011. Nenya: Subtle and Eyes-Free Mobile Input with a Magnetically-Tracked Finger Ring(CHI ’11). Association for Computing Machinery, New York, NY, USA, 2043–2046. https://doi.org/10.1145/1978942.1979238
[2]
aster94. 2021. Sensorfusion 1.0.3. https://www.arduino.cc/reference/en/libraries/sensorfusion/.
[3]
Alberto Boem and Giovanni Maria Troiano. 2019. Non-Rigid HCI: A Review of Deformable Interfaces and Input. In Proceedings of the 2019 on Designing Interactive Systems Conference (San Diego, CA, USA) (DIS ’19). Association for Computing Machinery, New York, NY, USA, 885–906. https://doi.org/10.1145/3322276.3322347
[4]
Bosch. 2021. Sensor Fusion. https://www.bosch-sensortec.com/software-tools/software/sensor-fusion-software/.
[5]
Jared Cechanowicz, Pourang Irani, and Sriram Subramanian. 2007. Augmenting the Mouse with Pressure Sensitive Input. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’07). ACM, New York, NY, USA, 1385–1394. https://doi.org/10.1145/1240624.1240835
[6]
Liwei Chan, Yi-Ling Chen, Chi-Hao Hsieh, Rong-Hao Liang, and Bing-Yu Chen. 2015. CyclopsRing: Enabling Whole-Hand and Context-Aware Interactions Through a Fisheye Ring. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (Charlotte, NC, USA) (UIST ’15). Association for Computing Machinery, New York, NY, USA, 549–556. https://doi.org/10.1145/2807442.2807450
[7]
Liwei Chan, Rong-Hao Liang, Ming-Chang Tsai, Kai-Yin Cheng, Chao-Huai Su, Mike Y. Chen, Wen-Huang Cheng, and Bing-Yu Chen. 2013. FingerPad: Private and Subtle Interaction Using Fingertips. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST ’13). Association for Computing Machinery, New York, NY, USA, 255–260. https://doi.org/10.1145/2501988.2502016
[8]
Ke-Yu Chen, Kent Lyons, Sean White, and Shwetak Patel. 2013. UTrack: 3D Input Using Two Magnetic Sensors(UIST ’13). Association for Computing Machinery, New York, NY, USA, 237–244. https://doi.org/10.1145/2501988.2502035
[9]
Ke-Yu Chen, Shwetak N. Patel, and Sean Keller. 2016. Finexus: Tracking Precise Motions of Multiple Fingertips Using Magnetic Sensing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 1504–1514. https://doi.org/10.1145/2858036.2858125
[10]
Victor Cheung, Jens Heydekorn, Stacey Scott, and Raimund Dachselt. 2012. Revisiting hovering: interaction guides for interactive surfaces. In Proceedings of the 2012 ACM international conference on Interactive tabletops and surfaces. 355–358.
[11]
Christian Corsten, Bjoern Daehlmann, Simon Voelker, and Jan Borchers. 2017. BackXPress: Using Back-of-Device Finger Pressure to Augment Touchscreen Input on Smartphones. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 4654–4666. https://doi.org/10.1145/3025453.3025565
[12]
Christian Corsten, Simon Voelker, and Jan Borchers. 2017. Release, Don’t Wait! Reliable Force Input Confirmation with Quick Release. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces(Brighton, United Kingdom) (ISS ’17). Association for Computing Machinery, New York, NY, USA, 246–251. https://doi.org/10.1145/3132272.3134116
[13]
Bruno Fruchard, Paul Strohmeier, Roland Bennewitz, and Jürgen Steimle. 2021. Squish This: Force Input on Soft Surfacesfor Visual Targeting Tasks. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 219, 9 pages. https://doi.org/10.1145/3411764.3445623
[14]
Ross B. Girshick. 2015. Fast R-CNN. CoRR abs/1504.08083(2015). arxiv:1504.08083http://arxiv.org/abs/1504.08083
[15]
Jun Gong, Yang Zhang, Xia Zhou, and Xing-Dong Yang. 2017. Pyro: Thumb-Tip Gesture Recognition Using Pyroelectric Infrared Sensing. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology(Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 553–563. https://doi.org/10.1145/3126594.3126615
[16]
Ken Hinckley, Seongkook Heo, Michel Pahud, Christian Holz, Hrvoje Benko, Abigail Sellen, Richard Banks, Kenton O’Hara, Gavin Smyth, and William Buxton. 2016. Pre-Touch Sensing for Mobile Interaction. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 2869–2881. https://doi.org/10.1145/2858036.2858095
[17]
Da-Yuan Huang, Liwei Chan, Shuo Yang, Fan Wang, Rong-Hao Liang, De-Nian Yang, Yi-Ping Hung, and Bing-Yu Chen. 2016. DigitSpace: Designing Thumb-to-Fingers Touch Interfaces for One-Handed and Eyes-Free Interactions(CHI ’16). Association for Computing Machinery, New York, NY, USA, 1526–1537. https://doi.org/10.1145/2858036.2858483
[18]
Yinghao Huang, Manuel Kaufmann, Emre Aksan, Michael J. Black, Otmar Hilliges, and Gerard Pons-Moll. 2018. Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time. ACM Trans. Graph. 37, 6, Article 185 (Dec. 2018), 15 pages. https://doi.org/10.1145/3272127.3275108
[19]
Snap Inc.2022. Spectacles. https://www.spectacles.com/fr/.
[20]
Yu Ishikawa, Buntarou Shizuki, and Junichi Hoshino. 2017. Evaluation of Finger Position Estimation with a Small Ranging Sensor Array. In Proceedings of the 5th Symposium on Spatial User Interaction (Brighton, United Kingdom) (SUI ’17). Association for Computing Machinery, New York, NY, USA, 120–127. https://doi.org/10.1145/3131277.3132176
[21]
Keiko Katsuragawa, Ju Wang, Ziyang Shan, Ningshan Ouyang, Omid Abari, and Daniel Vogel. 2019. Tip-Tap: Battery-Free Discrete 2D Fingertip Input. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 1045–1057. https://doi.org/10.1145/3332165.3347907
[22]
Minwoo Kim, Jaechan Cho, Seongjoo Lee, and Yunho Jung. 2019. IMU Sensor-Based Hand Gesture Recognition for Human-Machine Interfaces. Sensors 19, 18 (2019). https://doi.org/10.3390/s19183827
[23]
Kenneth Lai and Svetlana N Yanushkevich. 2018. CNN+ RNN depth and skeleton based dynamic hand gesture recognition. In 2018 24th International Conference on Pattern Recognition (ICPR). IEEE, 3451–3456.
[24]
Matthew Lakier, Dimcho Karakashev, Yixin Wang, and Ian Goldberg. 2020. Augmented Unlocking Techniques for Smartphones Using Pre-Touch Information. In Symposium on Spatial User Interaction (Virtual Event, Canada) (SUI ’20). Association for Computing Machinery, New York, NY, USA, Article 17, 5 pages. https://doi.org/10.1145/3385959.3418455
[25]
Ke Li, Raviraj Nataraj, Tamara L. Marquardt, and Zong-Ming Li. 2013. Directional Coordination of Thumb and Finger Forces during Precision Pinch. PLOS ONE 8, 11 (11 2013), 1–8. https://doi.org/10.1371/journal.pone.0079400
[26]
Z. Li, Z. Lei, A. Yan, E. Solovey, and K. Pahlavan. 2020. ThuMouse: A Micro-gesture Cursor Input through mmWave Radar-based Interaction. In 2020 IEEE International Conference on Consumer Electronics (ICCE). 1–9. https://doi.org/10.1109/ICCE46568.2020.9043082
[27]
Jaime Lien, Nicholas Gillian, M. Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous Gesture Sensing with Millimeter Wave Radar. ACM Trans. Graph. 35, 4, Article 142 (July 2016), 19 pages. https://doi.org/10.1145/2897824.2925953
[28]
Christian Loclair, Sean Gustafson, and Patrick Baudisch. 2010. PinchWatch: a wearable device for one-handed microinteractions. In Proc. MobileHCI, Vol. 10. Citeseer.
[29]
Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. arxiv:1711.05101 [cs.LG]
[30]
Jess McIntosh, Paul Strohmeier, Jarrod Knibbe, Sebastian Boring, and Kasper Hornbæk. 2019. Magnetips: Combining Fingertip Tracking and Haptic Feedback for Around-Device Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300638
[31]
Microsoft. 2022. Hololens. https://www.microsoft.com/en-us/hololens.
[32]
Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt. 2017. Real-Time Hand Tracking Under Occlusion From an Egocentric RGB-D Sensor. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
[33]
Florian ’Floyd’ Mueller, Richard Byrne, Josh Andres, and Rakesh Patibanda. 2018. Experiencing the Body as Play. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3173784
[34]
Aditya Shekhar Nittala, Anusha Withana, Narjes Pourjafarian, and Jürgen Steimle. 2018. Multi-touch skin: A thin and flexible multi-touch sensor for on-skin input. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–12.
[35]
Keunwoo Park, Sunbum Kim, Youngwoo Yoon, Tae-Kyun Kim, and Geehyuk Lee. 2020. DeepFisheye: Near-Surface Multi-Finger Tracking Technology Using Fisheye Camera. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology(Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 1132–1146. https://doi.org/10.1145/3379337.3415818
[36]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32. Curran Associates, Inc., 8024–8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
[37]
F. H. Raab, E. B. Blood, T. O. Steiner, and H. R. Jones. 1979. Magnetic Position and Orientation Tracking System. IEEE Trans. Aerospace Electron. Systems AES-15, 5 (1979), 709–718. https://doi.org/10.1109/TAES.1979.308860
[38]
Gonzalo Ramos, Matthew Boulos, and Ravin Balakrishnan. 2004. Pressure Widgets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria) (CHI ’04). ACM, New York, NY, USA, 487–494. https://doi.org/10.1145/985692.985754
[39]
Christina Salchow-Hömmen, Leonie Callies, Daniel Laidig, Markus Valtin, Thomas Schauer, and Thomas Seel. 2019. A Tangible Solution for Hand Motion Tracking in Clinical Applications. Sensors 19, 1. https://doi.org/10.3390/s19010208
[40]
Kang Shi, Pourang Irani, Sean Gustafson, and Sriram Subramanian. 2008. PressureFish: A Method to Improve Control of Discrete Pressure-based Input. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy) (CHI ’08). ACM, New York, NY, USA, 1295–1298. https://doi.org/10.1145/1357054.1357256
[41]
Mohamed Soliman, Franziska Mueller, Lena Hegemann, Joan Sol Roo, Christian Theobalt, and Jürgen Steimle. 2018. FingerInput: Capturing Expressive Single-Hand Thumb-to-Finger Microgestures. In Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces(Tokyo, Japan) (ISS ’18). Association for Computing Machinery, New York, NY, USA, 177–187. https://doi.org/10.1145/3279778.3279799
[42]
Paul Strohmeier, Jarrod Knibbe, Sebastian Boring, and Kasper Hornbæk. 2018. ZPatch: Hybrid Resistive/Capacitive ETextile Input. In Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction (Stockholm, Sweden) (TEI ’18). Association for Computing Machinery, New York, NY, USA, 188–198. https://doi.org/10.1145/3173225.3173242
[43]
Hsin-Ruey Tsai, Cheng-Yuan Wu, Lee-Ting Huang, and Yi-Ping Hung. 2016. ThumbRing: Private Interactions Using One-Handed Thumb Motion Input on Finger Segments. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct(Florence, Italy) (MobileHCI ’16). Association for Computing Machinery, New York, NY, USA, 791–798. https://doi.org/10.1145/2957265.2961859
[44]
Roel Vertegaal and Ivan Poupyrev. 2008. Organic user interfaces. Commun. ACM 51, 6 (2008), 26–30.
[45]
Nicolas Villar, Daniel Cletheroe, Greg Saul, Christian Holz, Tim Regan, Oscar Salandin, Misha Sra, Hui-Shyong Yeo, William Field, and Haiyan Zhang. 2018. Project Zanzibar: A Portable and Flexible Tangible Interaction Platform. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3174089
[46]
Saiwen Wang, Jie Song, Jaime Lien, Ivan Poupyrev, and Otmar Hilliges. 2016. Interacting with Soli: Exploring Fine-Grained Dynamic Gesture Recognition in the Radio-Frequency Spectrum. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST ’16). Association for Computing Machinery, New York, NY, USA, 851–860. https://doi.org/10.1145/2984511.2984565
[47]
Martin Weigel, Vikram Mehta, and Jürgen Steimle. 2014. More than Touch: Understanding How People Use Skin as an Input Surface for Mobile Computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 179–188. https://doi.org/10.1145/2556288.2557239
[48]
Martin Weigel, Aditya Shekhar Nittala, Alex Olwal, and Jürgen Steimle. 2017. Skinmarks: Enabling interactions on body landmarks using conformal skin electronics. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 3095–3105.
[49]
Eric Whitmire, Mohit Jain, Divye Jain, Greg Nelson, Ravi Karkar, Shwetak Patel, and Mayank Goel. 2017. DigiTouch: Reconfigurable Thumb-to-Finger Input and Text Entry on Head-Mounted Displays. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 113 (Sept. 2017), 21 pages. https://doi.org/10.1145/3130978
[50]
Graham Wilson, Craig Stewart, and Stephen A. Brewster. 2010. Pressure-based Menu Selection for Mobile Devices. In Proceedings of the 12th International Conference on Human Computer Interaction with Mobile Devices and Services (Lisbon, Portugal) (MobileHCI ’10). ACM, New York, NY, USA, 181–190. https://doi.org/10.1145/1851600.1851631
[51]
Zheer Xu, Pui Chung Wong, Jun Gong, Te-Yen Wu, Aditya Shekhar Nittala, Xiaojun Bi, Jürgen Steimle, Hongbo Fu, Kening Zhu, and Xing-Dong Yang. 2019. TipText: Eyes-Free Text Entry on a Fingertip Keyboard. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 883–899. https://doi.org/10.1145/3332165.3347865
[52]
Hui-Shyong Yeo, Erwin Wu, Juyoung Lee, Aaron Quigley, and Hideki Koike. 2019. Opisthenar: Hand Poses and Finger Tapping Recognition by Observing Back of Hand Using Embedded Wrist Camera. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 963–971. https://doi.org/10.1145/3332165.3347867
[53]
Shumin Zhai, Paul Milgram, and William Buxton. 1996. The Influence of Muscle Groups on Performance of Multiple Degree-of-Freedom Input. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, British Columbia, Canada) (CHI ’96). Association for Computing Machinery, New York, NY, USA, 308–315. https://doi.org/10.1145/238386.238534
[54]
Yang Zhang and Chris Harrison. 2015. Tomo: Wearable, Low-Cost Electrical Impedance Tomography for Hand Gesture Recognition. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (Charlotte, NC, USA) (UIST ’15). Association for Computing Machinery, New York, NY, USA, 167–173. https://doi.org/10.1145/2807442.2807480
[55]
Junhan Zhou, Yang Zhang, Gierad Laput, and Chris Harrison. 2016. AuraSense: Enabling Expressive Around-Smartwatch Interactions with Electric Field Sensing. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST ’16). Association for Computing Machinery, New York, NY, USA, 81–86. https://doi.org/10.1145/2984511.2984568

Cited By

View all
  • (2024)Thumb-to-Finger Gesture Recognition Using COTS Smartwatch AccelerometersProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3701600(184-195)Online publication date: 1-Dec-2024
  • (2024)vARitouch: Back of the Finger Device for Adding Variable Compliance to Rigid ObjectsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642828(1-20)Online publication date: 11-May-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SUI '22: Proceedings of the 2022 ACM Symposium on Spatial User Interaction
December 2022
233 pages
ISBN:9781450399487
DOI:10.1145/3565970
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 December 2022

Permissions

Request permissions for this article.

Check for updates

Badges

  • Honorable Mention

Author Tags

  1. Deep Learning
  2. Gestural Input
  3. Hover Input
  4. IMU
  5. LSTM
  6. Pressure Input
  7. RNN
  8. Thumb-to-Finger Interaction

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

Conference

SUI '22
SUI '22: Symposium on Spatial User Interaction
December 1 - 2, 2022
CA, Online, USA

Acceptance Rates

Overall Acceptance Rate 86 of 279 submissions, 31%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)107
  • Downloads (Last 6 weeks)1
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Thumb-to-Finger Gesture Recognition Using COTS Smartwatch AccelerometersProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3701600(184-195)Online publication date: 1-Dec-2024
  • (2024)vARitouch: Back of the Finger Device for Adding Variable Compliance to Rigid ObjectsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642828(1-20)Online publication date: 11-May-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media