Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

Keep the Phone in Your Pocket: Enabling Smartphone Operation with an IMU Ring for Visually Impaired People

Published: 15 June 2020 Publication History

Abstract

Previous studies have shown that visually impaired users face a unique set of pain points in smartphone interaction including locating and removing the phone from a pocket, two-handed interaction while holding a cane, and keeping personal data private in a public setting. In this paper, we present a ring-based input interaction that enables in-pocket smartphone operation. By wearing a ring with an Inertial Measurement Unit on the index finger, users can perform gestures on any surface (e.g., tables, thighs) using subtle, one-handed gestures and receive auditory feedback via earphones. We conducted participatory studies to obtain a set of versatile commands and corresponding gestures. We subsequently trained an SVM model to recognize these gestures and achieved a mean accuracy of 95.5% on 15 classifications. Evaluation results showed that our ring interaction is more efficient than some baseline phone interactions and is easy, private, and fun to use.

Supplementary Material

liu (liu.zip)
Supplemental movie, appendix, image and software files for, Keep the Phone in Your Pocket: Enabling Smartphone Operation with an IMU Ring for Visually Impaired People

References

[1]
Daniel Ashbrook, Patrick Baudisch, and Sean White. 2011. Nenya: subtle and eyes-free mobile input with a magnetically-tracked finger ring. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2043--2046.
[2]
Andrea Bianchi and Seungwoo Je. 2017. Disambiguating touch with a smart-ring. In Proceedings of the 8th Augmented Human International Conference. ACM, 27.
[3]
Roger Boldu, Alexandru Dancu, Denys JC Matthies, Thisum Buddhika, Shamane Siriwardhana, and Suranga Nanayakkara. 2018. FingerReader2. 0: Designing and Evaluating a Wearable Finger-Worn Camera to Assist People with Visual Impairments while Shopping. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 94.
[4]
Liwei Chan, Yi-Ling Chen, Chi-Hao Hsieh, Rong-Hao Liang, and Bing-Yu Chen. 2015. Cyclopsring: Enabling whole-hand and context-aware interactions through a fisheye ring. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 549--556.
[5]
Ke-Yu Chen, Kent Lyons, Sean White, and Shwetak Patel. 2013. uTrack: 3D input using two magnetic sensors. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 237--244.
[6]
Jungmin Chung, Changhoon Oh, SoHyun Park, and Bongwon Suh. 2018. PairRing: A Ring-Shaped Rotatable Smartwatch Controller. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, LBW635.
[7]
Nicola Dell, Krittika D'Silva, and Gaetano Borriello. 2015. Mobile Touch-Free Interaction for Global Health (HotMobile '15). ACM, New York, NY, USA, 15--20. https://doi.org/10.1145/2699343.2699355
[8]
Catherine Feng. 2016. Designing Wearable Mobile Device Controllers for Blind People: A Co-Design Approach. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 341--342.
[9]
Bogdan-Florin Gheran, Radu-Daniel Vatavu, and Jean Vanderdonckt. 2018. Ring x2: Designing Gestures for Smart Rings using Temporal Calculus. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 117--122.
[10]
Yizheng Gu, Chun Yu, Zhipeng Li, Weiqi Li, Xiaoying Wei, and Yuanchun Shi. 2019. Accurate and Low-Latency Sensing of Touch Contact on Any Surface with Finger-Worn IMU Sensor. In Proceedings of the 32th Annual Symposium on User Interface Software and Technology. ACM.
[11]
Sean Gustafson, Daniel Bierwirth, and Patrick Baudisch. 2010. Imaginary interfaces: spatial interaction with empty hands and without visual feedback. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 3--12.
[12]
Sean Gustafson, Christian Holz, and Patrick Baudisch. 2011. Imaginary phone: learning imaginary interfaces by transferring spatial memory from a familiar device. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 283--292.
[13]
Sean G Gustafson, Bernhard Rabe, and Patrick M Baudisch. 2013. Understanding palm-based imaginary interfaces: the role of visual and tactile cues when browsing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 889--898.
[14]
Chris Harrison, Hrvoje Benko, and Andrew D Wilson. 2011. OmniTouch: wearable multitouch interaction everywhere. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 441--450.
[15]
Chris Harrison, Desney Tan, and Dan Morris. 2010. Skinput: appropriating the body as an input surface. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 453--462.
[16]
Scott E Hudson, Chris Harrison, Beverly L Harrison, and Anthony LaMarca. 2010. Whack gestures: inexact and inattentive interaction with mobile devices. In Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction. ACM, 109--112.
[17]
Seungwoo Je, Minkyeong Lee, Yoonji Kim, Liwei Chan, and Andrea Bianchi. 2018. PokeRing: Notifications by Poking Around the Finger. In Chi Conference.
[18]
Lei Jing, Zixue Cheng, Yinghui Zhou, Junbo Wang, and Tongjun Huang. 2013. Magic Ring: a self-contained gesture input device on finger. In Proceedings of the 12th International Conference on Mobile and Ubiquitous Multimedia. ACM, 39.
[19]
Wolf Kienzle and Ken Hinckley. 2014. LightRing: always-available 2D input on any surface. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 157--160.
[20]
Lawrence H Kim and Sean Follmer. 2019. SwarmHaptics: Haptic Display with Swarm Robots. (2019), 688.
[21]
Gierad Laput, Robert Xiao, and Chris Harrison. 2016. Viband: High-fidelity bio-acoustic sensing using commodity smartwatch accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 321--333.
[22]
Hyunchul Lim, Jungmin Chung, Changhoon Oh, SoHyun Park, and Bongwon Suh. 2016. OctaRing: Examining Pressure-Sensitive Multi-Touch Input on a Finger Ring Device. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 223--224.
[23]
Guanhong Liu, Xianghua Ding, Chun Yu, Lan Gao, Xingyu Chi, and Yuanchun Shi. 2019. " I Bought This for Me to Look More Ordinary": A Study of Blind People Doing Online Shopping. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 372.
[24]
Sebastian Madgwick. 2010. An efficient orientation filter for inertial and inertial/magnetic sensor arrays. Report x-io and University of Bristol (UK) 25 (2010), 113--118.
[25]
Maia Naftali and Leah Findlater. 2014. Accessibility in context: understanding the truly mobile experience of smartphone users with motor impairments. In Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility. ACM, 209--216.
[26]
Suranga Nanayakkara, Roy Shilkrot, Kian Peen Yeo, and Pattie Maes. 2013. EyeRing: a finger-worn input device for seamless interactions with our surroundings. In Proceedings of the 4th Augmented Human International Conference. ACM, 13--20.
[27]
Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. Fingerio: Using active sonar for fine-grained finger tracking. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1515--1525.
[28]
Tao Ni and Patrick Baudisch. 2009. Disappearing mobile devices. In Proceedings of the 22nd annual ACM symposium on User interface software and technology. ACM, 101--110.
[29]
Masa Ogata, Yuta Sugiura, Hirotaka Osawa, and Michita Imai. 2012. iRing: intelligent ring using infrared reflection. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 131--136.
[30]
Uran Oh and Leah Findlater. 2015. A performance comparison of on-hand versus on-phone nonvisual input by blind and sighted users. ACM Transactions on Accessible Computing (TACCESS) 7, 4 (2015), 14.
[31]
Uran Oh, Lee Stearns, Alisha Pradhan, Jon E Froehlich, and Leah Findlater. 2017. Investigating microinteractions for people with visual impairments and the potential role of on-body interaction. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. 22--31.
[32]
Shishir G Patil, Don Kurian Dennis, Chirag Pabbaraju, Nadeem Shaheer, Harsha Vardhan Simhadri, Vivek Seshadri, Manik Varma, and Prateek Jain. 2019. Gesturepod: Enabling on-device gesture-based interaction for white cane users. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 403--415.
[33]
Sami Ronkainen, Jonna Häkkilä, Saana Kaleva, Ashley Colley, and Jukka Linjama. 2007. Tap input as an embedded interaction method for mobile devices. In Proceedings of the 1st international conference on Tangible and embedded interaction. ACM, 263--270.
[34]
T. Scott Saponas, Chris Harrison, and Hrvoje Benko. 2011. PocketTouch: Through-fabric Capacitive Touch Input (UIST '11). ACM, New York, NY, USA, 303--308. https://doi.org/10.1145/2047196.2047235
[35]
Lee Stearns, Uran Oh, Leah Findlater, and Jon E Froehlich. 2018. TouchCam: Realtime Recognition of Location-Specific On-Body Gestures to Support Users with Visual Impairments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 164.
[36]
Ke Sun, Chun Yu, Weinan Shi, Lan Liu, and Yuanchun Shi. 2018. Lip-Interact: Improving Mobile Device Interaction with Silent Speech Commands. In The 31st Annual ACM Symposium on User Interface Software and Technology. ACM, 581--593.
[37]
Hsin-Ruey Tsai, Min-Chieh Hsiu, Jui-Chun Hsiao, Lee-Ting Huang, Mike Chen, and Yi-Ping Hung. 2016. TouchRing: subtle and always-available input using a multi-touch ring. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct. ACM, 891--898.
[38]
Hsin-Ruey Tsai, Cheng-Yuan Wu, Lee-Ting Huang, and Yi-Ping Hung. 2016. ThumbRing: private interactions using one-handed thumb motion input on finger segments. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct. ACM, 791--798.
[39]
Ruolin Wang, Chun Yu, Xing-Dong Yang, Weijie He, and Yuanchun Shi. 2019. EarTouch: Facilitating Smartphone Use for Visually Impaired People in Mobile and Public Scenarios. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 24.
[40]
Hongyi Wen, Julian Ramos Rojas, and Anind K Dey. 2016. Serendipity: Finger gesture recognition using an off-the-shelf smartwatch. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 3847--3851.
[41]
Mathias Wilhelm, Daniel Krakowczyk, Frank Trollmann, and Sahin Albayrak. 2015. eRing: multiple finger gesture recognition with one ring using an electric field. In Proceedings of the 2nd international Workshop on Sensor-based Activity Recognition and Interaction. ACM, 7.
[42]
Katrin Wolf and Jonas Willaredt. 2015. PickRing: seamless interaction through pick-up detection. In Proceedings of the 6th augmented human international conference. ACM, 13--20.
[43]
Hanlu Ye, Meethu Malu, Uran Oh, and Leah Findlater. 2014. Current and future mobile and wearable device use by people with visual impairments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 3123--3132.
[44]
Sang Ho Yoon, Ke Huo, Vinh P Nguyen, and Karthik Ramani. 2015. TIMMi: Finger-worn Textile Input Device with Multimodal Sensing in Mobile Interaction. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction. ACM, 269--272.
[45]
Sang Ho Yoon, Ke Huo, and Karthik Ramani. 2014. Plex: finger-worn textile sensor for mobile interaction during activities. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication. ACM, 191--194.
[46]
Sang Ho Yoon, Yunbo Zhang, Ke Huo, and Karthik Ramani. 2016. TRing: Instant and Customizable Interactions with Objects Using an Embedded Magnet and a Finger-Worn Device. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 169--181.
[47]
Boning Zhang, Yiqiang Chen, Yueliang Qian, and Xiangdong Wang. 2011. A ring-shaped interactive device for large remote display and mobile device control. In Proceedings of the 13th international conference on Ubiquitous computing. ACM, 473--474.
[48]
Cheng Zhang, Anandghan Waghmare, Pranav Kundra, Yiming Pu, Scott Gilliland, Thomas Ploetz, Thad E Starner, Omer T Inan, and Gregory D Abowd. 2017. FingerSound: Recognizing unistroke thumb gestures using a ring. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 120.
[49]
Cheng Zhang, Xiaoxuan Wang, Anandghan Waghmare, Sumeet Jain, Thomas Ploetz, Omer T Inan, Thad E Starner, and Gregory D Abowd. 2017. FingOrbits: interaction with wearables using synchronized thumb movements. In Proceedings of the 2017 ACM International Symposium on Wearable Computers. ACM, 62--65.
[50]
Maotian Zhang, Ping Li, Panlong Yang, Jie Xiong, and Chang Tian. 2016. Poster: Sonicnect: Accurate Hands-Free Gesture Input System with Smart Acoustic Sensing (MobiSys '16 Companion). ACM, New York, NY, USA, 91--91. https://doi.org/10.1145/2938559.2948830
[51]
Yang Zhang, Junhan Zhou, Gierad Laput, and Chris Harrison. 2016. Skintrack: Using the body as an electrical waveguide for continuous finger tracking on the skin. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1491--1503.

Cited By

View all
  • (2023)OptiRing: Low-Resolution Optical Sensing for Subtle Thumb-to-Index Micro-InteractionsProceedings of the 2023 ACM Symposium on Spatial User Interaction10.1145/3607822.3614538(1-13)Online publication date: 13-Oct-2023
  • (2023)UnifiedSense: Enabling Without-Device Gesture Interactions Using Over-the-shoulder Training Between Redundant Wearable SensorsProceedings of the ACM on Human-Computer Interaction10.1145/36042777:MHCI(1-25)Online publication date: 13-Sep-2023
  • (2023)µGeT: Multimodal eyes-free text selection technique combining touch interaction and microgesturesProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614131(594-603)Online publication date: 9-Oct-2023
  • Show More Cited By

Index Terms

  1. Keep the Phone in Your Pocket: Enabling Smartphone Operation with an IMU Ring for Visually Impaired People

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
      Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 4, Issue 2
      June 2020
      771 pages
      EISSN:2474-9567
      DOI:10.1145/3406789
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 15 June 2020
      Published in IMWUT Volume 4, Issue 2

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. accessibility
      2. gestural input

      Qualifiers

      • Research-article
      • Research
      • Refereed

      Funding Sources

      • National Key Research and Development Plan
      • National Natural Science Foundation of China

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)114
      • Downloads (Last 6 weeks)7
      Reflects downloads up to 10 Nov 2024

      Other Metrics

      Citations

      Cited By

      View all
      • (2023)OptiRing: Low-Resolution Optical Sensing for Subtle Thumb-to-Index Micro-InteractionsProceedings of the 2023 ACM Symposium on Spatial User Interaction10.1145/3607822.3614538(1-13)Online publication date: 13-Oct-2023
      • (2023)UnifiedSense: Enabling Without-Device Gesture Interactions Using Over-the-shoulder Training Between Redundant Wearable SensorsProceedings of the ACM on Human-Computer Interaction10.1145/36042777:MHCI(1-25)Online publication date: 13-Sep-2023
      • (2023)µGeT: Multimodal eyes-free text selection technique combining touch interaction and microgesturesProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614131(594-603)Online publication date: 9-Oct-2023
      • (2023)EFRingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35694786:4(1-31)Online publication date: 11-Jan-2023
      • (2023)DRG-KeyboardProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35694636:4(1-30)Online publication date: 11-Jan-2023
      • (2022)Keep in Touch: Combining Touch Interaction with Thumb-to-Finger µGestures for People with Visual ImpairmentProceedings of the 2022 International Conference on Multimodal Interaction10.1145/3536221.3556589(105-116)Online publication date: 7-Nov-2022
      • (2022)FootUI: Designing and Detecting Foot Gestures to Assist People with Upper Body Motor Impairments to Use Smartphones on the BedProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3563285(1-11)Online publication date: 23-Oct-2022
      • (2022)A fine-grained gesture tracking system based on millimeter-waveCCF Transactions on Pervasive Computing and Interaction10.1007/s42486-022-00119-04:4(357-369)Online publication date: 28-Nov-2022
      • (2021)DualRingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/34781145:3(1-27)Online publication date: 14-Sep-2021
      • (2021)GestuRING: A Web-based Tool for Designing Gesture Input with Rings, Ring-Like, and Ring-Ready DevicesThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474780(710-723)Online publication date: 10-Oct-2021
      • Show More Cited By

      View Options

      Get Access

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media