Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3313831.3376198acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article
Open access

WATouCH: Enabling Direct Input on Non-touchscreen Using Smartwatch's Photoplethysmogram and IMU Sensor Fusion

Published: 23 April 2020 Publication History

Abstract

Interacting with non-touchscreens such as TV or public displays can be difficult and inefficient. We propose WATouCH, a novel method that localizes a smartwatch on a display and allows direct input by turning the smartwatch into a tangible controller. This low-cost solution leverages sensor fusion of the built-in inertial measurement unit (IMU) and photoplethysmogram (PPG) sensor on a smartwatch that is used for heart rate monitoring. Specifically, WATouCH tracks the smartwatch movement using IMU data and corrects its location error caused by drift using the PPG responses to a dynamic visual pattern on the display. We conducted a user study on two tasks -- a point and click and line tracing task -- to evaluate the system usability and user performance. Evaluation results suggested that our sensor fusion mechanism effectively confined IMU-based localization error, achieved encouraging targeting and tracing precision, was well received by the participants, and thus opens up new opportunities for interaction.

Supplementary Material

MP4 File (paper071vf.mp4)
Supplemental video
MP4 File (paper071pv.mp4)
Preview video

References

[1]
Rafael Ballagas, Michael Rohs, and Jennifer G. Sheridan. 2005. Sweep and Point and Shoot: Phonecam-Based Interactions for Large Public Displays. In CHI '05 Extended Abstracts on Human Factors in Computing Systems (CHI EA '05). Association for Computing Machinery, New York, NY, USA, 1200--1203. http://dx.doi.org/10.1145/1056808.1056876
[2]
Eric A. Bier, Maureen C. Stone, Ken Pier, William Buxton, and Tony D. DeRose. 1993. Toolglass and Magic Lenses: The See-through Interface. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '93). Association for Computing Machinery, New York, NY, USA, 73--80. http://dx.doi.org/10.1145/166117.166126
[3]
Sebastian Boring, Marko Jurmu, and Andreas Butz. 2009. Scroll, Tilt or Move It: Using Mobile Phones to Continuously Control Pointers on Large Public Displays. In Proceedings of the 21st Annual Conference of the Australian Computer-Human Interaction Special Interest Group: Design: Open 24/7 (OZCHI '09). Association for Computing Machinery, New York, NY, USA, 161--168. http://dx.doi.org/10.1145/1738826.1738853
[4]
Thisum Buddhika, Haimo Zhang, Samantha W. T. Chan, Vipula Dissanayake, Suranga Nanayakkara, and Roger Zimmermann. 2019. fSense: Unlocking the Dimension of Force for Gestural Interactions Using Smartwatch PPG Sensor. In Proceedings of the 10th Augmented Human International Conference 2019 (AH2019). ACM, New York, NY, USA, Article 11, 5 pages. http://dx.doi.org/10.1145/3311823.3311839
[5]
E. Foxlin. 2005. Pedestrian tracking with shoe-mounted inertial sensors. IEEE Computer Graphics and Applications 25, 6 (Nov 2005), 38--46. http://dx.doi.org/10.1109/MCG.2005.140
[6]
Jefferson Y. Han. 2005. Low-cost Multi-touch Sensing Through Frustrated Total Internal Reflection. In Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology (UIST '05). ACM, New York, NY, USA, 115--118. http://dx.doi.org/10.1145/1095034.1095054
[7]
Chris Harrison, Hrvoje Benko, and Andrew D. Wilson. 2011. OmniTouch: Wearable Multitouch Interaction Everywhere. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 441--450. http://dx.doi.org/10.1145/2047196.2047255
[8]
Ken Hinckley and Daniel Wigdor. 2002. Input technologies and techniques. The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications (2002), 151--168.
[9]
Steve Hodges, Shahram Izadi, Alex Butler, Alban Rrustemi, and Bill Buxton. 2007. ThinSight: Versatile Multi-touch Sensing for Thin Form-factor Displays. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST '07). ACM, New York, NY, USA, 259--268. http://dx.doi.org/10.1145/1294211.1294258
[10]
Meng-Ju Hsieh, Rong-Hao Liang, Da-Yuan Huang, Jheng-You Ke, and Bing-Yu Chen. 2018. RFIBricks: Interactive Building Blocks Based on RFID. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 189, 10 pages. http://dx.doi.org/10.1145/3173574.3173763
[11]
Microsoft in Business. 2015. Microsoft: Productivity Future Vision. (2015). Retrieved September 14, 2019 from https://www.youtube.com/watch?v=w-tFdreZB94.
[12]
Hiroshi Ishii and Brygg Ullmer. 1997. Tangible Bits: Towards Seamless Interfaces Between People, Bits and Atoms. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI '97). ACM, New York, NY, USA, 234--241. http://dx.doi.org/10.1145/258549.258715
[13]
Keiko Katsuragawa, Krzysztof Pietroszek, James R. Wallace, and Edward Lank. 2016. Watchpoint: Freehand Pointing with a Smartwatch in a Ubiquitous Display Environment. In Proceedings of the International Working Conference on Advanced Visual Interfaces (AVI '16). ACM, New York, NY, USA, 128--135. http://dx.doi.org/10.1145/2909132.2909263
[14]
U Kawamoto, Takeshi Kurata, N Sakata, Takashi Okuma, and Hideaki Kuzuoka. 2008. Position/orientation-aware physical tags using photo sensors and accelerometers for a tangible tabletop interface. In Proc. IEEE International Conference on Distributed Human-Machine Systems (DHMS). 397--404.
[15]
Daniel Kharlamov, Brandon Woodard, Liudmila Tahai, and Krzysztof Pietroszek. 2016. TickTockRay: Smartwatch-based 3D Pointing for Smartphone-based Virtual Reality. In Proceedings of the 22Nd ACM Conference on Virtual Reality Software and Technology (VRST '16). ACM, New York, NY, USA, 365--366. http://dx.doi.org/10.1145/2993369.2996311
[16]
Rushil Khurana, Mayank Goel, and Kent Lyons. 2019. Detachable Smartwatch: More Than A Wearable. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 2, Article 50 (June 2019), 14 pages. http://dx.doi.org/10.1145/3328921
[17]
Per-Ola Kristensson and Shumin Zhai. 2004. SHARK2: A Large Vocabulary Shorthand Writing System for Pen-based Computers. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (UIST '04). ACM, New York, NY, USA, 43--52. http://dx.doi.org/10.1145/1029632.1029640
[18]
Mikko Kytö, Barrett Ens, Thammathip Piumsomboon, Gun A. Lee, and Mark Billinghurst. 2018. Pinpointing: Precise Head- and Eye-Based Target Selection for Augmented Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 81, 14 pages. http://dx.doi.org/10.1145/3173574.3173655
[19]
Gierad Laput, Robert Xiao, and Chris Harrison. 2016. ViBand: High-Fidelity Bio-Acoustic Sensing Using Commodity Smartwatch Accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 321--333. http://dx.doi.org/10.1145/2984511.2984582
[20]
J. Lee, K. Matsumura, K. Yamakoshi, P. Rolfe, S. Tanaka, and T. Yamakoshi. 2013. Comparison between red, green and blue light reflection photoplethysmography for heart rate monitoring during motion. In 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 1724--1727. http://dx.doi.org/10.1109/EMBC.2013.6609852
[21]
Johnny C. Lee, Paul H. Dietz, Dan Maynes-Aminzade, Ramesh Raskar, and Scott E. Hudson. 2004. Automatic Projector Calibration with Embedded Light Sensors. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (UIST '04). ACM, New York, NY, USA, 123--126. http://dx.doi.org/10.1145/1029632.1029653
[22]
Johnny C. Lee, Scott E. Hudson, Jay W. Summet, and Paul H. Dietz. 2005. Moveable Interactive Projected Displays Using Projector Based Tracking. In Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology (UIST '05). ACM, New York, NY, USA, 63--72. http://dx.doi.org/10.1145/1095034.1095045
[23]
Sang-won Leigh, Philipp Schoessler, Felix Heibeck, Pattie Maes, and Hiroshi Ishii. 2015. THAW: Tangible Interaction with See-Through Augmentation for Smartphones on Computer Screens. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '15). ACM, New York, NY, USA, 89--96. http://dx.doi.org/10.1145/2677199.2680584
[24]
LEMFO. 2019. LEMFO LEM11 4G Smart Watch Android 7.1 3GB 32GB Video Call With 1200mah Power Bank Wireless Bluetooth Speaker Strap Replaceable. (2019). Retrieved December 26, 2019 from https://www.aliexpress.com/item/4000141268956.html.
[25]
Rong-Hao Liang, Kai-Yin Cheng, Liwei Chan, Chuan-Xhyuan Peng, Mike Y. Chen, Rung-Huei Liang, De-Nian Yang, and Bing-Yu Chen. 2013. GaussBits: Magnetic Tangible Bits for Portable and Occlusion-free Near-surface Interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 1391--1400. http://dx.doi.org/10.1145/2470654.2466185
[26]
S. O. H. Madgwick, A. J. L. Harrison, and R. Vaidyanathan. 2011. Estimation of IMU and MARG orientation using a gradient descent algorithm. In 2011 IEEE International Conference on Rehabilitation Robotics. 1--7. http://dx.doi.org/10.1109/ICORR.2011.5975346
[27]
Rafael Morales González, Caroline Appert, Gilles Bailly, and Emmanuel Pietriga. 2016. TouchTokens: Guiding Touch Patterns with Passive Tokens. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 4189--4202. http://dx.doi.org/10.1145/2858036.2858041
[28]
J. Müller, A. Schmidt, and F. Alt. 2012. Advertising on Public Display Networks. Computer 45, 05 (may 2012), 50--56. http://dx.doi.org/10.1109/MC.2012.150
[29]
Alex Olwal. 2006. LightSense: Enabling Spatially Aware Handheld Interaction Devices. In Proceedings of the 5th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR '06). IEEE Computer Society, Washington, DC, USA, 119--122. http://dx.doi.org/10.1109/ISMAR.2006.297802
[30]
James Patten, Hiroshi Ishii, Jim Hines, and Gian Pangaro. 2001. Sensetable: A Wireless Object Tracking Platform for Tangible User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '01). ACM, New York, NY, USA, 253--260. http://dx.doi.org/10.1145/365024.365112
[31]
Abdul Moiz Penkar, Christof Lutteroth, and Gerald Weber. 2012. Designing for the Eye: Design Parameters for Dwell in Gaze Interaction. In Proceedings of the 24th Australian Computer-Human Interaction Conference (OzCHI '12). ACM, New York, NY, USA, 479--488. http://dx.doi.org/10.1145/2414536.2414609
[32]
POISE. 2018. POISE Detachable Smartwatch. (2018). Retrieved September 14, 2019 from https://www.indiegogo.com/projects/poise-detachable-smartwatch#/.
[33]
Reza Rawassizadeh, Blaine A. Price, and Marian Petre. 2014. Wearables: Has the Age of Smartwatches Finally Arrived? Commun. ACM 58, 1 (Dec. 2014), 45--47. http://dx.doi.org/10.1145/2629633
[34]
Jun Rekimoto. 2002. SmartSkin: An Infrastructure for Freehand Manipulation on Interactive Surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '02). ACM, New York, NY, USA, 113--120. http://dx.doi.org/10.1145/503376.503397
[35]
Steven Schirra and Frank R. Bentley. 2015. "It's Kind of Like an Extra Screen for My Phone": Understanding Everyday Uses of Consumer Smart Watches. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '15). ACM, New York, NY, USA, 2151--2156. http://dx.doi.org/10.1145/2702613.2732931
[36]
Martin Schmitz, Jürgen Steimle, Jochen Huber, Niloofar Dezfuli, and Max Mühlh"auser. 2017. Flexibles: Deformation-Aware 3D-Printed Tangibles for Capacitive Touchscreens. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 1001--1014. http://dx.doi.org/10.1145/3025453.3025663
[37]
Shneiderman. 1983. Direct Manipulation: A Step Beyond Programming Languages. Computer 16, 8 (Aug 1983), 57--69. http://dx.doi.org/10.1109/MC.1983.1654471
[38]
Andrew Spielberg, Alanson Sample, Scott E. Hudson, Jennifer Mankoff, and James McCann. 2016. RapID: A Framework for Fabricating Low-Latency Interactive Objects with RFID Tags. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5897--5908. http://dx.doi.org/10.1145/2858036.2858243
[39]
R. Stanton, E. Pontelli, Z. Toups, and M. Manshad. 2018. Exploring a Novel Inexpensive Tangible Interface for Non-visual Math and Science. In Computers Helping People with Special Needs, Klaus Miesenberger and Georgios Kouroupetroglou (Eds.). Springer International Publishing, Cham, 619--627.
[40]
Yusuke Sugano, Xucong Zhang, and Andreas Bulling. 2016. AggreGaze: Collective Estimation of Audience Attention on Public Displays. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 821--831. http://dx.doi.org/10.1145/2984511.2984536
[41]
M. Sugimoto, K. Kodama, A. Nakamura, M. Kojima, and M. Inami. 2007. A Display-Based Tracking System: Display-Based Computing for Measurement Systems. In 17th International Conference on Artificial Reality and Telexistence (ICAT 2007). 31--38. http://dx.doi.org/10.1109/ICAT.2007.50
[42]
Jay Summet and Rahul Sukthankar. 2005. Tracking Locations of Moving Hand-Held Displays Using Projected Light. In Pervasive Computing, Hans W. Gellersen, Roy Want, and Albrecht Schmidt (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 37--46.
[43]
Hongyi Wen, Julian Ramos Rojas, and Anind K. Dey. 2016. Serendipity: Finger Gesture Recognition Using an Off-the-Shelf Smartwatch. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 3847--3851. http://dx.doi.org/10.1145/2858036.2858466
[44]
Wikipedia. 2019a. GPS/INS. (2019). Retrieved December 26, 2019 from https://en.wikipedia.org/wiki/GPS/INS.
[45]
Wikipedia. 2019b. Inertial navigation system. (2019). Retrieved December 26, 2019 from https://en.wikipedia.org/wiki/Inertial_navigation_system.
[46]
Wikipedia. 2019c. Pixel shifting. (2019). Retrieved September 14, 2019 from https://en.wikipedia.org/wiki/Pixel_shifting.
[47]
Andrew D. Wilson. 2005. PlayAnywhere: A Compact Interactive Tabletop Projection-vision System. In Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology (UIST '05). ACM, New York, NY, USA, 83--92. http://dx.doi.org/10.1145/1095034.1095047
[48]
Andrew D. Wilson and Raman Sarin. 2007. BlueTable: Connecting Wireless Mobile Devices on Interactive Surfaces Using Vision-based Handshaking. In Proceedings of Graphics Interface 2007 (GI '07). ACM, New York, NY, USA, 119--125. http://dx.doi.org/10.1145/1268517.1268539
[49]
Robert Xiao, Scott Hudson, and Chris Harrison. 2016. CapCam: Enabling Rapid, Ad-Hoc, Position-Tracked Interactions Between Devices. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces (ISS '16). ACM, New York, NY, USA, 169--178. http://dx.doi.org/10.1145/2992154.2992182
[50]
Robert Xiao, Gierad Laput, and Chris Harrison. 2014. Expanding the Input Expressivity of Smartwatches with Mechanical Pan, Twist, Tilt and Click. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 193--196. http://dx.doi.org/10.1145/2556288.2557017
[51]
Akio YAMAMOTO and Taku NAKAMURA. 2018. Performance of displayed-marker-based position tracking. Journal of Advanced Mechanical Design, Systems, and Manufacturing 12, 7 (2018), JAMDSM0125--JAMDSM0125. http://dx.doi.org/10.1299/jamdsm.2018jamdsm0125
[52]
Hui-Shyong Yeo, Juyoung Lee, Andrea Bianchi, and Aaron Quigley. 2016. WatchMI: Pressure Touch, Twist and Pan Gesture Input on Unmodified Smartwatches. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '16). ACM, New York, NY, USA, 394--399. http://dx.doi.org/10.1145/2935334.2935375
[53]
Yu Zhang, Tao Gu, Chu Luo, Vassilis Kostakos, and Aruna Seneviratne. 2018. FinDroidHR: Smartwatch Gesture Input with Optical Heartrate Monitor. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 1, Article 56 (March 2018), 42 pages. http://dx.doi.org/10.1145/3191788
[54]
J. Zhao, R. S. Allison, M. Vinnikov, and S. Jennings. 2017. Estimating the motion-to-photon latency in head mounted displays. In 2017 IEEE Virtual Reality (VR). 313--314. http://dx.doi.org/10.1109/VR.2017.7892302

Cited By

View all
  • (2024)A Meta-Bayesian Approach for Rapid Online Parametric Optimization for Wrist-based InteractionsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642071(1-38)Online publication date: 11-May-2024
  • (2023)Review on Wearable Technology in Sports: Concepts, Challenges and OpportunitiesApplied Sciences10.3390/app13181039913:18(10399)Online publication date: 17-Sep-2023
  • (2023)DRG-KeyboardProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35694636:4(1-30)Online publication date: 11-Jan-2023
  • Show More Cited By

Index Terms

  1. WATouCH: Enabling Direct Input on Non-touchscreen Using Smartwatch's Photoplethysmogram and IMU Sensor Fusion

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
      April 2020
      10688 pages
      ISBN:9781450367080
      DOI:10.1145/3313831
      Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 23 April 2020

      Check for updates

      Author Tags

      1. direct input
      2. public display
      3. smartwatch
      4. tangible input

      Qualifiers

      • Research-article

      Conference

      CHI '20
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

      Upcoming Conference

      CHI 2025
      ACM CHI Conference on Human Factors in Computing Systems
      April 26 - May 1, 2025
      Yokohama , Japan

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)278
      • Downloads (Last 6 weeks)33
      Reflects downloads up to 16 Jan 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)A Meta-Bayesian Approach for Rapid Online Parametric Optimization for Wrist-based InteractionsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642071(1-38)Online publication date: 11-May-2024
      • (2023)Review on Wearable Technology in Sports: Concepts, Challenges and OpportunitiesApplied Sciences10.3390/app13181039913:18(10399)Online publication date: 17-Sep-2023
      • (2023)DRG-KeyboardProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35694636:4(1-30)Online publication date: 11-Jan-2023
      • (2022)Enabling Tangible Interaction on Non-touch Displays with Optical Mouse Sensor and Visible Light CommunicationProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517666(1-14)Online publication date: 29-Apr-2022
      • (2022)Biomedical Sensors and Applications of Wearable Technologies on Arm and HandBiomedical Materials & Devices10.1007/s44174-022-00002-71:1(443-455)Online publication date: 4-Aug-2022
      • (2021)FieldSweep: A 2D Tracking System With Embedded Magnets and a SmartphoneExtended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411763.3451841(1-6)Online publication date: 8-May-2021

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Login options

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media