short-paper

fSense: Unlocking the Dimension of Force for Gestural Interactions using Smartwatch PPG Sensor

Authors:

Thisum Buddhika,

Samantha W. T. Chan,

Vipula Dissanayake,

Suranga Nanayakkara,

Roger ZimmermannAuthors Info & Claims

AH2019: Proceedings of the 10th Augmented Human International Conference 2019

Article No.: 11, Pages 1 - 5

https://doi.org/10.1145/3311823.3311839

Published: 11 March 2019 Publication History

Abstract

While most existing gestural interfaces focus on the static posture or the dynamic action of the hand, few have investigated the feasibility of using the forces that are exerted while performing gestures. Using the photoplethysmogram (PPG) sensor of off-the-shelf smartwatches, we show that, it is possible to recognize the force of a gesture as an independent channel of input. Based on a user study with 12 participants, we found that users were able to reliably produce two levels of force across several types of common gestures. We demonstrate a few interaction scenarios where the force is either used as a standalone input or to complement existing input modalities.

References

[1]

Zheng J. De La Rosa S. Guertler C. Dollar A. Bullock, I. 2013. Grasp frequency and usage in daily household and machine shop tasks. IEEE transactions on haptics 6, 3 (2013), 296--308.

Digital Library

[2]

Chen Y. Hsieh C. Liang R. Chen B. Chan, L. 2015. Cyclopsring: Enabling whole-hand and context-aware interactions through a fisheye ring. In In Proc UIST. 549--556.

Digital Library

[3]

Samsung Electronics. 2017. Gear S3. Retrieved Oct. 2, 2018 from https://www.samsung.com/global/galaxy/gear-s3/.

[4]

Wobbrock J. Patel S. Goel, M. 2012. GripSense: using built-in sensors to detect hand posture and pressure on commodity mobile phones. In Proc. UIST. 545--554.

Digital Library

[5]

Xiao R. Harrison C. Laput, G. 2016. Viband: High-fidelity bio-acoustic sensing using commodity smartwatch accelerometers. In Proc. UIST. 321--333.

Digital Library

[6]

Sekine M. Tamura T. Maeda, Y. 2011. Relationship between measurement site and motion artifacts in wearable reflected photoplethysmography. Journal of medical systems 35, 5 (2011), 969--976.

Digital Library

[7]

Marzo A. Fraser M. McIntosh, J. 2017. SensIR: Detecting Hand Gestures with a Wearable Bracelet using Infrared Transmission and Reflection. In Proc. UIST. 593--597.

Digital Library

[8]

McNeill C. Fraser M. Kerber F. Löchtefeld M. Krüger A McIntosh, J. 2016. EMPress: Practical hand gesture classification with wrist-mounted EMG and pressure sensing. In Proc. CHI. 2332--2342.

Digital Library

[9]

Wang Y. Yu C. Yan Y. Wen H. Shi Y. Sun, K. 2017. Float: One-Handed and Touch-Free Target Selection on Smartwatches. In Proc. CHI. 692--704.

Digital Library

[10]

Ramos Rojas J. Dey A. Wen, H. 2016. Serendipity: Finger gesture recognition using an off-the-shelf smartwatch. In Proc. CHI. 3847--3851.

Digital Library

[11]

Krakowczyk D. Trollmann F. Albayrak S. Wilhelm, M. 2015. eRing: multiple finger gesture recognition with one ring using an electric field. In Proc. iWOAR. 7.

Digital Library

[12]

Frank E. Hall M. A. Pal C. J. Witten, I. H. 2016. Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann.

Digital Library

[13]

Hinatsu S. Kuroda Y. Oshiro O. Yoshimoto, S. 2018. Hemodynamic Sensing of 3-D Fingertip Force by Using Nonpulsatile and Pulsatile Signals in the Proximal Part. IEEE transactions on biomedical circuits and systems 99 (2018), 1--10.

[14]

Gu T. Luo C. Kostakos V. Seneviratne A. Zhang, Y. 2018. FinDroidHR: Smartwatch Gesture Input with Optical Heartrate Monitor. Proc. IMWUT 2, 1 (2018), 56.

Digital Library

[15]

Harrison C. Zhang, Y. 2015. Tomo: Wearable, low-cost electrical impedance tomography for hand gesture recognition. In Proc. UIST. 167--173.

Digital Library

[16]

Liu J. Wang Y. Liu H. Chen Y. Zhao, T. 2018. PPG-based finger-level gesture recognition leveraging wearables. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications. 1457--1465.

Cited By

Villarreal-Narvaez SSluÿters AVanderdonckt JVatavu R(2023)Brave New GES World: A Systematic Literature Review of Gestures and Referents in Gesture Elicitation StudiesACM Computing Surveys10.1145/363645856:5(1-55)Online publication date: 7-Dec-2023
https://dl.acm.org/doi/10.1145/3636458
Kim HYoon BOh SWoo W(2023)Visualizing Hand Force with Wearable Muscle Sensing for Enhanced Mixed Reality Remote CollaborationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332021029:11(4611-4621)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3320210
Fang CRuan XZhang XChen XChen X(2023)Exploration on the Negative Effects of Sensor Shifts in Photoplethysmography-Based Gesture Recognition and a Solution Based on Transfer LearningIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2023.326846972(1-11)Online publication date: 2023
https://doi.org/10.1109/TIM.2023.3268469
Show More Cited By

Index Terms

fSense: Unlocking the Dimension of Force for Gestural Interactions using Smartwatch PPG Sensor
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools
  2. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing systems and tools

Recommendations

FinDroidHR: Smartwatch Gesture Input with Optical Heartrate Monitor

We present FinDroidHR, a novel gesture input technique for off-the-shelf smartwatches. Our technique is designed to detect 10 hand gestures on the hand wearing a smartwatch. The technique is enabled by analysing features of the Photoplethysmography (PPG) ...
A gaze interactive textual smartwatch interface
UbiComp/ISWC'15 Adjunct: Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers

Mobile gaze interaction is challenged by inherent motor noise. We examined the gaze tracking accuracy and precision of twelve subjects wearing a gaze tracker on their wrist while standing and walking. Results suggest that it will be possible to detect ...
Glance Awareness and Gaze Interaction in Smartwatches
CHI EA '15: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems

Smartwatches are widely available and increasingly adopted by consumers. The most common way of interacting with smartwatches is either touching a screen or pressing buttons on the sides. However, such techniques require using both hands. We propose ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

AH2019: Proceedings of the 10th Augmented Human International Conference 2019

March 2019

301 pages

ISBN:9781450365475

DOI:10.1145/3311823

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 March 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Conference

AH2019

AH2019: Augmented Human International Conference 2019

March 11 - 12, 2019

Reims, France

Acceptance Rates

AH2019 Paper Acceptance Rate 26 of 53 submissions, 49%;

Overall Acceptance Rate 121 of 306 submissions, 40%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
346
Total Downloads

Downloads (Last 12 months)24
Downloads (Last 6 weeks)1

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Villarreal-Narvaez SSluÿters AVanderdonckt JVatavu R(2023)Brave New GES World: A Systematic Literature Review of Gestures and Referents in Gesture Elicitation StudiesACM Computing Surveys10.1145/363645856:5(1-55)Online publication date: 7-Dec-2023
https://dl.acm.org/doi/10.1145/3636458
Kim HYoon BOh SWoo W(2023)Visualizing Hand Force with Wearable Muscle Sensing for Enhanced Mixed Reality Remote CollaborationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332021029:11(4611-4621)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3320210
Fang CRuan XZhang XChen XChen X(2023)Exploration on the Negative Effects of Sensor Shifts in Photoplethysmography-Based Gesture Recognition and a Solution Based on Transfer LearningIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2023.326846972(1-11)Online publication date: 2023
https://doi.org/10.1109/TIM.2023.3268469
Li DKang PZhu KLi JShull P(2023)Feasibility of Wearable PPG for Simultaneous Hand Gesture and Force Level ClassificationIEEE Sensors Journal10.1109/JSEN.2023.324112623:6(6008-6017)Online publication date: 15-Mar-2023
https://doi.org/10.1109/JSEN.2023.3241126
Iijima RShitara AOchiai Y(2022)Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing PeopleProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3544828(1-8)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1145/3517428.3544828
Rudolph JHolman DDe Araujo BJota RWigdor DSavage V(2022)Sensing Hand Interactions with Everyday Objects by Profiling Wrist TopographyProceedings of the Sixteenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3490149.3501320(1-14)Online publication date: 13-Feb-2022
https://dl.acm.org/doi/10.1145/3490149.3501320
Cafaro FBrady EChandra SPatil USaxena A(2021)E Pluribus Unum: Using Conceptual Metaphor Theory to Explore and Support Mixed-Ability WorkplacesProceedings of the ACM on Human-Computer Interaction10.1145/34795285:CSCW2(1-31)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3479528
Ling YChen XRuan YZhang XChen X(2021)Comparative Study of Gesture Recognition Based on Accelerometer and Photoplethysmography Sensor for Gesture Interactions in Wearable DevicesIEEE Sensors Journal10.1109/JSEN.2021.308171421:15(17107-17117)Online publication date: 1-Aug-2021
https://doi.org/10.1109/JSEN.2021.3081714
Hinatsu SSuzuki DIshizuka HIkeda SOshiro O(2021)Attack on PPG Biometrics: Presentation Attack by Stealth Recording and Waveform Estimation2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)10.1109/EMBC46164.2021.9630803(64-67)Online publication date: 1-Nov-2021
https://doi.org/10.1109/EMBC46164.2021.9630803
Yoon HSon J(2020)Collocated Wearable Interaction for Audio Book Application on Smartwatch and HearablesJournal of Multimedia Information System10.33851/JMIS.2020.7.2.1077:2(107-114)Online publication date: 30-Jun-2020
https://doi.org/10.33851/JMIS.2020.7.2.107
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten