research-article

Gesture Recognition Methods Using Sensors Integrated into Smartwatches: Results of a Systematic Literature Review

Authors:

Pedro Raphael Inácio Gomes,

Murillo Santos de Castro,

Thamer Horbylon NascimentoAuthors Info & Claims

IHC '23: Proceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems

Article No.: 55, Pages 1 - 11

https://doi.org/10.1145/3638067.3638082

Published: 24 January 2024 Publication History

Abstract

This work addresses the importance of gesture recognition in smartwatches and aims to conduct a systematic literature review to identify the most commonly used methods and sensors in this field. The selected works were obtained from renowned databases, such as ACM Digital Library, IEEE Xplore, Scopus, and ScienceDirect. Initially, 265 articles were identified, and after applying inclusion and quality criteria, 43 articles were selected for detailed analysis. The systematic review allowed for the consolidation of existing knowledge and the identification of the most relevant approaches in this area. The results revealed a diversity of algorithms employed, such as Naïve Bayes, Artificial Neural Network, Support Vector Machine, Dynamic Time Warping, Symbolic Aggregate approXimation, Long Short-Term Memory, Random Forest, Multiple Dimensional Dynamic Time Warping, Gated Recurrent Unit, K-nearest Neighbors, Hidden Markov Model, Latent Dirichlet Allocation, and Convolutional Neural Network. Additionally, we found that accelerometers were the most commonly used sensors, followed by gyroscopes, magnetometers, barometers, vital sensors (EMG, FMG, PPG), and infrared sensors. The findings highlight the relevance of gesture recognition in smartwatches and provide valuable insights for researchers and professionals interested in this constantly evolving field.

References

[1]

Amr Alanwar, Moustafa Alzantot, Bo-Jhang Ho, Paul Martin, and Mani Srivastava. 2017. Selecon: Scalable iot device selection and control using hand gestures. In Proceedings of the Second International Conference on Internet-of-Things Design and Implementation. 47–58.

Digital Library

[2]

Hongliang Bi, Jian Zhang, Yanjiao Chen, Chaoyang Zhou, and Zhibo Wang. 2021. Magic-hand: Turn a smartwatch into a mouse. Pervasive and Mobile Computing 78 (2021), 101487.

Digital Library

[3]

Vikram Cannanure, Xiang’Anthony’Chen, and Jennifer Mankoff. 2016. Twist’n’Knock: A One-handed Gesture for Smart Watches. In Graphics Interface. 189–193.

[4]

Feiyu Chen, Jia Deng, Zhibo Pang, Majid Baghaei Nejad, Huayong Yang, and Geng Yang. 2018. Finger angle-based hand gesture recognition for smart infrastructure using wearable wrist-worn camera. Applied Sciences 8, 3 (2018), 369.

[5]

Feiyu Chen, Honghao Lv, Zhibo Pang, Junhui Zhang, Yonghong Hou, Ying Gu, Huayong Yang, and Geng Yang. 2018. WristCam: A wearable sensor for hand trajectory gesture recognition and intelligent human–robot interaction. IEEE Sensors Journal 19, 19 (2018), 8441–8451.

[6]

Gabriele Costante, Lorenzo Porzi, Oswald Lanz, Paolo Valigi, and Elisa Ricci. 2014. Personalizing a smartwatch-based gesture interface with transfer learning. In 2014 22nd European Signal Processing Conference (EUSIPCO). 2530–2534.

[7]

Yufeng Deng, Dong Wang, Qian Zhang, Run Zhao, and Bo Chen. 2018. ReaderTrack: Reader-Book Interaction Reasoning Using RFID and Smartwatch. In 2018 27th International Conference on Computer Communication and Networks (ICCCN). IEEE, 1–9.

[8]

Paul Dourish. 2001. Where the action is: the foundations of embodied interaction. MIT press.

[9]

Assaf Feldman, Emmanuel Munguia Tapia, Sajid Sadi, Pattie Maes, and Chris Schmandt. 2005. ReachMedia: On-the-move interaction with everyday objects. In Ninth IEEE International Symposium on Wearable Computers (ISWC’05). IEEE, 52–59.

Digital Library

[10]

Alan Ferrari, Vanni Galli, Daniele Puccinelli, and Silvia Giordano. 2017. On the usage of smart devices to augment the user interaction with multimedia applications. In 2017 IEEE 18th International Symposium on A World of Wireless, Mobile and Multimedia Networks (WoWMoM). IEEE, 1–9.

[11]

Thamer Horbylon Nascimento, Fabrizzio Alphonsus Alves De Melo Nunes Soares, Marcos Alves Vieira, Juliana Paula Felix, Jaline Gonçalves Mombach, Livia Mancine Coelho De Campos, Wellington Galvão Rodrigues, Wesley Flavio De Miranda, and Ronaldo Martins Da Costa. 2019. Using Smartwatches as an Interactive Movie Controller: A Case Study with the Bandersnatch Movie. In 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), Vol. 2. 263–268. https://doi.org/10.1109/COMPSAC.2019.10217

[12]

Thamer Horbylon Nascimento, Cristiane B. R. Ferreira, Wellington G. Rodrigues, and Fabrizzio Soares. 2020. Interaction with Smartwatches Using Gesture Recognition: A Systematic Literature Review. In 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC). 1661–1666. https://doi.org/10.1109/COMPSAC48688.2020.00-17

[13]

Hiroshi Ishii and Brygg Ullmer. 1997. Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’97). Association for Computing Machinery, New York, NY, USA, 234–241. https://doi.org/10.1145/258549.258715

Digital Library

[14]

Shahram Jalaliniya, Jeremiah Smith, Miguel Sousa, Lars Büthe, and Thomas Pederson. 2013. Touch-less interaction with medical images using hand & foot gestures. In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication. 1265–1274.

Digital Library

[15]

Avnish Singh Jat and Tor-Morten Grønli. 2022. Smart watch for smart health monitoring: A literature review. In International Work-Conference on Bioinformatics and Biomedical Engineering. Springer, 256–268.

[16]

Shuo Jiang, Bo Lv, Weichao Guo, Chao Zhang, Haitao Wang, Xinjun Sheng, and Peter B. Shull. 2018. Feasibility of Wrist-Worn, Real-Time Hand, and Surface Gesture Recognition via sEMG and IMU Sensing. IEEE Transactions on Industrial Informatics 14, 8 (2018), 3376–3385. https://doi.org/10.1109/TII.2017.2779814

[17]

Mauri Kaipainen, Niklas Ravaja, Pia Tikka, Rasmus Vuori, Roberto Pugliese, Marco Rapino, and Tapio Takala. 2011. Enactive Systems and Enactive Media: Embodied Human-Machine Coupling beyond Interfaces. Leonardo 44, 5 (10 2011), 433–438. https://doi.org/10.1162/LEON_a_00244 arXiv:https://direct.mit.edu/leon/article-pdf/44/5/433/1580121/leon_a_00244.pdf

[18]

Peiqi Kang, Jinxuan Li, Bingfei Fan, Shuo Jiang, and Peter B Shull. 2021. Wrist-worn hand gesture recognition while walking via transfer learning. IEEE Journal of Biomedical and Health Informatics 26, 3 (2021), 952–961.

[19]

Victor Kartsch, Simone Benatti, Mattia Mancini, Michele Magno, and Luca Benini. 2018. Smart wearable wristband for EMG based gesture recognition powered by solar energy harvester. In 2018 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, 1–5.

[20]

Keiko Katsuragawa, Krzysztof Pietroszek, James R Wallace, and Edward Lank. 2016. Watchpoint: Freehand pointing with a smartwatch in a ubiquitous display environment. In Proceedings of the International Working Conference on Advanced Visual Interfaces. 128–135.

Digital Library

[21]

Keiko Katsuragawa, James R Wallace, and Edward Lank. 2016. Gestural text input using a smartwatch. In Proceedings of the International Working Conference on Advanced Visual Interfaces. 220–223.

Digital Library

[22]

Hyung-il Kim, Juyoung Lee, Hui Shyong Yeo, Aaron John Quigley, and Woontack Woo. 2019. SWAG demo: Smart watch assisted gesture interaction for mixed reality head-mounted displays. In Adjunct Proceedings-2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018. Institute of Electrical and Electronics Engineers Inc.

[23]

Barbara Kitchenham. 2004. Procedures for performing systematic reviews. Keele, UK, Keele University 33, 2004 (2004), 1–26.

[24]

Jonathan Knighten, Stephen McMillan, Tori Chambers, and Jamie Payton. 2015. Recognizing social gestures with a wrist-worn smartband. In 2015 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops). IEEE, 544–549.

[25]

Ce Li, Chunyu Xie, Baochang Zhang, Chen Chen, and Jungong Han. 2018. Deep Fisher discriminant learning for mobile hand gesture recognition. Pattern Recognition 77 (2018), 276–288.

Digital Library

[26]

Gen Li and Hiroyuki Sato. 2022. Sensing In-Air Signature Motions Using Smartwatch: A High-Precision Approach of Behavioral Authentication. IEEE Access 10 (2022), 57865–57879.

[27]

Yande Li, Taiqian Wang, Lian Li, Caihong Li, Yi Yang, Li Liu, 2018. Hand gesture recognition and real-time game control based on a wearable band with 6-axis sensors. In 2018 international joint conference on neural networks (IJCNN). IEEE, 1–6.

[28]

Yubing Li, Kun Zhao, Meichen Duan, Wei Shi, Liangliang Lin, Xinyi Cao, Yang Liu, and Jizhong Zhao. 2020. Control your home with a smartwatch. IEEE Access 8 (2020), 131601–131613.

[29]

Mateus M Luna, Thyago P Carvalho, Fabrizzio Alphonsus AMN Soares, Hugo AD Nascimento, and Ronaldo M Costa. 2017. Wrist player: a smartwatch gesture controller for smart TVs. In 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), Vol. 2. IEEE, 336–341.

[30]

Steven McGuckin, Soumyadeb Chowdhury, and Lewis Mackenzie. 2016. Tap’n’shake: gesture-based smartwatch-smartphone communications system. In Proceedings of the 28th Australian Conference on Computer-Human Interaction. 442–446.

[31]

Antigoni Mezari and Ilias Maglogiannis. 2017. Gesture recognition using symbolic aggregate approximation and dynamic time warping on motion data. In Proceedings of the 11th EAI International Conference on Pervasive Computing Technologies for Healthcare. 342–347.

Digital Library

[32]

Antigoni Mezari and Ilias Maglogiannis. 2018. An easily customized gesture recognizer for assisted living using commodity mobile devices. Journal of Healthcare Engineering 2018 (2018).

[33]

Danial Moazen, Seyed A Sajjadi, and Ani Nahapetian. 2016. AirDraw: Leveraging smart watch motion sensors for mobile human computer interactions. In 2016 13th IEEE Annual Consumer Communications & Networking Conference (CCNC). 442–446. https://doi.org/10.1109/CCNC.2016.7444820

Digital Library

[34]

David Moher, Alessandro Liberati, Jennifer Tetzlaff, Douglas G Altman, and the PRISMA Group*. 2009. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Annals of internal medicine 151, 4 (2009), 264–269.

[35]

Dania Murad, Fenyi Ye, Michael Barone, and Ye Wang. 2017. Motion initiated music ensemble with sensors for motor rehabilitation. In 2017 international conference on orange technologies (icot). IEEE, 87–90.

[36]

Kyeong-Beom Park and Jae Yeol Lee. 2019. New design and comparative analysis of smartwatch metaphor-based hand gestures for 3D navigation in mobile virtual reality. Multimedia Tools and Applications 78 (2019), 6211–6231.

Digital Library

[37]

Krzysztof Pietroszek, Liudmila Tahai, James R Wallace, and Edward Lank. 2017. Watchcasting: Freehand 3D interaction with off-the-shelf smartwatch. In 2017 IEEE Symposium on 3D User Interfaces (3DUI). IEEE, 172–175.

[38]

Blaine Reeder and Alexandria David. 2016. Health at hand: A systematic review of smart watch uses for health and wellness. Journal of biomedical informatics 63 (2016), 269–276.

Digital Library

[39]

Felix Schüssel, Frank Honold, Nikola Bubalo, and Michael Weber. 2016. Increasing robustness of multimodal interaction via individual interaction histories. In Proceedings of the Workshop on Multimodal Analyses enabling Artificial Agents in Human-Machine Interaction. 56–63.

Digital Library

[40]

Chao Shen, Yufei Chen, Gengshan Yang, and Xiaohong Guan. 2018. Toward hand-dominated activity recognition systems with wristband-interaction behavior analysis. IEEE Transactions on Systems, Man, and Cybernetics: Systems 50, 7 (2018), 2501–2511.

[41]

Alexandru-Ionut Siean and Radu-Daniel Vatavu. 2021. Wearable Interactions for Users with Motor Impairments: Systematic Review, Inventory, and Research Implications. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, USA) (ASSETS ’21). Association for Computing Machinery, New York, NY, USA, Article 7, 15 pages. https://doi.org/10.1145/3441852.3471212

Digital Library

[42]

Thanh-Hai Tran, Hoang-Nhat Tran, Hong-Quan Nguyen, Trung-Hieu Le, Van-Thang Nguyen, Trung-Kien Tran, Cuong Pham, Thi-Lan Le, Hai Vu, Thanh Phuong Nguyen, 2021. A pilot study on hand posture recognition from wrist-worn camera for human machine interaction. In 2021 International Conference on Advanced Technologies for Communications (ATC). IEEE, 201–206.

[43]

Ayush Tripathi, Arnab Kumar Mondal, Lalan Kumar, and AP Prathosh. 2022. ImAiR: Airwriting Recognition Framework Using Image Representation of IMU Signals. IEEE Sensors Letters 6, 10 (2022), 1–4.

[44]

Francisco J Varela, Evan Thompson, and Eleanor Rosch. 1992. The embodied mind: Cognitive science and human experience. MIT press.

[45]

Valeria Villani, Lorenzo Sabattini, Nicola Battilani, and Cesare Fantuzzi. 2016. Smartwatch-enhanced interaction with an advanced troubleshooting system for industrial machines. IFAC-PapersOnLine 49, 19 (2016), 277–282.

[46]

Valeria Villani, Lorenzo Sabattini, Giuseppe Riggio, Cristian Secchi, Marco Minelli, and Cesare Fantuzzi. 2017. A natural infrastructure-less human–robot interaction system. IEEE Robotics and Automation Letters 2, 3 (2017), 1640–1647.

[47]

Hong Wang, Peiqi Kang, Qinghua Gao, Shuo Jiang, and Peter B Shull. 2022. A Novel PPG-FMG-ACC Wristband for Hand Gesture Recognition. IEEE journal of biomedical and health informatics 26, 10 (2022), 5097–5108.

[48]

Mark Weiser. 1991. The Computer for the 21 st Century. Scientific American 265, 3 (1991), 94–105. http://www.jstor.org/stable/24938718

[49]

Yuki Yamato, Yutaro Suzuki, Kodai Sekimori, Buntarou Shizuki, and Shin Takahashi. 2020. Hand gesture interaction with a low-resolution infrared image sensor on an inner wrist. In Proceedings of the International Conference on Advanced Visual Interfaces. 1–5.

Digital Library

[50]

Geng Yang, Honghao Lv, Feiyu Chen, Zhibo Pang, Jin Wang, Huayong Yang, and Junhui Zhang. 2018. A novel gesture recognition system for intelligent interaction with a nursing-care assistant robot. Applied sciences 8, 12 (2018), 2349.

[51]

Hui-Shyong Yeo, Juyoung Lee, Andrea Bianchi, Alejandro Samboy, Hideki Koike, Woontack Woo, and Aaron Quigley. 2020. WristLens: Enabling Single-Handed Surface Gesture Interaction for Wrist-Worn Devices Using Optical Motion Sensor. In Proceedings of the Augmented Humans International Conference. 1–8.

Digital Library

[52]

Yufei Zhang, Bin Liu, and Zhiqiang Liu. 2019. Recognizing hand gestures with pressure-sensor-based motion sensing. IEEE transactions on biomedical circuits and systems 13, 6 (2019), 1425–1436.

[53]

Yidi Zhang, Mengyuan Ran, Jun Liao, Guoxin Su, Ming Liu, and Li Liu. 2021. Stacked LSTM-Based Dynamic Hand Gesture Recognition with Six-Axis Motion Sensors. In 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2568–2575.

[54]

Han Zhou, Yi Gao, Xinyi Song, Wenxin Liu, and Wei Dong. 2019. Limbmotion: Decimeter-level limb tracking for wearable-based human-computer interaction. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 4 (2019), 1–24.

Digital Library

Cited By

Silva LSoares FFelix JCardoso LAranha RNascimento T(2024)Gesture Recognition in Smartwatches Using LSTM for Interaction in Low-Cost Virtual EnvironmentsProceedings of the 26th Symposium on Virtual and Augmented Reality10.1145/3691573.3691615(284-288)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3691573.3691615

Index Terms

Gesture Recognition Methods Using Sensors Integrated into Smartwatches: Results of a Systematic Literature Review
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
    1. Redundancy
  2. Embedded and cyber-physical systems
    1. Embedded systems
    2. Robotics
2. Networks
  1. Network properties
    1. Network reliability

Recommendations

A systematic literature review on vision based gesture recognition techniques

Human Computer Interaction (HCI) technologies are rapidly evolving the way we interact with computing devices and adapting to the constantly increasing demands of modern paradigms. One of the most useful tools in this regard is the integration of Human-...
Guidelines for snowballing in systematic literature studies and a replication in software engineering
EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.

Objective: This paper presents guidelines for conducting literature reviews using ...
Towards Sustainability of Systematic Literature Reviews
ESEM '21: Proceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

Background: The software engineering community has increasingly conducted systematic literature reviews (SLR) as a means to summarize evidence from different studies and bring to light the state of the art of a given research topic. While SLR provide ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

IHC '23: Proceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems

October 2023

791 pages

ISBN:9798400717154

DOI:10.1145/3638067

Editors:
Ranilson Paiva
Universidade Federal de Alagoas (UFAL)
,
André Magno C. de Araújo
Universidade Federal de Alagoas (UFAL)
,
Taciana Pontual Falcão
Universidade Federal Rural de Pernambuco (UFRPE)
,
Yuska Paola Costa Aguiar
Universidade Federal da Paraíba (UFPB)

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 January 2024

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

IHC '23

IHC '23: XXII Brazilian Symposium on Human Factors in Computing Systems

October 16 - 20, 2023

Maceió, Brazil

Acceptance Rates

Overall Acceptance Rate 331 of 973 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
95
Total Downloads

Downloads (Last 12 months)95
Downloads (Last 6 weeks)13

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Silva LSoares FFelix JCardoso LAranha RNascimento T(2024)Gesture Recognition in Smartwatches Using LSTM for Interaction in Low-Cost Virtual EnvironmentsProceedings of the 26th Symposium on Virtual and Augmented Reality10.1145/3691573.3691615(284-288)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3691573.3691615

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents