research-article

LimbMotion: Decimeter-level Limb Tracking for Wearable-based Human-Computer Interaction

Authors:

Wei DongAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 3, Issue 4

Article No.: 161, Pages 1 - 24

https://doi.org/10.1145/3369836

Published: 14 September 2020 Publication History

Abstract

Wearable-based human-computer interaction is a promising technology to enable various applications. This paper aims to track the 3D posture of the entire limb, both wrist/ankle and elbow/knee, of a user wearing a smart device. This limb tracking technology can trace the geometric motion of the limb, without introducing any training stage usually required in gesture recognition approaches. Nonetheless, the tracked limb motion can also be used as a generic input for gesture-based applications. The 3D posture of a limb is defined by the relative positions among main joints, e.g., shoulder, elbow, and wrist for an arm. When a smartwatch is worn on the wrist of a user, its position is affected by both elbow and shoulder motions. It is challenging to infer the entire 3D posture when only given a single point of sensor data from the smartwatch. In this paper, we propose LimbMotion, an accurate and real-time limb tracking system. The performance gain of LimbMotion comes from multiple key technologies, including an accurate attitude estimator based on a novel two-step filter, fast acoustic ranging, and point clouds-based positioning. We implemented LimbMotion and evaluated its performance using extensive experiments, including different gestures, moving speeds, users, and limbs. Results show that LimbMotion achieves real-time tracking with a median error of 7.5cm to 8.9cm, which outperforms the state-of-the-art approach by about 32%.

References

[1]

Sonya Allin, Nancy Baker, Emily Eckel, and Deva Ramanan. Robust tracking of the upper limb for functional stroke assessment. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 18(5):542--550, 2010.

[2]

Edwin Valarezo Añazco, Patricio Rivera Lopez, Sangmin Lee, Kyungmin Byun, and Tae-Seong Kim. Smoking activity recognition using a single wrist imu and deep learning light. In Proceedings of the 2nd International Conference on Digital Signal Processing, pages 48--51. ACM, 2018.

Digital Library

[3]

Eric R Bachmann, I Duman, UY Usta, Robert B McGhee, XP Yun, and MJ Zyda. Orientation tracking for humans and robots using inertial sensors. In Proceedings 1999 IEEE International Symposium on Computational Intelligence in Robotics and Automation. CIRA'99 (Cat. No. 99EX375), pages 187--194. IEEE, 1999.

[4]

Ake Bjorck. Numerical methods for least squares problems, volume 51. Siam, 1996.

[5]

H. P. Bruckner, C. Spindeldreier, H. Blume, E. Schoonderwaldt, and E. Altenmuller. Evaluation of inertial sensor fusion algorithms in grasping tasks using real input data: Comparison of computational costs and root mean square error. In Ninth International Conference on Wearable and Implantable Body Sensor Networks, 2012.

Digital Library

[6]

Min Chen, Sergio Gonzalez, Athanasios Vasilakos, Huasong Cao, and Victor C Leung. Body area networks: A survey. Mobile networks and applications, 16(2):171--193, 2011.

[7]

J. L. Crassidis, F. L. Markley, and Y. Cheng. Survey of nonlinear attitude estimation methods. Journal of Guidance Control & Dynamics, 30(1):12--28, 2007.

[8]

Debraj De, Pratool Bharti, Sajal K Das, and Sriram Chellappan. Multimodal wearable sensing for fine-grained activity recognition in healthcare. IEEE Internet Computing, 19(5):26--35, 2015.

Digital Library

[9]

B. Fan, Q. Li, and T. Liu. How magnetic disturbance influences the attitude and heading in magnetic and inertial sensor-based orientation estimation:. Sensors, 18(1):76, 2018.

[10]

Kaiqiang Feng, Jie Li, Xiaoming Zhang, Chong Shen, Yu Bi, Tao Zheng, and Jun Liu. A new quaternion-based kalman filter for real-time attitude estimation using the two-step geometrically-intuitive correction algorithm. Sensors, 17(9):2146, 2017.

[11]

Fitbit. https://www.fitbit.com/.

[12]

Mohammad Ghamari, Balazs Janko, R Sherratt, William Harwin, Robert Piechockic, and Cinna Soltanpur. A survey on wireless body area networks for ehealthcare systems in residential environments. Sensors, 16(6):831, 2016.

[13]

Philip E Gill and Walter Murray. Quasi-newton methods for unconstrained optimization. IMA Journal of Applied Mathematics, 9(1):91--108, 1972.

[14]

Philip E Gill and Walter Murray. Newton-type methods for unconstrained and linearly constrained optimization. Mathematical Programming, 7(1):311--350, 1974.

Digital Library

[15]

Xiaonan Guo, Jian Liu, and Yingying Chen. Fitcoach: Virtual fitness coach empowered by wearable mobile devices. In IEEE INFOCOM 2017-IEEE Conference on Computer Communications, pages 1--9. IEEE, 2017.

[16]

L. Jiang, X. Lin, X. Liu, C. Bi, and G. Xing. Safedrive: Detecting distracted driving behaviors using wrist-worn devices. ACM IMWUT, 1(4):144:1--144:22, 2018.

[17]

Juha Kela, Panu Korpipää, Jani Mäntyjärvi, Sanna Kallio, Giuseppe Savino, Luca Jozzo, and Di Marca. Accelerometer-based gesture control for a design environment. Personal and Ubiquitous Computing, 10(5):285--299, 2006.

Digital Library

[18]

J. Kim, M. Kim, and K. Kim. Development of a wearable hci controller through semg & imu sensor fusion. In Proc. of IEEE Ubiquitous Robots and Ambient Intelligence (URAI), 2016.

[19]

M. Kos, J. Ženko, D. Vlaj, and I. Kramberger. Tennis stroke detection and classification using miniature wearable imu device. In Proc. of IEEE Systems, Signals and Image Processing (IWSSIP), 2016.

[20]

Gierad Laput, Robert Xiao, and Chris Harrison. Viband: High-fidelity bio-acoustic sensing using commodity smartwatch accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, pages 321--333. ACM, 2016.

Digital Library

[21]

Fan Li, Chunshui Zhao, Guanzhong Ding, Jian Gong, Chenxing Liu, and Feng Zhao. A reliable and accurate indoor localization method using phone inertial sensors. In Proceedings of the 2012 ACM conference on ubiquitous computing, pages 421--430. ACM, 2012.

Digital Library

[22]

S. Lindtner, S. Bardzell, and J. Bardzell. Reconstituting the utopian vision of making: Hci after technosolutionism. In Proc. of CHI Conference on Human Factors in Computing Systems, 2016.

Digital Library

[23]

T.-L. Liu and C.-P. Fan. Visible-light wearable eye gaze tracking by gradients-based eye center location and head movement compensation with imu. In Proc. of IEEE Consumer Electronics (ICCE), 2018.

[24]

P. Maciejasz, J. Eschweiler, K. Gerlach-Hahn, A. Jansen-Troy, and S. Leonhardt. A survey on robotic devices for upper limb rehabilitation. Journal of neuroengineering and rehabilitation, 11(1):3, 2014.

[25]

S. O. Madgwick, A. J. Harrison, and R. Vaidyanathan. Estimation of imu and marg orientation using a gradient descent algorithm. In Proc. of IEEE Rehabilitation Robotics (ICORR), 2011.

[26]

R. Mahony, T. Hamel, and J. M. Pflimlin. Nonlinear complementary filters on the special orthogonal group. IEEE Transactions on Automatic Control, 53(5):1203--1218, 2008.

[27]

W. Mao, J. He, and L. Qiu. Cat: high-precision acoustic motion tracking. In Proc. of ACM MobiCom, 2016.

Digital Library

[28]

D. P. McMullen, G. Hotson, K. D. Katyal, B. A Wester, M. S. Fifer, T. G. McGee, A. Harris, M. S. Johannes, R. J. Vogelstein, A. D. Ravitz, et al. Demonstration of a semi-autonomous hybrid brain--machine interface using human intracranial eeg, eye tracking, and computer vision to control a robotic upper limb prosthetic. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 22(4):784--796, 2014.

[29]

R. Nandakumar, V. Iyer, D. Tan, and S. Gollakota. Fingerio: Using active sonar for fine-grained finger tracking. In Proc. of ACM CHI Conference on Human Factors in Computing Systems, 2016.

Digital Library

[30]

Karen Otte, Bastian Kayser, Sebastian Mansow-Model, Julius Verrel, Friedemann Paul, Alexander U Brandt, and Tanja Schmitz-Hübsch. Accuracy and reliability of the kinect version 2 for clinical measurement of motor function. PloS one, 11(11):e0166532, 2016.

[31]

C. Peng, G. Shen, Y. Zhang, Y. Li, and K. Tan. Beepbeep: a high accuracy acoustic ranging system using cots mobile devices. In Proc. of the 5th international conference on Embedded networked sensor systems, 2007.

Digital Library

[32]

Lorenzo Peppoloni, Alessandro Filippeschi, Emanuele Ruffaldi, and Carlo Alberto Avizzano. A novel 7 degrees of freedom model for upper limb kinematic reconstruction based on wearable sensors. In 2013 IEEE 11th international symposium on intelligent systems and informatics (SISY), pages 105--110. IEEE, 2013.

[33]

Ananth Ranganathan. The levenberg-marquardt algorithm. Tutoral on LM algorithm, 11(1):101--110, 2004.

[34]

S. S Rautaray and A. Agrawal. Vision based hand gesture recognition for human computer interaction: a survey. Artificial Intelligence Review, 43(1):1--54, 2015.

Digital Library

[35]

Daniel Roetenberg, Henk Luinge, and Per Slycke. Xsens mvn: Full 6dof human motion tracking using miniature inertial sensors. Xsens Motion Technologies BV, Tech. Rep, 1, 2009.

[36]

Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747, 2016.

[37]

A. M. Sabatini. Quaternion-based extended kalman filter for determining orientation by inertial and magnetic sensing. IEEE Transactions on Biomedical Engineering, 53(7):1346--1356, 2006.

[38]

Chenguang Shen, Bo-Jhang Ho, and Mani Srivastava. Milift: Efficient smartwatch-based workout tracking using automatic segmentation. IEEE Transactions on Mobile Computing, 17(7):1609--1622, 2017.

[39]

S. Shen, H. Wang, and R. Roy Choudhury. I am a smartwatch and i can track my user's arm. In Proc. of ACM MobiSys, 2016.

Digital Library

[40]

Sheng Shen, Mahanth Gowda, and Romit Roy Choudhury. Closing the gaps in inertial motion tracking. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking, pages 429--444. ACM, 2018.

Digital Library

[41]

L. Song and M. Takatsuka. Real-time 3d finger pointing for an augmented desk. In Proc. of the Sixth Australasian conference on User interface-Volume 40, 2005.

Digital Library

[42]

Ke Sun, Yuntao Wang, Chun Yu, Yukang Yan, Hongyi Wen, and Yuanchun Shi. Float: one-handed and touch-free target selection on smartwatches. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pages 692--704. ACM, 2017.

Digital Library

[43]

Agnieszka Szczęsna, Przemysław Skurowski, Przemysław Pruszowski, Damian Pęszor, Marcin Paszkuta, and Konrad Wojciechowski. Reference data set for accuracy evaluation of orientation estimation algorithms for inertial motion capture systems. In International Conference on Computer Vision and Graphics, pages 509--520. Springer, 2016.

[44]

Weijun Tao, Tao Liu, Rencheng Zheng, and Hutian Feng. Gait analysis using wearable sensors. Sensors, 12(2):2255--2283, 2012.

[45]

Gary To and Mohamed R Mahfouz. Quaternionic attitude estimation for robotic and human motion tracking using sequential monte carlo methods with von mises-fisher and nonuniform densities simulations. IEEE Transactions on Biomedical Engineering, 60(11):3046--3059, 2013.

[46]

Leandra Vicci. Quaternions and rotations in 3-space: The algebra and its geometric interpretation. Physiological Research, 59(2):305--308, 2001.

[47]

Hongyi Wen, Julian Ramos Rojas, and Anind K Dey. Serendipity: Finger gesture recognition using an off-the-shelf smartwatch. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 3847--3851. ACM, 2016.

Digital Library

[48]

C. Zhang, Q. Xue, A. Waghmare, S. Jain, Y. Pu, S. Hersek, K. Lyons, K. A. Cunefare, O. T. Inan, and G. D. Abowd. Soundtrak: Continuous 3d tracking of a finger using active acoustics. ACM IMWUT, 1(2):30, 2017.

[49]

Pengfei Zhou, Mo Li, and Guobin Shen. Use it free: Instantly knowing your phone attitude. In Proceedings of the 20th annual international conference on Mobile computing and networking, pages 605--616. ACM, 2014.

Digital Library

[50]

A. R. Zweighaft, G. L. Slotness, A. L. Henderson, L. B. Osborne, S. M. Lightbody, L. M. Perhala, P. O. Brown, N. H. Haynes, S. M. Kern, P. N. Usgaonkar, et al. A physical workstation, body tracking interface, and immersive virtual environment for rehabilitating phantom limb pain. In IEEE Systems and Information Design Symposium (SIEDS), 2012.

Cited By

Boudlal HSerrhini MTahiri A(2024)A novel approach for simultaneous human activity recognition and pose estimation via skeleton-based leveraging WiFi CSI with YOLOv8 and mediapipe frameworksSignal, Image and Video Processing10.1007/s11760-024-03031-518:4(3673-3689)Online publication date: 29-Feb-2024
https://doi.org/10.1007/s11760-024-03031-5
Gomes PCastro MNascimento T(2023)Gesture Recognition Methods Using Sensors Integrated into Smartwatches: Results of a Systematic Literature ReviewProceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems10.1145/3638067.3638082(1-11)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3638067.3638082
DeVrio NMollyn VHarrison C(2023)SmartPoser: Arm Pose Estimation with a Smartphone and Smartwatch Using UWB and IMU DataProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606821(1-11)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606821
Show More Cited By

Index Terms

LimbMotion: Decimeter-level Limb Tracking for Wearable-based Human-Computer Interaction
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing systems and tools

Recommendations

Wearable-based Human-Computer Interaction with LimbMotion
SenSys '18: Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems

LimbMotion is a limb tracking system which enables accurate and real-time tracking with one wearable device on the wrist/ankle of a user. By integrating inertial sensing and acoustic sensing, LimbMotion significantly reduces the search space of a moving ...
With a flick of the wrist: stretch sensors as lightweight input for mobile devices
TEI '12: Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction

With WristFlicker, we detect wrist movement through sets of stretch sensors embedded in clothing. Our system supports wrist rotation (pronation/supination), and both wrist tilts (flexion/extension and ulnar/radial deviation). Each wrist movement is ...
A telerehabilitation framework for lower-limb functional recovery
BodyNets '14: Proceedings of the 9th International Conference on Body Area Networks

Wearable devices are driving the development of post-surgery rehabilitation procedures, also helping in reducing the recovery time and social costs. This paper presents a real-time monitoring framework aimed at supporting telerehabilitation sessions for ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 3, Issue 4

December 2019

873 pages

EISSN:2474-9567

DOI:10.1145/3375704

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 September 2020

Published in IMWUT Volume 3, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Zhejiang Provincial Natural Science Foundation for Distinguished Young Scholars
Fundamental Research Funds for the Central Universities
National Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
685
Total Downloads

Downloads (Last 12 months)98
Downloads (Last 6 weeks)9

Reflects downloads up to 28 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Boudlal HSerrhini MTahiri A(2024)A novel approach for simultaneous human activity recognition and pose estimation via skeleton-based leveraging WiFi CSI with YOLOv8 and mediapipe frameworksSignal, Image and Video Processing10.1007/s11760-024-03031-518:4(3673-3689)Online publication date: 29-Feb-2024
https://doi.org/10.1007/s11760-024-03031-5
Gomes PCastro MNascimento T(2023)Gesture Recognition Methods Using Sensors Integrated into Smartwatches: Results of a Systematic Literature ReviewProceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems10.1145/3638067.3638082(1-11)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3638067.3638082
DeVrio NMollyn VHarrison C(2023)SmartPoser: Arm Pose Estimation with a Smartphone and Smartwatch Using UWB and IMU DataProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606821(1-11)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606821
Khanna PFeiz SXu JRamakrishnan IJain SBi XBalasubramanian ACosta XAl Hassanieh HAsadi ACox LPerino DWidmer JGiustiniano D(2023)AccessWear: Making Smartphone Applications Accessible to Blind UsersProceedings of the 29th Annual International Conference on Mobile Computing and Networking10.1145/3570361.3592495(1-16)Online publication date: 2-Oct-2023
https://dl.acm.org/doi/10.1145/3570361.3592495
Streli PArmani RCheng YHolz C(2023)HOOV: Hand Out-Of-View Tracking for Proprioceptive Interaction using Inertial SensingProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581468(1-16)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581468
Niu XZou KShen DDrew SWu SGuo GChen R(2023)UltraMotion: High-precision Ultrasonic Arm Tracking for Real-world ExercisesIEEE Transactions on Mobile Computing10.1109/TMC.2023.3241077(1-18)Online publication date: 2023
https://doi.org/10.1109/TMC.2023.3241077
Bian SZhou BLukowicz PLiu MMagno MGuo K(2023)3D Hand Tracking with Induced Magnetic Field2023 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops)10.1109/PerComWorkshops56833.2023.10150286(589-594)Online publication date: 13-Mar-2023
https://doi.org/10.1109/PerComWorkshops56833.2023.10150286
Wang ZMa MFeng XLi XLiu FGuo YChen D(2022)Skeleton-Based Human Pose Recognition Using Channel State Information: A SurveySensors10.3390/s2222873822:22(8738)Online publication date: 11-Nov-2022
https://doi.org/10.3390/s22228738
Zhao XYang SDeng JZhang J(2022)Reversibly stealth QR code based on N-methylmaleimide-vinyl acetate copolymers under the condition of acid-base changeReactive and Functional Polymers10.1016/j.reactfunctpolym.2022.105241174(105241)Online publication date: May-2022
https://doi.org/10.1016/j.reactfunctpolym.2022.105241
Xu HZhou PTan RLi MShen G(2021)LIMU-BERTProceedings of the 19th ACM Conference on Embedded Networked Sensor Systems10.1145/3485730.3485937(220-233)Online publication date: 15-Nov-2021
https://dl.acm.org/doi/10.1145/3485730.3485937
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents