poster

Towards a data-driven method for RGB video-based hand action quality assessment in real time

Authors:

Mian LiAuthors Info & Claims

SAC '20: Proceedings of the 35th Annual ACM Symposium on Applied Computing

Pages 2117 - 2120

https://doi.org/10.1145/3341105.3374092

Published: 30 March 2020 Publication History

Abstract

In recent years, the research community has begun to explore Video-Based Action Quality Assessment on Human Body (VB-AQA), while few work focuses on Video-Based Action Quality Assessment on Human Hand (VH-AQA) yet. The current work on VB-AQA fails to deal with the inconsistency between captured features and the reality due to the changing angles of the camera, leaving a huge gap between VB-AQA and VH-AQA, while the computational efficiency is another critical problem. In this paper, a novel data-driven method for real-time VH-AQA is proposed. Features are formulated as spatio-temporal hand poses in this method and extracted via four steps: hand segmentation, 2D hand pose estimation, 3D hand pose estimation and hand pose organization. Based on the extracted features an assessment model is applied to evaluate the performance of actions and indicate the most promising adjustment as the feedback. We demonstrate the evaluation accuracy and computational efficiency of our method using our own Origami Video Dataset. For the latter, two new metrics are designed. It turns out that our method provides opportunities for real-time digital reconstruction of physical world activities and timely assessment.

References

[1]

N. Ahmed, T. Natarajan, and K. R. Rao. 1974. Discrete Cosine Transform. IEEE Trans. Comput. C-23, 1 (Jan. 1974), 90--93.

Digital Library

[2]

James W. Cooley and John W. Tukey. 1965. An Algorithm for the Machine Calculation of Complex Fourier Series. Math. Comp. 19, 90 (Apr. 1965), 297--301.

[3]

Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine Learning 20, 3 (Sep. 1995), 273--297.

[4]

Eugene d'Eon, David Luebke, and Eric Enderton. 2007. Efficient Rendering of Human Skin. In Proceedings of the 18th Eurographics Conference on Rendering Techniques. Eurographics Association, Aire-la-Ville, CH, 147--157.

[5]

Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (Nov. 1997), 1735--1780.

Digital Library

[6]

Yongjun Li, Xiujuan Chai, and Xilin Chen. 2018. End-To-End Learning for Action Quality Assessment. In Advances in Multimedia Information Processing - PCM 2018. Springer International Publishing, 125--134.

Digital Library

[7]

Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and Christian Theobalt. 2018. GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Press, 49--59.

[8]

Paritosh Parmar and Brendan Morris. 2017. Learning to Score Olympic Events. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE Press, Honolulu, HI, 76--84.

[9]

Fotini Patrona, Anargyros Chatzitofis, Dimitrios Zarpalas, and Petros Daras. 2018. Motion Analysis: Action Detection, Recognition and Evaluation based on motion capture data. Pattern Recognition 76 (Apr. 2018), 612 -- 622.

[10]

Hamed Pirsiavash, Carl Vondrick, and Antonio Torralba. 2014. Assessing the Quality of Actions. In 2014 European Conference on Computer Vision (ECCV). Springer International Publishing, Cham, DE, 556--571.

[11]

Tomas Simon, Hanbyul Joo, Iain A. Matthews, and Yaser Sheikh. 2017. Hand Keypoint Detection in Single Images Using Multiview Bootstrapping. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Press, 1145--1153.

[12]

Vinay Venkataraman, Ioannis Vlachos, and Pavan Turaga. 2015. Dynamical Regularity for Action Analysis. In Proceedings of the British Machine Vision Conference 2015. BMVA Press, Swansea, UK, 67.1--67.12.

[13]

Dibia Victor. 2017. Real-time Hand Tracking Using SSD on Tensorflow.

[14]

Yi Wang. 2018. Fitness Movement Recognition and Evaluation Based on Kinect. Computer Science and Application 8 (Jan. 2018), 1134--1145.

[15]

Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. 2016. Convolutional Pose Machines. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Press, Las Vegas, NV, 4724--4732.

[16]

Christian Zimmermann and Thomas Brox. 2017. Learning to Estimate 3D Hand Pose From Single RGB Images. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE Press, Venice, ITL, 4903--4911.

Cited By

Parmar PMorris B(2022)Win-Fail Action Recognition2022 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW)10.1109/WACVW54805.2022.00022(161-171)Online publication date: Jan-2022
https://doi.org/10.1109/WACVW54805.2022.00022
Parmar PReddy JMorris B(2021)Piano Skills Assessment2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP)10.1109/MMSP53017.2021.9733638(1-5)Online publication date: 6-Oct-2021
https://doi.org/10.1109/MMSP53017.2021.9733638

Index Terms

Towards a data-driven method for RGB video-based hand action quality assessment in real time
1. Applied computing
  1. Education
    1. Computer-assisted instruction
2. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Activity recognition and understanding
      2. Image and video acquisition
        Motion capture

Recommendations

A real time vision-based hand gestures recognition system
ISICA'10: Proceedings of the 5th international conference on Advances in computation and intelligence

Hand gesture recognition is an important aspect in Human-Computer interaction, and can be used in various applications, such as virtual reality and computer games. In this paper, we propose a real time hand gesture recognition system. It includes three ...
Real time eye movement identification protocol
CHI EA '10: CHI '10 Extended Abstracts on Human Factors in Computing Systems

This paper introduces a Real Time Eye Movement Identification (REMI) protocol designed to address challenges related to the implementation of the eye-gaze guided computer interfaces. The REMI protocol provides the framework for 1) eye position data ...
A novel finger and hand pose estimation technique for real-time hand gesture recognition

This paper presents a high-level hand feature extraction method for real-time gesture recognition. Firstly, the fingers are modelled as cylindrical objects due to their parallel edge feature. Then a novel algorithm is proposed to directly extract ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SAC '20: Proceedings of the 35th Annual ACM Symposium on Applied Computing

March 2020

2348 pages

ISBN:9781450368667

DOI:10.1145/3341105

Conference Chairs:
Chih-Cheng Hung
Kennesaw State University
,
Tomas Cerny
Baylor University
,
Program Chairs:
Dongwan Shin
New Mexico Tech
,
Alessio Bechini
University of Pisa, Italy

Copyright © 2020 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGAPP: ACM Special Interest Group on Applied Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 March 2020

Check for updates

Author Tags

Qualifiers

Poster

Funding Sources

Science and Technology Commission of Shanghai Municipality

Conference

SAC '20

Sponsor:

SIGAPP

SAC '20: The 35th ACM/SIGAPP Symposium on Applied Computing

March 30 - April 3, 2020

Brno, Czech Republic

Acceptance Rates

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
208
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)2

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Parmar PMorris B(2022)Win-Fail Action Recognition2022 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW)10.1109/WACVW54805.2022.00022(161-171)Online publication date: Jan-2022
https://doi.org/10.1109/WACVW54805.2022.00022
Parmar PReddy JMorris B(2021)Piano Skills Assessment2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP)10.1109/MMSP53017.2021.9733638(1-5)Online publication date: 6-Oct-2021
https://doi.org/10.1109/MMSP53017.2021.9733638

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents