research-article

Frame-Level Event Detection in Athletics Videos with Pose-Based Convolutional Sequence Networks

Authors:

Moritz Einfalt,

Charles Dampeyrou,

Rainer LienhartAuthors Info & Claims

MMSports '19: Proceedings Proceedings of the 2nd International Workshop on Multimedia Content Analysis in Sports

Pages 42 - 50

https://doi.org/10.1145/3347318.3355525

Published: 15 October 2019 Publication History

Abstract

In this paper we address the problem of automatic event detection in athlete motion for automated performance analysis in athletics. We specifically consider the detection of stride-, jump- and landing related events from monocular recordings in long and triple jump. Existing work on event detection in sports often uses manually designed features on body and pose configurations of the athlete to infer the occurrence of events. We present a two-step approach, where temporal 2D pose sequences extracted from the videos form the basis for learning an event detection model. We formulate the detection of discrete events as a sequence translation task and propose a convolutional sequence network that can accurately predict the timing of event occurrences. Our best performing architecture achieves a precision/recall of 92.3%/89.0% in detecting start and end of ground contact during the run-up and jump of an athlete at a temporal precision of +/- 1 frame at 200Hz. The results show that 2D pose sequences are a suitable motion representation for learning event detection in a sequence-to-sequence framework.

References

[1]

Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. 2018. An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. CoRR, Vol. abs/1803.01271 (2018). arxiv: 1803.01271 http://arxiv.org/abs/1803.01271

[2]

Yoshua Bengio, Patrice Simard, Paolo Frasconi, et almbox. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, Vol. 5, 2 (1994), 157--166.

[3]

Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .

[4]

Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun. 2018. Cascaded Pyramid Network for Multi-Person Pose Estimation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .

[5]

Chuan Wu, Yu-Fei Ma, Hong-Jiang Zhan, and Yu-Zhuo Zhong. 2002. Events recognition by semantic inference for sports video. In Proceedings. IEEE International Conference on Multimedia and Expo, Vol. 1. 805--808 vol.1.

[6]

Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language Modeling with Gated Convolutional Networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 (ICML'17). JMLR.org, 933--941. http://dl.acm.org/citation.cfm?id=3305381.3305478

[7]

Pablo Fernández de Dios, Qinggang Meng, and Paul WH Chung. 2013. A machine learning method for identification of key body poses in cyclic physical exercises. In 2013 IEEE International Conference on Systems, Man, and Cybernetics. IEEE, 1605--1610.

Digital Library

[8]

Claudio Marcio de Souza Vicente, Erickson R Nascimento, Luiz Eduardo C Emery, Cristiano Arruda G Flor, Thales Vieira, and Leonardo B Oliveira. 2016. High performance moves recognition and sequence segmentation based on key poses filtering. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1--8.

[9]

Benedikt Fasel, Jörg Spörri, Julien Chardonnens, Josef Kröll, Erich Müller, and Kamiar Aminian. 2018. Joint Inertial Sensor Orientation Drift Reduction for Highly Dynamic Movements. IEEE Journal of Biomedical and Health Informatics, Vol. 22, 1 (Jan 2018), 77--86. https://doi.org/10.1109/JBHI.2017.2659758

[10]

Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional Sequence to Sequence Learning. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 (ICML'17). JMLR.org, 1243--1252. http://dl.acm.org/citation.cfm?id=3305381.3305510

[11]

Ross Girshick. 2015. Fast R-CNN. In The IEEE International Conference on Computer Vision (ICCV) .

[12]

Georgia Gkioxari, Alexander Toshev, and Navdeep Jaitly. 2016. Chained Predictions Using Convolutional Neural Networks. In Computer Vision -- ECCV 2016, Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, Cham, 728--743.

[13]

Kohei Hakozaki, Naoki Kato, Masamoto Tanabiki, Junko Furuyama, Yuji Sato, and Yoshimitu Aoki. 2018. Swimmer's Stroke Estimation Using CNN and MultiLSTM. Journal of Signal Processing, Vol. 22, 4 (2018), 219--222.

[14]

Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. 2017. Mask R-CNN. In The IEEE International Conference on Computer Vision (ICCV) .

[15]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .

[16]

Peter J Huber et almbox. 1973. Robust regression: asymptotics, conjectures and Monte Carlo. The Annals of Statistics, Vol. 1, 5 (1973), 799--821.

[17]

Shariman Ismadi Ismail, Hiroyuki Nunome, Fatin Farhana Marzuki, and Izzat Su'aidi. 2018. The Influence of Additional Surface on Force Platform's Ground Reaction Force Data During Walking and Running. American Journal of Sports Science, Vol. 6, 3 (2018), 78--82.

[18]

Kyoungoh Lee, Inwoong Lee, and Sanghoon Lee. 2018. Propagating LSTM: 3D Pose Estimation based on Joint Interdependency. In The European Conference on Computer Vision (ECCV) .

Digital Library

[19]

Chen Li, Zhen Zhang, Wee Sun Lee, and Gim Hee Lee. 2018. Convolutional Sequence to Sequence Model for Human Dynamics. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .

[20]

Haojie Li, Jinhui Tang, Si Wu, Yongdong Zhang, and Shouxun Lin. 2010. Automatic Detection and Analysis of Player Action in Moving Background Sports Video Sequences. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 20, 3 (March 2010), 351--364. https://doi.org/10.1109/TCSVT.2009.2035833

Digital Library

[21]

Rainer Lienhart, Moritz Einfalt, and Dan Zecha. 2018. Mining Automatically Estimated Poses from Video Recordings of Top Athletes. International Journal of Computer Science in Sport, Vol. 17, 2 (2018), 94 -- 112. https://content.sciendo.com/view/journals/ijcss/17/2/article-p94.xml

[22]

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollá r, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. CoRR, Vol. abs/1405.0312 (2014). arxiv: 1405.0312 http://arxiv.org/abs/1405.0312

[23]

Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017. Feature Pyramid Networks for Object Detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .

[24]

Yue Luo, Jimmy Ren, Zhouxia Wang, Wenxiu Sun, Jinshan Pan, Jianbo Liu, Jiahao Pang, and Liang Lin. 2018. LSTM Pose Machines. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .

[25]

Diogo C. Luvizon, David Picard, and Hedi Tabia. 2018. 2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .

[26]

Alejandro Newell, Kaiyu Yang, and Jia Deng. 2016. Stacked hourglass networks for human pose estimation. In European Conference on Computer Vision. Springer, 483--499.

[27]

Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli. 2019. 3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .

[28]

Mir Rayat Imtiaz Hossain and James J. Little. 2018. Exploiting temporal information for 3D human pose estimation. In The European Conference on Computer Vision (ECCV) .

[29]

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Advances in Neural Information Processing Systems 28, C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (Eds.). Curran Associates, Inc., 91--99.

Digital Library

[30]

Ben Sapp and Ben Taskar. 2013. MODEC: Multimodal Decomposable Models for Human Pose Estimation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .

[31]

Long Sha, Patrick Lucey, Sridha Sridharan, Stuart Morgan, and Dave Pease. 2014. Understanding and analyzing a large collection of archived swimming videos. In IEEE Winter Conference on Applications of Computer Vision. IEEE, 674--681.

[32]

Brandon Victor, Zhen He, Stuart Morgan, and Dino Miniutti. 2017. Continuous Video to Simple Signals for Swimming Stroke Detection With Convolutional Neural Networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops .

[33]

Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. 2016. Convolutional pose machines. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4724--4732.

[34]

Kentaro Yagi, Kunihiro Hasegawa, Yuta Sugiura, and Hideo Saito. 2018. Estimation of Runners' Number of Steps, Stride Length and Speed Transition from Video of a 100-Meter Race. In Proceedings of the 1st International Workshop on Multimedia Content Analysis in Sports (MMSports'18). ACM, New York, NY, USA, 87--95. https://doi.org/10.1145/3265845.3265850

Digital Library

[35]

Dan Zecha, Christian Eggert, Moritz Einfalt, Stephan Brehm, and Rainer Lienhart. 2018. A Convolutional Sequence to Sequence Model for Multimodal Dynamics Prediction in Ski Jumps. In Proceedings of the 1st International Workshop on Multimedia Content Analysis in Sports (MMSports'18). ACM, New York, NY, USA, 11--19. https://doi.org/10.1145/3265845.3265855

Digital Library

[36]

Dan Zecha, Christian Eggert, and Rainer Lienhart. 2017. Pose Estimation for Deriving Kinematic Parameters of Competitive Swimmers. Electronic Imaging, Vol. 2017, 16 (2017), 21--29. https://doi.org/

Cited By

Moaveninejad SJanes APorcaro C(2024)Detection of Lowering in Sport Climbing Using Orientation-Based Sensor-Enhanced Quickdraws: A Preliminary InvestigationSensors10.3390/s2414457624:14(4576)Online publication date: 15-Jul-2024
https://doi.org/10.3390/s24144576
Moaveninejad SJanes APorcaro CBarletta LMucchi LPierobon M(2024)Climbing Routes Clustering Using Energy-Efficient Accelerometers Attached to the QuickdrawsBody Area Networks. Smart IoT and Big Data for Intelligent Health Management10.1007/978-3-031-72524-1_14(177-193)Online publication date: 27-Dec-2024
https://doi.org/10.1007/978-3-031-72524-1_14
Li XTao RWang YDing YLuo X(2024)Sport Action Evaluation Based on Human Pose Estimation: A Case of Evaluating Golf Swing ActionLearning Technology for Education Challenges10.1007/978-3-031-61678-5_6(65-78)Online publication date: 22-May-2024
https://doi.org/10.1007/978-3-031-61678-5_6
Show More Cited By

Index Terms

Frame-Level Event Detection in Athletics Videos with Pose-Based Convolutional Sequence Networks
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Object detection
      2. Computer vision tasks
        Visual content-based indexing and retrieval
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

Improving document-level event detection with event relation graph
Highlights
- An event correlation-based document-level event detection model is proposed.
- An ...
Abstract
The correlation between events within the same document plays a crucial role in event detection. Most existing detection models often ignore event correlations, which is not applicable to multi-event detection at the document level. In ...
Improving Event Detection by Automatically Assessing Validity of Event Occurrence in Text
CIKM '15: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management

Manually inspecting text to assess whether an event occurs in a document collection is an onerous and time consuming task. Although a manual inspection to discard the false events would increase the precision of automatically detected sets of events, it ...
Event Detection and Highlight Detection of Broadcasted Game Videos
HCMC '15: Proceedings of the 2nd Workshop on Computational Models of Social Interactions: Human-Computer-Media Communication

Efficient access of game videos is urgently demanded due to the emergence of live streaming platforms and the explosive numbers of gamers and viewers. In this work we facilitate efficient access from two aspects: game event detection and highlight ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MMSports '19: Proceedings Proceedings of the 2nd International Workshop on Multimedia Content Analysis in Sports

October 2019

120 pages

ISBN:9781450369114

DOI:10.1145/3347318

General Chairs:
Rainer Lienhart
University of Augsburg, Germany
,
Thomas B. Moeslund
Aalborg University, Denmark
,
Hideo Saito
Keio University, Japan

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '19

Sponsor:

SIGMM

MM '19: The 27th ACM International Conference on Multimedia

October 25, 2019

Nice, France

Acceptance Rates

Overall Acceptance Rate 29 of 49 submissions, 59%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
221
Total Downloads

Downloads (Last 12 months)28
Downloads (Last 6 weeks)5

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Moaveninejad SJanes APorcaro C(2024)Detection of Lowering in Sport Climbing Using Orientation-Based Sensor-Enhanced Quickdraws: A Preliminary InvestigationSensors10.3390/s2414457624:14(4576)Online publication date: 15-Jul-2024
https://doi.org/10.3390/s24144576
Moaveninejad SJanes APorcaro CBarletta LMucchi LPierobon M(2024)Climbing Routes Clustering Using Energy-Efficient Accelerometers Attached to the QuickdrawsBody Area Networks. Smart IoT and Big Data for Intelligent Health Management10.1007/978-3-031-72524-1_14(177-193)Online publication date: 27-Dec-2024
https://doi.org/10.1007/978-3-031-72524-1_14
Li XTao RWang YDing YLuo X(2024)Sport Action Evaluation Based on Human Pose Estimation: A Case of Evaluating Golf Swing ActionLearning Technology for Education Challenges10.1007/978-3-031-61678-5_6(65-78)Online publication date: 22-May-2024
https://doi.org/10.1007/978-3-031-61678-5_6
Kathan AAmiriparian SGebhard ATriantafyllopoulos AGerczuk MSchuller BLienhart RMoeslund TSaito H(2023)Personalised Speech-Based Heart Rate Categorisation Using Weighted-Instance LearningProceedings of the 6th International Workshop on Multimedia Content Analysis in Sports10.1145/3606038.3616155(9-13)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3606038.3616155
Gebhard ATriantafyllopoulos AAmiriparian SOttl SDieter VGerczuk MJaumann MHildner DSchneeweiß PRösel IKrauß ISchuller BLienhart RMoeslund TSaito H(2022)Improving Exertion and Wellbeing Prediction in Outdoor Running Conditions using Audio-based Surface RecognitionProceedings of the 5th International ACM Workshop on Multimedia Content Analysis in Sports10.1145/3552437.3555700(19-27)Online publication date: 14-Oct-2022
https://dl.acm.org/doi/10.1145/3552437.3555700
Roygaga CPatil DBoyle MPickard WReisers RBharati ABlanchard N(2022)APE-V: Athlete Performance Evaluation using Video2022 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW)10.1109/WACVW54805.2022.00076(691-700)Online publication date: Jan-2022
https://doi.org/10.1109/WACVW54805.2022.00076
Akhter IJaveed M(2022)Pedestrian Behavior Recognition via a Smart Graph-based Optimization2022 19th International Bhurban Conference on Applied Sciences and Technology (IBCAST)10.1109/IBCAST54850.2022.9990434(629-634)Online publication date: 16-Aug-2022
https://doi.org/10.1109/IBCAST54850.2022.9990434
Ghadi YAkhter IAlarfaj MJalal AKim K(2021)Syntactic model-based human body 3D reconstruction and event classification via association based features mining and deep learningPeerJ Computer Science10.7717/peerj-cs.7647(e764)Online publication date: 19-Nov-2021
https://doi.org/10.7717/peerj-cs.764
Stenum JCherry-Allen KPyles CReetzke RVignos MRoemmich R(2021)Applications of Pose Estimation in Human Health and Performance across the LifespanSensors10.3390/s2121731521:21(7315)Online publication date: 3-Nov-2021
https://doi.org/10.3390/s21217315
Badiola-Bengoa AMendez-Zorrilla A(2021)A Systematic Review of the Application of Camera-Based Human Pose Estimation in the Field of Sport and Physical ExerciseSensors10.3390/s2118599621:18(5996)Online publication date: 7-Sep-2021
https://doi.org/10.3390/s21185996
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents