research-article

Graph-based motion prediction for abnormal action detection

Authors:

Jian YangAuthors Info & Claims

MMAsia '20: Proceedings of the 2nd ACM International Conference on Multimedia in Asia

Article No.: 63, Pages 1 - 7

https://doi.org/10.1145/3444685.3446316

Published: 03 May 2021 Publication History

Abstract

Abnormal action detection is the most noteworthy part of anomaly detection, which tries to identify unusual human behaviors in videos. Previous methods typically utilize future frame prediction to detect frames deviating from the normal scenario. While this strategy enjoys success in the accuracy of anomaly detection, critical information such as the cause and location of the abnormality is unable to be acquired. This paper proposes human motion prediction for abnormal action detection. We employ sequence of human poses to represent human motion, and detect irregular behavior by comparing the predicted pose with the actual pose detected in the frame. Hence the proposed method is able to explain why the action is regarded as irregularity and locate where the anomaly happens. Moreover, pose sequence is robust to noise, complex background and small targets in videos. Since posture information is non-Euclidean data, graph convolutional network is adopted for future pose prediction, which not only leads to greater expressive power but also stronger generalization capability.

Experiments are conducted both on the widely used anomaly detection dataset ShanghaiTech and our newly proposed dataset NJUST-Anomaly, which mainly contains irregular behaviors happened in the campus. Our dataset expands the existing datasets by giving more abnormal actions attracting public attention in social security, which happen in more complex scenes and dynamic backgrounds. Experimental results on both datasets demonstrate the superiority of our method over the-state-of-the-art methods. The source code and NJUST-Anomaly dataset will be made public at https://github.com/datangzhengqing/MP-GCN.

References

[1]

Davide Abati, Angelo Porrello, Simone Calderara, and Rita Cucchiara. 2019. Latent space autoregression for novelty detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 481--490.

[2]

Sadegh Aliakbarian, Fatemeh Sadat Saleh, Mathieu Salzmann, Lars Petersson, and Stephen Gould. 2020. A stochastic conditioning scheme for diverse human motion prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5223--5232.

[3]

Judith Butepage, Michael J Black, Danica Kragic, and Hedvig Kjellstrom. 2017. Deep representation learning for human motion prediction and classification. In Proceedings of the IEEE conference on computer vision and pattern recognition. 6158--6166.

[4]

Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. 2015. Recurrent network models for human dynamics. In Proceedings of the IEEE International Conference on Computer Vision. 4346--4354.

Digital Library

[5]

Ross Girshick. 2015. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision. 1440--1448.

Digital Library

[6]

LiangYan Gui, YuXiong Wang, Xiaodan Liang, and José MF Moura. 2018. Adversarial geometry-aware human motion prediction. In Proceedings of the European Conference on Computer Vision (ECCV). 786--803.

Digital Library

[7]

Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).

[8]

Vineet Kosaraju, Amir Sadeghian, Roberto Martín-Martín, Ian Reid, Hamid Rezatofighi, and Silvio Savarese. 2019. Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. In Advances in Neural Information Processing Systems. 137--146.

[9]

Chen Li, Zhen Zhang, Wee Sun Lee, and Gim Hee Lee. 2018. Convolutional sequence to sequence model for human dynamics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5226--5234.

[10]

Maosen Li, Siheng Chen, Yangheng Zhao, Ya Zhang, Yanfeng Wang, and Qi Tian. 2020. Dynamic multiscale graph neural networks for 3D skeleton based human motion prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 214--223.

[11]

Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. 2020. Learning lane graph representations for motion forecasting. In Proceedings of the European Conference on Computer Vision (ECCV).

Digital Library

[12]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision. Springer, 740--755.

[13]

Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao. 2018. Future frame prediction for anomaly detection-a new baseline. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6536--6545.

[14]

Cewu Lu, Jianping Shi, and Jiaya Jia. 2013. Abnormal event detection at 150 fps in matlab. In Proceedings of the IEEE international conference on computer vision. 2720--2727.

Digital Library

[15]

Weixin Luo, Wen Liu, and Shenghua Gao. 2017. A revisit of sparse coding based anomaly detection in stacked rnn framework. In Proceedings of the IEEE International Conference on Computer Vision. 341--349.

[16]

Vijay Mahadevan, Weixin Li, Viral Bhalodia, and Nuno Vasconcelos. 2010. Anomaly detection in crowded scenes. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 1975--1981.

[17]

Wei Mao, Miaomiao Liu, Mathieu Salzmann, and Hongdong Li. 2019. Learning trajectory dependencies for human motion prediction. In Proceedings of the IEEE International Conference on Computer Vision. 9489--9497.

[18]

Julieta Martinez, Michael J Black, and Javier Romero. 2017. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2891--2900.

[19]

Romero Morais, Vuong Le, Truyen Tran, Budhaditya Saha, Moussa Mansour, and Svetha Venkatesh. 2019. Learning regularity in skeleton trajectories for anomaly detection in videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 11996--12004.

[20]

Asim Munawar, Phongtharin Vinayavekhin, and Giovanni De Magistris. 2017. Spatio-temporal anomaly detection for industrial robots through prediction in unsupervised feature space. In IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1017--1025.

[21]

Hyunjong Park, Jongyoun Noh, and Bumsub Ham. 2020. Learning memory-guided normality for anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 14372--14381.

[22]

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234--241.

[23]

Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. 2019. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5693--5703.

[24]

Yao Tang, Lin Zaho, Shanshan Zhang, Chen Gong, Guangyu Li, and Jian Yang. 2020. Integrating prediction and reconstruction for anomaly detection. Pattern Recognition Letters 129 (2020), 123--130.

[25]

Kalpit Thakkar and PJ Narayanan. 2018. Part-based graph convolutional network for action recognition. In British Machine Vision Conference (BMVC).

[26]

Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903 (2017).

[27]

Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert. 2017. The pose knows: Video forecasting by generating pose futures. In Proceedings of the IEEE international conference on computer vision. 3332--3341.

[28]

Borui Wang, Ehsan Adeli, Hsu kuang Chiu, De-An Huang, and Juan Carlos Niebles. 2019. Imitation learning for human pose prediction. In Proceedings of the IEEE International Conference on Computer Vision. 7124--7133.

[29]

Nicolai Wojke, Alex Bewley, and Dietrich Paulus. 2017. Simple online and realtime tracking with a deep association metric. In 2017 IEEE international conference on image processing (ICIP). IEEE, 3645--3649.

Digital Library

[30]

Sijie Yan, Yuanjun Xiong, and Dahua Lin. 2018. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Proceedings of the thirty-second AAAI conference on artificial intelligence. 1--9.

[31]

Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris N Metaxas. 2019. Semantic graph convolutional networks for 3D human pose regression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3425--3435.

[32]

Yiru Zhao, Bing Deng, Chen Shen, Yao Liu, Hongtao Lu, and Xian-Sheng Hua. 2017. Spatio-temporal autoencoder for video anomaly detection. In Proceedings of the 25th ACM international conference on Multimedia. 1933--1941.

Digital Library

[33]

Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. 2019. Objects as points. arXiv preprint arXiv:1904.07850 (2019).

Cited By

Gao XYang YWu YDu SQi G(2024)Multi-Condition Latent Diffusion Network for Scene-Aware Neural Human Motion PredictionIEEE Transactions on Image Processing10.1109/TIP.2024.341493533(3907-3920)Online publication date: 2024
https://doi.org/10.1109/TIP.2024.3414935
Yoo J(2023)A Golf Impact Detection Method based on High-resolution Mono CameraThe Journal of Korean Institute of Information Technology10.14801/jkiit.2023.21.1.10921:1(109-120)Online publication date: 31-Jan-2023
https://doi.org/10.14801/jkiit.2023.21.1.109
Liao CHwang DWu EKoike HWard JMcGill MMarky K(2023)AI Coach: A Motor Skill Training System using Motion Discrepancy DetectionProceedings of the Augmented Humans International Conference 202310.1145/3582700.3582710(179-189)Online publication date: 12-Mar-2023
https://dl.acm.org/doi/10.1145/3582700.3582710
Show More Cited By

Index Terms

Graph-based motion prediction for abnormal action detection
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Scene anomaly detection

Recommendations

An Improved Deep-learning Network for Abnormal Action Detection
ICCBN '22: Proceedings of the 10th International Conference on Communications and Broadband Networking

Detecting actions in untrimmed videos is a challenging task. In order to improve the accuracy of abnormal action detection, an improved network combining Boundary-Matching Network (BMN) and Structured Segment Network (SSN) is proposed in this paper to ...
Joint representation learning of appearance and motion for abnormal event detection

In this paper, we propose a joint learning of spatio-temporal representation based on 3D deep convolutional neural network for simultaneous representation of appearance and motion information in 3D volumes which are extracted from the multiple ...
Anomaly graph: leveraging dynamic graph convolutional networks for enhanced video anomaly detection in surveillance and security applications
Abstract
Video abnormality behavior identification plays a pivotal role in improving the safety and security of surveillance systems by identifying unusual events within video streams. However, existing methods face challenges in capturing complex ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MMAsia '20: Proceedings of the 2nd ACM International Conference on Multimedia in Asia

March 2021

512 pages

ISBN:9781450383080

DOI:10.1145/3444685

General Chairs:
Tat-Seng Chua
National University of Singapore
,
Jingdong Wang
Microsoft Research
,
Qi Tian
Huawei Noah's Ark
,
Program Chairs:
Cathal Gurrin
Dublin City University
,
Jia Jia
Tsinghua University
,
Hanwang Zhang
Nanyang Technological University
,
Qianru Sun
Singapore Management University

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 May 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Fund of China
China Postdoctoral Science Foundation
National Science Fund of Jiangsu Province
the Fundamental Research Funds for the Central Universities

Conference

MMAsia '20

Sponsor:

SIGMM

MMAsia '20: ACM Multimedia Asia

March 7, 2021

Virtual Event, Singapore

Acceptance Rates

Overall Acceptance Rate 59 of 204 submissions, 29%

Upcoming Conference

MM '24

Sponsor:
sigmm

The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
222
Total Downloads

Downloads (Last 12 months)29
Downloads (Last 6 weeks)2

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Gao XYang YWu YDu SQi G(2024)Multi-Condition Latent Diffusion Network for Scene-Aware Neural Human Motion PredictionIEEE Transactions on Image Processing10.1109/TIP.2024.341493533(3907-3920)Online publication date: 2024
https://doi.org/10.1109/TIP.2024.3414935
Yoo J(2023)A Golf Impact Detection Method based on High-resolution Mono CameraThe Journal of Korean Institute of Information Technology10.14801/jkiit.2023.21.1.10921:1(109-120)Online publication date: 31-Jan-2023
https://doi.org/10.14801/jkiit.2023.21.1.109
Liao CHwang DWu EKoike HWard JMcGill MMarky K(2023)AI Coach: A Motor Skill Training System using Motion Discrepancy DetectionProceedings of the Augmented Humans International Conference 202310.1145/3582700.3582710(179-189)Online publication date: 12-Mar-2023
https://dl.acm.org/doi/10.1145/3582700.3582710
Ma NWu ZCheung YGuo YGao YLi JJiang B(2022)A Survey of Human Action Recognition and Posture PredictionTsinghua Science and Technology10.26599/TST.2021.901006827:6(973-1001)Online publication date: Dec-2022
https://doi.org/10.26599/TST.2021.9010068
Liao CHwang DKoike H(2022)AI Golf: Golf Swing Analysis Tool for Self-TrainingIEEE Access10.1109/ACCESS.2022.321026110(106286-106295)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3210261

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents