Abstract
Action recognition aims to identify the action categories and features in the video by analyzing the actions and behavior patterns that are significant to the development of intelligent security, automatic driving, smart home, and other fields. However, current methods fail to adequately model the spatio-temporal relationships of actions in videos, and video annotation is a time-consuming and expensive process. This paper proposes a Self-Supervised Spatio-Temporal Attention Network (SSTA-Net) for action recognition to solve the above problems. Firstly, we use a self-supervised method for training, which does not require a large amount of labeled data and can explore unknown or hidden information in the data. Secondly, in the feature extraction part, Multi-Scale Convolution Attention Module (MC-AM) is proposed. By performing convolution operations on the input image at different scales, the details and edge information in the image are enhanced, and the image quality of the original sampling frame is improved. Finally, a Spatio-Temporal Attention Module (ST-A) is proposed. The module is used to capture the spatio-temporal signal sensitivity in the video, which effectively improves the accuracy of action recognition.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Beye, F., Shinohara, Y., Itsumi, H., Nihei, K.: Recognition-aware bitrate allocation for ai-enabled remote video surveillance. In: Proceedings of the IEEE 20th Consumer Communications & Networking Conference, pp. 158–163 (2023)
Gupta, S., Maple, C., Crispo, B., Raja, K., Yautsiukhin, A., Martinelli, F.: A survey of human-computer interaction (hci) & natural habits-based behavioural biometric modalities for user recognition schemes. Pattern Recogn. 139, 109453 (2023)
Herath, S., Harandi, M., Porikli, F.: Going deeper into action recognition: a survey. Image Vis. Comput. 60, 4–21 (2017)
Suh, S., Rey, V.F., Lukowicz, P.: Adversarial deep feature extraction network for user independent human activity recognition. In: Proceedings of the IEEE International Conference on Pervasive Computing and Communications, pp. 217–226 (2022)
Hwang, Y.M., Park, S., Lee, H.O., Ko, S.K., Lee, B.T.: Deep learning for human activity recognition based on causality feature extraction. IEEE Access 9, 112257–112275 (2021)
Bertasius, G., Wang, H., Torresani, L.: Is space-time attention all you need for video understanding? In: Proceedings of the 38th International Conference on Machine Learning, pp. 813–824 (2021)
Hou, Y., Ma, Z., Liu, C., Loy, C.C.: Learning lightweight lane detection cnns by self attention distillation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1013–1021 (2019)
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650–9660 (2021)
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738 (2020)
Han, T., Xie, W., Zisserman, A.: Self-supervised co-training for video representation learning. Adv. Neural. Inf. Process. Syst. 33, 5679–5690 (2020)
Qian, R., Meng, T., Gong, B., Yang, M.H., Wang, H., Belongie, S., Cui, Y.: Spatiotemporal contrastive video representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6964–6974 (2021)
Wang, J., Jiao, J., Liu, Y.H.: Self-supervised video representation learning by pace prediction. In: Proceedings of the European Conference on Computer Vision, pp. 504–521 (2020)
Luo, D., Liu, C., Zhou, Y., Yang, D., Ma, C., Ye, Q., Wang, W.: Video cloze procedure for self-supervised spatio-temporal learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 11701–11708 (2020)
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., Schmid, C.: Vivit: a video vision transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6836–6846 (2021)
Wang, Z., Fang, Z., Wang, J., Yang, Y.: Vitaa: visual-textual attributes alignment in person search by natural language. In: Proceedings of the European Conference on Computer Vision, pp. 402–420 (2020)
Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., Feichtenhofer, C.: Multiscale vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6824–6835 (2021)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Grill, J.B., et al.: Bootstrap your own latent-a new approach to self-supervised learning. Adv. Neural. Inf. Process. Syst. 33, 21271–21284 (2020)
Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299–6308 (2017)
Soomro, K., Zamir, A.R., Shah, M.: Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)
Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6450–6459 (2018)
Feichtenhofer, C., Fan, H., Malik, J., He, K.: Slowfast networks for video recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6202–6211 (2019)
Yang, Q., Lu, T., Zhou, H.: A spatio-temporal motion network for action recognition based on spatial attention. Entropy 24, 368 (2022)
Feichtenhofer, C.: X3d: Expanding architectures for efficient video recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 203–213 (2020)
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. In: Proceedings of the 9th International Conference on Learning Representations, pp. 132–141 (2021)
Tao, L., Wang, X., Yamasaki, T.: An improved inter-intra contrastive learning framework on self-supervised video representation. IEEE Trans. Circuits Syst. Video Technol. 32, 5266–5280 (2022)
Dave, I., Gupta, R., Rizve, M.N., Shah, M.: Tclr: temporal contrastive learning for video representation. Comput. Vis. Image Underst. 219, 103406 (2022)
Khorasgani, S.H., Chen, Y., Shkurti, F.: Slic: Self-supervised learning with iterative clustering for human action videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,. pp. 16091–16101 (2022)
Ranasinghe, K., Naseer, M., Khan, S., Khan, F.S., Ryoo, M.S.: Self-supervised video transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2874–2884 (2022)
Acknowledgement
This work was supported in part by the National Natural Science Foundation of China under Grant 61971273 and in part by the Fund Program for the Scientific Activities of Selected Returned Overseas Professionals in Shaanxi Province of China under Grant 2021-012.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, Y., Zhang, W., Pei, Z. (2023). SSTA-Net: Self-supervised Spatio-Temporal Attention Network for Action Recognition. In: Lu, H., et al. Image and Graphics. ICIG 2023. Lecture Notes in Computer Science, vol 14356. Springer, Cham. https://doi.org/10.1007/978-3-031-46308-2_32
Download citation
DOI: https://doi.org/10.1007/978-3-031-46308-2_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46307-5
Online ISBN: 978-3-031-46308-2
eBook Packages: Computer ScienceComputer Science (R0)