Abstract
In this paper, we propose a new attention model by integrating multi-scale features to recognize human action. We introduce multi-scale features through different sizes of convolution kernel on both spatial and temporal fields. The spatial attention model considers the relationship between detail and integral of the human action, therefore our model can focus on the significant part of the action on the spatial field. The temporal attention model considers the speed of action, in order that our model can concentrate on the pivotal clips of the action on the temporal field. We verify the validity of multi-scale features in the benchmark action recognition datasets, including UCF-101 (\(88.8\%\)), HMDB-51 (\(60.0\%\)) and Penn (\(96.3\%\)). As a result that the accuracy of our model outperforms the previous methods.
The first author of this paper is an undergraduate.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Sharma, S., Kiros, R., Salakhutdinov, R.: Action recognition using visual attention. Comput. Sci. (2015)
Yu, T.Z., Guo, C.X., Wang, L.F., Gu, H.X., Xiang, S.M., Pan, C.H.: Joint spatial-temporal attention for action recognition. Pattern Recogn. Lett. 112, 226–233 (2018)
Dempsey, P.W., Allison, M.E.D., Akkaraju, S., et al.: C3d of complement as a molecular adjuvant: bridging innate and acquired immunity. Science 271(5247), 348–350 (1996)
Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos (2014)
Balaguer, J.F., Gobbetti, E.: i3D: a high-speed 3D Web browser. In: Proceedings ACM Symposium on VRML, pp. 69–76 (1995)
Qiu, Z., Yao, T., Mei, T.: Learning spatio-temporal representation with pseudo-3D residual networks (2017)
Wang, L., Xiong, Y., Wang, Z., et al.: Towards good practices for very deep two-stream ConvNets. Comput. Sci. (2015)
Feichtenhofer, C., Pinz, A., Zisserman, A.: Convolutional two-stream network fusion for video action recognition. In: Computer Vision Pattern Recognition (2016)
Kosiorek, A.R., Bewley, A., Posner, I.: Hierarchical attentive recurrent tracking (2017)
Wang, F., Jiang, M., Qian, C., et al.: Residual attention network for image classification (2017)
Hu, J., Shen, L., Albanie, S., et al.: Squeeze-and-excitation networks (2017)
Mnih, V., Heess, N., Graves, A., et al.: Recurrent models of visual attention. In: Advances in Neural Information Processing Systems (2014)
Chen, L.C., Yi, Y., Jiang, W., et al.: Attention to scale: scale-aware semantic image segmentation. In: IEEE Conference on Computer Vision Pattern Recognition (2016)
Florack, L., Romeny, B.T.H., Viergever, M., et al.: The Gaussian scale-space paradigm and the multiscale local jet. Int. J. Comput. Vision 18(1), 61–75 (1996)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 640–651 (2014)
Lin, G.S., Shen, C.H., Hengel, A., Reid, I.: Efficient piecewise training of deep structured models for semantic segmentation. In: CVPR, pp. 3194–3203 (2016)
He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition (2015)
Clement, F., Couprie, C., Najman, L., et al.: Learning hierarchical features for scene labeling. IEEE Trans. Pattern Anal. Mach. Intell. (2013)
Wang, L., et al.: Temporal segment networks: towards good practices for deep action recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 20–36. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_2
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Feifei, L.: Large-scale video classification with convolutional neural networks. In: CVPR, pp. 1725–1732 (2014)
Donahue, J., et al.: Long-term recurrent convolutional networks for visual recognition and description. In: CVPR, pp. 2625–2634 (2015)
Sun, L., Jia, K., Shi, B.: Human action recognition using factorized spatio-temporal convolutional networks. In: ICCV, pp. 57–65 (2015)
Rohit, G., Deva, R.: Attentional pooling for action recognition. In: NIPS, pp. 33–44 (2017)
Cao, C., Zhang, Y., Zhang, C., Lu, H.: Body joint guided 3D deep convolutional descriptors for action recognition, CoRR. abs/1704.07160 (2017)
Yu, T., Gu, H., Wang, L., Xiang, S., Pan, C.: Cascaded temporal spatial features for video action recognition. In: ICIP, pp. 1552–1556 (2017)
Feichtenhofer, C., Pinz, A., Zisserman, A.: Convolutional two-stream network fusion for video action recognition. In: CVPR, pp. 1933–1941 (2016)
Soomro, K., Zamir, A.R., Shah, M.: UCF101: a dataset of 101 human actions classes from videos in the wild, CoRR. abs/1212.0402 (2012)
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., Serre, T.: HMDB: a large video database for human motion recognition. In: ICCV, pp. 2556–2563 (2011)
Zhang, W., Zhu, M., Derpanis, K.: From actemes to action: a strongly-supervised representation for detailed action understanding. In: ICCV, pp. 2248–2255 (2013)
Wang, H., Schmid, C.: Action recognition with improved trajectories. In: ICCV, pp. 3551–3558 (2013)
Iqbal, U., Garbade, M., Gall, J.: Pose for action-action for pose. In: FG, pp. 438–445 (2017)
Acknowledgments
This work is supported by the National Natural Science Foundation of China (Grant Number 61773377 and 61573352).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, Q., Yan, H., Wang, L. (2019). Multi-scale Spatial-Temporal Attention for Action Recognition. In: Lin, Z., et al. Pattern Recognition and Computer Vision. PRCV 2019. Lecture Notes in Computer Science(), vol 11857. Springer, Cham. https://doi.org/10.1007/978-3-030-31654-9_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-31654-9_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-31653-2
Online ISBN: 978-3-030-31654-9
eBook Packages: Computer ScienceComputer Science (R0)