Discriminative Segment Focus Network for Fine-grained Video Action Recognition
Authors: 
Baoli Sun, Xinchen Ye, Tiantian Yan, Zhihui Wang, Haojie Li, Zhiyong WangAuthors Info & Claims
Baoli Sun, Xinchen Ye, Tiantian Yan, Zhihui Wang, Haojie Li, Zhiyong WangAuthors Info & ClaimsArticle No.: 218, Pages 1 - 20
Abstract
Fine-grained video action recognition aims at identifying minor and discriminative variations among fine categories of actions. While many recent action recognition methods have been proposed to better model spatio-temporal representations, how to model the interactions among discriminative atomic actions to effectively characterize inter-class and intra-class variations has been neglected, which is vital for understanding fine-grained actions. In this work, we devise a Discriminative Segment Focus Network (DSFNet) to mine the discriminability of segment correlations and localize discriminative action-relevant segments for fine-grained video action recognition. Firstly, we propose a hierarchic correlation reasoning (HCR) module which explicitly establishes correlations between different segments at multiple temporal scales and enhances each segment by exploiting the correlations with other segments. Secondly, a discriminative segment focus (DSF) module is devised to localize the most action-relevant segments from the enhanced representations of HCR by enforcing the consistency between the discriminability and the classification confidence of a given segment with a consistency constraint. Finally, these localized segment representations are combined with the global action representation of the whole video for boosting final recognition. Extensive experimental results on two fine-grained action recognition datasets, i.e., FineGym and Diving48, and two action recognition datasets, i.e., Kinetics400 and Something-Something, demonstrate the effectiveness of our approach compared with the state-of-the-art methods.
References
[1]
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun,Mario Lucic, and Cordelia Schmid. 2021. ViViT: A video vision transformer. In Proceedings of the IEEE/CVF, ICCV, October. 6816–6826.
[2]
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. 2021. Is space-time attention all you need for video understanding?. In Proceedings of the ICML, July, Virtual Event. 813–824.
[3]
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2014. Spectral networks and locally connected networks on graphs. In Proceedings of the 2nd International Conference on Learning Representations, ICLR 2014.
[4]
Kiana Calagari, Tarek Elgamal, Khaled M. Diab, Krzysztof Templin, Piotr Didyk, Wojciech Matusik, and Mohamed Hefeeda. 2016. Depth personalization and streaming of stereoscopic sports videos. ACM Trans. Multim. Comput. Commun. Appl. 12, 3 (2016), 41:1–41:23.
[5]
João Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In Proceedings of the IEEE CVPR, July. 4724–4733.
[6]
Baian Chen, Zhilei Chen, Xiaowei Hu, Jun Xu, Haoran Xie, Jing Qin, and Mingqiang Wei. 2024. Dynamic message propagation network for RGB-D and video salient object detection. ACM Trans. Multim. Comput. Commun. Appl. 20, 1 (2024), 18:1–18:21.
[7]
Xin Chen, Anqi Pang, Wei Yang, Yuexin Ma, Lan Xu, and Jingyi Yu. 2021. SportsCap: Monocular 3D human motion capture and fine-grained understanding in challenging sports videos. Int. J. Comput. Vis. 129, 10 (2021), 2846–2864.
[8]
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In Proceedings of the Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, Barcelona, Spain. 3837–3845.
[9]
Ali Diba, Mohsen Fayyaz, Vivek Sharma, Mohammad Mahdi Arzani, Rahman Yousefzadeh, Juergen Gall, and Luc Van Gool. 2018. Spatio-temporal channel correlation networks for action classification. In Proceedings of the ECCV, September. 299–315.
[10]
Ali Diba, Mohsen Fayyaz, Vivek Sharma, Amir Hossein Karami, Mohammad Mahdi Arzani, Rahman Yousefzadeh, and Luc Van Gool. 2017. Temporal 3D convnets: New architecture and transfer learning for video classification. arXiv:1711.08200. Retrieved from https://arxiv.org/abs/1711.08200
[11]
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. 2019. SlowFast Networks for Video Recognition. In Proceedings of the IEEE ICCV, October. 6201–6210.
[12]
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. 2020. Multi-modal transformer for video retrieval. In Proceedings of the ECCV, August. 214–229.
[13]
Hongyang Gao and Shuiwang Ji. 2022. Graph U-Nets. IEEE Trans. Pattern Anal. Mach. Intell. 44, 9 (2022), 4948–4960.
[14]
Zan Gao, Leming Guo, Tongwei Ren, An-An Liu, Zhi-Yong Cheng, and Shengyong Chen. 2022. Pairwise two-stream convnets for cross-domain action recognition with small data. IEEE Trans. Neural Networks Learn. Syst. 33, 3 (2022), 1147–1161.
[15]
Pei Geng, Xuequan Lu, Chunyu Hu, Hong Liu, and Lei Lyu. 2023. Focusing fine-grained action by self-attention-enhanced graph neural networks with contrastive learning. IEEE Trans. Circuits Syst. Video Technol. 33, 9 (2023), 4754–4768.
[16]
Rohit Girdhar, João Carreira, Carl Doersch, and Andrew Zisserman. 2019. Video action transformer network. In Proceedings of the IEEE CVPR, June. 244–253.
[17]
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. 2017. The “something something” video database for learning and evaluating visual common sense. In Proceedings of the IEEE ICCV, October. 5843–5851.
[18]
William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December. 1024–1034.
[19]
Dongliang He, Zhichao Zhou, Chuang Gan, Fu Li, Xiao Liu, Yandong Li, Limin Wang, and Shilei Wen. 2019. StNet: Local and global spatial-temporal modeling for action recognition. In Proceedings of the AAAI, February. 8401–8408.
[20]
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015. ActivityNet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE CVPR, June. 961–970.
[21]
Boyuan Jiang, Mengmeng Wang, Weihao Gan, Wei Wu, and Junjie Yan. 2019. STM: Spatiotemporal and motion encoding for action recognition. In Proceedings of the IEEE ICCV, October. 2000–2009.
[22]
Zequn Jie, Xiaodan Liang, Jiashi Feng, Xiaojie Jin, Wen Feng Lu, and Shuicheng Yan. 2016. Tree-structured reinforcement learning for sequential object localization. In Proceedings of the NurIPS, December. 127–135.
[23]
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of the ICLR, April.
[24]
Peng Lei and Sinisa Todorovic. 2018. Temporal deformable residual networks for action segmentation in videos. In Proceedings of the IEEE CVPR, June. 6742–6751.
[25]
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang, and Yu Qiao. 2023. UniFormerV2: Spatiotemporal learning by arming image vits with video uniformer. In Proceedings of the IEEE/CVF, ICCV, October. 5373–5382.
[26]
Tianjiao Li, Lin Geng Foo, Qiuhong Ke, Hossein Rahmani, Anran Wang, Jinghua Wang, and Jun Liu. 2022. Dynamic spatio-temporal specialization learning for fine-grained action recognition. In Proceedings of the ECCV. 386–403.
[27]
Xiangpeng Li, Jingkuan Song, Lianli Gao, Xianglong Liu, Wenbing Huang, Xiangnan He, and Chuang Gan. 2019. Beyond RNNs: Positional self-attention with co-attention for video question answering. In Proceedings of the AAAI, February. 8658–8665.
[28]
Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. 2020. TEA: Temporal excitation and aggregation for action recognition. In Proceedings of the IEEE CVPR, June. 906–915.
[29]
Yingwei Li, Yi Li, and Nuno Vasconcelos. 2018. RESOUND: Towards action recognition without representation bias. In Proceedings of the ECCV, September. 520–535.
[30]
Zhenyang Li, Kirill Gavrilyuk, Efstratios Gavves, Mihir Jain, and Cees G. M. Snoek. 2018. VideoLSTM convolves, attends and flows for action recognition. Comput. Vis. Image Underst. 166 (2018), 41–50.
[31]
Shuang Liang, Wentao Ma, and Chi Xie. 2024. Relation with free objects for action recognition. ACM Trans. Multim. Comput. Commun. Appl. 20, 2 (2024), 58:1–58:19.
[32]
Zhenming Liang, Yingping Huang, and Zhenwei Liu. 2022. Efficient graph attentional network for 3D object detection from frustum-based LiDAR point clouds. J. Vis. Commun. Image Represent. 89 (2022), 103667.
[33]
Ji Lin, Chuang Gan, and Song Han. 2019. TSM: Temporal shift module for efficient video understanding. In Proceedings of the IEEE ICCV, October. 7082–7092.
[34]
Chuanbin Liu, Hongtao Xie, Zheng-Jun Zha, Lingfeng Ma, Lingyun Yu, and Yongdong Zhang. 2020. Filtration and distillation: Enhancing region attention for fine-grained visual categorization. In Proceedings of the AAAI, February. 11555–11562.
[35]
Jian Liu, Naveed Akhtar, and Ajmal Mian. 2022. Adversarial attack on skeleton-based human action recognition. IEEE Trans. Neural Networks Learn. Syst. 33, 4 (2022), 1609–1622.
[36]
Tianyu Liu, Yujun Ma, Wenhan Yang, Wanting Ji, Ruili Wang, and Ping Jiang. 2022. Spatial-temporal interaction learning based two-stream network for action recognition. Inf. Sci. 606 (2022), 864–876.
[37]
Yongxu Liu, Jinjian Wu, Leida Li, Weisheng Dong, and Guangming Shi. 2023. Quality assessment of UGC videos based on decomposition and recomposition. IEEE Trans. Circuits Syst. Video Technol. 33, 3 (2023), 1043–1054.
[38]
Chenxu Luo and Alan L. Yuille. 2019. Grouped spatial-temporal aggregation for efficient action recognition. In Proceedings of the IEEE ICCV, October. 5511–5520.
[39]
Yujun Ma and Ruili Wang. 2024. Relative-position embedding based spatially and temporally decoupled transformer for action recognition. Pattern Recognit. 145 (2024), 109905.
[40]
Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. 2016. Learning convolutional neural networks for graphs. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016. 2014–2023.
[41]
Sen Qiu, Tianqi Fan, Junhan Jiang, Zhelong Wang, Yongzhen Wang, Junnan Xu, Tao Sun, and Nan Jiang. 2023. A novel two-level interactive action recognition model based on inertial data fusion. Inf. Sci. 633 (2023), 264–279.
[42]
Zhaofan Qiu, Ting Yao, and Tao Mei. 2017. Learning spatio-temporal representation with pseudo-3D residual networks. In Proceedings of the IEEE ICCV, October. 5534–5542.
[43]
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedings of the NurIPS, December. 91–99.
[44]
Fabien Ringeval, Björn W. Schuller, Michel F. Valstar, Jonathan Gratch, Roddy Cowie, and Maja Pantic. 2018. Introduction to the special section on multimedia computing and applications of socio-affective behaviors in the wild. ACM Trans. Multim. Comput. Commun. Appl. 14, 1s (2018), 25:1–25:2.
[45]
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2009. The graph neural network model. IEEE Trans. Neural Networks 20, 1 (2009), 61–80.
[46]
Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In Proceedings of the Semantic Web - 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7. 593–607.
[47]
Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. 2020. FineGym: A hierarchical video dataset for fine-grained action understanding. In Proceedings of the IEEE CVPR, June. 2613–2622.
[48]
Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. 2020. Intra- and inter-action understanding via temporal action parsing. In Proceedings of the IEEE CVPR, June. 727–736.
[49]
Lingyun Song, Jun Liu, Mingxuan Sun, and Xuequn Shang. 2021. Weakly supervised group mask network for object detection. Int. J. Comput. Vis. 129, 3 (2021), 681–702.
[50]
Jonathan C. Stroud, David A. Ross, Chen Sun, Jia Deng, and Rahul Sukthankar. 2020. D3D: Distilled 3D networks for video action recognition. In Proceedings of the IEEE WACV, March. 614–623.
[51]
Baoli Sun, Xinchen Ye, Tiantian Yan, Zhihui Wang, Haojie Li, and Zhiyong Wang. 2022. Fine-grained action recognition with robust motion representation decoupling and concentration. In Proceedings of the 30th ACM MM, Lisboa, Portugal, October. 4779–4788.
[52]
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. VideoBERT: A joint model for video and language representation learning. In Proceedings of the IEEE ICCV, October. 7463–7472.
[53]
Yuan Tian, Yichao Yan, Guangtao Zhai, Guodong Guo, and Zhiyong Gao. 2022. EAN: Event adaptive network for enhanced action recognition. Int. J. Comput. Vis. 130, 10 (2022), 2453–2471.
[54]
Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015. Learning spatiotemporal features with 3D convolutional networks. In Proceedings of the IEEE ICCV, December. 4489–4497.
[55]
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. 2018. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE CVPR, June. 6450–6459.
[56]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the NurIPS, December. 5998–6008.
[57]
Haoran Wang, Yajie Wang, Baosheng Yu, Yibing Zhan, Chunfeng Yuan, and Wankou Yang. 2024. Attentional composition networks for long-tailed human action recognition. ACM Trans. Multim. Comput. Commun. Appl. 20, 1 (2024), 8:1–8:18.
[58]
Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu. 2021. TDN: Temporal difference networks for efficient action recognition. In Proceedings of the IEEE, CVPR, June. 1895–1904.
[59]
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. 2016. Temporal segment networks: Towards good practices for deep action recognition. In Proceedings of the ECCV, October. 20–36.
[60]
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. 2022. BEVT: BERT pretraining of video transformers. In Proceedings of the IEEE/CVF, CVPR, June. 14713–14723.
[61]
Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He. 2018. Non-local neural networks. In Proceedings of the IEEE, CVPR, June. 7794–7803.
[62]
Zhuhui Wang, Shijie Wang, Haojie Li, Zhi Dou, and Jianjun Li. 2020. Graph-propagation based correlation learning for weakly supervised fine-grained image classification. In Proceedings of the AAAI, February. 12289–12296.
[63]
Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krähenbühl, and Ross B. Girshick. 2019. Long-term feature banks for detailed video understanding. In Proceedings of the IEEE CVPR, June. 284–293.
[64]
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. 2018. Rethinking spatiotemporal feature learning: Speed-accuracy tradeoffs in video classification. In Proceedings of the ECCV, September. 318–335.
[65]
Haotian Xu, Xiaobo Jin, Qiufeng Wang, Amir Hussain, and Kaizhu Huang. 2022. Exploiting attention-consistency loss for spatial-temporal stream action recognition. ACM Trans. Multim. Comput. Commun. Appl. 18, 2s (2022), 119:1–119:15.
[66]
Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. 2022. Multiview transformers for video recognition. In Proceedings of the IEEE/CVF, CVPR, June. 3323–3333.
[67]
Sijie Yan, Yuanjun Xiong, and Dahua Lin. 2018. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Proceedings of the AAAI, February. 7444–7452.
[68]
Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. 2020. Temporal pyramid network for action recognition. In Proceedings of the IEEE CVPR, June. 588–597.
[69]
Ze Yang, Tiange Luo, Dong Wang, Zhiqiang Hu, Jun Gao, and Liwei Wang. 2018. Learning to navigate for fine-grained classification. In Proceedings of the ECCV, September. 438–454.
[70]
Junbo Yin, Jianbing Shen, Xin Gao, David J. Crandall, and Ruigang Yang. 2023. Graph neural network and spatiotemporal transformer attention for 3D video object detection from point clouds. IEEE Trans. Pattern Anal. Mach. Intell. 45, 8 (2023), 9822–9835.
[71]
Bo Zhang, Rui Zhang, Niccoló Bisagno, Nicola Conci, Francesco G. B. De Natale, and Hongbo Liu. 2021. Where are they going? predicting human behaviors in crowded scenes. ACM Trans. Multim. Comput. Commun. Appl. 17, 4 (2021), 123:1–123:19.
[72]
Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. 2021. Temporal query networks for fine-grained video understanding. In Proceedings of the IEEE CVPR, June. 4486–4496.
[73]
Yanyi Zhang, Xinyu Li, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe. 2021. VidTr: Video transformer without convolutions. In Proceedings of the IEEE/CVF, ICCV, October. 13557–13567.
[74]
Zhong Zhang, Haijia Zhang, and Shuang Liu. 2021. Person Re-identification using heterogeneous local graph attention networks. In Proceedings of the IEEE CVPR, June. 12136–12145.
[75]
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. 2018. Temporal relational reasoning in videos. In Proceedings of the ECCV, September. 831–846.
Index Terms
- Discriminative Segment Focus Network for Fine-grained Video Action Recognition
Recommendations
Few-shot Fine-Grained Action Recognition via Bidirectional Attention and Contrastive Meta-Learning
MM '21: Proceedings of the 29th ACM International Conference on MultimediaFine-grained action recognition is attracting increasing attention due to the emerging demand of specific action understanding in real-world applications, whereas the data of rare fine-grained categories is very limited. Therefore, we propose the few-...
Periodic-Aware Network for Fine-Grained Action Recognition
Pattern Recognition and Computer VisionAbstractRecently, skeleton-based action recognition has gained increasing attention and achieved remarkable results in coarse-grained action recognition. Despite the positive results shown in these attempts, they are less effective in scenarios that ...
Human Action Recognition by Mining Discriminative Segment with Novel Skeleton Joint Feature
Advances in Multimedia Information Processing – PCM 2013AbstractIn this paper, we present a “key segment” mining approach for human action recognition. Our model is able to locate discriminative segments for action samples via multiple instance learning. Moreover, we propose a dynamic pooling approach to ...
Comments
Information & Contributors
Information
Published In
July 2024
973 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 15 May 2024
Online AM: 26 March 2024
Accepted: 20 March 2024
Revised: 24 February 2024
Received: 20 December 2023
Published in TOMM Volume 20, Issue 7
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 182Total Downloads
- Downloads (Last 12 months)182
- Downloads (Last 6 weeks)17
Reflects downloads up to 10 Nov 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in