Abstract
The advanced action recognition technologies have attempted to adopt multimodal architecture based on powerful pretrained models (e.g., visual-text) to achieve higher accuracy. However, visual encoders treat all frames equally, which may lead to (i) loss of key information and (ii) poor robustness, as different video frames may contribute differently to the visual representation. The preliminary experiments show that the assessment of frame features could be highly correlated with the classification accuracy of the visual encoder. Based on this insight, this paper proposes VFS (Video Frame Selection) module to guide the model in selecting video frames, thereby enhancing the action recognition accuracy. Specifically, VFS module selects diverse representative frames through processes including preprocessing, dimensionality reduction, and clustering etc. Additionally, we introduce soft prompt tuning for domain adaptation as advanced design. Experimental results show that the proposed method achieves performance comparable to or better than existing competitive action recognition models on multiple datasets.
This work is supported by Science and Technology Innovation 2030 - Major Project “New Generation Artificial Intelligence (2030)” (2022ZD0116407) and National Natural Science Foundation of China (62101552).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bertasius, G., Wang, H., Torresani, L.: Is space-time attention all you need for video understanding? In: ICML, vol. 2, p. 4 (2021)
Chen, M., Han, X., Zhang, H., Lin, G., Kamruzzaman, M.: Quality-guided key frames selection from video stream based on object detection. JVCIR 65, 102678 (2019)
Cheng, M., Cai, K., Li, M.: Rwf-2000: an open large scale video database for violence detection. In: ICPR, pp. 4183–4190. IEEE (2021)
Dosovitskiy, A., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
Elahi, G.M.E., Yang, Y.H.: Online learnable keyframe extraction in videos and its application with semantic word vector in action recognition. Pattern Recogn. 122, 108273 (2022)
Feichtenhofer, C.: X3d: Expanding architectures for efficient video recognition. In: CVPR, pp. 203–213 (2020)
Feichtenhofer, C., Fan, H., Malik, J., He, K.: Slowfast networks for video recognition. In: ICCV, pp. 6202–6211 (2019)
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: CVPR, pp. 16000–16009 (2022)
Jia, M., Tang, L., Chen, B.C., Cardie, C., Belongie, S., Hariharan, B., Lim, S.N.: Visual prompt tuning. In: ECCV, pp. 709–727. Springer (2022)
Joao, C., Andrew, Z.: Quo vadis, action recognition? a new model and the kinetics dataset. In: CVPR pp. 6299–6308 (2017)
Lester, B., Al-Rfou, R., Constant, N.: The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021)
Li, K., et al.: Uniformer: unified transformer for efficient spatiotemporal representation learning. arXiv preprint arXiv:2201.04676 (2022)
Liu, X., et al.: Gpt understands, too. AI Open (2023)
Luo, H., et al.: Clip4clip: an empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing 508, 293–304 (2022)
Mahasseni, B., Lam, M., Todorovic, S.: Unsupervised video summarization with adversarial LSTM networks. In: CVPR, pp. 202–211 (2017)
Moon, W., Hyun, S., Park, S., Park, D., Heo, J.P.: Query-dependent video representation for moment retrieval and highlight detection. In: CVPR, June 2023
Radford, A., et al.: Learning transferable visual models from natural language supervision. In: ICML, pp. 8748–8763. PMLR (2021)
Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. Advances in NeurIPS 27 (2014)
Sudhakaran, S., Lanz, O.: Learning to detect violent videos using convolutional long short-term memory. In: AVSS, pp. 1–6. IEEE (2017)
Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks. In: ICCV, pp. 4489–4497 (2015)
Vaswani, A., et al.: Attention is all you need. NeurIPS 30 (2017)
Wang, L., et al.: Temporal segment networks: towards good practices for deep action recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 20–36. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_2
Wang, M., Xing, J., Liu, Y.: Actionclip: a new paradigm for video action recognition. arXiv preprint arXiv:2109.08472 (2021)
Zolfaghari, M., Singh, K., Brox, T.: ECO: efficient convolutional network for online video understanding. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 713–730. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8_43
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zhu, Q. et al. (2025). A Decoupling Video Frame Selection Method for Action Recognition. In: Hadfi, R., Anthony, P., Sharma, A., Ito, T., Bai, Q. (eds) PRICAI 2024: Trends in Artificial Intelligence. PRICAI 2024. Lecture Notes in Computer Science(), vol 15283. Springer, Singapore. https://doi.org/10.1007/978-981-96-0122-6_39
Download citation
DOI: https://doi.org/10.1007/978-981-96-0122-6_39
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-96-0121-9
Online ISBN: 978-981-96-0122-6
eBook Packages: Computer ScienceComputer Science (R0)