NJU MCG - Sensetime Team Submission to Pre-training for Video Understanding Challenge Track II
Authors: 
Liwei Jin, Haoyue Cheng, Su Xu, Wayne Wu, Limin WangAuthors Info & Claims
Liwei Jin, Haoyue Cheng, Su Xu, Wayne Wu, Limin WangAuthors Info & ClaimsPages 4799 - 4802
Abstract
This paper presents the method that underlies our submission to the Pre-training for Video Understanding Challenge Track II. We follow the basic pipeline of temporal segment networks [20] and further improve its performance in several aspects. Specifically, we use the latest transformer-based architectures, e.g., Swin Transformer, DeiT, CLIP-ViT, to enhance the representation power. We analyze different pre-training proxy tasks on the official pre-training datasets and other open-source video datasets. With these techniques, we derive an ensemble of deep models to attain a high classification accuracy (Top-1 accuracy 62.28%) on the testing set and secures first place in Track II of this challenge.
References
[1]
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. 2021 a. Facebook Timesformer. https://github.com/facebookresearch/TimeSformer
[2]
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. 2021 b. Is Space-Time Attention All You Need for Video Understanding? arXiv preprint arXiv:2102.05095 (2021).
[3]
MMAction2 Contributors. 2020. OpenMMLab's Next Generation Video Understanding Toolbox and Benchmark. https://github.com/open-mmlab/mmaction2.
[4]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et almbox. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).
[5]
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. 2019. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision. 6202--6211.
[6]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[7]
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et almbox. 2017. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 (2017).
[8]
Tianhao Li and Limin Wang. 2020. Learning spatiotemporal features via video and text pair discrimination. arXiv preprint arXiv:2001.05691 (2020).
[9]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021 a. Microsoft Swin Transformer. https://github.com/microsoft/Swin-Transformer
[10]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021 b. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030 (2021).
[11]
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. 2021 c. Microsoft Video Swin Transformer. https://github.com/SwinTransformer/Video-Swin-Transformer
[12]
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. 2021 d. Video Swin Transformer. arXiv preprint arXiv:2106.13230 (2021).
[13]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021 a. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 (2021).
[14]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et almbox. 2021 b. OpenAI CLIP. https://github.com/openai/CLIP
[15]
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2020. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12877 (2020).
[16]
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021. Facebook Deit. https://github.com/facebookresearch/deit
[17]
Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli. 2019. Video classification with channel-separated convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5552--5561.
[18]
Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu. 2021. TDN: Temporal difference networks for efficient action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1895--1904.
[19]
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. 2019. Temporal Segment Networks for Action Recognition in Videos. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 41, 11 (2019), 2740--2755.
[20]
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. 2016. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision. Springer, 20--36.
Index Terms
- NJU MCG - Sensetime Team Submission to Pre-training for Video Understanding Challenge Track II
Recommendations
MM21 Pre-training for Video Understanding Challenge: Video Captioning with Pretraining Techniques
MM '21: Proceedings of the 29th ACM International Conference on MultimediaThe quality of video representation directly decides the performance of video related tasks, for both understanding and generation. In this paper, we propose single-modality pretrained feature fusion technique which is composed of reasonable multi-view ...
Poster: Boosting Adversarial Robustness by Adversarial Pre-training
CCS '23: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications SecurityVision Transformer (ViT) shows superior performance on various tasks, but, similar to other deep learning techniques, it is vulnerable to adversarial attacks. Due to the differences between ViT and traditional CNNs, previous works designed new ...
A fast and efficient pre-training method based on layer-by-layer maximum discrimination for deep neural networks
In this paper, through extension of the present methods and based on error minimization, two fast and efficient layer-by-layer pre-training methods are proposed for initializing deep neural network (DNN) weights. Due to confrontation with a large number ...
Comments
Information & Contributors
Information
Published In
October 2021
5796 pages
ISBN:9781450386517
DOI:10.1145/3474085
- General Chairs:
- Heng Tao Shen,
- Yueting Zhuang,
- John R. Smith,
- Program Chairs:
- Yang Yang,
- Pablo Cesar,
- Florian Metze,
- Balakrishnan Prabhakaran
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 17 October 2021
Check for updates
Author Tags
Qualifiers
- Short-paper
Funding Sources
- National Natural Science Foundation of China
Conference
MM '21
Sponsor:
Acceptance Rates
Overall Acceptance Rate 995 of 4,171 submissions, 24%
Upcoming Conference
MM '24
- Sponsor:
- sigmm
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 124Total Downloads
- Downloads (Last 12 months)9
- Downloads (Last 6 weeks)1
Reflects downloads up to 18 Aug 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in