Detecting Temporal Proposal for Action Localization with Tree-structured Search Policy
Authors: 
Xinyang Jiang, Siliang Tang, Yang Yang, Zhou Zhao, Yin Zhang, Fei Wu, Yueting ZhuangAuthors Info & Claims
Xinyang Jiang, Siliang Tang, Yang Yang, Zhou Zhao, Yin Zhang, Fei Wu, Yueting ZhuangAuthors Info & ClaimsPages 1069 - 1077
Abstract
Understanding the semantics in videos is a complex but crucial task in video analysis. This paper focuses on localizing category-independent events, actions or other semantics in an untrimmed video, referred as salient temporal proposal localization. Traditional methods like sliding window have a high computational cost due to the densely sampling of different video segments. We propose a reinforcement learning based method, which trains a localizer that learns a search policy that, instead of exploring every video segment, finds an optimal search path to locate a salient proposal based on the currently observing video segment in a tree structure, therefore reduces the number of video segments fed into the proposal detector. In each search step, a localizer is trained to iteratively select the next sub-region containing salient proposals to continue the search, and a proposal detector is trained to recognize salient proposal from the sub-regions. The experiments demonstrate that our method is able to precisely detect salient proposals with a comparable recall and with much fewer candidate windows.
References
[1]
Miriam Bellver, Xavier Giro-i Nieto, Ferran Marques, and Jordi Torres. 2016. Hierarchical Object Detection with Deep Reinforcement Learning Deep Reinforcement Learning Workshop, NIPS.
[2]
Juan C Caicedo and Svetlana Lazebnik. 2015. Active object localization with deep reinforcement learning Proceedings of the IEEE International Conference on Computer Vision. 2488--2496.
[3]
Ming-Ming Cheng, Ziming Zhang, Wen-Yan Lin, and Philip Torr. 2014. BING: Binarized normed gradients for objectness estimation at 300fps Proceedings of the IEEE conference on computer vision and pattern recognition. 3286--3293.
[4]
jifeng dai, Yi Li, Kaiming He, and Jian Sun. 2016. R-FCN: Object Detection via Region-based Fully Convolutional Networks Advances in Neural Information Processing Systems 29. 379--387.
[5]
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation Proceedings of the IEEE conference on computer vision and pattern recognition. 580--587.
[6]
Georgia Gkioxari and Jitendra Malik. 2015. Finding action tubes Proceedings of the IEEE conference on computer vision and pattern recognition. 759--768.
[7]
F. C. Heilbron, V. Escorcia, B. Ghanem, and J. C. Niebles. 2015. ActivityNet: A large-scale video benchmark for human activity understanding 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 961--970.
[8]
Mihir Jain, Jan Van Gemert, Hervé Jégou, Patrick Bouthemy, and Cees GM Snoek. 2014. Action localization with tubelets from motion. In Proceedings of the IEEE conference on computer vision and pattern recognition. 740--747.
[9]
Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar. 2014. THUMOS Challenge: Action Recognition with a Large Number of Classes. http://crcv.ucf.edu/THUMOS14/. (2014).
[10]
Zequn Jie, Xiaodan Liang, Jiashi Feng, Xiaojie Jin, Wen Lu, and Shuicheng Yan. 2016. Tree-Structured Reinforcement Learning for Sequential Object Localization Advances in Neural Information Processing Systems. 127--135.
[11]
Stefan Mathe, Aleksis Pirinen, and Cristian Sminchisescu. 2016. Reinforcement learning for visual object detection Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2894--2902.
[12]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et almbox. 2015. Human-level control through deep reinforcement learning. Nature, Vol. 518, 7540 (2015), 529--533.
[13]
Dan Oneata, Jakob Verbeek, and Cordelia Schmid. 2013. Action and event recognition with fisher vectors on a compact feature set Proceedings of the IEEE international conference on computer vision. 1817--1824.
[14]
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks Advances in Neural Information Processing Systems 28. 91--99.
[15]
Zheng Shou, Dongang Wang, and Shih-Fu Chang. 2016. Temporal action localization in untrimmed videos via multi-stage cnns Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1049--1058.
[16]
Karen Simonyan and Andrew Zisserman. 2014. Two-Stream Convolutional Networks for Action Recognition in Videos. Advances in Neural Information Processing Systems 27. 568--576.
[17]
Chen Sun, Sanketh Shetty, Rahul Sukthankar, and Ram Nevatia. 2015. Temporal Localization of Fine-Grained Actions in Videos by Domain Transfer from Web Images Proceedings of the 23rd ACM International Conference on Multimedia (MM '15). 371--380.
[18]
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015. Learning spatiotemporal features with 3d convolutional networks Proceedings of the IEEE international conference on computer vision. 4489--4497.
[19]
J. R. R. Uijlings, K. E. A. van de Sande, T. Gevers, and A. W. M. Smeulders. 2013. Selective Search for Object Recognition. International Journal of Computer Vision Vol. 104, 2 (2013), 154--171.
[20]
Heng Wang and Cordelia Schmid. 2013. Action recognition with improved trajectories. In Proceedings of the IEEE international conference on computer vision. 3551--3558.
[21]
Zhongwen Xu, Yi Yang, and Alex G Hauptmann. 2015. A discriminative CNN video representation for event detection Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1798--1807.
[22]
Jimei Yang and Ming-Hsuan Yang. 2012. Top-down visual saliency via joint CRF and dictionary learning Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. 2296--2303.
[23]
Serena Yeung, Olga Russakovsky, Ning Jin, Mykhaylo Andriluka, Greg Mori, and Li Fei-Fei. 2015. Every moment counts: Dense detailed labeling of actions in complex videos. arXiv preprint arXiv:1507.05738 (2015).
[24]
Serena Yeung, Olga Russakovsky, Greg Mori, and Li Fei-Fei. 2016. End-to-end learning of action detection from frame glimpses in videos Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2678--2687.
Index Terms
- Detecting Temporal Proposal for Action Localization with Tree-structured Search Policy
Recommendations
Temporal Localization of Actions with Actoms
We address the problem of localizing actions, such as opening a door, in hours of challenging video data. We propose a model based on a sequence of atomic action units, termed "actoms," that are semantically meaningful and characteristic for the action. ...
Temporal-visual proposal graph network for temporal action detection
AbstractTemporal action detection is usually divided into two stages: temporal action proposal generation and proposal classification. Most methods consider the proposal classification stage as an action recognition task. However, compared with trimmed ...
CTAP: Complementary Temporal Action Proposal Generation
Computer Vision – ECCV 2018AbstractTemporal action proposal generation is an important task, akin to object proposals, temporal action proposals are intended to capture “clips” or temporal intervals in videos that are likely to contain an action. Previous methods can be divided to ...
Comments
Information & Contributors
Information
Published In
October 2017
2028 pages
ISBN:9781450349062
DOI:10.1145/3123266
- General Chairs:
- Qiong Liu,
- Rainer Lienhart,
- Haohong Wang,
- Program Chairs:
- Sheng-Wei "Kuan-Ta" Chen,
- Susanne Boll,
- Phoebe Chen,
- Gerald Friedland,
- Jia Li,
- Shuicheng Yan
Copyright © 2017 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 19 October 2017
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- NSFC
- Chinese Knowledge Center of Engineering Science and Technology
- Qianjiang Talents Program of Zhejiang Province 2015
- Key program of Zhejiang Province
- 973 program
Conference
MM '17
Sponsor:
Acceptance Rates
MM '17 Paper Acceptance Rate 189 of 684 submissions, 28%;
Overall Acceptance Rate 995 of 4,171 submissions, 24%
Upcoming Conference
MM '24
- Sponsor:
- sigmm
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 274Total Downloads
- Downloads (Last 12 months)2
- Downloads (Last 6 weeks)1
Reflects downloads up to 15 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in