Computer Science > Computer Vision and Pattern Recognition
[Submitted on 22 Jul 2018 (v1), revised 9 May 2019 (this version, v5), latest version 16 Dec 2019 (v6)]
Title:Correlation Net : spatio temporal multimodal deep learning
View PDFAbstract:This letter describes a network that is able to capture spatiotemporal correlations over arbitrary timestamps. The proposed scheme operates as a complementary, extended network over spatiotemporal regions. Recently, multimodal fusion has been extensively researched in deep learning. For action recognition, the spatial and temporal streams are vital components of deep Convolutional Neural Network (CNNs), but reducing the occurrence of overfitting and fusing these two streams remain open problems. The existing fusion approach is to average the two streams. To this end, we propose a correlation network with a Shannon fusion to learn a CNN that has already been trained. Long-range video may consist of spatiotemporal correlation over arbitrary times. This correlation can be captured using simple fully connected layers to form the correlation network. This is found to be complementary to the existing network fusion methods. We evaluate our approach on the UCF-101 and HMDB-51 datasets, and the resulting improvement in accuracy demonstrates the importance of multimodal correlation.
Submission history
From: Novanto Yudistira [view email][v1] Sun, 22 Jul 2018 14:48:32 UTC (718 KB)
[v2] Sat, 6 Oct 2018 06:59:46 UTC (354 KB)
[v3] Thu, 27 Dec 2018 05:01:31 UTC (910 KB)
[v4] Wed, 20 Mar 2019 10:28:43 UTC (313 KB)
[v5] Thu, 9 May 2019 01:40:45 UTC (830 KB)
[v6] Mon, 16 Dec 2019 06:57:10 UTC (1,424 KB)
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.