Combining Link and Content Correlation Learning for Cross-Modal Retrieval in Social Multimedia

Abstract

With the rapid growth of multimedia data, cross-modal retrieval has received great attention. Generally, learning semantics correlation is the primary solution for eliminating heterogeneous gap between modalities. Existing approaches usually focus on modeling cross-modal correlation and category correlation, which can’t capture semantic correlation thoroughly for social multimedia data. In fact, the diverse link information is complementary to provide rich hints for semantic correlation. In this paper, we propose a novel cross-modal correlation learning approach based on subspace learning by taking heterogeneous social link and content information into account. Both intra-modal and inter-modal correlation are simultaneously considered through explicitly modeling link information. Additionally, those correlations are incorporated into final representation, which further improve the performance of cross modal retrieval effectively. Experimental results demonstrate that the proposed approach performs better comparing with several state-of-the-art cross-modal correlation learning approaches.