Title:Top-down Visual Saliency Guided by Captions

Abstract:Top-down saliency methods based on deep neural networks have recently been proposed for task-driven visual search. However existing methods focus on object or scene classification tasks and cannot be used to compute saliency heatmaps using a natural language sentence as the top-down input. Current state-of-the-art image and video captioning models can generate accurate sentence captions but are difficult to understand, as they do not expose the internal process by which spatial and temporal regions are mapped to predicted words. In this paper, we expose this mapping and demonstrate that spatio-temporal saliency is learned implicitly by the combination of CNN and LSTM parts of modern encoder-decoder networks. Our approach, which we call Caption-Guided Visual Saliency, can produce spatial or spatio-temporal heatmaps for both given input sentences or sentences predicted by our model. Unlike recent efforts that introduce explicit "attention" layers to selectively attend to certain inputs while generating each word, our approach recovers saliency without the overhead of explicit attention layers, and can be used to analyze a variety of existing model architectures and improve their design. We evaluate our approach on large scale video and image datasets and make several interesting discoveries about the inner workings of captioning models. The source code is publicly available at this http URL.